Workflow for creating High Definition DivX compressed video from the output of the JVC JY-HD10/GR-HD1 camcorder Transport Stream MPEG-2
Goal: Sports video image quality without compromise
Investigating methods of compressing HD material to practical sizes has shown excellent results using DivX. See DivX website here. I understand the DivX standard to be similar to MPEG4. Until we see a solid emergence of the HD DVD standard, and encoding standard and tools, this is a great way to save current HD material. Using DivX I have gotten 40 minutes of material to fit in a 2.3Gbyte file, compressing at 6Mbit/sec.
A standard definition DVD uses MPEG-2, and has a max data rate of about 8Mbit/sec. This means 60 min of material, at max rate, creates a file of roughly 3.6Gbyte. I say roughly as I'm not allowing for audio data. A standard DVD holds about 4.25Gbyte of data.
HD video files, 720p format as downloaded from the JVC JY-HD10 camera, are in MPEG-2 Transport Stream format. They are encoded at about 18Mbit/sec video rate. This format means 60 min of material takes about 9Gbyte of space. Finding a better compression method, without quality loss, is very valuable.
Here are the before and after results of going to DivX.
The .mpg file should play in Windows Media Player. The DivX material requires either the DivX Player or a Windows codec. The DivX player is available here, as a free download. Try playing them side by side, pause and compare the image quality.
I started out this process intending to use Windows Media Encoder, creating .wmv files. However the required bitrate to preserve excellent HD quality seemed quite high. I found my way to DivX, and a tool chain based on VirtualDub. There are four stages to the tool chain, but the result is very functional. Here are the tools, with pointers to each:
||a video processing utility. Not intended for editing, rather for fast linear operations over video.
||codec (compressor/decompressor) for high quality video. An alternative to MPEG or WMV.
||a script driven frameserver, expanding a variety of video input types to uncompressed video, without using temporary file space.
|DGMPGDec, DGIndex, DGDecode, v1.4.0
||A plug-in for AVISynth, expanding MPEG-2 Transport Stream files.
||A plug-in for AVISynth, accepting .mpa audio as generated by DGIndex.
||http://www.avisynth.org/warpenterprises, search down the list for MPASource.
Frameserving video - processing uncompressed HD without huge file sizes
I can't justify all the portions of the following. I can just tell you how it works.
VirtualDub uses the DivX codec to generate the end result - .divx encoded high definition video, at a selectable bitrate. .divx files can be played by the DivX Player, or by Windows Media Player with a suitable codec add-on. The story would be simple here, but VirtualDub requires uncompressed video in a .avi file as input. Converting HD to uncompressed .avi creates files too large in size, and too great in expansion time, to be practical. Hence we see a technology called "frameserving" in common use for this problem. It consists of somehow (I'm not sure what happens here) using one program to feed frames, on demand, to another program. The consuming program thinks it is reading from a .avi file. The consuming program can read forward or backwards in the file, and the creating program can respond with exactly the frames requested. The creating program is building the uncompressed .avi frames as needed, without storing them in any intermediate (huge) file.
AviSynth is the frameserver. It in turn requires plug-ins which interpret the actual source file format, in this case Transport Stream MPEG-2 (.m2t) and audio (.mpa).
The procedure is basically to fire up VirtualDub, and open a special input file, which causes AviSynth to activate, and AviSynth in turn loads 2 plug-ins, and is prepared to read the .m2t source files. VirtualDub is used to specify the DivX codec compression parameters, and carry out 2 or 3 pass compression. Prior to starting VirtualDub, DGIDecode needs an index file, and MPASource needs an audio file. These two files are created by a short indexing process, by DGIndex.
The special file is named <anything>.avs, and is actually a short text file containing information for AviSynth about the plug-ins needed. This file has to be manually created in Notepad.
Here is a pictorial of the process.
Here is another walkthrough, which may better explain the process than I do below, here
Step by step procedure in WindowsXP: .m2t to .divx
VirtualDub installs simply as a set of files in a directory. It does not create a Windows Start menu item. So put a shortcut to VirtualDub.exe on your desktop.
Install AVISynth in a normal Windows fashion.
DivX installs in a normal Windows fashion.
DGMPGDec installs simply as a set of files in a directory. Copy and paste DGDecode.dll into the AVISynth subdirectory called plugins\. Create a desktop shortcut for DGIndex.exe.
MPASource installs simply as a set of files, of which you only need to extract from the .zip: mpasouce.dll (and the .avs files as examples). Disregard all the other source code files.
Test your installation
Make a test directory, to contain initially a few .m2t files. Use Windows Notepad to create this initial test.avs file for testing:
That's it, just one line of text. To make Notepad give it the proper extension, be sure to use Save-As and set Save_as_type to AllFiles. It must have a .avs extension. Start VirtualDub and File-Open_Video_File, and specify this test.avs file. VirtualDub should show a "video" which is just the version info for AVISynth. Now you know AVISynth is working!
Now we will fire up DGIndex to look at the .m2t file(s). Start DGIndex (Note 1 in pictorial above). Verify these settings:
-Audio, OutputMethod, is Demux_all_tracks.
-Audio, TrackNumber, is 1.
-Video, FieldOperation, is HonorPulldownFlags.
-Stream, SetPIDS, Video is: 1000
-Stream, setPIDS, Audio is: 1002
Now File, Open, and set Files_of_type to AllFiles, and select one .m2t file. You will get a FileList menu which enables ADDing more .m2t files, and ordering them. This concatenates multiple .m2t files into one effective file. OK when you have 1 or more .m2ts showing in the list.
Now you should see the video, and dragging the slider across the bottom should move smoothly through all the .m2t files as though it were one file.
Now do File, Play and see an information window open which tells you detailed information about the .m2t. Hit ESC to stop playback. You will not hear any audio here, but the Information window should show Audio, Track1, "MPEG Audio".
If you are not seeing information filled in for Video and Audio, then probably something is wrong with the PID settings. The values I show are for the JVC JY-HD10.
Now do File, Save_Project, and give it a name such as test, allowing the default extention to be .d2v. A Pentium 3GHz CPU can do 1 hour of .m2t in about 2 minutes.
Exit DGIndex and notice 2 new files created: test.d2v and test_PID_1002_DELAY_xxms.mpa. The first file is a textual content file. The second is the audio in an MPEG format. I normally rename the audio file to something shorter like: test-xxms.mpa (because I don't like spaces in filenames).
Now drag and drop the test.avs file onto Notepad and edit it to look like this (Note 2 in pictorial above):
(five lines of text as above). This is a script file, informing AVISynth of the plug-ins (that we put in the plugin\ directory during installation), and the video and audio input files. DGDecode.dll will read the .d2v and find the indexing information there for video from the set of .m2t files. mpasource.dll will read the .mpa file. Save again as test.avs
Start VirtualDub again and File, Open_Video_File, and select test.avs. Now you should see your first framserved frame! Click on VirtualDub <right triangle 1> at the bottom of the window and you should see the video playback with audio!
Compressing output with DivX
Next we will use the above video and compress it to DivX in VirtualDub. Do the steps above and get to the point of seeing your video + audio.
Next is to configure Divx parameters and do 2 passes of compression. In VirtualDub (Note 3 in pictorial above), select Video, Compression. You should see SelectVideoCompression with a list of codecs including DivX 6.0 Codec. Highlight DivX 6.0 Codec, and click Configure.
You should get a DivX Codec Properties window. In the CertificationProfile pulldown, select Unconstrained. Under BitRate, select "Multipass, 1st pass", then enter a value of 6000 kbit/s. It won't let you enter this large a value unless you have Unconstrained. Now OK, and OK from the Select window.
In VirtualDub, File, Save_as_AVI. Be sure you choose this one, as there are many save options. Give it a name, and I recommend changing the Save_as_type to AllFiles and selecting a filename suffix as: test.divx
You can let it take the default .avi suffix, but I find this confusing as .avi files can hold many different codec encodings. I prefer to make the .divx explicit, plus the DivX Player will be selected when you double click this file.
VirtualDub will do a first pass, which could take an hour for 1 hour of source material. You will see 2 progress windows.
When the first pass finishes, the progress windows disappear. Now do VirtualDub, Video, Compression, and Configure (the DivX 6.0 Codec), and set EncodeMode to Multipass, Nth pass. OK, OK.
File, Save_as_AVI, and select the same output filename as before, and confirm you want to replace it. The same progress windows appear, but now Pass shows as Pass 2.
When this finishes, I sometimes do another pass identiical to the above paragraph, to create a 3rd pass. If you do this, then select the output filename to be slightly different, so you can compare and decide if you need 2 or 3 passes. (to compare, double click VirtualDub again, to get 2 windows open, and OpenVideoFile the .divx files and compare them frame by frame at interesting points.
Double click the .divx file and see the DivX Player plays them! You have done it. Achieved video quality without compromise.
Each time you run DGIndex on a new set of .m2t files, remember to manually edit the xxx.avs file to put in the new filenames. You can keep comments in this file by starting a line with #.
I have picked the value of 6Mbits with some experimentation. Comparing side by side outputs, to my eye this bitrate retains the quality of the original HD material. You should try other, probably smaller, values. Sports video is challenging, with all the motion involved.
Please drop me a note with your preferred HD compression environment. DivX or otherwise! To richalt2 at yahoo dot com.