Using Deshaker Image Stabilization for Bicycle Video
|
You really want to perform some video processing functions only once -- for example, inverting upside-down video; image stabilization; resizing. The updated version of the freeware Windows video processing application VirtualDub, (now in enhanced, updated versionhttps://sourceforge.net/projects/vdfiltermod/) is lean and fast. By preprocessing in VirtualDub and saving the resulting file, you avoid having your main video editing suite perform the same processing again and again as it renders video.
There is another article on this site about VirtualDub. Please read it in connection with this article.
VirtualDub performs most functions using filters. One VirtualDub filter, Deshaker, is the best image stabilization software I have encountered. VirtualDub is supplied with a number of filters by default, but Deshaker is an add-on. You get it from the Deshaker Web page (at http://www.guthspot.se/video/deshaker.htm), and install it in the VirtualDub folder on your computer. Like VirtualDub, Deshaker is freeware, but the Web site requests a donation. I paid. There is good documentation on the site.
VirtualDub and Deshaker are available in 32-bit and 64-bit versions. The 64-bit version runs significantly faster, and so I recommend it, if your computer supports it. Some filters are available only in 32-bits, so you may want to keep a 32-bit version of VirtualDub as well.
Filters are the top item in VirtualDub's Video menu. When you click on that item, the filter dialog box will open. Click on "Add" to see the list of available filters. Then click on a filter in the list to add it. Filter information will appear in the filter dialog box. You may run more than one filter at a time, though certain combinations are not compatible. Think of the filter list as a sort of pipeline, working in order from top to bottom of the list. So, for example, you may invert the video image, and adjust color in one pass through VirtualDub.
Adding a filter in VirtualDub
There is another article on this site about image stabilization in general. While excellent documentation is available on the Deshaker site, here are some specifics relating to bicycle video.
Bicycle video is going to be shaky, whether you shoot it with a helmet camera or with a camera mounted on the bicycle. There will be small, rapid shakes and also wide sweeps from a turn of the head, or of the bicycle.
Unless you have a camera with very sophisticated built-in stabilization such as the GoPro 10 or 11, you will want to use stabilization in post-processing. The stabilization in earlier cameras such as my GitII and Sony HDR-AS100V can leave some shake in the image; Deshaker cleans that up.
Stabilization works very well with a camera mounted on the rider -- typically atop the helmet. The rider's body filters out the very rapid shakes which Deshaker cannot correct.
Shoot at the highest resolution your camera, editing suite and tolerance for processing time can support, to avoid much loss of sharpness. I typically shoot at 1080 x 1920. There is not much noticeable loss of quality with the slight enlargement of the image which I use to void black borders in Deshaker, displaying at the same resolution. Display at a lower resolution will avoid any loss of sharpness -- at that resolution. Shoot progressive video. Most action cameras do. Interlaced video doesn't process as well and should preferably be deinterlaced before running Deshaker -- see our article about deinterlacing. .
Deshaker runs two passes through the video file, analyzing it on the first pass and adjusting it on the second. In this way, Deshaker can look ahead in time, preventing it from confusing panning with shaking. Single-pass image stabilization can run to the end of its range, resulting in jerky motion during wide pans, and that is one reason in-camera stabilization can be unsatisfactory. .
The screen shot below is of the Deshaker interface, with my preferred settings for pass 1 and 1920 x 1080 HD video from a forward-facing helmet camera with a fisheye lens. A few settings (pixel aspect ratio, scan type, rolling shutter setting) are specific to the camera. Because of the fisheye lens, stabilization is different at the margins of the image. I use the "Ignore Pixels" setting to adjust stabilization for the center of the image. Ignoring the margins of the image also increases processing speed on Pass 1.
On Pass1, I select "Run Video Analysis Pass" in VirtualDub's File menu, and "Uncompressed RGB/YCbCr" in the Video menu. On this pass, Deshaker will save a log file, and VirtualDub will not save a video file. Still, processing will run much faster without a compressor -- more than twice as fast as with the XVid MPEG-4 compressor. Evidently, VirtualDub sends the video to the compressor for processing even if it isn't saving a file.
The rolling shutter setting is important. A camera with a rolling shutter scans the image from top to bottom and it will stretch or shrink the image if panning upward or downward. Deshaker corrects for this if the rolling shutter setting is correct. The Deshaker Web site gives instructions on how to test a camera's rolling shutter amount. I have more details about rolling shutter later in this article.
For each video that you stabilize, save a log file under a different name, so you can re-use it with different stabilization settings in Deshaker's pass 2. The video file must be the same geometrically and frame-by-frame to work with the same log file.
Processing on the second pass can be slow. In 1920 x 1080 HD resolution, on a computer with an Intel i7-3770 processor and using the XVid MPEG-4 codec, it's about 7.5 frames per second, so it takes about 4 times as long as the clip ran. Still, I prefer to have the entire stabilized video file to work with, and so I let my computer run Deshaker when I'm working on something else. My newer computer with an AMD 3900X processor runs Deshaker about 4 times as fast -- pretty much at the real-time display rate for 1920 x 1080 video.
Deshaker's default motion-smoothness values of 1000 produce floating-on-a cloud-smooth pans. For bicycle video, I recommend lower values, around 150 for horizontal and vertical panning and rotation. These settings will eliminate the rapid shake that is usual on a bicycle, while decentering the image less and causing minimal geometric distortion in images shot with a fisheye lens. I set motion smoothness for zooming to zero, to avoid "ballooning" of the image.
Uncompressed video files are so huge (5 GB per minute!) that you will quickly run out of disk space. These files won't play at normal speed, even from a fast hard drive. So, on the second pass, select "Save as AVI" (or another compressed format), and a compressor in the file menu. The free XVid MPEG-4 compressor is my favorite. If the original file is in any format other than AVI, you can save the stabilized version using the same filename and the different filetype. Keeping the original file, log file and processed file in the same folder (or the processed file in a parallel folder on a different hard drive, for less drive wear) makes them easy to find.
VirtualDub2 can save a file in several different formats. Choose the one that works for you.
It usually makes more sense, all in all, to run other processes after Deshaker. The exception would be if you are correcting for geometric distortions which affect the operation of Deshaker itself -- or if your camera scans from bottom to top. Then you need to invert the image before Pass 1,then invert it before and restore it after Pass 2..
As the image shifts position to remove shake, black border areas will appear unless you use one of the several tricks that Deshaker offers.
One is adaptive zoom, which enlarges the image to push the border areas off the screen. If adaptive zoom is turned on in Deshaker, the image will "balloon" in size whenever there is a big shake. If the camera is facing forward or rearward, this can make it look as if you are suddenly speeding up or slowing down on your bicycle -- even riding backwards.
Another option is to use earlier and later frames to fill in the border areas. This doesn't work perfectly with a moving camera. Images of objects closer to the camera change size faster, and so motion forward or backward results in poor matches where the other images fill in. Still, the fill works well enough that it's worth doing even if the results aren't perfect. The transitions between the current and other images are less distracting if you set Deshaker to create soft borders. If the camera is looking sideways, you can get a good match if the margins of the image are all at the same distance (for example, a wall).
Filling in borders using neighboring frames will work poorly if you run the geometric correction before Deshaker. With a rotated image, for example, the fill will be only at the corners, where the image reaches the edges of the frame.
On the second pass, setting motion smoothness for zooming to zero eliminates the "ballooning" effect. Setting 150 for horizontal, vertical and rotational motion smoothness (or 300 at 50 or 60 frames per second) will eliminate annoying rapid shake and produce a result more like your perceived head motion as you are riding -- while minimizing decentering of the image.
If the camera stays close to the same position and turns to look in different directions you can use the stabilization options more freely. Stabilized images can then look as smooth as if the camera is on a tripod.
An extra zoom factor of about 1.1 will keep border areas and fill out of the picture almost all the time with the motion-smoothness settings I recommend. As Deshaker itself resizes the image, there will be only one re-encoding. In case black borders remain, you may remove them later by zooming the image in your editing suite. It's time-consuming, but you can make better artistic decisions than Deshaker can, recentering the image if necessary.
I recommend that you have Deshaker look backward and forward the full 30 frames to fill in the border areas. When these areas fill in seamlessly, you have a larger image area to work with.
It is possible to play some interesting tricks in Pass 2. Setting Motion Smoothness for panning, zoom or rotation to -1 in pass 2 results in "infinite" smoothness: the position or rotation remains the same as in the first frame. (There can be a very slow drift, which you may have to correct in post-processing.) One use for this is to stabilize the image if the camera is on a tripod but the tripod is disturbed. I had a (non-bicycling) video where the camera was unintentionally zoomed in, then back out. All of the motion, fortunately, was in the zoomed-in area. Saving this segment of video as a file and setting Zoom to -1 kept the scale of the image constant. Fill using neighboring frames avoided black borders for the most part. Cropping the processed image and overlaying it on a still from the unzoomed section produced a result with only the slightest visible artifacting!
Another trick is to create 3D video from 2d video shot from a camera in motion. For this, you want zero vertical panning and very smooth horizontal panning. Video shot out the side window of an aircraft, for example, can produce hyperstereo of the landscape below, so it looks like a table-top model.
Many cameras have a "rolling shutter", which scans the image from top to bottom, rather than capturing it all at once. The image will then appear stretched vertically if the camera is panning upwards quickly, squished if the camera is panning downward and skewed if it is panning sideways.. If Deshaker knows the amount of rolling shutter for your camera, it can compensate for this, though it cannot compensate for very rapid shake which occurs during the scan of each video frame.
The Deshaker Web site lists rolling-shutter values for a number of cameras. I have contributed measurements for my cameras:
I also determined the settings for an old POV action camera, now defunct, by measuring from a video clip with rapid panning. The default 88% rolling shutter value works well, but this camera scans from bottom to top. Till I figured that out, results were terrible!
The Deshaker site describes a special test procedure to measure the rolling shutter value, but you can calculate it without having to do a special test, using a video clip with a rapid pan across a building or other object that has horizontal and vertical lines at right angles. This is common enough in helmet-camera videos, when the cyclist turns the head. If the camera has a rolling shutter, the object will appear skewed. Stop the image on the screen, so you can advance it a frame at a time, just as with the test described on the Deshaker Web site. Lay a straightedge -- the edge of a sheet of paper will do -- along a vertical edge which appears skewed in the image. Extend the line upward and downward to measure the horizontal positions at the top and bottom of the screen.
Checking for rolling shutter without having to do a special test shoot.
The camera was panning and so the buildings are skewed.
Draw a right angle across a horizontal line in the image you are measuring from, and carry it to the top and bottom of the screen to simulate the locations of the top and bottom of a still frame. A sheet of paper conveniently has square corners and can be used to derive the vertical line from the horizontal. The vertical and horizontal lines should cross near the center of the screen, to avoid geometric errors, especially with a fisheye lens.
Rolling-shutter correction is linear from top to bottom of the frame, and so a slight jitter may remain near the top and bottom if there was rapid shake.
Deshaker can produce astonishingly good results with a conventional lens -- one which images straight lines as straight lines. Telephoto and "normal" lenses are of this type, as are some wide-angle lenses. These lenses project an accurate geometric representation of the subject onto the flat sensor in the camera. This kind of lens cannot look behind itself, or extremely far out to the sides.
A fisheye image is easy to recognize: straight lines curve like parentheses near edges of the image. A fisheye lens can cover a wider angle, up to 180 degrees and sometimes even more. Because a fisheye lens looks in very different directions at the same time and compresses the image at the edges, shake will be different there.
There is a lot of image area out at the edges, and Deshaker takes an average when calculating how to reposition the image to stabilize it. Including the edges in the sensed area when using a fisheye lens will leave the center of the image shaky -- though the center is usually the main area of interest. With a fisheye lens, I recommend tracking an area only about half as high and wide as the image when running Deshaker's pass 1, the analysis pass. (For example, if you are shooting in 1080 x 1920 HD, you might exclude everything outside 270 pixels from the top and bottom, and 480 pixels from the right and left. If the horizon is high in your picture as it often is in a bicycle video, place the sensed rectangle closer to the top.) Some visible shake will still occur in your final result, but it will mostly be near the edges of the image. You may want to use a smaller sensing area if you are going to crop the image. Processing in pass 1 will be faster with a smaller analyzed area.
In a fisheye image with rapid panning,, a stabilized image can become "wavy" when panning direction reverses. This issue is nearly eliminated with the low motion-smoothness settings I recommend. Another way to minimize distortion is to crop the image after stabilizing it, but the best way of all is the old-fashioned way, to avoid shake as much as possible in the first place!
If a camera has a non-fisheye wide-angle lens (that is, if straight lines remain straight at the edges of the image)., objects appear larger near the edges of the image, and image geometry changes noticeably as the camera pans or shakes wildly. With a normal or telephoto lens, though, it's better to sense the entire image, giving Deshaker more to work with.
Deshaker's option to have the analyzed area follow motion in the image can be useful if objects are moving through the image and making it hard for Deshaker to track the stationary background. However, if the camera pans, this setting can move the analyzed area completely out of the image. If using this setting, monitor the display in Deshaker's Pass 1 to make sure that you haven't defeated stabilization in this way.
Deshaker numbers the frames in the log file and so you have the option to run more than one log file on the same video, in case you need to use different settings in tricky places, or make manual adjustments. This can be time-consuming, but if you are a perfectionist, there you go.
I describe how to use a camera's internal stabilization followed up by Deshaker, in my review of the Sony HDR-AS100V action camera.
A 360-degree camera such as the Garmin VIRB 360 needs to use its own image stabilization and/or the image needs to be cropped before running Deshaker. Deshaker cannot stabilize the entire 360-degree image, as the camera is looking in every direction at once.
In my experience, Deshaker running in VirtualDub sometimes inserts several blank frames at the start of the video, and shifts the audio track by a few frames. To address this issue, I import the complete, unedited file which VirtuaDub has produced into my editing suite, Pinnacle Studio Ultimate, detach the audio, placing it in a separate track, and then shift the start of that track to where it needs to be. Any good editing suite should be able to do this. Often, placing the start of the audio at the first non-blank frame of video will realign them. I use the two tracks together from that point onward. It is also possible to resynchronize the audio and video using a cue such as a hand clap, but unless you have intentionally created a cue, you may have to go hunting for one. We have an article covering that topic.
The image below shows the start of a one-minute-long clip as displayed in the timeline of Pinnacle Studio Ultimate 20. The display will be similar in any multi-track video editing suite.
The image above shows the end of the clips. The processed clip is missing the last several frames. The audio extends farther than shows with the original clip, but in fact, this audio was recorded!
So, if you are recording a clip on which you will use Deshaker, record a second or two extra at the start and end. And if you didn't do that, append extra frames before processing, so none of your original clip is lost.
The clip here was shot using a Samsung Galaxy S4 Mini Android smartphone. Oddly, VirtualDub running Deshaker identified the frame rate as 31.48 frames per second. I got jerky motion when I inserted the processed clip into Pinnacle Studio. AVS4YOU Video Converter correctly identified the clip as running at 30 frames per second (well, actually, 29.999, but close enough) and by first copying the clip using that software, I got a good result. I still had to shift the video and audio, though.