Using graphics cards for modifying video
Nowadays there is another reason for buying the latest generation of graphics cards than just for playing games. These cards can be used to manipulate high resolution images and videos in real-time. They make it possible to change the colors and applying a sharpening algorithm on the image slightly, while receiving instant feedback about the changes performed on a movie while playing it. The technique is already available, but to use it to actually perform the manipulations, the software controlling the graphics cards has to be enhanced.
Image manipulation software like Adobe’s Photoshop is wide-spread. It can be used to enhance the appearance of an image, for example by adjusting its colors and contrast. When changing the appearance it is important to get feedback fast, because that makes it easier to obtain the desired result. Just think of painting with a pencil on paper with a delay of a second before what is drawn appears on the paper. It would limit the precision and control. All modifications would have to be done in small steps to see what has been done.
The need for fast feedback applies to video as well. But in this case fast feedback is even more important compared to still images. If a running video fails to run smoothly, it can give a false impression of the modifications that have been made. To run a video smoothly, the video has to run at the video´s frame rate. In Europe (using PAL/SECAM) this is 25 images per second, and in North America (using NTSC) it is 30 images per second.
Graphics cards are extremely fast at doing parallel calculations for each pixel on the screen. These calculations are traditionally used for enhancing 3D calculations. The newest graphics cards are programmable (using pixel and vertex shaders), which means that they do not only have a small limited set of possible calculations. Now it is possible to do more complex effects that previously have required the flexibility of processors. In order to use the graphics cards for applying effects or algorithms, they have to be rewritten. This is because graphics cards have other instructions than processors. 
The actual code for utilizing the graphics card can be written once, as a general module that can be used in programming frameworks like DirectShow or Quicktime. This makes it easy to change existing programs or develop new ones that uses programmable graphics cards. Then the developers only has to rewrite the effects. A general module that can use programmable graphics cards for display purposes already exists in the latest version of DirectShow.
I have developed a DirectShow module that can be used to manipulate video using the graphics card. A program that uses this module was also developed to show how it could be done. This shows that the technique already is available.
For visualization, the method already works. The graphics card producer ATI has released a demo program that can run many effects simultaneously on high resolution clips without affecting the frame rate. It shows that it is possible to enhance the performance compared to using the processor for applying effects on a video stream. 
Due to slow software drivers, it is not yet possible to use programmable graphics cards for doing the actual rendering in real-time. This is because it requires an extra step compared to just displaying the images on screen, namely sending the image back to the computer’s main memory. The drivers have been enhanced a lot lately, making it plausible that the transfer speed from the graphics card to the main memory will soon be high enough for doing the modification without using the processor.
Using the graphics card for better performance will probably soon be used in photo and video manipulation software where there are demands on fast response times. The technique of modifying and analyzing images in real-time can be useful in many other areas, however. It could for example be used to enhance videos in medical applications or in surveillance cameras.

Klas Skogmar

This article is part of the master thesis “Real-time Video Effects Using Programmable Graphics Cards” from the Departement of Computer Science at Lund Institute of Technology