Overview

In some sense, computer graphics and computer vision are inverses of one another. Special purpose computer vision hardware is rarely found in typical mass-produced personal computers, but graphics processing units (GPUs) found on most personal computers, often exceed (in number of transistors as well as in compute power) the capabilities of the Central Processing Unit (CPU).

The present research involves implementing computer vision algorithms on modern computer graphics cards. This work investigates methods of efficiently mapping mathematical operations of computer vision onto modern computer graphics architecture.

As an example computer vision algorithm, a real--time projective camera motion tracking routine has been implemented on a modern, GeForce FX class GPU. This was done using using OpenGL and the nVIDIA Cg fragment shaders. Trade--offs between computer vision requirements and GPU resources were examined such as floating point accuracy, high framerate thoroughput and latency. The algorithm implementation was examined closely, and hardware bottlenecks were discovered and addressed. The performance of the GPU architecture for computer vision was examined and it was demonstrated that significant speedups can be achieved, while leaving the CPU free for other signal processing tasks.

Applications of our work include wearable, computer mediated reality systems that use both computer vision and computer graphics, and require realtime processing with low--latency and high throughput provided by modern GPUs. This ability to run many algorithms in real--time is an important step in creating a working mediated reality system, and allows others to easily build low cost systems capable of real--time computer vision.

void FragmentProgram(
    in float2 fptexCoord0 : TEXCOORD0,
    in float2 fptexCoord1 : TEXCOORD1,
    out float4 colorO       : COLOR0,
    const uniform samplerRECT FPE0,
    const uniform samplerRECT FPE1 )
{
   int i=0;
   float2 orig_coord;
   float2 new_coord, delta;

   float kappa =  -0.15;
   float2 center = {-0.05, 0.0}; //if known, focal point should be used

   //normalize coord [0,1.0]
   orig_coord =  (fptexCoord0)/float2( 320.0, 240.0);
   //shift coords [-0.5,0.5]
   orig_coord = orig_coord - float2(0.5,0.5);

   float2 radius = orig_coord - center;
   float2 r = distance(orig_coord, center);

   delta = orig_coord*kappa*r*r;
   new_coord = (fptexCoord0)/float2( 320.0, 240.0) + delta;

   colorO = texRECT(FPE1, new_coord*float2(320.0, 240.0));

}