Game Development Reference
can produce a colored surface and take full advantage of the Kinect's abilities.
Theprocessfordoingthisrequiresustoknow how to take a depth pixel, convert
it to world space, and then determine where that point would project to in the
color image. This is more or less converting from the depth camera's coordinate
system to the color camera's coordinate system.
We have already discussed most of the required concepts to do this. The
conversion from depth-image pixel to world space is already described above. We
also know the final step in the process, which is to project from world coordinates
to the color-image coordinates. However, we need to know what is different about
how the world space is perceived by each of the cameras. In Figure 2.1 we can
see that the depth and color cameras are offset from one another. This means
that an object that appears in the center of the depth-image stream will appear
off-center in the color-image stream. Our mapping process requires us to know
how to compensate for this offset—we want to know where in the color image our
depth pixels will end up!
This process is actually quite common in computer vision, and it is typically
referred to as stereo calibration [Bradski and Kaehler 08] and is used to find cor-
responding points in two cameras. The process itself can be somewhat complex,
although there are fairly good examples available in open-source libraries. How-
ever, the hard work of performing this calibration is already performed for us by
the Kinect SDK. A set of functions are provided to map between depth-image
coordinates and color-image coordinates that uses the factory calibration of the
Kinect. This is a very useful set of tools that can handle most of the heavy lifting
for us. We will use these functions in our example application later in this article.
Programming with the Kinect SDK
At this point, we are now ready to see in more detail how we can acquire the
Kinect data streams and use them in an example application. This section will
cover how to initialize the runtime and start acquiring data. Next we will discuss
how to map that data into a Direct3D 11 resource and finally how to use and
interpret those resources for a visualization of the Kinect data. The example
program has been written with the Hieroglyph 3 [Hieroglyph 3 12] open-source
Direct3D 11 framework. The framework, along with the example program and
utility code, is available for download from its Codeplex project page. The gen-
eral interfacing to the Kinect is available in the GlyphKinect project, while the
example application is available in the KinectPlayground project.
To enable your application to receive information from the Kinect, you must
first install the Kinect for Windows SDK. This SDK is freely available from Mi-
crosoft [Microsoft 12] and is provided with an installer that takes care of setting
up the drivers and the build environment for the Kinect (there are good instal-
lation instructions available with the SDK, so we won't repeat them here).