Blue and yellow purple hills: spatially augmented sandbox

We all played in the sand as kids. Be it in a sandbox in the back yard or at school, but preferably at the beach. We’d build mighty castles dominating the surrounding sandscape from atop their hill, lovingly decorated with shells. Or we’d dig intricate canal systems, ready to be flooded by the oncoming tide. In our minds, these were more than just mere piles of sand. They were lush green foothills, crowned by keeps made of stone and plaster, with shingled roofs and fluttering banners piercing the sky!

At least, usually that’s what we saw. Sometimes you’re in an environment where your imagination could do with a helping hand, such as when you’re at a museum teaching you about geography. In this case, a projector can work some magic.

For the museum “De wereld van Kina” (the world of Kina) in Ghent, I created a spatially augmented sandbox for my work at CREATE.

In an exhibition about earth quakes and minerals, the museum directors wanted the young visitors to be able to build their own volcanoes. I realized this idea using a projector, a Kinect, Unity and OpenCV. The Kinect provides the required depth information. OpenCV is used to find local maxima through blob detection, to be able to locate hills in the sandbox and pinpoint their peak. The Unity application does all the rest: render the topographical visuals based on the depth data, provide projection mapping to be able to properly align the projector and the Kinect to the sandbox, and show lava, fire and smoke when the volcano erupts. My suggestion to install a subwoofer under the sandbox to simulate rumbling was sadly cut due to budget constraints.

The projection mapping was done using the tool I wrote for projection mapping shows in Unity. I extended it to enable calibration of the depth info: on a second monitor, the raw depth output of the Kinect is displayed. On it, you can move the UV coordinates of the mesh corresponding to the sandbox, essentially telling the application where in the Kinect’s field of view the sandbox is located.

Getting correct depth data from the Kinect turned out to be a bit trickier than anticipated: the Kinect’s depth stream is a 16 bit integer per grayscale pixel, but there’s no built-in texture format that matches this – directly reading the depth stream into a texture ends up wrapping the depth info.

Wrapping depth when not reading the Kinect's depth stream correctly
Wrapping depth when not reading the Kinect’s depth stream correctly (image source)

Obviously, using the depth stream in this manner would not result in a continuous topological relief. I had to remap the 16 bit depth stream into something Unity could use. All I’d have to do to get a smooth grayscale gradient is turn each 16 bit integer into a floating point value, which could then be used as a texture in Unity. However, dividing 307200 (640×480) integers each frame is a killer on performance. To offload this work from the CPU, I wrote a compute shader to do this operation instead – the GPU is a piece of hardware far better suited to processing large amounts of data.

Direct use of Kinect depth leads to noisy results

A second caveat to Kinect usage is the very noisy depth info. In the gif above, you can see how the water is jittering – this is the direct output of the Kinect’s depth stream. In the final version of the application seen in the video, I opted to take a snapshot of the depth info every second, and crossfade to it. This delayed depth stream would be used for the water only, since the landscape has to be updated right away for a satisfying user experience. Smoothing out the water like this has the added benefit of fake water dynamics – the water gradually flows into a newly dug trench, much like it did in the canals we used to dig at the beach all those years ago.

Featured image source: Kina’s Facebook page