Production grade AR: Fake it ’till they make it

Development of AR is in full swing, with AR Kit & Core pushing the technology ahead. They have freed us from requiring predefined markers, and therefore opened up a new realm of possibilities. And though highly impressive, the technology is not without its limitations. At CREATE.eu, we had the following use case: a shiny, copper-paint-coated model of the North Sea Port with relatively little unique surface detail, surrounded by four iPads which would have to augment it with layers of information. Since this is a permanent installation, the AR layer has to be perfectly aligned all day, every day, without any intervention from the staff or from us. All these things are something that the consumer-facing AR technology we have available today is not particularly good at.

We did not want to take away from the stately presence of the Medieval guild house by resorting to sticking visible markers on the model, especially in an age where these are no longer socially acceptable. So instead, we designed our own solution. I started by experimenting with two Vive trackers – small devices which can determine their position and orientation very accurately with the help of two base stations, the same system used for the VR headset. The idea was simple enough: by sticking one tracker on the model and one tracker on the iPad, we could determine the relative camera position for the AR layer.

This worked quite well! Unfortunately though, the Vive trackers themselves turned out to be notoriously unreliable. They go into standby after not moving for a short time even when plugged in, often lose connection to the PC, and getting them connected again requires luck and patience in equal measure. Back to the drawing board.

The alternative solution sounded simple in theory: since the iPads would be on fixed mounts around the model and would only be rotating around one axis, it would suffice to know the angle at which the iPad is pointing, measured using a rotary encoder, to be able to position the AR camera. I was however wary of this solution: it requires knowing the exact position of the iPad’s camera in its resting position, and then being able to correctly rotate the virtual camera in line with the real camera. After much experimentation (and some desperation), it turned out to be impossible to get the virtual and physical camera aligned over the entire arc of rotation. There were too many imperfections in the mounts that we could not account for, and a few millimeters offset in the camera position results in a very large discrepancy 2 or 3 meters away.

Discussing this problem with my girlfriend over dinner, she gave me the idea of calibration, in the same way she would have to calibrate a camera during material laboratory testing. I figured I could just save the camera position that marker based AR computed, along with the angle the rotary encoder was at. This turned out to be a good approach: we would be able to have the absolute precision of old fashioned marker based AR, without the users ever having to see the thing. I believe that’s what they call “having your cake and eating it too”.

Close up of iPad with AR layer over model
Photo by Davy Hoskens

While we now had perfectly aligned stationary views, a problem remained when rotating the tablets. The AR layer would visibly lag behind.

It took quite a bit of digging to eliminate this issue. First, I had to make sure that the AR layer would only update in time with the physical camera. Then I improved the code for reading the rotary encoder’s angle so that it’s faster and more reliable. My colleagues gave the local network a hardware upgrade to ensure communication speed between the encoder PCs and the tablets wasn’t the issue. And finally, I added movement prediction to the virtual camera: while you’re rotating the tablet, the rotation of the virtual camera is sped up in function of the speed at which the tablet is being rotated. These changes combined resulted in a very satisfying, fully production grade AR layer.