Picking brains with Project Tango Augmented Reality.
Fantasy ideas for Augmented and Virtual Reality have never been in short supply, but our job is to make them work, so we’ve been spending time with Google’s Project Tango to get to grips with the practical applications.
In simple terms Project Tango allows an Android device to tell where it is in a room, which way it’s looking and to scan its surroundings in real-time 3D. The obvious place to start was to make simple demo to give an augmented tour of the Dootrix office and to pick the team’s brains. Here’s a video showing it in use.
How does it work?
Before filming the walk-around, we scanned the area so that the device could recognize the Dootrix Office. The scanning process involves walking around the area and looking in different directions for a couple of minutes. This builds up a picture of the surrounding visual features that make the space distinct.
Informed by an area definition file, a Project Tango app can now use localisation to understand not just how it’s moving around in space, but whereabouts in space the real world that space actually is. For example, it wouldn’t just know that you’re stood somewhere and have turned left, it would know that you’re standing two feet north of the kitchen door and have turned to the left.
The app combines the camera feed with rendered virtual elements to create an augmented reality experience. For demo purposes we chose to place in some interactive brains representing our staff, and a simple animated UI, but this could be much more sophisticated in a fully developed app. In this case, we recorded the relative location of where each member of staff sits in the office, and then position the virtual objects in those locations. As we walk around, these objects stay fixed in their real-world location.
What else could we make with it?
The long-view of possible applications is of course huge, but we’re looking at what can we build next week. Our demo process is a little like GPS-tagging locations, only the resolution is much finer and it works indoors. This makes it ideal for either guiding users around a known space, or making that space interactive. Imagine an interactive museum exhibit for example, or an in-store navigation aid where you search for a product and it draws the route you need to take to locate it.
In addition to the features we’re showing off here, Tango is also capable of understanding the depth of its surroundings. This could allow for features such as measuring or recording the rough shape of surfaces and objects.
These kind of useable and useful applications could be up and running in months rather than years, and we’re already applying these ideas in our client work.
If you think that this technology could be useful for you, please do get in touch to see how we could help you. If you would like to know more about the nuts and bolts behind our Project Tango demo, please read on.
How did we make the app?
The app was made in two parts in Unity with the Project Tango SDK.
First of all, a tool was made to record the locations of the staff relative to the scanned area. This involved simply walking round the area and tapping on each member of staff’s keyboard in space. The device then measured its current position and the angle and depth to that point in space. From there we can calculate the point in space that was tapped on.
The second part of the app was the display; the actual tour. This involved creating brains at the recorded locations, and presenting some basic data about each person. This part of the app was fairly conventional in function, only presented in an augmented reality context.
Motion tracking is the tracking of a device’s motion with respect to its surroundings. In this case ‘motion’ refers to both movement in terms of a changing location in space as well as a change in pose; that is the angle in which the device is ‘looking’.
With a normal mobile device approximate motion tracking is possible based on readings from the GPS, Gyrometer and the compass.
These readings allow a rough outdoors positioning (5m accuracy), a bearing (if calibrated) and relative rotation. GPS is widely used and very useful, but it has coarse accuracy and only works outdoors. The combination of a gyrometer and compass allow reasonable tracking of motion, but they lack a calibrated starting point of reference. They might be able to say you have rotated ‘this much’, but not necessarily say where you were to begin with.
The Project Tango sensors instead use visual-inertial odometry (VIO), where the device also analyses how the imagery it is viewing is changing. Essentially telling where it is not just with the usual sensors, but also based on what it can see. Images are analysed for features, which give an indication of not just the general pattern of what’s in front of the device, but also the distance. With this richer input the accuracy is much higher.
Motion Tracking Use Cases
The big use case for motion tracking (used in isolation from other Tango features) is virtual reality (VR) controllers. This means the device can be used to control a camera pose or 3D point selection with a full six degrees of freedom (6DoF). The device becomes a sort of 3D mouse, allowing natural input of position and rotation. Imagine how much more natural it would be for an everyday user to navigate a three-dimensional space and view an object from any chosen angle by walking around and looking rather than learning the confusing controls of a 3D modelling program.
Area Learning and Localisation
Project Tango area learning is a combination of mapping and localisation. Mapping is the construction and updating a map of an area, comprising a number of visual features. Localisation is the ability to determine one’s location within an area.
Whilst pure motion-tracking solutions suffer from ‘drift’, with inaccuracies increasing over time, localisation events mean that the device recognises its location; correcting its area-relative coordinates as necessary via a ‘loop closure’. In practice this means that a Project Tango device is more accurate in knowing its location than conventional means, and also remains accurate over time.
Combining area learning and localisation in real-time is a complex problem, known as Simultaneous Localisation And Mapping (SLAM). Thankfully Google do the hard work in the SDK, meaning developers don’t have to be experts in this area to benefit from this feature. Complications such as objects moving around after an area is scanned can be effectively ignored in most circumstances.
Area Learning and Localisation Use Cases
Area learning and localisation opens up possibilities such as guiding a user around a known location (indoor navigation), placing augmented reality objects at a specific location in the real world and recalling them in the same location later, and also increased accuracy for motion tracking.
Depth Sensing and Meshing
Depth sensing is the ability to ‘see’ depth; a technique also used by devices such as the Kinect, Leap Motion and RealSense. This is achieved by shining an invisible infra-red (IR) mesh into the environment and then capturing the IR light that is reflected back.
From any given perspective, a point cloud can be generated, which is a map of the depth for the pixels on the screen (albeit at a lower resolution). This means that it is possible to tell not just the X and Y coordinates of what the device is looking at, but also the Z. This means that a Tango device can look at an object and know exactly where it is and how big it is. This is something we’re used to as humans, but previous devices normally lack the context required in order to understand this.
Previous infra-red-sensing devices such as Kinect usually do not move, and so retain the same single perspective of their environment. As Tango devices are mobile, they can be moved around over time, building up a three-dimensional picture of their surroundings. This can be extended to a technique known as meshing, whereby the known points of depth are connected with a three-dimensional mesh, providing a form of object scanning.
Depth Sensing and Meshing Use Cases
Being able to understand the location and size of what is observed is perfect for augmented reality. Applications such as home furniture planners can now be based on your actual home rather than a generic model. The scanning aspect also gives potential for free-form mobile object scanning, so you could scan your cat and then 3D print it, for example.
Where are we going with this?
At Dootrix we specialise in making complex data and information easy to use, so anything that can help to simplify a potentially complicated user experience is incredibly interesting when you are designing mobile apps. The idea of being able to place access to specific information in very specific locations has almost unlimited possibilities for businesses of every kind.
The Project Tango apps we are experimenting with today don’t require new and expensive hardware like VR headsets – the next generation of Android phones or tablets are all that’s required – meaning that radically more efficient working processes could be deployed quickly and relatively cheaply.
Subscribe to our newsletter for free advice delivered to your inbox on a fortnightly basis.
This month’s most useful tools for developers — September 2017
Some are about new thinking or new techniques, some are about old or legacy technology and some are just useful resources for different languages or platforms. The brief is simple — if […]