Augmented reality in mobile applications

Recently our team was working on a tourist app that enhances the user experience of sightseeing. In some ways app was like all other tourist apps – it includes basic information about attractions and activities that can be found near sightseeings, list of upcoming events and so on. Since building mediocre apps isn’t our thing, we were thinking on how to improve the experience of sightseeing. We came up with the idea of using augmented reality for showing information about sightseeing. Before speaking about the specifics of this let’s first talk about what augmented reality is.

Introduction to Augmented reality (AR)

In basic words, augmented reality displays real-world objects that are processed and altered by the computer. For example, we could create 3D objects and displayed it in a specific place in the real world by placing tracker object or using phone’s GPS. The image below shows one of the most popular AR app. (image from macrumors.com)

Pokemon game in augmented reality

For more details on the augmented reality check this Wikipedia page.

How we built and Augmented reality app

Ok, and it’s cool stuff, but how does it work?
For augmented reality to work, you need a camera, accelerometer, rotation sensor and some math skills and GPS. The idea behind all of this is to use data from the real world, data gathered by the sensors, and process it in the way computer can use.
From the camera, we get information about view perspective – vertical and horizontal field of view, angles, and direction in which phone is facing. For calculating where your object should be placed you can use some sort of detection objects like placeholders or rely on sensor data like GPS. In our app, we wanted to use geolocation for displaying augmented objects so we went with using sensors instead of placeholders.
For displaying objects based on location GPS and accelerometer and rotation sensors are mandatory. Other sensors like gyroscope, magnetic field, and pressure sensors can be used to suppress sensor jitter and make your positioning more accurate by removing redundant sensor values.

Check our case study Plitvice Lakes

Finding proper object location

Most challenging part of making an AR is the math part. To be fair, it isn’t a rocket science but isn’t the elementary stuff either. Most math work can be done with integrated math function, you just need to make sure that you provide the right sensor data. The difficult part is deducing how to transform your sensor data into position on the screen. For that, you should be familiar with how 3D projection works and what azimuth is. Let’s explain this briefly on the object that is represented by a single dot. You need to calculate the distance between your phone(GPS location) and object (your dot) and find a bearing to that location. After you know the bearing (azimuth) you can calculate a location of an object on the screen. To make location on screen accurate you should take into account height difference. That problem was tricky to solve because every solution we found had a problem. We tried using GPS, Google API, and barometer sensor. Problem with GPS was that it is very inaccurate. Problem with getting height from Google API is that it’s only free for the limited amount of requests, which was a problem for us because we didn’t know how much client base would we have and what pricing plan would be good for us. The last solution was getting data from the barometer sensor which offered okey result but only a few phones had a barometer sensor. After a while, we decided to implement a GPS and barometer solution. If barometer sensor is available he will provide height, and if not, GPS will provide height.
Great, we finally have something to work with, an object location. But this “dot” doesn’t do much good for our sightseeing app. To make use of it we have created simple “Marker” that shows the name of location and distance to location and enables the user to open a single detail page for that location.

Improvements

This was our first attempt to build an AR, and we think that we have laid a good foundation for the future versions. In the future version, we will work on more accurate height value by implementing Google API paid solution or finding a new solution. Furthermore, we are keen on improving user experience by finding new ways of displaying information.