Creating a 3D topographical map in your HoloLens / Windows MR app with the Bing Maps Elevation API

Intro

All right, my friends. It's time for the real deal, the blog post I have been anticipating to write since I first published the 3D version of Walk the World like two months ago. It took me a while to extract the minimal understandable code from my rather convoluted app. Everyone who has ever embarked on a trip of 'exploratory programming' (meaning you have an idea what you want to do, but only some vague clues how) knows how you end up with a repo (and code) full of failed experiments, side tracks, etc. So I had to clean that up a little first. Also, my app does a lot more than just show the map, and those features would obscure the general idea. As a bonus - after creating this blog post I finally actually understand myself how and most importantly why the app works :).

So, without further ado, I am going to show you how to display a 3D map in your HoloLens or Windows MR headset. Just like in Walk the World. I will build upon my previous post, in which I showed you how to make a flat slippy map. This time, we are going 3D.

The general idea

As you can read in the previous post, I 'paste' the actual map tiles - mere images - on a Unity3D Plane. A Plane is a so-called mesh that exists out of a grid of 11x11 points, that form the vertices. If I somehow would be able to ascertain actual elevation on those locations, I can move those points up and down and actually get a 3D map. The tile itself will be stretched up and down.The idea of manipulating the insides of a mesh, which turns out to be very simple, is explained by the awesome Rick Bazarra in the first episode of his must-see "Unity Strikes Back" explanatory video series on YouTube, a follow-up to his Creative Coding with Unity series on Channel 9, that I consider a standard starting point for everyone who wants to get off the ground with Unity3D.

So where do we get those elevations? Enter the awesome Microsoft service called the Bing Maps Elevation API. It seems to be built-to-order for this task. Your first order of business - get yourself a Basic Bing Maps key.

Adding some geo-intelligence to the tile

The Bing Maps Elevation API documentation describes an endpoint GetElevations that allows you to get altitudes in several ways. One of them is a grid of altitudes in a bounding box. That is what we want - our tiles are square. The documentation says the bounding box should be specified as follows:

"A bounding box defined as a set of WGS84 latitudes and longitudes in the following order:
south latitude, west longitude, north latitude, east longitude"

If you envision a tile positioned so that north is up, we are required to calculate the geographical location of the top-right and bottom-left of the tile. The Open Street Maps Wiki provides code for the north-west corner of the tile, i.e. top left. I translated the code to C#...

//http://wiki.openstreetmap.org/wiki/Slippy_map_tilenames#C.23
private WorldCoordinate GetNorthWestLocation(int tileX, int tileY, int zoomLevel)
{
    var p = new WorldCoordinate();
    var n = Math.Pow(2.0, zoomLevel);
    p.Lon = (float)(tileX / n * 360.0 - 180.0);
    var latRad = Math.Atan(Math.Sinh(Math.PI * (1 - 2 * tileY / n)));
    p.Lat = (float) (latRad * 180.0 / Math.PI);
    return p;
}

... and it works fine, but we need the south west and the north east points. Well, if you consider how the tiles are stacked, you can easily see how the north east point is the north-west point of the tile right of our current tile, and the south-west point is the north west point of the tile below our tile. Therefore we can use the north-west points of those adjacent tiles to find the values we actually need - like this:

public WorldCoordinate GetNorthEast()
{
    return GetNorthWestLocation(X+1, Y, ZoomLevel);
}

public WorldCoordinate GetSouthWest()
{
    return GetNorthWestLocation(X, Y+1, ZoomLevel);
}

That was easy, right?

Size matters

Although Microsoft is a USA company, it fortunately has an international orientation so the Bing Maps Elevation API returns no yards, feet, inches, miles, furlongs, stadia, or any other deprecated distance unit – it returns plain old meters. Which is very fortunate, as the Windows Mixed Reality distance unit is – oh joy – meters too. But it returns elevation in real world values, and while it might be fun to show Kilimanjaro in real height, it will be a bit too big to fit in my room (or any room, for what matters). Open Street Map is shown at a definite scale per zoom level – and as a GIS guy, I like to be the height correctly scaled - to get for this real life feeling. Once again, referring to the Open Street Map Wiki – there is a nice table that shows how many meters a pixel is at any given zoom level. We will need the size per tile (which is 256 pixels, as I explained in the previous post), so we add the following code that will give you a scale factor for the available zoom levels for Open Street Map:

//http://wiki.openstreetmap.org/wiki/Zoom_levels
private static readonly float[] _zoomScales =
{
    156412f, 78206f, 39103f, 19551f, 9776f, 4888f, 2444f,
    1222f, 610.984f, 305.492f, 152.746f, 76.373f, 38.187f,
    19.093f, 9.547f, 4.773f, 2.387f, 1.193f, 0.596f, 0.298f
};

private const int MapPixelSize = 256;

public float ScaleFactor
{
    get { return _zoomScales[ZoomLevel] * MapPixelSize; }
}

Creating the request

Now we move to MapTile. We add the following code to download the Bing Maps Elevation API values

private string _mapToken = "your-map-token-here";

public bool IsDownloading { get; private set; }

private WWW _downloader;

private void StartLoadElevationDataFromWeb()
{
    if (_tileData == null)
    {
        return;
    }
    var northEast = _tileData.GetNorthEast();
    var southWest = _tileData.GetSouthWest();

    var urlData = string.Format(
    "http://dev.virtualearth.net/REST/v1/Elevation/Bounds?bounds={0},{1},{2},{3}&rows=11&cols=11&key={4}",
     southWest.Lat, southWest.Lon, northEast.Lat, northEast.Lon, _mapToken);
    _downloader = new WWW(urlData);
    IsDownloading = true;
}

This simply queries the TileInfo structure for the new methods we have just created. Notice it then builds the URL, containing the bounds, the hard coded 11x11 points that are in a Unity Plane, and the key. Then it calls a piece of Unity3D code called “WWW”  which is a sort of HttpClient named by someone with a lot of fantasy (NOT). And that’s it. We add a call to the existing SetTileData method like this:

public void SetTileData(TileInfo tiledata, bool forceReload = false)
{
    if (_tileData == null || !_tileData.Equals(tiledata) || forceReload)
    {
        TileData = tiledata;
        StartLoadElevationDataFromWeb();
    }
}

so that whenever tile data is supplied, it does not only initiate the downloading of the tile, but also the downloading of the 3D data.

Processing the 3D data

Next up is a method ProcessElevationDataFromWeb, that is called from Update (so about 60 times a second). In this method we check if a the MapTile is downloading – and if it’s ready, we process the data

protected override void OnUpdate()
{
    ProcessElevationDataFromWeb();
}

private void ProcessElevationDataFromWeb()
{
    if (TileData == null || _downloader == null)
    {
        return;
    }

    if (IsDownloading && _downloader.isDone)
    {
        IsDownloading = false;
        var elevationData = JsonUtility.FromJson<ElevationResult>(_downloader.text);
        if (elevationData == null)
        {
            return;
        }

        ApplyElevationData(elevationData);
    }
}

An ElevationResult is a class to deserialize a result from a call to the Bing Maps Elevation API in. I entered the result of a manual call in Json2CSharp and got a class structure back – only I changed all properties into public fields so the rather stupid limited Unity JsonUtility, that does not seem to understand the concept of properties, can handle it. I also initialized lists in the objects from the constructors. It’s not very interesting but if you want a look go here in the demo project.

Applying the 3D data.

So now it’s time to actually move the mesh points up an down. Mostly using code I stole from Rick Bazarra, with a few adaptions from me:

private void ApplyElevationData(ElevationResult elevationData)
{
    var threeDScale = TileData.ScaleFactor;

    var resource = elevationData.resourceSets[0].resources[0];

    var verts = new List<Vector3>();
    var mesh = GetComponent<MeshFilter>().mesh;
    for (var i = 0; i < mesh.vertexCount; i++)
    {
        var newPos = mesh.vertices[i];
        newPos.y = resource.elevations[i] / threeDScale;
        verts.Add(newPos);
    }
    RebuildMesh(mesh, verts);
}

private void RebuildMesh(Mesh mesh, List<Vector3> verts)
{
    mesh.SetVertices(verts);
    mesh.RecalculateNormals();
    mesh.RecalculateBounds();
    DestroyImmediate(gameObject.GetComponent<MeshCollider>());
    var meshCollider = gameObject.AddComponent<MeshCollider>();
    meshCollider.sharedMesh = mesh;
}

First we get the scale factor – that’s simply the value by which elevation data must be divided to make it match the current zoom level. Next, we get the elevation data itself, that is two levels down in de ElevationData. And then we go modify the elevation of the mesh points to match those of the elevation we got. For some reason - and that's why I said it looks like the Bing Maps Elevation API looks to be like built-to-order for this task - the points come in at exactly the right order for Unity to process in the mesh.

As I learned from Rick, you cannot modify the points of a mesh, you have to replace them. So we loop through the mesh points and fill a list with points that have their y – so the vertical direction – changed to a scaled value of the elevation. Then we call RebuildMesh, that simply replaces the entire mesh with new vertices, does some recalculation and rebuilds the collider, so your gaze cursor will actually play nice with the new mesh. I also noticed that it you don’t do the recalculate stuff, you will end up looking partly through tiles. I am sure people with a deeper understanding of Unity3D will understand why. I just found out that it needs to be done.

Don't press play yet! There a few tiny things left to do, to make the result look good.

Setting the right location and material

First of all, the map is kind of shiny, which was more or less okay-ish for the flat map, but if you turn the map into 3D you will get this over bright effect. So open up the project in Unity, create a new material “MapMaterial” and apply the properties as displayed here below left. The color of the material should be #BABABAFF. See left image. When it is done, drag it on top of the MapTile (see right image).

Then, the app is still looking at Redmond. While that’s an awesome place, there isn’t much spectacular to see as far as geography is concerned. So we mosey over to the MapBuilder script. There we change the zoom level to 14, the Latitude to 46.78403 and the Longitude to -121.7543

It's a little east and quite a bit more south from Redmond. In fact, when you press play, you will see a landmark that is very familiar if you live anywhere near the Seattle area or visited it:

famous Mount Rainier, the volcano sitting about 100 km from Seattle, very prominently visible from aircraft - weather permitting. To get this view, I had to fiddle with the view control keys a little after pressing play - if you press play initially you will see Rainier from a lot more close up.

And that, my friends, is how you make a 3D map in your HoloLens. Of almost any place in the world. Want to see Kilimanjaro? Change Latitude to -3.21508 and Longitude to 37.37316. Press play.

Niagra falls? Latitude 43.07306, Longitude -79.07561 and change zoom level to 17. Rotate the view forward a little with your mouse and pull back. You have to look down. But then, here you go.

GIS is cool, 3D GIS is ultra-cool! All that is left is to generate the UWP app and deploy it into your HoloLens or view it in your new Windows Mixed Reality device.

Caveat emptor

Awesome right?  Now there are a few things to consider. In my previous post I said this app was a bandwidth hog, as it downloads 169 tiles per map. In addition, it now also fires 169 requests per map to the Bing Maps Elevation API. For. Every. Single. Map. Every time. Apart from the bandwidth and power consequences, there's another thing to consider. If you go to this page and click "Basic Key", you will see something like this:

What is boils down to is - if your app is anywhere near successful, it will eat your allotted request limit very fast, you will get a mail from the Bing Team kindly informing you of this (been there, got that) - and then suddenly you will have a 2D app again. You will have to buy and enterprise key and those are not cheap. So I advise you to do some caching - both in the app and if possible on an Azure service. I employed a Redis cache to that extent.

Furthermore, I explained I calculate the north east and south west points of a tile using the north west points of the tiles left of and below the current tile. If those tiles are not present, because you are at the edge of the map, I have no idea what will happen - but presumably it won't work as expected. You can run into this when you are zoomed out sufficiently in the very south or east of the map. But then you are either at the International Date Line (that runs from the North Pole to the South Pole exactly on the side of Earth that is exactly opposite of the Greenwich Meridian) or at Antarctica. On the first spot, there’s mostly ocean (why else do you think they’ve put it there) and thus no (visible) geography to speak of. As far as Antarctica goes, you’ll hit another limitation, for it clearly says in the Bing Maps Elevation API documentation:

"There is a limitation in that latitude coordinates outside of the range of -85 and 85 are not supported."

So beware. Stay away from the Earth's edges. Your app might fall off :).

Some assembly required, batteries not included

Indeed, it does not look exactly like in the videos I showed. Walk the World employs a different map tile sets (plural indeed), and there's also all kinds of other stuff my app does - like sticking the map to the floor so that even at high elevations you have a nice overview, reverse geocoding so you can click on the map to see what's there, tracking the user's moves so it can make a new map appear where the user is walking off it  - connecting to the old one, zoom in/out centered on the user's location in the map, showing a map of the physical surroundings... there's a lot of math in there. I only showed you the basics. If you need a HoloDeveloper who knows and understand GIS to the core, you know who to contact now :)

Conclusion

Once you know the basics, it's actually pretty easy to create a 3D scaled map of about anywhere in the world, that is - anywhere where the Bing Maps Elevation API is supported. The 3D stuff is actually the easy part - knowing how to calculate tiles and build a slippy map is harder. But in the end, it is always easy when you know what to do. Like I said, I think GIS is a premier field in which Mixed Reality will shine. I am ready for it: I hope you are too. Innovate or die - there are certainly companies I know that could take that advice and get moving.

Get the demo project and get inspired!

Credits

Thanks to René Schulte, the wise man from 51.050409, 13.737262 ;)  for pointing me to the Bing Maps Elevation API. And of course to Rick Bazarra, who inspired me so often and actually provided some crucial code in his YouTube training video series.