Converting lat/lon coordinates to local coordinates for HoloLens apps

This is going to be a bit of a theoretical story, but I feel it’s a necessary as a precursor to my promise to explain in detail how I made the AMS HoloATC app. So bear with me.

One of the challenges when it comes to showing geo-positioned data in a HoloLens is that is most of this type of data comes in Lat/Lon (and optional Alt – for altitude) format. The UWP Map component knows how to handle it, but if you want to use this kind of data in a HoloLens you will need some way to convert to the X-Y-Z system Unity uses.

Now there are two approaches. The first one is to go for the full 3D experience and you project the coordinates relative to a globe. Although is awesome for demoes, it also has the drawback that it may not be easy to see relative heights over larger distances in what is in essence a curved plane. In my app I take the Netherlands – an area of 300 by 200 km, the largest part more or less North-South, and condense that by a factor of 15000 to about 20 x 13 meters. The curvature of the Earth would cause the airplanes to rise from the edges of the view, and then come down again as they head for approach and landing.

The second approach is to pretend the Earth is flat in the Netherlands (which is kind of true, but and in a different way-people who have ever visited us will understand why) and use a tangential plane that hits the Earth on a certain spot. This is the approach I took. For the spot where the plane hits the Earth I took what according to Google Maps is the center of Amsterdam airport (aka Schiphol) -  52.307687, 4.767424, 0 (lat/lon/alt)*. A very useful site for finding lat/lon coordinates of places on Earth is this one. Click on the map or enter a name and presto.

Projecting an airplane to a globe or this tangential plane requires more math than I know. Although I worked in GIS for over 20 years I was never formally trained for it that and I was never a math wizard anyway. Fortunately, some guy called Govert van Drimmelen – I presumed him to be Dutch as well based on his name, but he is actually from South Africa – has posted a GitHub gist that does exactly what you need. It actually supports both approaches (projection to a globe and to a tangential plane). I made a fork of it that only gets rid of the missing Util.DegreesToRadians, the tests and other stuff that is not used, but is essentially the same.

But there are still two caveats, and they both have to do with altitude. I put the center of Schiphol on the 0,0,0 position in Unity’s coordinate system, and then wrote this test code:

double x, y, z;
GpsUtils.GeodeticToEnu(52.307687, 4.767424, 0, 52.307687, 4.767424, 0, out x, out y, out z);
Assert.IsTrue(x == 0 && y == 0 && z == 0);

The first coordinate is the coordinate I want to project, the second one is the place where the tangential plane is hitting the ground. If I put both at the same place, the method should return 0,0,0. And indeed it does. Hurray.

Now let’s head over to the city of Leeuwarden, some 121 km North-East from Schiphol (this is a useful simple website for measuring distances) at lat, lon = 53.201233, 5.799913. As I have no idea what to expect, let’s first print out the results before testing

GpsUtils.GeodeticToEnu(53.201233, 5.799913, 0, 52.307687, 4.767424, 0, out x, out y, out z);
Debug.WriteLine($"{x},{y},{z}");

Result: 68991.988451593,99923.1412132109,-1155.45361490022. The output is apparently in meters. Nice. So… 100km to the North and 69km to the West. If you do Pythagoras on those first two values, you get indeed about 121000. Awesome. So that seems to work as well. But… 1155 down? Still the curvature of the Earth, I guess. Apparently when you go 121 km to the North-East, you end up 1155 below the horizon of someone standing on the original place. I think. So when I project my plane I use X for X, Y for Y, and the original altitude for Z. But this leads to another problem.

First of all, one unit is a meter in a HoloLens (or appears to be – let’s not get metaphysical). If I were to use X/Y/Alt directly, an airplane approaching from the direction of Leeuwarden at 3km would be some 121km from my point of view – and at 3km height. Even if I used 1:1 models it would be invisible. That does not help giving an Air Traffic Controller (ATC) a nice 3D view of the area around his or her airport of condern. So I divide X and Y by 15000. Result for an airplane about Leeuwarden is this: 4.59946589677287,6.6615427475474

So an airplane that is in real life about 121km from me appears about 7 meters from me, forward and quite a bit to the right. As airplanes on approach for an airport (at least around Schiphol) are moving within 20km around the airport this makes the busiest part of air traffic happen in an apparent space of about 2.5x2.5 meters. That looks good to me. But if I would use the same scale factor on the height, and airplane flying 3km would be 20cm from the ground. At final approach, say at about 500m, it would be a mere 3cm from the ground. At 10km – cruise altitude, and not particularly interesting to and ATC - it would still be a little short of 70cm. Our poor ATC would have to look very carefully to see height differences between aircraft on final approach ;). So I opted to scale down the height considerably less – by dividing that by only 2000. That puts an aircraft on 500m at 25cm, on 3km it is at 1.5m, and 10km is at 5m – still well within visual range, but literally flying way over your head, which is exactly what you want, as it is not of immediate concern for an ATC handling the arrivals and departures on an airport. The only drawback is that aircraft seem to climb at impossible steep trajectories when taking off, but I think that’s ok.

So this is how I convert and transform aircraft flight data into what I think a format and space that makes it usable for an ATC wearing a HoloLens. The fun part of it is that when I hook up the app to live data and put Schiphol to the side of the room, the city where I live is more or less where the living table is. It’s pretty awesome to see airplanes coming from Schiphol and moving over that table – because in certain conditions, and when I open a window, I can actually hear the rumble of engines of the actual airplane outside when it passes over my house at 3km height ;)

As I wrote earlier, a very theoretical and perhaps even dry piece of text. I hope it’s useful for other people thinking about using geo-positioned data in HoloLens. I am still a GIS nut at heart, although I don’t work in GIS anymore. I wonder if other people maybe have better approaches.

*^{Technically that is not correct - Schiphol is about 4.5 meters below sea level. Do not be alarmed. We have excellent dunes, dams, and other things to keep the wet bit where it belongs, i.e. not where we live. That is, for the time being ;)}

Sharing download links to (hidden) HoloLens apps

The Windows store has a very neat feature. You can send out direct http links to people that, when entered, show and app directly in the web version of the store, with a neat button next to it to initiate install. For instance, if you hit this URL: https://www.microsoft.com/store/apps/9NBLGGH08D4P

It will take you directly to my app Map Mania.

Even more handy is that this also works with hidden apps, so you can submit early versions of your app to the Store as hidden - and only hand out the link to a limited number of people. The store has more advanced methods for distributing betas these days, but as a low friction and easy way to send your POCs to customers this direct link feature is still very useful.

Unfortunately, this little trick does not work for HoloLens. For instance, my very first HoloLens app in the Store - “AMS HoloATC” - is accessible via URL: https://www.microsoft.com/store/apps/9NBLGGH52SZP

It will actually show you the app, but it will also say “This app does not work on your device”. Even on a HoloLens.

The solution for this is pretty simple – don’t use the http link, but use a store protocol link. Thus, you enter in Edge: ms-windows-store://pdp/?ProductId=9NBLGGH52SZP

And this will open the Windows Store App at the right place. And a button to install the app:

So you simply paste the product id behind “ms-windows-store://pdp/?ProductId=” and you can once again share links to selected audiences.

Credits go to my fellow MVP Tom Verhoeff who suggested trying this in an online conversation this afternoon, when I wanted him to try and download my app to see if it was available already. Incidentally, feedback on the app is also appreciated. In it’s current form it’s a one man spare time project. A video will appear shortly, and I will also document in detail how it’s built. Stay tuned.

How shipping an UWP app update can make your app unavailable

First of all – don’t do this. You might regret it dearly.  

One might consider the fact that this is possible at all to be a bug in the Store. For the moment I am just operating on the assumption that I, a Windows Development MVP, actually managed to mess up my best paying app’s availability by being distracted and not reading the wording in the submission carefully. The sole purpose of this little blog post is to prevent you falling into the same trap.

I happily created a new version of my app now supporting the Anniversary Update, so it would be available on all clients (most notably the XBox One!) In the package page, for some reason I misread the line next to the top checkbox:

It says, now in gray print because the submission is already done:

“Let Microsoft decide whether to make this app available to any future device families”

The key error I made here was missing the word future. I had played a little with the check boxes above the platforms and then I noticed the text above, got distracted for some reason, misread it, and unchecked all the boxes thinking “better let Microsoft handle this choice”.

Wrong. Very wrong. Microsoft handles future device families, not current device families. Set the checkboxes like this – and what you end up with is a submission that is not eligible for any of the mentioned devices. And it says so if you read the legend below the package. What struck me in hindsight as odd is that the app was certified and published without a hitch with what might be considered as a completely senseless set op options. Anyway, the net result was - when you tried to search in from the store, it never showed up, and if I used the direct link ($0.99, free trial included, thank you for supporting your faithful developer) it said, in the browser:

"This app does not work on your device"

On every device.

Fortunately the good people of the Windows Store were nice enough to point out my error, so I resubmitted (just the same package, just all checkboxes checked now!). So this is how a submission is supposed to look, and it will look that way if you don’t mess with those checkboxes to begin with:

Then the Store Team still had to do something to boot my previous submission out of the queue. When that was done, the app became available again. After five days of absence.

So I guess I stumbled upon an edge for which case the people of the Windows Store could not even imagine some stupid enough to actually stumble upon. ;) Quite a sobering experience for me, both as a user and a developer – even when your users are intelligent people. somehow, some way they will find a way to click on the wrong button and mess up. Usually I am on the other side of the fence. I will never say again “how can anyone be so stupid to do XYZ” because th8ings like this happen.

This is not the proudest article of my blog, but I figured that if I could fall into this trap, there is a remote possibility other people would do so as well. Bottom line: be careful with these checkboxes, read carefully what they mean, and don’t get distracted during a submission ;)