29 October 2016

Running a Hololens app on a Raspberry PI2–and controlling it with a keyboard

Intro

No, I am not drunk, nor did I smoke some peculiar produce our capital Amsterdam is famous for, and I did not bump my head a bit too hard on my car while getting out of it either. I was just curious how far the U of UWP would carry me. And it turns out, a lot further than I thought. It’s actually possible to run a HoloLens app on a Raspberry PI2. When I saw it happen, I actually had a bit trouble believing my own eyes, but it is possible indeed. With surprisingly little work. Although I added a little extra work to make it more fun.

Parts required & base material

  • One Raspberry PI2 or 3 running the latest version of Windows 10 IoT Core.
  • A keyboard that plays nice with the above
  • A screen to see the result on

I used a Raspberry PI2 running the latest Windows 10 IoT Core Insider’s build and a NexDock. Mainly because I have one, it’s handy and it looks cool. Be aware that to connect the NexDock keyboard you will need a Bluetooth dongle for your RP2 – or you will need use a RP3 which has it on board.

This post builds upon my (as of the time of this writing still incomplete) series about the Hololens Aircraft tracker, but that’s only because that’s a cool app to demo the idea on. It is no part of the series. I made a separate branch of the app at the end of the 6th episode. So this is like an interlude.

Some brute-force demolishing to get things to compile

Basically it’s as simple as setting the build target to ARM:

image

The only problem is, if you do that and then rebuild the project, it will complain about this

image

For the very simple reason this dll, which is part of the HoloToolkit, is not available for ARM. Hell, HoloLens runs x86 only, so why should it be available. No-one ever anticipated using a dolt peculiar person like me trying to run it on a bloody Raspberry PI2.

There is a rather crude way of fixing it. We are not using SpatialUnderstanding anyway, most certainly not on the Raspberry PI2. So I got rid of the plugin that Visual Studio complained about, by going to this folder in Unity:

image

And hitting delete. Rebuild project in Unity, try to compile it in Visual Studio. Alas. Still no success

image

But this is a different plugin dll – PlaneFinding. Also something we don’t use. Now this is a bit confusing, as there are three folder containing a Planefinding.dll. Maybe that’s an error in the HoloToolkit.

image 

Whatever. Let’s get rid of the whole plugins folder under SpatialMappping. Once again, rebuild in Unity, compile in Visual Studio. And wouldn’t you know it…

image

The sweet smell of success. Now I want you to understand this is no way to go about to make a HoloLens project compatible with a Raspberry PI2. This is using a mallet to hit very hard on a square peg to make it go through a round hole. I have demolished two components you might want to use in the future version of your HoloLens app. But this is not serious development – this is hacking for fun’s sake to prove a point. That is why I have made a separate branch ;).

Controlling the app’s viewpoint

When you run the app on the HoloLens this is no issue at all. If you want to see the airport and it’s planes from closer up or from a different angle, you just move your head, walk to the object of your interest – or around it. If if runs on a screen, things are a bit different. So I created this little behaviour (with “ou” indeed, which suggests the good folks at Unity have been educated in The Queen’s English) that more or less replicates the key mappings of the HoloLens emulator:

It’s crude, ugly, the result is a bit stuttering – but it does the job.

using UnityEngine;

public class KeyboardCameraController : MonoBehaviour
{
  public float Rotatespeed = 0.4f;
  public float MoveSpeed = 0.02f;
  public float FastSpeedAccleration = 7.5f;

  private Quaternion _initialRotation;
  private Vector3 _initialPosition;
  void Start()
  {
    _initialRotation = Camera.main.transform.rotation;
    _initialPosition = Camera.main.transform.position;
  }

  void Update()
  {
    var speed = 1.0f;
    if (Input.GetKey(KeyCode.LeftShift) || Input.GetKey(KeyCode.RightShift))
    {
      speed = FastSpeedAccleration * speed;
    }

    if (Input.GetKey(KeyCode.LeftArrow))
    {
      Camera.main.transform.RotateAround(Camera.main.transform.position,
          Camera.main.transform.up, -Rotatespeed * speed);
    }

    if (Input.GetKey(KeyCode.RightArrow))
      Camera.main.transform.RotateAround(Camera.main.transform.position,
          Camera.main.transform.up, Rotatespeed * speed);

    if (Input.GetKey(KeyCode.UpArrow))
    {
      Camera.main.transform.RotateAround(Camera.main.transform.position,
          Camera.main.transform.right, -Rotatespeed * speed);
    }

    if (Input.GetKey(KeyCode.DownArrow))
      Camera.main.transform.RotateAround(Camera.main.transform.position,
          Camera.main.transform.right, Rotatespeed * speed);

    if (Input.GetKey(KeyCode.A))
      Camera.main.transform.position +=
          Camera.main.transform.right * -MoveSpeed * speed;

    if (Input.GetKey(KeyCode.D))
      Camera.main.transform.position +=
          Camera.main.transform.right * MoveSpeed * speed;

    if (Input.GetKey(KeyCode.W))
      Camera.main.transform.position +=
          Camera.main.transform.forward * MoveSpeed * speed;

    if (Input.GetKey(KeyCode.S))
      Camera.main.transform.position +=
          Camera.main.transform.forward * -MoveSpeed * speed;

    if (Input.GetKey(KeyCode.PageUp))
      Camera.main.transform.position +=
          Camera.main.transform.up * MoveSpeed * speed;

    if (Input.GetKey(KeyCode.PageDown))
      Camera.main.transform.position +=
          Camera.main.transform.up * -MoveSpeed * speed;

    if (Input.GetKey(KeyCode.Escape))
    {
      Camera.main.transform.position = _initialPosition;
      Camera.main.transform.rotation = _initialRotation;
    }

  }
}

Drag this behaviour on top of the camera (or for what matters, anything at all). You will be able to control the camera’s standpoint via what I have been told is the the standard PC gamer’s WASD and arrow key mappings. PageUp/PageDown will move the camera standpoint up and down, ESC will bring you back to the original viewpoint when the app started, and using SHIFT will make things go faster.

Deploy and run

Deploy the app to your Raspberry PI2 or 3 using Visual Studio – and select your PI as remote machine. Use either “Release” or “Master” build configuration. The latter should - in theory – go faster, but takes much longer (as in very much longer) to compile and deploy. Also, if you choose “Master”, the app does not always start on my PI, it’s sometimes only deployed – so you have to get it going via the device’s Device Portal. This may have something to do with me running an Insider’s build.

Either way – if you have WiFi either built-in or via dongle, that’s fine, but unless you particularly like waiting, connect your PI2/3 to a wired connection while deploying. I also observed issues when the app runs while only having WiFi connectivity – data comes in pretty slow, it can take minutes before the first airplanes appear, while on wired it takes like 30 seconds, max. Apparently my 2.4Ghz network (which the PIs use) is not that strong compared to the 5G all the other devices in the house employ.

And it works. Here are some pictures and a video of the app in action. The performance is not stellar (big surprise here, matching a $35 device against a $3000 device that comes straight out of Star Trek), but still – it works pretty reasonable.

IMG_4786IMG_4792

Conclusion

Looks like the U in Universal Windows Platform is pretty universal indeed. Microsoft weren’t talking BS rubbish about this promise. This app can also be deployed to PCs (I learned that by accidentally choosing “Local Machine” as a target) and I don’t doubt it will run on phones and even XBox One, although I think I would have to ponder a little about the way to control the viewpoint on those devices as they don’t have a keyboard. Yet, an impressive result.

Code can be found here.

26 October 2016

A HoloLens airplane tracker 6–adding an airport (and a tower)

Intro

This episode is nearly codeless, it’s all Unity and general geeky GIS stuff. We are going to cut out a piece of OpenStreetMap to depict the location of Schiphol. We are also putting a real 3D ATC tower on the map. If you look closely, it has only a very superficial liking to that of the actual Schiphol ATC tower, but those are just minor details ;)

schiphol[3]Getting and positioning the map

Getting it was easy enough. As I wrote, I went about to cut out a piece of rectangular OpenStreetMap that contains Schiphol airport and, incidentally and certainly not coincidentally, the Wortell offices. We will deal with that in the final episode of the series.

Now positioning that map is a little tricky, as I have set the center Unity’s coordinate system to the center of Schiphol according to Google Maps. But the center of Schiphol, according to this nifty site which also displays coordinates, is found at a location that is about 25% from the bottom of the image and a little to the left.

image

The way I solved this was with a little bit of knowledge of geo coordinates and an advanced technique I learned in my years in Geo Information Systems, which can be described as ‘fiddling until it fits’ ;)

First order of business is to try and find out as closely as you can the lat/lon coordinates of the upper left and the lower right corner of the image. I also used latlong.net for that. As far as I could see, top/left is (52.368739, 4.706697) and bottom right (52.285193, 4.811196), while the center of Schiphol is (52.307687, 4.767424)

Now we need to go back to a precursor to this series in which I explained how to convert lat/lon coordinates to Unity’s X/Y/Z coordinates using a tangential plane. We can convert these coordinates using this code

double x, y, z;

GpsUtils.GeodeticToEnu(52.368739, 4.706697, 0, 52.307687, 4.767424, 0, 
  out x, out y, out z);
Debug.WriteLine($"{x / 15000},{y / 15000}");

GpsUtils.GeodeticToEnu(52.285193, 4.811196, 0, 52.307687, 4.767424, 0, 
  out x, out y, out z);
Debug.WriteLine($"{x / 15000},{y / 15000}");

Note I ignore the Z coordinate, and I divide the result by 15000 as that is how I scaled the airplane data as well. The result is

-0.275750329947415,0.453014630195657
0.199135521004053,-0.166804762707821

So we now know the map in Unity is 0.474885850951468 (east-west) by 0.619819392903478 (north-south) meters. We will need this knowledge soon. But first, drag your map cut-out into your App/Textures folder in Unity:

image

Then proceed to create a material based upon this texture. In the Materials folder, right-click, hit Create/Material and call the material “SchipholMaterial” (or whatever it is you fancy). Then drag your Schiphol texture on top of the “Albedo map” square

image

Next, click the white rectangle to the right of the Albedo square. That pops up a color picker – set the color to B3B3B3FF to make the Schiphol map a bit less overly bright.

image

Then, in HologramCollection, create a 3D Plane object. Initially this will look like this:

image

What is important to wrap your head around is that at scale 1:1, a plane’s size is 10x10 of these blocks which is 10x10 meters in HoloLens. We need to reduce that size so that it’s about 0.62 by 0.475 meters as that is 1:15000 scale. As 10m = scale 1, we need to scale the plane by 0.0475 for X, and 0.062 for Z (leave Y to 1).

Now drag your SchipholMaterial below the “Add Component” button on your Plane. Yes! Unity shows or Schiphol map, with the right dimensions, the right size…

image

… upside down. Life – and Unity is - full of these little surprises. I have no idea why this is necessary, but if you go to the transform section of the object and hit “180” in the Rotation Y box,

image

… that will fix it. Unfortunately, we are still not done, and I will show you why. Add a Sphere to the HologramCollection, size it 0.015 in all directions, give it the Trail Line as material. You will notice a blue dot in the middle of the image if you look from above. That’s 0,0,0 according to Unity.

image

(I toned down the light a little to make this better visible). Unfortunately, as I stated before, 0,0,0 should be on another place, quite a bit down and to the right, near the “Schiphol” label. If we continue to use this, airplanes taking off won’t be properly aligned with the runways :(.

Back to what we found earlier. The corners of the map should be

-0.275750329947415,0.453014630195657
0.199135521004053,-0.166804762707821

on the tangential plane we created and used the center of Schiphol as found by Google Maps as 0,0 in the horizontal plane. Now what follows this might look odd, but there is logic to that.

  • Summarize both X values, and divide the result by two. That is half of the difference in distance to what should be the X-center. This is about –0.038
  • Summarize both Y values, and divide that result by two as well. That is half of the difference in distance to what should be the Z-center (yeah, gets me every time too – what you tend to think of as Y from your Math class, is Z in Unity). This is about 0.143
  • Add those values to X and Z position in the Plane’s transform:

image

And boom. Compare that to the Google maps marker coordinate and that seems to fit pretty well.

imageimage

Not an exact fit, but that’s because we manually determined the corner’s lat/lon positions. But we are not talking zoning permits here (lengthy legal battles have been fought over mere centimeters on those in my lovely country) so it will do for this app.

I left the sphere in the project as a reference but disabled it, as we don’t need it anymore but you can enable it again and see it indeed shows up at the center of the map.

So. Now we have a properly sized and positioned map. There is a tiny detail missing.

Adding an ATC tower

Once again we go to CGTrader and browse around for a tower. Unfortunately the iconic Schiphol ATC tower is not available (paid or free) so I settled for the Hong Kong ATC tower that bears some superficial resemblance to it. Unfortunately there is some building in the attached to the actual tower in model - that I neither need or want. I followed this rather crude work flow to deal with that:

  • I downloaded the model fbx model zip file
  • Unpacked it, and renamed 3d-model.fbx to tower.fbx
  • I created a folder “Tower” in my Assets folder in the Unity editor
  • I dragged the tower.fbx into that folder
  • And then I dragged the resulting tower prefab on the HologramCollection. It will be very large.
  • Set its position to far far away, like 5000,5000,5000 to be out of the way of what we have already created.
  • Double click on “Tower” to get in into view. I got about this:

image

Rotate around it till you get about this:

image

Now what we want to get rid off, is the building next to the tower. I don’t know if I am legally entitled to change a model, so I limit myself to hiding parts of it. This works as follows.

  • First, make sure this button is selected.

image

  • Select the model by clicking on the building. Not the tower. This will select the whole model nonetheless
  • Click the building again. This will only select the building
  • Go to the inspector. Uncheck the checkbox all the way to the top left

image

That will get rid of most of the unwanted stuff:

image

See? Rinse and repeat until all the stuff you don’t want is invisible. Remember: click once selects the whole model, click twice to select only what you want. Hidden too much? CTRL-Z is your friend. It’s a bit of a hassle, especially the small floating bits just right of the tower, but in the end you should be left with just this:

image

When you are done, scale the tower to 0.001,0.001,0.001 and put it’s location to 0,0,0. Select the Main Camera, then GameObject/Align View to Selected and you will see Schiphol again, with a tiny ATC tower standing on top of it. If you look at it almost dead from above, you will see that it’s a bit off to the right and the bottom, according to this aerial picture from Google Maps that ironically enough blocks the view of the actual tower itself with a label – so I marked it with a red circle

imageimage

As it is the wrong tower – and as it’s not to scale, I did not bother with calculating it exact position but dragged it around a bit until it looked like it was in the same spot, which was –0.0283 for X and 0.0316 for Z. Fiddling a bit around with the view spot so I could see the tower’s base hitting the ground made it look like this:

image

And that’s fine with me. I’ll call it a day.

Conclusion

I have shown you how to orientate, scale and properly position (geographical) maps and objects on those maps in Unity, using some GIS knowledge, calculus, common sense and fiddling around. This makes your app look just that little extra pretty, plus it bases it in ‘real space’ which makes it extra cool.

Code – well, more like assets and config, can be found here. And I’d like to say hi to my fellow MVP Tom Verhoeff and his trainee Ryan, who seem to like this tutorial so I hope I made both their days ;)

17 October 2016

A HoloLens airplane tracker 5-smooth movement with iTween and adding a trail

Intro

In the video in the first post of this series  you can see the airplane models leaving a kind of trails in the air, while they move and rotate in a smooth fashion - in stead of merely hopping from place to place. The first is accomplished by adding a TrailRender to the AircraftHolder prefab – the second by using an awesome Unity plugin called iTween.

Adding the trail

imageBefore you can add the trail, it must have a material from which it’s made. This the color of the line. I will freely admit materials are still a bit vague to me, and after quite some fiddling I came to the following settings:

  • Shader: HoloToolkit/Fast
  • Albedo: hex color 120067FF
  • Metallic: 1
  • Emission: hex color 021F61

This leads to a kind of metallic light blue to dark blue trail line (depending on where the ‘directional light’) is shining, which is quite satisfactory. Feel free to mess around with the parameters. I did ;).

Then go to App/Scripts and open the AircraftHolder prefab. Click on “Add Component”, then select “Trail Render”. Use the following workflow/ setting:

image

  • Expand “Materials”. Drag the Trail Line Material that we just created on top of the “Element 0” field
  • Set start width to 0.003
  • Set end width to 1e-05.

This means the trail will be visible for 120 seconds, so the tail of the trail line will be at the place where the airplane was two minutes ago. Older segments are automatically deleted. Also, near the airplane the trail will be 0.003 meter wide, petering out to 0.00001 meter (1e-05) at the very end. And sure enough:

image

Unity takes care of the drawing, erasing, and making-smaller-toward-the-end part. The only thing we have to do is move the airplane. Which is what we were doing already anyway. It once again almost feels like cheating.

imageIntroducing and obtaining iTween

In a blog post about an earlier project, the CubeBouncer, I talked about using coroutines to achieve smooth animations. In the mean time I have stumbled upon iTween, an awesome Unity plugin created by one Bob Berkebile, that makes creating animations of all kinds a lot easier. Although some concepts are a bit odd – particularly the the use of the Hash object to pass in an arbitrary number of parameters into methods – once you wrap your head around it, it’s very easy to use. Oh. And it’s free too.

The easiest way to obtain it, is via the Unity Asset Store (Window/Asset Store or CTRL-9). Then enter iTween as search term. Click the iTween Logo, click import, and then the following dialog, as displayed to the right (below this paragraph), pops up. I usually deselect everything but the actual plugin, but if you would like to see samples and the Readme, feel free to leave everything checked.image

When the process is finished, your project tab should contain the iTween plugin like below

image

Once you have the iTween plugin installed, build the project, and move over to Visual Studio.

Using iTween for moving and rotating

So we head back to the AircraftController’s SetLocationOrientation method. We are going to change that a little:

private void SetLocationOrientation()
{
  SetNewFlightText();

  if (!_firstMove)
  {
    transform.localPosition = GetFlightLocation();
    if (_flightData.Heading != null)
    {
      transform.localEulerAngles = GetNewRotation();
    }
    transform.localScale = new Vector3(0.0015f, 0.0015f, 0.0015f);
    _firstMove = true;
  }
  else
  {
    iTween.MoveTo(gameObject, iTween.Hash("position", GetFlightLocation(), 
      "time", 7f, "islocal", true));
    if (_flightData.Heading != null)
    {
      iTween.RotateTo(gameObject, iTween.Hash("rotation", GetNewRotation(), 
        "time", 7f, "islocal", true));
    }
  }
}

So if the first set of data (or the initial move to a location) as has not been applied yet – that is, this is a newly created airplane – just do the same as we did before – plonk the airplane at it’s location with the proper rotation and attitude, and give it a size that makes sense. But if this aircraft already exists – move it smoothly to it’s new location in 7 seconds, using local position. And rotate it too, if needed, in the same time. And that is all you need for a smooth transition. Here you see the Hash in action. iTween has a few overloads for both Move and Rotate where you don’t need it, but in effect, when you want to do something only a little advanced – like using local space – say hello to Mr. Hash.

You can of course rant against these ‘stringly typed’ variables and talk about type safety and stuff like that – but this is how it works, and it’s perfectly usable, albeit a bit odd for those who are fans of tight architecture and clean code. But then again, HoloLens is a pretty out-of-the-ordinary device anyway. When in Rome, act like Romans. ;)

Making the text fade in and out

Now the only thing we are missing is that nice effect where the text fades out just before the airplane moves – and appears again – with the newly updated flight data. This prevents the text from flashing and also gives a nice indication whether or not we are receiving updates for an aircraft (you may have noticed by now that aircraft that have landed tend to hang around for a while just above the airport before disappearing).

First, we need to add another field:

private GameObject _label;

And of course, initialize that in Start:

void Start()
{
  _text = transform.GetComponentInChildren<TextMesh>();
  _label = transform.FindChild("Label").gameObject;
  _initComplete = true;
}

I already warned you Start would grow, didn’t I? ;) Then we add a new method:

private void StartSetLocationOrientation()
{
  if (_firstMove)
  {
    iTween.FadeTo(_label,
        iTween.Hash("alpha", 0.0f, "time", 0.5f, 
        "oncomplete", "SetLocationOrientation",
        "oncompletetarget", gameObject));
  }
  else
  {
    SetLocationOrientation();
  }
}

So when the first move is indeed done, it will fade the label’s alpha channel to 0 in half a second. When that is done, call the original SetLocationOrientation. So this is a callback. What is a very important iTween convention to understand – default the callback will be called on the object you are operating on – and that is the label. Not the airplane. Therefore I added an “oncompletetarget” entry with the current gameObject as value – so iTween knows it should call SetLocationOrientation and use my gameObject as a target, not the label.

Oh, in the else - if the airplane was just created and does not have an initial location - just move it to it’s initial location without using iTween.

Then we move to SetNewFlightData and change the call to SetLocationOrientation to StartSetLocationOrientation:

public void SetNewFlightData(Flight newFlightData)
{
  if (_initComplete)
  {
    var move = _flightData == null ||
                !_flightData.Location.LocationEquals(newFlightData.Location);
    _flightData = newFlightData;

    ExtractSpeedAndHeading();
    if (move)
    {
      StartSetLocationOrientation();
    }
    else
    {
      SetNewFlightText();
    }
  }
}

And finally, to SetLocationOrientation we add all the way to the end this line:

iTween.FadeTo(_label, iTween.Hash("alpha", 1f, "time", 0.5f, "delay", 0.2f));

This fades the label’s alpha channel (back) to 1, in 0.5 second, after a delay of 0.2 seconds. I found these values after experimenting. Feel free to play with times and stuff.

Conclusion

I have shown you how to add a trail line to an airplane, and how to use iTween for smooth transition effects. Once the basics of your HoloLens app stand, it’s quite easy to make just a little more effort and get it smooth and cool. Using a powerful tool like iTween makes it almost child’s play IMHO, once you get your head around how to using it.

Code, as always, can be found here.

15 October 2016

A HoloLens airplane tracker 4–Reading data and positioning airplanes

Time to rock and roll

All right my friends, it’s time to show the heart of the matter – how to read data from an Azure Mobile App Services data service and make the airplanes appear in the right places.

Adding the JSON data object scripts

As the projects are partially regenerated, and have a total different .NET baseline, I have not found a good way to share classes as binaries between full .NET code and Unity without getting in all kinds of trouble, so I choose the easy way out: I took

  • TrackType.cs
  • Coordinate.cs
  • Flight.cs

from the FlightDataService\DataObjects, moved them the App/Scripts/DataObjects folder in the Unity project. Then of course the service does not compile anymore, but I added the moved file as link. Then at least we have once source of truth as far as these classes are concerned.

Then we need a simple class to hold a list of flights with a timestamp (add that to the DataObjects folder as well):

using System;
using System.Collections.Generic;

namespace FlightDataService.DataObjects
{
  public class FlightSet
  {
    public FlightSet()
    {
    }

    public FlightSet(List<Flight> flights)
    {
      Flights = flights;
      TimeStamp = DateTimeOffset.Now;
    }
    public DateTimeOffset TimeStamp { get; set; }

    public List<Flight> Flights { get; set; }
  }
}

Adding a DataService to read data

I opted to add the DataService as a Singleton that can be accessed in the app. But we have a bit of an issue here -  Unity runs on Mono, e.g. .NET 3.something, and does not have things like packages for reading App services, or support async and Tasks and stuff. But remember – a HoloLens app is a two stage rocket. Unity does not generate an app for the HoloLens, but an UWP project, and that UWP project can use all the goodness from all the latest stuff. Meet the simple way to close the gap - my new friend UNITY_UWP.

using System;
using System.Collections.Generic;
using FlightDataService.DataObjects;
using HoloToolkit.Unity;
#if UNITY_UWP
using Microsoft.WindowsAzure.MobileServices;
using System.Net.Http;
using System.Threading.Tasks;
#endif

public class DataService : Singleton<DataService>
{
  public string DataUrl = "http://yourflightdataservice.azurewebsites.net/";

#if UNITY_UWP
    private MobileServiceClient _client;
#endif

  public DataService()
  {
#if UNITY_UWP
    _client = new MobileServiceClient(new Uri(DataUrl));
#endif
  }

#if UNITY_UWP
  public async Task<List<Flight>> GetFlights()
  {
    var result = await _client.InvokeApiAsync<List<Flight>>(
	                "FlightData", HttpMethod.Get, null);
    return result;
  }
#endif
}

Code between #if UNITY_UWP – #endif blocks is invisible to Unity – but will be accessible in the UWP HoloLens app that is generated by Unity. You will see Unity swallows it – no problems at all. There is only this tiny thing – you cannot add NuGet packages to Unity. So we have to do that in the UWP app, in Visual Studio. But… the UWP app is generated from Unity by File/Build and overwritten. The trick is to know not the whole app is overwritten, but only part of it. What I did was the following:

image

  • Generate the app in a subfolder “App” of the Unity project. This should be <your_root>\HoloATC_Demo\AMS HoloATC Demo\App
  • Open the solution in HoloATC_Demo\AMS HoloATC Demo\App (not the one in HoloATC_Demo\AMS HoloATC Demo)
  • Go to the project Assembly-Csharp
  • Add the NuGet Package Microsoft.Azure.Mobile.Client
  • Find the project.json file that belong to this project. You will find it’s not under AMS HoloATC Demo\App at all, but in AMS HoloATC Demo\UWP\Assembly-CSharp\project.json
  • This is the only file that has been changed with respect to the generated code. You will have to add this to source control. Even if Git claims it’s an ignored file.
  • After that – if you checkout the source in a different location or on a different machine, you will only have to regenerate the solution in the right place (the App folder), and revert this specific file. You will need to do this, or else the UWP app won’t compile.

Adding the AircraftLoader to create and update aircraft

I have created one class that is actually responsible for actually getting the aircraft data from the DataService – and one that is responsible for creating, updating and deleting aircraft. The updating – moving to a new position – is handled by the AircraftController – basically all the AircraftLoader says to the airplane is ‘here is new data for you – handle it’

Remember – it’s always best to start in Unity. So create an AircraftLoader and an AircraftController script in Assets/App/Scripts. Then proceed to double-click on AircraftLoader. This will open a Visual Studio instance. Be aware – this is not the project your will be running. This is the other solution, the one that is in the project root – I call this the ‘Edit’ solution. First, we are going to make sure the “usins” of the class are properly organized – that is, in a way that doesn’t make Unity go belly-up:

using System;
using UnityEngine;
using System.Collections.Generic;
using System.Linq;
using FlightServices.Data;
#if UNITY_UWP
using System.Threading.Tasks;
using Windows.Web.Http;
using Newtonsoft.Json;
#endif

Then we are going to add a whole lot of fields. And yes, some are public. These are things that can be set from the Unity editor. I have talked about that before.

public GameObject Aircraft;

public string TopLevelName = "HologramCollection";

private Dictionary<string, GameObject> _aircrafts;

private readonly TimeSpan _waitTime = TimeSpan.FromSeconds(5);

private DateTimeOffset _lastUpdate = DateTimeOffset.MinValue;

private Queue<FlightSet> _receivedData;

private GameObject _topLevelObject;
  • The Aircraft GameObject field will be used to drag our AircraftHolder onto – so this behaviour knows which game object to create
  • TopLevelName is the name of the parent object to which all the aircraft are added to. This  “HologramCollection” by default
  • _aircrafts is a dictionary of aircraft game objects with their id as key, so we can update/delete existing aircraft game objects based upon the data coming in
  • waitTime – minimal time to load new data
  • _lastUpdate – the time the last update of aircraft was completed
  • _receivedData – a queue with data coming from the service. I use this pattern all the time. I queue up data from the service, then read it in the Update method that Unity calls automatically 60 times a second. Don’t ever try to update game objects from .NET callback methods or events – you might regret it, or just get plain crashes – kind of like happens in UWP XAML apps, where you need the Dispatcher to take care of that
  • _topLevelObject – to store the game object with the TopLevelName in it. I have been told the method I use to find an object by name is quite heavy on performance – so better do it once and retain the result, right.

The Start method simply initializes some stuff:

void Start()
{
  _aircrafts = new Dictionary<string, GameObject>();
  _receivedData = new Queue<FlightSet>();
  _topLevelObject = GameObject.Find(TopLevelName);
}

And then there’s this little method – that is used as callback for the DataService’s GetFlight method:

#if UNITY_UWP

  private void ProcessData(Task<List<Flight>> flightData)
  {
    if (flightData.IsCompleted && !flightData.IsFaulted)
    {
      var set = new FlightSet(flightData.Result);
      _receivedData.Enqueue(set);
    }
  }
#endif

In short - when the data received and it is ok, just add it to the queue of received data. And then let Update handle it. As a matter of fact – like this:

private bool _isUpdating;
void Update()
{
  if ((_lastUpdate - DateTimeOffset.Now).Duration() > _waitTime)
  {
    _lastUpdate = DateTimeOffset.Now;
#if UNITY_UWP
    DataService.Instance.GetFlights().ContinueWith(ProcessData);
#endif
  }
  if (!_isUpdating)
  {
    _isUpdating = true;
    if (_receivedData.Any())
    {
      var set = _receivedData.Dequeue();
      var flightIds = set.Flights.Select(p => p.Id).ToList();

      var aircraftToDelete = 
        _aircrafts.Keys.Where(p => !flightIds.Contains(p)).ToList();
      DeleteAircraft(aircraftToDelete);

      var keysToUpdate = _aircrafts.Keys.Where(p => flightIds.Contains(p));
      var aircraftToUpdate = 
        set.Flights.Where(p => keysToUpdate.Contains(p.Id)).ToList();
      UpdateAircraft(aircraftToUpdate);

      var aircraftToAdd = 
        set.Flights.Where(p => !_aircrafts.Keys.Contains(p.Id)).ToList();
      CreateAircraft(aircraftToAdd);
    }

    _isUpdating = false;
  }
}

First it finds out if it’s necessary to download new data, but downloading and adding happens asynchronously. Then, if there’s any data in the queue, it first makes a list of all the id’s in the newly received flights.

  • Aircrafts that are in the aircraft game object dictionary (with the flight id as key) but are no longer in the list of flights, can be deleted (they have landed or moved out of Dutch airspace)
  • Aircraft that appear in that dictionary and in the list of flights need to be updated – possibly moved to a new position
  • Aircraft that do not have an entry in the game object dictionary are new, and need to be created.

It’s not that hard, see ;). The methods for creating, updating and deleting aircrafts are not that hard either.

private void CreateAircraft(IEnumerable<Flight> flights)
{
  foreach (var flight in flights)
  {
    var aircraft = Instantiate(Aircraft);
    aircraft.transform.parent = _topLevelObject.transform;
    aircraft.transform.localScale = new Vector3(0f, 0f, 0f);
    SetNewFlightData(aircraft, flight);
    _aircrafts.Add(flight.Id, aircraft);
  }
}

private void UpdateAircraft(IEnumerable flights)
{
  foreach (var flight in flights)
  {
    if (_aircrafts.ContainsKey(flight.Id))
    {
      var aircraft = _aircrafts[flight.Id];
      SetNewFlightData(aircraft, flight);
    }
  }
}

private void DeleteAircraft(IEnumerable<string> keys)
{
  foreach (var key in keys)
  {
    var aircraft = _aircrafts[key];
    Destroy(aircraft);
    _aircrafts.Remove(key);
  }
}

The CreateAircraft is the most interesting – it instantiates the aircraft game object, makes the HoloGramCollection it’s parent, passes the actual flight data to the object, and adds it to the dictionary of aircraft game objects using the flight id as the key. Oh, and it also sets the scale to 0, making the plane effectively invisible. This is because, without setting a location on instantiation, the aircraft will appear on 0,0,0 before – where in future episodes we will place the center of Schiphol, very near the tower, and that’s not a place where aircraft belong.

UpdateAircraft just sends flight data to the game object, and DeleteAircraft – well, deletes it. There is only this matter of SetNewFlightData:

private void SetNewFlightData(GameObject aircraft, Flight flight)
{
  var controller = aircraft.GetComponent<AircraftController>();
  if(controller != null)
  {
    //controller.SetNewFlightData(flight);
  }
}

But that has the actual method that moves the flight data to the AircraftController commented out, because that does not even exist. What is worse, the Aircraft does not even have to component. So let’s head over back to the Unity Editor.

Wiring up some stuff in Unity

First of all, we drag the DataService and AircraftLoader Script from the App/Scripts folder on top of the HologramCollection’s inspector page. Then we drag AircraftHolder prefab from App/Prefabs on top of the AircraftLoader’s “Aircraft” property. Net result should be this:

image

Please make sure the Data Url property of the Data Service indeed points to the place where your data service as created in the first post is published.

Then go to the App/Prefabs folder, open the AircraftHolder prefab itself, and drag the (still default) AircraftController on top of the AircraftHolder’s inspector page.

image

Save the scene, then rebuild the app with File/Build settings etc. Go back to Visual Studio once the building is finished.

Creating the AircraftController

Having programmed OO since the start of this century, I tend to put control where I logically think it belongs, so rather than programming the ‘flight logic’ into a class that loads and translates data too (the AircraftLoader) I opted for putting it into a separate class that would be part of the game object. I tend to conceptualize these combined game objects and components as OO objects - although that is not completely right, it helps me think about it. The start is simple enough

public class AircraftController : MonoBehaviour
{
  private Flight _flightData;
  private float? _speed;
  private float? _heading;
  private TextMesh _text;
  private bool _initComplete;
  private bool _firstMove;

  void Start()
  {
    _text = transform.GetComponentInChildren<TextMesh>();
    _initComplete = true;
  }
}

A few private fields to retain some data.

  • _flightData just keeps the last provide flight data available
  • _speed and _heading keep the last speed and heading. The stream of data sometimes misses a beat and provides no speed and/or heading – I prefer then to display the latest data, in stead of heaving the aircraft suddenly rotate to the North (heading 0) and back again when the next set of data is correct again
  • _text keeps a reference to the text mesh so I don’t have to look it up every update - potentially 60 times a second
  • the _initComplete boolean is a trick I use regularly to prevent other routines using variables like _text before the are initialized. Remember, the Update loop is called independently of what you do. It may look a bit overdone now with only one initialization statement, but believe me – there will be more.

Well then, finally the infamous SetNewFlightData method

public void SetNewFlightData(Flight newFlightData)
{
  if(_initComplete)
  {
    var move = _flightData == null ||
               !_flightData.Location.LocationEquals(
                   newFlightData.Location);
    _flightData = newFlightData;

    ExtractSpeedAndHeading();
    if (move)
    {
      SetLocationOrientation();
    }
    else
    {
      SetNewFlightText();
    }
  }
}

I like to write the code at high level as almost self-explanatory. First we determine if the aircraft needs to be moved, then we ingest the data, and extract speed and heading. Then, when the aircraft needs to be moved, change it’s location, if not, just update the label. The ExtractSpeedAndHeading is pretty straightforward. There is only the thing with heading - that sometimes comes in a negative value, and although Unity3D has no problem with that in positioning the aircraft, I think it looks ugly in the label. So I make sure it's always positive.

private void ExtractSpeedAndHeading()
{
  if (_flightData.Heading != null)
  {
    _heading = (float)_flightData.Heading;
  }
  if (_heading < 0)
  {
    _heading += 360;
  }
  if (_flightData.Speed != null)
  {
    _speed = (float)_flightData.Speed;
  }
}

SetFlightText is also rather trivial

private void SetNewFlightText()
{
  var speedText = 
    _speed != null ? string.Format("{0}km/h", _speed) : string.Empty;

    var headingText =
      _heading != null ? string.Format("{0}⁰", _heading) : string.Empty;

    var text = string.Format("{0} {1} {2}m {3} {4}", _flightData.FlightNr,
      _flightData.Aircraft, _flightData.Location.Alt, speedText, 
      headingText).Trim();
  _text.text = text;
}

Just some clever formatting to prevent empty postfixes like km/h and degrees in the label. By the way – stick to ye olde string.Format and don’t be tempted to use C# 6 string interpolation or you will be sorry (unless you put it between “#if UNITY_UWP – #endif). Anyway, on to the next routine, that actually does all the aircraft manipulation:

private void SetLocationOrientation()
{
  SetNewFlightText();

  transform.localPosition = GetFlightLocation();
  if (_flightData.Heading != null)
  {
    transform.localEulerAngles = GetNewRotation();
  }
  if (!_firstMove)
  {
    transform.localScale = new Vector3(0.0015f, 0.0015f, 0.0015f);
    _firstMove = true;
  }
}

It sets the text too, then set’s the location based on the flight’s location, and set rotation based upon the heading of the aircraft and whether it’s going up or down. Notice I use local position, heading, and scale. This means all those things are relative to the position, rotation and scale of the containing GameObject – HologramCollection. This has the advantage that I can move, rotate and scale the containing object and in one go everything that is in it follows suit. So you don’t have to do all calculations for that – Unity takes care of that form me. I don’t use it in this app just yet, but the actual app already has some experimental code to do that (although that code is not yet in the store version).

Also, note the fact the airplane gets it’s size here (when it’s created it’s 0,0,0). I have the feeling the model is actually a 1:1 scale representation of the actual aircraft – the first time I saw it with the HoloLens I could not find it at first, then turned around and had a “Glimly Glider experience” - a giant aircraft silently swooping down on me from what looked only 10-15 meters. I decide to scale it down to 0.0015 of it’s original size so it appears to be about 10-15cm in a HoloLens – a size more beneficial for getting an good overview, not to mention the blood pressure and heart rate of the average user ;).

Next up are these two routines:

private Vector3 GetFlightLocation()
{
  return GetLocalCoordinates(_flightData.Location);
}

private Vector3 GetLocalCoordinates(Coordinate c)
{
  return new Vector3((float)c.X / 15000,
    c.Alt != null ? (float)c.Alt / 2000.0f : 0f,
    (float)c.Y / 15000);
}

I already discussed the how and why of scaling the down coordinates 15000 times in horizontal direction and 2000 times in vertical direction in a recent blog post about converting lat/lon/alt coordinates into the Unity3D X/Y/Z system, so I am not going through that again, because the next part is a lot more interesting. First I will show the last method, GetVerticalAngle

private float GetVerticalAngle()
{
  var tracksize = _flightData.Track.Count;
  if (tracksize > 2)
  {
    var pLast = _flightData.Track[tracksize - 1];
    var pSecondLast = _flightData.Track[tracksize - 3];
    var delta = pLast.Alt - pSecondLast.Alt;
    if (Math.Abs(delta.Value) > 2.5f)
    {
      return delta < 0 ? 10 : -20;
    }
  }
  return 0;
}

I found that the data, although it provides information about whether the aircraft is actually ascending or descending, that data is not always correct. So I decided to calculate that myself, based upon the difference between the current location and an older location. Now since an aircraft usually descends a lot slower than it takes off (which is very fortunate for the passenger’s – or at least my – peace of mind) this method basically returns –20 when the aircraft is going up, and 10 when it’s going down. And then we get some beautiful Unity3D math again – or more accurately, methods that prevent you from having to use all kinds of advanced 3D math:

private Vector3 GetNewRotation()
{
  var heading = _heading ?? 0;
  var rotation = Quaternion.AngleAxis(heading, Vector3.up).eulerAngles +
      Quaternion.AngleAxis(GetVerticalAngle(), Vector3.right).eulerAngles;
  return rotation;
}

Try to picture in your mind how this works:

  • For the heading we have to rotate around the axis that is going up (and down, too) from the center of the aircraft – this what they call yaw in aviation
  • For the vertical angle – that makes it look whether the aircraft is going up or down – we have to rotate around the axis that goes to the right (and left – so basically over the wings) from the center of the aircraft. In aviation, this is called pitch 

You use summarize Quaternion.AngleAxis(angle, Vector3.<the axis you desire>).eulerAngles over multiple axes to get the combined rotation of an object and assign that in one go. Hence you see in SetLocationOrientation() the statement "transform.localEulerAngles = GetNewRotation();"

The final things

Go to AircraftLoader 's SetNewFlightData and uncomment the line

 //controller.SetNewFlightData(flight);   

Because we have implemented that in the previous section. Then there’s is the method LocationEquals in Coordinate, that we used in AircraftController but that was not implemented yet ;)

public bool LocationEquals(Coordinate other)
{
  if (other == null) return false;
  return (X == other.X && Y == other.Y && Alt != null &&
          other.Alt != null &&
          Alt.Value == other.Alt.Value);
}

And if you run this in a HoloLens or the Emulator, you will see aircraft!

image

You will notice they won't move through the air but jump from position to position. They also don't show their track, there is no Schiphol Airport map, no ATC tower, no church, no gaze cursor - and you cannot select anything yet - but that is because this post is long enough as it is. The base is here. 3D visualization of a JSON stream. What is left, is basically making things more slick :)

Conclusion

In hindsight I might better have splitted this episode in two blog post still, but I hope you have made it to the end. I feel this blog post is the heart of the matter – reading a data stream and turning it into a 3D model – making ‘dry records’ come to life, almost literally. I also showed you some key concepts about positioning and rotating stuff in 3D.

As usual, the code is here on GitHub.

08 October 2016

A HoloLens airplane tracker 3–Creating an annotated airplane

Intro

In this post we are going to create a game object holding the aircraft and a label showing aircraft data that always looks at the user. We are not going to worry about scaling yet – we will handle that from code, in the next blog post.

First some housekeeping

Just as in the CubeBouncer project, we are going to create some folders in the Assets first to keep our custom assets nicely separated. Create a folder App, and in this folder the following subfolders:

  • Audio
  • Materials
  • Prefabs
  • Scripts
  • Textures
  • UWPAssets

Creating the Aircraft Holder and including the aircraft

Inside the HologramCollection, create an empty GameObject called AircraftHolder. Then drag into that object the A320 object form the A320 folder. A rather large A320 appears in the Scene, as indicated in the previous blog post. Just ignore this for now.Open the A320 object (in the hierarchy, not the object in the A320 folder), disable the animator, and make sure X,Y,Z position and rotation are all 0,0,0 and scale 1,1,1:

image

At this point I have no idea what the Animator is supposed to do, but we don’t need it – I think.

Create a label

In this section we create the label above the airplane that displays flight information, like flight number, aircraft type, speed, altitude, etc. Right-click the AicraftHolder and create a “3D Text” Object. Call this “Label”. That label now sits at the bottom of the airplane, where apparently it’s origin is.

Open the label in the inspector, set the Y position to 15 and the Y rotation to 270. That places the text 15 meters above the airplane, placing it parallel above the fuselage, making the default text “Hello World” readable from the right side of the airplane

image

image

The go down to the “Text Mesh”. Change the following things:

  • Anchor to lower center
  • Change font size to 80
  • Change the color to red

image

image

Now hit “Add component”, and add a Mesh Collider. Select the “Convex” checkbox:

image

Of course, it’s not very practical if the airplane’s data is only readable from the left side. So to close off this article, we actually going to write code – by adding a simple script that keeps the text readable from every angle. Go to the Assets/App/Scripts folder, right-click it and hit “Create Script”. Call the script “LookAtCamera”. Double-click it – that will open Visual Studio – and change the update method to this:

void Update()
{	
  gameObject.transform.LookAt(Camera.main.transform);
  gameObject.transform.Rotate(Vector3.up, 180f);
}

imageSave the file, go back at Unity. Drag the new script on top of the Label inside the AircraftHolder game obejct To see this actually works: hit the Unity “Play” button

Now the fun thing is – as long as you are in play mode, changes in settings are not stored so this is great for fooling around. Initially your game screen looks like this – you are looking from under the plane right up to the front landing gear.

image

Select the AircraftHolder in the hierarchy, then change Z to 500.

image

The plane flicks forward… and the text is still readable although we are not looking form the left, but the back. Change rotation X to –20 and rotatation Y to –30:

image

The plane is rotated, we are now looking more or less form the right, but the text is still pointing straight to you, and is readable. Mission accomplished. Not bad for two lines of code!

Exit Play mode. Go to the label once more, and remove the text “Hello World”. The label seems to disappear, but it’s still there, it’s now just empty. It will be filled by code.

And finally…

Select the Prefab folder in Assets/App, and drag the AircraftHolder into it. After you have done it, remove it from the HologramCollection. And save the Scene.

Net result:

image

The reason why I am making the AircraftHolder in stead of adding components directly to the A320 model – which is perfectly possible – is that by using this way I can easily switch out the actual airplane model, while all logic and additional components are preserved. If you find a different (more pretty) aircraft 3D model this way saves you work.

Code so far can be downloaded from Github here.