A HoloLens airplane tracker 8–adding the church and the billboard

Intro

All the code to track aircraft in a HoloLens is already yours. Now the only thing I have left to write is how the Wortell Church and the billboard that appears on the map came to be. It’s not technically a part of the aircraft tracker per se, but a fun exercise in code that you might use if your company is near an airport too.

Required assets

We need two things:

• A Church
• A Billboard (and a logo to put on it)

Getting and importing the church

This was actually the hardest one. It turns on that there are lots of churches on cgtrader, but if you download for instance this church as FBX or OBJ and import it into Unity, you don’t get the promised church on the left– you get the ghost thing on the right!

As one of my colleagues - who lived in the UK for a long time before joining us - would put it: “it’s not quite what I had in mind”. And this one actually looks quite good – I have seen churches with not only textures but complete parts missing. Apparently not everyone checks the result of every format they convert to before uploading. Fortunately, the author writes he or she has created the model in Sketchup. Ain’t that nice, as you can download a free trial version of that program. So download that – and the original Sketchup model. Unpack the zip file, and rename the 3d-model.skp that’s in there to church.skp. Then double-click from the explorer and Sketchup will open, after possibly asking you about trial licensing first. I had to zoom out a little, but this is what I got

And that was indeed what I had in mind. Now hit File/Export/3D model, select FBX file (*.fbx) for “Save as “type”, choose a name that suits you and hit save

If you would import the resulting church.fbx into Unity you get this:

Although not nearly as pretty as in Sketchup, that’s more like it! To keep your project clean and make it easier to swap out this church for another one (or other building)later, don’t drag the FBX into the root of your assets, but first add a folder, then drag in the church, like displayed on the right.This way, the materials of all the imported objects don’t get mixed up.

To keep the church and the (future) billboard together, I created an empty GameObject “Wortell” in HoloGramCollection. I dragged the church in there. You will find the church too big (like way way too big), so scale it to 0.025 in all directions.Then set for location X to –0.076 and Z to 0.2915. Finally, set Y rotation to 158.

 

 

 

 

 

 

 

 

 

This will put the church nicely where it’s more or less supposed to be, although I think it’s about three times too big with respect to the map but otherwise it would be nearly invisible. For the record – this is not a model of the actual church that is converted to the Wortell offices, but it resembles it a little.

Getting a billboard and a logo

Getting this one is actually very easy. Go to the Unity Asset store – that is, click Window/Asset Store and type “Billboard” in filter. You will get an enormous amounts of hits but you just need the very first one, that I indicated in red. Click on it, then click “download” (or “import”, if you have downloaded it before) .

When you actually import, make sure to deselect the sample scene, as we really don’t need it:

Drag the billboard prefab from Assets/Billboard inside the Wortell object on the hierachy. You will see that once again this is way too big. Scale it to 0.03 for x, y and z, give it the same position as the church and set rotation to 0 for x, 0 for y, and 90 for z. The billboard will now look like this:

So the only thing missing is the Wortell logo – or whatever you want to put on it. I am not sure if this is the right procedure, but I went with this:

• Add a 512x512 image with your logo to Assets/App/Textures. I took this one. Notice the Wortell text is just about halfway – this should put the text at the bottom of the billboard (as is customary with our logo)
• Click on the Billboard in the hierarchy.
• Expand the “board” material in the inspector
• Drag your logo texture on the square that says “None (texture)”
• Set X and Y tiling both to 2

Net result should be like left, and the billboard standing above the church should now look to the right:

Getting the logo the way you want it to be takes a little fiddling. I really can’t tell you why you need to set those tiling values – I just found out it worked this way.

Adding some life to it

It looks pretty nice, but that billboard has all the interactivity of a wooden shed – it’s sitting there doing nothing. We want it to do two things:

• When it’s showing, it needs to be rotated to face the user – so it’s always readable
• Initially it won’t be visible at all, air tapping the church will make it visible (and invisible again, like a toggle)

After all, that’s what’s in the original video, and this very feature was even demonstrated live on Windows Weekly by none other than Mary Jo Foley herself  ;) when she and Paul Thurrott visited the Wortell offices on November 16, 2016.

Making the billboard face the user

This requires actually zero programming effort, as a billboard behaviour is already in the HoloToolkit. So basically all you have to do is:

• Got to Assets/HoloToolkit/Utilities/Scripts
• Locate the Billboard.cs script
• Drag it on top of your billboard game object in the hierarchy
• Set the script’s pivot axis to X

And you are done. If you deploy the app now to a HoloLens or emulator you will see the billboard now follows the camera. Now this may seem odd, as the billboard should rotate around the axis that runs from to top bottom – in other words, Y. But since the billboard is default rotated 90⁰ around it’s Z-axis and rotates around it’s own axis rather than a world axis, this makes sense. Kind of ;)

If you are smart, you are now are wondering why I did not use that for the aircraft label, as the billboard script also supports full free axis movement. That is simple – at the time I wrote the code for the label, I simply had not thought of looking into the HoloToolkit. So I wrote a custom script. And I kept it that way, as a kind of way to show you how I am still learning as well :)

Preparing the billboard folding in or out by air tapping

So what we want to achieve is to be able to air tap the church, but this should have an effect on the billboard. I have solved this as follows:

• Create a collider on the church
• Make a behaviour attached to the billboard that handles the folding in/out of the billboard
• Make a little behaviour attached to the church that receives the air tap and passes that to the behaviour mentioned in the previous bullet point.

First the collider. As explained in the previous episode , to be able to show a cursor on an object or select it, it needs to have a collider. I once again took the box collider for easy selection.

Only then we notice something peculiar when we look at the church from the top:

Meh. That’s annoying. Apparently the creator of this model has created it on an angle, or there’s something wrong with the conversion. Either way we will have to fix it. We do this by digging into the church object in the hierarchy and rotate the Group1 component 25⁰ over it’s Y-axis so it’s visually aligned with the collider:

Now of course church itself is no longer aligned with the road. But if you subtract the 25⁰ from the 158⁰ the church object it is rotated:

 

everything is now correctly rotated and aligned. And now for some action.

Adding the behaviours

We are almost there. First, we need the behavior that does the actual folding and unfolding of the billboard:

using System;
using UnityEngine;

public class BillBoardToggler : MonoBehaviour
{
  private float _startScale;
  void Start()
  {
    _startScale = transform.localScale.x;
    Hide();
  }

  public void Toggle()
  {
    if (Math.Abs(transform.localScale.x) < 0.001f)
    {
      // Pop up
      transform.localScale = new Vector3(0, _startScale, _startScale);
      iTween.ScaleTo(gameObject,
          new Vector3(_startScale, transform.localScale.y,
              transform.localScale.z), 1f);
    }
    else
    {
      // Pop down
      iTween.ScaleTo(gameObject, iTween.Hash("scale",
          new Vector3(0, transform.localScale.y, transform.localScale.z),
          "time", 1f, "oncomplete", "Hide"));
    }
  }

  private void Hide()
  {
    transform.localScale = new Vector3(0, 0, 0);
  }
}

So at start we save the x scale, and hide the complete billboard by scaling it to 0,0,0, effectively making it infinitely small – and thus invisible. Now if the Toggle method is called:

• If the scale bigger than 0.001, the billboard apparently was popped down. So we first set the Y and Z scale to their original value – and keep X to 0, then proceed to animate the X scale from 0 to its original value, using our old friend iTween, which we saw already in the 5th episode of this series. This lets the billboard rise out of the ground
• otherwise, the billboard was already popped up. So we animate the x scale to 0. If you look closely, you will see the billboard being reduced to its outline on the airport map. But just after that, Hide is called, letting the outline also pop out of existence.

Drag this behaviour on top of the billboard. Only one thing left to do – make sure this thing gets called when the user air taps the church. For that, we only need this simple behaviour:

using UnityEngine;

public class ChurchController : MonoBehaviour
{
  private BillBoardToggler _toggler;

  // Use this for initialization
  void Start()
  {
    _toggler = transform.parent.GetComponentInChildren<BillBoardToggler>();
  }

  private void OnSelect()
  {
    if (_toggler != null)
    {
      _toggler.Toggle();
    }
  }
}

Drag this on on top of the church. On start it will try to find a BillBoardToggler script on a sibling, and when the OnSelect method is called – a standard message sent by the HoloToolkit GestureManager – it will call the Toggle method on that, letting the billboard pop up (or make it go down again). So if you move your gaze cursor to the church and air tap on it:

Conclusion

So that’s it. No more beans to spill. This the the app and all of the app, except for the live data feed. In fact, it’s better then the app in the store, because backtracking the road I took making this app I skipped a few detours, false starts and messy things that have not turned up in this cleaned up tutorial code. That is why I can recommend everyone writing an extensive tutorial once in a while – because not only the students will learn, but the teacher as well.

The final and completed code of this app can be found here. I don’t rule out the possibility that I will write about extensions to this app at some later point, but this is what I wanted to share. I enjoyed immensely writing about it, and I hope you enjoyed reading. If you made it this far – kudos to you. I only ask you one thing – if this inspires you to write HoloLens apps too – consider writing a tutorial as well. Maye I will learn something from you as well ;)

A HoloLens airplane tracker 7–activating an aircraft by air tapping

Intro

In this episode I am going to describe how the airplanes are selected in the sky by air tapping them. This requires some changes made to both the Azure Mobile App service and the app. The change to the app service is necessary as I want to share the selected airplane between all possible HoloLenses using the app. Why? Because ;). You actually can do this with the app in the store. This is why sometimes airplanes get selected even when you don’t select them – that is because someone else, somewhere in the world, selects an airplane.

Adapting the service

We need to be able to post to the service which aircraft is active, and in the data that is pulled from the server the active airplane needs to be indicated. Remember in the very first episode the “Flight” object had an unused “IsActive” field? It won’t be unused very much longer. First, you add a reference to System.Runtime.Caching

Then, add using statements:

using System.Runtime.Caching;
using System.Net.Http;
using System.Net;

Proceed by adding the following two fields:

private ObjectCache Cache = MemoryCache.Default;

private const string ActiveIdKey = "ActiveId";

We will need to add a method to post the desired active plane’s Id to the server:

[HttpPost]
public HttpResponseMessage Post(string activeId)
{
  Cache[ActiveIdKey] = activeId;
  return Request.CreateResponse(HttpStatusCode.OK);
}

Then, in the Get method, insert right before the “return flights” statement the following code:

var activeKey = Cache[ActiveIdKey] as string;
if (activeKey != null)
{
  var activeFlight = flights.FirstOrDefault(p => p.Id == activeKey);
  if (activeFlight != null)
  {
    activeFlight.IsActive = true;
  }
}

This is a rather naive way of ‘storing’ data on a server but I am just do darn lazy to set up a database if I only want to have one simple id kept active. So I am using the MemoryCache. If you post an id to the Post method, it will simply set the cache key – which will be copied into the data in the next data download, provided it’s actually an id used in the data. Publish your service, and let’s go the Unity project again

Adding a gaze cursor – and messing a bit with it

The HoloToolkit has a number of gaze cursors, but I am not quite happy with those, so I made one for myself – by adding a bit into an existing one. The procedure is as follows:

• From HoloToolkit/Input/scripts, drag Handsmanager on top of the HologramCollection
• From HoloToolkit/Input/scripts, drag Gazemanager on top of the HologramCollection
• From HoloToolkit/Input/Prefabs, drag Cursor on top of the HologramCollection

Then we create a little extra behavior, called HandColorCursor

using UnityEngine;
using HoloToolkit.Unity;

public class HandColorCursor : Singleton<HandColorCursor>
{
  // Use this for initialization
  private GameObject _cursorOnHolograms;

  public Color HandDetectedColor;

  private Color _originalColor = Color.green;

  void Start()
  {
    _cursorOnHolograms = CursorManager.Instance.CursorOnHolograms;
    _originalColor = 
      _cursorOnHolograms.GetComponent<MeshRenderer>().material.color;
  }

  // Update is called once per frame
  void LateUpdate()
  {
    if (_cursorOnHolograms != null && HandsManager.Instance != null)
    {
      _cursorOnHolograms.GetComponent<MeshRenderer>().material.color =
        HandsManager.Instance.HandDetected ? Color.green : _originalColor;
    }
  }
}

Drag this behavior on top of the Cursor prefab. Now if you build and run the app and point the gaze cursor to the airport it should become a kind of blue circle. If you move your hand into view, the cursor should appear and go green.

If you move it back, the cursor should again become blue. Unfortunately, if you point it at anything else, you will still see the fuzzy ‘CursorOffHologram’ image. That it’s because the airport is the only thing that has a collider. To make aircraft selectable, we will need to do some work at AircraftHolder.

Updating the airplane

To be selectable, it needs to have a collider. I have chosen a box collider that is pretty big with respect to the actual airplane model – to make it easier to actually select the airplane (as it can be quite some distance away, or at least seem to be) and a box makes it possible to actually see the cursor sitting pretty stable on the selection area as it hits the collider (as opposed to a mesh collider that will make it wobble around). Go to your AircraftHolder prefab in the App/Prefab folder, and add an 80x80x80 Box Collider

And you will indeed see the cursor appear over the airplane

Getting some selecting done – code first

From HoloToolkit/Input/scripts, drag GestureManage on top of the HologramCollection. GestureManager sends an OnSelect message to any GameObject that happens to be hit by a gaze (and where, incidentally, the cursor is hanging out out) using this code:

private void OnTap()
{
  if (FocusedObject != null)
  {
    FocusedObject.SendMessage("OnSelect");
  }
}

So there needs to be an OnSelect method in the AircraftController to handle this. First we will need to add the following fields to the AircraftController:

private bool _isActive;
private AudioSource _sound;
private Dictionary<MeshRenderer, Color> _originalColors;

To the Start method, we add the following code, just in front of the “_initComplete = true;” statement

_sound = GetComponent<AudioSource>();
_originalColors = new Dictionary<MeshRenderer, Color>();
foreach (var component in GetComponentsInChildren<MeshRenderer>())
{
  _originalColors.Add(component, component.material.color);
}

So this actually initializes some things by getting the audio source and reads the color of all sub components – so we can revert it easily later. Then add this method:

private void SetHighlight()
{
  if (_isActive != _flightData.IsActive)
  {
    foreach (var component in GetComponentsInChildren<MeshRenderer>())
    {
      if (component.material.color == Color.white || 
          component.material.color == Color.red)
      {
        component.material.color = _flightData.IsActive ?
           Color.red : _originalColors[component];
      }
    }
    if (_flightData.IsActive)
    {
      _sound.Play();
    }
    else
    {
      _sound.Stop();
    }
    _isActive = _flightData.IsActive;
  }
}

If the _isActive field of the current airplane differs from the new _flightData.IsActive, then flip white things to red – or red things back to white, depending on the actual value of _flightData.IsActive. And it turns on the sound – the annoying sonar ping you saw in the movie I made in the first episode. And fortunately it can also turn in it off ;)

Next, add a call to this new method at the end of the already existing “SetNewFlightData” method. Then, add this code:

private void OnSelect()
{
  SendMessageUpwards("AircraftSelected", _flightData.Id);
}

This is like SendMessage, but then upwards through the object hierarchy, in stead of down to the children. So this will look for an AircraftSelected method in the parent.

Save the file. Now we go to the DataService, and add the following method. Make sure it is sitting between #if UNITY_UWP – #endif directives.

#if UNITY_UWP
public async Task SelectFlight(string flightId)
{
  var parms = new Dictionary<string, string>
  {
    { "activeId", flightId }
  };
  await _client.InvokeApiAsync<List<Flight>>("FlightData", 
           HttpMethod.Post, parms);
}
#endif

This does the actual calling of the App service. And then we need to go to AircraftLoader, which plays the select sound and passes the call trough to the DataService:

public void AircraftSelected(object id)
{
  _sound.Play();
#if UNITY_UWP
  DataService.Instance.SelectFlight(id.ToString());
#endif
}

Save everything, go back to Unity.

Getting some selecting done – connecting the dots in Unity

There are a few things we need to be doing in Unity, but one thing is very important. One I forgot in my first CubeBouncer demo app and in the app that’s in the store, simply because I was not aware of it. Fortunately someone who shall remain nameless here made me aware of it.

First, in the main Unity window, hit Edit/Project Settings/Audio. Then check “Ms HRTF Spatialization” for “Spatializer plugin”

Then open the AircraftHolder prefab again, and add an AudioSource. Important advice, also from the person mentioned above: hit the “Spatialize” checkbox. Then select the “Loop” checkbox and uncheck the “Play On Awake” checkbox. Finally, find a sound you would like to be played by a selected airplane. I took a sonar ping. I put it in the Assets/App/Audio folder, and dragged it on the Audio Clip property. I also moved the “Spatial Blend” slider all the way to the right but I don’t think that is necessary anymore now I use the Spatializer. Net result:

And sure enough, if you now air tap an airplane it shows like below, and it starts pinging:

Adding confirmation sound to selection

I think it’s good practice to add a confirmation sound of some sorts to any user input that is ‘understood’. This is especially necessary with speech commands, but I like to add it to air taps. Sometimes the effect of what you do is not immediately visible, and an audio confirmation the HoloLens has understood you and is working on making your command getting executed is very nice. I tend to add a kind of “pringgg” sound. I used the “Ready” file that is in Assets/App/Audio.

First, add an AudioSource to the HologramCollection and drag the “Ready” sound on top of the “AudioClip” field. Don’t forget to uncheck the “Play On Awake” checkbox.Then go to the AircraftLoader behaviour. Since this receives the message from every AircraftController and passes it through to the DataService, this is a nice central place to play the confirmation sound. So we add a new field:

private AudioSource _sound;

that needs to be initialized in the Start method:

_sound = GetComponent<AudioSource>();

and finally in the AircraftSelected method, insert this line at the top of the method:

_sound.Play();

And we are done. Now if you airtap on the aircraft your HoloLens will sound a ‘pringgg’ sound. That is, if you tapped correctly.

Conclusion

And that’s that. Selecting an airplane by by air tapping it it, and posting an id to the Azure Mobile App service is pretty easy indeed. A lot of the interaction components are basically dragged in from the HoloToolkit, very little actual programming is required. This is the power of the HoloToolkit – I use as much from what is already there, and make as little as possible myself. As any good programmer I am a lazy programmer ;)

This blog post written far, far away from home, in Sammamish, WA, USA, shortly after the MVP summit that left me very inspired – again. After posting this I will soon retire – and go to SEA-TAC tomorrow for my flight back home in the Netherlands.

Code can be found, as always, here on github.