Creating a geographical map on the floor in your Hololens / Windows MR app

Intro

If you make an awesome app like Walk the World, of course you are not going to spill the beans on how you built it, right? Well – wrong, because I think sharing knowledge is the basis of any successful technical community, because I like doing it, and last but not least – I feel it is one the primary things that makes an MVP an MVP. I can’t show you all the details, if only because the app is humongous and is badly in need of some overhaul / refactoring, but I can show you some basic principles that will allow you to make your own map. And that’s exactly what I am going to do.

‘Slippy map’?

Getting on my GIS hobby horse here :) Gather around the fire ye youngsters and let this old GIS buff tell you all about it. ;)

Slippy maps are maps made out of pre rendered square images that together form a continuous looking map. Well-known examples are Bing Maps and Google Maps, as well as the open source map Open Street Maps – that we use for this example, The images are usually 256x256 pixels. Slippy maps have a fixed number of discrete zoom levels. For every zoom level there is a number of tiles. Zoom level 0 is typically 1 image showing the whole world. Zoom level 1 is 4 tiles each showing 1/4th of the world. Zoom level 2 is 16 tiles each showing 1/16th of the world. You get the idea. Generally speaking a zoom level has 2^zoomlevel x 2^zoomlevel tiles.

You can see the number of tiles (and the amount of data servers need to store those) go up very quickly. Open Street Maps’ maximum zoom level is 19 – which all by itself is 274.9 billion tiles, and that is on top of all the other levels. Google Map’s maximum zoom level is 21 at some places and I think Bing Maps goes even further. The amount of tiles of level 21 alone would be 4398046511104, a little short of 5 trillion. And that is not all - they even have multiple layers – map, terrain, satellite. Trillions upon trillions of tiles - that is even too much to swallow for Microsoft and Google, which is why they vary the maximum zoom level depending on the location – in the middle of the ocean there’s considerable less zoom levels available ;). But still: you need really insane amounts of storage and bandwidth to serve a slippy map of the whole world with some amount of detail, which is why you have so little parties actually meeting this challenge – mostly Google, Microsoft, and Open Street Maps.

Anyway - in a slippy map tiles are defined in rows and columns per zoom level. The takeaway from this is to realize that a map tile can be identified by three parameters and three parameters only: X, Y and zoom level. And those need to be converted into a unique URL per tile. The way these tiles are exactly organized depends on the actual map supplier. And if your starting point is a Lat/Lon coordinate, you will have to do some extra math. Basically all you need to know you can find here at the Open Streep Maps wiki. But I am going to show in more detail anyway.

Setting up the project

For this project we will use Unity 2017.1.0.f3. Do not forget to install the Windows Store (.NET) Target Support as well. If you are finished, clone my basic setup from Setting up a HoloLens project with the HoloToolkit - June 2017 edition. Rename the folder to “SlippyMapDemo”, then

• Open the project in Unity
• Open the Build Settings window (CTRL+B)
• Make sure it looks like this:

 

• Hit the "Player Settings..." button
• Change "June2017HoloLensSetup" in "SlippyMapDemo" where ever you see it. Initially you will see only one place, but please expand the "Icon" and "Publishing Settings" panels as well, there are more boxes to fill in. 

Upgrading the HoloToolkit

To make a working app on the new Unity version, we will need a new HoloToolkit. So delete everything from the Assets/Holotoolkit but do it from Unity, as this will leave the .gitignore in place.

Select all items, press delete. Then, go to http://holotoolkit.download/ and download the latest HoloToolkit. If you click “Open” while downloading, Unity will automatically start to import it. I would once again suggest de-selecting Holotoolkit-Tests as they are not useful for an application project.

Some basic stuff first

We will need to have a simple class that can hold a Lat/Lon coordinate. UWP supplies those, but that means we cannot test in the editor and, well, we don’t need all what they can do. So we start off with this extremely simple class:

public class WorldCoordinate
{
    public float Lon { get; set; }
    public float Lat { get; set; }

    public override string ToString()
    {
        return string.Format("lat={0},lon={1}", Lat, Lon);
    }
}

Lat/Lon to tile

A map has a location – latitude and longitude, so that is our starting point. So we need to have a way to get the tile on which the desired location is. A tile is defined by X, Y and zoom level, remember? We start like this:

using System;
using UnityEngine;

public class TileInfo : IEquatable<TileInfo>
{
    public float MapTileSize { get; private set; }

    public TileInfo(WorldCoordinate centerLocation, int zoom, float mapTileSize)
    {
        SetStandardValues(mapTileSize);

        var latrad = centerLocation.Lat * Mathf.Deg2Rad;
        var n = Math.Pow(2, zoom);
        X = (int)((centerLocation.Lon + 180.0)/360.0*n);
        Y = (int)((1.0 - Mathf.Log(Mathf.Tan(latrad) + 1 / Mathf.Cos(latrad)) / Mathf.PI) / 2.0 * n);
        ZoomLevel = zoom;
    }

    private void SetStandardValues(float mapTileSize)
    {
        MapTileSize = mapTileSize;
    }

    public int X { get;  set; }
    public int Y { get;  set; }

    public int ZoomLevel { get; private set; }

}

Via a simple constructor X and Y are calculated from latitude, longitude and zoomlevel, The MapTileSize is the apparent physical size of the map tile - that we will need in the future (it will be 0.5 meters, as we will see later).

Wow. That seems like some very high brow GIS calculation, that only someone like me understands, right? ;)Maybe, maybe not, but you find this formula on the Open Street Map wiki, more specifically, here.

So now we have the center tile, and to calculate the tiles next to it, we simply need another constructor

public TileInfo(int x, int y, int zoom, float mapTileSize)
{
    SetStandardValues(mapTileSize);
    X = x;
    Y = y;
    ZoomLevel = zoom;
}

in a simple loop, as we also will see later, we can find the tiles next, above and below it and we can define a MapBuilder class that can loop over this information, and make map tiles grid. BTW, this class also has some equality logic, but that’s not very exiting in this context, so you can look that up in the demo project.

Tile to URL

As I have explained, a tile can be identified by three numbers: X, Y and zoom level but to actually show the tile, you need to convert that to a URL. So I defined this interface to make the tile retrieval map system agnostic:

public interface IMapUrlBuilder
{
    string GetTileUrl(TileInfo tileInfo);
}

and this single class implementing it for Open Street Map:

using UnityEngine;

public class OpenStreetMapTileBuilder : IMapUrlBuilder
{
    private static readonly string[] TilePathPrefixes = { "a", "b", "c" };

    public string GetTileUrl(TileInfo tileInfo)
    {
        return string.Format("http://{0}.tile.openstreetmap.org/{1}/{2}/{3}.png",
                   TilePathPrefixes[Mathf.Abs(tileInfo.X) % 3],
                   tileInfo.ZoomLevel, tileInfo.X, tileInfo.Y);
    }
}

this is not new code. Regular readers (or better – long time readers) of this blog may have seen it as early as 2010, when I showed how to do this for Window Phone 7.

Some (very little) re-use

Basically we are going to download images from the web again, using the URL that the IMapUrlBuilder can calculate from the TileData. Downloading an showing images in a HoloLens app – been there, done that, and it fact, that’s what made the idea of Walk the World popup up in my mind in the first place. So I am going to reuse the DynamicTextureDownloader from this post. In the demo project, it sits in the HolotoolkitExtensions. We make a simple child class:

using HoloToolkitExtensions.RemoteAssets;
using System.Collections;
using UnityEngine;

public class MapTile : DynamicTextureDownloader
{
    public IMapUrlBuilder MapBuilder { get; set; }

    private TileInfo _tileData;

    public MapTile()
    {
        MapBuilder = MapBuilder != null ? MapBuilder : new OpenStreetMapTileBuilder();
    }

    public void SetTileData(TileInfo tiledata, bool forceReload = false)
    {
        if (_tileData == null || !_tileData.Equals(tiledata) || forceReload)
        {
            TileData = tiledata;
        }
    }

    public TileInfo TileData
    {
        get { return _tileData; }
        private set
        {
            _tileData = value;
            ImageUrl = MapBuilder.GetTileUrl(_tileData);
        }
    }
}

So what happens – if you set TileData using SetTileInfo, it will ask the MapBuilder to calculate the tile image URL, the result will be assigned to the parent class’ ImageUrl property, and the image will automatically be drawn as a texture on the Plane it’s supposed to be added to.

Creating the MapTile prefab

In HologramCollection, create a Plane and call it MapTile. Change it’s X and Z scale to 0.05 so the 10x10 meter plane will show as 0.5 meters indeed. Then add the MapTile script to it as a component. Finally, drag the MapTile Plane (with attached script) from the HologramCollection to the Prefabs folder in Assets.

If you are done, remove the MapTile from the HologramCollection in the Hierarchy.

A first test

To the HologramCollection we add another empty element called “Map”. Set it’s Y position to –1.5, so the map will appear well below our viewpoint. To that Map game object that we add a first version of our MapBuilder script:

using UnityEngine;

public class MapBuilder : MonoBehaviour
{
    public int ZoomLevel = 12;

    public float MapTileSize = 0.5f;

    public float Latitude = 47.642567f;
    public float Longitude = -122.136919f;

    public GameObject MapTilePrefab;


    void Start()
    {
        ShowMap();
    }

    public void ShowMap()
    {
        var mapTile = Instantiate(MapTilePrefab, transform);
        var tile = mapTile.GetComponent<MapTile>();
       tile.SetTileData(
           new TileInfo(
               new WorldCoordinate{ Lat = Latitude, Lon = Longitude }, 
               ZoomLevel, MapTileSize)
           );
    }
}

This simple script basically creates one tile based upon the information you have provided. Since it does nothing with location, it will appear at 0,0,0 in the Map, which itself is at 1.5 meters below your viewpoint. If you were to run the result in the HoloLens, a 0.5x0.5m map tile map should appear right around your feet.

Anyway, drag the prefab we created in the previous step on the Map Tile Prefab property of the Map Tile Builder …

… and press the Unity Play button.You will see nothing at all, but if you rotate the Hololens Camera 90 degrees over X (basically looking down) while being in play mode a map tile will appear, showing Redmond.

Only it will be upside down, thanks to the default way Unity handles Planes - something I still don't understand the reason for. Exit play mode, select the Map game object and change it’s Y rotation to 180, hit play mode again and rotate the camera once again 90 over X.

That’s more like it.

The final step

Yes, there is only one step left. In stead of making one tile, let’s make a grid of tiles.

We add three more properties to the MapBuilder script:

public float MapSize = 12;

private TileInfo _centerTile;
private List<MapTile> _mapTiles;

And then here’s the body of MapBuilder v2

void Start()
{
    _mapTiles = new List<MapTile>();
    ShowMap();
}

public void ShowMap()
{
    _centerTile = new TileInfo(new WorldCoordinate { Lat = Latitude, Lon = Longitude }, 
        ZoomLevel, MapTileSize);
    LoadTiles();
}

private void LoadTiles(bool forceReload = false)
{
    var size = (int)(MapSize / 2);

    var tileIndex = 0;
    for (var x = -size; x <= size; x++)
    {
        for (var y = -size; y <= size; y++)
        {
            var tile = GetOrCreateTile(x, y, tileIndex++);
            tile.SetTileData(
                 new TileInfo(_centerTile.X - x, _centerTile.Y + y, ZoomLevel, MapTileSize),
                 forceReload);
            tile.gameObject.name = string.Format("({0},{1}) - {2},{3}", x, y, tile.TileData.X,
                tile.TileData.Y);
        }
    }
}

private MapTile GetOrCreateTile(int x, int y, int i)
{
    if (_mapTiles.Any() && _mapTiles.Count > i)
    {
        return _mapTiles[i];
    }

    var mapTile = Instantiate(MapTilePrefab, transform);
    mapTile.transform.localPosition = new Vector3(MapTileSize * x - MapTileSize / 2, 0, 
                                                  MapTileSize * y + MapTileSize / 2);
    mapTile.transform.localRotation = Quaternion.identity;
    var tile = mapTile.GetComponent<MapTile>();
    _mapTiles.Add(tile);
    return tile;
}

In ShowMap we first calculate the center tile’s data, and then in LoadTiles we simply loop over a square matrix from –(MapSize/ 2) to +(MapSize/2) and yes indeed, if you define a MapSize of 12 you will actually get a map of 13x13 tiles because there has to be a center tile. If there is a center tile, the resulting matrix by definition has an uneven number of tiles.

In LoadTiles you see TileInfo’s second constructor in action: just X, Y, and Zoomlevel. Once you have the first center tile, calculating adjacent tiles is extremely easy. GetOrCreateTiles does the actual instantiation and positioning in space of the tiles – that’s what we need the MapTileSize for. Note, that for an extra performance gain (and a prevention of memory leaks), it actually keeps the instantiated game objects in memory once they are created, so if you call ShowMap from code after you have changed one of the MapBuilder’s parameters, it will re-use the existing tiles in stead of generating new ones. LoadTiles itself also creates a name for the MapTile but that’s only for debugging / educational purposes – that way you can see which tiles are actually downloaded in the Hierachy while being in Unity Play Mode.

If you deploy this into a HoloLens and look down you will see a map of 6.5x6.5 meters flowing over the floor.

To make this completely visible I had to move the camera up over 20 meters ;) but you get the drift.

Some words of warning

• The formula in TileInfo calculates a tile from Latitude and Longitude. That only guarantees that location will be on that tile but it doesn’t say anything about where on the tile it is. You can see Redmond on the center tile, but it’s not quite in the center of that center tile. It may have well been op the top left. The more you zoom out, the more this will be a factor.
• This sample shows Open Street Map and Open Street Map only. You will need to build IMapUrlBuilder implementations yourself if you want to use other map providers. Regular readers of this blog know this very easy to do. But please be aware of the TOS of map providers.
• Be aware that the MapBuilder with map setting of 12 downloads 13x13=169 tiles from the internet. On every call. This app is quite the bandwidth hog - and probably a power hog as well.

Conclusion and some final thoughts

Building basic maps in Unity for use in your HoloLens / Windows Mixed Reality apps is actually pretty easy. I will admit I don’t understand all the fine details of the math either, but I do know how slippy maps are supposed to work, and once I had translated the formula to C#, the rest was actually not that hard, as I hope to have shown.

Almost all data somehow has a relation with a location, and being able to generate a birds’ eye environment in which you can collaborate with you peers, will greatly enhance productivity and correct interpretation of data. Especially if you add 3D geography and/or buildings to it. It looks like reality, it shows you what’s going on, and you less and less have to interpret 2D data into 3D data. We are 3D creatures, and it’s high time our data jumps off the screen to join us in 3D space.

I personally think GIS and geo-apps are one of the premier fields in which AR/VR/MR will shine – and this will kick off an awesome revolution in the GIS world IMHO. Watch this space. More to come!

Demo project here. Enjoy!

Styles in Xamarin Forms don't work properly in UWP .NET Native - here is how to fix it

Intro

Xamarin Forms is awesome. If you have learned XAML from WPF, Silverlight, Windows Phone, Universal Windows Apps or UWP, you can jump right in using the XAML you know (or at least something that looks remarkably familiar) and start to make apps that will run cross platform on iOS, Android and UWP. So potentially your app cannot only run on phones but also on XBox, HoloLens and PCs.

OnPlatform FTW!

One of the coolest thing is the OnPlatform construct. For instance, you can have something this:

<Style TargetType="Label" x:Key="OtherTextStyle" >
    <Setter Property="FontSize">
        <OnPlatform x:Key="FontSize" x:TypeArguments="x:Double" >
            <On Platform="Windows" Value="100"></On>
            <On Platform="Android" Value="30"></On>
            <On Platform="iOS" Value="30"></On>
        </OnPlatform>
    </Setter>
</Style

This indicates the label that has this style applied to it, should have a font size of 100 on Windows, 30 on Android, and 30 on iOS. In the demo project I have defined some styles in the App.xaml, and the net result is that is looks like this on Android (left), iOS(right) and Windows (below).

The result is not necessarily very beautiful, but if you look in the MainPage.xaml you will see everything has a style and no values are hard coded. You can also see that although the Android and iOS apps are mobile apps and the Windows app is essentially an app running on a tablet or a PC (the demarcation line between these is becoming hazier with the day) it will still work out using OnPlatform.

I have used various constructs. Apart from the inline construct as I showed above, there's also this one

<OnPlatform x:Key="ImageSize" x:TypeArguments="x:Double" >
    <On Platform="Windows" Value="150"></On>
    <On Platform="Android" Value="100"></On>
    <On Platform="iOS" Value="90"></On>
</OnPlatform>

<Style TargetType="Image" x:Key="ImageStyle" >
    <Setter Property="HeightRequest" Value="{StaticResource ImageSize}" />
    <Setter Property="WidthRequest" Value="{StaticResource ImageSize}" />
    <Setter Property="VerticalOptions" Value="Center" />
    <Setter Property="HorizontalOptions" Value="Center" />
</Style

A construct I would very much recommend, as it enables you to re-use the ImageSize value for other things, for instance the height of button, in another style. You can also use these doubles directly in Xaml, like I did with SomeOtherTextFontSize in the last label in MainPage.xaml

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://xamarin.com/schemas/2014/forms"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             x:Class="UWPStyleIssue.MainPage">
    <Grid VerticalOptions="Center" HorizontalOptions="Center" >
        <Grid.RowDefinitions>
            <RowDefinition Height="*"></RowDefinition>
            <RowDefinition Height="*"></RowDefinition>
            <RowDefinition Height="*"></RowDefinition>
            <RowDefinition Height="*"></RowDefinition>
        </Grid.RowDefinitions>
        <Image Grid.Row="0"
            Source=
               "https://media.licdn.com/mpr/mpr/shrinknp_400_400/[abbreviated]jpg"
           Style="{StaticResource ImageStyle}"></Image>
        <Label Text="Welcome to                       Xamarin Forms!" Grid.Row="1"
             Style="{StaticResource TextStyle}"/>
        <Label Text="Yet another line" Grid.Row="2" Style="{StaticResource OtherTextStyle}"/>
        <Label Text="Last Line" Grid.Row="3" FontSize="{StaticResource SomeOtherTextFontSize}"/>
    </Grid>
</ContentPage

Although I do not recommend this practice - styles are much cleaner - sometimes needs must and this can be handy.

I can hear you think by now: "your point please, kind sir?" (or most likely something less friendly). Well... it works great on Android, as you have seen. It also works great on iOS. And yes, on Windows too...

OnPlatform WTF?

... until you think "let's get this puppy into the Windows Store". As every Windows Developer knows, if you compile for the Store, you compile for Release, which kicks off the .NET Native toolchain. This is very easy to spot as the compilation process takes much longer. The result is not Intermediary Language (IL),  but binary code - an exe - which makes UWP apps so much faster than their predecessors. Unfortunately, it also means the release build is an entirely different beast than a debug build, which can have some unexpected side effects. In our application, if you run the Release build, you will end up with this.

That is quite some 'side effect'. No margin to pull the first text up, no font size (just default), no image... WTF indeed.

Analysis

Unfortunately I had some issues with another library (FFImageLoading) which took me on the wrong track for quite a while, but after I had fixed that I noticed that when I changed the styles from Onplatform to hard coded values the styling started to work again - even in .NET Native. So if I did this

<x:Double x:Key="ImageSize">150</x:Double>
<!--<OnPlatform x:Key="ImageSize" x:TypeArguments="x:Double"  >
    <On Platform="Windows" Value="150"></On>
    <On Platform="Android" Value="100"></On>
    <On Platform="iOS" Value="90"></On>
</OnPlatform>-->

at least my image showed up again:

With a deadline looming and an ginormous style sheet in my app I really had no time to make a branch with separate styles for Windows. We had to go to the store and we had to go now. Time for a cunning plan. I came up with this:

A solution/workaround/hack/fix ... sort of

So it works when the styles do contain direct values, not OnPlatform, right... ? If you look at App.xaml.cs in the portable project you will see a line in the constructor that's usually not there, and it's commented out

public App()
{
    InitializeComponent();
    //this.FixUWPStyling();

    MainPage = new UWPStyleIssue.MainPage();
}

If you remove the slashes and run the app again in Release....

magic happens. All styles seem to work again. This is because of an extension method that's in the file ApplicationExtensions, that you will find in the Portable project in de Extensions folder

public static void FixUWPStyling(this Application app)
{
    if (Device.RuntimePlatform == Device.Windows)
    {
        app.ConvertAllOnPlatformToExplict();
        app.ConvertAllOnDoubleToPlainDouble();
    }
}

The first method, ConvertAllOnPlatformToExplict, does the following:

• Loop trough all the styles
• Loop through all the setters in a style
• Check if the setters 's property name is either "HeightRequest", "WidthRequest", or "FontSize"
• If so, extract the Windows value from the OnPlatform struct
• Set the setter's value to a plain double with as value the extracted Windows value

It's crude, it requires about everything to be in OnPlatform, but it does the trick. I am not going to write it all out here, it's not very great code, and you can see it all on GitHub anyway.

Then, for good measure it calls ConvertAllOnDoubleToPlainDouble, which loops trough the all the doubles, like

<OnPlatform x:Key="ImageSize" x:TypeArguments="x:Double" >...</OnPlatform>

It extracts the Windows value, removes the OnPlatform from the resource dictionary and adds a new plain double with the Windows style only to the resource dictionary. For some reason, replacement is not possible.

Conclusion

There is apparently a bug in the Xamarin Forms .NET Native UWP tooling, which causes OnPlatform values being totally ignored. With my dirty little trick, you can at least get your styles to work without having to rewrite the whole shebang for Windows or have a separate style file for it. Note this does not fix everything, if you have other value types (like GridHeights) you will need to add your own conversion to ConvertAllOnPlatformToExplict  What I have given you was enough to fix my problems, but not all potential issues that may arise from this bug.

I hope this drives the Xamarin for UWP adoption forward, and I also hopes this helps the good folks in Redmond fix the bug. I've pretty much identified what goes wrong, now they 'only' have to take are of the how ;)

Demo project with fix can be found here.

Building a dynamic floating clickable menu for HoloLens/Windows MR

Intro

In the various XAML-based platforms (WPF, UWP, Xamarin) that were created by or are now part of Microsoft we have the great capability to perform databinding and templating - essentially saying to for instance a list 'this is my data, this is how a single item should look, good luck with it' and the UI kind of creates itself. This we don't quite have in Unity projects for Windows Mixed Reality. But still I gave it my best shot when I created a dynamic floating menu for my app Walk the World (only the first few seconds are relevant, the rest is just showing off a view of Machu Picchu)

Starting point

We actually start using the end result of my previous post, as I don't really like to do things twice. So copy that project to another folder, or make a branch, whatever. I called the renamed folder FloatingDynamicMenuDemo. Then proceed as follows:

• Delete FloatingScreenDemo.* from the project's root
• Empty the App sub folder - just leave the .gitignore
• Open the project in Unity
• Open the Build Settings window (CTRL+B)
• Hit the "Player Settings..." button
• Change "FloatingScreenDemo" in "FloatingDynamicMenuDemo" whereever you see it. Initially you will see only one place, but please expand the "Icon" and "Publishing Settings" panels as well, there are more boxes to fill in.
• Rename the HelpHolder to MenuHolder
• Remove the Help Text Controller from the HelpHolder.
• Change the text in the 3DTextPrefab from the Lorum ipsum to "Select a place too see"
• Change the text's Y position from 0.84 to 0.23 so it will end up at the top of the 'screen'

So now we have a workspace with most of the stuff we need already in it. Time to fill in the gaps.

Building a Menu Item part 1 - graphics

So, think templating. We first need to have a template before we can instantiate it. But the only thing I can instantiate are game objects. So... we need to make one... a combination of graphics and code. That sounds like - a prefab indeed!

First, we will make a material for the menu items, as this will be easier for debugging. Go to the App/Materials folder, find HelpScreenMaterial, hit CTRL-D, and rename HelpScreenMaterial 1 to MenuItemMaterial. Then, change its color to a kind of green, for instance 00B476FF. Also, change the rendering mode to "Opaque". This is so we can easily see the plane.

Inside the MenuHolder we make a new empty game object. I called it - d'oh - MenuItem. Inside that MenuItem, we first make a 3DTextPrefab, then a Plane. The plane will be of course humongous again, and very white. So first drag the green MenuItemMaterial on it. Then change it's X Rotation to 270 so it will be upright again. Then you have to experiment a little with the X and Z scale until it is more or less the same width as your blue Plane, and a little over 1 line of text height, as showed to the left.The values I got were X = 0.065 and Z = 0.004 but this depends of course on the font size you take. Make sure there is some extra padding between the left and right edges of the green Plane and the blue Plane.

As you can see in the top panel, the text and the menu pane are invisible - they are only visible when looked upon dead right from the camera in the game view. This is because they basically are at the same distance as the screen. So we need to set -0.02 to the Z of the green Plane - so it appears in front of the blue screen - and -0.04 to the Z of the 3DTextPrefab so it will appear in front of the green Plane, and you will see the effect in the Scene pane as well now.

Since this is a Menu, we want the text to appear from the left. The Anchor is now middle center and it's Alignment Center, and that is not desirable. So we have to set Alignment to Left and Anchor to Middle Left, and then we drag the text prefab to the left till the edge of the green plane. I found an X position value of -0.32.

Now create a folder "Prefabs" in your App folder in the Assets pane, and drag the MenuItem object from there. This will create a Prefab. The text MenuItem in the Hierarchy will turn blue.

You can now safely delete the MenuItem from the Hierarchy. Mind you, the Hierarchy. Make sure it stays in Prefabs.

Building a Menu Item part 2 - code

Our 'menu' needs some general data structure helper. So we start with an interface for that:

public interface IMenuItemData
{
    object SelectMessageObject { get; set; }

    string Title { get; set; }

    int MenuId { get; set; }
}

And a default implementation:

public class MenuItemData : IMenuItemData
{
    public object SelectMessageObject { get; set; }

    public string Title { get; set; }

    public int MenuId { get; set; }
}

The SelectedMessageObject is the payload - the actual data. The Title contains the text we want to have displayed on the menu, and the MenuId we need so we can distinguish select events coming from multiple menus, should your application have such. For the distribution of events we once again use the Messenger that I introduced before (and have used extensively ever since).

To send a selected object around we need a message class:

public class MenuSelectedMessage
{
    public IMenuItemData MenuItem { get; set; }
}

And then we only need to add this simple MenuItemController, a behaviour that handles when the MenuItem is tapped:

using HoloToolkit.Unity.InputModule;
using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class MenuItemController : MonoBehaviour, IInputClickHandler
{
    private TextMesh _textMesh;
    private IMenuItemData _menuItemData;

    public IMenuItemData MenuItemData
    {
        get { return _menuItemData; }
        set
        {
            if (_menuItemData == value)
            {
                return;
            }
            _menuItemData = value;
            _textMesh = GetComponentInChildren<TextMesh>();
            if (_menuItemData != null && _textMesh != null)
            {
                _textMesh.text = _menuItemData.Title;
            }
        }
    }

    public void OnInputClicked(InputClickedEventData eventData)
    {
        if (MenuItemData != null)
        {
            Messenger.Instance.Broadcast(
                new MenuSelectedMessage {MenuItem = MenuItemData});
            PlayConfirmationSound();
        }
    }

    private AudioSource _audioSource;
    private void PlayConfirmationSound()
    {
        if (_audioSource == null)
        {
            _audioSource = GetComponent<AudioSource>();
        }
        if (_audioSource != null)
        {
            _audioSource.Play();
        }
    }
}

There is a property MenuItemData that accepts an IMenuItemData. If you set it, it will retain the value in a private field but also shows the value of Title in a TextMesh component. This behaviour is also an IInputClickHandler, so if the user taps this, the OnInputClicked method is called. Essentially all it does, is sending off it's MenuItemData object - that was used to fill the text with a value - to the Messenger. And it tries to play a sound. You should decide for yourself if you want that.

So all we have to to is add this behaviour MenuItem prefab, as this is the thing we are going to click on. That way, if you click next to the text but at the correct height (the menu 'row'), it's still selected. So select MenuItem, hit the "Add Component" button and add the Menu Item Controller.

Now if you like, you can add an with AudioSource with a special sound that signifies the selection of a menu. As I have stated before, immediate (audio) feedback is very important in immersive applications. I have done not so. I usually let the receiver of a MenuSelectedMessage do the notification sound.

Building the menu itself

This is done by a surprisingly small and simple behavior. All it does is instantiate a number of game object on a certain positions.

using System.Collections.Generic;
using System.Linq;
using UnityEngine;public class MenuBuilder :MonoBehaviour
{
    public float MaxNumber = 10;

    public float TopMargin = 0.1f;

    public float MenuItemSize = 0.1f;

    private List<GameObject> _createdMenuItems = new List<GameObject>();
    public MenuBuilder()
    {
        _menuItems = new List<IMenuItemData>();
    }
    public GameObject MenuItem;

    private IList<IMenuItemData> _menuItems;

    public IList<IMenuItemData> MenuItems
    {
        get { return _menuItems; }
        set
        {
            _menuItems = value;
            BuildMenuItems();
        }
    }

    private void BuildMenuItems()
    {
        foreach (var menuItem in _createdMenuItems)
        {
            DestroyImmediate(menuItem);
        }
        if (_menuItems == null || !_menuItems.Any())
        {
            return;
        }
        for (var index = 0; index < MenuItems.Count; index++)
        {
            var newMenuItem = MenuItems[index];
            var newGameObject = Instantiate(MenuItem, gameObject.transform);
            newGameObject.transform.localPosition -= 
                new Vector3(0,(MenuItemSize * index) - TopMargin, 0);

            var controller = newGameObject.GetComponent<MenuItemController>();
            controller.MenuItemData = newMenuItem;
            _createdMenuItems.Add(newGameObject);
        }
    }
}

All the important work happens in BuildMenuItems. Any existing items are destroyed first, then we simply loop through the list of menu items - these are IMenuItemData objects. Then a game object provided in MenuItem is instantiated inside the current game object, and it's vertical position is calculated and set. Then it gets the MenuItemController from the instantiated game object - it just assumes it must be there - and puts the newMenuItem in it - so the MenuItem will show the associated text.

So now add the MenuBuilder to the HelpHolder. Then, from prefabs, drag the MenuItem prefab onto the Menu Item property. Net result:

Now let's add an initialization behaviour to actually make stuff appear in the menu. This behavior has some hard coded data in it, but you can imagine this coming from some Azure data source

using System.Collections.Generic;
using UnityEngine;

public class VistaMenuController : MonoBehaviour
{
    void Start()
    {
        var builder = GetComponent<MenuBuilder>();

        IList<IMenuItemData> list = new List<IMenuItemData>();
        list.Add(new MenuItemData
        {
            Title = "Mount Everest, Nepal",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(27.91282f, 86.94221f)
        });
        list.Add(new MenuItemData
        {
            Title = "Kilomanjaro, Tanzania",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(-3.21508f, 37.37316f)
        });
        list.Add(new MenuItemData
        {
            Title = "Mount Rainier, Washington, USA",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(46.76566f, -121.7554f)
        });
        list.Add(new MenuItemData
        {
            Title = "Niagra falls (from Canada)",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(43.07306f, -79.07561f)
        });
        list.Add(new MenuItemData
        {
            Title = "Mount Robson, British Columbia, Canada",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(53.061809f, -119.168358f)
        });
        list.Add(new MenuItemData
        {
            Title = "Athabasca Glacier, Alberta, Canada",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(52.18406f, -117.257f)
        });
        list.Add(new MenuItemData
        {
            Title = "Etna, Sicily, Italy",
            MenuId = 1,
            SelectMessageObject = new WorldCoordinate(37.67865f, 14.9964f)
        });

        builder.MenuItems = list;
    }
}

This comes straight from Walk the World - these 7 of it's 10 vistas with a location to look from it. Add this behaviour to the Help Holder as well. Now it's time to run the code and see our menu for the very first time!

Some tweaking and fiddling

If you press the play button in Unity, you will get something like displayed to the left. A former British colleague would say something among the lines of "It's not quite what I had in mind". But we can fix this, fortunately.

In the Menu Item Builder that you have added to HelpHolder, there's two more properties:

The first one is the relative location where the first item should appear, and the second the size allotted for each menu. Clearly the first MenuItem is placed too low. The only way to really get this done is by trial an error. The higher you make Top Margin, the higher up the first item moves. A value of 0.18 gives about this and that seems about right:

And 0.041 for Menu Item Size gives this:

Which is just what you want - a tiny little space between the menu items. Like I said, just trial and error.

Testing if its works

Once again, a bit lame: a simple behaviour to listen to the menu selection messages:

using HoloToolkitExtensions.Messaging;

public class MenuListener : MonoBehaviour
{
    // Use this for initialization
    void Start()
    {
        Messenger.Instance.AddListener<MenuSelectedMessage>(ProcessMenuMessage);
    }

    private void ProcessMenuMessage(MenuSelectedMessage msg)
    {
        if (msg.MenuItem.MenuId == 1 )
        {
            Debug.Log("Taking you to " + msg.MenuItem.Title);
            Debug.Log(msg.MenuItem.SelectMessageObject.ToString());
        }
    }
}

Add this behaviour to the Managers object, click play, and sure enough if you click menu items, you will see in Unity's debug console:

Yes, I know, that's a lame demo - you connect something to the message that actually does something. Speak out the name. Have a dancing popup. The point is that it works and the messages get out when you click :)

Some final look & feel bits

Yah! We have a more or less working menu but it looks kind of ugly and not everything works - the close button, for instance. Let's fix the look & feel first. We needed the greenish background of the menu item to properly space and align the items, but now we do not need it anymore. So go to the MenuItemMaterial. Select a new shader: under HoloToolkit, you will find "Vertex Lit Configurable Transparent".

Then go all the way down, to "Other" and set "Cull" to front. That way, the front part of the plane - the green strips will be invisible - but still hittable.

If you press play, the menu should now look like this:

Getting the button to work

As stated above, the button is not working - and for a very simple reason: in my previous post I showed that it looks for a component in it's parent that is a BaseTextScreenController (or a child class of that). There are none.

So let's go back to the VistaMenuController again. The top says

public class VistaMenuController : MonoBehaviour

Let's change that into

public class VistaMenuController : BaseTextScreenController

You will need to add "using HoloToolkitExtensions.Animation;" to top to get this to work. You will also need to change

void Start()
{

into

public override void Start()
{
  base.Start();

If you now hit "Play" in Unity you will end up with this

Right. Nothing at all :). This is because the base Start method (which is the Start method of BaseTextScreenController) actually hides the menu, on the premises that you don't want to see the menu initially. So we have to have a way to make it visible. Fortunately, that's very easy. We will just re-use the ShowHelpMessage from the previous post again to make this work. Go back one more time to the VistaMenuController "Start" method, and add one more statement:

public override void Start()
{
  base.Start();
  Messenger.Instance.AddListener<ShowHelpMessage>(m => Show());

If you now press play, you will still see nothing. But if you yell "Show help" to your computer (or press "0" - zero) the menu pops up and comes into view. With, I might add, the for my apps now iconic "pling" sound. And if you click the button, the menu will disappear with the equally iconic "clonk" .

Some concluding remarks

Of course, this is still pretty primitive. With the current font size and menu item size, stuff will be happily rendered outside of the actual menu screen if your texts are too long or you have more than 7 menu items. That is because the screen is just a floating backdrop. Scrolling for more items? Nope. Dynamic or manual resizing? Nope. But it is a start, and I have used it with great success.

Let me know if this was valuable to you, and what you used it for. Full demo project at GitHub, as always.

Building a floating HoloLens 'info screen' - 2: adding the C# behaviours

Intro

In the first installment of this 2-part blog post we have created the UI of the info screen, now we are going to build the dynamics.

• First we will make the app recognize a speech command
• Then we will add the dynamics to make the screen appear and disappear when we want.
• Then we will make the close button work
• As a finishing touch we will add some spatial sound.

Adding a speech command – the newest new way

For the second or maybe even the third time since I have started using the HoloToolkit, the was the way speech commands are supposed to work has changed. The keyword manager is now obsolete, you now have to use SpeechInputSource and SpeechInputHandler.

First, we add a Messenger, as already described in this blog post, to the Managers game object. It sits in Assets/HoloToolkitExtensions/Scripts/Messaging.

Then, since we are good boy scouts that like to keep things organized, a create a folder “Scripts” under “Assets/App”. In “Scripts” we add a “Messages” folder, and in that we create the following highly complicated message class ;)

public class ShowHelpMessage
{
}

In Scripts we create the SpeechCommandExectutor, which is simply this:

using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class SpeechCommandExecutor : MonoBehaviour
{
    public void OpenHelpScreen()
    {
        Messenger.Instance.Broadcast(new ShowHelpMessage());
    }
}

Add this SpeechCommandExecutor to the Managers game object. Also add a SpeechInputSource script from the HoloToolkit, click they tiny plus-button on the right and add “show help” as keyword:

Also, select a key in “key shortcut”. Although they Unity3D editor supports voice commands, you can now also use a code to test the flow. And believe me – your colleagues will thank you for that. Although lots of my colleagues are now quite used to me talking to devices and gesturing in empty air, repeatedly shouting at a computer because it was not possible to determine if there’s a bug in the code or the computer just did not hear you… is still kind of frowned upon.

Anyway. To connect the SpeechCommandExecutor to the SpeechInputSource we need a SpeechInputHandler. That is also in the HoloToolkit. So drag it out of there into the Managers objects. Once again you have to click a very tiny plus-button:

And then the work flow is a follows

1. Check the “Is Global Listener” checkbox (that is there because of a pull request by Yours Truly)
2. Select the plus-button under “Responses”
3. Select “Show help” from the keyword drop down
4. Drag the Managers object from the Hierachy to the box under “Runtime only”
5. Change “Runtime only” to “Editor and Runtime”
6. Select “SpeechCommandExecutor” and then “OpenHelpScreen” from the right dropdown.

To test you have done everything ok:

In Assets/App/Scripts, double-click SpeechCommandExecutor.

This will open Visual Studio, on the SpeechCommandExecutor. Set a breakpoint on

Messenger.Instance.Broadcast(new ShowHelpMessage());

Hit F5, and return to Unity3D. Click the play button, and press “0”, or shout “Show help” if you think that’s funny (on my machine, speech recognition in the editor does not work on most occasions, thus I am very happy with the keys options).

If you have wired up everything correctly, the breakpoint should be hit. Stop Visual Studio and leave Unity Play Mode again. This part is done.

Making the screen follow your gaze

Another script from my HoloToolkitExtensions, that I already mentioned in some form, is MoveByGaze. It looks like this:

using UnityEngine;
using HoloToolkit.Unity.InputModule;
using HoloToolkitExtensions.SpatialMapping;
using HoloToolkitExtensions.Utilities;

namespace HoloToolkitExtensions.Animation
{
    public class MoveByGaze : MonoBehaviour
    {
        public float MaxDistance = 2f;

        public float DistanceTrigger = 0.2f;

        public float Speed = 1.0f;

        private float _startTime;
        private float _delay = 0.5f;

        private bool _isJustEnabled;

        private Vector3 _lastMoveToLocation;

        public BaseRayStabilizer Stabilizer = null;

        public BaseSpatialMappingCollisionDetector CollisonDetector;

        // Use this for initialization
        void Start()
        {
            _startTime = Time.time + _delay;
            _isJustEnabled = true;
            if (CollisonDetector == null)
            {
                CollisonDetector = new DefaultMappingCollisionDetector();
            }
        }

        void OnEnable()
        {
            _isJustEnabled = true;
        }

        // Update is called once per frame
        void Update()
        {
            if ( _isBusy || _startTime > Time.time)
                return;

            var newPos = LookingDirectionHelpers.GetPostionInLookingDirection(2.0f, 
                GazeManager.Instance.Stabilizer);
            if ((newPos - _lastMoveToLocation).magnitude > DistanceTrigger || _isJustEnabled)
            {
                _isJustEnabled = false;
                var maxDelta = CollisonDetector.GetMaxDelta(newPos - transform.position);
                if (maxDelta != Vector3.zero)
                {
                    _isBusy = true;
                    newPos = transform.position + maxDelta;
                    LeanTween.moveLocal(gameObject, transform.position + maxDelta, 
                        2.0f * maxDelta.magnitude / Speed).setEaseInOutSine().setOnComplete(MovingDone);
                    _lastMoveToLocation = newPos;
                }
            }
        }

        private void MovingDone()
        {
            _isBusy = false;
        }

        private bool _isBusy;

    }
}

This is an updated, LeanTween (in stead of iTween) based version of a thing I already described before in this post so I won’t go over it in detail. You will find it in the Animation folder of the HoloToolkitExtensions in the demo project. It uses helper classes BaseSpatialMappingCollisionDetector, DefaultMappingCollisionDetector and SpatialMappingCollisionDetector that are also described in the same post – these are in the HoloToolkitExtensions/SpatialMapping folder of the demo project.

The short workflow, for if you don’t want to go back to that article:

• Add a SpatialMappingCollisionDetector to the Plane in the HelpHolder
• Add a MoveByGaze to the HelpHolder itself
• Drag the InputManager on top of the “Stabilizer” field in the MoveByGaze script
• Drag the Plane on top of the “Collision Detector” field

The result should look like this

I would suggest updating “Speed” to 2.5 because although the screen moves nice and fluid, the default value is a bit slow for my taste. If you now press the Play Button in Unity, you will see the screen already following the gaze cursor if you move around with the mouse or the keyboard.

The only thing is, it is not always aligned to the camera. For that, we have the LookAtCamera script I already wrote about in October in part 3 of the HoloLens airplane tracker app, but I will show it here anyway:

using UnityEngine;

namespace HoloToolkitExtensions.Animation
{
    public class LookatCamera : MonoBehaviour
    {
        public float RotateAngle = 180f;

        void Update()
        {
            gameObject.transform.LookAt(Camera.main.transform);
            gameObject.transform.Rotate(Vector3.up, RotateAngle);
        }
    }
}

because it’s so small. The only change between this and the earlier version is that you know can set the the rotate angle in the editor ;). Drag it on top of the HelpHolder now the screen will always face the user after moving to a place right in front of it.

Fading in/out the help screen

In the first video you can see the screen fades nicely in on the voice command, and out when it’s clicked. The actual fading is done by no less than three classes, two of whom are inside the HoloToolkitExtensions. First is this simple FadeInOutController, that is actually usable all by itself:

using UnityEngine;

namespace HoloToolkitExtensions.Animation
{
    public class FadeInOutController : MonoBehaviour
    {
        public float FadeTime = 0.5f;

        protected bool IsVisible { get; private set; }

        private bool _isBusy;

        public virtual void Start()
        {
            Fade(false, 0);
        }

        private void Fade(bool fadeIn, float time)
        {
            if (!_isBusy)
            {
                _isBusy = true;
                LeanTween.alpha(gameObject, fadeIn ? 1 : 0, time).setOnComplete(() 
                    => _isBusy = false);
            }
        }

        public virtual void Show()
        {
            IsVisible = true;
            Fade(true, FadeTime);
        }

        public virtual void Hide()
        {
            IsVisible = false;
            Fade(false, FadeTime);
        }
    }
}

So this is a pretty simple behaviour that fades the current gameobject in or out, in a configurable timespan, and it makes sure it will not get interrupted while doing the fade. Also – notice it initially fades the gamobject out in zero time, so initially any gameobject with this behavior will be invisible

Next up is BaseTextScreenController, that is a child class of FadeInOutController:

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using System.Linq;

namespace HoloToolkitExtensions.Animation
{
    public class BaseTextScreenController : FadeInOutController
    {
        private List<MonoBehaviour> _allOtherBehaviours;

        // Use this for initialization
        public override void Start()
        {
            base.Start();
            _allOtherBehaviours = GetAllOtherBehaviours();
            SetComponentStatus(false);
        }

        public override void Show()
        {
            if (IsVisible)
            {
                return;
            }
            SetComponentStatus(true);
            var a = GetComponent<AudioSource>();
            if (a != null)
            {
                a.Play();
            }
            base.Show();
        }

        public override void Hide()
        {
            if (!IsVisible)
            {
                return;
            }
            base.Hide();
            StartCoroutine(WaitAndDeactivate());
        }

        IEnumerator WaitAndDeactivate()
        {
            yield return new WaitForSeconds(0.5f);
            SetComponentStatus(false);
        }
    }
}

So this override, on start, gathers all other behaviors, then de-activates components (this will be explained below). When Show is called, it first activates the the components, then tries to play a sound, then calls the base Show to unfade the control. If Hide is called, it first calls the base fade, then after a short wait starts to de-activate all components again.

So what is the deal with this? The other two missing routines are like this:

private List<MonoBehaviour> GetAllOtherBehaviours()
{
    var result = new List<Component>();
    GetComponents(result);
    var behaviors = result.OfType<MonoBehaviour>().Where(p => p != this).ToList();
    GetComponentsInChildren(result);
    behaviors.AddRange(result.OfType<MonoBehaviour>());
    return behaviors;
}

private void SetComponentStatus(bool active)
{
    foreach (var c in _allOtherBehaviours)
    {
        c.enabled = active;
    }
    for (var i = 0; i < transform.childCount; i++)
    {
        transform.GetChild(i).transform.gameObject.SetActive(active);
    }
}

As you can see, the first method simply finds all behaviors in the gameobject – the screen - and its immediate children, except for this behavior. If you supply “false” for “active”, it will first disable all behaviours (except the current one), and then it will set all child gameobjects to inactive. The point of this is that we have a lot of things happening in this screen. It’s following your gaze, checking for collisions, it’s spinning a button, and it’s waiting for clicks – all in vain as the screen is invisible. So this setup makes the whole screen dormant, disables all behaviors except the current one – and also can bring it back ‘to life’ again by supplying ‘true’. The important part is to do the right order (first the behaviours, then the gameobjects). It’s also important to gather the behaviours at the start, because once gameobjects are deactivated, you can’t get to their behaviors anymore.

The final class does nearly nothing – but this is the only app-specific class

using HoloToolkitExtensions.Animation;
using HoloToolkitExtensions.Messaging;

public class HelpTextController : BaseTextScreenController
{
    public override void Start()
    {
        base.Start();
        Messenger.Instance.AddListener<ShowHelpMessage>(ShowHelp);
    }

    private void ShowHelp(ShowHelpMessage arg1)
    {
        Show();
    }
}

Basically the only thing this does is make sure the Show method is called when a ShowHelpMessage is received. If you drag this HelpTextController on top of the HelpHolder and press the Unity play button, you see an empty screen in stead of the help screen. But if you press 0 or yell “show help” the screen will pop up.

Closing the screen by a button tap

So now the screen is initially invisible, it appears on a speech command – now how do we get rid of it again? With this very simple script the circle is closed:

using HoloToolkit.Unity.InputModule;
using UnityEngine;

namespace HoloToolkitExtensions.Animation
{
    public class CloseButton : MonoBehaviour, IInputClickHandler
    {
        private void Start()
        { }

        void Awake()
        {
            gameObject.SetActive(true);
        }

        public void OnInputClicked(InputClickedEventData eventData)
        {
            var h = gameObject.GetComponentInParent<BaseTextScreenController>();
            if (h != null)
            {
                h.Hide();
                var a = gameObject.GetComponent<AudioSource>();
                if (a != null)
                {
                    a.Play();
                }
            }
        }
    }
}

This a standard HoloToolkit IInputClickHandler – when the user clicks, it tries to find a BaseTextScreenController in the parent and calls the Hide method, effectively fading out the screen. And it tries to play a sound, too.

Some finishing audio touches

Two behaviours – the CloseButton and the BaseTextScreenController – try to play sound when they are activated. As I have stated multiple times before, having immediate audio feedback when a HoloLens ‘understands’ a user initiated action is vital, especially when that action’s execution may take some time. At no point you want the user to have a ‘huh it’s not doing anything’ feeling.

In the demo project I have included two audio files I use quite a lot – “Click” and “Ready”. “Click” should be added to the Sphere in HelpHolder. That is easily done by dragging it onto the Sphere from App/Scripts/Audio onto the Sphere. That will automatically create an AudioSource.

Important are the following settings:

• Check the “Spatialize” checkbox
• Uncheck the “Play on awake checkbox
• Move the “Spatial Blend” slider all the way to the right
• In the 3D sound settings section, set “Volume Roloff” to “Custom Rolloff”

Finally, drag “Ready” on top of the HelpHolder itself, where it will be picked up by the HelpTextController (which is a child class of BaseTextScreenController ) and apply the same settings. Although you might consider not using spatial sound here, because it’s not a sound that is particularly attached to a location – it’s a general confirmation sound

Conclusion

To be honest, a 2d-ish help screen feels a bit like a stopgap. You can also try to have a kind of video of audio message showing/telling the user about the options that are available. Ultimately you can think of an intelligent virtual assistant that teach you the intricacies of an immersive app. With the advent of ‘intelligent’ bots and stuff like LUIS it might actually become possible to have an app help you through it’s own functionality by having a simple questions-and-answers like conversation with it. I had quite an interesting discussion about this subject at Unity Unite Europe last Wednesday. But then again, since Roman times we have pointed people in right directions or conveyed commercial messages by using traffic signs and street signs – essentially 2D signs in a 3D world as well. Sometimes we used painted murals, or even statue like things. KISS sometimes just works.

The completed demo project can be downloaded here.