29 May 2019

Migrating to MRTK2 - looking a bit closer at tapping, and trapping 'duplicate' events

Intro

In my previous post I wrote about how game objects can be made clickable (or 'tappable') using the Mixed Reality Toolkit 2, and how things changed from MRTK1. And in fact, when you deploy the app to a HoloLens 1, my demo actually works as intended. But then I noticed something odd in the editor, and made a variant of the app that went with the previous blog post to see how things work- or might work- in HoloLens 2.

Debugging ClickyThingy ye olde way

Like I wrote before, it's possible to debug the C# code of a running IL2CPP C++ app running on a HoloLens. To debug using breakpoints is a bit tricky when you are dealing with rapidly firing event - stopping through the debugger might actually have some influence on the order events play out. So I resorted to the good old "Console.WriteLine-style" of debugging, and added a floating text in the app that shows what's going on.

The ClickableThingy behaviour I made in the previous post then looks like this:

using Microsoft.MixedReality.Toolkit.Input;
using System;
using TMPro;
using UnityEngine;

public class ClickableThingyGlobal : BaseInputHandler, IMixedRealityInputHandler
{
    [SerializeField]
    private TextMeshPro _debugText;

    public void OnInputUp(InputEventData eventData)
    {
        GetComponent<MeshRenderer>().material.color = Color.white;
        AddDebugText("up", eventData);
    }

    public void OnInputDown(InputEventData eventData)
    {
        GetComponent<MeshRenderer>().material.color = Color.red;
        AddDebugText("down", eventData);
    }

    private void AddDebugText( string eventPrefix, InputEventData eventData)
    {
        if( _debugText == null)
        {
            return;
        }
        var description = eventData.MixedRealityInputAction.Description;
        _debugText.text += 
            $"{eventPrefix} {gameObject.name} : {description}{Environment.NewLine}";
    }
}



Now in the HoloLens 1, things are exactly like you expect. Air tapping the sphere activates Up and Down events exactly once for every tap (because the Cube gets every tap, even when you don't gaze at it - see my previous post for an explanation)

When you run the same code in the editor, though, you get a different result:

Tap versus Grip - and CustomPropertyDrawers

The interesting thing is, when you 'air tap' in the editor (using the space bar and the left mouse button), thumb and index finger of the simulated hand come together. This, now, is recognized as a tap followed by a grip, apparently.

So we need to filter the events coming in through OnInputUp and OnInputDown to respond to the actual events we want. This is where things get a little bit unusual - there is no enumeration of sorts that you can use to compare you actual event against. The available events are all in the configuration, so they are dynamically created.

The way to do some actual filtering is to add a property of type MixedRealityInputAction to your behaviour (I used _desiredInputAction). Then the MRTK2 automatically creates a drop down with possible to events to select from:

How does this magic work? Well, the MRTK2 contains a CustomPropertyDrawer called InputActionPropertyDrawer that automatically creates this drop down whenever you add a property of type MixedRealityInputAction to your behaviour. The values in this list are pulled from the configuration. This fits with the idea of the MRTK2 that everything must be configurable ad infinitum. Which is cool but sometimes it makes things confusing.

Anyway, you select the event you want to test for in the UI, in this case "Select":

And then you have to check if the event methods if the event matches your desired event.

if (eventData.MixedRealityInputAction != _desiredInputAction)
{
    return;
}

And then everything works as you expect: only the select event results in an action by the app.

How about HoloLens 2?

I could only test this in the emulator. The odd things is, even without the check on the input action, only the select action was fired, even when I pinched the hand using the control pane:

So I have no idea if this is actually necessary on a real live HoloLens 2, but my friends and fellow MVPs Stephen Hodgson and Simon 'Darkside' Jackson have both mentioned this kind of event type check as being necessary in a few on line conversations (although I then did not understand why). So I suppose it is :)

Conclusion

Common wisdom has it that the best thing about teaching is that you learn a lot yourself. This post is excellent proof of that wisdom. If you think this here old MVP is the end-all and know-all of this kind of stuff, think again. I knew of customer editors, but I literally just learned the concept of CustomPropertyDrawer while I was writing this post. I had no idea it existed, but I found it  because I wanted to know how the heck the editor got all the possible MixedRealityInputAction from the configuration and show that in such a neat list. Took me quite some searching, actually - which is logical, if you don't know what exactly you are looking for ;).

I hope this benefits you as well. Demo project here (branch TapCloseLook).

22 May 2019

Migrating to MRTK2 - IInputClickHandler and SetGlobalListener are gone. How do we tap now?

Intro

Making something 'clickable' (or actually more 'air tappable') was pretty easy in the Mixed Reality Toolkit 1. You just added the IInputClickHandler interface like this:

using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class ClickableThingy: MonoBehaviour, IInputClickHandler
{
    public void OnInputClicked(InputClickedEventData eventData)
    {
        // Do something
    }
}

You dragged this behaviour on top of any game object you want to act on being air tapped and OnInputClicked was being activated as soon as you air tapped. But IInputClickHandler does no longer exist in MRTK2. How does that work now?

Tap – just another interface

To support the air tap in MRTK2, it's simply a matter of switching out one interface for another:

using Microsoft.MixedReality.Toolkit.Input;
using UnityEngine;

public class ClickableThingy : MonoBehaviour, IMixedRealityInputHandler
{
    public void OnInputUp(InputEventData eventData)
    {
        //Do something else
    }

    public void OnInputDown(InputEventData eventData)
    {
        //Do something
    }
}

I don't have HoloLens 2, but if you put whatever was in OnInputClicked in OnInputDown it's being executed on a HoloLens 1 when you do an air tap and the object is selected by a the gaze cursor.. So I guess that's a safe bet if you want to make something that runs on both HoloLens 1 and 2.

‘Global tap’ – add a base class

In the MRTK 1 days, when you wanted to do a ‘global tap’, you could simply add a SetGlobalListener behaviour to the game object that contained your IInputClickHandler implementing behaviour:

Adding this object meant that any airtap would be routed to this IImputClicked object - even without the gaze cursor touching the game it, or touching anything anything at all, for what matters. This could be very useful in situations where you for instance were placing objects on the spatial map and some gesture is needed to stop the movement. Or some general confirmation gesture in a situation where some kind of UI was not feasible because it would get in the way. But the SetGlobalListener behaviour is gone as well, so how do get that behavior now?

Well, basically you make your ClickableThingy not only implement IMixedRealityInputHandler, but also be a child class of BaseInputHandler.

using Microsoft.MixedReality.Toolkit.Input;
using UnityEngine;

public class ClickableThingyGlobal : BaseInputHandler, IMixedRealityInputHandler
{
    public void OnInputUp(InputEventData eventData)
    {
        // Do something else
    }

    public void OnInputDown(InputEventData eventData)
    {
        // Do something
    }
}

This has a property isFocusRequired that you can set to false in the editor:

And then your ClickableThingy will get every tap. Smart people will notice it makes sense to always make a child class of BaseInputHandler as the IsFocusRequired property is default true – so the default behavior ClickableThingyGlobal is to act exactly the same as ClickableThingy, but you can configure it’s behavior in the editor, which makes your behavior applicable to more situations. Whatever you can make configurable saves code. So I'd always go for a BaseInputHandler for anything that handles a tap.

Proof of the pudding

This is exactly what the demo project shows: a cube that responds to a tap regardless whether there is a gaze or hand cursor on it, and a sphere that only responds to a tap when there is a hand or gaze cursor on it. Both use the ClickableThingyGlobal: the cube has the IsFocusRequired check box unselected, on the sphere it is selected. To this end I have adapted the ClickableThingyGlobal to actually do something usable:

using Microsoft.MixedReality.Toolkit.Input;
using UnityEngine;

public class ClickableThingyGlobal : BaseInputHandler, IMixedRealityInputHandler
{
    public void OnInputUp(InputEventData eventData)
    {
        GetComponent<MeshRenderer>().material.color = Color.white;
    }

    public void OnInputDown(InputEventData eventData)
    {
        GetComponent<MeshRenderer>().material.color = Color.red;
    }
}

or at least something visible, which is to change the color of the elements from white to red on a tap (and back again).

In an HoloLens 1 it looks like this.

The cube will always flash red, the sphere only when there is some cursor pointing to it. In the HoloLens 2 emulator it looks like this:

The fun thing now is that you can act on both InputUp and InputDown, which I use to revert the color setting. To mimic the behavior of the old OnInputClicked, adding code in OnInputDown and leaving OnInputUp is sufficient I feel.

Conclusion

Yet another part of moved cheese, although not dramatically so. Demo code is very limited, but can still be found here. I hope me documenting finding my way around Mixed Reality Toolkit 2 helps you. If you have questions about specific pieces of your HoloLens cheese having been moved and you can't find them, feel free to ask me. In any case I intend to write lots more of these posts.

15 May 2019

Migrating to MRTK2 - MS HRTF Spatializer missing (and how to get it back)

Intro

One the many awesome (although sadly underutilized) capabilities of HoloLens is Spatial Audio. With just a few small speakers and some very nifty algorithms it allows you to connect audio to moving Holograms that sound as if they are coming from a Hologram . Microsoft have applied this with such careful precision that you can actually hear Holograms moving above and behind you, which greatly enhances the immersive experience in a Mixed Reality environment. It also has some very practical uses - for instance, alerting the user something interesting is happening outside of their field of vision - and the audio also provides a clue where the user is supposed to look.

Upgrade to MRKT2 ... where is my Spatializer?

In the process op upgrading AMS HoloATC to Mixed Reality Toolkit 2 I noticed something odd. I tried - in the Unity editor - to click an airplane, that should then start to emit a pinging sound. In stead, I saw this error in the editor pop up:

"Audio source failed to initialize audio spatializer. An audio spatializer is specified in the audio project settings, but the associated plugin was not found or initialized properly. Please make sure that the selected spatializer is compatible with the target."

Then I looked into the project's audio settings (Edit/Project Settings/Audio) and saw that the Spatializer Plugin field was set to "None" - and that the MS HRTF Spatializer (that I normally expect to be in the drop down) was not even available!

Now what?

The smoking - or missing - gun

The solution is rather simple. If you look in the Mixed Reality Toolkit 2 sample project, you will notice the MS HRFT Spatializer is both available and selected. So what is missing?

Look at the Packages node in your Assets. It's all the way to the bottom. You will probably see this;

But what you are supposed to see is this:

See what's missing? Apparently the spatializer has been moved into a Unity Package. When you install the Mixed Reality Toolkit 2 and click "Mixed Reality Toolkit/Add to Scene and configure" it is supposed to add this package automatically (at least I think it is) - but for some reason, this does not always happen.

Use the Force Luke - that is, the Unity Package Manager

Fortunately, it's easy to fix. In the Unity Editor, click Window/Package Manager. This will open the Package Manager Window. Initially it will only show a few entries, but then, near the bottom, you will see "Windows Mixed Reality" appear. Hit the "Install" button top right. And when its done the Windows Mixed Reality entry will appear in the Packages will appear.

And now, if you go to Edit/Project Settings/Audio, you will see that MS HRTF Spatializer has appeared again. If this a migrated project and you have not messed with the audio settings, it will probably be selected automatically again.

Conclusion

No code this time, as there is little to code. I do need to add a little word of warning here - apparently these packages are defined in YourProject/Packages/manifest.json. Make sure this gets added to your repo and checked in as well.

10 May 2019

Migrating to MRTK2–NewtonSoft.JSON (aka JSON.Net) is gone

Intro

In ye olde days, if you set up a project using the Mixed Reality Toolkit 1, NewtonSoft.JSON (aka JSON.Net) was automatically included. This was because part of the MRTK1 had a dependency on it –something related to the gLTF stuff used it. This is (apparently) no longer the case. So if you had a piece of code that previously used something like this

public class DeserializeJson : MonoBehaviour
{
    [SerializeField]
    private TextMeshPro _text;

    void Start()
    {
        var jsonstring = @"
{
   ""Property1"" : ""Hello"",
   ""Property2"" : ""Folks""
}";
        var deserializedObject = JsonConvert.DeserializeObject<DemoJson>(jsonstring);

        _text.text = string.Concat(deserializedObject.Property1,
            Environment.NewLine, deserializedObject.Property2);
    }
}

It will no longer compile when you use the MRTK2. You will need to get it elsewhere. There are two ways to solve this: the right way and the wrong way.

The wrong way

The wrong way, that I was actually advised to do, is to get a copy of an MRTK1 and drag the JSON.Net module from there into your project. It's under HoloToolkit\Utilities\Scripts\GLTF\Plugins\JsonNet. And it will appear to work, too. In the editor. And as long as you use the .NET scripting backend. Unity has announced, though, the .NET backend will disappear – you will need to use IL2CPP soon. And when you do so, you will notice your app will mysteriously fail to deserialize JSON. If you run the C++ app in debug mode from Visual Studio you will see something cryptic like this:

The reason why is not easy to find. If you dig deeper, you will see it complaining about it trying to use Reflection.Emit and this apparently is not allowed in the C++ world. Or not in the way it's done. Whatever.

The right way

Fortunately there is an another way - and a surprisingly one to boot. There is a free JSON.Net package in the Unity store, and it seems to do the trick for me – I can compile the C++ app, deploy it on the HoloLens 2 emulator and it actually parses JSON.

QED:

But will this work on a HoloLens 2?

The fun thing is of course the HoloLens 2 has an ARM processor, so the only way to test if this really works is to run in on an HoloLens 2. Unlike a few very lucky individuals, I don't have access to the device. But I do have something else - an ARM based PC that I was asked to evaluate in 2018.  I compiled for ARM, made a deployment package, powered up the ARM PC and wouldn't you know it...

So. I think we can be reasonably sure this will work on a HoloLens 2 as well.

Update - this has been verified.

Conclusion

I don't know whether all the arcane and intricate things JSON.Net supports are supported by this package, but it seems to do the trick as far as my simple needs are concerned. I guess you should switch to this package to prepare for HoloLens 2.

Code as usual on GitHub:

And yes, the master is still empty but I intend to use that for demonstrating a different issue.

06 May 2019

Migrating to MRTK2 - Mixed Reality Toolkit Standard Shader 'breaks'

Intro

At this moment I am trying to learn as much as possible about the new Mixed Reality Toolkit 2, to be ready for HoloLens 2 when it comes. I opted for using a rather brutal cold turkey learning approach: I took my existing AMS HoloATC app, ripped out the ye goode olde HoloToolkit, and replaced it by the new MRTK2 - fresh from GitHub. Not surprisingly this breaks a lot. I am not sure if this is the intended way of migrating - it's like renovating the house by starting with bulldozering a couple of walls away. But this is the way I chose do it, as it forces me to adhere to the new style and learn how stuff works, without compromises. It also makes me very clear where things are going to break when I do this to customer apps.

So I am starting a series of short blog posts, that basically documents the bumps in the road as I encounter them, as well as how I swerved around them or solved them. I hope other people will benefit from this, especially as I will showing a lot of moved cheese. And speaking of...

Help! My Standard Shader is broken!

So you had this nice Mixed Reality app that showed these awesome holograms:

and then you decided to upgrade to the Mixed Reality Toolkit 2

and you did not expect to see this. This is typically the color Unity shows when a material is missing or something in the shader is thoroughly broken. And indeed, if you look at the materials:

something indeed is broken.

How to fix this

There is good new, bad news, and slightly better news.

  • The good news - it's easy to fix.
  • The bad news is - you have to do this for every material in your apps that used the 'old' HTK Standard shader
  • The slightly better news - you can do this for multiple materials in one go. Provided they are all in one folder, or you do something nifty with search

So, in your assets select your materials:

Then in the inspector select the Mixed Reality Toolkit Standard Shader (again) :

And boom. Everything looks like it should.

Or nearly so,because although it carries the same name, it's actually a different shader. Stuff actually might look a wee bit different. In my sample app, especially the blue seems to look a bit different.

So what happened?

If you look what Git marks as changed, only the tree materials themselves are marked changed:

and if you look at a diff, you will see the referenced file GUID for the shader is changed. So indeed, although it carries the same name (Mixed Reality Toolkit Standard), as far as Unity is concerned it's a different shader.

(you might want to click on the picture to be able to actually read this).

As you scroll down through the diff, you will see lots of additions too, so this is not only a different shader id, it's actually a different or new shader as well. Why they deliberately chose to break the shader ID - beats me. Maybe to make upgrading from one shader to another possible, or have both the old and the new one simultaneously work in one project, making upgrade easier. But since they have the same name, this might also induce confusion. Whatever- but this is what causes the shader to 'break' at upgrade, an now you know how to fix it, too.

Conclusion

I hope to have eliminated once source of confusion today, and I wish you a lot of fun watching the //BUILD 2019 keynote in a few hours.

You can find a demo project here.

  • Branch "master" shows the original project with HoloToolkit
  • Branch "broken" shows the project upgraded to MRTK2 - with broken shaders
  • Branch "fixed" shows the fixed project