12 February 2020

HoloLens 2 - it's the interaction, stupid!

Intro

Tuesday, February 7, 2020 marked an important occasion to me. A HoloLens 2 arrived at my door. For the first time since March 2019, where I got to try a HoloLens 2 during the MVP Summit for a few minutes, I actually had my hands on a device. And what's more - I got a month to play, test and develop with it, courtesy of fellow MVP and Regional Director Philipp Bauknecht of MediaLesson GbmH, a real community hero, who has graciously provided me with this learning opportunity. I hope I will be someday be able to repay him this enormous favor.

Just having this device around and being able to test and develop with it, quite changed my views of it,  what's actually important - and what makes it a game changer.

Display

First of all, I am going to bring your hopes down a little. To an extent, HoloLens 2 suffers a bit from what I would like to coin as "the Apollo 12 effect". The whole world followed Neil Armstrong, Buzz Aldrin and Michael Collins to the Moon and where glued to very bad black & white screen while the first two men took their steps to the Moon. But a lot less people watched Apollo 12. Successive flights got even less attention (bar Apollo 13, but that was not because they landed, but almost died). People had literally seen this before and were - I kid you not - complaining about footage of men on the Moon eating up precious TV time from the football games. Subsequent flights after 17 were cancelled. People are extremely well equipped at accepting 'magic' and then getting bored with it.

As far as display goes, HoloLens 2 shows you virtual objects in 3D space that can interact with reality. This, my friends, is exactly what HoloLens 1 did.

It does this a lot faster, the view is brighter, the holograms are a lot more stable, and the thing almost everyone harped on - the field of view - has been considerably increased. I can almost imagine Alex Kipman shouting "we gave you bloody magic and all you kept telling me was the view was not big enough - are you happy now???"

There are other things: the device is a lot more ergonomic, it feels lighter but actually isn't very much - it's just better balanced. Donning it as easy as cake, taking it off as well, and charging via USB-C is a godsend - no more fiddling with MicroUSB on a wobbly end. I have seen more than one HoloLens 1 with a damaged charging port.

That's all very fine and welcome. But that's not what I mean by game changing. If we stay in space terminology - HoloLens 1 was like we suddenly had a fusion rocket. It is awesome, but parts of it were messy.

HoloLens 2 has a warp drive.

Interaction, interaction, interaction

Everyone who has ever used HoloLens 1 - or better still, tried instructing a newbie user to use one - knows the challenge. You can select something by pointing your head to a Hologram, then perform an air tap. Just tap your finger and thumb together. Easy as cake. And yet I have witnessed people who for some reason could not perform this simple task successfully. Either they pointed the cursor not correctly, or they made gestures that were almost but not quite an air tap, made it to slow or too fast, contorted their hands in a way that apparently confused the device, or started to make up gestures - that of course did not work at all. Whatever. Most people got it, but between 10-20% just never could get it to work reliably, if at all. The HoloLens 1 came with a little clicker for those people, apparently a last-minute addition - that almost no-one ever used. It either lost its charge at an inconvenient time, or (in most cases) got completely lost at all, it being a small device that easily was forgotten or dropped somewhere.

HoloLens 2 does not come with a clicker, and that's for a reason. If you make a gesture that even remotely resembles an air tap, it registers it as such - with such ferocity and accuracy that if you have a large contact surface you might even get some inadvertent air taps in (I will have to look into that for my app Walk the World, for instance). 

In addition, what everyone saw being demoed first by the amazing Julia Schwarz - the ability just to touch, grab and move Holograms works amazingly well. To such an extent that you can push buttons like they are real, grab, move and rotate things like they are real... everything with amazing accuracy. You can even have your hand visualized and then it looks like you have some computer-generated glove on your hands - it follows every little movement. The resulting interaction model is very natural. So natural that you actually at first expect haptic feedback when you push a 'button'. Maybe something for HoloLens 3 ;).

There are a few things you might want to explain when you instruct someone to use the device for the first time to speed things up - like that if you want to use the start button, that's on your wrist. No more bloom gestures – the Italians will appreciate that ;). You might want to explain how an air tap works, but it's likely people find that out by themselves as it is so easy now. Also, the device goes out of it's way to explain itself on first startup.The fact that it can not only track your hand but also recognizes all kinds of hand postures and gestures allows for much more detailed control , and my personal favorite is having menus popup when you hold one hand in a certain position. These hand palm menus can be very easily made, using no code at all, just using stuff that's included in the MRKT2 out of the box.

But wait, there's more

Voice commands, remember that? The thing that everyone used like crazy and then quickly came back from, as it did not always work in noisy environments, especially with a lot of talk around. And making an odd gesture in empty space and looking weird is one thing, but shouting repeatedly at a device makes you feel very awkward indeed. Whatever they did to it, it's now way more accurate and confident at recognizing speech. Even in a very loud room with people talking. Speech control is everywhere in HoloLens 2, and very easy to use reliably.

And then there's eye tracking. Remember you had to move your whole head to point the gaze cursor? It now tracks your eyes. I knows what your are looking at. I use this in AMS HoloATC to make an image of the actual airplane pop up when you look at the model. There are four (or five, depending on what you include) events that you can easily track. I also learned that on a real life device I make that happen way too fast and too nervously. Having a real device, I will be able to fix this in the near future.

Eye tracking also has some extra benefits - first of all, it allows the device to use Windows Hello login using iris recognition. Second, calibrating is a lot easier and faster. No longer do you have first close your one eye, then very very precisely move your finger in the right slot for a couple of times, and repeat that for the other eye - you now simply have to track a few holograms with your eyes, as they move though your view. And you really should do that - Microsoft pushed the envelope a lot further when it comes to display technology, so if you don't calibrate properly, there's a lot more chance of having a fuzzy view. Fortunately the device has a setting that automatically starts the calibration routine when it detects the user has changed (which it presumably does using the iris scan).

In conclusion

HoloLens 2 is an amazing device, with an amazing display technology - but it's the interaction model that makes it really special. This is what takes in over the top, makes it natural, simple to learn and easy to use. The hand/eye tracking removes the barrier of artificial gestures and make wandering around and interacting with Holograms at lot easier. This will make use of the device in business settings - especially industrial and manufacturing environments - a lot easier.

I love living in the future:)

22 January 2020

Mixed Reality apps failing the WACK - revisited

Intro

Last September I wrote about various hoops you had to jump through to get a Mixed Reality app to appear in the store and not have it to fail  the Windows Application Certification Toolkit (WACK). It's time for a little update, since part of what I wrote is no longer necessary, and some things have changed subtly.

The good news

I wrote how you manually had to mess with Package.AppManifest.xml, particularly how you had to remove the DirectX 10 dependency to make it (still) downloadable for HoloLens 1 devices. Provided you have been diligently installing the service updates that are issued for this device, by now your HoloLens 1 should have been updated to no longer reject apps that have this requirement. So no need to take this step anymore.

The bad news

The trick I wrote how to make sure some unsupported DLLs were not included does not work anymore. We have to resort to more drastic measures.

The problem

To recap: if you generate a C++ solution from the recommended Unity version (2018.4.x, LTS branch) you get the following errors in the WACK

  • HolographicAppRemoting.dll has failed the AppContainerCheck check.
  • PerceptionDevice.dll has failed the AppContainerCheck check.
  • UnityRemotingWMR.dll has failed the AppContainerCheck check.
  • The Supported API test will list 10 errors concerning UnityRemotingWMR calling unsupported APIs
  • The Debug configuration test will tell youUnity RemotingWMR is only built in debug mode
  • And if you try to build for x86 or ARM, the Package sanity test will tell you HolographicAppRemoting.dll, PerceptionDevice.dll and UnityRemotingWMR.dll are only available for x64.

I suggested editing the “Unity Data.vcxitems” file that is inside your store projects and change the contents of the DeploymentContent-tags for these files to false. That worked swell. Until, somewhere between Unity and Visual Studio versions upgrade, some brilliant person thought it a good idea to recreate that file whenever you change the architecture. So when you change the architecture from "x86" to "ARM" - something that happens automatically while creating multi-platform packages - you are greeted with this:

M previous solution does not work anymore, as the three offending DLLs are automatically added back to the file, and thus the WACK fails again. Some more brute force is being required.

The solution

If you expand the Il2CppOutputProject, you can simply see the offending DLL's. The solution is as simple as selecting and deleting them.

And then you can generate your packages, run the WACK again... and may get another failure:

This depends on the version of the WACK you have. As as you can read here, the Store has been updated to ignore this error. In other words, if you are down to this error, you are good to go for submission and (successful) certification in the Windows Store.

Mind you - as soon as you regenerate the solution from Unity, the project file may or may not be overwritten and you will need to repeat this. So - always always run the WACK before you submit - but then again, that's what you should do anyway.

Conclusion

The ever changing constellations of Unity and Visual Studio keep us throwing curveballs, but fortunately there's always been a way around. So far. Happy submitting!

15 January 2020

Mixed Reality apps failing MS Store validation on "10.5.1 Personal information" - and how to fix it

Intro

After fellow MVP (and RD, although not fellow-) Philipp Bauknecht discovered a bug in my AMS HoloATC app while testing it on his HoloLens 2 (what lunatic runs HoloLenses with DE-DE settings anyway :P) I created a fix for that and submitted the fixed app to the Microsoft Store on Saturday January 12th. I assumed it would pass right though. It did not. Monday morning I got a failure message:

That was not what I hoped, and certainly not what I expected

Parsing the message

I will admit to being rather confused, because I have a privacy policy link. In the Store. But apparently something has changed. If you want all the details, you can find them in the new App Developer Agreement, but the text in the failure basically says it all:

"For in-product include the privacy policy URL under the settings section"

So I need to add a privacy policy in the app. Okay. But they are expecting it under the settings section. This is clearly written with a desktop app in mind. But I don't have a main screen, let alone I can put a hamburger menu somewhere and put a "Settings" entry in this.

Solution part 1

So I could clearly not obey the letter of the law, but I can try to act in spirit. So I ventured out to see if I could get a privacy policy in the app in a way that would be acceptable to the Store testers. All my apps have like a "help" function, that you can activate by saying "help" or "show help". I actually advertise that on start-up with a floating text that is shortly visible after starting up the app.

This help screen looked like this:

And now looks like this

I actually added a text and a quad behind it, just ahead of the quad that is the main backdrop.

And on that quad there a very simple and crude behavior

using Microsoft.MixedReality.Toolkit.Input;
using UnityEngine;
using UnityEngine.WSA;

public class BrowserActivator : MonoBehaviour, IMixedRealityPointerHandler
{
    [SerializeField]
    private string _urlToOpen;

    public void OnPointerDown(MixedRealityPointerEventData eventData)
    {
        Launcher.LaunchUri(_urlToOpen,false);
    }

    public void OnPointerDragged(MixedRealityPointerEventData eventData)
    {
    }

    public void OnPointerUp(MixedRealityPointerEventData eventData)
    {
    }

    public void OnPointerClicked(MixedRealityPointerEventData eventData)
    {
    }
}

that launches a browser window when you tap on the text. In the app it looks like this:

And if you tap the indicated text, the app will move to the background and you will see this:

Or whatever URL you put into _urlToOpen in the editor. I took the exact same URL as in the Store. I am pretty sure this won't win me any design awards - but that's not the goal here. Note: these images where created with a HoloLens 1, as I am still lacking a HoloLens 2 (if anyone from Microsoft reads this - yes, that's a hint ;) ).

Solution part 2

Since I have no settings section, I thought it prudent to tell the testers where they could find the link to the privacy policy. So under "Submission options" I explained the why, where and how like this:

"IMPORTANT NOTE: this app failed App Policies: 10.5.1 Personal Information. Specifically, it missed an in app privacy policy. I was suggested to add it under section "settings" but there is no sections "settings", in fact, there are not sections at all, since this is not running in a window.
I have added a privacy policy tappable text in the help screen. You can access that by saying "help" or "show help", after you have initially placed the airport. This opens a browser and shows the privacy policy.
"

Proof of the pudding

As you can see, following this procedure took me through Store validation successfully. In fact, it took me through it twice - as it's Atlanta twin app ATL HoloATC got certified within hours after submitting with this addition.

Conclusion

Resuming:

  • Have some way in your app to popup the privacy policy web page.
  • Explain in Submission options to the testers where they can find it.

We can discuss at length how useful things like this are, if and how this could be communicated better to developers, if there needs to be guidance first before this is implement for the nice market of Mixed Reality apps - that don't have a window nor a settings section - whatever, this works, and while there is no official guidance on how to deal with privacy policies in apps like this, this article show you a way forward. Presumably, for the testers it's also just simply a matter of being mandated to tick off a box required by the legal department. And this is apparently an acceptable way of getting the box ticked. Job done.

No code sample this time - I assume everyone making MR apps will be able to re-create the two lines of actual code in the behaviour. And possible come up with a more elegant solution. But that was not the point of this - the point was getting in the store without a legal blocker.