Centralized reusable audio feedback mechanisms for Mixed Reality apps

Intro – feedback is key

Although speech recognition in Mixed Reality apps is very good, sometimes the best recognition fails, or you slightly mispronounced something. Or the command you just said is recognized, but not applicable in the current state. Silence,  nothing happens and you wonder - did the app just not understand me, is the response slow or what? The result is always undesirable – users wait for something to happen and nothing does. They start to repeat the command and halfway the app executes the first command after all, or even worse – they start start shouting, which makes for a quite embarrassing situation (both for user and bystanders). Believe me, I’ve been there. So – it’s super important to inform your Mixed Reality app’s user  that a voice command has been understood and is being processed right away. And if you can’t process it, inform the user of that as well.

What kind of feedback?

Well, that’s basically up to you. I usually choose a simple audio feedback sound – if you have been following my blog or downloading my apps you are by now very familiar with the ‘pringggg’ sound I use in every app, be it an app in the Windows Store or one of my many sample apps on GitHub. If someone uses a voice command that’s not appropriate in the current context or state of the app, I tend to give some spoken feedback, telling the user that although the app has understood the command, can’t be executed now and if possible for what reason. Or prompt for some additional action. For both mechanisms I use a kind of centralized mechanism that uses my Messenger behaviour, that already has played a role in multiple samples.

Project setup overview

The hierarchy of the project is as displayed below, and all is does is showing the user interface on the right:

If you say “Test command”, you will hear the “pringggg” sound I already described, and if you push the button the spoken feedback “Thank you for pressing this button”. Now this is rather trivial, but it only serves to show the principle. Notice, by the way, the button comes from the Mixed Reality Toolkit examples – I described before how to extract those samples and use them in your app. 

The Audio Feedback Manager and Spoken Feedback Manager look like this:

The Audio Feedback Manager contains an Audio Source that just contains the confirmation sound, and a little script “Confirm Sound Ringer” by yours truly, which will be explained below. This sound is intentionally not spatialized, as it’s a global confirmation sound. If it was spatialized, it would also be localized, and the user would be able to walk away from confirmation sounds or spoken feedback, which is not what we want.

The Spoken Feedback Manager contains an empty Audio Source (also not spatialized), a Text To Speech Script from the Mixed Reality Toolkit, and a the “Spoken Feedback Manager’ script also by me.

ConfirmSoundRinger

using HoloToolkitExtensions.Messaging;
using UnityEngine;

namespace HoloToolkitExtensions.Audio
{
    public class ConfirmSoundRinger : MonoBehaviour
    {
        void Start()
        {
            Messenger.Instance.AddListener<ConfirmSoundMessage>(ProcessMessage);
        }

        private void ProcessMessage(ConfirmSoundMessage arg1)
        {
            PlayConfirmationSound();
        }

        private AudioSource _audioSource;
        private void PlayConfirmationSound()
        {
            if (_audioSource == null)
            {
                _audioSource = GetComponent<AudioSource>();
            }
            if (_audioSource != null)
            {
                _audioSource.Play();
            }
        }
    }
}

Not quite rocket science. If a message of type ConfirmSoundMessage arrives, try to find an Audio Source. If found, play the sound. ConfirmSoundMessage  is just an empty class with not properties or methods whatsoever – it’s a bare signal class.

SpokenFeedbackManager

Marginally more complex, but not a lot:

using HoloToolkit.Unity;
using HoloToolkitExtensions.Messaging;
using System.Collections.Generic;
using UnityEngine;

namespace HoloToolkitExtensions.Audio
{
    public class SpokenFeedbackManager : MonoBehaviour
    {
        private Queue<string> _messages = new Queue<string>();
        private void Start()
        {
            Messenger.Instance.AddListener<SpokenFeedbackMessage>(AddTextToQueue);
            _ttsManager = GetComponent<TextToSpeech>();
        }

        private void AddTextToQueue(SpokenFeedbackMessage msg)
        {
            _messages.Enqueue(msg.Message);
        }

        private TextToSpeech _ttsManager;

        private void Update()
        {
            SpeakText();
        }

        private void SpeakText()
        {
            if (_ttsManager != null && _messages.Count > 0)
            {
                if(!(_ttsManager.SpeechTextInQueue() || _ttsManager.IsSpeaking()))
                {
                    _ttsManager.StartSpeaking(_messages.Dequeue());
                }
            }
        }
    }
}

If a SpokenFeedbackMessage comes in, it’s added to the queue. In the Update method, SpeakText is called, which first checks if there are any messages to process, then checks if the TextToSpeech is available – and if so, it pops the message out of the queue and actually speaks it. The queue has two functions. First, the message may come from a background thread, and by having SpeakText called from Update, it’s automatically transferred to the main loop. Second, it prevents messages being ‘overwritten’ before they are even spoken.

The trade-off of course is that you might stack up messages if the user quickly repeats an action, resulting in the user getting a lot of talk while the action is already over.

On the Count > 0 in stead of any – apparently you are to refrain from using LINQ extensively in Unity apps, as this is deemed inefficient. It hurts my eyes to see it used this way, but when in Rome…

Wiring it up

There is a script SpeechCommandExecuter sitting in Managers, next to a Speech Input Source and a Speech Input Handler, that is being called by the Speech Input Handler when you say “Test Command”. This is not quite rocket science, to put it mildly:

public class SpeechCommandExecuter : MonoBehaviour
{
    public void ExecuteTestCommand()
    {
        Messenger.Instance.Broadcast(new ConfirmSoundMessage());
    }
}

As is the ButtonClick script, that’s attached to the ButtonPush:

using HoloToolkit.Unity.InputModule;
using HoloToolkitExtensions.Audio;
using HoloToolkitExtensions.Messaging;
using UnityEngine;

public class ButtonClick : MonoBehaviour, IInputClickHandler
{
    public void OnInputClicked(InputClickedEventData eventData)
    {
        Messenger.Instance.Broadcast(
            new SpokenFeedbackMessage { Message = "Thank you for pressing this button"});
    }
}

The point of doing it like this

Anywhere you now have to give confirmation or feedback, you now just need to send a message – and you don’t have to worry about setting up an Audio Source, a Text To Speech and wiring that up correctly. Two reusable components take care of that. Typically, you would not send the conformation directly from the pushed button or the speech command – you would first validate if the command can be processed in the component that holds the logic, and then give confirmation or feedback from there.

Conclusion

I hope to have convinced you of the importance of feedback, and I showed you a simple and reusable way of implementing that. You can find the sample code, as always, on GitHub.

Short tip: using the UI components from the Mixed Reality Toolkit examples

The Mixed Reality Toolkit is the foundation for nearly all apps built for HoloLens and Windows Mixed Reality immersive head sets. It has a lot of indispensable components that make building these apps not so much a breeze, but al least a lot less tedious. It’s a bit sparse as far as actual UI components is concerned. They are now in the process of merging the awesome UI stuff from the Mixed Reality Design Labs project in it. This process is not ready by far, so far there’s only some simple buttons in it.

But if you look into this part of the code in the Mixed Reality Toolkit on GitHub , you will notice there being three folders:

The last one contains the actual Mixed Reality Toolkit, the 2^nd the Unit Test which are not very interesting for us right now, but the first contains a whole lot of examples. And in those examples, a whole lot of nice UI controls like a host of buttons, a slider, a check box, toggles, toolbars, a very cool loader animation, helpers to draw lines and curves and probably a lot I have not noticed yet. There’s also a lot of demo scenes to show them off. But the Examples folder contains 114 MB of assets, almost doubling the size of already hefty MRKT itself.

Extracting only the UI elements is pretty simple:

• Clone the project from Github
• Copy the whole HoloToolkit-Examples into your projects Assets folder, next to the HoloToolkit
• Delete everything but the these two folders:
• HoloToolkit-Examples\UX
• HoloToolkit-Examples\Prototyping
• [optional] remove the “Scenes” subfolder from both the UX and the Prototyping folder. But maybe you should have look around first.

So you will end up with the following asset folder structure:

And then you can use cool stuff like this, as shown in the InteractiveButtonComponents scene:

Or these holographic buttons as you can find in ObjectCollectionExample scene (amongst a lot of other stuff)

And these samples of how to draw lines in the ‘air’ in the LineExamples scene

So far I have only used the icon buttons, but those are pretty cool in itself because they quite resemble the buttons in the Holographic shell, so your app’s user should immediately recognize what they are for.

No separate code this time, as all the code is in the MRTK repo itself. I do encourage your to root around through it, there’s quite some cool surprises in there.

Getting your logos and splash screens right for Mixed Reality apps

Intro

Way too often I still see Mixed Reality apps with default (white) splash screens, app windows, or even worse – the default Unity Icon. When Mixed Reality was HoloLens-only and brand new, you kinda got away with that, as the WOW factor of the actual app eclipsed almost everything else, and the iconography did not matter much as there was only the Holographic Start menu. Also, although judging from my download numbers a fair number of HoloLenses are in use worldwide, the number is pretty limited compared to, say, PCs.

These days of leeway are over. With the advent of the Fall Creators Update, Mixed Reality apps can run on PCs with the right hardware – and in stead of a $3000+ device, a $400 headset will do. And as soon as I made an app available for immersive headsets, my download numbers exploded. Much more eyeballs so much more visibility, and in Windows, the flexible start menu and it’s different tile sizes makes omissions in the iconography glaringly visible. I also notice ‘ordinary’ users are a lot less forgiving. So, time to step up your game.

Note: I am currently using Unity 2017.2.0p2 MRTP5. Things may change. They tend to do that very much in this very new and exiting field :)

So what do you need?

At the very minimum, and icon of size 620x620 and a 1240x600 splash screen. 620x620 is the 310x310 icon at 200% scale. I tend to used 200% icons as they are the default sizes that are created by the Unity editor. I give the start 620x620 file the default name “Square310x310Logo.scale-200.png” and go from there.

In my Unity Assets folder I create a folder “UWPassets” and I drag the “Square310x310Logo.scale-200.png” in there. I then proceed to make the following file from the first one

• Wide310x150Logo.scale-200.png (620x310 pixels)
• Square150x150Logo.scale-200.png (300x300 pixels)
• Square71x71Logo.scale-200.png (142x142 pixels)
• Square44x44Logo.scale-200.png (88x88 pixels)
• StoreLogo.scale-100.png (50x50 pixels)

Do pay attention to the last one – that‘s 100% scale, so it’s actual size. To the right you see my actual icon.

These are just the standard sizes for UWP apps, if you are coming in from a Unity perspective you may not be familiar with them.

And then, as I said, you need the splash screen. This needs to be 1240x600. I have created this ‘awesome’ splash screen:

I suggest you take something that says something more about your app, but this will to for an example.

The result in the Unity editor:

Putting icons in the right place in Unity

This is not rocket science: Hit File/Build Settings and then the button “Player Settings”. This will open the “PlayerSettings” pane on the right side of your editor. First, you open the pane “Icon” and then the section “Store logo”. From the UWPAssets folder, drag the “StoreLogo.scale-100” into the top box.

You can leave the rest empty. Then skip all the other sections, scroll all the way down to the section “Universal 10 tiles and logos”. Open “Square 44x44 Logo”

Drag the “Square44x44Logo.scale-200.png” into the 4th box from the top, marked 200%. Then open section “Square 71x71 Logo” and drag  “Square71x71Logo.scale-200.png” in. Proceed similarly with the remaining three sizes till all the 200% boxes are filled.

Splash screen, splash screen and splash screen

Now dig this – there are actually three places to put splash screens in. Potentially you can have different splash images - I tend to use the same one everywhere. So scroll further down to the Splash Screen section. First up, on top, is the “Virtual Reality Splash Image”. You can drag your splash screen in there if you want.

Next up, a bit further down, is the Windows splash screen:

This is the most important one. This appears on your desktop when the app starts, it also is shown in a rectangle in Immersive Headsets during the app ‘transition scene’ (when your Cliff House is replaced by the app), and it’s displayed on your default app screen:

And finally, there’s Windows Holographic Splash screen:

This is actually important in a HoloLens. As this does not have a ‘transition scene’, it shows a floating splash screen for a few seconds. You can see it in an immersive headset too, but usually for a very short time – most of the time you see it flash by in a split second before the actual app appears.

So where is the very first splash screen for? To be honest, I have no idea. Possibly it’s for other, non Windows Mixed Reality based apps.

Just tiny thing left

I tend to generate the app in a subfolder “App” in the root of the project. Typically, I place a .gitignore in there that, look like this:

*.*
*

true to it’s name, it ignores – everything. Because, well, it’s a generated app, right? But if you go into the generated app’s assets (App\SplashScreen\Assets) you will notice something odd:

Although we nicely created an icon for every size, there’s still this pesky “Square44x44Logo.targetsize-24_altform-unplated.png” icon with the default logo that – interestingly, is 24x24 pixels big. So where is this used? Apparently in the task bar and – as I have noticed – in the developer portal. So although your customers won’t see it in the Store (I think), you will, it will clutter op your app list in the developer portal, and that’s annoying. I have found no way to actually create this from Unity, so I take the easy approach: I create yet another icon, now 24x24, overwrite that “Square44x44Logo.targetsize-24_altform-unplated.png” in  App\SplashScreen\Assets and add it to Git manually there. Unity won’t overwrite it, and if it get’s deleted or overwritten, I can always revert. You just need to remember to check the icon before actually submitting.

Conclusion

So now you know how to properly assign the minimum iconography and splash screens, and I let’s agree on no longer publishing stuff with default icons or empty splash screens to the store, right? ;)

Project code – although it does not actually do anything – can be found here. But it’s useful reference material.