HoloLens CubeBouncer application part 5-using gravity, adding more speech & sound, and app icons

Preface

In the previous post I showed you how to integrate speech recognition. This post mostly builds upon stuff we now already know. It introduces actually one new concept. From the very first post you might recall I instructed you to turn off gravity in the rigid body component. Hence, the cubes float in the air. But they still have mass, so like any body a cube has inertia. You will notice that when you bounce two cubes – they keep on moving, although they slowly lose speed (because we also defined drag). If you were to set drag to zero, you get the effect as if the cubes move through vacuum – they just keep on moving, as Sir Isaac Newton described in the first of his three laws of motion. So if you were to suddenly switch on gravity, the floating cubes would drop to the floor. Of course, in real life you would not be able to switch gravity on and off at will, but in AR magic is just a matter of setting the right property. And that is exactly what we are going to do.

Down she goes!

We open the CubeManipulator script, and add a one line method:

public void OnDrop()
{
  _rigidBody.useGravity = true;
}

Did I mention using Unity feels a lot like cheating? I did, right? We also need to add a line to the top of the OnRevert method, or else the cube will move to it’s old position when we call “go to start”, but after that it will immediately drop to the floor again. That’s not what we want. when the cube returns, we turn on gravity off for that particular cube.

public IEnumerator OnRevert()
{
  _rigidBody.useGravity = false;
  //rest of method

And then it’s a matter of adding a new command to the SpeechManager:

if (cmd == DropCommand)
{
  if (GazeManager.Instance.Hit)
  {
    GazeManager.Instance.HitInfo.collider.gameObject.SendMessage("OnDrop");
  }
}

Of course you have to define “DropCommand” and add it to the the keyword recognizer like in the previous post, but that’s all. Rebuild the project from Unity, deploy the Visual Studio solution, look at a cube, say “drop” and it falls to the floor. And if you say “go to start” it will still go back to it’s old position and stay there. Easy, right?

Dropping and returning all cubes

By now I think you won’t find this very hard to understand anymore. First, we need actually implement the code in the MainStarter, as this is the object that knows which cubes are available:

public void RevertAll()
{
  foreach (var c in _cubes)
  {
    c.SendMessage("OnRevert");
  }
}

public void DropAll()
{
  foreach (var c in _cubes)
  {
    c.SendMessage("OnDrop");
  }
}

Every cube knows where it came from, so we only have to call “OnRevert” on every cube. In the same line, calling “OnDrop” for every cube will drop all cubes. And in the SpeechManager we just add two commands to recognize: “drop” and “total recall”, or whatever for phrases you choose for these commands. When the phrases are recognize, simply call the right method:

if (cmd == RevertCommand)
{
    _mainStarter.RevertAll();
}

if (cmd == DropAllCommand)
{
   _mainStarter.DropAll();
}

Of course you have to define the RevertCommand and the DropCommand fields again and add them to the KeywordRecognizer, but once again – very little code, powerful functionality. Clever use of and already very clever the SDK goes a long way.

Adding some missing sounds

So in the video I showed in the first post, “create new grid” gave a kind of ping-sound, a returning cube a whistling sound, and “total recall” made a kind of  “tadadaah” sound. I find these kind of affirmative sounds very useful for giving feedback to the user that the app understands you, although in practice you might go for a little less garish sounds than I did ;). 

I added three sounds to Assets/Custom/Audio in Unity, then added an audio source to the Managers game object. I only changed the settings “Play on awake” (to off) and Volume (to 0.5)

Then we proceed to MainStarter.cs, and we add two public AudioClips fields and a private AudioSource clip:

public AudioClip ReadyClip;

public AudioClip ReturnAllClip;

private AudioSource _audioSource;

The Audio source of course needs to be initialized in Start:

_audioSource = GetComponent();

At the top of the CreateGrid method we add one line:

private void CreateGrid(Vector3 hitPosition)
{
  _audioSource.PlayOneShot(ReadyClip);

And we do something similar at the top of the RevertAll method

public void RevertAll()
{
  _audioSource.PlayOneShot(ReturnAllClip);

Go back to Unity, drag the Audio assets “Ready” and “Return all” on top of the Main Starter Script in Managers. We have done that a couple of times already but the be sure, one more time a picture that shows what I mean here:

Then rebuild the project, re-deploy from Visual Studio, and sure enough you will hear the sounds the “pringgg!” sound when the grid is created (at app start-up, and when you say “create new grid”), and the “tadadaah” when you say “totall recall”

Then for the final sound – the whistling sound the cube makes when it returns. Well, that’s almost the same, only now we go to the CubeManipulator script, and add a field:

public AudioClip ComeBackClip;

And on top of the of the OnRevert method we just need this?

public IEnumerator OnRevert()
{
  _audioSource.PlayOneShot(ComeBackClip);

Go back to Unity, select the WortellCube prefab and drag the “Return” Audio assets on top of the field “Come Back Clip” that now has appeared on the CubeManipulator script component. Rebuild your project, deploy the Visual Studio solution and indeed, the cube now returns with a whistling sound. when your say “go to start”.

The problem now only is – when you say “total recall” now, the app will play the “tadadaah” sound – and a lot of time the whistling sound simultaneously, resulting in quite a cacophony. After all, we are calling the OnRevert method for every single cube. This is not a good user experience. So we will have to change a few things.

Some tweaks to improve the sound experience

We need some way to tell the cubes not to play their sound on “total recall”. To that end, we first need to change the OnRevert method as follows:

public IEnumerator OnRevert(object doPlaySound)
{
  if ((bool) doPlaySound)
  {
    _audioSource.PlayOneShot(ComeBackClip);
  }

Then we go back to the SpeechManager, and in the method KeywordRecognizer_OnPhraseRecognized we add “true” to the OnRevert message that is fired when “go to start” is recognized

if (cmd == GoToStartCommand)
{
  if (GazeManager.Instance.Hit)
  {
    GazeManager.Instance.HitInfo.collider.gameObject.SendMessage("OnRevert", true);
  }
}

In contrast, in the MainStarter script we need to add false in OnRevert:

public void RevertAll()
{
  _audioSource.PlayOneShot(ReturnAllClip);
  foreach (var c in _cubes)
  {
    c.SendMessage("OnRevert", false);
  }
}

And now if you say “totall recall” you will once again only hear the “tadadaa”, while a “go to start” still lets one cube return with the whistling sound.

Tiles and naming

I have added 5 files to the “Assets/Custom/UWPAssets” folder in Unity (see right). These are all 200% assets. So what you need to do is:

• Hit File/Build settings/Player settings
• Go to the right, to the inspector.
• Expand the Icon pane
• Scroll all the way down to “Universal 10 tiles and Logos”

• Expand the Square 44x44 Logo pane
• Scroll down to the “Scale 200%, (88x88 pixels)”
• Hit the “select” button in the square on the right
• Select “Square44x44Logo” from the popup

Your net result should be this.

Repeat this for the other for logo formats. When you are done and you collapse all the image panes, they should all have the suffix”(200)” now.

 

 

Finally, we scroll all the way to the top again, and change the following settings:

• Change the company name into whatever you like – I took “LocalJoost Ltd.” This not not mandatory
• Change “Product name” to “CubeBouncer”
• Hit the “Select” button on the “Default icon” square and once again select the 310x310 square logo
• Under “Icon”, change short name into “CubeBouncer”
• Check “Large tile” and “Wide tile” under “Show name on”.

Net result:

At this point it’s best to close Visual Studio, and delete the generated solution from the App folder. We have effectively changed the app title, yet you will notice that it does not have any effect if you keep the solution. If you generate the solution anew, you will see it’s name is different too – it’s now called CubeBouncer.sln, no longer CubeBouncerDemo.sln. If you deploy it to a HoloLens, you will see either this in the all apps menu

and this one when you have pinned it:

And that’s it! Were done!

Concluding remarks

After setting up Unity, adding air taps, force, spatial sound, voice recognition, gravity and some imagery we have a simple but functional HoloLens app that demonstrates a lot (but not nearly all) of the HoloLens interaction model. I hope you enjoyed this trip, I sure had a lot of fun building and documenting it. It was a very educating experience for me and I found it a very fitting subject for the 256th post on this blog ;). Should you have any remarks, questions or improvements, let me know.

The source code for the final project can be downloaded here.

Oh… There is still a very minor issue with it, though. I wonder if anyone actually finds it. András Velvárt is excluded from competition ;). Let me know if you find it.