07 February 2018

Building a floating audio player in Mixed Reality

Intro

imageAs I promised in my previous blog post, I would write about how I created the floating audio player designed to easily demonstrate how to download and play audio files in Mixed Reality (or actually, just Unity, because the code is not MR specific). I kind of skipped over the UI side. In this post I am going to talk a little more about the floating audio player itself. This code is using the Mixed Reality Toolkit and so actually is Mixed Reality specific.

Dissecting the AudioPlayer prefab

The main game object

imageThe AudioPlayer consists out of two other prefabs, a SquareButton and a Slider. I have talked about this button before, so I won’t go over that one in detail again. The main game object of the AudioPlayer has an AudioSource and two extra scripts. The simple version of the Sound Playback Controller was already described in the previous blog post, and will be handled in great detail here. The other script is a standard Billboard script from the Mixed Reality toolkit. It essentially keeps the object rotated towards the camera, so you will never see it from the side of the backside where it’s hard to read and operate. Note I have restricted pivot axis to Y, so it only rotates over a vertical axis.

The button

imageIt’s a fairly standard SquareButton, and I have set the text and icon as I described here. Now that button only shows in the editor, the runtime text and the icon are set by a simple script that toggles icon and text, so that the button cycles between being a “Play” and a “Pause” button. That script is pretty easy:

using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class IconToggler : MonoBehaviour, IInputClickHandler
{
    public Texture2D Icon1;

    public Texture2D Icon2;

    public string Text1;

    public string Text2;

    private TextMesh _textMesh;

    private GameObject _buttonFace;

    void Awake ()
    {
        _buttonFace = gameObject.transform.
           Find("UIButtonSquare/UIButtonSquareIcon").gameObject;
        var text = gameObject.transform.Find("UIButtonSquare/Text").gameObject;
        _textMesh = text.GetComponent<TextMesh>();
        SetBaseState();
    }

    public void SetBaseState()
    {
       _textMesh.text = Text1;
       _buttonFace.GetComponent<Renderer>().sharedMaterial.mainTexture = Icon1;
    }

    private float _lastClick;

    public void OnInputClicked(InputClickedEventData eventData)
    {
        if (Time.time - _lastClick > 0.1)
        {
            _lastClick = Time.time;
            Toggle();
        }
    }

    public void Toggle()
    {
        var material = _buttonFace.GetComponent<Renderer>().sharedMaterial;
        material.mainTexture = material.mainTexture == Icon1 ? Icon2 : Icon1;
       _textMesh.text = _textMesh.text == Text1 ? Text2 : Text1;
    }
}

It has four public properties, as already is visible in the image: Image1 and Text1 for the default image and text (“Play”), Image 2 and Text 2 for the alternate image and text (“Pause”). The Awake method grabs some objects within the button itself, then sets the base state – which is, the default icon and text.

It also implements IInputClickHandler, so the user can tap it. In OnInputClicked it calls the Toggle method. That then toggles both text and image. Notice there’s simple time based guard OnInputClicked. This is to prevent the button from sending a burst of click events. In the Unity editor, I mostly get two clicks every time I press the XBox controller A button, and then nothing happens. Annoying, but easily mitigated this way.

The Slider

I can be short about that one. I did not create that, but simply nicked it from the Mixed Reality Toolkit Examples. It sits in HoloToolkit-Examples\UX\Prefabs. I like making stuff, but I like stealing reusing stuff even better.

The extended Sound Playback Controller

Let’s start at Start ;). Note: the BaseMediaLoader was handled in the previous blog post,

public class SoundPlaybackController : BaseMediaLoader
{
    public AudioSource Audio;

    public GameObject Slider;

    public GameObject Button;

    private SliderGestureControl _sliderControl;

    private IconToggler _iconToggler;

    public AudioType TypeAudio = AudioType.OGGVORBIS;

    void Start()
    {
        _sliderControl = Slider.GetComponent<SliderGestureControl>();
        _sliderControl.OnUpdateEvent.AddListener(ValueUpdated);
        Slider.SetActive(false);
        Button.SetActive(false);
        _iconToggler = Button.GetComponent<IconToggler>();
    }
}

In the Start method, we first grab a bunch of stuff. Note the fact that we not only turn off the slider control but also actually attach an event handler to that.

We continue with StartLoadMedia and LoadMediaFromUrl

protected override IEnumerator StartLoadMedia()
{
    Slider.SetActive(false);
    Button.SetActive(false);
    yield return LoadMediaFromUrl(MediaUrl);
}
private IEnumerator LoadMediaFromUrl(string url) { var handler = new DownloadHandlerAudioClip(url, TypeAudio); yield return ExecuteRequest(url, handler); if (handler.audioClip.length > 0) { Audio.clip = handler.audioClip; _sliderControl.SetSpan(0, Audio.clip.length); Slider.SetActive(true); Button.SetActive(true); _iconToggler.SetBaseState(); } }

The override from StartLoadMedia in this version turns off the whole UI while we are actually loading data, and turns it on when we are done loading. Since that fails when we load MP3, the MP3 player in the demo project disappears and on startup. The others one disappear too, in fact, but immediately appear again since we are loading small clips. This goes so fast you can’t even see it.

LoadMediaFromUrl not only executes the request and sets the downloaded clip to the Audio Souce, as we saw before, but we also set the span of the Slider Control between 0 and the length of the AudioClip in seconds. Easy, right?

Now the Update method, which as you know is called 60 times per second, is the trick to keeping the slider equal to the the current time of the clips that’s now playing:

protected override void Update()
{
    base.Update();
    if (Audio.isPlaying)
    {
        _sliderControl.SetSliderValue(Audio.time);
    }
    if (Mathf.Abs(Audio.time - _sliderControl.MaxSliderValue) < 0.1f)
    {
        Audio.Stop();
        Audio.time = 0;
        _iconToggler.SetBaseState();
        _sliderControl.SetSliderValue(0);
    }
}

Thus if the audio clip plays, the slider moves along. It’s not quite rocket science. If the clip has nearly finished playing, it is stopped and everything is set to the base state: the icon, the time of the audio clip, and the slider is set to 0 again.

And finally – remember that event handler we added to the OnValueUpdated event of the slider? Guess what:

private void ValueUpdated()
{
    Audio.time = _sliderControl.SliderValue;
}

It’s the opposite of the third line of Update – now we set the Audio time to the Slider value.

Conclusion

And that’s it. You can simply use some out-of-the-box components in the Mixed Reality Toolkit and/or it’s examples to build a simple but effective control to play audio. You can grab the demo project (it’s still the same) from here.

04 February 2018

Downloading audio files from Azure blob storage and playing them in Mixed Reality apps

Intro

All but the most trivial apps have some kind of back end. In the past, I have written about accessing all kinds of services from HoloLens and Mixed Reality apps, But apart from mere data, you can download all kind of media from external sources and use those in you Mixed Reality apps. This is highly useful if you want to change used media, use different files for like instructions depending on some factor. Or heck, some random background music. In fact, what I will be describing is not even Mixed Reality specific – the principle can be used in any Unity app, although the code that sits around to demonstrate the workings definitely only works for Mixed Reality.

Oh, and by the way – for us Microsoft geeks “back end” equals to “Microsoft Azure” – if it’s not for the competitive pricing, then for the way Microsoft makes it easy for developers to get going (heaven knows this was quite different in the early days). But to be clear: this will work with any backend that hosts files.

This post will be 2-part: the first part concentrates on the actual technique of downloading and playing audio files, the second part will explain the ‘floating audio player’ I built to make this easily demonstrable. The floating part in ‘floating player’ should be taken in the most literal way possible:

image

imageYou can find it in the demo project. You can just jump to there if you want to skip all my mumbling.

Global overview

If you open the project in Unity, you will not see one but three floating players, next to each other. They will all attempt to play on of my ringtones – an excerpt from the Doctor Who theme song (one is a nerd or one is not). But it will attempt to use one of three different audio formats – the well-known MP3, Ogg Vorbis, and WAVE audio format (aka ye good ole’ WAV).

image

To hear them, either build the app or just hit the play button. If you choose the second option, you will need to attach an XBox One controller to you PC, to steer the cursor and click on the buttons. You will notice the Ogg and the WAV file playing nicely. The mp3 one won’t. In fact, the player just disappears before you even can get to it. We will get to that later.

Using UnityWebRequest

In previous posts I have shown you either how to use the Unity WWW class, or how to resort to pure UWP code and use HttpClient/HttpRequestMessage. It seems like the WWW class is being deprecated, although I have no official information on that. However,the new (third) kid on the block seems to be UnityWebRequest.

This is a bit on an oddball. It uses a handler. Generally, a UnityWebRequest looks like this:

var request = UnityWebRequest.Get(url);
request.downloadHandler = handler;
yield return request.SendWebRequest();

The second line is optional – if the you don’t set the handler, it’s the default DownloadHandler class, which sports a “text” property you can query. This is very useful for accessing data services like an API app on Azure App Service. If you want to download audio files, you will need a DownloadHandlerAudioClip handler.

Putting it in code

To make this all work easily, I have created the following base class for download loading media:

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;

public abstract class BaseMediaLoader : MonoBehaviour
{
    public string MediaUrl;

    private string _currentMediaUrl;

    protected virtual void Update()
    {
        if (_currentMediaUrl != MediaUrl)
        {
            _currentMediaUrl = MediaUrl;
            StartCoroutine(StartLoadMedia());
        }
    }

    protected abstract IEnumerator StartLoadMedia();

    protected IEnumerator ExecuteRequest(string url, DownloadHandler handler)
    {
        var request = UnityWebRequest.Get(url);
        request.downloadHandler = handler;
        yield return request.SendWebRequest();
    }
}

MediaUrl is an url that points to a place where the actual audio file can be downloaded – in this case, as SAS link to a file in my own Azure blob storage. Every time Update is called, it checks if the MediaUrl has been changed, and if so, it starts the StartLoadMedia in the background. ExecuteRequest is a helper method that can be used from a StartLoadMedia. Like this:

using System.Collections;
using UnityEngine;
using UnityEngine.Networking;

public class SoundPlaybackController : BaseMediaLoader
{
    public AudioSource Audio;

    public AudioType TypeAudio = AudioType.OGGVORBIS;

    protected override IEnumerator StartLoadMedia()
    {
        yield return LoadMediaFromUrl(MediaUrl);
    }

    private IEnumerator LoadMediaFromUrl(string url)
    {
        var handler = new DownloadHandlerAudioClip(url, TypeAudio);
        yield return ExecuteRequest(url, handler);
        if (handler.audioClip.length > 0)
        {
            Audio.clip = handler.audioClip;
        }
    }
}

This is a very reduced version of the SoundPlaybackController that is in the demo project, concentrating only on the actual downloading and playing of the data.

The override of StartLoadMedia simply calls StartLoadMedia, which proceeds to create a DownloadHandlerAudioClip handler. Now the odd thing is, this handler wants and url as well as the request. Why this is so, I have no idea. Also notice the fact you need to supply the handler with the type of audio you are going to download – there’s a AudioType enumeration for that.

If the audio has been downloaded successfully, you only have to set the “clip” property of an AudioSource to the handler’s “audioClip” property, and call the Audiosource’s “Play” method. And you are good to go.

Audio types are important

Since the TypeAudio property is public, the Unity editor makes a nice dropdown for us to select the type of audio:

image

I’ll be the first one to admit I haven’t heard of most of these file types, let alone know them. Actually, before I started, I only knew MP3 and WAV. I learned to know Ogg Vorbis. And for a good reason too. I already mentioned the fact the left (mp3) player disappears when you start the code. The SoundPlaybackController that I actually created (not the simple version above) hides the UI of the whole player while actually downloading the audio file. You might have noticed an error in the Unity status bar when you run the code. That actually says:

Streaming of 'mp3?st=2018-02-02t15%3a51%3a00z&se=2020-02-03t17%3a51%3a00z&sp=rl&sv=2017-04-17&sr=b&sig=kymvql5q1yyr%2bqilcxhfc3popwo56vd0ejibennldzw%3d' on this platform is not supported
UnityEngine.Networking.DownloadHandlerAudioClip:get_audioClip()
<LoadMediaFromUrl>c__Iterator0:MoveNext() (at Assets/App/Scripts/SoundPlaybackController.cs:34)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

imageThe code in that player crashes, and never gets to showing the UI again. Well, that’s a bummer. Unity simply does not seem to support MP3 on ‘this platform’, which is apparently UWP. You can convert it to WAV, but unless you like to burn a lot of battery downloading stuff, you must be pretty much out of your mind doing so – Ogg Voribs is a much better option, as this simple list of files shows.

And Audacity, that good old workhorse of audio-artists and podcasters all around the globe makes conversion easy, so why not use it, right.

image

Conclusion

The very short version of this blog post: use UnityWebRequest and DownloadHandlerAudioClip to download an audio clip from Azure blob storage, and if you value your users’ bandwidth, their devices’ battery life, and your own sanity – use Ogg Vorbis audio files.

I have not tried all the other audio files types, simply because I did not have to do so – Ogg Vorbis works fine. WAV too, but is absurdly big compared to MP3 and Ogg Vorbis. In addition, I have no idea what those audio types are, how I should create/convert them, and why I should use them. To paraphrase Star Trek’s TOS medical officer Dr. Leonard “Bones” McCoy – I am a developer, not an audio engineer. 

Next time, I will explain the workings of the floating audio player around this code.