Improving Azure Custom Vision Object Recognition by using and correcting the prediction pictures

Intro

Last month I wrote about integrating Azure Custom Vision Object Recognition with HoloLens to recognize to label objects in 3D space. I wrote the prediction went pretty well, although I used only 35 pictures. I also wrote the process of taking, uploading and labeling pictures is quite tedious.

Improving on the go

It turns out Custom Vision retained all the pictures I uploaded in the cause of testing. So every time I used my HoloLens and asked Custom Vision to locate toy aircraft, it stored that in the cloud with it's prediction. And the fun thing is, you can use those pictures to actually improve your model again.

After some playing around with my model my (for the previous blog post about this subject) , I clicked the prediction tab, and I found about 30 pictures - each for every time I used the model from my HoloLens. I could use those to improve my model. After that, I did some more testing using the HoloLens to show you how it's done. So, I clicked the Predictions tab and there were a couple of more pictures:

If we select the first picture, we see this:

The model has already annotated the areas where it thinks is a an airplane in red. Interestingly now the model is a lot better than it originally was (when it only featured my pre-loaded images) as it now recognizes the DC-3 Dakota on top - that it has never seen before - as an airplane! And even the X-15 (the black thing on the left ) is recognized. Although the X-15 had a few entries in the training images it barely looks like an airplane (for all intents and purpose it was more a spaceship with wings to facilitate a landing).

I digress. You need to click every area you want to confirm:

And when you are done, and all relevant areas are white:

Simply click the X top right. The image will now disappear from the "Predictions" list and end up in the "Training images" list

Some interesting things to note

The model really improved from adding the new images. Not only did it recognize the DC3 'Dakota' that had not been in the training images, but also this Tiger Moth model (the bright yellow one) that it had never  seen before:

Also, it stopped recognizing or doubting things like the HoloLens pouch that's lying there, and my headphones and hand were also recognized as 'definitely not an airplane'

Yet, I also learned it's dangerous to take the same background over and over again. Apparently the model starts to rely on that. If I put the Tiger Moth on a dark blue desk chair in stead of a light blue bed cover

Yes...  the model is quite confident airplane in the picture but it's not very good at pinpointing it.

And as far as the Curtiss P40 'Kittyhawk' goes - even though it has been featured extensively in both the original training pictures and the ones I added from the Predictions, this no is success either. The model is better at pinpointing the aircraft, but considerably less sure it is an aircraft. And the outer box, that includes the chair, gives a 30.5%. So in looks that to make this model even more reliable I still need more pictures but then on other background, more different lighting, etc.

Conclusion

You don't have to take very much pictures up front to incrementally improve a Custom Vision Object Recognition model - you can just iterate on it's predictions and improve them. It feels a bit like teaching a toddler how to build something from Legos - you first show the principle, then let them muck around, and every time things goes wrong, you show how it should have been done. Gradually they get the message. Or at least, that's what you hope. ;)

No (new) code this time, as the code from last time is unchanged.

Disclaimer - I have no idea how much prediction pictures are stored and for how long - I can imagine not indefinitely, and not an unlimited amount. But I can't attach numbers to that.

Mixed Reality Toolkit vNext–dependency injection with extension services

Intro

The Mixed Reality Toolkit vNext comes with an awesome mechanism for dependency injection. This also takes away a major pain point – all kinds of behaviors that are singletons that are called from everywhere, leading to all kind of interesting timing issues - and tightly coupled classes. This all ends wit extension services, which piggyback on the plugin structure of the MRKT-vNext. In this post I will describe how you make, configure and use such an extension service

Creating an extension service

A service that can be used by the extension service framework (and be found by the inspector dropdown that I will show later) needs to implement IMixedRealityExtensionService at the very least. But of course we want to have the service make do something useful so I made a child interface:

using Microsoft.MixedReality.Toolkit.Core.Interfaces;

namespace Assets.App.Scripts
{
    public interface ITestDataService : IMixedRealityExtensionService
    {
        string GetTestData();
    }
}

the method GetTestData is the method we want to use.

Any class implementing IMixedRealityExtensionService needs to implement six methods and two properties. And to be usable by the framework, it needs to have this constructor:

<ClassName>(string name, uint priority)

To make this a little more simple, the MRKT-vNext contains a base class BaseExtensionService that provides default implementation for all the required stuff. And thus we can make a TestDataService very simple, as it a) implements all properties and b) forces us to provide the necessary constructor:

using Microsoft.MixedReality.Toolkit.Core.Services;
using UnityEngine;

namespace Assets.App.Scripts
{
    public class TestDataService : BaseExtensionService, ITestDataService
    {
        public TestDataService(string name, uint priority) : base(name, priority)
        {
        }

        public string GetTestData()
        {
            Debug.Log("GetTestData called");
            return "Hello";
        }
    }
}

Registering the service in the framework

Check if a custom profile has been selected. Assuming you have followed the procedure I described in my previous post, you can do you this by selecting the MixedRealityToolkit game object in your scene and then double-clicking the “Active Profile” field

If the UI is read-only, there’s no active custom profile. Check if there’s a profile in MixedRealityToolkit-Generated/CustomProfiles and drag that on top of the ActiveProfile field of the MixedRealityTool object. If there’s no custom profile at all, Please refer to my previous blog post.

Scroll all the way down to Additional Service Providers.

Click the </> button. This creates a MixedRealityRegisteredServiceProvidersProfile in
MixedRealityToolkit-Generated/CustomProfiles and shows this editor.

Click “+ Register a new Service Provider”. This results in a “New Configuration 8” that if you expand it, looks like this:

If you click the “Component Type” drop down you should be able to select “Assets.Apps.Scripts” and then “TestDataService”.

I also tend to give this component a bit more understandable name so the final result looks like this:

Calling the service from code

A very simple piece of code shows how you can then retrieve the and use the service from the MixedRealityToolkit:

using Microsoft.MixedReality.Toolkit.Core.Services;
using UnityEngine;

namespace Assets.App.Scripts
{
    public class TestCaller : MonoBehaviour
    {
        private void Start()
        {
            var service  = MixedRealityToolkit.Instance.GetService<ITestDataService>();
            Debug.Log("Service returned " + service.GetTestData());
        }
    }
}

Notice I can retrieve the implementation using my own interface type. This very is similar to what we are used to do in ‘normal’ IoC containers like Unity (the other one), AutoFac, SimpleIoC. If you attach this behaviour to any game object in the hierarchy (I created an empty object “Managers” to this extent), and run this project, you will simply see this in the console:

It’s not spectacular, but it proves the point that this is working as expected

Conclusion

MRTK-vNext provides a very neat visual select mechanism for wiring up dependency injection that is very easy to use. I can also easily retrieve implementations of the service using an interface, just like any other IoC platform. The usage of profiles makes it very flexible and reusable. This alone makes it a great framework, and then I have not even looked into the cross-platform stuff. That I will do soon. Stay tuned.

In the mean time, the demo project can be found here.

Mixed Reality Toolkit vNext–setting up a project

Intro

You might have heard it – the Mixed Reality Toolkit folks are busy with a major rewrite. The original MRTK was intended to accelerate applications targeted toward HoloLens and (later) Windows Mixed Reality immersive headsets. The new version “aims to further extend the capabilities of the toolkit and also introduce new features, including the capability to support more VR/AR/XR platforms beyond Microsoft's own Mixed Reality setup”.

I’ve been quite busy but finally found some time to play with it. And while I am doing that, I am going to shoot off some smaller and bigger blogs about things I learned – both for myself to remember and you to enjoy.

Be advised: it’s still heavy in development. The first beta release explicitly states:

This is a pre-release version and is not feature complete.

• This release is not intended for production use.
• This release contains many breaking changes from previous HoloToolkit and Mixed Reality Toolkit vNext releases.

So let’s dive in. In this first blog post on this subject, I will simply describe how to set up an empty project.

Cloning the latest MRTK-vNext from GitHub

This is pretty standard procedure. I prefer to use TortoiseGit for this, as I seem to be of the dying creed that’s not particularly fond of command lines. The repo is here (as it has been for a while)

After you have cloned the project, check out branch mrtk_development. This is the bleeding edge. This is where the most things happen (and the most things break ;) ).

Creating a new Unity project

Also fairly standard. You will need Unity 2018.2.18f1 for that. Or, by the time you will read this, probably an ever newer version. After you have created the project, close Unity again.

Adding the MRTK to you project

From the MRKT repo, copy the following folders and files to the assets folder of your project:

• MixedRealityToolkit
• MixedRealityToolkit-SDK
• MixedRealityToolkit.meta
• MixedRealityToolkit-SDK.meta

Configuring the MRKT-vNext components in your scene

Configuring the MRTK has become a whole lot easier.

• Open and existing scene or create one.
• Click Mixed Reality Toolkit/Configure
• You will see two game objects appear in your scene
  

Now Unity starts to complain about there being no camera tagged as MainCamera. This can easily be fixetd by tagging it manually:

Basically you know have an empty project

Preparing a custom configuration

If you click the MixedRealityToolkit game object, on the DefaultMixedRealityToolkitConfigurationProfile field:

you will see this this appear in the inspector

If you then click the button “Copy & Customize” it will create a new folder MixedRealityToolkit-Generated, and in that a CustomProfiles folder. And in that a MixedRealityToolkitConfigurationProfile

Make sure to check that the new profile is actually applied:

You will now see that the settings are no long greyed out, and now you can change and swap out components.

The fun thing is, these are not longer all MonoBehaviours and (most of all) no longer singletons. The MixedRealityToolkit class is the only ‘singleton’ left. The MixedRealityToolkitConfigurationProfile is a so called ‘scriptable object’ that can hold the configuration of the whole MRTK. But the MixedRealityToolkitConfigurationProfile  is more or less a hat stand for all kinds of other partial configurations, all of which will end up in a profile as well.

Concluding words

We have taken the first (very small) baby steps into configuring the MRTK-vNext. We actually did not write any code and therefore, unlike most of my other posts, this does not come with a sample project. The next one will, though.