Scanning physical objects with an Xbox One Kinect to use as Holograms in HoloLens

Intro

There are several demos out there that show obviously scanned models of people or physical objects used in HoloLens applications. I think it was fellow MVP and Dresdener 3D genius René Schulte who first used his bust in a demo or at least went public with it. Unfortunately I have not been able to find very much about the actual process used to do the scanning and get to this result, so I have been trying to kludge together a procedure to make a full color 3d scan of a physical object – myself – and show it in a HoloLens.

Shopping list

For this you will need the following hardware:

• A PC with a USB3 port. A Surface 4 Pro i7 has been successfully used for this goal
• A Kinect sensor for Xbox one, aka Kinect 2
• A Kinect Adapter for Xbox One S and Windows PC

And the following software:

• The 3D Scan app (see below)
•  CloudCompare Stereo
• This special shader that I nicked from this thread on the Unity forms

Setting up the hardware

The Kinect adapter, when you remove it from the box, seems to be quite an intricate contraption of two boxes and three wires, one of them permanently connected to one of the boxes. That box – I’ll call it box #1 - is about the size of a package of cigarettes, and is the power supply. There is a wire with a mains plug that needs to be connected to this box. The other wire – that is permanently attached to box # 1– has a round plug on the other side.That plug goes into box #2 (the one without fixed wire). It’s about the size of an overly thick Mars bar. Now we have one wire unaccounted for – one with a weird squarish plug on one side, and a USB-3 plug on the other side. The squarish part also goes into box #2, next to the round plug coming from the power supply. Now there’s only one hole unaccounted for - on the other side box #1 is another hole – that’s where you need to connect the actual Kinect 2.

Plug in the mains connected to box #1, and connect the USB plug coming from box #2 to your computer’s USB3.0 port. This will start the installation of a couple of drivers, and I seem to recall it also automatically installed the 3D Scan app that you also can find in the store here.

Downloading additional software

If the 3D scan app did not install automatically, you will need to install it from the Windows Store. Also, download and install CloudCompare. Finally download the shader. That’s a zip file – we will need that later in the process. So store that in your downloads folder and unblock it.

The actual scanning

The 3D Scanning app is actually rather straightforward. You can set a few options, and I noticed that you have to fiddle a lot with the settings to get the effect you want. A higher scan resolution give a lower depth, for instance. Also, Kinect sometimes just loses track of the object it tries to track. What you do have to make sure is that there is plenty of light and prevent shadows, because although the actual object tracking is done via Kinect’s magic sensors, the overlaying of color is using the camera and that just needs light. And move slowly. Very slowly.

I scanned myself sitting on a rotating chair and a 120 seconds settings, rotating very very slowly. For this I used the “Kinect Sensor-stationary” setting. When it’s done, it will open the 3D builder app. That usually complains about invalid geometries that need to be fixed. Click the popup, prepare to wait for quite a while, and then finally the 3D object will be ready for the next step

Converting to a format usable for Unity

The annoying thing is that although the 3D builder can save the scanned object as a WaveFront Object (obj) file – that is readable by Unity -  it will strip any color from it. So we will need to take an in-between step. To that effect, don’t save the file in OBJ but in PLY format. Then start CloudCompare, and open the PLY file in that you just created. Hit file/save as and save the file now as an FBX file. That gives you a number of options – I usually just use FBX binary.

Importing in Unity and showing the colors

When you import this into Unity and drag the scanned object on the canvas, you will soon notice a few things about me

• I’m rather large – like statue-of-Roman-emperor-with-overly-inflated-ego large
• I seem to hang at a random place at a random angle in the sky, even if position and rotation are 0,0,0
• I am … rather pale.

Getting to let me look less like a roman statue uses requires the special shader. Using my standard folder structure, I added a folder “Shaders” and copied the contents of the UnityVC zip file into it. Net result:

 

Then in your assets folder, go to the folder where you have imported the scanned model into. In my case that’s App/Models:

In that you will find a Material ColorMaterial

Select that Material, go into it’s properties in the Inspector over on the right and select either “Standard (Vertex Color)” or “Standard Specular (Vertex Color)”

And boom. There I am, in full color color and glory looking like a zombie.

Now to make me appear in front of the HoloLens view and not seem like some giant ancient, balding and grey god of… whatever descending from Heaven, I used the following settings on the scanned Hologram:

And then you get a more or less life-sized floating ghost/zombie/Borg me-like appearance

I had the dubious pleasure of walking around myself, noticing the back part was missing.

Lessons learned

• First of all – already mentioned in passing, make sure there is enough light, and prevent shadows. This is harder than it sounds. You will notice scanned Holograms tend to look rather pale when created with insufficient light.
• The handheld setting is way harder to use than the stationary setting. Consider placing objects on a rotary platform rather than moving around it with the Kinect
• Move or rotate the object you want to scan slowly (or move the Kinect slowly)
• You will have to fiddle a lot with settings before you get the result you want
• Larger objects (like humans) are way easier to scan than small objects
• Make anything you don’t want on the scan as black as and non-reflective as possible

Concluding remarks

Unity will complain regularly about an error in the shader in the editor, but it still seems to work fine. Having no clue about shaders and how to write them yet, I tend to ignore it. The resulting project, although containing no code written by me at all, can be found here.