This blog attempts to be a collection of how-to examples in the Microsoft software stack - things that may take forever to find out, especially for the beginner. I see it as my way to return something to the Microsoft community in exchange for what I learned from it.
This is a little tidbit I use in a lot of apps, for instance in the CubeBouncer and for floating info screens in my other apps. Basically I want to know a position dead ahead of the user, at a certain maximum distance, or closer by if the user is looking at a physical object that is closer by. Think of an invisible ray coming out of the HoloLens – I want to have a point where it strikes a physical object – and if there is no such thing within a certain maximum distance, I want a point along that ray at a maximum distance.
I made a little helper class for that, and it’s called LookingDirectionHelpers
using HoloToolkit.Unity.InputModule;
using HoloToolkit.Unity.SpatialMapping;
using UnityEngine;
namespace HoloToolkitExtensions.Utilities
{
public static class LookingDirectionHelpers
{
public static Vector3 GetPostionInLookingDirection(float maxDistance = 2,
BaseRayStabilizer stabilizer = null )
{
RaycastHit hitInfo;
var headReady = stabilizer != null
? stabilizer.StableRay
: new Ray(Camera.main.transform.position, Camera.main.transform.forward);
if (SpatialMappingManager.Instance != null &&
Physics.Raycast(headReady, out hitInfo, maxDistance, SpatialMappingManager.Instance.LayerMask))
{
return hitInfo.point;
}
return CalculatePositionDeadAhead(maxDistance);
}
public static Vector3 CalculatePositionDeadAhead(float distance = 2,
BaseRayStabilizer stabilizer = null)
{
return (stabilizer != null
? stabilizer.StableRay.origin + stabilizer.StableRay.direction
: Camera.main.transform.position + Camera.main.transform.forward)
.normalized * distance;
}
}
}
Although it’s a small thing, it actually does quite a lot. If you call GetPostionInLookingDirection
It first tries to determine a so-called head ray. This can either come from the stabilizer or be directly calculated from the location and angle of the camera (that is, your head). I would recommend feeding it a stabilizer, as that makes for getting a much more reliable location.
If it actually finds a hit, it returns that point
If it does not, it uses CalculatePositionDeadAhead to, well, calculate a position dead ahead. It takes once again either the stabilizer head ray or calculates one from the camera, then normalizes it (i.e. makes it’s length 1) and then multiplies it with the desired distance. This effectively gives a point distance meters right before the user’s eyes.
This script requires the presence of a SpatialMappingManager prefab, for that’s the only way to find out which Unity layer contains the spatial mesh. If you want to call this script using a stabilizer (and I think you should), the InputManager should be present as well, as that will create a GazeManager singleton, which contains the stabilizer. So you can call this helper class like this:
var pos = LookingDirectionHelpers.GetPostionInLookingDirection(2.0f, GazeManager.Instance.Stabilizer);
And that will return you a point at either a Spatial Mesh, or at 2 meters distance from the user.
Although and the code involved is not very voluminous, I have gone all the way and made and – albeit pretty lame – minimalist demo for it that you can download here. This is based upon a HoloToolkit based setup as described here. All it does it, when you move around, is print the location onto the debug log. So this will only work in the Unity editor, on the HoloLens emulator, or on an app running in debug mode on an actual HoloLens – but you won’t see much happening on the device itself, just in debug windows on your screen.
But it shows how it works and can be used, and that is the point. Later in this blog we will see a better application for this.
Those who have seen my HoloLens apps (most notably Walk the World) have noticed I tend to use floating "info screens", especially for help screens. My apps are mostly voice command driven as I don't like to have floating controls that are in view all of the time. They stress the fact that you are in an virtual environment, and that degrades the actual immersive experience, IMHO. So I go as much for gestures and voice as possible.
But where there are no visual clues for functionality, there's also lack of discoverability. So I tend to include a voice command "help" or "show help" that brings up a simple floating screen that shows what the app can do.
A few important things that you might not see right away:
The screen follows your gaze
The screen tries to move away from you to be readable, but will stop moving if it get's pushed against an obstacle. So it won't disappear into another hologram or a physical object, like the wall or a floor. Or at least it tries to. I must admit it does not always works perfectly.
Notice that at first it will appears like 1 meter before you and move into view, next time it will appears where you last left it and then move into view.
In a two-part post I will describe how I have created such a screen.
The first part will handle building the actual visual structure in Unity (and one little behaviour)
The second part describes the code for all other Unity behaviours.
I am going to assume you know a bit about Unity but not too much, so there's going to be lot of images.
Setting up a base project
That's easy, as I described that in my previous blog post. Just make sure you download a HoloToolkit from June 16, 2017, or later. This includes this pull request by yours truly that we will need in this app. And while you are importing stuff, also import LeanTween from the Unity Asset Store (hit CTRL-9 to open it immediately without having to hunt it down in the menu). When doing so, make sure you deselect everything but the Plugisn checkbox.
The basic setup of an info screen
My info screens basically exist out of three simple components:
Inside the HologramCollection that we inherited from my previous post we will first make an empty game object that I called "HelpHolder" as this will be a help screen, but you can call it anything you like. To make designing a little easier, set it's Z position to 1, else it will be sitting over the camera, which is always on 0,0,0 in a HoloLens app. That kind of obscures the view.
Inside that HelpHolder we first make a common Plane. This gives the standard, way too big 10x10m horizontal square. Change it's rotation to 270 and change X and Z scale to 0.07m (changing the Y scale makes no sense as a Plane essentially has no Y dimension).
Double click the Plane in the HelpHolder - this will make your scene zoom in. Now use that hand button and left top the scene screen and the CTRL key to rotate around to you get to see the white side of the Plane (the other side is translucent). Notice the HoloLens cursor ;)
Now a white background for a text doesn't look good for me, I find it too bright. So we are going to make a material to make it look better.
To keep things organized, we first create an "App" folder in "Assets", and within that a "Materials" folder. In that Materials folder we create a HelpScreenMaterial Material
Setting some color and reflection
Now over on the right side:
Set "Shader" to "HoloToolkit/StandardFast"
Set "Rendering Mode" to "Transparent"
Set "Color" to a background color you fancy, I took a kind of sky blue (#0080FFFF)
Move the "Smoothness" slider all the way to the left - we don't want any reflections or stuff from this 'screen'
Now you only have to drag the material on your plane and it will turn blueish.
Rather standard Unity stuff this, but I thought it nice to point it out for beginners.
Changing the collider
A Collider is a component that determines how a game object collides with other objects. When you use primitive objects, those are of the same type as actual shape you use, although the names are not always the same. So a Sphere typically has a Sphere Collider, but a Cube has a Box Collider. There is no Plane collider, as a Plane is a generic Mesh - so it uses a Mesh Collider. And here we run into an issue, because a Plane typically has one side and it looks the Mesh Collider has that as well - and if not, it does not prevent the help window from getting pushed though the floor or a wall. As I found out making this demo :D.
So select the Plane, hit the Add Component button at the bottom and add a Box Collider.
Then
Unselect the checkmark next to "Mesh Collider". This will disable the old Mesh Collider. A game object may have only one active Collider so we want to get rid of this. You can also delete it if you want using the dropdown that appears if you click the gear icon all the way to the right.
Put "0.02" in the Y filed in the "Size" Section. This will make the Collider as big as the blue plane, and 2 cm thick.
What may seem confusing is that the Collider claims it's 10 x10 meters in size. That is true, but it is also scaled to 0.07 in X direction, and 0.050535134 in Z direction. If you remember the default size of a Plane is 10x10, this makes the screen about 70 cm wide and 50 cm height, which looks like the size you saw on the video. A Plane has no thickness, so if the scale of Y is set to 1, the colliders width will be the actual size in the Y field.
If you look at the screen on edge, you can see the green indicator lines showing the outline of the Collider:
Adding text
Find the 3DTextPrefab and drag it onto the HelpHolder:
It should end up under the Plane. Zoom a little in on the plane to see the text clearly.
Now change the text into the help text you want (I took some Lorum Ipsum) and change some of the text settings:
Change Y to 0.133 (this will move the text towards the top of the 'screen', making room for the button later on).
Change Z to -0.02 (this will move the text to 2cm before the 'screen', this will prevent rendering issues later on
Change "Alignment" to left
I wish to stress the Y value hugely depends on the size of your text, the the font size, and the size of your 'help screen'. To get it right, it requires some fiddling around (as we will see later on).
Building the button - part 1
Right click the HelpHolder and add a Sphere. This will - like almost everything is initially - way too large. To change it's scale to 0.08 in all three dimensions. Then change it's Z-value to -0.045 (this will put the button in front of the 'screen' and also change the Z value to -0.01
This results in the following and now you can see where the fiddling starts, because that screen is too big for the text and the button it not quite where we want it
Some in-between fiddling around
With these two buttons you can very easily move (left) or resize (right) objects. Select the Plane, then select the desired function.
By dragging the blue block in the left image you can change the screen size in vertical direction, the red block will do so in horizontal. With the yellow arrow (right image) you can move the plane upward until it is where you like it.
In my Inspector pane on the right it said this when I was done:
But... now all of or stuff is quite off center as far as the HelpHolder, the root object, is concerned. It's center point is pretty low on our screen, which means the screen it too high.
This can be fixed by selecting all three denizens of HelpHolder (using the CTRL button), select the Move button again, grab the yellow arrow and move the whole combination downward until the read arrow is more or less on the horizon.
It does not have to be a super precise hit, as long as it's not so much off center as it first was.
Building the button - part 2
A white sphere is not a button, so add some that makes sure you can click it. I think a real designer might have to say something about it, but I have found that a red texture with a large OK text on it works - in the sense that I never had to explain to anyone that it's something you can air tap and that will act like something of a button. So I created this awesome :D picture, created a "Texture" folder under app and put it there
It's odd shape will become clear soon.
First, create a "DoneMaterial" in Materials. Then:
Set "Shader" once again to "HoloToolkit/StandardFast"
Set "Rending Mode" to "Fade" (not Transparent)
Select the "Textures" folder in the App/Assets folder, and drag the "Done" texture on top of the little square left of the "Albedo" text
Change the X value of "Emissions/Tiling" to 3. This will morph three images on the Sphere, repeating them horizontally.
If you have done everything correct, you will see the material looks like above.
Now drag the DoneMaterial from the Materials folder onto the Sphere in the Hierarchy
And the button on the screen looks already familiar :). I left the default Shader settings as it's a bit smooth, so it looks like it reflects a little light adding to it's 3D appearance.
Turn off shadows
This is a thing trick I learned from Dennis Vroegop, long time Microsoft MVP, my unofficial MVP 'mentor' who learned me to deal with suddenly being a Microsoft MVP too way back in 201,1 and long time expert on Natural User Interface. The trick is this: unless your app really really uses them for some reasons, turn off "receive shadows" and "cast shadows" for the renderers of all three objects. As you can see the actual real light sources in your room through HoloLens, shadows will appear on the wrong places anyway and only give cause for confusion at best - at worse they will 'break the spell'. As a bonus, the Unity engine won't need to calculate the shadows so you will save some performance as well.
So select all three objects (use the CTRL key for that like in any other normal Windows program) and turn this off in one go:
A little code for dessert
This is part of my HoloToolkitExtensions libary, that I one day will publish in full, but, well, time. It is called HorizontalSpinner, and basically is the grandchild of the SimpleRotator that briefly passed by (without explaination) in my post about a generic toggle component for HoloLens. It uses LeanTween and looks like this:
using UnityEngine;
namespace HoloToolkitExtensions.Animation
{
public class HorizontalSpinner: MonoBehaviour
{
public float SpinTime = 2.5f;
public bool AbsoluteAxis = false;
private void Start()
{
if (AbsoluteAxis)
{
LeanTween.rotateAround(gameObject, Vector3.up, 360 / 3.0f,
SpinTime / 3.0f).setLoopClamp();
}
else
{
var rotation = Quaternion.AngleAxis(360f / 3.0f, Vector3.up).eulerAngles;
LeanTween.rotateLocal(gameObject, rotation, SpinTime / 3.0f).setLoopClamp();
}
}
private void OnEnable()
{
LeanTween.resume(gameObject);
}
private void OnDisable()
{
LeanTween.pause(gameObject);
}
}
}
Default this spins an object around for 120° in local space. Since our the tiling of our 'button' is 3 in X direction, this will look like the button spins around in 2.5 seconds, while in reality, it will jump back to it's original position. But as you can see if you press the Unity Play button, it looks like the button is rotating around endlessly. LeanTween is doing most of the work: it rotates the game object around and the setLoopClamp at the end makes it a loop. No need for coroutines, lerps, and heaven knows what.
I am using local space because I want the button to perpendicular to the user's view at any given moment, but since the screen moves around with the user's gaze and gaze angle, it needs to be in the local space of the HelpHolder.
The HorizontalSpinner is in HoloToolkitExtensions/Scripts/Animations. Simply drag it on top of the Sphere, then change values as you like, although I would recommend not changing the Absolute Axis setting.
Conclusion
We have built the visual parts of a help screen but with very little functionality or interaction. It's actually not hard to do, if you know what you are doing. I hope I have helped you getting a bit more feeling for that.
In the next installment, which will hardly contain images, we will see WAY more code.
My apologies for the longest blogging hiatus in the history of this blog, I was very busy with things both HoloLens and not-HoloLens related. Chief culprit though was my HoloLens app "Walk the World", whose development completely captured and absorbed me. Passion for tech sometimes does that with me ;). Then came some work-related issues, but now I finally am up to speed again.
Intro
A consequence of the hiatus is a lot of change (the only thing that changes faster than your average JavaScript-framework-of-the-month is the HoloToolkit) so any HoloLens how-to blog post I'd start with I would have to start with 'how to set up a project'. So I make a little separate post of that subject, thus I can refer to it often. Until the HoloToolkit changes again ;)
File/new project and Getting what you need
First, start a new Unity Project. I have called mine June2017HoloLensSetup.
Then, you open a browser and go to Rafael Rivera's HoloToolkit download site and you click the top HoloToolkit-somenumber-unitypackage. At the time of this writing the top looks like this:
.
Downlaod and double click the resulting .unitypackage file - Unity will automatically import it into your project. This will give you a window with a lot of checkboxes:
This looks a bit intimidating, but just collapse the HoloToolkit node in the list, that will reveal another node "HoloToolkit-test" that you don't need, and you can de-select it
And then hit the Import-button. Then very soon, Unity will show a popup:
Select "Force Text Serialization". This is very important if you want to be able to easily upgrade the HoloToolkit and be able to check it in into your source control of choice.
The HoloToolkit will now be imported. Well, that was not so hard, wasn't it? This is time to get some coffee as Unity will churn a little on this.
Setting up the camera
HoloLens apps need some settings applied to the camera, and the HoloToolkit folks have taken care of that. So first of all, you delete the Main Camera from your Hierarchy - just select it and delete it.
Then, find the HoloLensCamera by entering "Camera" into the search box indicated with a red ellipse, and drag the new Camera into your hierarchy
This is also a good moment to save your scene. Hit CTRL+S (or click File/Save Scenes), enter an name for your scene (I usually take "Main") and you are done with the this part.
Setting a whole lot of properties - the easy way
Now you may not have noticed it, but after importing the HoloToolkit you got an extra extra menu options.
This was added because you have to set a whole lot of properties necessary for or beneficial to a HoloLens projects - scattered all over Unity, and good luck to you determining what you forgot if something doesn't work and where you have to correct that. Those were the bad old days (like a year ago ;) ). Now it's easy.
Clicking "Apply HoloLens Scene Settings" gives you this popup. Make sure everything is selected, then hit Apply. Always hit apply!
Next up is "Apply HoloLens Project Settings"
Same procedure: make sure everything is selected, hit apply. Unity asks if you want to reload the project - click Yes and wait for the project to reload. Finally, "Apply HoloLens Capability Settings"
This is what I always select - my HoloLens apps all tend to use voice commands, all use Spatial Perception (that is, after all, HoloLens' biggest USP) and most of them download some data from the internet. I never use the camera. Make your selections according to your necessary capabilities and hit apply.
Sound should be Spatial!
For some reason this is not part of these nice HoloToolkit setup screens - but if you use spatial sound in your apps (and you should!) there is an extra step. You select "Edit/Project Settings/Audio" like this
"Project settings is almost all the way down in a long Edit menu. Now on the right side of the editor, all the way on the top, a new menu has appeared:
The Spatializer Plugin is default set to "none", which will give you the default Unity stereo sound - nice, but that's sub-optimal usage of a $3K (or $5K) HoloLens - we want real spatial sound. So set it to MS HRTF Spatializer. This is, by the way, a very important and often overlooked part of immersive experiences. Human beings are 3D aware not only by vision, but also by sound. Think for instance of how you move around in a city, you hear cars coming to you even if they are outside of your vision (and that has more than once saved my life). In your house, there is the hum of the air conditioner, the bubbling of the fish tank filter, the voice of your partner - you don't have to see any one of those, but still you can tell where they are. It all contributes to your 3D awareness - and so does spatial sound contribute heavily to an immersive holographic experience.
So far for this soapbox moment ;).
Some organizing
It tend to create two empty gameobjects - HologramCollection and Managers. The HologramCollection will actually hold Holograms that are part of the application, the Managers object is like a container for global functions, object and standard stuff.
Some standard stuff to always add
What (I think) you will always need is an InputManager (the thing that acts as a kind of hub for all input, be it gestures, speech or whatever), SpatialMapping (duh) and some kind of cursor. I usually take the object that is just called "Cursor", there's also "DefaultCursor" and "CursorWithFeedback".
The easiest way to find them is once again entering their name in the search box. Now this can sometimes give cause for some confusion. For instance, if you enter "Cursor" in the search box you will get a lot of hits.
Only choose the so-called prefabs, the things that get the simple sky blue cube next to it. The same goes, by the way, for InputManager and SpatialMapping. Always take the prefabs.
When you are done adding SpatialMapping, find the property "Surface Material" and change it to "Occlusion" by clicking the little circle button behind it. Then, select it from the "Select Material" window that pops up
Finally some stubbornness
The current settings make that stuff is clipped when you are within 85cm of a Hologram. There are good technical and performance reasons for it, yet I am a stubborn *** ** * ***** so I tend to make this 15 cm.
Simply because I think it's cool to get that close to a Hologram.
A final word on source control
Assuming you use Git, making a good .gitignore is pretty hard. Especially the HoloToolkit contains stuff you would expect not to reside in a source repository, there's all kinds of dlls in it and folders called x86 and x64, typically folders that are part of a build path and should usually be ignored. I've been bitten so many times by just that little piece of the HoloToolkit, especially when upgrading. Therefore a hard-won piece of advice: if you value your sanity, put a .gitignore file in Assets\HoloToolkit containing just these lines:
!*.*
!*
That should always add everything, I think.
You can download the (empty) project for reference, with my .gitignore files (there are no less than three) here.
