18 July 2018

An advanced gaze-following behaviour to place objects on top or in front of obstructions (and scale if necessary)

Intro

In my previous post I described some complicated calculations using a BoxCastAll to determine how to place a random object on top or in front of some obstruction in the looking direction of the user, be it another object or the Spatial Mesh. Because the post was long enough as it was, I described the calculations separately. They are in an extra method "GetObjectBeforeObstruction" in my HoloToolkitExtension LookingDirectionHelpers, and I wrote a very simple Unity behaviour to show how it could be used. But that behaviour simply polls every so many seconds (2 is default) where the user looks, then calls GetObjectBeforeObstruction and moves the object there. This gives a kind of nervous result. I promised a more full fledged behaviour, and here it is: AdvancedKeepInViewController. It’s basically sitting in a project that looks remarkably like the demo project in the previous post: the same ‘scenery’, only there’s a 4th element you can toggle using the T button or by saying “Toggle”.

image

Features

The behaviour

  • Only moves the object if the head is rotated more than a certain number of degrees per second, or the user moves a certain number of meters per second. It use the CameraMovementTracker from Walk the World that I described in an earlier post.
  • Optionally fine tunes the location where the object is placed after doing an initial placement (effectively doing a BoxCastAll twice per movement)
  • Optionally scales the object to have a more or less constant viewing size. This is indented for 'billboards' like objects - i.e. floating screens.
  • Optionally makes an object appear right in front of the user if it's enabled, in stead of moving it in view the first time from the last place where it was before it got disabled.
  • Optionally makes the object disappear when the user is moving a certain number of metes per second, to prevent objects from blocking the view or providing distractions. This is especially useful when running an app in a HoloLens while you are on a factory floor where you really want to see things like handrails, electricity cables, or holes in the floor (possibly with a smelter in it).

The code is not that complicated, but I thought it best to explain it step by step. I skip the part where all the editor-settable properties are listed - you can find them in AdvancedKeepInViewController's source in the demo project. I have added explanatory tooltip description to almost all of them.

Starting up

The start is pretty simple:

void Start()
{
    _objectMaterial = GetComponentInChildren<Renderer>().material;
    _initialTransparency = _objectMaterial.color.a;
}

void OnEnable()
{
    _startTime = Time.time + _delay;
    DoInitialAppearance();
    _isJustEnabled = true;
}

private void DoInitialAppearance()
{
    if (!AppearInView)
    {
        return;
    }

    _lastMoveToLocation = GetNewPosition();
    transform.position = _lastMoveToLocation;
}

We get the material of the first Render's material we can find and it's initial transparency, as we need be able to revert to that later. Then we need to check if the user has selected to object to initially appear in front, and if so, do the initial appearance. At the end you see GetNewPosition being called, that's a simple wrapper around LookingDirectionHelpers.GetObjectBeforeObstruction. It tries to project the object to hit an obstruction at a certain max direction; if there is no obstruction in that range, just give a point at the maximum distance. Since it's called multiple times and I am lazy, I made a little method of it

private Vector3 GetNewPosition()
{
    var newPos = LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
        DistanceBeforeObstruction, LayerMask, _stabilizer);
    if (Vector3.Distance(newPos, CameraCache.Main.transform.position) < MinDistance)
    {
        newPos = LookingDirectionHelpers.CalculatePositionDeadAhead(MinDistance);
    }
    return newPos;
}

Moving around

The main thing is, of course, driven by the Update loop. The Update method therefore the heart of the matter:

void Update()
{
    if (_startTime > Time.time)
        return;
    if (_originalScale == null)
    {
        _originalScale = transform.localScale;
    }

    if (!CheckHideWhenMoving())
    {
        return;
    }

    if (CameraMovementTracker.Instance.Distance > DistanceMoveTrigger ||
        CameraMovementTracker.Instance.RotationDelta > DeltaRotationTrigger || 
        _isJustEnabled)
    {
        _isJustEnabled = false;
        MoveIntoView();
    }
#if UNITY_EDITOR
    if (_showDebugBoxcastLines)
    {
        LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
            DistanceBeforeObstruction, LayerMask, _stabilizer, true);
    }
#endif
}

After the startup timeout (0.1 second) has been expired, we first gather the original scale of the object (needed if we actually scale). If the user is moving fast enough, hide the object and stop doing anything. Else, use the CameraMovementTracker that I wrote about two posts ago to determine if the user has moved or rotated enough to warrant a new location for the object (and the first time the code gets here, repositioning should happen anyway). And then it simply shows the Box Cast debug lines, that I already extensively showed off in my previous post.

So the actual moving around is done by these two methods (using once again good old LeanTween), and the second one is pretty funky indeed:

private void MoveIntoView()
{
    if (_isMoving)
    {
        return;
    }

    _isMoving = true;
    var newPos = GetNewPosition();
    MoveAndScale(newPos);
}

private void MoveAndScale(Vector3 newPos, bool isFinalAdjustment = false)
{
    LeanTween.move(gameObject, newPos, MoveTime).setEaseInOutSine().setOnComplete(() =>
    {
        if (!isFinalAdjustment && EnableFineTuning)
        {
            newPos = GetNewPosition();
            MoveAndScale(newPos, true);
        }
        else
        {
            _isMoving = false;
            DoScaleByDistance();
        }
    });
    _lastMoveToLocation = newPos;
}

So the move MoveIntoView method first check if a move action is not already initiated. Then it gets a new position using - duh - GetNewPosition again, and calls MoveAndScale. MoveAndScale moves the object to it's new position, then calls itself an extra time. The idea behind this is a follows: the actual bounding box of the object might have changed between the original cast in MoveIntoView and the eventual positioning if the object you move is locked to be looking at the Camera while it moved, using something like the Mixed Reality Toolkit's BillBoard or (as in my sample) my very simple LookAtCamera behaviour . So a second 'finetuning' call is done, using the 'isFinalAdjustment' parameter. And if we are done moving, optionally we do some scaling. And this looks like this:

You might also notice the cubes appear from the camera’s origin, the floating screen appears initially in the right place. This is another option you can select.

Scale it up. Or down

For an object like a floating screen with text, you might want to ensure readability. So if your text is projected too far away, it might become unreadable. If it is projected too close, the text might become huge, the user can only see a small portion of it - and effectively it's unreadable too. Hence this little helper method

private void DoScaleByDistance()
{
    if (!ScaleByDistance || _originalScale == null || _isScaling)
    {
        return;
    }
    _isScaling = true;
    var distance = Vector3.Distance(_stabilizer ? _stabilizer.StablePosition : 
        CameraCache.Main.transform.position,
        _lastMoveToLocation);
    var newScale = _originalScale.Value * distance / MaxDistance;
    LeanTween.scale(gameObject, newScale, MoveTime).setOnComplete(() => _isScaling = false);
}

I think it only makes sense for 'text screens', not for 'natural' objects, therefore it's an option you can turn off in the editor. But if you do turn it on, it determines the scale by multiplying the original scale by the distance divided by the MaxDistance, assuming that is the distance you want to see you object on it's original scale as defined in the editor. Be aware that the autoscaling can make the screen appear inside other objects again, so use wisely and with caution.

Fading away when necessary

This method should return false whenever the object is faded out, or fading in or out – that way, MoveIntoView does not get called by Update.

private bool CheckHideWhenMoving()
{
    if (!HideWhenMoving || _isFading)
    {
        return true;
    }
    if (CameraMovementTracker.Instance.Speed > HideSpeed &&
        !_isHidden)
    {
        _isHidden = true;
        StartCoroutine(SetFading());
        LeanTween.alpha(gameObject, 0, FadeTime);
    }
    else if (CameraMovementTracker.Instance.Speed <= HideSpeed && _isHidden)
    {
        _isHidden = false;
        StartCoroutine(SetFading());
        LeanTween.alpha(gameObject, _initialTransparency, FadeTime);
        MoveIntoView();
    }

    return !_isHidden;
}

private IEnumerator SetFading()
{
    _isFading = true;
    yield return new WaitForSeconds(FadeTime + 0.1f);
    _isFading = false;
}

Basically this method says: if this thing should be hidden at high speed and is not already fading in or out:

  • If the user’s speed is higher than the threshold value and the object is visible, hide it
  • If the user’s speed is lower than the threshold value and the object is invisible, show it.

The way of hiding and showing is once againdone with LeanTween, but I found that using the .setOnComplete was a bit unreliable for detecting the fading in or out came to an end. So I simply use a coroutine that sets the blocking _isFading, waits a wee bit longer than the FadeTime, and then clears _isFading again. That way, no multiple fades can start or stop.

The tiny matter of transparency

The HideWhenMoving feature has a dependency – for it to work, the material needs to support transparency. That is to say – it’s rendering mode needs to be set to transparent (or at least not opaque). As you move around quickly, the semi transparent box and the double boxes will fade out and in nicely:

But if you move around and the floating screen wants to fade, you will see only the text fade out – the window outline stays visible. This has a simple reason: the material’s rendering mode is set is opaque, not transparent

image

The background of the screen with the button fades out nicely though because it uses a different material – actually a copy, but with only the rendering mode set to transparent:

image

But if you look really carefully, you will see not the entire screen fades out. Part of the button seems to remain visible. The culprit is the button’s backplate:

image

Now it’s up to you – you can change the opacity this material, and then it will be fixed for all buttons. The problem is that this material is part of the Mixed Reality Toolkit. So if you update that, it will most likely be overwritten. And then you will have to keep track of changes like this. Or you can manually change every backplate of every button, or do that once and make your own prefab button. There are multiple ways to Rome in this case.

Nice all those Unity demos…

But how does it look in real life? Well, like in this video. First it shows you the large semi-transparent cube actually disappears if you move quickly enough, then it shows the moving and scaling of the "Hello World" screen, but it also shows that when you move quickly enough, it will try to fade, but only the text will fade. The two cubes show nothing special other than that they appear more or less on the spatial mesh, and the "Screen with button" shows shrinking and growing as well, and it will fade completely - but the back plate. I have told you how to fix that.

Some tidbits

If you try to run the project in an HoloLens or Immersive Headset and wonder where the hell the cubes, capsule and other 'scenery' is that is clearly visible in the Unity editor - the are explicitly turned off by something called the HideInRuntime behaviour that sits in the "Scenery" game object, where all the 'scenery'  resides it. This is because in a HoloLens, you already have real obstructions. If you want to try this in an Immersive headset, please remove or disable this feature otherwise you will be in a void with almost nothing to test the behaviour at all.

Conclusion

Unlike the previous one, this behaviour makes full use of the possibilities GetObjectBeforeObstruction offers. I think there’s still room for improvement here and tweaks. For instance, if you want to use this behaviour to move and place stuff, simply disable the the behaviour when it’s done. But this behaviour as it is, is very usable and in fact I use it myself in various apps now.

13 July 2018

Calculate Mixed Reality object locations on top or in front of obstructions using BoxCastAll

Intro

It was quite difficult to name this article. Usually I try to find a title that more or less describes a good search term that I used myself when I was looking for ‘something like this’, but I could not really find what I was looking for. What I have here is code that calculates locations for objects to be in front or on top of other objects and/or the Spatial Mesh. It does so using a BoxCastAll, something I have tried to use before, but not very successfully.  I have tried using Rigidbody.SweepTest and although it works for some scenarios, it did not work for all. My floating info screens ended up half a mountain anyway (in Walk the World), or the airport could not move over the ground because of some tiny obstacle blocking it (in AMS HoloATC). So I tried a new approach.

This is part one of a two-post blog post. Explaining how the BoxCast works and what extra tricks and calculations were necessary to get it to work properly proved to need quite a long text, so I will leave the behaviour actually using this code in good way for the next post.

BoxCast magic

So what is a BoxCast, actually? It’s comparable to a normal RayCast. But where a RayCast gives you the intersection of a line and an obstruction, a BoxCast does that – suprise - with a box. You essentially throw a box from a point along a vector until it hits something – or some things, as a BoxCastAll potentially returns more than one hit. If you take the one that is closest to your camera (a hit has a convenient “distance” property) you potentially have a place where you can place the object.

… except that it does not take into account the following things:

  • An object’s (center) position and the center of it’s bounding box are not always the same; this will make the BoxCast not always happen at the place you think it does
  • The vector from the camera to the hit may or may not be parallel to the direction of the BoxCast; therefore, we need to project the vector from the camera to the hit on the vector of the BoxCast.
  • The BoxCast hit detection happens at the edge of the casted box, and an objects position is determined by it’s center – so, we need so we need to move back a little towards the camera, or else about half of our object – determined by it’s actual orientation - will end up inside the obstruction.

My code takes all of that into account. It was quite hard won knowledge before I uncovered all the lovely pitfalls.

First a new utility method

For a BoxCast to work, you need a box. You typically accomplish that by getting the bounds of all the Renderers in the object you want to cast and combine those to one big bounding box. I hate typing or copying code more than once, so I create this little extension method to GameObject

public static class GameObjectExtensions
{
    public static Bounds GetEncapsulatingBounds(this GameObject obj)
    {
        Bounds totalBounds = new Bounds();

        foreach (var renderer in obj.GetComponentsInChildren<Renderer>())
        {
            if (totalBounds.size.magnitude == 0f)
            {
                totalBounds = renderer.bounds;
            }
            else
            {
                totalBounds.Encapsulate(renderer.bounds);
            }
        }

        return totalBounds;
    }
}

BoxCast Magic

In LookingDirectionHelpers, a static class containing utilities to calculate directions and places the direction the user is looking (duh) at I have created a method that does the BoxCast magic. It does quite a lot, and I am going through it step by step. It starts like this:

public static Vector3 GetObjectBeforeObstruction(GameObject obj, float maxDistance = 2,
    float distanceFromObstruction = 0.02f, int layerMask = Physics.DefaultRaycastLayers,
    BaseRayStabilizer stabilizer = null, bool showDebugLines = false)
{
    var totalBounds = obj.GetEncapsulatingBounds();

    var headRay = stabilizer != null
        ? stabilizer.StableRay
        : new Ray(CameraCache.Main.transform.position, CameraCache.Main.transform.forward);

     var hits = Physics.BoxCastAll(GetCameraPosition(stabilizer),
                                  totalBounds.extents, headRay.direction,
                                  Quaternion.identity, maxDistance, layerMask)
                                  .Where(h => !h.transform.IsChildOf(obj.transform)).ToList();

As you can see, the method accepts quite some parameters, most of them optional:

  • obj – the actual object to cast and place against or on top of the obstruction
  • maxDistance – the maximum distance to place the object from the camera (if it does not hit another object first)
  • distanceFromObstruction – the distance to keep between the object and the obstruction
  • layerMask – what layers should we ‘hit’ when we are looking for obstructions (default is everything)
  • stabilizer – used to get a more stable location and viewpoint source than the camera itself
  • showDebugLines – use some awesome help classes I nicked from the Unity Forums from “HiddenMonk” to show how the BoxCast is performed. Without these, I sure as hell would not have been able to identify all issues that I had to address.

Well then – first we get the total encapsulating bounds, then we check if we can either use the Stabilizer that we need to use the camera to define a ray in the direction we want to cast. The we calculate a point dead ahead of the camera – this is the.

And then we do the actual BoxCast, or actually a BoxCastAll. The cast is done:

  • From the Camera position
  • Using the total extents of the object
  • In the direction of the viewing ray (so a line from your head to where the gaze cursor is)
  • using no rotation (we used the Render's bounds, that already takes any rotation into account)
  • over a maximum distance
  • against the layers described by the layer mask (default is all)

Notice the Where clause at the end. BoxCasts hit everything, including child objects of the cast object itself, as it may be in the path of it’s own cast. So we need to weed out any hits that apply to the object itself or its children.

The next piece of visualizes how the BoxCast is performed, using HiddenMonk's code:

if (showDebugLines)
{
    BoxCastHelper.DrawBoxCastBox(GetCameraPosition(stabilizer),
        totalBounds.extents, headRay.direction,
        Quaternion.identity, maxDistance, Color.green);
}

This uses Debug.Draw – these lines are only visible in the Unity editor, in Play mode. They will not show up in the Game pane but in the Scene pane. Which makes sense, as you can them look at the result from every angle without affecting the actual scene in the game.

This looks like this:

image

Now to address the issues I listed on top of this article, we need to do a few things.

Giving it the best shot cast

The next line is a weird one but is explained by the fact that there may be a difference between the center of the actual bounding box (and thus the cast) and center of the object as reported by Unity. I am not entirely sure why this is, but trust me, it's happens with some objects. We need to compensate for that.

var centerCorrection = obj.transform.position - totalBounds.center;

Below you see an example of such an object. I typically happens when an object is composed of one or more other objects that are off center, and especially when the object is asymmetrical. Like this 'floating screen'. You will see it's an empty game object containing a Quad and a 3DTextPrefab that are moved upwards in local space. Without the correction factor, you get the situation on the left - the BoxCast happens 'too low'

imageimage

On the right side, you see the the desired effect. I opted to change the location to of the object to the center of the BoxCast – you might also consider changing the start location of the BoxCast, but that a side effect: the ray won’t start at the user’s viewpoint (but in this case, a little bit above it) which might be confusing or produce undesirable results.

Hit or miss - projection

We need to find the closest hit… but that hit might not be right in front us, along the viewing vector. So  we need to create a vector from the camera to the hit, then make a (longer) vector that follows the user’s gaze, and finally project the ‘hit vector’ to the ‘gaze vector’. Then and only then we know how much room there is in front of us.

if (hits.Any())
{
    var closestHit = hits.First(p => p.distance == hits.Select(q => q.distance).Min());
    var hitVector = closestHit.point - GetCameraPosition(stabilizer);
    var gazeVector = CalculatePositionDeadAhead(closestHit.distance * 2) - 
                       GetCameraPosition(stabilizer);
    var projectedHitVector = Vector3.Project(hitVector, gazeVector);

To show what happens, I have made a screenshot where I made Unity draw debug lines for every calculated vector:

if (showDebugLines)
{
    Debug.DrawLine(GetCameraPosition(stabilizer), closestHit.point, Color.yellow);
    Debug.DrawRay(GetCameraPosition(stabilizer), gazeVector, Color.blue);
    Debug.DrawRay(GetCameraPosition(stabilizer), projectedHitVector, Color.magenta);
}

Which results in the following view (for clarity I have disabled the code that draws the BoxCast for this screenshot)

image

A little magnification shows the area of interest a little bit better:

image

You can clearly see the the yellow line from the camera to the original hit, the blue line which is the viewing direction of the user, and the magenta line projected on that.

Keep your distance please

Now this all works fine for a flat object like a Quad (posing as an 'info screen' here. But not on a box like this for instance (which I made partially translucent for clarity).

image

The issue here is simple, although it took me some time to figure out what was causing it. Like I said before, the hit takes place at the edge of the shape, but the object's position is tied to it's center, so if I set the object's position to that hit, it will end up halfway the obstruction. QED.

So what we need to do is make yet another ray, that will go from the center of the object to the edge, following the same direction as the projected hit vector (the magenta line). Now RayCasts don't work from inside an object, but fortunately there's another way - the Bounds class supports an IntersectRay method. It works a bit kludgy IMHO but it does the trick:

var edgeRay = new Ray(totalBounds.center, projectedHitVector);
float edgeDistance;
if(totalBounds.IntersectRay(edgeRay,  out edgeDistance))
{
    if (showDebugLines)
    {
        Debug.DrawRay(totalBounds.center, 
            projectedHitVector.normalized * Mathf.Abs(edgeDistance + distanceFromObstruction),
            Color.cyan);
    }
}

So we intersect the projected hit vector from the center of the bounds to the edge of the bounds. This will give us the distance from the center to the part of the object that hit the obstruction, and we can move the object 'backwards' to the desired position. Since I specified a 'distanceFromObstruction' we can add that to the distance the object needs to be moved 'back'  as well to keep a distance from an obstruction, in stead of touching it (although for this object it's 0). Yet another debug line, cyan this time, shows what's happening:

image

The cyan line is the part over which the object is moved back. Now the only thing left is to calculate the new position and return it, this time using the centerCorrection we used before to make the object actually appear within the BoxCast's 'outlines':

return GetCameraPosition(stabilizer) +
            projectedHitVector - projectedHitVector.normalized * 
Mathf.Abs(edgeDistance + distanceFromObstruction) + centerCorrection;

Nobody is perfect

If you think "hey, it looks like it is not completely perfectly aligned", you are right. This is because Unity has it's limits in determining volumes and bounding boxes. This is probably because the main concern of a game is performance, not 100% accuracy. If I add this line to the code

BoxCastHelper.DrawBox(totalBounds.center, totalBounds.extents, Quaternion.identity, Color.red);

it actually shows the bounding box:

image

So this explains a bit more what is going on. With all the debug lines enabled it looks like this, which I can imagine is as confusing as helpful ;)

image

Show and tell

It’s actually not really easy to properly show you how this method can be utilized. As I said in the beginning, I will save that for the next post. In the mean time, I have cobbled together a demo project that uses the GetObjectBeforeObstruction in a very simple way. I have created a SimpleKeepInViewController that polls every so many seconds (2 is default) where the user looks, then calls GetObjectBeforeObstruction and moves the object there. This gives a bit of a nervous result, but you get the idea.

public class SimpleKeepInViewController : MonoBehaviour
{
    [Tooltip("Max distance to display object before user")]
    public float MaxDistance = 2f;

    [Tooltip("Distance before the obstruction to keep the current object")]
    public float DistanceBeforeObstruction = 0.02f;

    [Tooltip("Layers to 'see' when detecting obstructions")]
    public int LayerMask = Physics.DefaultRaycastLayers;

    [Tooltip("Time before calculating a new position")]
    public float PollInterval = 2f;

    [SerializeField]
    private BaseRayStabilizer _stabilizer;

    [SerializeField]
    private bool _showDebugBoxcastLines = true;

    private float _lastPollTime;


    void Update()
    {
        if (Time.time > _lastPollTime)
        {
            _lastPollTime = Time.time + PollInterval;
            LeanTween.move(gameObject, GetNewPosition(), 0.5f).setEaseInOutSine();
        }
#if UNITY_EDITOR
        if (_showDebugBoxcastLines)
        {
            LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
                DistanceBeforeObstruction, LayerMask, _stabilizer, true);
        }
#endif
    }

    private Vector3 GetNewPosition()
    {
        return LookingDirectionHelpers.GetObjectBeforeObstruction(gameObject, MaxDistance,
            DistanceBeforeObstruction, LayerMask, _stabilizer);
    }
}

There is only one oddity here – you see I actually call GetObjectBeforeObstruction twice. But the first time only happens in the editor, and only if you select the Show Debug Boxcast Line checkbox:

imageIf I did not add this, you would see the lines flash for one frame every 2 seconds, which is hardly enlightening. This way, you can see them all the time in the editor



imageIn the demo project you will find three objects – in the images above you have already seen a single block (the default), a rotating ‘info screen’ that shows “Hello World” and there’s also this composite object on the left (two cubes off-center), here displayed with all debug lines enabled ;). You can toggle between the three objects by saying “Toggle” or by pressing the “T”. The latter will actually also work in a HoloLens if you have a Bluetooth keyboard attached - and believe me, I tried ;-)


Conclusion

Yet another way to make an object appear next to on on top of an obstruction ;). This code actually took me way too much time to complete, but I learned a lot from it and at some point it became a matter of honor to get the bloody thing to work.

Fun factoid: most of the code, and a big part of the blog post, was actually written on train trips to an from an awesome HoloLens project I am currently involved in. Both the demo project and this blog post were actually published while being on my way, courtesy of the Dutch Railways free WiFi service ;)