23 July 2011

Speed and distance calculation extension methods for Windows Phone 7

Upgrading stuff to from Windows Phone 7 to “Mango” is sometimes like cleaning your desk drawer before going on a holiday – you stumble upon stuff you forgot it was even there. For the ‘game engine’ I made for Catch’em Birds I created a few very handy extension methods for calculating speed and distance, which I would like to share with the rest of the community:

using System;
#if WINDOWS_PHONE
using System.Windows;
#else
using System.Drawing;
using System.Windows;
#endif


namespace Wp7nl.Utilities
{
  public static class PointExtensions
  {
    /// <summary>
    /// Distances from point 1 to point 2
    /// </summary>
    /// <param name="p1">The first point.</param>
    /// <param name="p2">The second point.</param>
    /// <returns></returns>
    public static double DistanceFrom(this Point p1, Point p2)
    {
      var dX = p2.X - p1.X;
      var dY = p2.Y - p1.Y;
      return Math.Sqrt(dX * dX + dY * dY);
    }

    /// <summary>
    /// Calculates the speed in pixels per second
    /// </summary>
    /// <param name="p1">The first point.</param>
    /// <param name="p2">The second point.</param>
    /// <param name="duration">The duration</param>
    /// <returns>Speed in pixels per second</returns>
    public static double CalculateSpeed(this Point p1, Point p2, Duration duration)
    {
      return p1.DistanceFrom(p2) / duration.TimeSpan.TotalSeconds;
    }

    /// <summary>
    /// Calculates the duration given a distance and a speed.
    /// </summary>
    /// <param name="p1">The first point.</param>
    /// <param name="p2">The second point.</param>
    /// <param name="speed">The speed in pixels per second.</param>
    /// <returns>Time it takes to get from p1 to p2</returns>
    public static Duration CalculateDuration(this Point p1, Point p2, double speed)
    {
      return new Duration(TimeSpan.FromSeconds(p1.DistanceFrom(p2) / speed));
    }
  }
}

I will discuss these in a not quite logical order:

  • The first method is of course good ole' Pythagoras caught in an extension method, basically there as a helper method for the other two.
  • The third method is the one I used most: given that my object needs to move from p1 to p2 with a given speed in pixels per second, what’s the Duration I need to apply to my Storyboard?
  • The second one is the inverse – given the fact that on object was moving from p1 to p2 in duration “duration”, calculate its speed in pixels per second.

As you can see at the #if on top it also work on System.Drawing.Points under full .NET 4.0. Maybe these methods are useful there as well, I put that only there to facilitate some unit tests.

All these methods are part of the wp7nl CodePlex library – or at least the will so at the upcoming Mango release.

Update: thanks to Rene Schulte for pointing out to me that my original DistanceFrom method using Math.Pow is very slow compared to simple multiplications. Rene is a Silverlight MVP from Dresden, Germany and has some very nice Point extension methods on his blog “Kodierer” as well - in English, don’t worry ;-)

20 July 2011

Concatenating a long set of objects in a string with a separator

It still amazes me when I see people concatenating a long set of repetitive strings with a StringBuilder, adding a separator to each string it, and then remove the last separator – or doing some complex if-statement comparing a count or index to prevent a trailing separator or something.

This is a very simple trick I use a lot of times when I need, for instance, to change a list of points into it’s Well Known Text (WKT) representation. So let’s assume we have a variable points of IList<System.Windows.Point>, that I want to convert to a list of coordinates that fit in WKT string. That is: X and Y separated by a space, and coordinates separated by a comma. I can do that with this simple statement that would have been a one-liner if it had not been for lack of space ;-) :

var pointString = 
  string.Join(", ",
    points.Select(p => 
      string.Format(CultureInfo.InvariantCulture, "{0} {1}", p.X, p.Y)));

If I had points X=10, Y=15, X=20, Y=25 and X=30 ,Y=40 this would nicely translate to

10 15,20 25,30 40

That's all. No loops, no truncating trailing comma’s: just a Select and a Join

16 July 2011

Using Kinect and MVVMLight 4 for some basic Google Maps manipulation

Being a new tech junkie, I of course wanted to try the waters when the Kinect beta SDK was released on June 16, 2011. I thought it best to avoid domestic trouble and let the Kinect my wife bought back in December sit nicely connected to the XBox360 downstairs, and ordered a second Kinect – which just happened to be on sale for only €99.

The global idea

I wanted to create a simple application that allows you to both pan and zoom on a standard Google Maps web site, using hand gestures and some speech commands. I wanted to show visual feedback projected over the browser, like moving hands and stuff like that. Because I wanted to re-use some of my hard-won Windows Phone 7 knowledge, I wanted to use MVVM as well. So enter MVVMLight and while I was at it, version 4 too. The SDK is not usable from Silverlight, so the choice fell logically on Windows Presentation Foundation (WPF) to host a browser and see how things went from there. The result is below. It’s pretty crude and the user experience leaves much to be desired, but it’s a nice start. I will explain what I’ve done and why, and as usual include the whole sample application.

Basic operation

When the application opens up, it shows Google Maps in a full screen browser, and pretty much nothing else. To enable a demo I am planning to give, the Kinect is not immediately initialized. Forgive this old Trekkie - I could not resist. For years I’ve envied Kirk and Picard for being able to command their computer by talking to it. To get the application to start tracking your hands, you have to say “Kinect engage”. :-)

If Kinect understands your command, it will show the command to the left. If you move your hands, to yellow semi-transparent hand symbols should appear. If you move your hands from left to right they follow you – moving them forward will make them smaller, moving them backward bigger.

If you say “Kinect track”  you can do the following:

rightzoominIf you hold your left hand closer to your body than your right hand by some 20 cm, it will zoom in on the location of your right hand (green “plus” symbol on the right hand symbol). 

rightzoomoutIf you hold your left hand further from your body than your right hand by some 20 cm, it will zoom out on the location of your right hand (red “minus” symbol on the right hand symbol”)

rightpanIf you stretch out both arms and then move your hands, you can pan the map (blue circle with cross will appear on both of the hand symbols).

Below is a partial screenshot showing Kinect zooming in on Ontario.

kinecttrack

Miscellaneous voice commands:

  • “Kinect stop tracking” will keep following your hands but disable zoom and pan until you say “Kinect track” again
  • “Kinect video on” will show what the Kinect video cam is seeing on the top left of the screen
  • “Kinect video off” will hide video again (you might want to do this as it’s heavy on performance)
  • “Kinect shutdown” will exit the application

 

The application in action. Since my computer choked on running both Kinect and a screen recorder I actually did this the old fashioned way using a video camera recording the screen ;)

Setting the stage

To get this sample to work, you need quite a lot of stuff, both hard- and software. First, of course, you will need a Kinect sensor. Be aware you cannot use a Kinect that comes with an XBox360s – that only has the proprietary orange Kinect connector that may look like and USB connector, but most certainly is not. You will need a retail Kinect for Xbox360 sensor, which includes special USB/power cabling – that will connect to your PC’s ‘ordinary’ USB port

A word to the wise: if you have a desk top PC, connect Kinect to a back USB port and never, ever use an USB extension cable. Unless you are entertained by very frequent BSOD’s. I learned it the hard way - make sure you don’t.

Then you need quite some stuff to download and install:

I would also recommend very much installing the Kinect project templates for Visual Studio 2010 by the brilliant Dennis Delimarsky. I started out by using his Kinect Skeleton template application.

MVVMLight 4

MVVMLight 4 does not have a binary release yet, as far as I understand, so I pulled the sources from CodePlex, compiled the whole stuff and took the following parts:

  • GalaSoft.MvvmLight.WPF4.dll
  • GalaSoft.MvvmLight.Extras.WPF4.dll
  • Microsoft.Practices.ServiceLocation.dll
  • System.Windows.Interactivity.dll

After building you will find the first file in GalaSoft.MvvmLight\GalaSoft.MvvmLight (NET4)\bin\Release, the others in GalaSoft.MvvmLight\GalaSoft.MvvmLight.Extras (NET4)\bin\Release

Coding4Fun Kinect Toolkit

I used only a tiny bit of it, and still have to investigate what it can do, but you can find it here on CodePlex.

Setting up the project

KinectSolutionA good programmer is a lazy programmer, so I just selected Dennis Delimarsky’s KinectSkeletonApplication template (you will find it under Visual C#\Windows\Kinect). That creates a nice basic WPF-based Kinect skeleton tracking application that works out of the box – it projects the video image Kinect sees, and tracks your hands with two red circles. This actually got me off the ground in no time at all, so kudos and thanks to you, Dennis!

After peeking in Dennis’ code I cleaned the MainPage.xaml and MainPage.xml.cs. Then I added the following references:

  • Coding4Fun.Kinect.Wpf.dll
  • GalaSoft.MvvmLight.WPF4.dll
  • GalaSoft.MvvmLight.Extras.WPF4.dll
  • Microsoft.Practices.ServiceLocation.dll
  • System.Windows.Interactivity.dll
  • Microsoft.Speech.dll (on my computer it’s sitting in C:\Program Files\Microsoft Speech Platform SDK\Assembly)

As usual, I put assemblies like this in a solution folder called “Binaries” first, to ensure they become part of the solution itself. The net result is displayed right:

Main viewmodel

MainViewModel is very simple and is basically nothing more than a Locater:

using GalaSoft.MvvmLight;

namespace MapController.ViewModel
{
  public class MainViewModel : ViewModelBase
  {
    private static PoseViewModel _poseViewModelInstance;
    public static PoseViewModel PoseViewModel
    {
      get { return _poseViewModelInstance ?? 
        (_poseViewModelInstance = new PoseViewModel()); }
    }
  }
}

This is a pretty standard pattern that makes sure that now matter how much MainViewModels you create by XAML instantiation, there will be one and only one PoseViewModel.

The Pose viewmodel

PoseViewModel is currently a kind of God class and probably should be broken up should this become a real application one day. This is where most of the Kinect stuff lives. And it’s remarkably small for all the things it actually does. It starts out like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows;
using Coding4Fun.Kinect.Wpf;
using GalaSoft.MvvmLight;
using GalaSoft.MvvmLight.Messaging;
using MapController.Messages;
using Microsoft.Research.Kinect.Nui;
using Vector = Microsoft.Research.Kinect.Nui.Vector;

namespace MapController.ViewModel
{
  public class PoseViewModel : ViewModelBase, IDisposable
  {
    private Runtime _runtime;
    private const int Samples = 2;
    private const string DefaultleftHand = "Resources/lefthand.png";
    private const string DefaultrightHand = "Resources/righthand.png";

    private readonly List<SkeletonData> _skeletons;

    public PoseViewModel()
    {
      if (!IsInDesignMode)
      {
        _leftHandImage = DefaultleftHand;
        _rightHandImage = DefaultrightHand;
        _skeletons = new List<SkeletonData>();
        _runtime = new Runtime();
        _runtime.SkeletonFrameReady += RuntimeSkeletonFrameReady;
        _runtime.VideoFrameReady += RuntimeVideoFrameReady;
        _runtime.Initialize(
          RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);
        _runtime.VideoStream.Open(ImageStreamType.Video, 
          2, ImageResolution.Resolution640x480, ImageType.Color);
        Messenger.Default.Register<CommandMessage>(this, ProcessSpeechCommand);
        SpeechController.Initialize();
      }
    }

    void RuntimeVideoFrameReady(object sender, ImageFrameReadyEventArgs e)
    {
      if (ShowVideo)
      {
        Messenger.Default.Send(new VideoFrameMessage {Image = e.ImageFrame.Image});
      }
    }
  }
}

Important to note is the Vector alias on top – there is also a System.Windows.Vector and you sure don’t want to use that one. This basically initializes the Kinect “Runtime”, instructs it to use skeleton and video tracking and defines callbacks for that. The SpeechController – I will get to that later – is initialized as well. Note the _skeletons list – I use that to take the average of a more than one (actually two) skeletons to make the hand tracking a bit more stable.

Note also the callback “RuntimeVideoFrameReady”- it just shuttles off a frame in a message. A behavior - the DisplayVideoBehavior - will take care of that. I tried to do this with data binding; it works, but gave a less than desirable performance, to put it mildly.

The ViewModel has 12 properties, which all follow the new MVVMLight 4 syntax:

private Vector _lefthandPosition;
public Vector LeftHandPosition
{
  get { return _lefthandPosition; }
  set
  {
    if (!_lefthandPosition.Equals(value))
    {
       _lefthandPosition = value;
       RaisePropertyChanged(() => LeftHandPosition);
    }
  }
}

To prevent this blog post challenging Tolstoy’s "War and Peace'” for length I will only name the rest by name and type:

  • Vector RightHandPosition
  • double LeftHandScale
  • double RightHandScale
  • Visibility HandVisibility
  • string LeftHandImage
  • string RightHandImage
  • bool IsInitialized
  • bool IsTracking
  • bool ShowVideo
  • string LastCommand

Except for this one, since is does something more – it changes the images for the hands, if neccesary:

private bool _isPanning;
public bool IsPanning
{
  get { return _isPanning; }
  set
  {
    if (_isPanning != value)
    {
      _isPanning = value;
      LeftHandImage = _isPanning ? "Resources/leftpan.png" : DefaultleftHand;
      RightHandImage = _isPanning ? "Resources/rightpan.png" : DefaultrightHand;
      RaisePropertyChanged(() => IsPanning);
    }
  }
}

The actual skeleton and pose processing code is surprisingly simple, then:

void RuntimeSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
  if (IsInitialized)
  {
    var skeletonSet = e.SkeletonFrame;

    var data = (from s in skeletonSet.Skeletons
                where s.TrackingState == SkeletonTrackingState.Tracked
                select s).FirstOrDefault();

    lock (new object())
    {
      _skeletons.Add(data);
    }
    if (_skeletons.Count < Samples)
    {
      return;
    }
    lock (new object())
    {
      LeftHandPosition = 
        ScaleJoint(_skeletons.Average(JointID.HandLeft)).Position;
      RightHandPosition = 
        ScaleJoint(_skeletons.Average(JointID.HandRight)).Position;
      var spinePosition = 
        ScaleJoint(_skeletons.Average(JointID.Spine)).Position;
      _skeletons.Clear();

      LeftHandScale = RelativePositionToScale(spinePosition, LeftHandPosition);
      RightHandScale = RelativePositionToScale(spinePosition, RightHandPosition);
      HandVisibility = Visibility.Visible;

      if (IsTracking)
      {
        // Both arms stretched: initiate pan
        IsPanning = ((spinePosition.Z - LeftHandPosition.Z) > 0.5 && 
                    (spinePosition.Z - RightHandPosition.Z) > 0.5);

        if (!IsPanning)
        {
          // Send a zoom in/out message depending on the hand's relative positions 
          var handDist = (LeftHandPosition.Z - RightHandPosition.Z);
          if (handDist > 0.2)
          {
            // left hand pulled back: zoom in
            Messenger.Default.Send(new ZoomMessage 
             { X = RightHandPosition.X, Y = RightHandPosition.Y, Zoom = 1 });

            RightHandImage = "Resources/rightzoomin.png";
          }
          else if (handDist < -0.2)
          {
            // left hand pushed forward back: zoom out
            Messenger.Default.Send(new ZoomMessage 
              { X = RightHandPosition.X, Y = RightHandPosition.Y, Zoom = -1 });
            RightHandImage = "Resources/rightzoomout.png";
          }
          else
          {
            RightHandImage = DefaultrightHand;
          }
        }
      }
      else
      {
        IsPanning = false;
      }
    }
  }
}

This is really the core of the whole pose recognition performed by this application. The method basically takes a skeleton and puts it in a list. If the desired number of samples are obtained, the average of joints is calculated and scale to the screen. Then it checks if both hands are 50 cm or more before the main body (‘Spine’) – this is assumed to be a panning pose. If that’s not recognized, the method tries to detect if one hand is more than 20 cm before another, so it should initiate a zoom action – this is done by sending a ZoomMessage. If that’s so, the right hand’s image is changed accordingly. The ZoomMessage is so simple I will omit it's source here

This uses a few other methods:

/// <summary>
/// Convert a relative position to a scale
/// </summary>
/// <param name="spinePos"></param>
/// <param name="handPos"></param>
/// <returns></returns>
private double RelativePositionToScale( Vector spinePos, Vector handPos )
{
  // 30 cm before the chest is 'zero position'
  return 1 - ((spinePos.Z - handPos.Z - 0.3) * 1.8);
}

/// <summary>
/// Scale a joint
/// </summary>
/// <param name="joint"></param>
/// <returns></returns>
private Joint ScaleJoint(Joint joint)
{
  return joint.ScaleTo(
    (int)SystemParameters.PrimaryScreenWidth, 
    (int)(SystemParameters.PrimaryScreenHeight), 0.5f, 0.5f);
}
Both speak pretty much for themselves. The Joint.ScaleTo extension method is coming from Coding4Fun. The ‘Average’ extension methods are my own, and are defined in a separate class:
using System.Collections.Generic;
using System.Linq;
using Microsoft.Research.Kinect.Nui;
using Vector = Microsoft.Research.Kinect.Nui.Vector;

namespace MapController
{
  public static class KinectExtensions
  {
    /// <summary>
    /// Calculates the average Vectors of a any number of Vectors
    /// </summary>
    public static Vector Average(this IEnumerable<Vector> vectors)
    {
      return new Vector {
        X = vectors.Select(p => p.X).Average(),
        Y = vectors.Select(p => p.Y).Average(),
        Z = vectors.Select(p => p.Z).Average(),
        W = vectors.Select(p => p.W).Average(),
      };
    }

    /// <summary>
    /// Calculates the average of a specific Joint in a number of vectors
    /// </summary>
    public static Joint Average( this IEnumerable<SkeletonData> data, 
      JointID joint)
    {
      return new Joint
      {
        Position = data.Select(skeleton => 
          skeleton.Joints[joint].Position).Average()
      };
    }
  }
}

Now let's have a look at the GUI.

Transparent overlay window

WPF poses some unique challenges, one of those being the fact that no WPF objects can be drawn on top of Win32 objects – like, for instance, a WebBrowser control. To circumvent this I created second child window – full screen, like the first window, but Transparent. So I added KinectOverlay.xaml. This window is launched from MainWindow.xaml.cs as a child window like this:

namespace MapController
{
  public partial class MainWindow : Window
  {
    public MainWindow()
    {
      InitializeComponent();
      Loaded += MainWindowLoaded;
    }

    void MainWindowLoaded(object sender, RoutedEventArgs e)
    {
      var w = new KinectOverlay {Owner = this};
      w.Show();
    }
  }
}

MainWindow

There's not much in there, actually. Just a browser and some behaviors.

<Window xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" 
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
    xmlns:i="http://schemas.microsoft.com/expression/2010/interactivity" 
    xmlns:MapController_Behaviors="clr-namespace:MapController.Behaviors" 
    WindowStyle="None" WindowState="Maximized" 
    x:Class="MapController.MainWindow" Height="355" Width="611" 
    ShowInTaskbar="True" AllowsTransparency="False"
    DataContext="{Binding PoseViewModel, Source={StaticResource MainViewModel}}">
  <i:Interaction.Behaviors>
    <MapController_Behaviors:ZoomBehavior/>
    <MapController_Behaviors:PanBehavior 
        RightHandPosition="{Binding RightHandPosition, Mode=TwoWay}" 
        LeftHandPosition="{Binding LeftHandPosition, Mode=TwoWay}"
        IsPanning="{Binding IsPanning, Mode=TwoWay}">
    </MapController_Behaviors:PanBehavior>
  </i:Interaction.Behaviors>
   <Grid x:Name="Grid">
     <WebBrowser x:Name="Browser" Source="http://maps.google.com" />
   </Grid>
</Window>

As you can see, the actual zoom and pan actions are performed by two behaviors – named “Zoombehavior” and “PanBehavior”. Windows Phone 7 developers, please take note of the fact that WPF allows you to directly bind to behavior dependency properties (this will be possible in the next Windows Phone 7 release as well, since that runs Silverlight 4-ish). Here we come down to the dirty parts of the solution, for the actual map manipulation is performed by simulating mouse position and buttondowns for dragging, and mouse wheel rotations for zoomin in and out. If you are interested in the actual innards of these things, please refer to the sample solution. Below is just a short description

  • The Zoombehavior just waits for a ZoomMessage – and then simulates a zoom in or out by moving the mouse cursor to the position of the right hand, and then simulating a scroll wheel rotation by one position. It only accepts one zoom per 2 seconds to prevent zooming in or out at an uncontrollable rate.
  • The PanBehavior has three dependency properties bound to both hand positions, and IsPanning. If the model says its panning, it calculates the point between left and right hand, and uses that as a dragging origin.

Both use a NativeWrapper static class that uses some Win32 api calls – courtesy of pinvoke.net

KinectOverlay

Also pretty simple:  A Grid containing a Canvas, which contains standard behavior, and then three images and a text: left hand, right hand, video output, and text place holder to show the last command – all bound to the PoseViewModel. Don’t you just love the power of XAML? ;-)

<Window
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"  
    xmlns:i="http://schemas.microsoft.com/expression/2010/interactivity" 
    xmlns:ei="http://schemas.microsoft.com/expression/2010/interactions" 
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008" 
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" 
    xmlns:MapController_Behaviors="clr-namespace:MapController.Behaviors" 
    mc:Ignorable="d" x:Class="MapController.KinectOverlay"
    Title="KinectOverlay" Height="300" Width="300" WindowStyle="None" 
    WindowState="Maximized" Background="Transparent" AllowsTransparency="True">
  <Grid>
  <i:Interaction.Behaviors>
    <ei:FluidMoveBehavior AppliesTo="Children" Duration="0:0:0.5">
    <ei:FluidMoveBehavior.EaseX>
      <SineEase EasingMode="EaseInOut"/>
    </ei:FluidMoveBehavior.EaseX>
    <ei:FluidMoveBehavior.EaseY>
      <SineEase EasingMode="EaseInOut"/>
    </ei:FluidMoveBehavior.EaseY>
    </ei:FluidMoveBehavior>
  </i:Interaction.Behaviors>
  <Canvas Background="Transparent" 
  DataContext="{Binding PoseViewModel, Source={StaticResource MainViewModel}}">

    <!-- Left hand -->
    <Image Source="{Binding LeftHandImage}" x:Name="leftHand" Stretch="Fill"
       Canvas.Left="{Binding LeftHandPosition.X, Mode=TwoWay}"  
       Canvas.Top="{Binding LeftHandPosition.Y, Mode=TwoWay}"
       Visibility="{Binding HandVisibility}" Opacity="0.75"
       Height="118" Width="80" RenderTransformOrigin="0.5,0.5">
    <Image.RenderTransform>
      <TransformGroup>
      <ScaleTransform ScaleX="{Binding LeftHandScale}" 
        ScaleY="{Binding LeftHandScale}"/>
      <SkewTransform/>
      <RotateTransform/>
      <TranslateTransform X="-40" Y="-59"/>
      </TransformGroup>
    </Image.RenderTransform>
    </Image>

    <!-- Right hand -->
    <Image x:Name="righthand" Source="{Binding RightHandImage}" Stretch="Fill"
       Canvas.Left="{Binding RightHandPosition.X, Mode=TwoWay}"  
       Canvas.Top="{Binding RightHandPosition.Y, Mode=TwoWay}"
       Visibility="{Binding HandVisibility}" Opacity="0.75"
       Height="118" Width="80" RenderTransformOrigin="0.5,0.5">
    <Image.RenderTransform>
      <TransformGroup>
      <ScaleTransform ScaleX="{Binding RightHandScale}" 
        ScaleY="{Binding RightHandScale}"/>
      <SkewTransform/>
      <RotateTransform/>
      <TranslateTransform X="-40" Y="-59"/>
      </TransformGroup>
    </Image.RenderTransform>
    </Image>

    <!-- Video -->
    <Image Canvas.Left="0" Canvas.Top="100" Width ="360" 
     Visibility="{Binding ShowVideo, Converter={StaticResource booleanToVisibilityConverter}}">
    <i:Interaction.Behaviors>
      <MapController_Behaviors:DisplayVideoBehavior/>
    </i:Interaction.Behaviors>
    </Image>
    
    <!-- Shows last speech command -->
    <TextBlock Canvas.Left="10" Canvas.Top="500" Text="{Binding LastCommand}"
      FontSize="36" Foreground="#FF001900"></TextBlock>
  </Canvas>
  </Grid>
</Window>

Note that both MainPage.xaml and KinectOverlay.xaml are full-screen with no Window-style but that KinectOverlay.xaml has Background="Transparent" and AllowsTransparency="True" attributes. This actually creates the ‘transparent overlay.’

Speech recognition

If you think, after reading this, that skeleton tracking with Kinect is almost embarrassingly simple, wait till you see speech recognition. I barely scratched the surface I think, but apart from a lot of initialization bruhaha getting Kinect to recognize basic phrases is a simple matter of feeding a couple of strings to a “Choices” object, feeding that to a “GrammarBuilder” and finally loading that into the “SpeechRecognitionEngine” object. Add a callback to it’s SpeechRecognized property and out come your recognized strings. No calibration, initialization, no hours of ‘training’, nothing – it just works.

Moving back to the last part of the PoseViewModel: at the very end of it is a static property for the speech controller and a simple switch recognizing the speech commands.

#region Speech commands
private static SpeechController _speechControllerInstance;
public SpeechController SpeechController
{
  get { return _speechControllerInstance ?? 
    (_speechControllerInstance = new SpeechController()); }
}

private void ProcessSpeechCommand(CommandMessage command)
{
  switch (command.Command)
  {
    case VoiceCommand.Shutdown:
      {
        Application.Current.Shutdown(); break;
      }
    //etc etc rest of the commands omitted
    case VoiceCommand.VideoOff:
      {
        ShowVideo = false; break;
      }
  }

  LastCommand = command.Command.ToString();
}
#endregion

The method ProcessSpeechCommand is called whenever the model receives a CommandMessage – that is issued by the SpeechController. As you can see, the spoken commands are contained in a simple enumeration “VoiceCommand”. Being too lazy to make a proper factory for all the commands, I include the factory methods in CommandMessage as well:

using System.Collections.Generic;
using Microsoft.Speech.Recognition;

namespace MapController.Messages
{
  /// <summary>
  /// Command the application is supposed to understand
  /// </summary>
  public class CommandMessage
  {
    public VoiceCommand Command { get; set; }

    /// <summary>
    /// Factory methods
    /// </summary>
    private static Dictionary<string, VoiceCommand> _commands;
    public static IDictionary<string, VoiceCommand> Commands
    {
      get
      {
        if (_commands == null)
        {
          _commands = new Dictionary<string, VoiceCommand>
          {
            {"kinect engage", VoiceCommand.Engage},
            {"kinect stop tracking", VoiceCommand.StopTracking},
            {"kinect track", VoiceCommand.Track},
            {"kinect shutdown", VoiceCommand.Shutdown},
            {"kinect video off", VoiceCommand.VideoOff},
            {"kinect video on", VoiceCommand.VideoOn}
          };
        }
        return _commands;
      }
    }

    public static Choices Choices
    {
      get
      {
        var choices = new Choices();
        foreach (var speechcommand in Commands.Keys)
        {
          choices.Add(speechcommand);
        }
        return choices;
      }
    }
  }
}

The static Commands property is simple translation table from actual text to VoiceCommand enumeration values. The voice commands themselves (as you see, it’s just strings) are fed into a Choices object. Now the only thing that’s missing is the actual speech recognition ‘engine’:

using System;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Threading;
using System.Windows;
using System.Windows.Threading;
using GalaSoft.MvvmLight.Messaging;
using MapController.Messages;
using Microsoft.Research.Kinect.Audio;
using Microsoft.Speech.AudioFormat;
using Microsoft.Speech.Recognition;

namespace MapController.ViewModel
{
  public class SpeechController : IDisposable
  {
    private const string RecognizerId = "SR_MS_en-US_Kinect_10.0";
    private SpeechRecognitionEngine _engine;
    private KinectAudioSource _audioSource;
    private Stream _audioStream;
    private Thread _audioThread;

    public void Initialize()
    {
      // Audio recognition needs to happen on a separate thread
      _audioThread = new Thread(InitSpeechRecognition);
      _audioThread.Start();
    }

    private void InitSpeechRecognition()
    {
      // All kinds on initialization bruhaha directly taken from sample
      _audioSource = new KinectAudioSource
                  {
                    FeatureMode = true,
                    AutomaticGainControl = false,
                    SystemMode = SystemMode.OptibeamArrayOnly
                  };
      var ri =
        SpeechRecognitionEngine.InstalledRecognizers().
          Where(r => r.Id == RecognizerId).FirstOrDefault();
      _engine = new SpeechRecognitionEngine(ri.Id);
      var gb = new GrammarBuilder { Culture = new CultureInfo("en-US") };

      // Building my command list
      gb.Append(CommandMessage.Choices);

      // More initialization bruhaha directly taken from sample
      var g = new Grammar(gb);
      _engine.LoadGrammar(g);
      _engine.SpeechRecognized += SreSpeechRecognized;

      _audioStream = _audioSource.Start();
      _engine.SetInputToAudioStream(_audioStream,
                                    new SpeechAudioFormatInfo(
                                    EncodingFormat.Pcm, 16000, 16, 1,
                                    32000, 2, null));
      _engine.RecognizeAsync(RecognizeMode.Multiple);
    }

    private void SreSpeechRecognized(object sender, 
      SpeechRecognizedEventArgs e)
    {
      // Convert spoken text into a command
      if (CommandMessage.Commands.ContainsKey(e.Result.Text))
      {
        Application.Current.Dispatcher.Invoke(DispatcherPriority.Normal,
          new Action(() =>
            Messenger.Default.Send(
               new CommandMessage { 
               Command = CommandMessage.Commands[e.Result.Text] })));
      }
    }
  }
}

This is mostly converted from the basic Speech sample. Only the red things are actually (mostly) mine. Most interesting to note is the fact that for some reason speech recognition needs to be done on a separate thread. Don’t ask me why – it just needs to. In the middle you can see I feed my Choices-built-from-commands  to a ‘SpeechRecognitionEngine’ going via the ‘GrammarBuilder’ as described earlier, and the SreSpeechRecognized that fires when a speech command is recognized. I call the Messenger back on the UI thread (remember, we are on a separate thread here so nothing bound can be accessed directly) and the result is fired back into the ProcessSpeechCommand of the model, that acts on it as described above.

Caveat: this speech recognition does not work with an ordinary microphone. Kinect is doing the recognition, apparently. Speech recognition samples without a Kinect connected to your computer simply do not start up.

Conclusion and lessons learned

As said before, the solution is pretty crude, and so is the user experience. For instance, the application might start with measuring body dimensions. A person with arms shorter that 50 cm would have trouble getting the application to pan, for instance ;-).  But for a first try – with no prior experience – I think it’s a nice start of getting off the ground with Kinect development. I had tremendous fun experimenting with it, although I got a bit distracted from my main passion, i.e. Windows Phone 7. I hope to show this very soon at Vicrea and who knows, maybe this will turn into actual work ;-)

Apart from the actual knowledge and concepts of the API and Kinect controlling, I have learned the following lessons from this application

  1. A good technical implementation of controlling applications with gestures needs an API that is supplied by the controlled application,  or some kind of wrapper. I now use a pretty crude trick, by simulating mouse actions. For a real gesture controlled application something more is needed
  2. When it comes to gesture control, you are basically on your own, without guidelines on ‘how to do things’. For the past 20 years we’ve been using mouse and keyboard to control our computers. This led to a well defined ‘language’ of ‘concepts that has firmly taken root in our consciousness – things like clicking left and right buttons, dragging, using the mouse wheel for zooming in our out, using cursor keys, menu structures (like about/help and tools/properties) – heck, things like CTRL-ALT-DELETE even have become a figure of speech. But when in comes to gestures, there are no rules, written or unwritten, that describe the ‘logical’ way of zooming and panning a map. Apart from creating the application itself, I had to ‘invent’ the actual poses or gestures. So the logic in zooming out by pulling a hand toward me is mine, but not necessarily yours. A very odd experience. But a fun one.

Sample solution can be obtained here.

18 June 2011

Mogade powered Live Tile high score service for Windows Phone 7

Those who are following my twitter timeline cannot have missed the fact that I recently published a little game called “Catch’em birds” for Windows Phone 7 in the Marketplace. This was mainly a trial project to see if I could bend MVVMLight far enough to use in an action  game (result: yes you can). It’s not exactly a smashing success in terms of downloads and sales, and it does not use any of the Windows Phone exclusive features. So I decided to add a Live Tile showing the overall high score and if possible, the player’s rank. I’ve noticed that some people in the Dutch Windows Phone developer community (I won’t mention names here) tend to get particularly competitive when high scores are involved, so maybe this would boost download. And if not, then at least I’d have a lot of fun developing it.

Catch’em birds features an online leaderboard using the services of Mogade. They have a nice and very simple to use library that you can include in your Windows Phone 7 project. You make an account, you define a game, and you define a leaderboard. This will give you 3 keys that you must include in your App and you are ready to go. Samples on their site are quite adequate so I won’t bore you with a repeat of that.

But now I want to use the same leaderboard services from a Live Tile service. So I defined a solution with an empty ASP.NET Web application project, and deleted all source files except for web.config. In the web.config, I defined four keys:

<appSettings>
  <add key="gamekey" value="your gamekey here"/>
  <add key="secret" value="your game secret here"/>
  <add key="leaderboard" value="your leaderboard key here"/>
  <add key="baseimageurl" value="Catchembirds/background.png"/>
</appSettings>

The first three are the Mogade keys and are supposed to be secret, so you are not going to include them in your tile request, but ‘somewhere safe’. How safe a plain ole’ web.config is, is subject of discussion but not the point of this article.

To make Mogade work, you will need Mogade.Core.dll and Newtonsoft.Json.Net35.dll. Beware: use the full .NET version. This a .NET web application, not a Windows Phone application ;-). You can download these from Mogade or just nick them out of my sample solution. Make a reference to those two dll’s and then it’s simply a case of defining a Generic Handler (ashx) and filling in the ProcessRequest method.

First, I get all the variables I need. My leaderboard is configured to use player name and anonymous live id to recognize the player, so I have to pick those from an URL. The rest comes from the web.config.

var userName = context.Request["username"];
var anid = context.Request["anid"];
var gameKey = ConfigurationManager.AppSettings["gamekey"];
var secret = ConfigurationManager.AppSettings["secret"];
var leaderboard = ConfigurationManager.AppSettings["leaderboard"];
var baseImageUrl = ConfigurationManager.AppSettings["baseImageUrl"];

The second part opens the base image, calls the Mogade service, waits for it to return data, and draws the result on the image. If the user has not played or has no name entered into the game settings yet, there may be no ranking, so that’s optional. To show that it actually works and the tile is indeed updated multiple times I’ve included code to draw a timestamp on the tile as well. This you will never do in real life, of course.

using (var myBitmap = new Bitmap(context.Server.MapPath(baseImageUrl)))
{
  try
  {
    var readyCalled = new ManualResetEvent(false);
    var driver = new Driver(gameKey, secret);

    // Get high score and user ranking
    driver.GetLeaderboard(leaderboard, LeaderboardScope.Overall, 1, 50,
      userName, anid,
      p =>
      {
        if (p.Success)
        {
          using (var g = Graphics.FromImage(myBitmap))
          {
            var font = new Font("Segoe", 5, FontStyle.Bold);
            // Draw high score on the tile
            var points = p.Data.Scores[0].Points;
            g.DrawString("h: " + points.ToString(),
                          font, Brushes.White, new PointF(10, 5));

            // If user has ranking, draw ranking
            if (p.Data.User != null)
            {
              var rank = "r: " + p.Data.User.Rank.ToString();
              var size = g.MeasureString(rank, font);
              g.DrawString(rank.ToString(),font,
                            Brushes.White, 
                            new PointF(173 - size.Width - 10, 5));
            }
            // Draw timestamp - for debugging purposes only
            g.DrawString(
              String.Format(
                new CultureInfo("en-US"),
                "{0:HH:mm\ndd-MM-yyyy}", DateTime.UtcNow),
                  font, Brushes.White, new PointF(10, 110));
          }
        }
        // Indicate Mogade call is complete
        readyCalled.Set();
      });

    // Service waits here for Mogade call to complete
    readyCalled.WaitOne();
  }

To make the code wait for the call to Mogade to complete, I use a ManualResetEvent. Calling its “WaitOne” method basically hangs the main thread until something else – in this case the anonymous callback from the Mogade GetLeaderBoard method – calls it’s Set method. This little trick I learned from Olaf Conijn.

The final part writes the image back to the stream. There are two important things to note here. First, I output the image as a JPEG of pretty low quality. There are three good reasons for that:

  1. The most important one: I use a ShellTileSchedule to retrieve the tiles, which simply refuses to download anything larger than 80 kb. When I output the tile as PNG, the result is over 120kb and nothing seems to happen. It took me some time to figure out why.
  2. You hardly see the difference on a small screen between 80kb PNG and 16kb JPEG
  3. Microsoft have tried very hard to make bandwidth and battery usage of Windows Phone 7 as economical as possible, so be user, battery and data plan friendly and keep this kind of repetitive network access as small as possible.
    finally
    {
      // Output as JPEG to stay below the 80 kb
      context.Response.ContentType = "image/jpg";
      using (var ms = new MemoryStream())
      {
        // Select JPEG encoder
        var imgCodec = ImageCodecInfo.GetImageEncoders().Where(
          codec => codec.FormatID.Equals(ImageFormat.Jpeg.Guid)).FirstOrDefault();
        var parameters = new EncoderParameters();
        parameters.Param[0] = new EncoderParameter(Encoder.ColorDepth, 8);

        // Save on memory stream first, then write bytes to Response
        // Neccesary to get this working on AppHarbor
        myBitmap.Save(ms, imgCodec, parameters);
        var bmpBytes = ms.GetBuffer();
        ms.Close();

        // Add content-length for good measure
        context.Response.AppendHeader("Content-Length", bmpBytes.Length.ToString());
        context.Response.BinaryWrite(bmpBytes);
        context.Response.Flush();
        context.Response.End();
      }
    }
  }
}

The second thing to note is that I use a little detour writing the resulting image to the Response via a MemoryStream. I could just as easy have written it directly to the context.Response.OutputStream – which I originally did - but I host the service on AppHarbor and for some reason this gives me a "A generic error occurred in GDI+" error message. Using the MemoryStream solves that problem.

The result looks something like this:livetile

 

 

 

 

 

For my Windows Phone 7 solution I use Mark Monster’s SmartShellTileSchedule – with a little adaption, i.e. I put a try/catch block in the UpdateTileBeforeOperation because that tended to crash on my phone – but not on my wife’s. I still don’t understand why. I initialize it like this:

var mogadeClient = MogadeClient.Initialize(gameId, gameKey);
mogadeClient.Update(null);
new SmartShellTileSchedule
{
  Interval = UpdateInterval.EveryHour,
  MaxUpdateCount = 0,
  RemoteImageUri = new Uri(
    string.Format(
      @"http://www.yourservernamehere.com/LiveTile.ashx?username={0}&anid={1}",
      userName, mogadeClient.GetUniqueIdentifier()), UriKind.Absolute),
  StartTime = DateTime.Now,
  Recurrence = UpdateRecurrence.Interval
};

gameId, gameKey and userName are all variables loaded from the settings ‘somewhere’ in your phone app.

I had a little doubt whether or not the App Hub test team would object to me sending the anonymous live id as part of the tile request but yesterday my live tile enabled version 1.2.0 of Catch'em birds was certified for the Marketplace, so apparently that's allowed. So now there's no reason not to add a Live Tile to your game as well. Make use of this Windows Phone 7 exclusive feature and get your game pinned to start screens all over the world!

Sample code can be downloaded here.

17 May 2011

Debugging Windows Phone 7 device network access with just a laptop

It’s a strange world. Today I needed to debug an iPad’s web traffic, and ended up getting instructions from the Dutch Windows Phone 7 DPE Matthijs Hoekstra. To dry run the solution I tried hooking up my own Windows Phone 7 an so was able to see Birdsong actually sending and downloading messages.

Now there are numerous sources around the internet describing what you should do to get this far but most only show pieces of the puzzle, so I decided to do a quick write-up on how to do this for Windows Phone 7.

Prerequisites

  • One laptop running Windows 7 with a WiFi card. I used a HP Probook 4720s
  • Connectify
  • Fiddler2
  • One Windows Phone 7 device (duh)

Make sure you laptop has internet connectivity, either by WiFi or by cable.

Install and configure Connectify

Download and install Connectify. Now installing this might be a little scary – it wants to install an unverified driver, and the first time I tried it the install got stuck, I had no network connectivity anymore, rebooted, uninstalled Connectify and then painlessly re-installed.

ConnectifyAnyway, it comes with a simple wizard, that asks you for a network name, as password, a connection to share. Connectify runs in the system tray and if you click on it, it pops up like this showed left.

You can see that my WiFi card now is used by the laptop itself to connect to the internet and as hotspot as well :-). But I could just as easily share a cabled internet connection (as I did this afternoon at my work). You can even see my hotspot’s password, so now you can steal my internet connection when you are in the neighborhood and I leave Connectify running (and then still it gives a bubble notification saying someone connected).

Your Windows Phone 7 can now already share your laptop’s WiFi connection – if you go to Settings/WiFi you can already see the network showing up. As you can see on the image my HTC 7 Pro has connected to it successfully

 

 

 

Install and configure Fiddler

Just install Fiddler the usual way, but then click Tools/Fiddler options and click tab “Connections”. Right under the box “Fiddler listens on port 8888” there’s a checkbox “Allow remote computers to connect”. Tick that one. close down Fiddler, start it again, and this will bring a Windows Firewall check box to your screen. Check all three checkboxes, and click “Allow Acces”.

IMG_8515Configure your Windows Phone 7

Pick up your phone, again, go to Settings/WiFi, connect to the Connectify network if you have not already done so, using the password you picked for it. As the phone is connected, tap your network, locate the “proxy” toggle, click it, use your laptops’ name as proxy and port 8888 as port, and you are done. Note – both my access point and my laptop are called “JSC7”, but they could be different. Always use your computers’ name for proxy.

 

 

 

 

 

Done

You can now spy on your Windows Phone 7’s network. I fired up Birdsong and saw it actually do somethingfiddler

Some final words

If you have an access point handy, you might skip the whole Connectify stuff, just define your PC as a proxy for that network in your Windows Phone 7 settings and only configure and fire up Fiddler on that PC. But I did not have a wireless AP I could connect my Windows Phone 7 to so I had to resort to Connectify.

Oh by the way, this works for a iPad as well :-)

20 April 2011

Safe event detachment ‘pattern’ for behaviors

For the past few weeks I’ve been hacking away at my Windows Phone 7 game and noticed something odd. The objective is to intercept moving objects on the screen and drag/flick them to a target – but as the levels progressed interception and movement became ever more difficult and slow. At least that was reported by my #wp7nl fellow members who were kind (or crazy) enough to test drive the game into the higher levels. I won’t mention names here – yet ;-).

I followed the usual pattern for a behavior – at least, I think it is the usual pattern

using System.Windows;
using System.Windows.Interactivity;

namespace SomeNamespace
{
  public class DemoBehavior : Behavior<FrameworkElement>
  {
    protected override void OnAttached()
    {
      base.OnAttached();
      AssociatedObject.Loaded += AssociatedObjectLoaded;
    }

    void AssociatedObjectLoaded(object sender, RoutedEventArgs e)
    {
      // Hook up a bunch of events to the AssociatedObject
    }

    protected override void OnDetaching()
    {
      AssociatedObject.Loaded -= AssociatedObjectLoaded;
      // Unhook the other events from the AssociatedObject
      base.OnDetaching();
    }
  }
}

I had seen this problem before while programming in WPF and indeed – when I put a breakpoint in the first line of the OnDetaching  override it was never called. Apparently Windows Phone 7 has the same problem. So here was my game, happily creating a lot of moving objects with each 1-2 behaviors attached to it listening to a bunch of events. They were never unhooked when the objects were deleted. So within a few levels the game started eating resources like there’s no tomorrow.

I modified my behaviors to try to do a general cleanup not only when OnDetaching is called but also when the AssociatedObject is unloaded. This gave me a pattern which I think might be beneficial to everyone who is working with behaviors in general:

using System.Windows;
using System.Windows.Interactivity;

namespace SomeNamespace
{
  public class BetterDemoBehavior : Behavior<FrameworkElement>
  {
    #region Setup
    protected override void OnAttached()
    {
      base.OnAttached();
      AssociatedObject.Loaded += AssociatedObjectLoaded;
      AssociatedObject.Unloaded += AssociatedObjectUnloaded;
    }

    void AssociatedObjectLoaded(object sender, RoutedEventArgs e)
    {
      // Hook up a bunch of events to the AssociatedObject
    }
    #endregion

    #region Cleanup
    private bool _isCleanedUp;

    private void Cleanup()
    {
      if (!_isCleanedUp)
      {
        _isCleanedUp = true;
        AssociatedObject.Loaded -= AssociatedObjectLoaded;
        AssociatedObject.Unloaded -= AssociatedObjectUnloaded;
        // Unhook the other events from the AssociatedObject
      }
    }

    protected override void OnDetaching()
    {
      Cleanup();
      base.OnDetaching();
    }

    void AssociatedObjectUnloaded(object sender, RoutedEventArgs e)
    {
      Cleanup();
    }
    #endregion
  }
}

And because I am not very partial to repetitive work, I created a snippet for this piece of code.

I guess that if both and Windows Phone 7 and WPF have this issue, Silverlight will have it as well. I hope at least this will help people who are, like me, stubborn enough to use Silverlight and MVVMLight to develop games for Windows Phone 7 ;-)

13 April 2011

Preventing objects made invisible by data binding to show up initially

Currently I am in the process of writing a small Windows Phone 7 game . It sports a ‘game over’ panel with a ‘play again’ button and it’s visibility is controlled by a boolean value ‘IsGameOver’ in the ViewModel. My VisibilityConverter converts the true and false to Visiblity.Visible and Visibility.Collapsed.

Although this works well, data binding apparently takes place only after the complete GUI is loaded, for when the game starts, you see the panel for about half a second before it disappears again. Ugh.

I did not actually solve this problem, but found a workaround in the following little behavior:

using System.Windows;
using System.Windows.Interactivity;

namespace Wp7nl.Behaviors
{
  public class SetInitialOpacityBehavior : Behavior<FrameworkElement>
  {
    protected override void OnAttached()
    {
      AssociatedObject.Loaded += AssociatedObjectLoaded;
      base.OnAttached();
      AssociatedObject.Opacity = InitialOpacity;
    }

    void AssociatedObjectLoaded(object sender, RoutedEventArgs e)
    {
      AssociatedObject.Loaded -= AssociatedObjectLoaded;
      AssociatedObject.Opacity = OpacityAfterLoading;
    }

    #region InitialOpacity
    public const string InitialOpacityPropertyName = "InitialOpacity";

    public int InitialOpacity
    {
      get { return (int)GetValue(InitialOpacityProperty); }
      set { SetValue(InitialOpacityProperty, value); }
    }

    public static readonly DependencyProperty InitialOpacityProperty = 
	  DependencyProperty.Register(
        InitialOpacityPropertyName,
        typeof(int),
        typeof(SetInitialOpacityBehavior),
        new PropertyMetadata(0));
    #endregion

    #region OpacityAfterLoading
    public const string OpacityAfterLoadingPropertyName = "OpacityAfterLoading";

    public int OpacityAfterLoading
    {
      get { return (int)GetValue(OpacityAfterLoadingProperty); }
      set { SetValue(OpacityAfterLoadingProperty, value); }
    }

    public static readonly DependencyProperty OpacityAfterLoadingProperty = 
	DependencyProperty.Register(
      OpacityAfterLoadingPropertyName,
      typeof(int),
      typeof(SetInitialOpacityBehavior),
      new PropertyMetadata(1));
    
    #endregion
  }
}

It basically sets the opacity to 0 when the behavior is attached to the FrameworkElement (which apparently takes place before the element is actually displayed), thereby making the element invisible. After the Loading event has fired, it sets the opacity to the value of the dependency property OpacityAfterLoading - which is default 1. Apparently this takes place after data binding. Whatever – my ‘game over’ panel does not show up at the start anymore, but only after the game ends. Like it should.

So if you notice an element showing up initially while it should not, just fire up Blend, drag this behavior on top of it and be done with it.

This behavior is by no means Windows Phone 7 specific – it would work perfectly in Silverlight and probably WPF too. Be aware, however, that although OpacityAfterLoading is a dependency property, in Windows Phone 7 you cannot data bind it, because it’s currently based on Silverlight 3 which only allow you to bind to classes that do descend from DependencyObject – which a behavior obviously does not ;-)

One final concluding note – although this behavior is in the Wp7nl namespace, it currently not in the Wp7nl library nor the NuGet package. But it will be in the next ;-)