Skeletal Tracking Fundamentals

This video covers the basics of skeletal tracking using the Kinect sensor.  You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides.

• [00:31] Skeleton Tracking API
• [01:24] Understanding Skeleton Quality and Joint data
• [03:27] Setup skeleton tracking
• [03:44] Adding a basic hand tracked cursor
• [09:12] Using TransformSmoothing to remove ?skeletal jitter?

Setup

The steps below assume you have setup your development environment as explained in the "Setting Up Your Development Environment" video.

Task: Setup skeleton tracking

Create the Window_Loaded event

Go to the properties window (F4), select the MainWindow, select the Events tab, and double click on the Loaded event to create the Window_Loaded event

 

Initializing the runtime

Create a new variable outside of the Window_Loaded event to reference the Kinect runtime.

C#

Runtime nui = new Runtime();

Visual Basic

Dim nui As New Runtime()

In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseSkeletalTracking to receive skeletal data and register for the SkeletonFrameReady event.

C#

nui.Initialize(RuntimeOptions.UseSkeletalTracking);  nui.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);  void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) { }

Visual Basic

nui.Initialize(RuntimeOptions.UseSkeletalTracking)  AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReady  Private Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs)  End Sub

Running the application

Add a breakpoint inside the SkeletonFrameReady event and run the application.

Note ? You will need to stand far enough away that the Kinect can see all or most of your skeleton for the SkeletonFrameReady event to fire.

 

When the breakpoint fires, a skeleton position is now being tracked. You can inspect the SkeletonFrameReadyEventArgs to see that the SkeletonFrame returns a collection of six skeletons as shown below.

 

Task: Add a basic hand tracked cursor

In this example, we're going to use the position of the tracked skeleton's, head, left hand, and right hand to move ellipse controls.

Designing your UI

Starting from the project above, switch to MainWindow.xaml and make sure you can see the XAML code. We will add three three ellipses of varying color onto a Canvas control named MainCanvas as shown in the XAML below.

XAML

<Canvas Name="MainCanvas">     <Ellipse Canvas.Left="0" Canvas.Top="0" Height="50" Name="headEllipse" Stroke="Black" Width="50" Fill="Orange" />     <Ellipse Canvas.Left="50" Canvas.Top="0" Height="50" Name="rightEllipse" Stroke="Black" Width="50" Fill="SlateGray" />     <Ellipse Canvas.Left="100" Canvas.Top="0" Fill="SpringGreen" Height="50" Name="leftEllipse" Stroke="Black" Width="50" /> </Canvas>

Get the Tracked Skeleton

In the SkeletonFrameReady event, we'll use a LINQ query to get the first tracked skeleton.

C#

 void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) {     SkeletonFrame allSkeletons = e.SkeletonFrame;      //get the first tracked skeleton     SkeletonData skeleton = (from s in allSkeletons.Skeletons                              where s.TrackingState == SkeletonTrackingState.Tracked                              select s).FirstOrDefault(); }

Visual Basic

Private Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs)     Dim allSkeletons As SkeletonFrame = e.SkeletonFrame      'get the first tracked skeleton     Dim skeleton As SkeletonData = ( _         From s In allSkeletons.Skeletons _         Where s.TrackingState = SkeletonTrackingState.Tracked _         Select s).FirstOrDefault() End Sub

Getting a Joint position

A Joint position returns X,Y,Z values as explained below

• X = Horizontal position between ?1 and +1
• Y = Vertical position between ?1 and +1
• Z = Distance from Kinect measured in meters

Scaling a Joint value

Given that the X and Y positions are between ?1 and +1, we can use the Coding4Fun Kinect Toolkit ScaleTo method to scale the value to a maximum X and Y position as shown below.

C#

Joint HandRight =  skeleton.Joints[JointID.HandRight].ScaleTo(640, 480);

Visual Basic

Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480)

You can also constrain the maximum values for the skeletal joints to a range smaller than -1 to +1.  For example, if you were designing an application where the user can reach around the screen to touch things, you may not want them to have to step left or right to reach the edges.  By using the second version of the ScaleTo method, you can specify the range of the specified joint to someting smaller as shown:

C#

Joint HandRight =  skeleton.Joints[JointID.HandRight].ScaleTo(640, 480, .5f, .5f);

Visual Basic

Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480, .5f, .5f)

Essentially, this code will make the joint range of -0.5 to +0.5 map to the pixels at 0 to 640.

Also, as you can see above, you can get a particular Joint, like the HandRight Joint, by using the skeleton indexer for the Joints collection. 

Setting an Ellipse Position

To move the ellipses in our MainWindow  to the location of a Joint, we will use the method below that sets the Canvas.Left and Canvas.Top position to the X (Left) and Y (Top) value from the Joint parameter.

C#

private void SetEllipsePosition(FrameworkElement ellipse, Joint joint) {     var scaledJoint = joint.ScaleTo(640, 480, .5f, .5f);      Canvas.SetLeft(ellipse, scaledJoint.Position.X);     Canvas.SetTop(ellipse, scaledJoint.Position.Y); }

Visual Basic

Private Sub SetEllipsePosition(ByVal ellipse As FrameworkElement, ByVal joint As Joint)     Dim scaledJoint = joint.ScaleTo(640, 480,.5f,.5f)      Canvas.SetLeft(ellipse, scaledJoint.Position.X)     Canvas.SetTop(ellipse, scaledJoint.Position.Y) End Sub

 

Putting it all together

Finally, in the nui_SkeletonFrameReady event, call the SetEllipsePosition methods for the three joints as shown below:

C#

SetEllipsePosition(headEllipse, skeleton.Joints[JointID.Head]); SetEllipsePosition(leftEllipse, skeleton.Joints[JointID.HandLeft]); SetEllipsePosition(rightEllipse, skeleton.Joints[JointID.HandRight]);

Visual Basic

SetEllipsePosition(headEllipse, skeleton.Joints(JointID.Head)) SetEllipsePosition(leftEllipse, skeleton.Joints(JointID.HandLeft)) SetEllipsePosition(rightEllipse, skeleton.Joints(JointID.HandRight))

 

 

Task: Using TransformSmoothing to for less skeletal jitter

Using the same application as above, you may notice jitteriness in the hand positions as small changes between updates to the SkeletalFrameReady event change the location of the ellipses.

Using TransformSmoothParameters

To do this, set the TransformSmooth property of the SkeletonEngine to true.  This will use a default set of smoothing parameters to reduce jitter in the skeletal tracking engine.  If these defaults don't provide you with the result you're looking for, then you can build the set of TransformSmoothParameters yourself to meet the needs for your application.

You must set the TransformSmoothParameters after calling the nui.Initialize method.

Note: Since every application is different, you will need to experiment with the parameters to understand what's right for your application.

C#

private void Window_Loaded(object sender, RoutedEventArgs e) {                //Initialize to do skeletal tracking     nui.Initialize(RuntimeOptions.UseSkeletalTracking);      //Must set to true and set after call to Initialize     nui.SkeletonEngine.TransformSmooth = true;      //Use to transform and reduce jitter     var parameters = new TransformSmoothParameters     {         Smoothing = 0.75f,         Correction = 0.0f,         Prediction = 0.0f,         JitterRadius = 0.05f,         MaxDeviationRadius = 0.04f     };      nui.SkeletonEngine.SmoothParameters = parameters;

Visual Basic

Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs)      'Initialize to do skeletal tracking     nui.Initialize(RuntimeOptions.UseSkeletalTracking)      'Must set to true and set after call to Initialize     nui.SkeletonEngine.TransformSmooth = True      'Use to transform and reduce jitter     Dim parameters = New TransformSmoothParameters With {.Smoothing = 0.75f, .Correction = 0.0f, .Prediction = 0.0f, .JitterRadius = 0.05f, .MaxDeviationRadius = 0.04f}      nui.SkeletonEngine.SmoothParameters = parameters      'add event to receive skeleton data     AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReady  End Sub

windows 7 computer speakers running shoes nikon d5000 droid x