姿势和手势通常会混淆，但是他们是两个不同的概念。当一个人摆一个姿势时，他会保持身体的位置和样子一段时间。但是手势包含有动作，例如用户通过手势在触摸屏上，放大图片等操作。
身体以及各个关节点的位置定义了一个姿势。更具体的来说，是某些关节点相对于其他关节点的位置定义了一个姿势。姿势的类型和复杂度决定了识别算法的复杂度。通过关节点位置的交叉或者关节点之间的角度都可以进行姿势识别。
节点交叉并不需要使用X,Y的所有信息。一些姿势只需要使用一个坐标轴信息。例如：立正姿势，在这个姿势中，手臂和肩膀近乎在一个垂直坐标轴内而不用考虑用户的身体的大小和形状。在这个姿势中，逻辑上只需要测试手和肩部节点的X坐标的差值，如果在一个阈值内就可以判断这些关节点在一个平面内。但是这并不能保证用户是立正姿势。应用程序还需要判断手在Y坐标轴上应该低于肩部。
并不是所有的姿势识别都适合使用节点交叉法，一些姿势使用其他方法识别精度会更高。例如，用户伸开双臂和肩膀在一条线上这个姿势，称之为T姿势。可以使用节点相交技术，判断手、肘、以及肩膀是否在Y轴上处于近乎相同的位置。另一种方法是计算某些关节点连线之间的角度。骨骼追踪引擎能够识别多达20个关节点数据。任何三个关节点就可以组成一个三角形。使用三角几何就可以计算出他们之间的角度。
响应识别到的姿势
识别姿势的目的是触发一些操作。最简单的方法是当探测到某一姿势后立即响应一些类似鼠标点击之类的事件。
应用程序要使用姿势识别必须知道什么时候该忽略什么时候该响应特定的姿势。如前所述，最简单的方法是当识别到某一姿势时立即响应。如果这是应用程序的功能，需要选择一个用户不可能会在休息或者放松时会产生的姿势。选择一个姿势很容易，但是这个姿势不能是户自然而然或者大多数情况下都会产生的姿势。这意味着姿势必须是有意识的，就像是鼠标点击那样，用户需要进行某项操作才会去做某种特定的姿势。除了马上响应识别到的某个姿势外，另一种方法是触发一个计时器。只有用户保持这一姿势一段时间，应用程序才会触发相应的操作。
另一种方法是当用户摆出某一系列的姿势时才触发某一动作。这需要用户按照特定的序列摆出一些列的姿势，才会执行某一操作。使用系列姿势和一些不常用的姿势可以使得应用程序知道用户有意想进行某一项操作，而不是误操作。换句话说，这能够帮助用户减少误操作。
Simon Says 游戏中使用姿势识别
Simon指令时让用户按照顺序做一系列的姿势，而不是触摸那四个矩形。使用关节点角度进行姿势识别可以给予应用程序更多的姿势选择。
为了让游戏好玩，需要尽可能多的选择可识别的姿势。另外，还要能比较容易的将新的姿势添加进来。为了创建一个姿势库，需要创建一个新的PoseAngle类和名为Pose的结构。如下面的代码所示。Pose存储了一个姿势的名称和一个PoseAngle数组。PoseAngle的有两个JointType类型的成员变量用来计算角度，Angle为期望角度，Threshold 阈值。
开始姿势和姿势库定义好了之后，下面来开始改写游戏的逻辑代码。当游戏GameOver时，会调用ProcessGameOver方法。在前篇文章中，这个方法用来判断用户的双手是否在指定的对象上，现在替换为识别用户的姿势是否是指定的姿势。如下代码展示了如何处理游戏开始和姿势识别，IsPose方法判断是否和指定的姿势匹配，这个方法在多个地方都可能会用到。IsPost方法遍历一个姿势中的所有PoseAngle，如果任何一个关节点角度和定义的不一致，方法就返回false，表示不是指定的姿势。方法中的if语句用来判断角度是否在360度范围内，如果不在，则转换到该范围内。
IsPost方法调用GetJointAngle方法来计算两个关节点之间的角度。GetJointAngle调用GetJointPoint方法来获取每一个节点在主UI布局空间中的坐标。这一步其实没有太大必要，原始的位置信息也可以用来计算角度。但是，将关节点的坐标转换到主UI界面上来能够帮助我们进行调试。获得了节点的位置后，使用余弦定理计算节点间的角度。Math.Acos返回的值是度，将其转换到角度值。If语句处理角度值在180-360的情况。余弦定理返回的角度在0-180度内，if语句将在第三和第四象限的值调整到第一第二象限中来。
程序还必须识别姿势并启动程序。当程序识别到启动的姿势是，将游戏的状态切换到SimonInstructing。这部分代码和GenerateInstructions及DisplayInstructions是分开的。将GenerateInstructions产生的指令改为随机的从姿势库中选取某一个姿势。然后使用选择的姿势填充指令集合。DisplayInstructions方法可以使用自己的方法比如图片来给用户以提示。一旦游戏显示完指令，游戏转入PlayerPerforming阶段。这个阶段给了游戏者一定的时间来摆出特定的姿势，当程序识别到需要的姿势时，转到下一个姿势，并重启计时器。如果超过给定时间仍然没有给出指定的姿势，游戏结束。WPF中System.Windows.Threading命名空间下的DispatcherTimer类可以简单的完成计时器的功能。
提升
namespace SimonSayAction
{
public enum GamePhase
{
GameOver = ,
SimonInstructing = ,
PlayerPerforming =
} public partial class MainWindow : Window
{
#region Member Variables
private KinectSensor kinectDevice;
private Skeleton[] frameSkeletons;
private GamePhase currentPhase;
private int[] instructionSequence;
private int instructionPosition;
private int currentLevel;
private Random rnd = new Random();
private Pose[] poseLibrary;
private Pose startPose;
private DispatcherTimer poseTimer;
#endregion Member Variables #region Constructor
public MainWindow()
{
InitializeComponent(); this.currentLevel = ; this.poseTimer = new DispatcherTimer();
this.poseTimer.Interval = TimeSpan.FromSeconds();
// 返回表示指定秒数的 System.TimeSpan，其中对秒数的指定精确到最接近的毫秒
this.poseTimer.Tick += (s, e) => { ChangePhase(GamePhase.GameOver); };
this.poseTimer.Stop(); PopulatePoseLibrary(); // 初始化游戏姿势库
ChangePhase(GamePhase.GameOver); KinectSensor.KinectSensors.StatusChanged += KinectSensors_StatusChanged;
this.KinectDevice = KinectSensor.KinectSensors.FirstOrDefault(x => x.Status == KinectStatus.Connected);
}
#endregion Constructor #region Methods
private void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
{
switch (e.Status)
{
case KinectStatus.Initializing:
case KinectStatus.Connected:
case KinectStatus.NotPowered:
case KinectStatus.NotReady:
case KinectStatus.DeviceNotGenuine:
this.KinectDevice = e.Sensor;
break;
case KinectStatus.Disconnected:
//TODO: Give the user feedback to plug-in a Kinect device.
this.KinectDevice = null;
break;
default:
//TODO: Show an error state
break;
}
} private void KinectDevice_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
using (SkeletonFrame frame = e.OpenSkeletonFrame())
{
if (frame != null)
{
frame.CopySkeletonDataTo(this.frameSkeletons);
Skeleton skeleton = GetPrimarySkeleton(this.frameSkeletons); // 获取最近骨架 if (skeleton == null)
{
ChangePhase(GamePhase.GameOver);
}
else
{
if (this.currentPhase == GamePhase.SimonInstructing)
{
LeftHandElement.Visibility = System.Windows.Visibility.Collapsed;
RightHandElement.Visibility = System.Windows.Visibility.Collapsed;
}
else
{
TrackHand(skeleton.Joints[JointType.HandLeft], LeftHandElement, LayoutRoot);
TrackHand(skeleton.Joints[JointType.HandRight], RightHandElement, LayoutRoot); switch (this.currentPhase)
{
case GamePhase.GameOver:
ProcessGameOver(skeleton);
break; case GamePhase.PlayerPerforming:
ProcessPlayerPerforming(skeleton);
break;
}
}
}
}
}
} private void TrackHand(Joint hand, FrameworkElement cursorElement, FrameworkElement container)
{
if (hand.TrackingState == JointTrackingState.NotTracked)
{
cursorElement.Visibility = Visibility.Collapsed;
}
else
{
cursorElement.Visibility = Visibility.Visible;
Point jointPoint = GetJointPoint(this.KinectDevice, hand, container.RenderSize, new Point(cursorElement.ActualWidth / 2.0, cursorElement.ActualHeight / 2.0));
Canvas.SetLeft(cursorElement, jointPoint.X);
Canvas.SetTop(cursorElement, jointPoint.Y);
}
} private void ProcessGameOver(Skeleton skeleton)
{
if (IsPose(skeleton, this.startPose)) // 是否是开始姿势
{
ChangePhase(GamePhase.SimonInstructing);
}
} private static Point GetJointPoint(KinectSensor kinectDevice, Joint joint, Size containerSize, Point offset)
{
// 虽然骨骼点在同一坐标空间，但是通过转换成UI相关，可以方便调试
DepthImagePoint point = kinectDevice.MapSkeletonPointToDepth(joint.Position, kinectDevice.DepthStream.Format);
point.X = (int)((point.X * containerSize.Width / kinectDevice.DepthStream.FrameWidth) - offset.X);
point.Y = (int)((point.Y * containerSize.Height / kinectDevice.DepthStream.FrameHeight) - offset.Y); return new Point(point.X, point.Y);
} private double GetJointAngle(Joint centerJoint, Joint angleJoint)
{
// 用余弦定理来求角度
Point primaryPoint = GetJointPoint(this.KinectDevice, centerJoint, this.LayoutRoot.RenderSize, new Point());
Point anglePoint = GetJointPoint(this.KinectDevice, angleJoint, this.LayoutRoot.RenderSize, new Point());
Point x = new Point(primaryPoint.X + anglePoint.X, primaryPoint.Y); double a;
double b;
double c; a = Math.Sqrt(Math.Pow(primaryPoint.X - anglePoint.X, ) + Math.Pow(primaryPoint.Y - anglePoint.Y, ));
b = anglePoint.X;
c = Math.Sqrt(Math.Pow(anglePoint.X - x.X, ) + Math.Pow(anglePoint.Y - x.Y, )); double angleRad = Math.Acos((a * a + b * b - c * c) / ( * a * b));
double angleDeg = angleRad * / Math.PI; if (primaryPoint.Y < anglePoint.Y)
{
angleDeg = - angleDeg;
} return angleDeg;
} private void PopulatePoseLibrary() // 游戏姿势库
{
this.poseLibrary = new Pose[]; //游戏开始 Pose - 伸开双臂 Arms Extended
// 肩，轴，髋
this.startPose = new Pose();
this.startPose.Title = "Start Pose";
this.startPose.Angles = new PoseAngle[];
this.startPose.Angles[] = new PoseAngle(JointType.ShoulderLeft, JointType.ElbowLeft, , ); // 角度，阈值
this.startPose.Angles[] = new PoseAngle(JointType.ElbowLeft, JointType.WristLeft, , );
this.startPose.Angles[] = new PoseAngle(JointType.ShoulderRight, JointType.ElbowRight, , );
this.startPose.Angles[] = new PoseAngle(JointType.ElbowRight, JointType.WristRight, , ); //Pose 1 -举起手来 Both Hands Up
// Wrist 部位比较固定，可以作为基准
this.poseLibrary[] = new Pose();
this.poseLibrary[].Title = "举起手来(Arms Up)";
this.poseLibrary[].Angles = new PoseAngle[];
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderLeft, JointType.ElbowLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowLeft, JointType.WristLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderRight, JointType.ElbowRight, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowRight, JointType.WristRight, , ); //Pose 2 - 把手放下来 Both Hands Down
this.poseLibrary[] = new Pose();
this.poseLibrary[].Title = "把手放下来（Arms Down）";
this.poseLibrary[].Angles = new PoseAngle[];
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderLeft, JointType.ElbowLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowLeft, JointType.WristLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderRight, JointType.ElbowRight, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowRight, JointType.WristRight, , ); //Pose 3 - 举起左手 Left Up and Right Down
this.poseLibrary[] = new Pose();
this.poseLibrary[].Title = "（举起左手）Left Up and Right Down";
this.poseLibrary[].Angles = new PoseAngle[];
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderLeft, JointType.ElbowLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowLeft, JointType.WristLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderRight, JointType.ElbowRight, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowRight, JointType.WristRight, , ); //Pose 4 - 举起右手 Right Up and Left Down
this.poseLibrary[] = new Pose();
this.poseLibrary[].Title = "（举起右手）Right Up and Left Down";
this.poseLibrary[].Angles = new PoseAngle[];
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderLeft, JointType.ElbowLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowLeft, JointType.WristLeft, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ShoulderRight, JointType.ElbowRight, , );
this.poseLibrary[].Angles[] = new PoseAngle(JointType.ElbowRight, JointType.WristRight, , );
} private bool IsPose(Skeleton skeleton, Pose pose)
{
// 判断一个骨架中是否包含指定姿势 —— 注意传入的参数PoseAngle(JointType centerJoint, JointType angleJoint, double angle, double threshold)
bool isPose = true;
double angle;
double poseAngle;
double poseThreshold;
double loAngle;
double hiAngle; for (int i = ; i < pose.Angles.Length && isPose; i++)
{
poseAngle = pose.Angles[i].Angle;
poseThreshold = pose.Angles[i].Threshold;
angle = GetJointAngle(skeleton.Joints[pose.Angles[i].CenterJoint], skeleton.Joints[pose.Angles[i].AngleJoint]); hiAngle = poseAngle + poseThreshold;
loAngle = poseAngle - poseThreshold; if (hiAngle >= || loAngle < )
{
// 如果角度出现大于180度，需要调整
loAngle = (loAngle < ) ? + loAngle : loAngle;
hiAngle = hiAngle % ; isPose = !(loAngle > angle && angle > hiAngle);
}
else
{
isPose = (loAngle <= angle && hiAngle >= angle);
}
} return isPose;
} private void ProcessPlayerPerforming(Skeleton skeleton)
{
int instructionSeq = this.instructionSequence[this.instructionPosition]; if (IsPose(skeleton, this.poseLibrary[instructionSeq])) // 姿势是否符合要求
{
this.poseTimer.Stop();
this.instructionPosition++; if (this.instructionPosition >= this.instructionSequence.Length)
{
ChangePhase(GamePhase.SimonInstructing);
}
else
{
//TODO: Notify the user of correct pose
this.poseTimer.Start();
}
}
} private void ChangePhase(GamePhase newPhase)
{
if (newPhase != this.currentPhase)
{
this.currentPhase = newPhase;
this.poseTimer.Stop(); // 计时器重置 switch (this.currentPhase)
{
case GamePhase.GameOver:
this.currentLevel = ;
GameStateElement.Text = "GAME OVER!";
GameInstructionsElement.Text = "Place hands over the targets to start a new game.";
break; case GamePhase.SimonInstructing:
this.currentLevel++;
GameStateElement.Text = string.Format("Level {0}", this.currentLevel);
GameInstructionsElement.Text = "Watch for Simon's instructions";
GenerateInstructions(); // 产生一级的游戏
DisplayInstructions(); // 显示
break; case GamePhase.PlayerPerforming:
this.poseTimer.Start();
this.instructionPosition = ;
break;
}
}
} private void GenerateInstructions()
{
this.instructionSequence = new int[this.currentLevel]; for (int i = ; i < this.currentLevel; i++)
{
this.instructionSequence[i] = rnd.Next(, this.poseLibrary.Length - );
}
} private void DisplayInstructions()
{
GameInstructionsElement.Text = string.Empty;
StringBuilder text = new StringBuilder();
int instructionsSeq; for (int i = ; i < this.instructionSequence.Length; i++)
{
instructionsSeq = this.instructionSequence[i];
text.AppendFormat("{0}, ", this.poseLibrary[instructionsSeq].Title);
} GameInstructionsElement.Text = text.ToString();
ChangePhase(GamePhase.PlayerPerforming);
} private static Skeleton GetPrimarySkeleton(Skeleton[] skeletons)
{
Skeleton skeleton = null; if (skeletons != null)
{
//Find the closest skeleton
for (int i = ; i < skeletons.Length; i++)
{
if (skeletons[i].TrackingState == SkeletonTrackingState.Tracked)
{
if (skeleton == null)
{
skeleton = skeletons[i];
}
else
{
if (skeleton.Position.Z > skeletons[i].Position.Z)
{
skeleton = skeletons[i];
}
}
}
}
} return skeleton;
}
#endregion Methods #region Properties
public KinectSensor KinectDevice
{
get { return this.kinectDevice; }
set
{
if (this.kinectDevice != value)
{
//Uninitialize
if (this.kinectDevice != null)
{
this.kinectDevice.Stop();
this.kinectDevice.SkeletonFrameReady -= KinectDevice_SkeletonFrameReady;
this.kinectDevice.SkeletonStream.Disable();
SkeletonViewerElement.KinectDevice = null;
this.frameSkeletons = null;
} this.kinectDevice = value; //Initialize
if (this.kinectDevice != null)
{
if (this.kinectDevice.Status == KinectStatus.Connected)
{
this.kinectDevice.SkeletonStream.Enable();
this.frameSkeletons = new Skeleton[this.kinectDevice.SkeletonStream.FrameSkeletonArrayLength];
this.kinectDevice.Start(); SkeletonViewerElement.KinectDevice = this.KinectDevice;
this.KinectDevice.SkeletonFrameReady += KinectDevice_SkeletonFrameReady;
}
}
}
}
}
#endregion Properties
}
}