kinect基本手势识别.docx

译Kinect for Windows SDK开发入门(十一)：手势识别 下：基本手势识别 上文简要介绍了手势识别的基本概念和手势识别的基本方法，并以八种手势中的挥手(wave)为例讲解了如何使用算法对手势进行识别，本文接上文，继续介绍如何建立一个手部追踪类库，并以此为基础，对剩余7中常用的手势进行识别做一些介绍。1. 基本的手势追踪 手部追踪在技术上和手势识别不同，但是它和手势识别中用到的一些基本方法是一样的。在开发一个具体的手势控件之前，我们先建立一个可重用的追踪手部运动的类库以方便我们后续开发。这个手部追踪类库包含一个以动态光标显示的可视化反馈机制。手部追踪和手势控件之间的交互高度松耦合。 首先在Visual Studio中创建一个WPF控件类库项目。然后添加四个类： KinectCursorEventArgs.cs，KinectInput.cs，CusrorAdorner.cs和KinectCursorManager.cs这四个类之间通过相互调用来基于用户手所在的位置来完成光标位置的管理。KinectInput类包含了一些事件，这些事件可以在KinectCursorManager和一些控件之间共享。KinectCursorEventArgs提供了一个属性集合，能够用来在事件触发者和监听者之间传递数据。KinectCursorManager用来管理从Kinect传感器中获取的骨骼数据流，然后将其转换到WPF坐标系统，提供关于转换到屏幕位置的可视化反馈，并寻找屏幕上的控件，将事件传递到这些控件上。最后CursorAdorner.cs类包含了代表手的图标的可视化元素。 KinectCursorEventArgs继承自RoutedEventArgs类，它包含四个属性：X、Y、Z和Cursor。X、Y、Z是一个小数，代表待转换的用户手所在位置的宽度，高度和深度值。Cursor用来存储CursorAdorner类的实例，后面将会讨论，下面的代码展示了KinectCursorEventArgs类的基本结构，其中包含了一些重载的构造器。public class KinectCursorEventArgs:RoutedEventArgs public double X get; set; public double Y get; set; public double Z get; set; public CursorAdorner Cursor get; set; public KinectCursorEventArgs(double x, double y) X = x; Y = y; public KinectCursorEventArgs(Point point) X = point.X; Y = point.Y; RoutedEventArgs基类有一个构造函数能够接收RoutedEvent作为参数。这是一个有点特别的签名，WPF中的UIElement使用这种特殊的语法触发事件。下面的代码是KinectCursorEventArgs类对这一签名的实现，以及其他一些重载方法。public KinectCursorEventArgs(RoutedEventroutedEvent) : base(routedEvent) publicKinectCursorEventArgs(RoutedEventroutedEvent, doublex, doubley, doublez) : base(routedEvent) X = x; Y = y; Z = z; publicKinectCursorEventArgs(RoutedEventroutedEvent, Pointpoint) : base(routedEvent) X = point.X; Y = point.Y; publicKinectCursorEventArgs(RoutedEventroutedEvent, Pointpoint,doublez) : base(routedEvent) X = point.X; Y = point.Y; Z = z; publicKinectCursorEventArgs(RoutedEventroutedEvent, objectsource) : base(routedEvent, source) publicKinectCursorEventArgs(RoutedEventroutedEvent,objectsource,doublex,doubley,doublez) : base(routedEvent, source) X = x; Y = y; Z = z; publicKinectCursorEventArgs(RoutedEventroutedEvent, objectsource, Pointpoint) : base(routedEvent, source) X = point.X; Y = point.Y; publicKinectCursorEventArgs(RoutedEventroutedEvent, objectsource, Pointpoint,doublez) : base(routedEvent, source) X = point.X; Y = point.Y; Z = z; 接下来，要在KinectInput类中创建事件来将消息从KinectCursorManager中传递到可视化控件中去。这些事件传递的数据类型为KinectCursorEventArgs类型。 在KinectInput类中添加一个KinectCursorEventHandler的代理类型：(1) 添加一个静态的routed event声明。(2) 添加KinectCursorEnter，KinectCursorLeave，KinectCursorMove，KinectCursorActive和KinectCursorDeactivated事件的add和remove方法。下面的代码展示了三个和cursor相关的事件，其他的如KinectCursorActivated和KinectCursorDeactivated事件和这个结构相同：public delegate void KinectCursorEventHandler(object sender,KinectCursorEventArgs e);public static class KinectInput public static readonly RoutedEvent KinectCursorEnterEvent=EventManager.RegisterRoutedEvent(KinectCursorEnter,RoutingStrategy.Bubble, typeof(KinectCursorEventHandler),typeof(KinectInput); public static void AddKinectCursorEnterHandler(DependencyObject o, KinectCursorEventHandler handler) (UIElement)o).AddHandler(KinectCursorEnterEvent, handler); public static void RemoveKinectCursorEnterHandler(DependencyObject o, KinectCursorEventHandler handler) (UIElement)o).RemoveHandler(KinectCursorEnterEvent, handler); public static readonly RoutedEvent KinectCursorLeaveEvent=EventManager.RegisterRoutedEvent(KinectCursorLeave,RoutingStrategy.Bubble, typeof(KinectCursorEventHandler),typeof(KinectInput); public static void AddKinectCursorLeaveHandler(DependencyObject o, KinectCursorEventHandler handler) (UIElement)o).AddHandler(KinectCursorEnterEvent,handler); public static void RemoveKinectCursorLeaveHandler(DependencyObject o, KinectCursorEventHandler handler) (UIElement)o).RemoveHandler(KinectCursorEnterEvent, handler); 注意到以上代码中没有声明任何GUI编程中的Click事件。这是因为在设计控件类库时，Kinect中并没有点击事件，相反Kinect中两个重要的行为是enter和leave。手势图标可能会移入和移出某一个可视化控件的有效区域。如果要实现普通GUI控件的点击效果的话，必须在Kinect中对这一事件进行模拟，因为Kinect原生并不支持点击这一行为。 CursorAdorner类用来保存用户手势图标可视化元素，它继承自WPF的Adorner类型。之所以使用这个类型是因为它有一个特点就是总是在其他元素之上绘制，这在我们的项目中非常有用，因为我们不希望我们的光标会被其他元素遮挡住。代码如下所示，我们默认的adorner对象将绘制一个默认的可视化元素来代表光标，当然也可以传递一个自定义的可视化元素。public class CursorAdorner:Adorner private readonly UIElement _adorningElement; private VisualCollection _visualChildren; private Canvas _cursorCanvas; protected FrameworkElement _cursor; StroyBoard _gradientStopAnimationStoryboard; readonly static Color _backColor = Colors.White; readonly static Color _foreColor = Colors.Gray; public CursorAdorner(FrameworkElement adorningElement) : base(adorningElement) this._adorningElement = adorningElement; CreateCursorAdorner(); this.IsHitTestVisible = false; public CursorAdorner(FrameworkElement adorningElement, FrameworkElement innerCursor) : base(adorningElement) this._adorningElement = adorningElement; CreateCursorAdorner(innerCursor); this.IsHitTestVisible = false; public FrameworkElement CursorVisual get return _cursor; public void CreateCursorAdorner() var innerCursor = CreateCursor(); CreateCursorAdorner(innerCursor); protected FrameworkElement CreateCursor() var brush = new LinearGradientBrush(); brush.EndPoint = new Point(0, 1); brush.StartPoint = new Point(0, 0); brush.GradientStops.Add(new GradientStop(_backColor, 1); brush.GradientStops.Add(new GradientStop(_foreColor, 1); var cursor = new Ellipse() Width=50, Height=50, Fill=brush ; return cursor; public void CreateCursorAdorner(FrameworkElement innerCursor) _visualChildren = new VisualCollection(this); _cursorCanvas = new Canvas(); _cursor = innerCursor; _cursorCanvas.Children.Add(this._cursorCanvas); _visualChildren.Add(this._cursorCanvas); AdornerLayer layer = AdornerLayer.GetAdornerLayer(_adorningElement); layer.Add(this); 因为继承自Adorner基类，我们需要重写某些基类的方法，下面的代码展示了基类中的方法如何和CreateCursorAdorner方法中实例化的_visualChildren和_cursorCanvas字段进行绑定。protected override int VisualChildrenCount get return _visualChildren.Count; protected override Visual GetVisualChild(int index) return _visualChildrenindex;protected override Size MeasureOverride(Size constraint) this._cursorCanvas.Measure(constraint); return this._cursorCanvas.DesiredSize;protected override Size ArrangeOverride(Size finalSize) this._cursorCanvas.Arrange(new Rect(finalSize); return finalSize; CursorAdorner对象也负责找到手所在的正确的位置，该对象的UpdateCursor方法如下，方法接受X，Y坐标位置作为参数。然后方法在X，Y上加一个偏移量以使得图像的中心在X，Y之上，而不是在图像的边上。另外，我们提供了该方法的一个重载，该重载告诉光标对象一个特殊的坐标会传进去，所有的普通方法调用UpdateCursor将会被忽略。当我们在磁性按钮中想忽略基本的手部追踪给用户更好的手势体验时很有用。public void UpdateCursor(Pointposition, boolisOverride) _isOverriden = isOverride; _cursor.SetValue(Canvas.LeftProperty,position.X-(_cursor.ActualWidth/2); _cursor.SetValue(Canvas.LeftProperty, position.Y - (_cursor.ActualHeight / 2); public void UpdateCursor(Pointposition) if(_isOverriden) return; _cursor.SetValue(Canvas.LeftProperty, position.X - (_cursor.ActualWidth / 2); _cursor.SetValue(Canvas.LeftProperty, position.Y - (_cursor.ActualHeight / 2); 最后，添加光标对象动画效果。当Kinect控件需要悬浮于一个元素之上，在用户等待的时候，给用户反馈一些信息告知正在发生的事情，这一点很有好处。下面了的代码展示了如何使用代码实现动画效果:public virtual void AnimateCursor(doublemilliSeconds) CreateGradientStopAnimation(milliSeconds); if(_gradientStopAnimationStoryboard != null) _gradientStopAnimationStoryboard.Begin(this, true); public virtual void StopCursorAnimation(doublemilliSeconds) if(_gradientStopAnimationStoryboard != null) _gradientStopAnimationStoryboard.Stop(this); public virtual void CreateGradientStopAnimation(doublemilliSeconds) NameScope.SetNameScope(this, newNameScope(); varcursor = _cursor asShape; if(cursor = null) return; varbrush = cursor.Fill asLinearGradientBrush; varstop1 = brush.GradientStops0; varstop2 = brush.GradientStops1; this.RegisterName(GradientStop1, stop1); this.RegisterName(GradientStop2, stop2); DoubleAnimationoffsetAnimation = newDoubleAnimation(); offsetAnimation.From = 1.0; offsetAnimation.To = 0.0; offsetAnimation.Duration = TimeSpan.FromMilliseconds(milliSeconds); Storyboard.SetTargetName(offsetAnimation, GradientStop1); Storyboard.SetTargetProperty(offsetAnimation, newPropertyPath(GradientStop.OffsetProperty); DoubleAnimationoffsetAnimation2 = newDoubleAnimation(); offsetAnimation2.From = 1.0; offsetAnimation2.To = 0.0; offsetAnimation2.Duration = TimeSpan.FromMilliseconds(milliSeconds); Storyboard.SetTargetName(offsetAnimation2, GradientStop2); Storyboard.SetTargetProperty(offsetAnimation2, newPropertyPath(GradientStop.OffsetProperty); _gradientStopAnimationStoryboard = newStoryboard(); _gradientStopAnimationStoryboard.Children.Add(offsetAnimation); _gradientStopAnimationStoryboard.Children.Add(offsetAnimation2); _gradientStopAnimationStoryboard.Completed += delegate _gradientStopAnimationStoryboard.Stop(this); ; 为了实现KinectCursorManager类,我们需要几个帮助方法,代码如下，GetElementAtScreenPoint方法告诉我们哪个WPF对象位于X,Y坐标下面，在这个高度松散的结构中，GetElementAtScreenPoint方法是主要的引擎，用来从KinectCurosrManager传递消息到自定义控件，并接受这些事件。另外，我们使用两个方法来确定我们想要追踪的骨骼数据以及我们想要追踪的手。private static UIElement GetElementAtScreenPoint(Point point, Window window) if (!window.IsVisible) return null; Point windowPoint = window.PointFromScreen(point); IInputElement element = window.InputHitTest(windowPoint); if (element is UIElement) return (UIElement)element; else return null;private static Skeleton GetPrimarySkeleton(IEnumerable skeletons) Skeleton primarySkeleton = null; foreach (Skeleton skeleton in skeletons) if (skeleton.TrackingState != SkeletonTrackingState.Tracked) continue; if (primarySkeleton = null) primarySkeleton = skeleton; else if (primarySkeleton.Position.Z skeleton.Position.Z) primarySkeleton = skeleton; return primarySkeleton;private static Joint? GetPrimaryHand(Skeleton skeleton) Joint leftHand=skeleton.JointsJointType.HandLeft; Joint rightHand=skeleton.JointsJointType.HandRight; if (rightHand.TrackingState = JointTrackingState.Tracked) if (leftHand.TrackingState != JointTrackingState.Tracked) return rightHand; else if (leftHand.Position.Z rightHand.Position.Z) return rightHand; else return leftHand; if (leftHand.TrackingState = JointTrackingState.Tracked) return leftHand; else return null; KinectCursorManager应该是一个单例类。这样设计是能够使得代码实例化起来简单。任何和KinectCursorManager工作的控件在KinectCursorManager没有实例化的情况下可以独立的进行KinectCursorManager的实例化。这意味着任何开发者使用这些控件不需要了解KinectCursorManager对象本身。相反，开发者能够简单的将控件拖动到应用程序中，控件负责实例化KinectCursorManager对象。为了使得这种自服务功能能和KinectCursorMange类一起使用，我们需要创建一个重载的Create方法来将应用程序的主窗体类传进来。下面的代码展示了重载的构造函数以及特殊的单例模式的实现方法。public class KinectCursorManager private KinectSensor kinectSensor; private CursorAdorner cursorAdorner; private readonly Window window; private UIElement lastElementOver; private bool isSkeletonTrackingActivated; private static bool isInitialized; private static KinectCursorManager instance; public static void Create(Window window) if (!isInitialized) instance = new KinectCursorManager(window); isInitialized = true; public static void Create(Window window,FrameworkElement cursor) if (!isInitialized) instance = new KinectCursorManager(window,cursor); isInitialized = true; public static void Create(Window window, KinectSensor sensor) if (!isInitialized) instance = new KinectCursorManager(window, sensor); isInitialized = true; public static void Create(Window window, KinectSensor sensor, FrameworkElement cursor) if (!isInitialized) instance = new KinectCursorManager(window, sensor, cursor); isInitialized = true; public static KinectCursorManager Instance get return instance; private KinectCursorManager(Window window) : this(window, KinectSensor.KinectSensors0) private KinectCursorManager(Window window, FrameworkElement cursor) : this(window, KinectSensor.KinectSensors0, cursor) private KinectCursorManager(Window window, KinectSensor sensor) : this(window, sensor, null) private KinectCursorManager(Window window, KinectSensor sensor, FrameworkElement cursor) this.window = window; if (KinectSensor.KinectSensors.Count 0) window.Unloaded += delegate if (this.kinectSensor.SkeletonStream.IsEnabled) this.kinectSensor.SkeletonStream.Disable(); ; window.Loaded += delegate if (cursor = null) cursorAdorner = new CursorAdorner(FrameworkElement)window.Content); else cursorAdorner = new CursorAdorner(FrameworkElement)window.Content, cursor); this.kinectSensor = sensor; this.kinectSensor.SkeletonFrameReady += SkeletonFrameReady; this.kinectSensor.SkeletonStream.Enable(new TransformSmoothParameters(); this.kinectSensor.Start(); ; 下面的代码展示了KinectCursorManager如何和窗体上的可视化元素进行交互。当用户的手位于应用程序可视化元素之上时，KinectCursorManager对象始终保持对当前手所在的可视化元素以及之前手所在的可视化元素的追踪。当这一点发生改变时，KinectCursorManager会触发之前控件的leave事件和当前控件的enter事件。我们也保持对KinectSensor对象的追踪，并触发activated和deactivated事件。private void SetSkeletonTrackingActivated() if (lastElementOver != null & isSkeletonTrackingActivated = false) lastElementOver.RaiseEvent(new RoutedEventArgs(KinectInput.KinectCursorActivatedEvent); isSkeletonTrackingActivated = true;private void SetSkeletonTrackingDeactivated() if (lastElementOver != null & isSkeletonTrackingActivated = false) lastElementOver.RaiseEvent(new RoutedEventArgs(KinectInput.KinectCursorDeactivatedEvent); isSkeletonTrackingActivated = false ;private void HandleCursorEvents(Point point, double z) UIElement element = GetElementAtScreenPoint(point, window); if (element != null) element.RaiseEvent(new KinectCursorEventArgs(KinectInput.KinectCursorMoveEvent, point, z) Cursor=cursorAdorner ); if (element != lastElementOver) if (lastElementOver != null) lastElementOver.RaiseEvent(new KinectCursorEventArgs(KinectInput.KinectCursorLeaveEvent, point, z) Cursor = cursorAdorner ); element.RaiseEvent(new KinectCursorEventArgs(KinectInput.KinectCursorEnterEvent, point, z) Cursor = cursorAdorner ); lastElementOver = element; 最后需要两个核心的方法来管理KinectCursorManger类。SkeletonFrameReady方法与之前一样，用来从Kinect获取骨骼数据帧时触发的事件。在这个项目中，SkeletonFrameReady方法负责获取合适的骨骼数据，然后获取合适的手部关节点数据。然后将手部关节点数据传到UpdateCusror方法中，UpdateCursor方法执行一系列方法将Kinect骨骼空间坐标系转化到WPF的坐标系统中，Kinect SDK中MapSkeletonPointToDepth方法提供了这一功能。SkeletonToDepthImage方法返回的X,Y值，然后转换到应用程序中实际的宽和高。和X,Y不一样，Z值进行了不同的缩放操作。简单的从Kinect深度摄像机中获取的毫米数据。代码如下，一旦这些坐标系定义好了之后，将他们传递到HandleCursorEvents方法然后CursorAdorner对象将会给用户以反馈。相关代码如下：private void SkeletonFrameReady(objectsender, SkeletonFrameReadyEventArgse) using(SkeletonFrameframe = e.OpenSkeletonFrame() if(frame = null| frame.SkeletonArrayLength = 0) return; Skeleton skeletons = newSkeletonframe.SkeletonArrayLength; frame.CopySkeletonDataTo(skeletons); Skeletonskeleton = GetPrimarySkeleton(skeletons); if(skeleton = null) SetHandTrackingDeactivated(); else Joint? primaryHand = GetPrimaryHand(skeleton); if(primaryHand.HasValue) UpdateCursor(primaryHand.Value); else SetHandTrackingDeactivated(); private voidSetHandTrackingDeactivated() cursorAdorner.SetVisibility(false); if(lastElementOver != null& isHandTrackingActivated = true) lastElementOver.RaiseEvent(newRoutedEventArgs(KinectInput.KinectCursorDeactivatedEvent); ; isHandTrackingActivated = false; private voidUpdateCursor(Jointhand) varpoint = kinectSensor.MapSkeletonPointToDepth(hand.Position, kinectSensor.DepthSt