×
Namespaces

Variants
Actions
Revision as of 03:42, 5 September 2013 by hamishwillee (Talk | contribs)

Detecting Faces on Windows Phone

From Nokia Developer Wiki
Jump to: navigation, search

This article explains how to detect faces from images using the FaceDetectionWP8 library.

WP Metro Icon Multimedia.png
SignpostIcon XAML 40.png
WP Metro Icon WP8.png
SignpostIcon WP7 70px.png
Article Metadata
Code Example
Source file: FaceDetectionWP8 Example (github)
Tested with
Devices(s): Lumia 820
CompatibilityArticle
Created: summeli (09 Aug 2013)
Last edited: hamishwillee (05 Sep 2013)

Contents

Introduction

This article gives an example how to detect faces on Windows Phone platform using the FaceDetectionWP8 library. FaceDetectionWP8 is a port to Windows Phone 8 of Face Detection For Windows Phone 7 by Julia Schwarz, which in turn is largely derived from the Java-based library at jviolajones.

The library uses the same algorithms as the OpenCV Face Detection library, and hence should also be useful for detecting forms other than faces. It is licensed under the New BSD License so you can use it even on commercial projects.

The code described here is FaceDetectionWP8s example application. This app allows you to select a picture from the Camera roll, identify faces, and superimpose a rectangle over them.

At the end of the article we also briefly review RealtimeExample (the original WP7 port example code). This is more advanced and requires some image processing tricks before using the FaceDetection library. We first look at the basic example, since it's easier to understand.

Face detection example app

Creating an example app for WP8

The basic layout of the example app is shown above in the introduction

Let's start the example application with the XAML below:

 <Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0" RenderTransformOrigin="0.393,0.461">
<Button Name="btnPhotoChoose" Click ="btnPhotoChoose_Clicked" Margin="67,41,60,475">Choose Photo</Button>
<Image Name="facesPic" HorizontalAlignment="Left" Height="388" Margin="19,170,0,0" VerticalAlignment="Top" Width="427"/>
</Grid>

Pressing the button will launch PhotoChooser from the gallery, so let's implement that.

private void btnPhotoChoose_Clicked(object sender, RoutedEventArgs e)
{
PhotoChooserTask photo = new PhotoChooserTask();
photo.Completed += new EventHandler<PhotoResult>(photoChooserTask_Completed);
photo.ShowCamera = true;
photo.Show();
}

After the photo is chosen we can display the image in the facesPic bitmap in the current layout:

void photoChooserTask_Completed(object sender, PhotoResult e)
{
if (e.TaskResult == TaskResult.OK)
{
BitmapImage bmp = new BitmapImage();
bmp.SetSource(e.ChosenPhoto);
WriteableBitmap btmMap = new WriteableBitmap(bmp);
facesPic.Source = btmMap;
}
}

Detecting faces

The first step is to construct the FaceDetector object in the application. Notice that you'll need haarcascade_frontalface_alt.xml file to detect faces (you can copy this file from the Example application).

const string MODEL_FILE = "haarcascade_frontalface_alt.xml";
FaceDetectionWinPhone.Detector m_detector;
public MainPage()
{
InitializeComponent();
m_detector = new FaceDetectionWinPhone.Detector(System.Xml.Linq.XDocument.Load(MODEL_FILE));
}

After previous chapter we have an image in a bitmap, so the next step is to detect faces in the image. In this example I'm using WriteableBitmapEx library to draw rectangles into bitmap to show results in the image. So let's update the photosChooserTas_Completed() function.

void photoChooserTask_Completed(object sender, PhotoResult e)
{
if (e.TaskResult == TaskResult.OK)
{
BitmapImage bmp = new BitmapImage();
bmp.SetSource(e.ChosenPhoto);
WriteableBitmap btmMap = new WriteableBitmap(bmp);
 
//find faces from the image
List<FaceDetectionWinPhone.Rectangle> faces =
m_detector.getFaces(
btmMap, 10f, 1f, 0.05f, 1, false, false);
 
//go through each face, and draw a red rectangle on top of it.
foreach (var r in faces)
{
int x = Convert.ToInt32(r.X);
int y = Convert.ToInt32(r.Y);
int width = Convert.ToInt32(r.Width);
int height = Convert.ToInt32(r.Height);
btmMap.FillRectangle(x, y, x + height, y + width, System.Windows.Media.Colors.Red);
}
//update the bitmap before drawing it.
btmMap.Invalidate();
facesPic.Source = btmMap;
}
}

Face Detection Algorithm

The face detection algorithm is based on the geometric features of a face. The vector set in this example "haarcascade_frontalface_alt.xm" is trying to recognize frontal faces by finding ears, eyes, nose etc. from the image. Therefore is't not really optimal to find faces from side profiles of humans, and it requires enough light to detect all the geometrical parts of a face. It's really not perfect, sometimes it just can't find a face for some reason. I wouldn't recommend using this library in an application where it's critical to find faces from an image.

The face detection also consumes quite a lot of CPU time, so it would be a good idea to run it on it's own thread to keep the UI responsive. At least with bigger images the waiting time for the task to complete could become too long.

RealtimeExample

RealtimeExample is the example code for the original Face Detection For Windows Phone 7 library. As the name suggests, it uses the library to do "real time" detection of faces. Here is a video of the example in action.


In this section I'm going to go though the NewCameraFrame() function which is called once for each frame captured by the camera viewfinder. In this function we use the FaceDetection library to find faces, and then draw a red square around the face to the canvas. The processing power in Phone is quite limited compared to the modern PC, so we're going to use 4x downscaled greyscale image for detecting the face. After we have detected the face, we'll multiply the x and y coordinates with 4 to get the size of the face box in the real camera frame.

In the fist step the frame is down-sampled by factor of 4. Then it's turned into a greyscale image, and the contrast of the grayscale image is adjusted with HistogramEqualization. Finally the greyscale image is transformed into ARGB format for the the FaceDetection library, and it's copied into the _pixelDataGrayInt buffer.

int _downsampleFactor = 4;
private void NewCameraFrame(object sender, CameraFrameEventArgs cameraFrameEventArgs)
{
var w = cameraViewer.CameraWidth;
var h = cameraViewer.CameraHeight;
 
//create temporary buffers for the frame
if (_pixelDataGray == null || _pixelDataGray.Length != h * w)
{
_pixelDataGray = new byte[w / _downsampleFactor * h / _downsampleFactor];
_pixelDataDownsampled =
new byte[w / _downsampleFactor * h / _downsampleFactor];
_pixelDataGrayInt = new int[w / _downsampleFactor * h / _downsampleFactor];
_wb = new WriteableBitmap(w / _downsampleFactor, w / _downsampleFactor);
//The dbgImg is the image on the upper left corner in the demo, and it's now pointing into _wb
dbgImg.Source = _wb;
}
 
_lastUpdate = DateTime.Now;
 
Utils.DownSample(cameraFrameEventArgs.ARGBData, w, h, ref _pixelDataGrayInt, _downsampleFactor);
Utils.ARGBToGreyScale(_pixelDataGrayInt, ref _pixelDataGray);
Utils.HistogramEqualization(ref _pixelDataGray);
Utils.GrayToARGB(_pixelDataGray, ref _pixelDataGrayInt);

The next step is to detect the faces from the image we just prepared in the previous step. In here we're also updating the timer text running on top of the image, and we're copying the grayscale image sent to the face detection into the image on the top left corner of the example application for debugging.

List<FaceDetectionWinPhone.Rectangle> faces =
_detector.getFaces(
_pixelDataGrayInt,
w / _downsampleFactor,
h / _downsampleFactor,
2f, 1.25f, 0.1f, 1, false, false);
 
var elapsed = (DateTime.Now - _lastUpdate).TotalMilliseconds;
cameraResolution.Text = w + " x " + h + " " + elapsed + " ms";
 
//copy the grayscale image into the frame shown in the upper left corner
_pixelDataGrayInt.CopyTo(_wb.Pixels, 0);
_wb.Invalidate();

Now we have the faces in the List, so the final step is to draw the red square to the canvas. The image sent to the FaceDetection library was downscaled by factor of 4, so in here we're multiplying the x and the y coordinates with 4 to draw the square to correct place on the canvas.

Dispatcher.BeginInvoke(delegate()
{
cnvsFaceRegions.Children.Clear();
foreach (var r in faces)
{
Rectangle toAdd = new Rectangle();
TranslateTransform loc = new TranslateTransform();
loc.X = r.X * _downsampleFactor / (double)w * cnvsFaceRegions.ActualWidth;
loc.Y = r.Y * _downsampleFactor / (double)w * cnvsFaceRegions.ActualHeight;
toAdd.RenderTransform = loc;
toAdd.Width = r.Width * _downsampleFactor;
toAdd.Height = r.Height * _downsampleFactor;
toAdd.Stroke = new SolidColorBrush(Colors.Red);
cnvsFaceRegions.Children.Add(toAdd);
}
});

Face SDK for Windows Phone 7

The Face SDK for Windows Phone 7 is an alternative face-recognition library provided by Microsoft Research. The Face SDK provides few extra features that are not included in Face Detection Library, but is bound by a licensing agreement that only allows it to be used with Windows Phone 7. Below is a short comparison of Face SDK and Face Detecting library.

Warning.pngWarning: Read the Face SDK licencing agreement carefully before including the SDK in your own apps.

Feature Face Detection Library Microsoft Face SDK
Face Detection Included Included
Detect other forms besides faces Included Not Included
Face Alignment Not Included Included
Face Tracking Not Included Included
Cartoon generation Not Included Included
Windows Phone 7 Support Included Included
Windows Phone 8 Support Included Prohibited in license agreement
License New BSD License Microsoft Research License Agreement

562 page views in the last 30 days.