Detecting Faces on Windows Phone
This article explains how to detect faces from images using the FaceDetectionWP8 library.
This article gives an example how to detect faces on Windows Phone platform using the FaceDetectionWP8 library. FaceDetectionWP8 is a port to Windows Phone 8 of Face Detection For Windows Phone 7 by Julia Schwarz, which in turn is largely derived from the Java-based library at jviolajones.
The library uses the same algorithms as the OpenCV Face Detection library, and hence should also be useful for detecting forms other than faces. It is licensed under the New BSD License so you can use it even on commercial projects.
The code described here is FaceDetectionWP8s example application. This app allows you to select a picture from the Camera roll, identify faces, and superimpose a rectangle over them.
At the end of the article we also briefly review RealtimeExample (the original WP7 port example code). This is more advanced and requires some image processing tricks before using the FaceDetection library. We first look at the basic example, since it's easier to understand.
Creating an example app for WP8
The basic layout of the example app is shown above in the introduction
Let's start the example application with the XAML below:
<Grid x:Name="ContentPanel" Grid.Row="1" Margin="12,0,12,0" RenderTransformOrigin="0.393,0.461">
<Button Name="btnPhotoChoose" Click ="btnPhotoChoose_Clicked" Margin="67,41,60,475">Choose Photo</Button>
<Image Name="facesPic" HorizontalAlignment="Left" Height="388" Margin="19,170,0,0" VerticalAlignment="Top" Width="427"/>
Pressing the button will launch PhotoChooser from the gallery, so let's implement that.
After the photo is chosen we can display the image in the facesPic bitmap in the current layout:
The first step is to construct the FaceDetector object in the application. Notice that you'll need haarcascade_frontalface_alt.xml file to detect faces (you can copy this file from the Example application).
const string MODEL_FILE = "haarcascade_frontalface_alt.xml";
m_detector = new FaceDetectionWinPhone.Detector(System.Xml.Linq.XDocument.Load(MODEL_FILE));
After previous chapter we have an image in a bitmap, so the next step is to detect faces in the image. In this example I'm using WriteableBitmapEx library to draw rectangles into bitmap to show results in the image. So let's update the photosChooserTas_Completed() function.
void photoChooserTask_Completed(object sender, PhotoResult e)
if (e.TaskResult == TaskResult.OK)
BitmapImage bmp = new BitmapImage();
WriteableBitmap btmMap = new WriteableBitmap(bmp);
//find faces from the image
List<FaceDetectionWinPhone.Rectangle> faces =
btmMap, 10f, 1f, 0.05f, 1, false, false);
//go through each face, and draw a red rectangle on top of it.
foreach (var r in faces)
int x = Convert.ToInt32(r.X);
int y = Convert.ToInt32(r.Y);
int width = Convert.ToInt32(r.Width);
int height = Convert.ToInt32(r.Height);
btmMap.FillRectangle(x, y, x + height, y + width, System.Windows.Media.Colors.Red);
//update the bitmap before drawing it.
facesPic.Source = btmMap;
Face Detection Algorithm
The face detection algorithm is based on the geometric features of a face. The vector set in this example "haarcascade_frontalface_alt.xm" is trying to recognize frontal faces by finding ears, eyes, nose etc. from the image. Therefore is't not really optimal to find faces from side profiles of humans, and it requires enough light to detect all the geometrical parts of a face. It's really not perfect, sometimes it just can't find a face for some reason. I wouldn't recommend using this library in an application where it's critical to find faces from an image.
The face detection also consumes quite a lot of CPU time, so it would be a good idea to run it on it's own thread to keep the UI responsive. At least with bigger images the waiting time for the task to complete could become too long.
RealtimeExample is the example code for the original Face Detection For Windows Phone 7 library. As the name suggests, it uses the library to do "real time" detection of faces. Here is a video of the example in action.
In this section I'm going to go though the NewCameraFrame() function which is called once for each frame captured by the camera viewfinder. In this function we use the FaceDetection library to find faces, and then draw a red square around the face to the canvas. The processing power in Phone is quite limited compared to the modern PC, so we're going to use 4x downscaled greyscale image for detecting the face. After we have detected the face, we'll multiply the x and y coordinates with 4 to get the size of the face box in the real camera frame.
In the fist step the frame is down-sampled by factor of 4. Then it's turned into a greyscale image, and the contrast of the grayscale image is adjusted with HistogramEqualization. Finally the greyscale image is transformed into ARGB format for the the FaceDetection library, and it's copied into the _pixelDataGrayInt buffer.
int _downsampleFactor = 4;
private void NewCameraFrame(object sender, CameraFrameEventArgs cameraFrameEventArgs)
var w = cameraViewer.CameraWidth;
var h = cameraViewer.CameraHeight;
//create temporary buffers for the frame
if (_pixelDataGray == null || _pixelDataGray.Length != h * w)
_pixelDataGray = new byte[w / _downsampleFactor * h / _downsampleFactor];
new byte[w / _downsampleFactor * h / _downsampleFactor];
_pixelDataGrayInt = new int[w / _downsampleFactor * h / _downsampleFactor];
_wb = new WriteableBitmap(w / _downsampleFactor, w / _downsampleFactor);
//The dbgImg is the image on the upper left corner in the demo, and it's now pointing into _wb
dbgImg.Source = _wb;
_lastUpdate = DateTime.Now;
Utils.DownSample(cameraFrameEventArgs.ARGBData, w, h, ref _pixelDataGrayInt, _downsampleFactor);
Utils.ARGBToGreyScale(_pixelDataGrayInt, ref _pixelDataGray);
Utils.GrayToARGB(_pixelDataGray, ref _pixelDataGrayInt);
The next step is to detect the faces from the image we just prepared in the previous step. In here we're also updating the timer text running on top of the image, and we're copying the grayscale image sent to the face detection into the image on the top left corner of the example application for debugging.
List<FaceDetectionWinPhone.Rectangle> faces =
w / _downsampleFactor,
h / _downsampleFactor,
2f, 1.25f, 0.1f, 1, false, false);
var elapsed = (DateTime.Now - _lastUpdate).TotalMilliseconds;
cameraResolution.Text = w + " x " + h + " " + elapsed + " ms";
//copy the grayscale image into the frame shown in the upper left corner
Now we have the faces in the List, so the final step is to draw the red square to the canvas. The image sent to the FaceDetection library was downscaled by factor of 4, so in here we're multiplying the x and the y coordinates with 4 to draw the square to correct place on the canvas.
foreach (var r in faces)
Rectangle toAdd = new Rectangle();
TranslateTransform loc = new TranslateTransform();
loc.X = r.X * _downsampleFactor / (double)w * cnvsFaceRegions.ActualWidth;
loc.Y = r.Y * _downsampleFactor / (double)w * cnvsFaceRegions.ActualHeight;
toAdd.RenderTransform = loc;
toAdd.Width = r.Width * _downsampleFactor;
toAdd.Height = r.Height * _downsampleFactor;
toAdd.Stroke = new SolidColorBrush(Colors.Red);
Face SDK for Windows Phone 7
The Face SDK for Windows Phone 7 is an alternative face-recognition library provided by Microsoft Research. The Face SDK provides few extra features that are not included in Face Detection Library, but is bound by a licensing agreement that only allows it to be used with Windows Phone 7. Below is a short comparison of Face SDK and Face Detecting library.
|Feature||Face Detection Library||Microsoft Face SDK|
|Detect other forms besides faces||Included||Not Included|
|Face Alignment||Not Included||Included|
|Face Tracking||Not Included||Included|
|Cartoon generation||Not Included||Included|
|Windows Phone 7 Support||Included||Included|
|Windows Phone 8 Support||Included||Prohibited in license agreement|
|License||New BSD License||Microsoft Research License Agreement|