×
Namespaces

Variants
Actions
(Difference between revisions)

Real-time camera viewfinder filters in Native code

From Nokia Developer Wiki
Jump to: navigation, search
Mansewiz (Talk | contribs)
m (Mansewiz - - Introduction)
hamishwillee (Talk | contribs)
m (Hamishwillee - Add competition winner note)
(23 intermediate revisions by 5 users not shown)
Line 1: Line 1:
[[Category:Draft]][[Category:Windows Phone]][[Category:Silverlight]][[Category:Multimedia]][[Category:Code Examples]][[Category:Code Snippet]]
+
[[Category:Windows Phone]][[Category:Silverlight]][[Category:Multimedia]][[Category:Code Examples]][[Category:Code Snippet]][[Category:Windows Phone 8]]
{{Abstract|This article explains how to create real-time camera filters for Windows Phone 8, using native code (C++). }}  
+
{{Abstract|This article explains how to create real-time camera filters for Windows Phone 8, using native code (C++). }}
 
+
{{Note|This article was a winner in the [[Windows Phone 8 Wiki Competition 2012Q4]].}}
 +
{{SeeAlso|[[Creating a Lens application that uses HLSL effects for filters]]}}
 
{{ArticleMetaData <!-- v1.2 -->
 
{{ArticleMetaData <!-- v1.2 -->
|sourcecode= http://projects.developer.nokia.com/NativeFilterDemo/browser
+
|sourcecode= [http://projects.developer.nokia.com/NativeFilterDemo/browser NativeFilterDemo] (Nokia Projects)
 +
|installfile= <!-- Link to installation file (e.g. [[Media:The Installation File.sis]]) -->
 
|devices= Lumia 810, Lumia 820, Lumia 822, Lumia 920
 
|devices= Lumia 810, Lumia 820, Lumia 822, Lumia 920
 
|sdk= <!-- SDK(s) built and tested against (e.g. [http://linktosdkdownload/ Qt SDK 1.1.4]) -->
 
|sdk= <!-- SDK(s) built and tested against (e.g. [http://linktosdkdownload/ Qt SDK 1.1.4]) -->
|platform= Windows Phone 8
+
|platform= Windows Phone 8 and later
 +
|devicecompatability= <!-- Compatible devices e.g.: All* (must have internal GPS) -->
 +
|dependencies= <!-- Any other/external dependencies e.g.: Google Maps Api v1.0 -->
 +
|signing= <!-- Signing requirements - empty or one of: Self-Signed, DevCert, Manufacturer -->
 +
|capabilities= <!-- Capabilities required by the article/code example (e.g. Location, NetworkServices. -->
 
|keywords= <!-- APIs, classes and methods (e.g. QSystemScreenSaver, QList, CBase -->
 
|keywords= <!-- APIs, classes and methods (e.g. QSystemScreenSaver, QList, CBase -->
 +
|language= <!-- Language category code for non-English topics - e.g. Lang-Chinese -->
 
|translated-by= <!-- [[User:XXXX]] -->
 
|translated-by= <!-- [[User:XXXX]] -->
|translated-from-title= <!-- Title only -->  
+
|translated-from-title= <!-- Title only -->
 
|translated-from-id= <!-- Id of translated revision -->
 
|translated-from-id= <!-- Id of translated revision -->
|review-by=<!-- After re-review: [[User:username]] -->
+
|review-by= <!-- After re-review: [[User:username]] -->
 
|review-timestamp= <!-- After re-review: YYYYMMDD -->
 
|review-timestamp= <!-- After re-review: YYYYMMDD -->
 
|update-by= <!-- After significant update: [[User:username]]-->
 
|update-by= <!-- After significant update: [[User:username]]-->
 
|update-timestamp= <!-- After significant update: YYYYMMDD -->
 
|update-timestamp= <!-- After significant update: YYYYMMDD -->
|creationdate= <!-- Format YYYYMMDD -->
+
|creationdate= 20121124
|author= Mansewiz
+
|author= [[User:Mansewiz]]
 
}}
 
}}
 
{{Note|This is an "internal" entry in the [[Windows Phone 8 Wiki Competition 2012Q4]]. The author is a Nokia / Microsoft employee.}}
 
  
 
== Introduction ==
 
== Introduction ==
One of the big new feature of Windows Phone 8 SDK is the support for C and C++, also known as native code support.  In this article, we will have a brief look how one can leverage that support to create real-time filters for the camera. For sake of simplicity, the example will implement a simple gray filter, that will convert on the fly the camera input and show the result on the phone display.  <br/><br/>
+
One of the big new features of Windows Phone 8 SDK is the support for C and C++, also known as native code support.  In this article, we will have a brief look at how one can exploit that support to create real-time filters for the camera. For simplicity's sake, the example will implement a simple gray filter, that will convert camera input on the fly and show the result onscreen.  <br/><br/>
 
The demo application will look like this:  
 
The demo application will look like this:  
 
[[File:NativeFilterDemo.png|400px|none]]
 
[[File:NativeFilterDemo.png|400px|none]]
 
.
 
.
  
== Why Native filters? ==
+
== Why native filters? ==
Microsoft has published a very similar example, where they do live conversion of the camera viewfinder images to grayscale. The example was written for Windows Phone 7, but it also works well in WP8. [http://msdn.microsoft.com/en-us/library/hh202982%28v=vs.92%29.aspx Get it here.] This works well, but the gray filter is quite simple, more complicated filters will require more computation, and the CPU is quickly maxed out when we try to process the camera input at 30 frame per seconds. The speed gain by going closer to the metal may be needed for complex algorithm. Also, you might already have your own image filters written in C and/or C++ for other platforms, that you can reuse without converting them to C#. Finally, as we will see in other wiki entries, the native side opens further optimization possibilities, like using DirectX or the ARM Neon instruction set.
+
Microsoft has published a very similar example, where they do live conversion of the camera viewfinder images to grayscale. The example was written for Windows Phone 7 (you can download it [http://msdn.microsoft.com/en-us/library/hh202982%28v=vs.92%29.aspx here]), but it also works well in WP8. This works well, but the gray filter is quite simple; more complicated filters will require more computation, and the CPU is quickly maxed out when we try to process the camera input at several frames per second. The speed gain by going closer to the metal may be needed for more complex algorithms. Also, you might already have your own image filters written in C and/or C++ for other platforms, that you can reuse without converting them to C#. Finally, as we will see in other wiki entries, the native side opens further optimization possibilities, like using DirectX or the ARM Neon instruction set.
  
== Setting up the viewfinder ==
+
== Setting up the viewfinder ==
 
Our UI will be XAML based. The UI will control everything, while the C++ side is rather dumb, simply executing the filtering when asked to. Let's first create the projects for both the XAML and the C++ components:
 
Our UI will be XAML based. The UI will control everything, while the C++ side is rather dumb, simply executing the filtering when asked to. Let's first create the projects for both the XAML and the C++ components:
 
<br/>
 
<br/>
Line 36: Line 41:
 
* Add a new project to your solution, of type '''''Windows Phone Runtime Component'''''. That template is under the '''Visual C++/Windows Phone''' category. That will become our image filter.  
 
* Add a new project to your solution, of type '''''Windows Phone Runtime Component'''''. That template is under the '''Visual C++/Windows Phone''' category. That will become our image filter.  
 
<br/>
 
<br/>
We then add a live camera stream to our UI. That is easily done by :  
+
We then add a live camera stream to our UI. That is easily done by:  
* In your XAML, define a rectangle that will be painted using a video brush:  
+
* In your XAML, define a rectangle that will be painted using a {{Icode|VideoBrush}}:  
 
<code xml>
 
<code xml>
 
<Grid x:Name="LayoutRoot" Background="Transparent">
 
<Grid x:Name="LayoutRoot" Background="Transparent">
Line 47: Line 52:
 
   </Grid>
 
   </Grid>
 
</code>
 
</code>
* In the page loaded event, Create a PhotoCaptureDevice, and set it as the source of the video brush:
+
* In the page loaded event, create a PhotoCaptureDevice, and set it as the source of the {{Icode|VideoBrush}}:
 
<code csharp>
 
<code csharp>
 
Windows.Foundation.Size resolution = new Windows.Foundation.Size(640, 480);
 
Windows.Foundation.Size resolution = new Windows.Foundation.Size(640, 480);
Line 53: Line 58:
 
ViewfinderBrush.SetSource(m_camera);
 
ViewfinderBrush.SetSource(m_camera);
 
</code>
 
</code>
By now, with these 10 lines of codes, you should have an application with a functional camera ! Note that in the last step, we use the Windows PRT class '''Windows.Phone.Media.Capture.PhotoCaptureDevice''' which is new to Windows Phone 8. In WP7, one would have to use the Silverlight/.NET class '''Microsoft.Devices.PhotoCamera'''. Because it's a Win PRT class, it can be accessed by both managed and native code, we will soon take advantage of that possibility.
+
By now, with these 10 lines of codes, you should have an application with a functional camera! Note that in the last step, we use the Windows PRT class '''Windows.Phone.Media.Capture.PhotoCaptureDevice''' which is new to Windows Phone 8. In WP7, one would have to use the Silverlight/.NET class '''Microsoft.Devices.PhotoCamera'''. Because it's a Win PRT class, it can be accessed by both managed and native code; we will soon take advantage of that possibility.
  
== Talking with the native(s) ==  
+
== Displaying the filtered frames ==
So far, that was easy, but now we need to be a bit careful. We will handle quite huge amount of data. The viewfinder frames, 640x480 pixels refreshed at a rate of 30 frames per second, will go back and forth between managed code (our UI) and the native site (our C++ filter). That's a lot of pixels per seconds! We have to avoid any useless copy operations, as copies (memory accesses) will hurt our performance. <br/>
+
The {{Icode|VideoBrush}} is easy to use but it doesn't offer us a way to get in-between the camera and the brush to apply our filter. We need to find another way to display the modified frames coming from the camera. Several strategies are possible, but the strategy chosen must have a reasonable camera lag (the time between the photons enters the camera until the scene is displayed on the screen) as well as a decent frame rate. Let's have a quick look at 3 possible strategies:
We will implement the following sequence diagram, for each new viewfinder frame coming from the camera:
+
# Using an {{Icode|Image}} control, with {{Icode|WriteableBitmap}} as a source.
 +
# Using a {{Icode|MediaElement}} control, with a custom {{Icode|MediaStreamSource}} as a source.
 +
# Using a DirectX texture.
 +
<br/>
 +
=== Using an Image control ===
 +
This is the strategy used in the [http://msdn.microsoft.com/en-us/library/hh202982%28v=vs.92%29.aspx How to: Work with Grayscale in a Camera Application for Windows Phone] MSDN sample. The UI control defined in XAML is:
 +
<code xml>
 +
<Image x:Name="MyCameraViewfiner"  Width="640" Height="480">
 +
</code>
 +
 
 +
The source of the {{Icode|Image}}} is set to a {{Icode|WriteableBitmap}}. When the frames from the camera have been filtered, the resulting array of pixels is copied into to that {{Icode|WriteableBitmap}}. 
 +
<code csharp>
 +
Deployment.Current.Dispatcher.BeginInvoke(delegate()  // Switch to UI thread
 +
{
 +
    // Copy to WriteableBitmap.
 +
    ARGBPx.CopyTo(wb.Pixels, 0);
 +
    wb.Invalidate();
 +
});
 +
</code>
 +
The problem with this is the switch to UI thread ({{Icode|Deployment.Current.Dispatcher.BeginInvoke}}). The switch is required so that the application does not update the bitmap while the UI thread is doing something with it. Switching between threads is quite slow, and the lag associated with that solution was too high for the application.
 +
 
 +
=== Using a MediaElement ===
 +
The [http://msdn.microsoft.com/en-us/library/system.windows.controls.mediaelement.aspx MediaElement] is 'the' UI control that is used for media playback, videos or audio, streamed media or from a local file. It takes care of all the buffering, audio sync and displaying logic and it's highly optimized. By defining a custom {{Icode|MediaStreamSource}}, we can feed the filtered camera frames to the {{Icode|MediaElement}} that will take care of displaying them properly.
 +
<code xml>
 +
<MediaElement x:Name="MyCameraMediaElement"
 +
                            IsHitTestVisible="False"
 +
                            Margin="4" Width="640" Height="480" />
 +
</code>
 +
<br/>
 +
Defining your own {{Icode|MediaStreamSource}} requires a little bit of work, but it's all worth it. The lag with this solution is small, and the frame rate we can achieve is pretty good. This is the solution we will take into use.
 +
 
 +
=== Using a DirectX texture ===
 +
When it comes to camera lag and performance, this is probably the best solution. However DirectX has a fairly steep learning curve, and to keep things simple we will not use it. Look over [[Creating a Lens application that uses HLSL effects for filters|this article]] if you're interested in using DirectX.
 +
 
 +
== Our own MediaStreamSource==
 +
To feed video frames to the {{Icode|MediaElement}} we need to define our custom {{Icode|MediaStreamSource}}. Pete Brown walks us through how this should be done in [http://10rem.net/blog/2010/06/22/book-excerpt-creating-raw-media-audio-and-video his blog]. Check it out for more details, here we will only go through the big lines.
 +
<br/>
 +
Our {{Icode|MediaStreamSource}} will define a single stream of type video (RGBA). Since our real source for video data is the camera, we have a source of infinite length and that can't be seeked. Our initialization will look like this:
 +
<code csharp>
 +
mediaStreamAttributes[MediaStreamAttributeKeys.VideoFourCC] = "RGBA";
 +
mediaStreamAttributes[MediaStreamAttributeKeys.Width] = _frameWidth.ToString();
 +
mediaStreamAttributes[MediaStreamAttributeKeys.Height] = _frameHeight.ToString();
 +
mediaStreamDescriptions.Add(_videoStreamDescription);
 +
 
 +
// a zero timespan is an infinite video
 +
mediaSourceAttributes[MediaSourceAttributesKeys.Duration] =
 +
TimeSpan.FromSeconds(0).Ticks.ToString(CultureInfo.InvariantCulture);
 +
// Can't seek.
 +
mediaSourceAttributes[MediaSourceAttributesKeys.CanSeek] = false.ToString();
 +
</code>
 +
 
 +
Whenever the {{Icode|MediaElement}} decides it needs a new video frame, it will call the {{Icode|GetSampleAsync()}} method from our {{Icode|MediaStreamSource}}.
 +
 
 +
We will implement the following sequence diagram :
 
[[File:SequenceDiagram.png|none]]
 
[[File:SequenceDiagram.png|none]]
 
To get the data of the frames coming from the camera, we will have to call to the platform functionality '''Windows::Phone::Media::Capture::ICameraCaptureDevice::GetPreviewBufferArgb'''. Let's look at its definition:
 
To get the data of the frames coming from the camera, we will have to call to the platform functionality '''Windows::Phone::Media::Capture::ICameraCaptureDevice::GetPreviewBufferArgb'''. Let's look at its definition:
Line 63: Line 121:
 
void GetPreviewBufferArgb(Platform::WriteOnlyArray<int, 1U>^ pixels)  
 
void GetPreviewBufferArgb(Platform::WriteOnlyArray<int, 1U>^ pixels)  
 
</code>
 
</code>
That function fills an array that we provide with the camera data. That's a copy operation we can't avoid. The buffer is of type Platform::WriteOnlyArray, which according to [http://msdn.microsoft.com/en-us/library/windows/apps/hh700131.aspx MSDN documentation], is to be used when the caller passes an array for the method to fill . Makes sense. We will also use that buffer type to communicate between our managed component and our native component<br/>
+
That method fills an array of our choice with the camera data. That's a copy operation we can't avoid. The buffer is of type {{Icode|Platform::WriteOnlyArray}}, which according to the [http://msdn.microsoft.com/en-us/library/windows/apps/hh700131.aspx MSDN documentation], is to be used when the caller passes an array for the method to fill. We will also use that buffer type to communicate between our managed component and our native component.<br/>
  
The public interface of our native component will be:
+
The public interface of our native component will be:
 
<code cpp>
 
<code cpp>
 
     public ref class WindowsPhoneRuntimeComponent sealed
 
     public ref class WindowsPhoneRuntimeComponent sealed
Line 72: Line 130:
 
         WindowsPhoneRuntimeComponent();
 
         WindowsPhoneRuntimeComponent();
 
         void Initialize(Windows::Phone::Media::Capture::PhotoCaptureDevice^ captureDevice);
 
         void Initialize(Windows::Phone::Media::Capture::PhotoCaptureDevice^ captureDevice);
         void NewViewfinderFrame( Platform::WriteOnlyArray<int,1U>^ frameData);
+
         void NewViewfinderFrame( Platform::WriteOnlyArray<int,1U>^ inputBuffer,
 +
          Platform::WriteOnlyArray<uint8,1U>^ outputBuffer);
 
     ...
 
     ...
 
     };
 
     };
 
</code>
 
</code>
On the managed side, we allocate the buffer when the application is initialized, and reuse that buffer for every frames:
 
<code>m_frameData = new int[(int)m_camera.PreviewResolution.Height * (int)m_camera.PreviewResolution.Width]; </code>
 
  
Every time the camera subsystem fires a '''PreviewFrameAvailable''' event we will call the native '''NewViewfinderFrame''' method who will take care of filling the buffer with camera data and filter that data.
+
On the managed side, we allocate the {{Icode|inputBuffer}} and the {{Icode|outputBuffer}} when the application is initialized, and reuse those buffers for each frame:
<br/>
+
<code cpp>
When the UI side receives back the filtered camera data buffer, it is ready to be displayed. The UI side will do that with the following :
+
_cameraData = new int[_frameWidth * _frameHeight];
<code csharp>
+
_frameBufferSize = _frameWidth * _frameHeight * _framePixelSize;
Deployment.Current.Dispatcher.BeginInvoke(delegate()
+
_cameraFilteredData = new byte[_frameBufferSize]
{
+
</code>
      m_frameData.CopyTo(m_wb.Pixels, 0);
+
      m_wb.Invalidate();
+
      m_processingFrame = false;
+
});
+
</code>  
+
  
Again, an unavoidable copy! This time the data is copied to a WriteableBitmap, '''m_wb'''. At the initialization phase, we defined '''m_wb''' as the source for the XAML Image component that displays our filtered viewfinder. <br/>
+
Note that for this example, my data buffer is in RGBA format. It's a format easy to handle, and the most familiar for most of us. However, it is not very efficient in terms of size and image manipulation performance. Using YUV format/color space would make more sense.  
 
+
So, we end up copying two times the camera data, first from camera subsystem into our application, then from our application to the display subsystem. <br/>
+
Note that for this example, my data buffer is in the RGBA format. It's a format easy to handle, and the most familiar for most of us. However, it is not very efficient in terms of size and image manipulation performance. Using YUV format/color space would make more sense.  
+
 
<br/>
 
<br/>
 
That covers the big lines of the data handling of the application. I skipped the not-so-interesting code, get the source code of the full project from the link in top right corner of this page. The last thing to do is the filtering itself.
 
That covers the big lines of the data handling of the application. I skipped the not-so-interesting code, get the source code of the full project from the link in top right corner of this page. The last thing to do is the filtering itself.
  
 
== The filtering in C++ ==
 
== The filtering in C++ ==
For the filtering, I took the code from [http://hilbert-space.de/?p=22 Nils Pipenbrinck's excellent blog entry on Neon optimization].  
+
For the filtering, I took the code from Nils Pipenbrinck's excellent [http://hilbert-space.de/?p=22 blog entry] on Neon optimization.  
<code c++>
+
<code cpp>
 
void WindowsPhoneRuntimeComponent::ConvertToGrayOriginal( Platform::WriteOnlyArray<int,1U>^ frameData)
 
void WindowsPhoneRuntimeComponent::ConvertToGrayOriginal( Platform::WriteOnlyArray<int,1U>^ frameData)
 
{
 
{
Line 127: Line 176:
 
}
 
}
 
</code>
 
</code>
If you look in the project code, you will find the same filter, but optimized with the ARM Neon instruction set. I'll let you pick the one you prefer.  
+
If you look in the project code, you will find the same filter, but optimized with the ARM Neon instruction set. I'll let you pick the one you prefer.
 +
 
 
== Wrapping it up ==
 
== Wrapping it up ==
Hopefully this short article will help you getting started writing native filters with Windows Phone 8. Remember to always keep the performance in mind, these types of applications are very CPU intensive. Using C++ for your filters is one way to improve the performance. If the performance gain is not enough for your use cases, you might have a look at hooking your camera directly into DirectX , or optimizing your filter with Neon.
+
Hopefully this short article will help you getting started writing native filters with Windows Phone 8. Remember to always keep the performance in mind, these types of applications are very CPU intensive. Using C++ for your filters is one way to improve the performance. If the performance gain is not enough for your use cases, you might have a look at hooking your camera directly into DirectX, or optimizing your filter with Neon instructions.
  
 
== Source code ==  
 
== Source code ==  
 
The source code is maintained in [http://projects.developer.nokia.com/NativeFilterDemo/browser#NativeComponent Nokia Project]. Look for the zip file at the bottom of the page.
 
The source code is maintained in [http://projects.developer.nokia.com/NativeFilterDemo/browser#NativeComponent Nokia Project]. Look for the zip file at the bottom of the page.

Revision as of 02:11, 23 January 2013

This article explains how to create real-time camera filters for Windows Phone 8, using native code (C++).

Note.pngNote: This article was a winner in the Windows Phone 8 Wiki Competition 2012Q4.

WP Metro Icon Multimedia.png
SignpostIcon XAML 40.png
WP Metro Icon DirectX.png
WP Metro Icon WP8.png
Article Metadata
Code Example
Source file: NativeFilterDemo (Nokia Projects)
Tested with
Devices(s): Lumia 810, Lumia 820, Lumia 822, Lumia 920
Compatibility
Platform(s): Windows Phone 8 and later
Windows Phone 8
Article
Created: Mansewiz (24 Nov 2012)
Last edited: hamishwillee (23 Jan 2013)

Contents

Introduction

One of the big new features of Windows Phone 8 SDK is the support for C and C++, also known as native code support. In this article, we will have a brief look at how one can exploit that support to create real-time filters for the camera. For simplicity's sake, the example will implement a simple gray filter, that will convert camera input on the fly and show the result onscreen.

The demo application will look like this:

NativeFilterDemo.png

.

Why native filters?

Microsoft has published a very similar example, where they do live conversion of the camera viewfinder images to grayscale. The example was written for Windows Phone 7 (you can download it here), but it also works well in WP8. This works well, but the gray filter is quite simple; more complicated filters will require more computation, and the CPU is quickly maxed out when we try to process the camera input at several frames per second. The speed gain by going closer to the metal may be needed for more complex algorithms. Also, you might already have your own image filters written in C and/or C++ for other platforms, that you can reuse without converting them to C#. Finally, as we will see in other wiki entries, the native side opens further optimization possibilities, like using DirectX or the ARM Neon instruction set.

Setting up the viewfinder

Our UI will be XAML based. The UI will control everything, while the C++ side is rather dumb, simply executing the filtering when asked to. Let's first create the projects for both the XAML and the C++ components:

  • Start by creating a new project, of type Windows Phone App. You will find the template under the Visual C#/Windows Phone category. That's where our UI will be coded.
  • Add a new project to your solution, of type Windows Phone Runtime Component. That template is under the Visual C++/Windows Phone category. That will become our image filter.


We then add a live camera stream to our UI. That is easily done by:

  • In your XAML, define a rectangle that will be painted using a VideoBrush:
<Grid x:Name="LayoutRoot" Background="Transparent">
<Rectangle Width="640" Height="480" Canvas.ZIndex="1">
<Rectangle.Fill>
<VideoBrush x:Name="viewfinderBrush" />
</Rectangle.Fill>
</Rectangle>
</Grid>
  • In the page loaded event, create a PhotoCaptureDevice, and set it as the source of the VideoBrush:
Windows.Foundation.Size resolution = new Windows.Foundation.Size(640, 480);
m_camera = await PhotoCaptureDevice.OpenAsync(CameraSensorLocation.Back, resolution);
ViewfinderBrush.SetSource(m_camera);

By now, with these 10 lines of codes, you should have an application with a functional camera! Note that in the last step, we use the Windows PRT class Windows.Phone.Media.Capture.PhotoCaptureDevice which is new to Windows Phone 8. In WP7, one would have to use the Silverlight/.NET class Microsoft.Devices.PhotoCamera. Because it's a Win PRT class, it can be accessed by both managed and native code; we will soon take advantage of that possibility.

Displaying the filtered frames

The VideoBrush is easy to use but it doesn't offer us a way to get in-between the camera and the brush to apply our filter. We need to find another way to display the modified frames coming from the camera. Several strategies are possible, but the strategy chosen must have a reasonable camera lag (the time between the photons enters the camera until the scene is displayed on the screen) as well as a decent frame rate. Let's have a quick look at 3 possible strategies:

  1. Using an Image control, with WriteableBitmap as a source.
  2. Using a MediaElement control, with a custom MediaStreamSource as a source.
  3. Using a DirectX texture.


Using an Image control

This is the strategy used in the How to: Work with Grayscale in a Camera Application for Windows Phone MSDN sample. The UI control defined in XAML is:

<Image x:Name="MyCameraViewfiner"  Width="640" Height="480">

The source of the Image} is set to a WriteableBitmap. When the frames from the camera have been filtered, the resulting array of pixels is copied into to that WriteableBitmap.

Deployment.Current.Dispatcher.BeginInvoke(delegate()  // Switch to UI thread
{
// Copy to WriteableBitmap.
ARGBPx.CopyTo(wb.Pixels, 0);
wb.Invalidate();
});

The problem with this is the switch to UI thread (Deployment.Current.Dispatcher.BeginInvoke). The switch is required so that the application does not update the bitmap while the UI thread is doing something with it. Switching between threads is quite slow, and the lag associated with that solution was too high for the application.

Using a MediaElement

The MediaElement is 'the' UI control that is used for media playback, videos or audio, streamed media or from a local file. It takes care of all the buffering, audio sync and displaying logic and it's highly optimized. By defining a custom MediaStreamSource, we can feed the filtered camera frames to the MediaElement that will take care of displaying them properly.

<MediaElement x:Name="MyCameraMediaElement"
IsHitTestVisible="False"
Margin="4" Width="640" Height="480" />


Defining your own MediaStreamSource requires a little bit of work, but it's all worth it. The lag with this solution is small, and the frame rate we can achieve is pretty good. This is the solution we will take into use.

Using a DirectX texture

When it comes to camera lag and performance, this is probably the best solution. However DirectX has a fairly steep learning curve, and to keep things simple we will not use it. Look over this article if you're interested in using DirectX.

Our own MediaStreamSource

To feed video frames to the MediaElement we need to define our custom MediaStreamSource. Pete Brown walks us through how this should be done in his blog. Check it out for more details, here we will only go through the big lines.
Our MediaStreamSource will define a single stream of type video (RGBA). Since our real source for video data is the camera, we have a source of infinite length and that can't be seeked. Our initialization will look like this:

mediaStreamAttributes[MediaStreamAttributeKeys.VideoFourCC] = "RGBA";
mediaStreamAttributes[MediaStreamAttributeKeys.Width] = _frameWidth.ToString();
mediaStreamAttributes[MediaStreamAttributeKeys.Height] = _frameHeight.ToString();
mediaStreamDescriptions.Add(_videoStreamDescription);
 
// a zero timespan is an infinite video
mediaSourceAttributes[MediaSourceAttributesKeys.Duration] =
TimeSpan.FromSeconds(0).Ticks.ToString(CultureInfo.InvariantCulture);
// Can't seek.
mediaSourceAttributes[MediaSourceAttributesKeys.CanSeek] = false.ToString();

Whenever the MediaElement decides it needs a new video frame, it will call the GetSampleAsync() method from our MediaStreamSource.

We will implement the following sequence diagram :

SequenceDiagram.png

To get the data of the frames coming from the camera, we will have to call to the platform functionality Windows::Phone::Media::Capture::ICameraCaptureDevice::GetPreviewBufferArgb. Let's look at its definition:

void GetPreviewBufferArgb(Platform::WriteOnlyArray<int, 1U>^ pixels)

That method fills an array of our choice with the camera data. That's a copy operation we can't avoid. The buffer is of type Platform::WriteOnlyArray, which according to the MSDN documentation, is to be used when the caller passes an array for the method to fill. We will also use that buffer type to communicate between our managed component and our native component.

The public interface of our native component will be:

    public ref class WindowsPhoneRuntimeComponent sealed
{
public:
WindowsPhoneRuntimeComponent();
void Initialize(Windows::Phone::Media::Capture::PhotoCaptureDevice^ captureDevice);
void NewViewfinderFrame( Platform::WriteOnlyArray<int,1U>^ inputBuffer,
Platform::WriteOnlyArray<uint8,1U>^ outputBuffer);
...
};

On the managed side, we allocate the inputBuffer and the outputBuffer when the application is initialized, and reuse those buffers for each frame:

_cameraData = new int[_frameWidth * _frameHeight];
_frameBufferSize = _frameWidth * _frameHeight * _framePixelSize;
_cameraFilteredData = new byte[_frameBufferSize]

Note that for this example, my data buffer is in RGBA format. It's a format easy to handle, and the most familiar for most of us. However, it is not very efficient in terms of size and image manipulation performance. Using YUV format/color space would make more sense.
That covers the big lines of the data handling of the application. I skipped the not-so-interesting code, get the source code of the full project from the link in top right corner of this page. The last thing to do is the filtering itself.

The filtering in C++

For the filtering, I took the code from Nils Pipenbrinck's excellent blog entry on Neon optimization.

void WindowsPhoneRuntimeComponent::ConvertToGrayOriginal( Platform::WriteOnlyArray<int,1U>^ frameData)
{
uint8 * src = (uint8 *) frameData->Data;
uint8 * dest = (uint8 *) frameData->Data;
int n = frameData->Length;
int i;
for (i=0; i<n; i++)
{
int r = *src++; // load red
int g = *src++; // load green
int b = *src++; // load blue
src++; //Alpha
 
// build weighted average:
int y = (r*77)+(g*151)+(b*28);
 
// undo the scale by 256 and write to memory:
 
*dest++ = (y>>8);
*dest++ = (y>>8);
*dest++ = (y>>8);
dest++;
 
}
}

If you look in the project code, you will find the same filter, but optimized with the ARM Neon instruction set. I'll let you pick the one you prefer.

Wrapping it up

Hopefully this short article will help you getting started writing native filters with Windows Phone 8. Remember to always keep the performance in mind, these types of applications are very CPU intensive. Using C++ for your filters is one way to improve the performance. If the performance gain is not enough for your use cases, you might have a look at hooking your camera directly into DirectX, or optimizing your filter with Neon instructions.

Source code

The source code is maintained in Nokia Project. Look for the zip file at the bottom of the page.

315 page views in the last 30 days.
×