Real-time camera viewfinder filters in Native code
This article explains how to create real-time camera filters, using native code (C++).
Windows Phone 8
Note: This is an "internal" entry in the Windows Phone 8 Wiki Competition 2012Q4. The author is a Nokia / Microsoft employee.
In this article, we will have a quick look how one can create real-time filters for the camera viewfinder, using native code (C++). For sake of simplicity, the example will implement a simple gray filter. For that we will use the new Windows PRT camera API that are now available in the WP8 SDK. We will also use the possibility to write C++ code, another new functionality of the WP8 SDK. The end result will look like this:
The source code of the full project can be downloaded from the link in top right corner of this page.
Why Native filters?
Microsoft has published a very similar example to this project, where they convert the camera viewfinder images to grayscale, which works well in WP7 and WP8. Get it here. This works well, but the gray filter is quite simple, more complicated filters will require more computation, and the CPU limits are quickly reached when processing VGA frames (604x480) at 30 frame per seconds. The speed gain by going closer to the metal may be needed for complex algorithm. Also, you might already have your own image filters written in C++ for other platforms, that you can reuse without converting them to C#. Finally, as we will see in other wiki entries, the native side opens further optimization possibilities, like using DirectX or the ARM Neon instruction set.
Setting up the viewfinder
Our UI will be XAML based. The UI will control everything, while the C++ side is rather dumb, simply executing the filtering when it is asked to. Let's first create the projects for both the XAML and the C++ components:
- Start by creating a new project, of type Windows Phone App. You will find the template under the Visual C#/Windows Phone category. That will be our UI.
- Add a new project to your solution, of type Windows Phone Runtime Component. That template is under the Visual C++/Windows Phone category. That will be our image filter.
We then add a live camera stream to our UI. That is easily done by :
- In your XAML, define a rectangle that will be painted using a video brush:
<Grid x:Name="LayoutRoot" Background="Transparent">
<Rectangle Width="640" Height="480" Canvas.ZIndex="1">
<VideoBrush x:Name="viewfinderBrush" />
- In the page loaded event, Create a PhotoCaptureDevice, and set it as the source of the video brush:
Windows.Foundation.Size resolution = new Windows.Foundation.Size(640, 480);
m_camera = await PhotoCaptureDevice.OpenAsync(CameraSensorLocation.Back, resolution);
By now, with these 10 lines of codes, you should have an application with a functional camera ! Note that in the last step, we have used the Windows PRT class Windows.Phone.Media.Capture.PhotoCaptureDevice which is new to Windows Phone 8. In WP7, one would use the Silverlight/.NET class Microsoft.Devices.PhotoCamera().
Talking with the native(s)
So far, that was easy, but now we need to be a bit careful. We will have quite huge amount of data. The viewfinder frames, even with VGA frames that makes 640x480 pixels at 30 frame per seconds, will go back and forth between managed code (our UI) and the native site (our C++ filter). We have to avoid any useless copy operations, as copies (memory accesses) will hurt our performance.
Windows Phone: [[Category:Windows Phone]]
[[Category:Windows Phone 7.5]]
[[Category:Windows Phone 8]]
Nokia Asha: [[Category:Nokia Asha]]
[[Category:Nokia Asha Platform 1.0]]
Series 40: [[Category:Series 40]]
[[Category:Series 40 1st Edition]] [[Category:Series 40 2nd Edition]]
[[Category:Series 40 3rd Edition (initial release)]] [[Category:Series 40 3rd Edition FP1]] [[Category:Series 40 3rd Edition FP2]]
[[Category:Series 40 5th Edition (initial release)]] [[Category:Series 40 5th Edition FP1]]
[[Category:Series 40 6th Edition (initial release)]] [[Category:Series 40 6th Edition FP1]] [[Category:Series 40 Developer Platform 1.0]] [[Category:Series 40 Developer Platform 1.1]] [[Category:Series 40 Developer Platform 2.0]]
[[Category:S60 1st Edition]] [[Category:S60 2nd Edition (initial release)]] [[Category:S60 2nd Edition FP1]] [[Category:S60 2nd Edition FP2]] [[Category:S60 2nd Edition FP3]]
[[Category:S60 3rd Edition (initial release)]] [[Category:S60 3rd Edition FP1]] [[Category:S60 3rd Edition FP2]]
[[Category:S60 5th Edition]]
[[Category:Symbian^3]] [[Category:Symbian Anna]] [[Category:Nokia Belle]]