Namespaces

Variants
Actions

Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries over the next few weeks. Thanks for all your past and future contributions.

(Difference between revisions)

Template universal app for video recording with MediaCapture using Imaging SDK Filters

From Wiki
Jump to: navigation, search
leemcpherson (Talk | contribs)
(Leemcpherson - - The Transform Class)
leemcpherson (Talk | contribs)
(Leemcpherson - - The Transform Class)
Line 103: Line 103:
  
 
To the Shared C++ project Add both a .h header and .cpp c++ file.  Creating these two files from scratch is beyond the scope of this article.  Instead copy the contents of the ImagingEffect.cpp and ImagingEffect.h from the sample to your project.  (These files are too large to post directly in this article.)  There are several functions within this class that this article will highlight in the next section so that you can modify or add to this class.
 
To the Shared C++ project Add both a .h header and .cpp c++ file.  Creating these two files from scratch is beyond the scope of this article.  Instead copy the contents of the ImagingEffect.cpp and ImagingEffect.h from the sample to your project.  (These files are too large to post directly in this article.)  There are several functions within this class that this article will highlight in the next section so that you can modify or add to this class.
 +
 +
==== Static Transform Functions for NV12 & YUY2 ====
 +
Most of the work of the transform happens within the non-class functions TransformImage_NV12 and TransformImage_YUY2.  These functions transform the video frame at the pointer pSrc and copy the new, transformed image to the pointer pDest.  Another important variable within these functions is the pointer to a wide string, filterParams.  This string contains parameters from the C# project that define which filters to use and what parameters to use with them.
 +
<code cpp>
 +
auto size = Windows::Foundation::Size(dwWidthInPixels, dwHeightInPixels);
 +
auto totalbytes = (int)dwHeightInPixels * (int)dwWidthInPixels * 2;  //each macropixel of 4 bytes creates 2 pixels (YUYV)
 +
Bitmap^ m_BitmapToProcess = AsBitmapYUY2(pSrc, (unsigned int)size.Width, (unsigned int)size.Height);
 +
ApplyImagingFilters(m_BitmapToProcess, pDest, totalbytes, filterParams, ColorMode::Yuv422_Y1UY2V);
 +
</code>
 +
 +
In order to use the Nokia Imaging SDK, the first step is to transform the raw byte data into a Bitmap.  This is accomplished for the NV12 function using another function pulled directly from
 +
 +
 +
NV12 and YUY2 are two different formats for video that uses luminance and chroma data instead of straight RGBA data.  These are the only two formats used by the Nokia Lumia 920 and the LifeCam HD-5000 webcam.
  
 
== Summary ==
 
== Summary ==

Revision as of 02:26, 8 June 2014

This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied. The article will also describe key sections of code that are requirements for creating your own version of this app.

Enter article metadata as described below. Note that this template can be placed anywhere in the article. Do not remove parameters that you do not use

WP Metro Icon Multimedia.png
SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Compatibility
Platform(s):
Windows Phone 8
Article
Created: leemcpherson (09 Jun 2014)
Last edited: leemcpherson (08 Jun 2014)

Contents

Introduction

With the introduction of universal apps and WinRT for Windows Phone 8.1, the Silverlight/WP8.0 method of accessing the camera has changed. Within the latest Nokia Imaging SDK is a IImageProvider called CameraPreviewImageSource which allows you to process frames from a video stream before capturing a final image to process. However, unless you were to provide your own video encoder, you cannot record the video being output by the CameraPreviewImageSource.

WinRT provides a MediaCapture class which provides video preview, image capture, and video record capabilities. The preview can be output to a new XAML class called a CaptureElement. There is no traditional way to apply the Nokia Imaging SDK to the preview (or record) output of the MediaCapture class. However, the MediaCapture class will accept Media Foundation Transforms through the AddEffectAsync method. This article will describe the key sections of code that are required to add a Media Foundation Transform to the app and how to incorporate the Nokia Imaging SDk into it.

Many parts of this app were inspired by or even pulled directly from code provided by other developers on this site.

References coming soon....

C++ Universal Component

The Media Foundation Transform must be coded in C++ and the interfaces that must be implemented within the transform class are quite complicated. Thankfully, there exist several samples of a Media Foundation transform that do not require much modification for our purposes. A good sample to start with is the MediaExtensions sample from the Univeral Apps sample set from Microsoft. The sample contains a GrayscaleTransform component from which this app's transform is based.

http://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096#content

Preparing the C++ Component

Add Media Foundation Libraries

Next, the Media Foundation libraries need to be added to the linker. For each C++ project (WP8.1 and Win8), right-click on the project name and select properties. Expand the Linker group and select Input. Change the Configuration (at the top) from Active to All Configurations and change Platform from Active to All Platforms so that the following changes are applied to every configuration and platform.

To the end of the "Additional Dependencies" block, add mfplat.lib and mfuuid.lib separated by a semi-colon(;).

Add Common Folder from sample & Include Directories

At the root of the sample app (this app or the MediaExtensions app) is a folder labeled Common which contains several C++ header files (AsyncCB.h, CritSec.h, ExtensionsDefs.h, etc) . These are necessary for the media transform code to compile. Copy the entire Common folder to the root of your own solution.

Next, go to the property page for each C++ project in your app and change the configuration and platform to all as before. Expand the C/C++ menu and click on General. To the end of the Addition Include Directories, add a semicolon (;) and the path "..\..\Common" so that VS2013 can find the header files we just copied.

Modify pch.h

Replace the contents of your pch.h file with the following:

#pragma once
 
#include <collection.h>
#include <ppltasks.h>
 
// Windows Header Files:
#include <windows.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mfapi.h>
#include <mferror.h>
#include <d3d11.h>
#include <D2d1helper.h>
 
#include <assert.h>
 
#include <tchar.h>
#include <Strsafe.h>
#include <objidl.h>
#include <new>
 
#include <wrl\client.h>
#include <wrl\implements.h>
#include <wrl\ftm.h>
#include <wrl\event.h>
#include <wrl\wrappers\corewrappers.h>
#include <windows.media.h>
 
#include <ppltasks.h>
 
#include <ExtensionsDefs.h>
 
using namespace Platform;
using namespace Microsoft::WRL;
using namespace Microsoft::WRL::Wrappers;


Add the Nokia Imaging SDK

Add the Nokia Imaging SDK to all projects in the solution using NuGet. You do NOT have to manually add a reference to the Imaging SDK to the C++ projects. They will not appear as a reference under the properties, but they will be included in the project anyways.

Add NativeBuffer.h

This class is copied entirely from galazzo 's article here: http://developer.nokia.com/community/wiki/Nokia_Imaging_SDK_in_native_code#Wrapping_buffers_into_a_Bitmap

The Transform Class

To the Shared C++ project Add both a .h header and .cpp c++ file. Creating these two files from scratch is beyond the scope of this article. Instead copy the contents of the ImagingEffect.cpp and ImagingEffect.h from the sample to your project. (These files are too large to post directly in this article.) There are several functions within this class that this article will highlight in the next section so that you can modify or add to this class.

Static Transform Functions for NV12 & YUY2

Most of the work of the transform happens within the non-class functions TransformImage_NV12 and TransformImage_YUY2. These functions transform the video frame at the pointer pSrc and copy the new, transformed image to the pointer pDest. Another important variable within these functions is the pointer to a wide string, filterParams. This string contains parameters from the C# project that define which filters to use and what parameters to use with them.

auto size = Windows::Foundation::Size(dwWidthInPixels, dwHeightInPixels);
auto totalbytes = (int)dwHeightInPixels * (int)dwWidthInPixels * 2; //each macropixel of 4 bytes creates 2 pixels (YUYV)
Bitmap^ m_BitmapToProcess = AsBitmapYUY2(pSrc, (unsigned int)size.Width, (unsigned int)size.Height);
ApplyImagingFilters(m_BitmapToProcess, pDest, totalbytes, filterParams, ColorMode::Yuv422_Y1UY2V);

In order to use the Nokia Imaging SDK, the first step is to transform the raw byte data into a Bitmap. This is accomplished for the NV12 function using another function pulled directly from


NV12 and YUY2 are two different formats for video that uses luminance and chroma data instead of straight RGBA data. These are the only two formats used by the Nokia Lumia 920 and the LifeCam HD-5000 webcam.

Summary

Remove Category:Draft when the page is complete or near complete


The "platform categories" will be displayed here in preview only - Copy paste relevant categories into text here

Version Hint

Windows Phone: [[Category:Windows Phone]]
[[Category:Windows Phone 7.5]]
[[Category:Windows Phone 8]]

Nokia Asha: [[Category:Nokia Asha]]
[[Category:Nokia Asha Platform 1.0]]

Series 40: [[Category:Series 40]]
[[Category:Series 40 1st Edition]] [[Category:Series 40 2nd Edition]]
[[Category:Series 40 3rd Edition (initial release)]] [[Category:Series 40 3rd Edition FP1]] [[Category:Series 40 3rd Edition FP2]]
[[Category:Series 40 5th Edition (initial release)]] [[Category:Series 40 5th Edition FP1]]
[[Category:Series 40 6th Edition (initial release)]] [[Category:Series 40 6th Edition FP1]] [[Category:Series 40 Developer Platform 1.0]] [[Category:Series 40 Developer Platform 1.1]] [[Category:Series 40 Developer Platform 2.0]]

Symbian: [[Category:Symbian]]
[[Category:S60 1st Edition]] [[Category:S60 2nd Edition (initial release)]] [[Category:S60 2nd Edition FP1]] [[Category:S60 2nd Edition FP2]] [[Category:S60 2nd Edition FP3]]
[[Category:S60 3rd Edition (initial release)]] [[Category:S60 3rd Edition FP1]] [[Category:S60 3rd Edition FP2]]
[[Category:S60 5th Edition]]
[[Category:Symbian^3]] [[Category:Symbian Anna]] [[Category:Nokia Belle]]

Add categories below using category selector.

616 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×