×
Namespaces

Variants
Actions
(Difference between revisions)

Template universal app for video recording with MediaCapture using Imaging SDK Filters

From Nokia Developer Wiki
Jump to: navigation, search
leemcpherson (Talk | contribs)
(Leemcpherson -)
leemcpherson (Talk | contribs)
(Leemcpherson -)
Line 1: Line 1:
 
[[Category:Draft]]
 
[[Category:Draft]]
 +
{{Note|This is an entry in the [[Nokia Original Effect Wiki Challenge 2014Q2]]}}
 +
 
This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied.  The article will also describe key sections of code that are requirements for creating your own version of this app.
 
This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied.  The article will also describe key sections of code that are requirements for creating your own version of this app.
  
''Enter article metadata as described below. Note that this template can be placed anywhere in the article. Do not remove parameters that you do not use''
+
 
{{ArticleMetaData <!-- v1.3 -->
+
{{ArticleMetaData v1.0
 
|sourcecode= <!-- Link to example source code e.g. [[Media:The Code Example ZIP.zip]] -->
 
|sourcecode= <!-- Link to example source code e.g. [[Media:The Code Example ZIP.zip]] -->
 
|installfile= <!-- Link to installation file (e.g. [[Media:The Installation File.wgt]]) -->
 
|installfile= <!-- Link to installation file (e.g. [[Media:The Installation File.wgt]]) -->
|devices= <!-- Devices tested against - e.g. Nokia Lumia 928, Nokia Asha 501) -->
+
|devices= Nokia Lumia 920, Nokia Lumia 820, Windows 8.1 PC
|sdk= <!-- SDK(s) built and tested against (e.g. Windows Phone 8.0 SDK) -->
+
|sdk= Windows Phone 8.1, Windows 8.1
|dependencies= <!-- Any other/external dependencies e.g.: Google Maps Api v1.0 -->
+
|dependencies= Nokia Imaging SDK
 
|signing=<!-- Special Signing requirements -->
 
|signing=<!-- Special Signing requirements -->
|capabilities= <!-- Required capabilities for code (e.g. ID_CAP_LOCATION, ID_CAP_NETWORKING) -->
+
|capabilities= webcam, microphone
 
|language= <!-- Language category code for non-English topics - e.g. Lang-Chinese -->
 
|language= <!-- Language category code for non-English topics - e.g. Lang-Chinese -->
 
|translated-by= <!-- [[User:XXXX]] -->
 
|translated-by= <!-- [[User:XXXX]] -->
Line 19: Line 21:
 
|update-by= <!-- After significant update: [[User:username]]-->
 
|update-by= <!-- After significant update: [[User:username]]-->
 
|update-timestamp= <!-- After significant update: YYYYMMDD -->
 
|update-timestamp= <!-- After significant update: YYYYMMDD -->
|creationdate= <!-- Format YYYYMMDD -->
+
|creationdate= 20140607
|author= leemcpherson  
+
|author= [[User:leemcpherson]]
 
}}
 
}}
  
Line 206: Line 208:
 
}
 
}
 
</code>
 
</code>
 +
 +
== C# Universal App ==
 +
=== Preparing the main App ===
 +
In order to use the MF Transform that was just created, there are two steps that must be taken.
 +
# The C++ app must be referenced in the C# app.  WP8.1 to WP8.1 and Win8 to Win8.
 +
# The MF Transform must be registered. 
 +
 +
In a desktop app, this would be done in the registry.  In a WinRT app, this is done in the Manifest file in an XML editor.  Right-click on the Package.appxmanifest file in the WP8.1 app, select "Open with...", and choose an XML editor.  In between Applications and Capabilities, insert:
 +
<code xml>
 +
<Extensions>
 +
    <Extension Category="windows.activatableClass.inProcessServer">
 +
      <InProcessServer>
 +
        <Path>ImagingEffects.WindowsPhone.dll</Path>
 +
        <ActivatableClass ActivatableClassId="ImagingEffects.ImagingEffect" ThreadingModel="both" />
 +
      </InProcessServer>
 +
    </Extension>
 +
  </Extensions>
 +
</code>
 +
The corresponding code for Win8 app should just change WindowsPhone.dll to Windows.dll. 
  
 
== Summary ==
 
== Summary ==

Revision as of 06:17, 8 June 2014

Note.pngNote: This is an entry in the Nokia Original Effect Wiki Challenge 2014Q2

This article provides an example app which is intended to be used as a template for applying filters to a real-time video with the ability to record video with the filters applied. The article will also describe key sections of code that are requirements for creating your own version of this app.


Template:ArticleMetaData v1.0

Contents

Introduction

With the introduction of universal apps and WinRT for Windows Phone 8.1, the Silverlight/WP8.0 method of accessing the camera has changed. Within the latest Nokia Imaging SDK is a IImageProvider called CameraPreviewImageSource which allows you to process frames from a video stream before capturing a final image to process. However, unless you were to provide your own video encoder, you cannot record the video being output by the CameraPreviewImageSource.

WinRT provides a MediaCapture class which provides video preview, image capture, and video record capabilities. The preview can be output to a new XAML class called a CaptureElement. There is no traditional way to apply the Nokia Imaging SDK to the preview (or record) output of the MediaCapture class. However, the MediaCapture class will accept Media Foundation Transforms through the AddEffectAsync method. This article will describe the key sections of code that are required to add a Media Foundation Transform to the app and how to incorporate the Nokia Imaging SDk into it.

Many parts of this app were inspired by or even pulled directly from code provided by other developers on this site.

References coming soon....

C++ Universal Component

The Media Foundation Transform must be coded in C++ and the interfaces that must be implemented within the transform class are quite complicated. Thankfully, there exist several samples of a Media Foundation transform that do not require much modification for our purposes. A good sample to start with is the MediaExtensions sample from the Univeral Apps sample set from Microsoft. The sample contains a GrayscaleTransform component from which this app's transform is based.

http://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096#content

Preparing the C++ Component

Add Media Foundation Libraries

Next, the Media Foundation libraries need to be added to the linker. For each C++ project (WP8.1 and Win8), right-click on the project name and select properties. Expand the Linker group and select Input. Change the Configuration (at the top) from Active to All Configurations and change Platform from Active to All Platforms so that the following changes are applied to every configuration and platform.

To the end of the "Additional Dependencies" block, add mfplat.lib and mfuuid.lib separated by a semi-colon(;).

Add Common Folder from sample & Include Directories

At the root of the sample app (this app or the MediaExtensions app) is a folder labeled Common which contains several C++ header files (AsyncCB.h, CritSec.h, ExtensionsDefs.h, etc) . These are necessary for the media transform code to compile. Copy the entire Common folder to the root of your own solution.

Next, go to the property page for each C++ project in your app and change the configuration and platform to all as before. Expand the C/C++ menu and click on General. To the end of the Addition Include Directories, add a semicolon (;) and the path "..\..\Common" so that VS2013 can find the header files we just copied.

Modify pch.h

Replace the contents of your pch.h file with the following:

#pragma once
 
#include <collection.h>
#include <ppltasks.h>
 
// Windows Header Files:
#include <windows.h>
#include <mfapi.h>
#include <mfidl.h>
#include <mfapi.h>
#include <mferror.h>
#include <d3d11.h>
#include <D2d1helper.h>
 
#include <assert.h>
 
#include <tchar.h>
#include <Strsafe.h>
#include <objidl.h>
#include <new>
 
#include <wrl\client.h>
#include <wrl\implements.h>
#include <wrl\ftm.h>
#include <wrl\event.h>
#include <wrl\wrappers\corewrappers.h>
#include <windows.media.h>
 
#include <ppltasks.h>
 
#include <ExtensionsDefs.h>
 
using namespace Platform;
using namespace Microsoft::WRL;
using namespace Microsoft::WRL::Wrappers;


Add the Nokia Imaging SDK

Add the Nokia Imaging SDK to all projects in the solution using NuGet. You do NOT have to manually add a reference to the Imaging SDK to the C++ projects. They will not appear as a reference under the properties, but they will be included in the project anyways.

Add NativeBuffer.h

This class is copied entirely from galazzo 's article here: http://developer.nokia.com/community/wiki/Nokia_Imaging_SDK_in_native_code#Wrapping_buffers_into_a_Bitmap

The Transform Class

To the Shared C++ project Add both a .h header and .cpp c++ file. Creating these two files from scratch is beyond the scope of this article. Instead copy the contents of the ImagingEffect.cpp and ImagingEffect.h from the sample to your project. (These files are too large to post directly in this article.) There are several functions within this class that this article will highlight in the next section so that you can modify or add to this class.

Static Transform Functions for NV12 & YUY2

Most of the work of the transform happens within the non-class functions TransformImage_NV12 and TransformImage_YUY2. These functions transform the video frame at the pointer pSrc and copy the new, transformed image to the pointer pDest. Another important variable within these functions is the pointer to a wide string, filterParams. This string contains parameters from the C# project that define which filters to use and what parameters to use with them.

auto size = Windows::Foundation::Size(dwWidthInPixels, dwHeightInPixels);
auto totalbytes = (int)dwHeightInPixels * (int)dwWidthInPixels * 2; //each macropixel of 4 bytes creates 2 pixels (YUYV)
Bitmap^ m_BitmapToProcess = AsBitmapYUY2(pSrc, (unsigned int)size.Width, (unsigned int)size.Height);
ApplyImagingFilters(m_BitmapToProcess, pDest, totalbytes, filterParams, ColorMode::Yuv422_Y1UY2V);

In order to use the Nokia Imaging SDK, the first step is to transform the raw byte data into a Bitmap. This is accomplished for the NV12 function using another function pulled directly from galazzo 's article again [[1]]. For YUY2 to Bitmap, the function was modified for the YUY2 format.

ApplyingImagingFilters function

The use of the Nokia Imaging SDK in a C++ app is similar to a C# app. First, a source must be created. Then, several Effects or Filters are applied. Finally, the result is rendered to a designated target. In this app, the source is a BitmapImageSource that uses the previously created Bitmap in its constructor.

        Nokia::Graphics::Imaging::BitmapImageSource^ bis = ref new BitmapImageSource(sourceBitmap);
FilterEffect^ filterEffect = ref new FilterEffect(bis);
 
IVector<IFilter^>^ Filters = ref new Platform::Collections::Vector<IFilter^>();
 
wchar_t* token = NULL;
wchar_t *next_token = NULL;
std::wstring s = std::wstring(filterParams);
 
token = wcstok_s(&s[0], L";", &next_token);
while (token != NULL)
{
auto filter = GetFilter(token);
Filters->Append(filter);
token = wcstok_s(NULL, L";", &next_token);
}
 
filterEffect->Filters = Filters;
auto renderer = ref new Nokia::Graphics::Imaging::BitmapRenderer(filterEffect, colorMode);
 
auto renderOp = renderer->RenderAsync();
auto renderTask = create_task(renderOp);
 
renderTask.then([pDest, totalBytes](Nokia::Graphics::Imaging::Bitmap^ bitmap)
{
auto count = bitmap->Buffers->Length;
auto data = FromIBuffer(bitmap->Buffers[0]->Buffer);
//auto data = GetPointerToPixelData(bitmap->Buffers[0]->Buffer);
CopyMemory(pDest, data, totalBytes);
}).wait();

Next, filters are added to a FilterEffect via a loop run using the filterParams parameter. This will be described in the next section.

Finally, a BitmapRenderer is created using a specified ColorMode so that the output matches the format of the input. In the Nokia Imaging SDK, NV12 matches to Yuv420Sp and YUY2 matches to Yuv422_Y1UY2V. A pointer to the raw byte data is created from the IBuffer internal to the newly created Bitmap. This byte data is then copied to the pointer, pDest.

Note.pngNote: NV12 and YUY2 are two different formats for video that uses luminance and chroma data instead of straight RGBA data. These are the only two formats used by the Nokia Lumia 920 and the LifeCam HD-5000 webcam.

Parsing the Filter Parameters & Creating the Filters

In this app, the filter parameters are passed solely as text data. Since C++ does not have Reflection function like C# does, the strings have to be matched to class names manually. In the string parameter, filterParamters, filters are separated by semicolons (;) and the specific arguments are separated by comma (,). Within the GetFilter function, each filter name is matched up and the number of parameters is counted to determine which filter is returned to the main function

Warning.pngWarning: Currently, only LomoFilter and part of SolarizeFilter are ready to use with this solution. The template for using any other filter can be copied directly from these two.

CImagingEffect::SetProperties

This function allows the C++ transform to receive data from the C# portion of the app. When the transform is applied to the MediaCapture class via AddEffectAsync, only a string variable is passes and no instance of the transform is returned to the C# portion of the app. Therefore, one cannot pass data to the transform via conventional means. However, one of the parameters of the AddEffectAsync method takes an IPropertySet interface. This is essentially a dictionary of KeyValuePairs that must use WinRT-types.

This parameter comes through to the MF Transform through the SetProperties function.

        HRESULT hr = S_OK;
try
{
IPropertySet^ properties = reinterpret_cast<IPropertySet^>(pConfiguration);
IVector<String^>^ filterList = safe_cast<IVector<String^>^>(properties->Lookup(L"filterList"));
auto size = filterList->Size;
wchar_t temp[100] = { 0 };
for (int i = 0; i < size; i++)
{
auto commandKey = filterList->GetAt(i);
auto commandArray = safe_cast<String^>(properties->Lookup(commandKey));
wchar_t* tempWide = const_cast< wchar_t* >(commandArray->Data());
wcscat_s(temp, commandKey->Data());
wcscat_s(temp, L",");
wcscat_s(temp, tempWide);
}
m_altFilterParams = std::wstring(temp);
}
catch (Exception ^exc)
{
hr = exc->HResult;
}
return hr;

This function parses the content of the IPropertySet and converts it to a std::wstring in the class called m_altFilterParams. However, the transform functions cannot access this class-member variable, so it must be passed via a pointer as an argument to the transform functions. This is done in OnProcessOutput:

        // Invoke the image transform function.
assert(m_pTransformFn != nullptr);
if (m_pTransformFn)
{
auto s = m_altFilterParams;
(*m_pTransformFn)(m_rcDest, outputLock.GetTopRow(), outputLock.GetStride(), inputLock.GetTopRow(), inputLock.GetStride(), m_imageWidthInPixels, m_imageHeightInPixels, m_altFilterParams.data());
}
else
{
ThrowException(E_UNEXPECTED);
}

C# Universal App

Preparing the main App

In order to use the MF Transform that was just created, there are two steps that must be taken.

  1. The C++ app must be referenced in the C# app. WP8.1 to WP8.1 and Win8 to Win8.
  2. The MF Transform must be registered.

In a desktop app, this would be done in the registry. In a WinRT app, this is done in the Manifest file in an XML editor. Right-click on the Package.appxmanifest file in the WP8.1 app, select "Open with...", and choose an XML editor. In between Applications and Capabilities, insert:

<Extensions>
<Extension Category="windows.activatableClass.inProcessServer">
<InProcessServer>
<Path>ImagingEffects.WindowsPhone.dll</Path>
<ActivatableClass ActivatableClassId="ImagingEffects.ImagingEffect" ThreadingModel="both" />
</InProcessServer>
</Extension>
</Extensions>

The corresponding code for Win8 app should just change WindowsPhone.dll to Windows.dll.

Summary

Remove Category:Draft when the page is complete or near complete


The "platform categories" will be displayed here in preview only - Copy paste relevant categories into text here

Version Hint

Windows Phone: [[Category:Windows Phone]]
[[Category:Windows Phone 7.5]]
[[Category:Windows Phone 8]]

Nokia Asha: [[Category:Nokia Asha]]
[[Category:Nokia Asha Platform 1.0]]

Series 40: [[Category:Series 40]]
[[Category:Series 40 1st Edition]] [[Category:Series 40 2nd Edition]]
[[Category:Series 40 3rd Edition (initial release)]] [[Category:Series 40 3rd Edition FP1]] [[Category:Series 40 3rd Edition FP2]]
[[Category:Series 40 5th Edition (initial release)]] [[Category:Series 40 5th Edition FP1]]
[[Category:Series 40 6th Edition (initial release)]] [[Category:Series 40 6th Edition FP1]] [[Category:Series 40 Developer Platform 1.0]] [[Category:Series 40 Developer Platform 1.1]] [[Category:Series 40 Developer Platform 2.0]]

Symbian: [[Category:Symbian]]
[[Category:S60 1st Edition]] [[Category:S60 2nd Edition (initial release)]] [[Category:S60 2nd Edition FP1]] [[Category:S60 2nd Edition FP2]] [[Category:S60 2nd Edition FP3]]
[[Category:S60 3rd Edition (initial release)]] [[Category:S60 3rd Edition FP1]] [[Category:S60 3rd Edition FP2]]
[[Category:S60 5th Edition]]
[[Category:Symbian^3]] [[Category:Symbian Anna]] [[Category:Nokia Belle]]

Add categories below using category selector.

2431 page views in the last 30 days.
×