×
Namespaces

Variants
Actions

Template app for developing Imaging SDK Effects

From Nokia Developer Wiki
Jump to: navigation, search
Featured Article
02 Jun
2014

This article provides an example app which is intended to be used as a template for creating and testing custom effects, custom filters, and filter recipes.

Announcements.pngNokia Original Imaging Effect Wiki Challenge (21 Jul 2014): Winners have been announced for the Nokia Original Imaging Effect Wiki Challenge. The Tweet-Off to decide which authors win a Nokia Lumia 630 starts today. Follow the Nokia Developer Twitter feed and retweet your favorite entries.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code ExampleTested with
SDK: Windows Phone 8.0 SDK
Devices(s): Lumia 1020, Lumia 520
Compatibility
Platform(s):
Windows Phone 8
Dependencies: Nokia Imaging SDK 1.2.115
Platform Security
Capabilities: ID_CAP_ISV_CAMERA, ID_CAP_MEDIALIB_PHOTO,ID_FUNCCAP_EXTEND_MEM
Article
Created: yan_ (25 May 2014)
Last edited: BuildNokia (06 Jun 2014)

Contents

Introduction

The Nokia Imaging SDK provides several interfaces that can be used to customize the rendering pipeline. Developers can combine built-in filters to create filter recipes, or create completely new custom filters and custom effects. They can create all of these types of recipes and filters in both C# and CX/C++.

This article provides a code example which demonstrates most of these types of filters and effects, and which provides an environment to test your recipe with a picture and the camera viewfinder. Specifically, the code includes:

  • An example of custom filter in CX/C++.
  • An example of custom filter in C#.
  • An example of custom effect in CX/C++.
  • An example of custom effect in C#.
  • A demonstration class which can embed a complex effects pipeline (a recipe, including custom and built in filters).
  • An application to test the recipe with a picture or camera viewfinder.

The example app may be used by developers as a template for creating and testing custom effects, custom filters, and filter recipes. The code is released under MIT license and is available as a module of the wp8-sample repository on Github and as a zip archive on this Wiki: File:ImagingSDKFilterTemplate.zip

Custom Effect

A custom effect processes an entire image at once. This makes it possible to access all pixels at once, which can be useful for some effects. However, a side effect of this is that custom effects are sometimes memory inefficient.

C++/CX

To implement a custom effect with C++/CX, you must implement an ICustomEffect interface and use it with DelegatingEffect.

Warning.pngWarning: DelegatingEffect use a weak pointer on ICustomEffect. You are responsible of ICustomEffect life.

The ICustomFilter interface functions are as follows:

  • LoadAsync: Used to load effect data. To load data asynchronously, use concurrency::create_async. If you don't need to load data, return nullptr.
  • ProvideSourceBuffer: The Imaging SDK copies input pixels to the memory allocated by this method.
  • ProvideTargetBuffer: Your effect should copy output pixels to the memory allocated by this method.
  • Process: Must contain your algorithm. This method is called to process the image.

The Imaging SDK uses the windows::storage::streams::IBuffer interface to transfer memory between entities. A class which implements the IBuffer interface must implement the IBufferByteAccess COM interface. It's highly recommended that you use Windows::Storage::Streams::Buffer. When you instantiate a Buffer, you must set its capacity using the Length property as follows or the SDK will think it's empty and throw an exception.

m_sourceBuffer = ref new Windows::Storage::Streams::Buffer (4*imageSize.Width*imageSize.Height);
m_sourceBuffer->Length = m_sourceBuffer->Capacity;

Note.pngNote: A pixel is a BGRA type with a size of 4 bytes. Therefore, your buffer must be sized 4*width*height bytes.

Warning.pngWarning: Windows phone is Little Endian. When you manipulate a byte array, the pixel component order is BGRA. However, when you manipulate an int or uint array, the pixel component order is ARGB.

Access to the IBuffer memory is provided by the IBufferByteAccess COM interface. The simplest way to do this is by following the approach described in Obtaining pointers to data buffers (C++/CX).

byte* GetPointerToPixelData(Windows::Storage::Streams::IBuffer^ pixelBuffer, unsigned int *length = nullptr)
{
if (length != nullptr)
{
*length = pixelBuffer ->Length;
}
// Query the IBufferByteAccess interface.
Microsoft::WRL::ComPtr< Windows::Storage::Streams::IBufferByteAccess> bufferByteAccess;
reinterpret_cast<IInspectable*>( pixelBuffer)->QueryInterface(IID_PPV_ARGS(&bufferByteAccess));
 
// Retrieve the buffer data.
byte* pixels = nullptr;
bufferByteAccess->Buffer(&pixels);
return pixels;
}

Tip.pngTip: The Effect/MyEffect class in the CPP project library provides an implementation of ICustomEffect in C++/CX.

C#

In C#, the Imaging SDK provides the class CustomEffectBase. Inheriting from this is highly recommended. When you inherit, you have just to re-implement the OnProcess function so it follows your effect algorithm.

Note.pngNote: CustomEffectBase constructors have a Boolean called isInplace. If you set this to True, the same memory allocation will be used for input and output.

Tip.pngTip: The Effect/MyEffect class in the CSharp project library provides an implementation of CustomEffectBase in C#.

Custom Filter

A filter processes pixels by tiles. This means that the implementation only has access to a small part of the image at any given time. A filter is very memory efficient, compared to an effect. It's used like other SDK filters: you add it to a IFilter container and use FilterEffect.

C++/CX

To implement a custom filter with C++/CX, you must implement an ICustomFilter interface and use it with DelegatingFilter.

Warning.pngWarning: DelegatingFilter use a weak pointer on ICustomFilter. You are responsible of ICustomFilter life.

The ICustomFilter interface functions are as follows:

  • BeginProcessing: Takes an ICustomFilterRequest as a parameter. ICustomFilterRequest contains information about the block. Instantiates an ICustomFilterResponse, which contains the memory used by the filter. This method is called before pixel processing.
  • PrepareAsync: Used to load filter data. To load data asynchronously, use concurrency::create_async. If you don't need to load data, return nullptr.
  • ProcessBlock: Must contain your filter algorithm, which should be processed on a block of pixels.

Note.pngNote: SDK memory access is explained above.

Tip.pngTip: The Effect/MyEffect class in the CPP project library provides an implementation of ICustomEffect in C++/CX.

C#

In C#, the Imaging SDK provides the class CustomFilterBase. Inheriting from this is highly recommended. When you inherit, you have just to re-implement the OnProcess function so it follows your effect algorithm.

The CustomFilterBase constructor has three parameters :

  • blockMargins: If you need an overlap between input block, specify the size of the border pixel you need for your filter. By default this is set to 0.
  • wrapBorders: Specify whether the pixel data for the margin should be taken from inside the image (true) or whether the margin pixels should be transparent black (false).
  • supportedColorModes: a filter can works with Bgra or Ayuv pixels. It's recommended to use Brga color modes.

Tip.pngTip: The Filter/MyFilter class in the CSharp project library provides an implementation of CustomFilterBase in C#.

RecipeTemplate

When you code the rendering pipeline, your code between the Image Source and the Rendering could become complicated due to daisy chaining and instantiating the FilterEffect and set parameters, among others.

To render this part reusable, use the provided sample in the Recipe/RecipeTemplate class in the CSharp project library. It implements these interfaces:

  • IImageConsumer: recipe input functions
  • IImageProvider: recipe output functions
  • IDisposable: dispose effect elements
Recipe template corresponding to the central block.

When you implement your recipe, you must call SetPipelineBeginEnd once you have instantiated or modified recipe elements. This method indicates to the RecipeTemplate which element is the recipe input and which element is the recipe output. (The input and output can be the same element). It is used like an SDK Effect and can contain complex pipeline creation.

For example:

  1. public class RecipeDaisyChain : CSharp.Recipe.RecipeTemplate
  2. {
  3. 	HdrEffect effect_1;
  4. 	FilterEffect    effect_2;
  5. 	public RecipeDaisyChain(IImageProvider source, double factor)
  6. 		:base(source)
  7. 	{
  8. 		//first effect pipeline element
  9. 		effect_1 = new HdrEffect(source);
  10.  
  11. 		//second effect pipeline element
  12. 		effect_2 = new FilterEffect(effect_1);
  13.  
  14. 		if(factor>2) factor = 2;
  15. 		//filters creation
  16. 		effect_2.Filters = new IFilter[] { new HueSaturationFilter(-1 + factor, 0), new LomoFilter() };
  17.  
  18. 		//set the input and output element
  19. 		SetPipelineBeginEnd(effect_1, effect_2);
  20. 	}
  21. 	 #region IDispose
  22.         // Flag: Has Dispose already been called? 
  23.         bool disposed = false;
  24.  
  25.         // Protected implementation of Dispose pattern. 
  26.         protected override void Dispose(bool disposing)
  27.         {
  28.             if (disposed)
  29.                 return;
  30.  
  31.             if (disposing)
  32.             {
  33.                 if (effect_1 != null)
  34.                 {
  35.                     effect_1.Dispose();
  36.                     effect_1 = null;
  37.                 }
  38.                 if (effect_2 != null)
  39.                 {
  40.                     effect_2.Dispose();
  41.                     effect_2 = null;
  42.                 }
  43.             }
  44.             disposed = true;
  45.             // Call base class implementation. 
  46.             base.Dispose(disposing);
  47.         }
  48.         #endregion
  49. }

With this recipe, this pipeline code:

  1. var filters = new IFilter[] { 
  2. new HueSaturationFilter(-1 + factor, 0), 
  3. new LomoFilter() };
  4.  
  5. using (var source = new StreamImageSource(stream))
  6. using (var  effect_1 = new HdrEffect(source))
  7. using (var  effect_2 = new FilterEffect(effect_1) { Filters = filters })
  8. using (var renderer = new WriteableBitmapRenderer(effect_2, writeableBitmap))
  9. {
  10.     await renderer.RenderAsync();
  11. }

becomes:

  1. using (var source = new StreamImageSource(stream))
  2. using (var recipe = new RecipeDaisyChain (source))
  3. using (var renderer = new WriteableBitmapRenderer(MyRecipe, writeableBitmap))
  4. {
  5.     await renderer.RenderAsync();
  6. }

Tip.pngTip: You can find several recipe implementations in ImagingSDKFilterTemplate/Recipe.

RecipeTemplate use the recommended IDispose implementation explained in this article: IDisposable Interface. When you inherit RecipeTemplate you must override Dispose(bool) function like this :

class DerivedClass : BaseClass
{
// Flag: Has Dispose already been called?
bool disposed = false;
 
// Protected implementation of Dispose pattern.
protected override void Dispose(bool disposing)
{
if (disposed)
return;
 
if (disposing) {
// Free any other managed objects here.
//
}
 
// Free any unmanaged objects here.
//
disposed = true;
// Call base class implementation.
base.Dispose(disposing);
}
}

Sample

The sample application provides an environment to test your recipe. It is composed of two pages. The main page applies the recipe to a picture. You can use the slider to change recipe parameters. The application uses the Interactive state machine to keep the GUI responsive.

MainPage
MainPage menu

The Main Page Application Bar actions are:

  • image: opens the image selection.
  • live: opens a Live Page to use the recipe with the viewfinder.
  • info: hides/displays the top left panel, which contains information such as memory use.
  • bench: launches the rendering several times with the full image resolution to estimate rendering times.
  • save: saves the current picture.

The Live Page applies the recipe effect on the camera viewfinder. You can use the slider to change recipe parameters. The page supports all orientations.

Live page
Live page menu

The Live Page Application Bar actions are:

  • Fast shot: take preview image and apply recipe effect. Picture is saved in Saved Pictures folder.
  • shot: take a picture and apply recipe. Picture is saved in Saved Pictures folder.
  • switch: switches which camera is being used if a device has two cameras.
  • info: hides/displays the top left panel, which contains information such as memory use.

Interface your recipe

The application uses the ImagingSDKFilterTemplate/Recipe/RecipeFactory to instantiate the recipe for rendering. This class is a singleton composed of:

  • Param: property used as recipe parameter. Its value corresponds to the slider value in Main Page and Live Page.
  • CreatePipeline: function which instantiates the recipe.
  1. public class RecipeFactory
  2. {
  3. 	#region singleton part
  4. 	   private static Lazy<RecipeFactory> mSingleton =  new Lazy<RecipeFactory>(true);
  5. 	   static public RecipeFactory Current { get { return mSingleton.Value; } }
  6. 	   public RecipeFactory() { }
  7. 	#endregion
  8.  
  9.  
  10. 	public double Param { get; set; }
  11.  
  12. 	public IImageProvider CreatePipeline(IImageProvider source)
  13. 	{
  14. 		var value = 0.5 + 3 * Param;
  15. 		//return new RecipeCSharpEffect(source, value); //Recipe with C# custom effect
  16. 	        //return new RecipeCSharpFilter(source, value);//Recipe with C# custom Filter
  17. 	        // return new RecipeCPPEffect(source, value);   //Recipe with CPP custom effect   
  18. 		//return new RecipeCPPFilter(source, value);     //Recipe with CPP custom effect  
  19. 		return new RecipeDaisyChain(source, value);    //Recipe Daisy chain 
  20. 	}
  21.  
  22. }

To create a new filter recipe, you just have to put your code into CreatePipeline.

Note.pngNote: If you don't use RecipeTemplate, you must return the last element of your effect pipeline. If you use a daisy chain, you could get a memory error, like "element could not be disposed."

When the slider is moved in the MainPage, the callback must

  1. set the new value to the RecipeFactory
  2. call requestProcessing() to schedule rendering with the interactive state machine.
private void FilterParam_ValueChanged(object sender, RoutedPropertyChangedEventArgs<double> e)
{
//update factory param value with slider value
RecipeFactory.Current.Param = e.NewValue;
//schedule rendering
requestProcessing();
}

If you add a GUI element, you must implement these steps.

When the slider is moved to the Live Page, the callback must:

  1. set the new value to the RecipeFactory
  2. call UpdateEffect(). This function indicates that the recipe should be recreated for the next preview image.
private void FilterParam_ValueChanged(object sender, RoutedPropertyChangedEventArgs<double> e)
{
//update factory param value with slider value
RecipeFactory.Current.Param = e.NewValue;
//indication recipe should be updated
UpdateEffect();
}

Interesting links

This page was last modified on 6 June 2014, at 20:25.
735 page views in the last 30 days.
×