×
Namespaces

Variants
Actions
Revision as of 04:32, 14 October 2013 by hamishwillee (Talk | contribs)

Filter and Custom Filter Management Framework for the Nokia Imaging SDK

From Nokia Developer Wiki
Jump to: navigation, search

This article explains how to use Nokia Imaging SDK with custom filters and image processors.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code Example
Installation file: File:Imaging Project.zip (ARM and x86)
Tested with
SDK: Windows Phone 8.0 SDK, Nokia Imaging SDK Beta 1
Devices(s): Windows Phone 8 Emulator, Nokia Lumia 920
Compatibility
Platform(s):
Windows Phone 8
Dependencies: Nokia Imaging SDK
Article
Created: Engin Kırmacı (31 Aug 2013)
Last edited: hamishwillee (14 Oct 2013)

Contents

Introduction

The Nokia Imaging SDK provides more than 50 filters and effects for developers to use and combine them to create new filters. At some point these filters doesn’t meet developer’s needs.

This article mainly explains a helper library that helps to combine Nokia Imaging SDK filters, custom filters, and image processors with some example filters and processors. Moreover some usefull methods covered in this article.

Architecure

Interfaces

  • ICustomFilter

Custom filters and image processors must use this interface. It needed to determine it'll be processed by Nokia Imaging SDK or not.

public interface ICustomFilter {  }
  • IPerPixelManipulation

It used for basic filtering, which process image pixel by pixel. "color" parametre is Alpha, Red, Green and Blue byte values of pixel.

public interface IPerPixelManipulation
{
byte[] CalculatePixel(byte[] color);
}
  • IProcesors

Advance filtering, which process pixels together such as Convolution-based filters

public interface IProcessors
{
WriteableBitmap Process(WriteableBitmap pixels);
}

ImagingSession.cs

ImagingSession class combines Nokia Imaging SDK and custom filter. It stores original image in "jpegData" as IBuffer which provided by Nokia Imaging SDK and output is "Image" as WriteableBitmap so that custom filters can access pixels data.

public class ImagingSession
{
EditingSession _editsession { get; set; }
IBuffer _jpegData { get; set; }
 
 
public List<IFilter> History { get; set; }
public WriteableBitmap Image { get; set; }

To start session, there is two options;

  • Load image to WriteableBitmap then start session with it
  • Load image directly using MemoryStream (MemoryStream method implemented for getting stream from camera, but can be used for different scenarios)

When starting ImagingSession, it startsEditingSession and original state to IBuffer which are provided by Nokia Imaging SDK. Then renders image to WriteableBitmap to be used further.

async public Task StartSession(WriteableBitmap writeableBitmap)
{
if (_editsession != null)
_editsession.Dispose();
 
Image = writeableBitmap;
 
using (MemoryStream ms = new MemoryStream())
{
System.Windows.Media.Imaging.Extensions.SaveJpeg(Image, ms, (int)Image.PixelWidth, (int)Image.PixelHeight, 0, 100);
 
_jpegData = ms.GetWindowsRuntimeBuffer();
}
 
_editsession = new EditingSession(_jpegData);
 
await _editsession.RenderToWriteableBitmapAsync(Image);
}
 
async public Task StartSession(MemoryStream stream)
{
if (_editsession != null)
_editsession.Dispose();
 
_jpegData = stream.GetWindowsRuntimeBuffer();
 
stream.Close();
stream = null;
 
_editsession = new EditingSession(_jpegData);
 
Image = new WriteableBitmap((int)_editsession.Dimensions.Width, (int)_editsession.Dimensions.Height);
 
await RenderImage(Size.Empty);
}

Adding filter to session,

public void AddFilter(IFilter filter)
{
if (filter != null)
History.Add(filter);
}

Applying Filters and Processors to session;

async public Task ApplyFilters()
{
IFilter[] tmpHistory = new IFilter[History.Count];
History.CopyTo(tmpHistory);
 
int position = 0;
while (position < tmpHistory.Length)
{
var currFilter = tmpHistory[position];

There is three conditions;

  • If filter is not ICustomFilter, the following code passes filter to EditingSession
 if (currFilter is ICustomFilter)
{
//some code here
}
else
{
for (int i = position; i < tmpHistory.Length; i++)
{
var tmpCurrFilter = tmpHistory[i];
 
if (!(tmpCurrFilter is IPerPixelManipulation) && !(tmpCurrFilter is IProcessors))
{
_editsession.AddFilter(tmpCurrFilter);
position++;
}
else
break;
}
 
await RenderImage(Size.Empty);
}
  • If filter is ICustomFilter and IPixelManipulation,
 if (currFilter is ICustomFilter)
{
if (currFilter is IPerPixelManipulation)
{
List<IFilter> perPixelManipulationFilters = new List<IFilter>();
 
for (int i = position; i < tmpHistory.Length; i++)
{
var tmpCurrFilter = tmpHistory[i];
 
if (tmpCurrFilter is IPerPixelManipulation)
{
perPixelManipulationFilters.Add(tmpCurrFilter);
position++;
}
else
break;
}
 
for (int x = 0; x < Image.PixelWidth; x++)
for (int y = 0; y < Image.PixelHeight; y++)
{
var color = Image.GetPixel(x, y).ExtractColor();
foreach (IPerPixelManipulation filter in perPixelManipulationFilters)
color = filter.CalculatePixel(color);
 
Image.SetPixel(x, y, color.InsertColor());
}
}
//some code here
 
SetProcessedImageToSession();
}
  • If filter is ICustomFilter and IProcessors
if (currFilter is ICustomFilter)
{
if (currFilter is IPerPixelManipulation)
{
//some code here
}
else if (currFilter is IProcessors)
{
Image = ((IProcessors)currFilter).Process(Image);
position++;
}
 
SetProcessedImageToSession();

Rendering image;

  • Custom filters immediately sets filtered pixels to WriteableBitmap, after that to use Nokia Imaging SDK filters, it needs to initialize EditSession according to filtered image. The following code does this.
public void SetProcessedImageToSession()
{
IBuffer tmp = null;
using (MemoryStream ms = new MemoryStream())
{
System.Windows.Media.Imaging.Extensions.SaveJpeg(Image, ms, (int)Image.PixelWidth, (int)Image.PixelHeight, 0, 100);
 
tmp = ms.GetWindowsRuntimeBuffer();
}
 
_editsession = new EditingSession(tmp);
}
  • In the other hand, Nokia Imaging SDK filters must be rendered to WriteableBitmap.

Note: The following code also can change image size

async public Task RenderImage(Size size)
{
if (Image == null)
{
if (size != Size.Empty)
Image = new WriteableBitmap((int)size.Width, (int)size.Height);
else
Image = new WriteableBitmap((int)_editsession.Dimensions.Width, (int)_editsession.Dimensions.Height);
}
else if (size != Size.Empty && size != new Size(Image.PixelWidth, Image.PixelHeight))
Image = new WriteableBitmap((int)size.Width, (int)size.Height);
 
await _editsession.RenderToWriteableBitmapAsync(Image);
 
GC.Collect();
}

Undoing filters;

  • If last applied filter is a Nokia Imaging SDK filter; then it calls EditSession Undo() method and renders image
  • If last applied filter is an ICustomFilter; then it calls ApplyFilters() method for filters expect the last filter in History.
async public Task Undo()
{
if (History.Count == 0)
return;
 
var lastFilter = History.Last();
if (lastFilter != null)
{
History.Remove(lastFilter);
 
if (lastFilter is ICustomFilter)
{
SetOrginalImageToSession();
await RenderImage(Size.Empty);
 
await ApplyFilters();
}
else
{
_editsession.Undo();
 
await RenderImage(Size.Empty);
}
}
}

How to create custom filters and processors

Custom Filters

To create this type of filter,

  • IFilter, ICustomFilter and IPerPixelManipulation must be defined as interfaces
  • void Apply(IEditingSession editingSession), void Undo(IEditingSession editingSession) methods must be implemented in order to use IFilter
public class CustomFilter : IFilter, ICustomFilter, IPerPixelManipulation
{
//TODO implement constructor to pass in/out values
double _filterValue = 0;
double _sumColor = 0;
public CustomFilter (double filterValue, out sumColor)
{
this._filterValue = filterValue;
sumColor = this._sumColor;
}
 
public byte[] CalculatePixel(byte[] color)
{
byte a = color[ColorValues.A];
byte r = color[ColorValues.R];
byte g = color[ColorValues.G];
byte b = color[ColorValues.B];
 
//some manipulation for pixel here
 
color[ColorValues.A] = a;
color[ColorValues.R] = r;
color[ColorValues.G] = g;
color[ColorValues.B] = b;
 
return color;
}
 
public void Apply(IEditingSession editingSession)
{
throw new System.NotImplementedException();
}
 
public void Undo(IEditingSession editingSession)
{
throw new System.NotImplementedException();
}
}

Processors

To create this type of filter,

  • IFilter, ICustomFilter and IProcessors must be defined as interfaces
  • void Apply(IEditingSession editingSession), void Undo(IEditingSession editingSession) methods must be implemented in order to use IFilter
public class Processor : IFilter, ICustomFilter, IPerPixelManipulation
{
//TODO implement constructor to pass in/out values
double _filterValue = 0;
double _sumColor = 0;
public Processor (double filterValue, out sumColor)
{
this._filterValue = filterValue;
sumColor = this._sumColor;
}
 
public WriteableBitmap Process(WriteableBitmap Image)
{
WriteableBitmap pixelated = new WriteableBitmap(Image.PixelWidth, Image.PixelHeight);
 
//some manipulation for image
 
return pixelated;
}
 
public void Apply(IEditingSession editingSession)
{
throw new System.NotImplementedException();
}
 
public void Undo(IEditingSession editingSession)
{
throw new System.NotImplementedException();
}
}

CustomFilterFactory.cs

In order to use custom filters and processors, there is a helper class CustomFilterFactory. It works as a FilterFactory in Nokia Imaging SDK. After implement a custom filter, implement it in this class as below.

public class CustomFilterFactory
{
//some other custom filters and processors
 
static public IFilter CreateCustomFilter(double filterValue, out sumColor)
{
return new CustomFilter(filterValue, out sumColor);
}
}

Sample Project

MainPage.xaml

<Grid x:Name="LayoutRoot" Background="Transparent">  
<StackPanel Canvas.ZIndex="999" VerticalAlignment="Top" Width="480" Orientation="Horizontal">
<Button x:Name="loadButton" Content="Load Image" Click="Load_Click" FontSize="16" />
<Button x:Name="applyButton" Content="Apply Filters" Click="ApplyFilters_Click" FontSize="16" IsEnabled="False" />
<Button x:Name="undoButton" Content="Undo" Click="Undo_Click" FontSize="16" IsEnabled="False" />
<Button x:Name="undoAllButton" Content="Undo All" Click="UndoAll_Click" FontSize="16" IsEnabled="False" />
</StackPanel>
<Image x:Name="ImageViewer" />
</Grid>

MainPage.xaml.cs

  • Add following code at the top of page.
ImagingSession _imagingSession = new ImagingSession();
async void Load_Click(object sender, RoutedEventArgs e)
{
var photoChooserTask = new PhotoChooserTask();
photoChooserTask.Completed += photoChooserTask_Completed;
photoChooserTask.Show();
}
 
async void photoChooserTask_Completed(object sender, PhotoResult e)
{
applyButton.IsEnabled = true;
undoButton.IsEnabled = true;
undoAllButton.IsEnabled = true;
 
BitmapImage bi = new BitmapImage();
bi.SetSource(e.ChosenPhoto);
var image = new WriteableBitmap(bi);
 
await _imagingSession.StartSession(image);
bi = null;
image = null;
 
ImageViewer.Source = _imagingSession.Image;
}
  • Adding filters and applying
async void ApplyFilters_Click(object sender, RoutedEventArgs e)
{
int[] Red = null;
int[] Green = null;
int[] Blue = null;
int[] Luminance = null;
//IPixelManipulation Filter
_imagingSession.AddFilter(CustomFilterFactory.CreateHistogramFilter(out Red, out Green, out Blue, out Luminance));
 
////Nokia Imaging SDK Filters
_imagingSession.AddFilter(FilterFactory.CreateAntiqueFilter());
_imagingSession.AddFilter(FilterFactory.CreateCartoonFilter(true));
 
//Nokia Imaging SDK Filter Group
_imagingSession.AddFilter(new FilterGroup(new IFilter[]
{
FilterFactory.CreateAntiqueFilter(),
FilterFactory.CreateLomoFilter(0.7,0.7, LomoVignetting.Medium, LomoStyle.Green)
}));
 
//IPixelManipulation Filters
_imagingSession.AddFilter(CustomFilterFactory.CreateThresholdFilter(150, 100, 0));
_imagingSession.AddFilter(CustomFilterFactory.CreateColorPaletteFilter(AccentColorsPalette()));
 
//IProcessors Filter
_imagingSession.AddFilter(CustomFilterFactory.CreatePixelInterpolation(128));
 
//Nokia Imaging SDK Filter
_imagingSession.AddFilter(FilterFactory.CreateBlurFilter(BlurLevel.Blur4));
 
await _imagingSession.ApplyFilters();
ImageViewer.Source = _imagingSession.Image;
}
 
int[] AccentColorsPalette()
{
List<int> ColorList = new List<int>();
ColorList.Add(Colors.Black.GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xE3, 0xC8, 0x00).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xAA, 0x00, 0xFF).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x00, 0xAB, 0xA9).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x87, 0x79, 0x4E).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x64, 0x76, 0x87).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xE5, 0x14, 0x00).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xF4, 0x72, 0xD0).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xFA, 0x68, 0x00).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x6D, 0x87, 0x64).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x76, 0x60, 0x8A).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xD8, 0x00, 0x73).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xA4, 0xC4, 0x00).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x6A, 0x00, 0xFF).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x60, 0xA9, 0x17).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x00, 0x8A, 0x00).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x1B, 0xA1, 0xE2).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xA2, 0x00, 0x25).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x00, 0x50, 0xEF).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0x82, 0x5A, 0x2C).GetAsInteger());
ColorList.Add(Color.FromArgb(0xFF, 0xF0, 0xA3, 0x0A).GetAsInteger());
 
return ColorList.ToArray();
}
  • Undo and UndoAll operations
async void Undo_Click(object sender, RoutedEventArgs e)
{
await _imagingSession.Undo();
ImageViewer.Source = _imagingSession.Image;
}
 
async void UndoAll_Click(object sender, RoutedEventArgs e)
{
await _imagingSession.UndoAll();
ImageViewer.Source = _imagingSession.Image;
}

Sample Filters

  • IPixelManipulation

and HistogramFilter - IPixelManipulation (collects Red, Green, Blue and Luminance data for histogram of image)

  • IProcessors

Summary

It's my approach to use custom filters and processors with Nokia Imaging SDK. Maybe it isn't the best way to implement custom filters and apply them to image but I think that it's easier to use. It'll speed up development of filter based application. Also that I didn't cover some useful tools in this article, you can check in project file DevTools.cs and Extensions folder.

1211 page views in the last 30 days.
×