×
Namespaces

Variants
Actions
(Difference between revisions)

Filter and Custom Filter Management Framework for the Nokia Imaging SDK

From Nokia Developer Wiki
Jump to: navigation, search
Engin Kırmacı (Talk | contribs)
(Engin Kırmacı -)
hamishwillee (Talk | contribs)
m (Hamishwillee - Minor subedit for wiki formatting)
Line 1: Line 1:
 
[[Category:Nokia Imaging SDK]][[Category:Code Examples]][[Category:Tutorial]][[Category:XAML]][[Category:Windows Phone 8]]
 
[[Category:Nokia Imaging SDK]][[Category:Code Examples]][[Category:Tutorial]][[Category:XAML]][[Category:Windows Phone 8]]
 +
{{Abstract|This article explains Imaging Framework that implement for to use custom filters and built-in filters. }} 
 +
{{Note|This is an entry in the [[Nokia Imaging and Big UI Wiki Competition 2013Q4]].}}
 
{{ArticleMetaData <!-- v1.3 -->
 
{{ArticleMetaData <!-- v1.3 -->
 
|sourcecode= [https://github.com/enginkirmaci/WP_Imaging Imaging Framework for Windows Phone at GitHub]
 
|sourcecode= [https://github.com/enginkirmaci/WP_Imaging Imaging Framework for Windows Phone at GitHub]
Line 19: Line 21:
 
|author= [[User:Engin Kırmacı]]
 
|author= [[User:Engin Kırmacı]]
 
}}
 
}}
 
{{Announcement|height=41|title=Note|This is an entry in the [http://developer.nokia.com/Community/Wiki/Nokia_Imaging_and_Big_UI_Wiki_Competition_2013Q4 Nokia Imaging and Big UI Wiki Competition 2013Q4].}}
 
 
{{Abstract|This article explains Imaging Framework that implement for to use custom filters and built-in filters. }} 
 
 
 
== Introduction ==
 
== Introduction ==
  
The Nokia Imaging SDK provides more than [http://www.developer.nokia.com/Resources/Library/Imaging_API_Ref/#!index.html 50 filters and effects] for developers to use and combine them to create new filters. Moreover with SDK 1.0, It allows us to implement custom filters and image providers. <br />
+
The Nokia Imaging SDK provides more than [http://www.developer.nokia.com/Resources/Library/Imaging_API_Ref/#!index.html 50 filters and effects] for developers to use and combine them to create new filters. Moreover with SDK 1.0, It allows us to implement custom filters and image providers.  
  
 
I implemented a framework which helps easier use of custom filters and built-in filters with extended capabilities (Preprocess and Postprocess filter for custom filter, historize filters etc...) Also I try to demonstrate how to implement different filters which can be used for image processing.
 
I implemented a framework which helps easier use of custom filters and built-in filters with extended capabilities (Preprocess and Postprocess filter for custom filter, historize filters etc...) Also I try to demonstrate how to implement different filters which can be used for image processing.
Line 35: Line 32:
  
 
* You can't use custom filters as built-in filters. You use custom filters as you use FilterEffect. It's ImageConsumer and ImageProvider. So you can pipeline with filter effect. More details about pipe lining explained in [http://developer.nokia.com/Resources/Library/Lumia/#!nokia-imaging-sdk/core-concepts.html;#toc_CustomEffectBase Core concepts]
 
* You can't use custom filters as built-in filters. You use custom filters as you use FilterEffect. It's ImageConsumer and ImageProvider. So you can pipeline with filter effect. More details about pipe lining explained in [http://developer.nokia.com/Resources/Library/Lumia/#!nokia-imaging-sdk/core-concepts.html;#toc_CustomEffectBase Core concepts]
<br />
+
* {{Icode|ForEachRow}} method in {{Icode|PixelRegion}} class:
 +
*: Use this method to iterate over the pixels. If you want to iterate every pixels, don't use for loop, it's the most efficient way. Unless you want to iterate some pixels, not all of them or multiple iteration for each pixel, use your own for loop.
 +
* '''IsInPlace''' property of CustomEffectBase constructor:
 +
** Default value is false. Which means {{Icode|sourcePixel}}, {{Icode|targetPixel}} is different array and you have to get pixel from {{Icode|sourcePixel}}, process it and set to {{Icode|targetPixel}}. Also note that If you don't set targetPixel, you'll see black image.
 +
** If you set {{Icode|IsInPlace()}} to {{Icode|true}}, {{Icode|sourcePixel}} and {{Icode|targetPixel}} is same array. Documentation says that it's more efficient. But you have to be careful if you want to process pixel and don't want to change source pixel.
  
* '''ForEachRow''' method in PixelRegion class:
 
: Use this method to iterate over the pixels. If you want to iterate every pixels, don't use for loop, it's the most efficient way. Unless you want to iterate some pixels, not all of them or multiple iteration for each pixel, use your own for loop.
 
<br />
 
  
* '''IsInPlace''' property of CustomEffectBase constructor:
+
* {{Icode|ToColor}} and {{Icode|FromColor}} methods:
:* Default value is false. Which means sourcePixel, targetPixel is different array and you have to get pixel from sourcePixel, process it and set to targetPixel. Also note that If you don't set targetPixel, you'll see black image.
+
*: converts between a {{Icode|Windows.UI.Color}} and the color format expected in the pixel arrays.
:* If you set IsInPlace to true, sourcePixel and targetPixel is same array. Documentation says that it's more efficient. But you have to be careful if you want to process pixel and don't want to change source pixel.  
+
<br />
+
  
* '''ToColor''' and '''FromColor''' methods:
 
: converts between a Windows.UI.Color and the color format expected in the pixel arrays.
 
<br />
 
  
 
== Imaging Framework ==
 
== Imaging Framework ==
 
=== Interfaces ===
 
=== Interfaces ===
 
* {{Icode|IHistoryable}}
 
* {{Icode|IHistoryable}}
It used for factory classes, when factory able to have more than one filters. Example is NokiaFilterFactory which stores every built-in filters.
+
It used for factory classes, when factory able to have more than one filters. Example is {{Icode|NokiaFilterFactory}} which stores every built-in filters.
 
This interface provides following methods;
 
This interface provides following methods;
 
<code csharp>
 
<code csharp>
        bool CanUndo();
+
bool CanUndo();
 
+
void Undo();
        void Undo();
+
void UndoAll();
 
+
        void UndoAll();
+
 
</code>
 
</code>
  
 
* {{Icode|IPostProcess}}
 
* {{Icode|IPostProcess}}
Use to extend custom filters. It provides custom filter to post process factory. So after custom filter processed, post process factory's filters processed. For example, QuadTransform filter implemantation.
+
Use to extend custom filters. It provides custom filter to post process factory. So after custom filter processed, post process factory's filters processed. For example, {{Icode|QuadTransform}} filter implementation.
  
 
<code csharp>
 
<code csharp>
Line 95: Line 86:
  
 
* {{Icode|FactoryBase}}
 
* {{Icode|FactoryBase}}
Factories derives from this class. It has {{Icode|Input}} and {{Icode|Output}} as ImageProvider so that it builds pipeline with another factory. There is already two factories; CustomFilterFactory for customfilter and NokiaFilterFactory for built-in filters. In future, there'll be more for same type of custom filters like convulation filters used for smooting, bluring, etc.
+
Factories derives from this class. It has {{Icode|Input}} and {{Icode|Output}} as {{Icode|ImageProvider}} so that it builds pipeline with another factory. There is already two factories; {{Icode|CustomFilterFactory}} for customfilter and {{Icode|NokiaFilterFactory}} for built-in filters. In future, there'll be more for same type of custom filters like convulation filters used for smoothing, blurring, etc.
  
 
* {{Icode|CustomFilterFactory}}
 
* {{Icode|CustomFilterFactory}}
This factory stores only one custom filter. If custom filter has IPreProcess or/and IPostProcess interface, with ApplyFilter method, it builds pipeline inside it.
+
This factory stores only one custom filter. If custom filter has {{Icode|IPreProcess}} or/and {{Icode|IPostProcess}} interface, with {{Icode|ApplyFilter}} method, it builds pipeline inside it.
  
 
* {{Icode|FilterHistory}}
 
* {{Icode|FilterHistory}}
This class used in ImagingManager. It historize built-in filters and custom filters added. It provides {{Icode|Undo}} and {{Icode|UndoAll}} method for filters.
+
This class used in {{Icode|ImagingManager}}. It historize built-in filters and custom filters added. It provides {{Icode|Undo}} and {{Icode|UndoAll}} method for filters.
  
 
=== How to implement custom filters? ===
 
=== How to implement custom filters? ===
Line 107: Line 98:
  
 
* Implementation of custom filter for '''Calculation Purpose'''  
 
* Implementation of custom filter for '''Calculation Purpose'''  
{{Icode|HistogramFilter}} is the only sample for this type of filter. This filters calculates Red, Green, Blue and Luminance of image. These parameters can be used as filter parametres then.
+
{{Icode|HistogramFilter}} is the only sample for this type of filter. This filters calculates Red, Green, Blue and Luminance of image. These parameters can be used as filter parameters then.
<br />
+
 
Trick is here, {{Icode|IsInPlace}} property in base constructor is setted to true, so that no need to set {{Icode|targetPixelRegion.ImagePixels}}.
+
Trick is here, {{Icode|IsInPlace}} property in base constructor is set to true, so that no need to set {{Icode|targetPixelRegion.ImagePixels}}.
 
<code csharp>
 
<code csharp>
 
public class HistogramFilter : CustomEffectBase
 
public class HistogramFilter : CustomEffectBase
Line 200: Line 191:
  
 
* '''Complex''' custom filter  
 
* '''Complex''' custom filter  
{{Icode|PixelInterpolation}} is one of the sample for this type of filter. This filter pixaletes image without changing the image size. It divides to image to rectangles like table and get top left pixel of each cell. Then changes every pixels in cell to corner.
+
{{Icode|PixelInterpolation}} is one of the sample for this type of filter. This filter pixelates image without changing the image size. It divides to image to rectangles like table and get top left pixel of each cell. Then changes every pixels in cell to corner.
 
<br />
 
<br />
For that purpose, iteration every pixels with ForEachRow is not efficent. First of all, use following to get width and height of an image.
+
For that purpose, iteration every pixels with {{Icode|ForEachRow}} is not efficient. First of all, use following to get width and height of an image.
 
<code csharp>
 
<code csharp>
 
int width = (int)sourcePixelRegion.Bounds.Width;
 
int width = (int)sourcePixelRegion.Bounds.Width;
Line 277: Line 268:
 
* {{Icode|imageControl}} for viewing purpose. Use image control which is in page.
 
* {{Icode|imageControl}} for viewing purpose. Use image control which is in page.
  
<br />
+
 
After that, add filters to be applied. The only restriction here, when initializing custom filters, you have to pass ImageProvider. But you don't have to worry about provider, use DummySource. ImagingManager handles and changes it.  
+
After that, add filters to be applied. The only restriction here, when initializing custom filters, you have to pass {{Icode|ImageProvider}}. But you don't have to worry about provider, use {{Icode|DummySource}}. {{Icode|ImagingManager}} handles and changes it.  
 
<code csharp>
 
<code csharp>
 
var effects2 = new FilterEffect();
 
var effects2 = new FilterEffect();
Line 310: Line 301:
 
* {{Icode|HistogramFilter}} (collects Red, Green, Blue and Luminance data for histogram of image)
 
* {{Icode|HistogramFilter}} (collects Red, Green, Blue and Luminance data for histogram of image)
 
* {{Icode|CornerDetection}} (detects corners of quadratical shape in black&white image)
 
* {{Icode|CornerDetection}} (detects corners of quadratical shape in black&white image)
* {{Icode|QuadTransformation}} (gets quadratical shape using corner points, then transforms shape to rectangular form)
+
* {{Icode|QuadTransformation}} (gets quadratic shape using corner points, then transforms shape to rectangular form)
  
 
== Contribution ==
 
== Contribution ==

Revision as of 05:59, 10 December 2013

This article explains Imaging Framework that implement for to use custom filters and built-in filters.

Note.pngNote: This is an entry in the Nokia Imaging and Big UI Wiki Competition 2013Q4.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code Example
Installation file: File:WP Imaging.zip (ARM and x86)
Tested with
SDK: Windows Phone 8.0 SDK, Nokia Imaging SDK v1.0
Devices(s): Windows Phone 8 Emulator, Nokia Lumia 920
Compatibility
Platform(s):
Windows Phone 8
Dependencies: Nokia Imaging SDK
Article
Created: Engin Kırmacı (31 Aug 2013)
Last edited: hamishwillee (10 Dec 2013)

Contents

Introduction

The Nokia Imaging SDK provides more than 50 filters and effects for developers to use and combine them to create new filters. Moreover with SDK 1.0, It allows us to implement custom filters and image providers.

I implemented a framework which helps easier use of custom filters and built-in filters with extended capabilities (Preprocess and Postprocess filter for custom filter, historize filters etc...) Also I try to demonstrate how to implement different filters which can be used for image processing.

Please read all documents about Nokia Imaging SDK to understand this article.

Useful information about custom filters in Nokia imaging SDK

  • You can't use custom filters as built-in filters. You use custom filters as you use FilterEffect. It's ImageConsumer and ImageProvider. So you can pipeline with filter effect. More details about pipe lining explained in Core concepts
  • ForEachRow method in PixelRegion class:
    Use this method to iterate over the pixels. If you want to iterate every pixels, don't use for loop, it's the most efficient way. Unless you want to iterate some pixels, not all of them or multiple iteration for each pixel, use your own for loop.
  • IsInPlace property of CustomEffectBase constructor:
    • Default value is false. Which means sourcePixel, targetPixel is different array and you have to get pixel from sourcePixel, process it and set to targetPixel. Also note that If you don't set targetPixel, you'll see black image.
    • If you set IsInPlace() to true, sourcePixel and targetPixel is same array. Documentation says that it's more efficient. But you have to be careful if you want to process pixel and don't want to change source pixel.


  • ToColor and FromColor methods:
    converts between a Windows.UI.Color and the color format expected in the pixel arrays.


Imaging Framework

Interfaces

  • IHistoryable

It used for factory classes, when factory able to have more than one filters. Example is NokiaFilterFactory which stores every built-in filters. This interface provides following methods;

bool CanUndo();
void Undo();
void UndoAll();
  • IPostProcess

Use to extend custom filters. It provides custom filter to post process factory. So after custom filter processed, post process factory's filters processed. For example, QuadTransform filter implementation.

public class QuadTransformation : CustomEffectBase, IPostProcess
{
.
.
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
.
.
}
 
public FactoryBase PostProcessFactory()
{
var factory = new NokiaFilterFactory(null);
factory.Filters.Add(new ReframingFilter(new Rect(0, 0, Size.Width, Size.Height), 0));
 
return factory;
}
}
  • IPreProcess

Use to extend custom filters. It provides custom filter to pre process factory. So before custom filter processed, pre process factory's filters processed.

Classes

  • ImagingManager

This class is the main core, everything happens here.

  • FactoryBase

Factories derives from this class. It has Input and Output as ImageProvider so that it builds pipeline with another factory. There is already two factories; CustomFilterFactory for customfilter and NokiaFilterFactory for built-in filters. In future, there'll be more for same type of custom filters like convulation filters used for smoothing, blurring, etc.

  • CustomFilterFactory

This factory stores only one custom filter. If custom filter has IPreProcess or/and IPostProcess interface, with ApplyFilter method, it builds pipeline inside it.

  • FilterHistory

This class used in ImagingManager. It historize built-in filters and custom filters added. It provides Undo and UndoAll method for filters.

How to implement custom filters?

I'm categorizing my sample custom filters in 3 category; calculation purpose, simple ones and complex ones. I'll try to explained over sample filters.

  • Implementation of custom filter for Calculation Purpose

HistogramFilter is the only sample for this type of filter. This filters calculates Red, Green, Blue and Luminance of image. These parameters can be used as filter parameters then.

Trick is here, IsInPlace property in base constructor is set to true, so that no need to set targetPixelRegion.ImagePixels.

public class HistogramFilter : CustomEffectBase
{
public int[] Red { get; set; }
 
public int[] Green { get; set; }
 
public int[] Blue { get; set; }
 
public int[] Luminance { get; set; }
 
public HistogramFilter(IImageProvider source, out int[] red, out int[] green, out int[] blue, out int[] luminance)
: base(source, true)
{
Red = new int[byte.MaxValue + 1];
Green = new int[byte.MaxValue + 1];
Blue = new int[byte.MaxValue + 1];
Luminance = new int[byte.MaxValue + 1];
 
red = Red;
green = Green;
blue = Blue;
luminance = Luminance;
}
 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
sourcePixelRegion.ForEachRow((index, width, pos) =>
{
for (int x = 0; x < width; ++x, ++index)
{
Color c = ToColor(sourcePixelRegion.ImagePixels[index]);
 
Red[c.R]++;
Green[c.G]++;
Blue[c.B]++;
 
double luminance = 0.299 * c.R + 0.587 * c.G + 0.114 * c.B;
if (luminance < byte.MinValue)
luminance = byte.MinValue;
else if (luminance > byte.MaxValue)
luminance = byte.MaxValue;
 
Luminance[(byte)luminance]++;
}
});
}
  • Simple custom filter

ThresholdFilter is one of the sample for this type of filter. It's the same implementation as described here

public class ThresholdFilter : CustomEffectBase
{
private byte _thresholdR = 0;
private byte _thresholdG = 0;
private byte _thresholdB = 0;
 
public ThresholdFilter(IImageProvider source, byte R, byte G, byte B)
: base(source)
{
_thresholdR = R;
_thresholdG = G;
_thresholdB = B;
}
 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
sourcePixelRegion.ForEachRow((index, width, pos) =>
{
for (int x = 0; x < width; ++x, ++index)
{
Color c = ToColor(sourcePixelRegion.ImagePixels[index]);
 
if (c.R < _thresholdR)
c.R = 0;
 
if (c.G < _thresholdG)
c.G = 0;
 
if (c.B < _thresholdB)
c.B = 0;
targetPixelRegion.ImagePixels[index] = FromColor(c);
}
});
}
}
  • Complex custom filter

PixelInterpolation is one of the sample for this type of filter. This filter pixelates image without changing the image size. It divides to image to rectangles like table and get top left pixel of each cell. Then changes every pixels in cell to corner.
For that purpose, iteration every pixels with ForEachRow is not efficient. First of all, use following to get width and height of an image.

int width = (int)sourcePixelRegion.Bounds.Width;
int height = (int)sourcePixelRegion.Bounds.Height;

Then, it iterates only corner pixels.

int offsetX = _pixelSize / 2;
int offsetY = _pixelSize / 2;
 
for (int x = 0; x < width; x += _pixelSize)
{
for (int y = 0; y < height; y += _pixelSize)
{
//calculation here
}
}

Final code looks as below

public class PixelInterpolation : CustomEffectBase
{
private int _pixelSize = 0;
 
public PixelInterpolation(IImageProvider source, int PixelSize)
: base(source)
{
_pixelSize = PixelSize;
}
 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
int offsetX = _pixelSize / 2;
int offsetY = _pixelSize / 2;
 
int width = (int)sourcePixelRegion.Bounds.Width;
int height = (int)sourcePixelRegion.Bounds.Height;
 
for (int x = 0; x < width; x += _pixelSize)
{
for (int y = 0; y < height; y += _pixelSize)
{
// make sure that the offset is within the boundry of the image
while (x + offsetX >= width) offsetX--;
while (y + offsetY >= height) offsetY--;
 
// get the pixel color in the center of the soon to be pixelated area
var pixel = sourcePixelRegion.ImagePixels[DimensionConverter.To1D(x + offsetX, y + offsetY, width)];
 
// for each pixel in the pixelate size, set it to the center color
for (int i = x; i < x + _pixelSize && i < width; i++)
for (int j = y; j < y + _pixelSize && j < height; j++)
targetPixelRegion.ImagePixels[DimensionConverter.To1D(i, j, width)] = pixel;
}
}
}
}

How to use?

As far as now, I described confusing things. The most easy part of this Imaging Framework is how to use. It's really simple.

First of all, add ImagingManager to page;

ImagingManager ImagingManager = new ImagingManager();

Then, start a new session;

await ImagingManager.StartSession(imageStream, imageControl);
  • ImageStream is whatever you want to use from them.
  • imageControl for viewing purpose. Use image control which is in page.


After that, add filters to be applied. The only restriction here, when initializing custom filters, you have to pass ImageProvider. But you don't have to worry about provider, use DummySource. ImagingManager handles and changes it.

var effects2 = new FilterEffect();
effects2.Filters = new IFilter[]
{
new BlurFilter(32)
};
 
ImagingManager.AddFilter(effects2);
 
ImagingManager.AddFilter(new ThresholdFilter(ImagingManager.DummySource, 150, 100, 0));
ImagingManager.AddFilter(new ColorPaletteFilter(ImagingManager.DummySource, AccentColorsPalette()));
ImagingManager.AddFilter(new PixelInterpolation(ImagingManager.DummySource, 64));

Finally, render it.

await ImagingManager.Render();

Note that in sample project, you can see more detailed use of framework.

Sample Filters

  • IPixelManipulation
  • HistogramFilter (collects Red, Green, Blue and Luminance data for histogram of image)
  • CornerDetection (detects corners of quadratical shape in black&white image)
  • QuadTransformation (gets quadratic shape using corner points, then transforms shape to rectangular form)

Contribution

If you want to contribute the project, please contact me over github page.

Conclusion

It's my approach for custom filter and built-in filters for Nokia Imaging SDK. I'm not an expert in imaging process, I tried my best and continue to learn new things. I believe Imaging Framework will speed up development of filter based application. I'm also using it with my applications and constantly updated and added new things to it. I'm planning to implement more filters that used for image processing.

413 page views in the last 30 days.