Namespaces

Variants
Actions

Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries over the next few weeks. Thanks for all your past and future contributions.

Pop Art filter effect Halftoning using Imaging SDK

From Wiki
Jump to: navigation, search
Featured Article
04 Aug
2014

This article explains how to implement a filter effect inspired to Andy Warhol's Pop Art using Halftoning technique.

WP Metro Icon Multimedia.png
WP Metro Icon WP8.png
Article Metadata
Code Example
Source file: GitHub - Pop Art
Tested with
Devices(s): Nokia Lumia 925, Nokia Lumia 1020, Nokia Lumia 1520, Nokia Lumia 630
Compatibility
Platform(s):
Windows Phone 8
Article
Created: galazzo (07 Jun 2014)
Last edited: chintandave_er (11 Aug 2014)

Contents

Pop art

Example of Pop Art

Pop art is an art movement that emerged in the mid-1950s in Britain and in the late 1950s in the United States. Pop art presented a challenge to traditions of fine art by including imagery from popular culture such as advertising, news, etc.

Pop art employs aspects of mass culture, such as advertising, comic books and mundane cultural objects. Pop art is aimed to employ images of popular as opposed to elitist culture in art, emphasizing the banal or kitschy elements of any given culture, most often through the use of irony. It is also associated with the artists' use of mechanical means of reproduction or rendering techniques.

Andy Warhol is probably the most famous figure in Pop Art, he is the father of the effect shown in the side image that we are going to reproduce through the Nokia Imaging SDK.

Halftone

Halftone example.
Basically the effect we are going to create is made by composing the cartoon and Halftone effect. Nokia imaging SDK provide a good implementation of Cartoon filter, but not of the Halftone who is the one on which we will focus.


Halftone is a technique that simulates continuous tone of an image through the use of dots, varying either in size, in shape or in spacing, thus generating a gradient like effect.

Where continuous tone images contains an infinite range of colors or greys, the halftone process reduces visual reproductions to an image that is printed with only one color of ink, in dots of differing size. This reproduction relies on a basic optical illusion that these tiny halftone dots are blended into smooth tones by the human eye.

Dot shapes

Though round dots are the most common used, there are different dot types available, each of them having their own characteristics.

  • Round dots: most common, suitable for light images, especially for skin tones. They meet at a tonal value of 70%.
  • Elliptical dots: appropriate for images with many objects. Elliptical dots meet at the tonal values 40% (pointed ends) and 60% (long side), so there is a risk of a pattern.
  • Square dots: best for detailed images, not recommended for skin tones. The corners meet at a tonal value of 50%. The transition between the square dots can sometimes be visible to the human eye.

In this article we will focus on round dots as the algorithm is not dependent by the shape so once learned the technique the user can use the one he prefer, also the most strange ones.

Creating the effect

There are plenty of tutorials and well done documentation explaining how to create a custom effect. As quick recap let's remember that to create a custom effect in managed code we need to inherit from CustomEffectBase while from DelegatingEffect in C++/CX.

In managed code, the basic looks like that:

public class DoubleEffect : CustomEffectBase 
{
public DoubleEffect(IImageProvider source) : base(source)
{
}
 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
}
}

In C++/Cx there is much less documentation, we must implement a couple of more functions, but things are almost the same.

namespace NativeFilters
{
public ref class Halftone sealed : ICustomEffect
{
public:
Halftone();
 
virtual IAsyncAction^ LoadAsync();
virtual void Process(Windows::Foundation::Rect rect);
virtual IBuffer^ ProvideSourceBuffer(Windows::Foundation::Size imageSize);
virtual IBuffer^ ProvideTargetBuffer(Windows::Foundation::Size imageSize);
 
private:
static byte* GetPointerToPixelData(IBuffer^ pixelBuffer, unsigned int *length);
};
}
  • LoadAsync Load/prepare for rendering.
  • Process Called when the image should be processed.
  • ProvideSourceBuffer Provide an IBuffer sufficiently sized to hold an image of the specified size.
  • ProvideTargetBuffer Provide an IBuffer sufficiently sized to hold an image of the specified size.

Drawing dots

As we learned in the description of the effect, basically the image to process is divided into cells, computed the average luminosity of the area then plot on the corresponding cell a dot ( or the shape you decided ) of the size related to the luminosity of the area. The darker is the area the bigger is the dot size and vice versa.

The cell size is a crucial choice for a good effect. If too small we risk simply to dirty the image as the effect is not well visible, if too big we loose all details. A good default value is a cell 20x20 pixels, but based on the image size the user should be able to set the size best fit his tastes.

The approach proposed divide the pixel's luminosity range into ten smaller ranges. This choice comes from a lot of tests and seems to be the best one, but the user is free to change it. The luminosity mapping is linear:

Range Dot size
[0,25] 20 pixels
[26, 50] 18 pixels
[51,75] 16 pixels
[76,100] 14 pixels
[101,125] 12 pixels
[126,150] 10 pixels
[151,175] 8 pixels
[176,200] 6 pixels
[201,225 4 pixels
[226,255] 2 pixels

To speed-up performances and avoid creating a new dots mask matrix each time a new region is processed the approach followed and suggested is to pre-computing the matrix's so when computed the luminosity of the area the corresponding matrix will be just copied.

So what we need is a function that is able to build the dot template with the cell size and shape we decided that we call CreateDot. The class constructor initializes the ten matrix's, then inside the OnProcess we will implement the filter algorithm.

The CellSize, as suggested by the name set the size of the area is being to processed. This the most important parameter as much more then the shape will be responsible of a great or very bad result.

A too big or too small parameter will produce just a dirty image. Is not a topic of Based on tests for a 5mpx image a CellSize of 20 produce good results, but for sure the best solution would be a dynamic computation of this value based on image size. In my opinion this computation must not be delegated to filter to allow to developer the max freedom range when using it in his app.

For that reason the filter is initialized with a default value, then to be modified dynamically (or statically) by the developer.

To give a better global effect to the image we add a "movement" to dots topology to avoid a too regular matrix looking of dots, so each row is shifted by a value of 8px. This value could become dynamic in order to be controlled by the developer, but is not mandatory.

Once computed area's luminosity we compy the tone matrix at {Icode|toneIndex}} over the CellSize area. As the tome matrix has a value of [0,1] the pixel is filled with the black value were we want to draw part of the dot else will be kept the original value. The choice to use {Icode|double}} against bool come from the fact the algorithm can be improved adding alpha feature.

The proposed code looks like the following:

public class HalftoneEffect : CustomEffectBase
{
// Array of matrixes
private double[][,] tone;
 
/*
* Commodity enum to identify the shape you want to implement for each region
* Just the Circle shape was implemented as demonstration. Feel free to implement your own
*/

public enum Shape
{
Circle,
Diamond,
Square
}
 
public Shape CellShape { get; set; }
 
/*
* The cell size, as suggested by the name set the size of the area is being to processed.
*/

private uint m_cell_size = 20;
public uint CellSize {
get
{
return m_cell_size;
}
 
/*
* Each time the the value is changed all matrix's will be initialized again
*/

set
{
m_cell_size = value;
tone = null;
tone = new double[10][,];
uint radius = m_cell_size;
uint range = (uint)Math.Floor((double)(m_cell_size / 10));
 
/*
* Initialize matrixex from bigger to smaller
* reducing the radius by range value each cycle
*/

for (int i = 0; i < 10; i++)
{
tone[i] = new double[m_cell_size, m_cell_size];
CreateDot(ref tone[i], m_cell_size, radius);
radius -= range;
}
}
}
 
public HalftoneEffect(IImageProvider source)
: base(source)
{
CellSize = 20;
}
 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
int m_width = (int) sourcePixelRegion.ImageSize.Width;
int m_height = (int) sourcePixelRegion.ImageSize.Height;
uint luma = 0;
uint toneIndex = 0;
 
/*
* Scan all the image by a block size of CellSize x CellSize
* To avoid buffer overflow do not forget to subtract CellSize from boudaries
*/

for (uint y = 0; y < m_height - CellSize; y += CellSize)
{
for (uint x = 0; x < m_width - CellSize; x += CellSize)
{
/*
* The first step is to compute the average luminance value of the area
* It's known there is a best function to compute pixel luminosity from RGB
* anyway in that case the accuracy provided is useless falling down performances.
* In that context the old dummy average makes it's job very well.
*/

luma = 0;
for (uint m_y = y; m_y < (y+CellSize); m_y++)
{
for (uint m_x = x; m_x < (x + CellSize); m_x++)
{
uint color = sourcePixelRegion.ImagePixels[m_x + (m_y * m_width)];
var r = (byte)((color >> 16) & 255);
var g = (byte)((color >> 8) & 255);
var b = (byte)((color) & 255);
 
luma += (uint)( (double)(r+g+b) / 3.0 );
}
}
luma /= (CellSize * CellSize);
 
// Compute the index of array of matrix's to select the more suitable one
toneIndex = (uint)Math.Floor((float)luma / 25);
if (toneIndex >= 10) toneIndex = 9;
 
// To give a better global effect to the image each row is shifted by a value of 8px
var offset = 8;
var m_xxx = 0;
 
/*
* Once computed area's luminosity we compy the tone matrix at toneIndex over the CellSize area.
* As the tome matrix has a value of [0,1] the pixel is filled with the black value were we want to draw
* part of the dot else will be kept the original value.
*/

for (uint m_y = y, m_yy = 0; m_y < (y+CellSize); m_y++, m_yy++)
{
for (uint m_x = x, m_xx = 0; m_x < (x + CellSize); m_x++, m_xx++)
{
m_xxx = (int)((m_xx + ((((y/CellSize) % 2) == 1) ? offset : 0)) % CellSize);
 
uint color = sourcePixelRegion.ImagePixels[m_x + (m_y * m_width)];
var r = (byte)((color >> 16) & 255);
var g = (byte)((color >> 8) & 255);
var b = (byte)((color) & 255);
 
r = (byte) ((double) r * tone[toneIndex][m_yy, m_xxx]);
g = (byte) ((double) g * tone[toneIndex][m_yy, m_xxx]);
b = (byte) ((double) b * tone[toneIndex][m_yy, m_xxx]);
 
targetPixelRegion.ImagePixels[m_x + ((int)(m_y * m_width))] =
(uint)(b | (g << 8) | (r << 16) | (0xFF << 24));
}
}
}
}
}
 
private static void CreateDot(ref double[,] dot, uint outersize, uint innersize)
{
uint cx = outersize / 2;
uint cy = outersize / 2;
 
double _x = 0;
double _y = 0;
 
double distance = 0;
 
innersize /= 2;
 
for (int y = 0; y < outersize; y++)
{
for (int x = 0; x < outersize; x++)
{
_x = x - cx;
_y = y - cy;
 
distance = Math.Sqrt(_x * _x + _y * _y);
 
if (distance <= innersize)
{
dot[y, x] = 0;
}
else
{
dot[y, x] = 1;
}
}
}
}
}

Implementing in C++

Once described the algorithm and the code in C# finally we can introduce the C++/CX implementation that differs a bit from C#, while the algorithm is exactly the same.

The main difference is that in C++/CX we need to implement the function GetPointerToPixelData. In C++/CX you can get a raw pointer to the underlying byte array by using the Windows Runtime Library IBufferByteAccess interface that is defined in robuffer.h. By using this approach you can modify the byte array in-place without making any unnecessary copies of the data.

This argument is well documented in the article Obtaining pointers to data buffers (C++/CX).

Another important difference is the pixel access. In C# accessing to a matrix entry [x,y] gives an integer, to access to each channel component we must use shift operators then the width of the image matches the sourceBuffer width. In C++/CX accessing to the byte valued raw buffer, each [x,y] entry is don't correspond to a pixel but to a signle channel as each pixel is stored as a sequence of four byte in the format of Bgra8888.

This implies when scan each row to take into account is size is 'image width' x 4.

using namespace Windows::Foundation;
using namespace Windows::Storage::Streams;
 
using namespace Nokia::Graphics::Imaging;
using namespace Nokia::InteropServices::WindowsRuntime;
 
namespace NativeFilters
{
public ref class Halftone sealed : ICustomEffect
{
public:
Halftone();
 
// set and get the size of the region to process
property unsigned int CellSize
{
unsigned int get()
{
return m_cell_size;
}
 
void set(unsigned int value)
{
m_cell_size = value;
}
}
 
virtual IAsyncAction^ LoadAsync();
virtual void Process(Windows::Foundation::Rect rect);
virtual IBuffer^ ProvideSourceBuffer(Windows::Foundation::Size imageSize);
virtual IBuffer^ ProvideTargetBuffer(Windows::Foundation::Size imageSize);
 
private:
// array of ten matrixes as pointers to pointers
double** tone[10];
unsigned int m_cell_size;
unsigned int imageWidth;
Buffer^ sourceBuffer;
Buffer^ targetBuffer;
 
private:
static byte* GetPointerToPixelData(IBuffer^ pixelBuffer, unsigned int *length);
static void CreateDot(double*** dot, unsigned int outersize, unsigned int innersize);
};
}
#include "pch.h"
#include <ppltasks.h>
#include <wrl.h>
#include <robuffer.h>
#include <ppltasks.h>
#include <math.h>
 
#include "NativeFilters.h"
 
using namespace NativeFilters;
using namespace Platform;
using namespace concurrency;
 
using namespace Windows::Storage::Streams;
using namespace Microsoft::WRL;
 
Halftone::Halftone() : m_cell_size(20) // the default cell size is initialized to 20
{
}
 
/*
* Code executed when the filter is being to be prepared
*/

IAsyncAction^ Halftone::LoadAsync()
{
return create_async([this]
{
unsigned int radius = CellSize;
unsigned int range = (unsigned int) floor((double)(CellSize / 10));
 
/*
* Initialize matrices from bigger to smaller
* reducing the radius by range value each cycle
*/

for (int i = 0; i < 10; i++)
{
/*
* memory allocation for i^ matrix.
* First is allocated the memory for array representing rows then for each row allocate memory for each column
*/

tone[i] = new double*[CellSize];
for (unsigned int j = 0; j < CellSize; j++)
{
tone[i][j] = new double[CellSize];
}
 
/*
* Initialize the matrix. Value are passed by reference for performances reasons
*/

CreateDot(&tone[i], CellSize, radius);
radius -= range;
}
});
}
 
void Halftone::Process(Windows::Foundation::Rect rect)
{
unsigned int sourceLength, targetLength;
byte* sourcePixelRegion = GetPointerToPixelData(sourceBuffer, &sourceLength);
byte* targetPixelRegion = GetPointerToPixelData(targetBuffer, &targetLength);
 
/*
* Pixels are stored as a sequence of four bytes for each entry.
* Pre-compute the correct dot matrix size
*/

int CellSizeABGR = CellSize * 4;
 
unsigned int m_width = (unsigned int) rect.Width*4;
unsigned int m_height = (unsigned int) rect.Height;
unsigned int luma = 0;
unsigned int toneIndex = 0;
unsigned int xOffset = 0;
 
/*
* Scan all the image by a block size of CellSize x CellSize
* To avoid buffer overflow do not forget to subtract CellSize from boudaries
*/

for (unsigned int y = 0; y < m_height - CellSize; y += CellSize)
{
for (unsigned int x = 0; x < m_width - CellSizeABGR; x += CellSizeABGR)
{
/*
* The first step is to compute the average luminance value of the area
* It's known there is a best function to compute pixel luminosity from RGB
* anyway in that case the accuracy provided is useless falling down performances.
* In that context the old dummy average makes it's job very well.
* Compared to C# we don't need shift operators as we can access directly to each channel, but always remember
* the different approach to widths adding += 4
*/

luma = 0;
for (unsigned int m_y = y; m_y < (y + CellSize); m_y++)
{
xOffset = m_y * m_width;
for (unsigned int m_x = x; m_x < (x + CellSizeABGR); m_x += 4)
{
byte b = (byte)sourcePixelRegion[xOffset + m_x + 0];
byte g = (byte)sourcePixelRegion[xOffset + m_x + 1];
byte r = (byte)sourcePixelRegion[xOffset + m_x + 2];
 
luma += (unsigned int)((double)((int)r + (int)g + (int)b) / 3.0);
}
}
luma /= (CellSize * CellSize);
 
// Compute the index of array of matrix's to select the more suitable one
toneIndex = (unsigned int) floor((float)luma / 25);
if (toneIndex >= 10) toneIndex = 9;
 
// To give a better global effect to the image each row is shifted by a value of 8px
int offset = 8;
int m_xxx = 0;
 
/*
* Once computed area's luminosity we compy the tone matrix at toneIndex over the CellSize area.
* As the tome matrix has a value of [0,1] the pixel is filled with the black value were we want to draw
* part of the dot else will be kept the original value.
*/

for (unsigned int m_y = y, m_yy = 0; m_y < (y + CellSize); m_y++, m_yy++)
{
xOffset = m_y * m_width;
for (unsigned int m_x = x, m_xx = 0; m_x < (x + CellSizeABGR); m_x+=4, m_xx++)
{
m_xxx = (int)((m_xx + ((((y / CellSize) % 2) == 1) ? offset : 0)) % CellSize);
 
byte b = (byte)sourcePixelRegion[xOffset + m_x + 0];
byte g = (byte)sourcePixelRegion[xOffset + m_x + 1];
byte r = (byte)sourcePixelRegion[xOffset + m_x + 2];
 
r = (byte)((double)r * tone[toneIndex][m_yy][m_xxx]);
g = (byte)((double)g * tone[toneIndex][m_yy][m_xxx]);
b = (byte)((double)b * tone[toneIndex][m_yy][m_xxx]);
 
targetPixelRegion[xOffset + m_x + 0] = b;
targetPixelRegion[xOffset + m_x + 1] = g;
targetPixelRegion[xOffset + m_x + 2] = r;
targetPixelRegion[xOffset + m_x + 3] = 0xFF;
}
}
}
}
}
 
void Halftone::CreateDot(double*** dot, unsigned int outersize, unsigned int innersize)
{
int cx = outersize / 2;
int cy = outersize / 2;
 
double _x = 0;
double _y = 0;
 
double distance = 0;
 
innersize /= 2;
 
for (int y = 0; y < (int)outersize; y++)
{
for (int x = 0; x < (int)outersize; x++)
{
_x = x - cx;
_y = y - cy;
 
distance = sqrt(_x * _x + _y * _y);
 
if (distance <= innersize)
{
(*dot)[x][y] = 0;
}
else
{
(*dot)[x][y] = 1.0;
}
}
}
}
 
IBuffer^ Halftone::ProvideSourceBuffer(Windows::Foundation::Size imageSize)
{
unsigned int size = (unsigned int)(4 * imageSize.Height * imageSize.Width);
sourceBuffer = ref new Windows::Storage::Streams::Buffer(size);
sourceBuffer->Length = size;
imageWidth = (unsigned int)imageSize.Width;
return sourceBuffer;
}
 
IBuffer^ Halftone::ProvideTargetBuffer(Windows::Foundation::Size imageSize)
{
unsigned int size = (unsigned int)(4 * imageSize.Height * imageSize.Width);
targetBuffer = ref new Windows::Storage::Streams::Buffer(size);
targetBuffer->Length = size;
return targetBuffer;
}
 
byte* Halftone::GetPointerToPixelData(Windows::Storage::Streams::IBuffer^ pixelBuffer, unsigned int *length)
{
if (length != nullptr)
{
*length = pixelBuffer->Length;
}
 
// Query the IBufferByteAccess interface.
ComPtr<Windows::Storage::Streams::IBufferByteAccess> bufferByteAccess;
reinterpret_cast<IInspectable*>(pixelBuffer)->QueryInterface(IID_PPV_ARGS(&bufferByteAccess));
 
// Retrieve the buffer data.
byte* pixels = nullptr;
bufferByteAccess->Buffer(&pixels);
return pixels;
}

Using the filter

Asserting everybody knows how to apply a filter in C# using Nokia Imaging SDK we will focus on how to use it when written in native code. As implemented as Windows Runtime Component with his own namespace we must add reference to our project then include the namespace.

A fully working prototype has been published on github and availabe for download.

A bit difference when working with filters written in native code is that we must also use the delegatingeffect class to really create an effect that can be used in the rendering pipeline.

using NativeFilters;
 
async void cam_CaptureImageAvailable(object sender, Microsoft.Devices.ContentReadyEventArgs e)
{
string dateformat = Convert.ToString(DateTime.Now.Year) +
DateTime.Now.Month.ToString("d2") +
DateTime.Now.Day.ToString("d2") + "_" +
DateTime.Now.Hour.ToString("d2") + "_" +
DateTime.Now.Minute.ToString("d2") + "_" +
DateTime.Now.Second.ToString("d2");
 
string fileName = "WP_" + dateformat + "_POPART.jpg";
 
try
{
var halftoneNative = new NativeFilters.Halftone();
halftoneNative.CellSize = 20;
using (var halftoneEffect = new DelegatingEffect(new StreamImageSource(e.ImageStream), halftoneNative))
 
using (var filterEffect = new FilterEffect(halftoneEffect))
using (var wbRender = new JpegRenderer(filterEffect))
{
if (PopArtCartoon == true)
{
filterEffect.Filters = new[] { new CartoonFilter(true) };
}
 
var result = await wbRender.RenderAsync();
 
// Save photo to the media library camera roll.
library.SavePictureToCameraRoll(fileName, result.AsStream());
}
}
finally
{
// Close image stream
e.ImageStream.Close();
}
}

As we are accustomed to, the original Pop Art filter is created using the cartoon filter.

Anyway each scene and subject is different and based also on your and users tastes, a different implementation could be to use the Posterize filter or the Oily filter as well the Paint filter.

Results

Note.pngNote: Click on thumb to visualize the full screen image without dots artifacts

Original image Result
Original image
Cartoon Filter
Original image
Cartoon Filter
Original image
Posterize Filter (Value 6)
Original image
Posterize Filter (Value 8)

Improvements

The performances of the algorithm can be improved with a different approach computing luminosity as the nested for cycles has a N^2 complexity.

The basic idea is to work on a reduced image and replace the code compute the luminosity

luma = 0;
for (uint m_y = y; m_y < (y+CellSize); m_y++)
{
for (uint m_x = x; m_x < (x + CellSize); m_x++)
{
uint color = sourcePixelRegion.ImagePixels[m_x + (m_y * m_width)];
var r = (byte)((color >> 16) & 255);
var g = (byte)((color >> 8) & 255);
var b = (byte)((color) & 255);
 
luma += (uint)( (double)(r+g+b) / 3.0 );
}
}
luma /= (CellSize * CellSize);

simply with a direct access to the reduced image taking into account the reducing size ratio. For example if we reduce the original image by 8 times the pseudo-code could look like

luma = reduced_image[x/8, y/8];

This approach improves performances but it is less accurate and uses much memory as must create a second reduced support image that however small consume memory and this could be a problem on low memory devices.

For a lot of developers this is not a problem at all compared to performance improvement, but to risk to cut off low-men devices that has the 50% of market could be not good, but this is a forum discussion topic, not related to this article.

Up to developer to use the approach best fit his needs.

Pop Art filter on published apps

This code has been already implemented and published on 4Blend HDR app and available for free.

Some 4Blend HDR user's shots

Pop-art Images Pop-art Images
Cartoon Filter
Shot by Shubham Kumar
Shot by Aloysius Ting
Cartoon Filter
This page was last modified on 11 August 2014, at 06:16.
527 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×