Namespaces

Variants
Actions

Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries over the next few weeks. Thanks for all your past and future contributions.

Creating a beautiful mosaic effect using the Nokia Imaging SDK

From Wiki
Jump to: navigation, search
Featured Article
19 Jan
2014

This article shows how to create a simple mosaic effect using the Nokia Imaging SDK.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code ExampleTested with
SDK: Windows Phone 8.0,
Compatibility
Platform(s):
Windows Phone 8
Dependencies: Nokia Imaging SDK 1.0
Article
Created: pkovacevic (17 Dec 2013)
Last edited: pavan.pareta (22 Jan 2014)

Note.pngNote: This is an entry in the Nokia Imaging and Big UI Wiki Competition 2013Q4.

Contents

Introduction

A photographic mosaic (photomosaic) is typically made by dividing up an image into many sections, and then replacing each part with other images ("tiles") that have approximately the same colour as the original image at the same location. The resulting photomosaic looks like the original image from a distance, but the individual images become visible at higher magnification.

This article describes a clever trick which uses the Nokia Imaging SDK to mimic the mosaic effect. Instead of picking a tile image which colour-matches the original picture at each point, we arrange arbitrary tile images in a grid, and then "tint" them to match the colour of the image at all points. This tinting is easy, achieved by simply blending the tile grid with the original image using the Imaging SDK.

This article shows how to use the Nokia Imaging SDK to:

  • make your own custom effects that handles multiple input images
  • apply effects to an image source.
  • chain multiple effects together.



Pre-requisites

Readers must be comfortable with C# code. A basic knowledge of the Nokia Imaging SDK concepts is recommended (Core concepts and Quick Start).

Nokia Imaging Concepts

Before exploring this specific filter, let me introduce two required concepts: image provider and image consumer.

  • Image provider can be all sorts of things - images, streams, filter effects or any custom effect for that matter. They provide some kind of image information, they represent the image source. All image providers implement IImageProvider interface.
  • Image consumers are classes that consume the image providers - they do something with the information provided by the image provider.
using (var filterEffect = new FilterEffect(imageSource))
using (var blendFilter = new BlendFilter(foregroundFrame))
{
blendFilter.BlendFunction = BlendFunction.Normal;
filterEffect.Filters = new IFilter[] { new CartoonFilter(), blendFilter };
}

For example, FilterEffect is an image consumer and an image provider. As the image consumer, the FilterEffect class can consume any image source and apply some kind of processing to it. After the image is processed, FilterEffect can output the resulting image and therefore present itself to other consumers as an image provider.

FilterEffect class comes with the SDK, and can combine multiple filters on the image provider at hand. In the example above, we've added a cartoon and a blend filter to its filter list. BlendFilter takes a foreground image source as a parameter - an image that will blend itself with the main image. For this example, a foreground image is a simple transparent frame. If we render this effect on the screen, we get the following output.

Example1.png

Mosaic Effect Introduction

This description separates the problem into two parts: arranging multiple tile images on a large canvas, and then blending with the original image to create the mosaic effect.

First we want to build our own custom effect class (derived from CustomEffectBase) to encapsulate the tile arranging operation. This will take an array of image providers, arrange them in tiles, and output the resulting image. The way we’d like to use our class is represented below:

// Define all funky image sources that will be used as image providers
var ImageSourcesList = new List<IImageProvider>() { imageSource1, imageSource2, imageSource3, imageSource4 };
 
// Construction of the Tile Effect, taking image providers list to use, size of the individual tile and output resolution (how many tiles are contained in the output)
var tileEffect = new TileEffect(ImageSourcesList, tileSize: new Size(400, 300), outputResolution: new Size(800, 600));

The first parameter we supply are the four image sources that will provide our tiles. With tile size 400x300 and output resolution of 800x600, we will get two rows and two columns of tiles. The output for this example can be seen on the image below.

Example3.png

Every image source was resized and translated to one tile position. We have four image sources and four tile positions.

Note that if there are less image providers than expected tiles, we expect the class to reuse the available providers to fill the remaining tiles. Note also that if a tile doesn't fall cleanly in the output canvas, we expect the class to ignore tile pixels that fall out of range.

var tileEffect = new TileEffect(new List<IImageProvider>(){mainImageSource}, tileSize: new Size(300, 250), outputResolution: new Size(750, 250));

With tile size 300x250and output resolution of 750x250, we expect to see two and a half tiles on the output.

Example2.png


We can then take the above images and blend them with the original image to create the mosaic effect.

TileEffect Implementation

Custom effect classes must inherit from CustomEffectBase class and they must implement the OnProcess method where the image processing logic is held. We define our class below.

class TileEffect : CustomEffectBase
{
 
public TileEffect(IList<IImageProvider> tileImageProviders, Size tileSize, Size outputResolution, bool shuffle = false)
: base(new BitmapImageSource(new Bitmap(outputResolution, ColorMode.Bgra8888)))
{

}
}

Public constructor takes a list of image providers, tile resolution and output resolution parameter. Last two parameters effectively tell us how many tiles will be contained in the output.

Note.pngNote: Base class constructor expects single source image provider as the parameter which is unfortunate for our class purpose and design. We will send one empty bitmap image source to do our side of the deal. This action will also in return define our output image resolution because CustomEffectBase class will make sure that for every source pixel we get one output pixel. With that in mind, we initialize our dummy bitmap resolution be equal to the wanted output resolution.

Next thing we have to define is how to process the image providers and implement the OnProcess operation. Overriding this function we will define how to combine our image sources inside one image.

protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion outputPixelRegion)
{
// Logic on the pixels
}

PixelRegion is a class that encapsulates pixels and methods for accessing them. sourcePixelRegion therefore contains pixel information from the source image, and outputPixelRegion contains the output pixels we have to fill.

Note.pngNote: sourcePixelRegion is not important for our case. Pixels interesting to us are contained in the image providers list provided in the constructor. Also, we know that sourcePixelRegion will be filled with empty pixels because we sent an empty bitmap to the base class as the source parameter.

Things we will handle in OnProcess function:

  • Resizing the image providers to individual tile resolution. This means we will create one resized bitmap for each image provider. Converting image providers to bitmap will also provide us individual pixel access.
  • If the number of bitmaps (image providers sent) is less than the number of tiles, we’ll have to reuse our bitmaps to fill the remaining space.
  • We have to map and copy individual pixels from our image providers to the appropriate output pixels.

We can resize and convert IImageProvider type class with its public method GetBitmapAsync. OnProcess is already running in the background thread, so we will run this method synchronously.

// How many tiles can fit horizontally
var horizontalTilesNumber = (int)Math.Ceiling(targetPixelRegion.ImageSize.Width / _tileSize.Width);
// How many tiles can fit vertically
var verticalTilesNumber = (int)Math.Ceiling(targetPixelRegion.ImageSize.Height / _tileSize.Height);
 
foreach (var imageProvider in _tileImageProviders)
{
// Initialize empty bitmap
var tileBitmap = new Bitmap(_tileSize, ColorMode.Bgra8888);
 
// Create new bitmap from image source (resized to tile size), synchronous operation
imageProvider.GetBitmapAsync(tileBitmap, OutputOption.Stretch).AsTask().Wait();
 
// Save to tile bitmap sources for later use
_tileBitmapSources.Add(tileBitmap);
 
imageProcessedCounter++;
 
if (imageProcessedCounter >= horizontalTilesNumber * verticalTilesNumber)
{
// We have enough bitmaps to fill all tiles -> stop
break;
}
 
}


If we don’t have enough bitmap sources to cover all tiles, we have to reuse bitmaps until all tiles are covered.

int bitmapPointer = 0;
// If some tile places are empty -> Add existing bitmaps to fill remaining tiles
while (_tileBitmapSources.Count < horizontalTilesNumber * verticalTilesNumber)
{
_tileBitmapSources.Add(_tileBitmapSources.ElementAt(bitmapPointer);
bitmapPointer++;
}

Accessing bitmap pixels

When we have a bitmap, we can access its individual pixels using bitmaps internal buffer.

var buffer = bitmap.Buffers[0].Buffer;
 
buffer.GetByte(i) -> A
buffer.GetByte(i + 1) -> R
buffer.GetByte(i + 2) -> G
buffer.GetByte(i + 3) -> B

To make this efficient, we can load all pixels into memory using ToArray method.

var pixelArray = bitmap.Buffers[0].Buffer.ToArray();

To address the pixel value (4 bytes) at two-dimensional point (x,y) inside a one-dimensional array, we use the following transformation

uint bitmapPixelIndex = (uint) (y * tWidth + x) * 4;
pixels[bitmapPixelIndex] // Pixel red value at x, y
pixels[bitmapPixelIndex +1 ] // Pixel green value at x, y
pixels[bitmapPixelIndex +2 ] // Pixel blue value at x, y
pixels[bitmapPixelIndex +3 ] // Pixel alpha value at x, y
  • Variable y represents bitmap pixel row position
  • Variable x represents bitmap pixel column position
  • Constant tWidth represents number of pixels inside a bitmap row.

Mapping bitmaps pixels to the output image

Every pixel of every bitmap must find its unique position on the output canvas. Where the pixel falls depends on which tile position bitmap holds (tile row and tile column represent the tile position) and the position index of the pixel inside the bitmap.

This step will include little more thinking.

  • We have to map bitmaps to the tile positions on the output image. The position index in the bitmaps list will define a tile's position on the output image. (The first bitmap in the list will be put as the first tile in the upper left corner, second will be right next to it and so on.)
  • We have to map and copy individual bitmap pixels to the appropriate output pixels. We have to be careful not to escape the array boundaries (case when not all tiles completely fall in the output canvas)
Mapping.png

We use following transformation to map a bitmap pixel position (x,y) to the output pixel position outputPixelIndex

var outputPixelIndex = outputWidth * tHeight * tRow + tCol * tWidth + y * outputWidth + x;
  • Variable y represents bitmap pixel row position
  • Variable x represents bitmap pixel column position
  • Variable tRow represents targeted tile row index.
  • Variable tCol represents targeted tile colum index.
  • Constant tHeight represents how many pixels are in the tile colum (bitmap column).
  • Constant tWidth represents how many pixels are in the tile row (bitmap column).
  • Constant outputWidth represents how many pixels are in the output image row.

Secure index out of range cases

We have to ignore all the pixels that go beyond the output range. We have two cases to secure:

  • Tile width goes beyond the output boundaries.
  • Tile height goes beyond the output boundaries.

For every tile we are evaluating, we have to calculate how many pixels will fall out of range. When traversing a tile, we will acknowledge only pixels that are safe to copy.

var widthOutOfRangePixels = 0;
// If this tile will make us go out of output source width,
// mark how many pixels aren't safe to copy
if ((tCol + 1) * tWidth > outputWidth)
{
widthOutOfRangePixels = (tCol + 1) * tWidth - outputWidth;
}
 
var heightOutOfRangePixels = 0;
// If this tile will make us go out of output source height,
// mark how many pixels aren't safe to copy
if ((tRow + 1) * tHeight > outputHeight)
{
heightOutOfRangePixels = (tRow + 1) * tHeight - outputHeight;
}
 
for (int y = 0; y < _tileSize.Height - heightOutOfRangePixels; y++)
{
for (int x = 0; x < _tileSize.Width - widthOutOfRangePixels; x++)
{
// Pixel mapping
}
}


TileEffect Complete Source

Below you can find a complete TileEffect source. Because every bitmap maps to its own personal area on the output canvas, we can enrole multiple threads to do the mapping and copying in parallel. Code below is using portable TPL library to make this easy.

Note.pngNote: You can replace Parallel.For loop with regular for loop

    class TileEffect : CustomEffectBase
{
private readonly IList<IImageProvider> _tileImageProviders;
private IList<Bitmap> _tileBitmapSources = new List<Bitmap>();
private readonly bool _shuffle;
private readonly Size _tileSize;
 
public TileEffect(IList<IImageProvider> tileImageProviders, Size tileSize, Size outputResolution, bool shuffle = false)
: base(new BitmapImageSource(new Bitmap(outputResolution, ColorMode.Bgra8888)))
{
if (tileSize.Width > outputResolution.Width || tileSize.Height > outputResolution.Height)
{
throw new ArgumentException("Tile dimensions can't be larger than output resolution");
}
_tileSize = tileSize;
_tileImageProviders = tileImageProviders;
_shuffle = shuffle;
}
 
 
 
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion outputPixelRegion)
{
var horizontalTilesNumber = (int)Math.Ceiling(outputPixelRegion.ImageSize.Width / _tileSize.Width);
var verticalTilesNumber = (int)Math.Ceiling(outputPixelRegion.ImageSize.Height / _tileSize.Height);
 
 
var imageProcessedCounter = 0;
foreach (var imageProvider in _tileImageProviders)
{
// Create new bitmap from image source (resized to tile size), synchronous
var tileBitmap = new Bitmap(_tileSize, ColorMode.Bgra8888);
imageProvider.GetBitmapAsync(tileBitmap, OutputOption.Stretch).AsTask().Wait();
 
// Add to tile bitmap sources
_tileBitmapSources.Add(tileBitmap);
imageProcessedCounter++;
 
if (imageProcessedCounter >= horizontalTilesNumber * verticalTilesNumber)
{
// We can fill all tiles -> stop
break;
}
}
 
int bitmapPointer = 0;
// If some tile places are empty -> Add existing bitmaps to fill remaining tiles
while (_tileBitmapSources.Count < horizontalTilesNumber * verticalTilesNumber)
{
_tileBitmapSources.Add(_tileBitmapSources.ElementAt(bitmapPointer % _tileBitmapSources.Count));
bitmapPointer++;
}
 
if (_shuffle)
{
var random = new Random();
_tileBitmapSources = _tileBitmapSources.OrderBy(x => random.Next()).ToList();
}
 
 
Parallel.For(0, _tileBitmapSources.Count, i =>
{
// Tile column index
int tCol = i % horizontalTilesNumber;
// Tile row index
int tRow = i / horizontalTilesNumber;
 
int outputWidth = (int)outputPixelRegion.ImageSize.Width;
int outputHeight = (int)outputPixelRegion.ImageSize.Height;
 
// Tile width in pixels
int tWidth = (int)_tileSize.Width;
// Tile Height in pixels
int tHeight = (int)_tileSize.Height;
 
// Get bitmap pixels array. Faster than accessing
// individual bytes from the buffer
var bitmapPixels = _tileBitmapSources[i].Buffers[0].Buffer.ToArray();
 
var widthOutOfRangePixels = 0;
// If this tile will make us go out of output source width,
// mark how many pixels aren't safe to copy
if ((tCol + 1) * tWidth > outputWidth)
{
widthOutOfRangePixels = (tCol + 1) * tWidth - outputWidth;
}
var heightOutOfRangePixels = 0;
// If this tile will make us go out of output source height,
// mark how many pixels aren't safe to copy
if ((tRow + 1) * tHeight > outputHeight)
{
heightOutOfRangePixels = (tRow + 1) * tHeight - outputHeight;
}
 
for (int y = 0; y < _tileSize.Height - heightOutOfRangePixels; y++)
{
for (int x = 0; x < _tileSize.Width - widthOutOfRangePixels; x++)
{
// Location of the pixel we will copy?
uint bitmapTileIndex = (uint)(y * tWidth + x) * 4;
// What pixel will we copy?
var r = bitmapPixels[bitmapTileIndex];
var g = bitmapPixels[bitmapTileIndex + 1];
var b = bitmapPixels[bitmapTileIndex + 2];
var a = bitmapPixels[bitmapTileIndex + 3];
 
// Location on the output image we will copy pixel to
var outputPixelIndex = outputWidth * tHeight * tRow + tCol * tWidth +
y * outputWidth + x;
 
// Copy pixel
outputPixelRegion.ImagePixels[outputPixelIndex] = FromColor(new Color() { A = a, R = r, B = b, G = g });
}
 
}
});
 
}
}

Mosaic Effect Implementation

The main building blocks for our mosaic are the tiles introduced in the TileEffect class. The final stage of creating the photomosaic is to blend its output with the original image using the Nokia Imaging SDK's blend filter. The process of blending mixes the arbitrarily placed tiles with the parts of the main image, which will effectively tint the tiles at each position to the correct colour. It will look as if they are constructing the main image.

We have already seen blend in action, but we have many more blend options to play with.

Note.pngNote: You can experiment with different blend options and blend levels. This article will take blend option overlay to achieve the best mosaic effect.

The following code takes our tile effect image provider (output is a grid of arbitrarily placed images) and blends it with the main (original) image source.

// Create a tile effect with tile size 60x60 and output resolution 600x600
using (var tileEffect = new TileEffect(_imageSources, new Size(60, 60), new Size(600, 600)))
// Create new blend filter with blend overlay function and use our tile source as the foreground image
using (var blendFilter = new BlendFilter(tileEffect) { BlendFunction = BlendFunction.Overlay, Level = 1 })
// Create new filter effect that will use blend filter to process our main image
using (var filterEffect = new FilterEffect(mainImageSource) { Filters = new[] { blendFilter } })
// Instantiate renderer that will render filter effect on screen
using (var renderer = new WriteableBitmapRenderer(filterEffect, result))
{
await renderer.RenderAsync();
OriginalImage.Source = result;
}

We can see that we chained multiple components one to another. Blend filter took our TileEffect as the foreground image provider.

Chained components


The output image is shown below.

Mosaic2.png


Gallery of different blend effects

We can use different blend effects. Some examples of possible outputs are shown below:

Summary

In this article we showed how to use the Imaging SDK filters and effects on the image providers. We created a new custom effect and showed how to use it together with the existing SDK components. The focus of this article was on creating a custom effect class that could combine multiple images on the same canvas. The article explored and explained some of the problems one would face when trying to implement a similar feature. In the end, we used this custom effect class together with the built-in blend filter to achieve a simple yet beautiful mosaic effect.

This page was last modified on 22 January 2014, at 09:31.
286 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×