×
Namespaces

Variants
Actions
(Difference between revisions)

Green Screen Magic (Chroma Key) with Nokia Imaging SDK

From Nokia Developer Wiki
Jump to: navigation, search
hamishwillee (Talk | contribs)
m (Hamishwillee - - Introduction)
summeli (Talk | contribs)
(Summeli - - Image processing chain for Green Screen helper)
Line 78: Line 78:
 
The image processing chain is created as follows:  
 
The image processing chain is created as follows:  
 
<code csharp>
 
<code csharp>
 +
//the image provider has been set as the cameraImageSource from the previous code snippet.
 +
var _imageProvider = cameraImageSource;
 +
 
//This function sets up the filter settings that are used later to provide filtered images to the stream  
 
//This function sets up the filter settings that are used later to provide filtered images to the stream  
 
public void SetFilterSettings(Windows.UI.Color color, double distance)
 
public void SetFilterSettings(Windows.UI.Color color, double distance)

Revision as of 15:24, 26 November 2013

This article shows how to use the Chroma Key (Green Screen) with Nokia Imaging SDK.

Note.pngNote: This is an entry in the Nokia Imaging and Big UI Wiki Competition 2013Q4.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code Example
Source file: GreenScreenMagic (github)
Tested with
Devices(s): Nokia Lumia 820
Compatibility
Platform(s):
Windows Phone 8
Dependencies: Nokia Imaging SDK 1.0
Article
Created: summeli (23 Nov 2013)
Last edited: summeli (26 Nov 2013)

Contents

Introduction

Chroma Keying (also known as "green screen" filtering) is a special effect that is often used for replacing the background from the subject of a photo or video based on its colour hues – for example replacing the green screen behind a weather reporter with a weather map.

This article demonstrates how to use the Chroma Key filter from the Nokia Imaging SDK 1.0, how to use the camera as an image source, how to create a custom filter and add it to the image processing chain, how to illustrate the alpha channel (after chroma key) to the user by painting the transparent area black (all in real time), and how to blend two images together after one area has been marked as transparent with the chroma key.

The two code example apps described in the article are:

  • Green Screen Helper, an app to help users set up a high quality green screen in preparation for Chroma Key photography. The application uses the chroma key filter in real time to make a selected colour black - enabling the user to see which parts of the screen will not be properly filtered when applying the chroma key filter. This is helpful, since it then becomes easy to see where the editing scene needs to be fixed - by adjusting the lights, or fixing wrinkles. In addition, the hue range sensitivity accepted by the filter can be adjusted: a broader hue range will be more tolerant of poor lighting and wrinkles in the background, but may result in some of the subject also being selected.
  • Green Screen Magic, a simple photo editing app which will replace the green screen area with a background. This app uses the same approach as the previous app to help the user to select the chroma key, and adjust sensitivity. After that it uses the blend functionality to combine two images.


Note that the UIs of both apps are very simple: this article concentrates on showing how to use the Nokia Imaging SDK effectively.

Pre-requisites

Nokia Imaging SDK Core concepts should be read before going further with this article. A basic knowledge of Windows Phone UI system is also recommended but not essential.


Green Screen Helper

A brief video demo of the Green Screen Helper app is given below.


The UI is really simple. It is comprised of a Canvas to display the camera preview and a Slider to set the hue sensitivity of the chroma key filter.

GreenScreen Helper UI

The canvas has a background of VideoBrush, which will show the filtered viewfinder output as video

 <Canvas x:Name="viewfinderCanvas" Tap="viewfinderCanvas_Tap">
<!--Camera viewfinder -->
<Canvas.Background >
<VideoBrush x:Name="viewfinderBrush" />
</Canvas.Background>
</Canvas>

The Slider is used to set the colour distance for the chroma key. The SDK allows a value between 0 and 1, but the "usable" range for this type of application is somewhere between 0 and 0.5, so we limit the Slider maximum value to 0.5.

<Slider Name="accuracySlider"  Minimum="0" Maximum="0.5" Value="0.2" SmallChange="0.05" LargeChange="0.1" ValueChanged="accuracySlider_ValueChanged"/>

Using the Camera as Image Provider

Nokia Imaging SDK has a CameraPreviewImageSource which allows you to filter image data directly from the camera. Using it is really straightforward:

var photoDevice = await PhotoCaptureDevice.OpenAsync(sensorLocation, new Windows.Foundation.Size(640, 480));
var cameraImageSource = new CameraPreviewImageSource(_photoDevice);

Image processing chain for Green Screen helper

The image processing chain is shown below. We use the Camera as an image provider, and then follow it with the chroma key filter, transparent to black filter. Finally we render the result into new buffer.

Green Screen Helper image processing chain

The image processing chain is created as follows:

//the image provider has been set as the cameraImageSource from the previous code snippet.
var _imageProvider = cameraImageSource;
 
//This function sets up the filter settings that are used later to provide filtered images to the stream
public void SetFilterSettings(Windows.UI.Color color, double distance)
{
var filters = new List<IFilter>();
//color and distance are set by the user
filters.Add(new ChromaKeyFilter(color, distance, 1, false));
_filterEffect = new FilterEffect(_imageProvider)
{
Filters = filters
};
 
_transparentEffect = new TransparentToBlackFilter(_filterEffect);
}
 
//Here we use the filter chain from previous step, and rendering the result into frameBuffer
private async Task GetNewFrameAndApplyEffect(IBuffer frameBuffer)
{
if (_efectSemaphore.WaitOne(500))
{
uint scanlineByteSize = (uint)_frameSize.Width * 4; // 4 bytes per pixel in BGRA888 mode
var bitmap = new Nokia.Graphics.Imaging.Bitmap(_frameSize, ColorMode.Bgra8888, scanlineByteSize, frameBuffer);
 
 
if (_transparentEffect != null)
{
var renderer = new BitmapRenderer(_transparentEffect, bitmap);
await renderer.RenderAsync();
}
_efectSemaphore.Release();
}
}

Custom filter to convert the transparent parts of the image into black

The Chroma key filter will provide only alpha channel values 0 and 255. This is great in the green screen use case, since we want to make the selected colour range totally transparent (the weather presenter doesn't want the green screen showing through the weather map!)

Creating custom filters for Nokia Imaging SDK is simple. Just implement the OnProcess() function from the CustomEffectBase base.

My filter to convert the transparent alpha channel into black looks like this.

public class TransparentToBlackFilter : CustomEffectBase
{
public TransparentToBlackFilter(IImageProvider imageprovider)
: base(imageprovider)
{
 
}
 
 
protected override void OnProcess(PixelRegion source, PixelRegion target)
{
source.ForEachRow((index, width, pos) =>
{
for (int x = 0; x < width; ++x, ++index)
{
Color c = ToColor(source.ImagePixels[index]);
//read the alpha channel, and if it's transparent, then convert the pixel into black
if (c.A == 0)
{
c.A = 255;
c.B = 0;
c.G = 0;
c.R = 0;
}
 
target.ImagePixels[index] = FromColor(c);
}
});
}
}

Displaying the filtered stream to the UI

Finally we're just using the Image source created in the previous step, and we're displaying the results on the UI.

_filtteredImageSource = new AlphaFiltteredImageSource(_cameraImageSource, new Windows.Foundation.Size(640, 480));
 
_mediaElement = new MediaElement();
_mediaElement.Stretch = Stretch.UniformToFill;
_mediaElement.BufferingTime = new TimeSpan(0);
_mediaElement.SetSource(_filtteredImageSource);
 
// Display camera viewfinder data in videoBrush element
viewfinderBrush.SetSource(_mediaElement);

Selecting the transparent colour for the Chroma Key filter from the viewfinder

Now that we actually have the filtered image processing chain going into the viewfinder (with default pure green colour) we can detect what colour the user actually prefers. This is done by rendering the canvas from the UI into a bitmap, and reading the pixel values from tapped area. At the same time we are also reading the current slider value, and applying those settings to the image processing chain.

private void viewfinderCanvas_Tap(object sender, GestureEventArgs e)
{
//pause filter until we have selected the new colour to be removed from the stream
_filtteredImageSource.pause();
int height = (int) viewfinderCanvas.ActualHeight;
int width = (int)viewfinderCanvas.ActualWidth;
var bitmap = new WriteableBitmap(width, height);
bitmap.Render(viewfinderCanvas, null);
bitmap.Invalidate();
 
Point pt = e.GetPosition(viewfinderCanvas);
 
//read the pixel color at pointer position
int[] Pixels = bitmap.Pixels;
int pos = (int) pt.X * (int) pt.Y;
int px = Pixels[pos];
byte[] colorArray = BitConverter.GetBytes(px);
byte blue = colorArray[0];
byte green = colorArray[1];
byte red = colorArray[2];
byte alpha = colorArray[3];
_currentColor = Windows.UI.Color.FromArgb(alpha, red, green, blue);
 
_filtteredImageSource.SetFilterSettings(_currentColor, _currentDelta);
_filtteredImageSource.resume();
}

Green Screen Magic

A brief video demo of the Green Screen Magic app is given below.


The UI for Green Screen Magic is pretty simple. It's has three buttons that are used to

  1. choose the front image with the green screen.
  2. select the background
  3. view the results.


The UI for setting the chroma key is similar to the Green Screen helper UI. It has canvas to select the image, and a button to press when the settings are done.

GreenScreen Helper UI

Using the Selected image as Image Provider

In this example we are using the selected image from the photo library as image provider for the image processing chain.

Here's how to launch the PhotoChooserTask:

PhotoChooserTask photo = new PhotoChooserTask();
photo.Completed += new EventHandler<PhotoResult>(imageChooserTaskCompleted);
photo.Show();

Now we're going to use the result as an image source for the image processing chain

private void imageChooserTaskCompleted(object sender, PhotoResult e) 
{
if (e.TaskResult == TaskResult.OK)
{
MemoryStream stream = new MemoryStream();
e.ChosenPhoto.CopyTo(stream);
//The imageBuffer is IBuffer
App.GSModel.ImageBuffer = stream.GetWindowsRuntimeBuffer();
}
}
 
//later in the model-class we're using the IBuffer like this to create the image sources
BufferImageSource frontSource = new BufferImageSource(imageBuffer)

ImageProcessing chains for Green Screen Magic

The first chain is same as with the green screen helper: ImageSource->Chroma Key ->Transparent to black -> bitmap renderer, but since we're only doing it once, it's much simpler to write.

public async Task RenderBitmapTransparentAndBlackAsync(WriteableBitmap bitmap, Windows.UI.Color color, double distance)
{
_currentDelta = distance;
_currentColor = color;
var filters = new List<IFilter>();
filters.Add(new ChromaKeyFilter(_currentColor, _currentDelta, 1, false));
 
using (BufferImageSource source = new BufferImageSource(_imageBuffer))
using (FilterEffect effect = new FilterEffect(source) { Filters = filters })
using (TransparentToBlackFilter blackEffect = new TransparentToBlackFilter(effect))
using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(blackEffect, bitmap))
{
await renderer.RenderAsync();
bitmap.Invalidate();
}
}

The second image processing buffer is a bit different. Instead of the transparent to black filter we are using the blend filter with background to combine two images together.

Green Screen Magic image processing chain

The image processing chain can be created like this:

public async Task RenderResultBitmap(WriteableBitmap bitmap)
{
//use currentColor and delta from the preview
var chomafilters = new List<IFilter>();
chomafilters.Add(new ChromaKeyFilter(_currentColor, _currentDelta, 1, false));
 
// First remove the chroma key from the fron image, and then combine images 1, and 2
using (BufferImageSource frontSource = new BufferImageSource(_imageBuffer))
using (FilterEffect front = new FilterEffect(frontSource) { Filters = chomafilters })
//combine images
using (BufferImageSource background = new BufferImageSource(_bgBuffer))
using (FilterEffect filters = new FilterEffect(background))
using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(filters, bitmap))
{
filters.Filters = new IFilter[] { new BlendFilter(front, BlendFunction.Normal) };
await renderer.RenderAsync();
bitmap.Invalidate();
}
}


Summary

This article has delivered a method of illustrating the alpha channel (after chroma key) to the user by painting the transparent area black. It has also shown how to do this in real time, as well as for the single images. It has also shown how to blend two images together after one area has been marked as transparent with the chroma key.

The source code for this example is licensed under MIT license.

363 page views in the last 30 days.