×
Namespaces

Variants
Actions
(Difference between revisions)

Green Screen Magic (Chroma Key) with Nokia Imaging SDK

From Nokia Developer Wiki
Jump to: navigation, search
summeli (Talk | contribs)
(Summeli - - Custom filter to convert the transparent parts of the image into black)
hamishwillee (Talk | contribs)
m (Hamishwillee - Bot update - Fix metadata)
Line 7: Line 7:
 
|devices= Nokia Lumia 820 <!-- Devices tested against - e.g. Nokia Lumia 928, Nokia Asha 501) -->
 
|devices= Nokia Lumia 820 <!-- Devices tested against - e.g. Nokia Lumia 928, Nokia Asha 501) -->
 
|sdk= <!-- SDK(s) built and tested against (e.g. Windows Phone 8.0 SDK) -->
 
|sdk= <!-- SDK(s) built and tested against (e.g. Windows Phone 8.0 SDK) -->
|dependencies= <!-- Any other/external dependencies e.g.: Google Maps Api v1.0 -->  
+
|dependencies= <!-- Any other/external dependencies e.g.: Google Maps Api v1.0 -->
|signing=<!-- Special Signing requirements -->
+
|signing= <!-- Special Signing requirements -->
 
|capabilities= <!-- Required capabilities for code (e.g. ID_CAP_LOCATION, ID_CAP_NETWORKING) -->
 
|capabilities= <!-- Required capabilities for code (e.g. ID_CAP_LOCATION, ID_CAP_NETWORKING) -->
 
|language= <!-- Language category code for non-English topics - e.g. Lang-Chinese -->
 
|language= <!-- Language category code for non-English topics - e.g. Lang-Chinese -->
 
|translated-by= <!-- [[User:XXXX]] -->
 
|translated-by= <!-- [[User:XXXX]] -->
|translated-from-title= <!-- Title only - not link -->  
+
|translated-from-title= <!-- Title only - not link -->
 
|translated-from-id= <!-- Id of translated revision -->
 
|translated-from-id= <!-- Id of translated revision -->
|review-by=<!-- After re-review: [[User:username]] -->
+
|review-by= <!-- After re-review: [[User:username]] -->
 
|review-timestamp= <!-- After re-review: YYYYMMDD -->
 
|review-timestamp= <!-- After re-review: YYYYMMDD -->
 
|update-by= <!-- After significant update: [[User:username]]-->
 
|update-by= <!-- After significant update: [[User:username]]-->

Revision as of 07:02, 26 November 2013

This article shows how to use the Chroma Key (Green Screen) with Nokia Imaging SDK.

SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code Example
Source file: GreenScreenMagic (github)
Tested with
Devices(s): Nokia Lumia 820
Compatibility
Platform(s):
Windows Phone 8
Article
Created: summeli (23 Nov 2013)
Last edited: hamishwillee (26 Nov 2013)

Contents

Introduction

Note.pngNote: This is an entry in the Nokia Imaging and Big UI Wiki Competition 2013Q4.

In this article we're going to create two simple apps to demonstrate how to use the Chroma Key filter with Nokia Imaging SDK 1.0. The first app is called Green Screen Helper, which will help the user to set up high quality green screen. This could be quite handy, if you're building your home studio, and trying to set up the green screen for the first time. Basically the application uses the chroma key filter in real time, to make the selected color transparent. After that we're going to change the transparent pieces of the frame into black so the user can see the end result of applying the chroma key filter into the scene. This is really helpful, since you can fix editing scene easily by adjusting the lights, or fixing wrinkles from the green sceen.


The second app is called Green Screen Magic, it's a simple photo editing app witch will replace the green screen area with a background (like a weather map). This app will use the same transparency to black filters as the previous app to help the user to select the chroma key, and adjust sensitivity etc. After that it will use the blend functionality to combine two images.


Pre-requisites

This article is focused into creating application with the Nokia Imaging SDK, so it's not covering the UI creation parts in details. The UIs are very simple, so basic knowledge of Windows Phone UI system is recommended (not essential). I also recommend reading the Nokia Imaging SDK Core concepts before going further with this article.

Green Screen Helper

The UI is really simple in this app. It's basically just a canvas and a slider to set the color distnace to the chroma key in Nokia imaging sdk.

GreenScreen Helper UI

The canvas has a background of videoBrush, which will show the filtered viewfinder as video

 <Canvas x:Name="viewfinderCanvas" Tap="viewfinderCanvas_Tap">
<!--Camera viewfinder -->
<Canvas.Background >
<VideoBrush x:Name="viewfinderBrush" />
</Canvas.Background>
</Canvas>

The slider will be used to set the color distance for the chroma key. The distance range in the SDK goes from 0 into 1, but the real "usable" range for this type of application is somewhere between 0, and 0.5, so we're going to limit the slider into maximum of 0.5

 <Slider Name="accuracySlider"  Minimum="0" Maximum="0.5" Value="0.2" SmallChange="0.05" LargeChange="0.1" ValueChanged="accuracySlider_ValueChanged"/>

Using the Camera as Image Provider

Nokia Imaging SDK has a CameraPreviewImageSource witch is made for the camera. Using it is really straightforward

var photoDevice = await PhotoCaptureDevice.OpenAsync(sensorLocation, new Windows.Foundation.Size(640, 480));
var cameraImageSource = new CameraPreviewImageSource(_photoDevice);

Image processing chain for Green Screen helper

Now we are going to use the Camera as an imageproceder like described in the previous chapter. After the camera we are putting the chroma key filter, transparent to black filter, and after that we're going to render the result into new buffer.

Green Screen Helper image processing chain

The image processing chain can be created like this:

//this function sets up the filter settings that are used later to provide filttered images to the stream 
public void SetFilterSettings(Windows.UI.Color color, double distance)
{
var filters = new List<IFilter>();
//color and distance are set by the user
filters.Add(new ChromaKeyFilter(color, distance, 1, false));
_filterEffect = new FilterEffect(_imageProvider)
{
Filters = filters
};
 
_transparentEffect = new TransparentToBlackFilter(_filterEffect);
}
 
//in here we are using the filter chain from previous step, and rendering the result into framebuffer
private async Task GetNewFrameAndApplyEffect(IBuffer frameBuffer)
{
if (_efectSemaphore.WaitOne(500))
{
uint scanlineByteSize = (uint)_frameSize.Width * 4; // 4 bytes per pixel in BGRA888 mode
var bitmap = new Nokia.Graphics.Imaging.Bitmap(_frameSize, ColorMode.Bgra8888, scanlineByteSize, frameBuffer);
 
 
if (_transparentEffect != null)
{
var renderer = new BitmapRenderer(_transparentEffect, bitmap);
await renderer.RenderAsync();
}
_efectSemaphore.Release();
}
}

Custom filter to convert the transparent parts of the image into black

The Chroma key filter will provide only alpha channel values 0 and 255. This is great in the green screen use case, since we want to make the selected color range totally transparent. Imagine a weather forecast where there would be some leftover from the green screen on top of the weather map.

Creating custom filters for Nokia Imaging SDK is simple. Just implement the OnProcess function from the CustomEffectBase base. My filter to convert the transparent alpha channel into black looks like this.

public class TransparentToBlackFilter : CustomEffectBase
{
public TransparentToBlackFilter(IImageProvider imageprovider)
: base(imageprovider)
{
 
}
 
 
protected override void OnProcess(PixelRegion source, PixelRegion target)
{
source.ForEachRow((index, width, pos) =>
{
for (int x = 0; x < width; ++x, ++index)
{
Color c = ToColor(source.ImagePixels[index]);
//read the alpha channel, and if it's transparent, then convert the pixel into black
if (c.A == 0)
{
c.A = 255;
c.B = 0;
c.G = 0;
c.R = 0;
}
 
target.ImagePixels[index] = FromColor(c);
}
});
}
}

Displaying the filtered stream to the UI

Finally we're just using the Image source created in the previous step, and we're displaying the results on the UI.

_filtteredImageSource = new AlphaFiltteredImageSource(_cameraImageSource, new Windows.Foundation.Size(640, 480));
 
_mediaElement = new MediaElement();
_mediaElement.Stretch = Stretch.UniformToFill;
_mediaElement.BufferingTime = new TimeSpan(0);
_mediaElement.SetSource(_filtteredImageSource);
 
// Display camera viewfinder data in videoBrush element
viewfinderBrush.SetSource(_mediaElement);

Selecting the transparent color for the Chroma Key filter from the viewfinder

Now that we actually have the filtered image processing chain going into the viewfinder (with default pure green color) we can detect what color the user actually prefers. This is done with a simple strategy where we are just rendering the canvas from the UI into a bitmap, and reading the pixel values from tapped area. At the same time we are also reading the current slider value, and applying those settings to the image processing chain as well.

private void viewfinderCanvas_Tap(object sender, GestureEventArgs e)
{
//pause filter untill we have selected the new color to be removed from the stream
_filtteredImageSource.pause();
int height = (int) viewfinderCanvas.ActualHeight;
int width = (int)viewfinderCanvas.ActualWidth;
var bitmap = new WriteableBitmap(width, height);
bitmap.Render(viewfinderCanvas, null);
bitmap.Invalidate();
 
Point pt = e.GetPosition(viewfinderCanvas);
 
//read the pixel color at pointer position
int[] Pixels = bitmap.Pixels;
int pos = (int) pt.X * (int) pt.Y;
int px = Pixels[pos];
byte[] colorArray = BitConverter.GetBytes(px);
byte blue = colorArray[0];
byte green = colorArray[1];
byte red = colorArray[2];
byte alpha = colorArray[3];
_currentColor = Windows.UI.Color.FromArgb(alpha, red, green, blue);
 
_filtteredImageSource.SetFilterSettings(_currentColor, _currentDelta);
_filtteredImageSource.resume();
}

Green Screen Magic

The UI for Green Screen Magic is pretty simple. It's only three buttons that the user can to 1. choose the front image with the green screen. 2. to select the background, and 3. button to view the results.

The UI for setting the chroma key is similar to the Green Screen helper UI. It has canvas to select the image, and a button to press when the settings are done.

GreenScreen Helper UI

Using the Selected image as Image Provider

In this example we are using the selected image from the photo library as image provider for the image processing chain.

Here's how to launch the photochooser task to the UI

PhotoChooserTask photo = new PhotoChooserTask();
photo.Completed += new EventHandler<PhotoResult>(imageChooserTaskCompleted);
photo.Show();

Now we're going to use the result as an image source for the image processing chain

private void imageChooserTaskCompleted(object sender, PhotoResult e) 
{
if (e.TaskResult == TaskResult.OK)
{
MemoryStream stream = new MemoryStream();
e.ChosenPhoto.CopyTo(stream);
//The imageBuffer is IBuffer
App.GSModel.ImageBuffer = stream.GetWindowsRuntimeBuffer();
}
}
 
//later in the model-class we're using the IBuffer like this to create the image sources
BufferImageSource frontSource = new BufferImageSource(imageBuffer)

ImageProcessing chains for Green Screen Magic

The first chain is same as with the green screen helper; ImageSource->Chroma Key ->Transparent to black -> bitmap renderer, but since we're only doing it once, it's much simpler to write.

public async Task RenderBitmapTransparentAndBlackAsync(WriteableBitmap bitmap, Windows.UI.Color color, double distance)
{
_currentDelta = distance;
_currentColor = color;
var filters = new List<IFilter>();
filters.Add(new ChromaKeyFilter(_currentColor, _currentDelta, 1, false));
 
using (BufferImageSource source = new BufferImageSource(_imageBuffer))
using (FilterEffect effect = new FilterEffect(source) { Filters = filters })
using (TransparentToBlackFilter blackEffect = new TransparentToBlackFilter(effect))
using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(blackEffect, bitmap))
{
await renderer.RenderAsync();
bitmap.Invalidate();
}
}

The second image processing buffer is a bit different. Instead of the transparent to black filter we are using the blend filter with background to combine two images together.

Green Screen Magic image processing chain

The image processing chain can be created like this:

public async Task RenderResultBitmap(WriteableBitmap bitmap)
{
//use currentColor and delta from the preview
var chomafilters = new List<IFilter>();
chomafilters.Add(new ChromaKeyFilter(_currentColor, _currentDelta, 1, false));
 
// First remove the chroma key from the fron image, and then combine images 1, and 2
using (BufferImageSource frontSource = new BufferImageSource(_imageBuffer))
using (FilterEffect front = new FilterEffect(frontSource) { Filters = chomafilters })
//combine images
using (BufferImageSource background = new BufferImageSource(_bgBuffer))
using (FilterEffect filters = new FilterEffect(background))
using (WriteableBitmapRenderer renderer = new WriteableBitmapRenderer(filters, bitmap))
{
filters.Filters = new IFilter[] { new BlendFilter(front, BlendFunction.Normal) };
await renderer.RenderAsync();
bitmap.Invalidate();
}
}


Summary

This article has delivered a method of illustrating the alpha channel (after chroma key) to the user by painting the transparent area black. It has also shown how to do this in real time, as well as for the single images. It has also shown how to blend two images together after one area has been marked as transparent with the chroma key.

The source code for this example is licensed under MIT license.

201 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×