×
Namespaces

Variants
Actions
(Difference between revisions)

Optimizing Imaging SDK use for rapidly changing filter parameters

From Nokia Developer Wiki
Jump to: navigation, search
hamishwillee (Talk | contribs)
m (Hamishwillee - Remove competition entry note - no longer relevant)
croozeus (Talk | contribs)
m (Croozeus - Adding to the FA List)
Line 1: Line 1:
 
[[Category:Imaging on Windows Phone]][[Category:Optimization on Windows Phone]][[Category:Code Examples]][[Category:Windows Phone 8]][[Category:XAML]]
 
[[Category:Imaging on Windows Phone]][[Category:Optimization on Windows Phone]][[Category:Code Examples]][[Category:Windows Phone 8]][[Category:XAML]]
 +
{{FeaturedArticle|timestamp=20130929}}
 
{{Abstract|This article explains how to use the Nokia Imaging SDK with user interaction efficiently. }}
 
{{Abstract|This article explains how to use the Nokia Imaging SDK with user interaction efficiently. }}
 
{{ArticleMetaData <!-- v1.3 -->
 
{{ArticleMetaData <!-- v1.3 -->

Revision as of 18:54, 29 September 2013

Featured Article
29 Sep
2013

This article explains how to use the Nokia Imaging SDK with user interaction efficiently.

WP Metro Icon Graph1.png
SignpostIcon XAML 40.png
WP Metro Icon WP8.png
Article Metadata
Code ExampleTested with
Devices(s): Lumia 920, 820, 620,
Compatibility
Platform(s):
Windows Phone 8
Dependencies: Nokia Imaging SDK
Article
Created: yan_ (30 Jul 2013)
Last edited: croozeus (29 Sep 2013)

Contents

Introduction

Applying image effects with the Nokia Imaging SDK is both easy and efficient. However there are use-cases where a naive application of the filters can result in unnecessary memory use and processing, which can in turn result in less-than-smooth UI behavior. One such example is when a user connects image processing code directly to the value of a slider, and generates a new version of the image on every slider event.

This article provides an an overview of filter use and "limitations", an explanation of how they can be misused (taking the slider case as an example) and provides a state machine that can be used to handle the user interaction more effectively.

Tip.pngTip: The "Filter effects" example in the Imaging SDK uses a "naive" implementation. An improved version is provided at the end of this article.

Pre-requisites

A basic understanding of how the Imaging SDK is used is recommended (but not essential). The links below provide a good starting point:

EditingSession

To manipulate a picture you must create an EditingSession. This class can use a picture pixel buffer or work directly with an image file. To work with high resolution pictures, you must use the encoded file (the SDK will decode only the pixels it needs).

If you want to use an encoded file stream or StorageFile, you should use EditingSessionFactory. This factory function will copy the encoded file to a IBuffer object and instantiate an EditingSession asynchronously.

Unmanaged resources

The Nokia Imaging SDK is a WinPRT component coded in C++ and allocating unmanaged resources. Since C# uses a Garbage Collector, you don't know when an EditingSession, with its unmanaged resources, will be deleted. Application memory can grow quickly if developers do not take specific action to release them promptly.

For this reason, EditingSession implements the IDisposable interface. This interface indicates that you can de-allocate unmanaged resources by calling the Dispose() function.

To simplify IDisposable class use, C# provides the using keyword. This keyword is equivalent to try/finally where Dispose() is called in the finally section.

With using keyword
//define session variable and instantiate an EditingSession
using(var session = await EditingSessionFactory.CreateEditingSessionAsync(inputStream))
{
session.addFilter( FilterFactory.CreateAntiqueFilter() );
await _session.RenderToImageAsync(resultImage);
}//session.Dispose is called
Without using keyword
{
EditingSession session = null;
try
{
//instantiate an EditingSession
session = await EditingSessionFactory.CreateEditingSessionAsync(inputStream));
 
session.addFilter( FilterFactory.CreateAntiqueFilter() );
await _session.RenderToImageAsync(resultImage);
}
finally
{
if(session != null) session.Dispose(); //call Dispose
session = null;
}
}

Please call Dipose() manually or use using keyword - depending on your application context. The using keyword is preferred if you want to apply an effect on your picture and then won't need to further re-use the EditingSession.

Filters

After you have instantiated an EditingSession, you need to stack filters with the AddFilter() function. Each filter is instantiated with FilterFactory and the instance is Invariant (i.e. its parameters can't be modified).

So, if you want to modify a filter in the stack, you must remove it and add the modified filter. EditingSession provides only two methods to remove a filter:

  • Undo(): remove the last filter added to the stack.
  • UndoAll() : remove all filters.

If you want to modify only the last filter added to the stack, use Undo(). If you want to modify another filter, it is simpler to use UndoAll() and recreate the filter stack.

Once filters have been added to the session, you can asynchronously generate the final image to a Bitmap with RenderToBitmapAsync() methods, or to a stream with RenderToJpegAsync() methods.

Warning.pngWarning: Rendering is asynchronous, and you can't reuse the session until processing completes.

User interaction

Filters are invariant - every time a parameter is changed a new filter(s) needs to be created. Furthermore, after we start rendering an image we can't reuse a EditingSession until it completes. Applying image effects with the Nokia Imaging SDK is easy and efficient, so usually these limitations have little effect on app design.

There are however user interactions where a naive design can have an impact - for example if parameters can change rapidly resulting in creation of unnecessary EditingSession and filter objects.

The following sections explain this problem, and the solution, in the context of a slider controlling a filter parameter on an image. To simplify code, we use stream file picture as EditingSession input and Image control as rendering output.

Naive method

The Slider raises ValueChanged events when the user changes the slider position. A naive implementation would generate new images (and hence new filters) when the event is raised.

private void filterparam_ValueChanged(object sender, RoutedPropertyChangedEventArgs<double> e)
{
using (EditingSession session = await EditingSessionFactory.CreateEditingSessionAsync(input))
{
//create filter stack
session.AddFilter(FilterFactory.CreateSolarizeFilter(filterValue));
 
//process the rendering
await session.RenderToImageAsync(output);
}
}

Warning.pngWarning: The session can't be reused while it is processing. Therefore in this case a new EditingSession is created at each raised event.

When a user moves the slider, many events are raised. With the above implementation each event results in recreation of the session and filters and a new asynchronous (parallel!) process to apply the effect to the image. Not only is generation of the parallel images not necessary, but this approach has the following implications:

  • The app will use 100% of CPU and allocate a lot of unmanaged resources.
  • The application GUI can be affected, resulting in "jerky" rendering of the image.
  • Processing time is not constant and you don't know which is the real parameters apply on the final displayed image.

Naive implementation code example

Sample code implement this solution in NokiaImagingFilter.cs file.

This class is composed by three properties and one function:

  • Input: file stream
  • Output: target Image controls.
  • filterValue: filter parameter. Range [0.0, 1.0]
  • processRendering(): process the rendering to the Image control
async void processRendering()
{
if (output == null || input == null)
return;
try
{
//reset stream position
input.Seek(0, SeekOrigin.Begin);
//create a new session
using (EditingSession session = await EditingSessionFactory.CreateEditingSessionAsync(input))
{
//create filter pipeline
session.AddFilter(FilterFactory.CreateSolarizeFilter(filterValue));
session.AddFilter(FilterFactory.CreateTemperatureAndTintFilter((int)(100 - 200 * filterValue), 0));
 
//process rendering to an Image control.
await session.RenderToImageAsync(output, OutputOption.PreserveAspectRatio);
}
}
catch (Exception)
{
}
}

Note.pngNote: To manipulate a picture, we use the file stream. So we need to reset its position before the EditionSession creation.

When a property is modified, processRendering() function is called.

Interactive State Machine

The naive method should be corrected to remove unnecessary processing, render the image display smoothly, and ensure the final image uses the correct parameters.

The approach we use to improve naive method is to create a very simple state machine.

Interactive State Machine Diagram

Note.pngNote: Interactive State Machine is really simple to implement and you can easily adapt it for other context.

This State machine has three States:

  • WAIT: wait new parameters to process a new rendering.
  • APPLY: process the rendering.
  • SCHEDULE: save new parameters for the new rendering.


And a Transition can be caused by two events :

  • requestProcessing: parameters are updated. Request a new rendering processed.
  • processFinished: rendering process is finished.


When the user begins to move the slider:

  1. the parameter is updated => APPLY State become active => process the image using current parameter
  2. if the parameter is updated and a rendering process has not finished => SCHEDULE State become active and the parameter is saved
  3. when the rendering process is finished => APPLY State become active => process a new rendering

Note.pngNote: If the user continues to move the slider, the state machine will loop between APPLY and SCHEDULE to avoid unnecessary processing.

Once the user finishes moving the slider :

  • rendering process is finished and SCHEDULE State is active => APPLY State become active => process the image using saved parameter.
  • rendering process is finished and APPLY State is active => WAIT State become active => no more process is necessary.

Note.pngNote: As the SCHEDULE state saves the last parameters every time it is updated, rendering is always done using the correct/most recent parameters.

State machine sample code

Sample code implement this solution in InteractiveNokiaImagingFilter.cs file.

The class is composed of three properties and three functions:

  • Input: file stream
  • Output: target Image controls.
  • filterValue: filter parameter. Range [0.0, 1.0]
  • requestProcessing(): A parameter is updated.
  • processRendering(): process asynchronously the rendering to the Image control.
  • processFinished(): Process rendering is finished. Called at the end of processRendering() function.


The processRendering() function is similar to the one used in the naive method described above, but with processFinished() called at the end and the EditingSession reused.

async void processRendering()
{
try
{
if (output != null && session != null)
{
//reset filter pipeline
session.UndoAll();
//create filter pipeline
session.AddFilter(FilterFactory.CreateSolarizeFilter(filterValue));
session.AddFilter(FilterFactory.CreateTemperatureAndTintFilter((int)(100 - 200 * filterValue), 0));
 
await session.RenderToImageAsync(output, OutputOption.PreserveAspectRatio);
}
}
catch (Exception)
{
}
finally
{
processFinished();
}
}

To Implement the interactive State Machine, we represent :

  • States with the enum STATE.
  • the active state with the currentState member.
enum STATE
{
WAIT,
APPLY,
SCHEDULE
};
//Current State
STATE currentState = STATE.WAIT;

Transitions are managed by requestProcessing() and processFinished() functions. These functions update the active State and call processRendering when active State is APPLY.

void requestProcessing()
{
switch (currentState)
{
//State machine transition : WAIT -> APPLY
case STATE.WAIT:
currentState = STATE.APPLY;
//enter in APPLY STATE => apply the filter
processRendering();
break;
 
//State machine transition : APPLY -> SCHEDULE
case STATE.APPLY:
currentState = STATE.SCHEDULE;
break;
 
//State machine transition : SCHEDULE -> SCHEDULE
case STATE.SCHEDULE:
currentState = STATE.SCHEDULE;
break;
}
}
void processFinished()
{
switch (currentState)
{
//State machine transition : APPLY -> WAIT.
case STATE.APPLY:
currentState = STATE.WAIT;
break;
//State machine transition : SCHEDULE -> APPLY.
case STATE.SCHEDULE:
currentState = STATE.APPLY;
//enter in APPLY STATE => apply the filter
processRendering();
break;
}
}

Note.pngNote: When SCHEDULE state is active, we need to save parameters. In this implementation we use the last updated parameters. Since parameters are the class properties, we don't need to save it again.

Warning.pngWarning: This implementation is not thread safe like slider event are called in UI thread. If you have a parameter which can be updated by another thread, you can use Deployment.Current.Dispatcher.BeginInvoke to move param update to UI Thread.

Optimizing rendering duration (optional)

This section explains an optimization, where the perceived performance is improved by trading off the output resolution for reduced calculation time.

Warning.pngWarning: This approach may or may not be useful in your particular application or use-case.
The Imaging SDK is very optimized, but unfortunately the SDK doesn't explain its optimizations. As a result, for some use-cases output resolution is an important factor, in others it may have no effect at all.

Imaging SDK rendering duration depends on a number of different factors, including:

  • Input type - encoded file or decoded buffer,
  • Input resolution,
  • Output resolution,
  • Filter(s) used,
  • Selected picture area,
  • SDK cache,
  • Hardware, memory, system
  • etc


This section explores the effects of output resolution on the rendering time for a number of different picture input resolutions. The example uses the context of the user moving the slider - rendering an intermediate "low resolution" picture much faster. To do it, we use two Writeablebitmap as Image source:

  • bitmapHR: full resolution bitmap
  • bitmapLR: low resolution bitmap


Full resolution must be the output dimension in real pixels. Since .NET control dimensions are in logical pixels these must be converted to real pixels. To convert logical pixel dimension to real pixel dimension you only need the factor given by System.Windows.Application.Current.Host.Content.ScaleFactor  :

     bitmapHR = new WriteableBitmap(
(int)(output.Width * System.Windows.Application.Current.Host.Content.ScaleFactor / 100.0f + 0.5f),
(int)(output.Height * System.Windows.Application.Current.Host.Content.ScaleFactor / 100.0f + 0.5f)
);

Low resolution is simply the full resolution dived by a factor.

    bitmapLR = new WriteableBitmap(
bitmapHR.PixelWidth / 2,
bitmapHR.PixelHeight / 2
);

Unfortunately, when a bitmap is set as Image control source, a concurrent access between SDK and Image control appeared (see #Another correction). To fix this we added a temporary WriteableBitmap as the output target. After rendering, we copy this to the Image bitmap that is displayed.

 public async Task RenderToBitmapAsync(EditingSession session)
{
// process rendering to the temporary WriteableBitmap
await session.RenderToWriteableBitmapAsync(_tmpBitmap, OutputOption.PreserveAspectRatio);
//Copy pixels
_tmpBitmap.Pixels.CopyTo(_previewBitmap.Pixels,0);
_previewBitmap.Invalidate(); // Force a redraw
}

To finish, we must know when the application must use low resolution or high resolution. If you use a slider, you can process low resolution rendering between its ManipulationStarted and ManipulationCompleted events.

 private void slider_ManipulationStarted(object sender, System.Windows.Input.ManipulationStartedEventArgs e)
{
interactiveFilter2.HRrendering = false;
}
 
private void slider_ManipulationCompleted(object sender, System.Windows.Input.ManipulationCompletedEventArgs e)
{
interactiveFilter2.HRrendering = true;
}

Note.pngNote: when rendering resolution change, requestProcessing is called.

When processRendering() is called, we can now choose between low resolution and high resolution rendering.

Sample code implement this solution in InteractiveNokiaImagingFilter2.cs file which update InteractiveNokiaImagingFilter.cs with rendering time optimization.

As stated in the beginning of this section, the benefit of this optimization will depend on a number of factors:

  • With an 8Mp picture rendering time decreases from 120ms to 90ms
  • With a 41Mp picture, rendering time is constant 90ms.
  • With another example using gestures for picture navigation with a 41Mp picture and half resolution, rendering time decreased from 500-160 ms to 180-60 ms.

Sample code

Naive method
Interactive State Machine

Sample show interactivity between the two methods and with optimized rendering duration :

  1. Run application in release.
  2. Click image icon bar and select a picture.
  3. Select a pivot page :
    • Naive page use the naive method,
    • Interactive page use Interactive State Machine,
    • Interactive2 page use Interactive State Machine with optimized rendering duration.
  4. Move the slider.

The Interactive State Machine provides better interaction with the user.

Note.pngNote: To increase the difference, you can test a 41 MP picture taken with Nokia Lumia 1020 or Nokia Pureview 808.

Improved Filter Effects SDK sample

The Nokia Imaging SDK example Filter Effects can be installed from Windows Phone Store here. This sample applies filters on a picture using a slider. All filters are based on AbstractFilter class, which implements the naive method described here, with all its problems.

To improve this sample, I've modified AbstractFilter class with Interactive State Machine. You can find modified code to test here: Media:FilterEffects_InteractiveStateMachine.zip.

Another correction

The updated version of the AbstractFilter class has another correction. The sample uses the same WriteableBitmap as Image source and for output rendering. There is concurrent access between the Imaging SDK thread and the UI, so when the user moves the slider it is possible for the image to partially display the result from a number of slider positions. The snapshot below shows this phenomena - the top and bottom part of the displayed image (separated by a red line) are the result of different parameters.

Concurrent access phenomena when user move a slider

Using the state machine removes parallel rendering of images, but the concurrent access between SDK and Image control can still result in the Image showing the result of a number of renderings. To fix this I've added a temporary WriteableBitmap as the output target. After rendering, I copy this to the Image bitmap that is displayed.

 public async Task RenderToBitmapAsync(EditingSession session)
{
// process rendering to the temporary WriteableBitmap
await session.RenderToWriteableBitmapAsync(_tmpBitmap, OutputOption.PreserveAspectRatio);
//Copy pixels
_tmpBitmap.Pixels.CopyTo(_previewBitmap.Pixels,0);
_previewBitmap.Invalidate(); // Force a redraw
}

Real example

The techniques described in this article have been used in my commercial "monsterification" (monster image editing) app: MonsterCam.

Monster Cam Tag

MonsterCam is one of first applications based on the Imaging SDK, and offers:

  • Real time ROI extraction with gestures (will be explained in a future article)
  • Applying effects with user control.
  • Monsterification is done with DirectX.

This application provide an unlimited trial version (You don't need to pay) and can be found on Windows Phone Store here

Reference

680 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×