Please note that as of October 24, 2014, the Nokia Developer Wiki will no longer be accepting user contributions, including new entries, edits and comments, as we begin transitioning to our new home, in the Windows Phone Development Wiki. We plan to move over the majority of the existing entries. Thanks for all your past and future contributions.

Talk:Custom Filter QuickStart for Nokia Imaging SDK

From Wiki
Jump to: navigation, search


RatishPhilip - Getting color components

In the working with colors section, an alternative way is to use the ToColor method of CustomEffectBase to convert the uint to Windows.UI.Color object. From this object you can obtain the RGBA components.

Similarly, you can use the FromColor method of CustomEffectBase to convert the Color object to uint.

RatishPhilip (talk) 07:19, 16 December 2013 (EET)

Hamishwillee - Thanks Ratish

Hi Rob

This looks like it could be quite fun to play with. This is a nice intro to custom filters, and in particular the format of the arrays.

I like that you've started with making it easy for people to play with custom effects by a few very simple changes to the example code. In itself that is a pretty useful article for someone who wants to play with filter code, but not to have to play with the whole framework. For this though I would probably leave the existing code in place in your example but comment it out - so that its easier to see the removal/addition.

I like the various effects you documented in the appendix. I think though that you should have some before and after shots here to demonstrate. If you got the algorithm from somewhere, you should reference those.You should also link to imaging SDK docs if there are already filters for these available.

For boilerplate code I would prefer that whatever can be documented with inline comments is. In the text what is useful to do is to say what the parameters are, any limitations - basically "what is it that makes up a custom filter". Most important, link to relevant sections in library for more information.

Lastly, this article title is not quite right. It is not "all about custom filters", though it is an interesting and useful document. A better title would be one that captures that this shows how to get up and running playing with custom effects quickly or one that captures that you've got some common algorithms (not?) in the SDK.



hamishwillee (talk) 07:59, 18 December 2013 (EET)

Hamishwillee - PS Ideally should have a code example

So that its easy to play with the filters you've shown off. Not strictly necessary given that you've started off an ND example

hamishwillee (talk) 08:01, 18 December 2013 (EET)

Rob.Kachmar - RE: Getting color components


Thanks for the suggestion; the ToColor and FromColor methods definitely make the code more intuitive. At the same time, they also have a significant performance impact to keep converting to and from a Windows.UI.Color object. I ran a very basic test through the RealtimeFilterDemo project and found that the code performs about 50% faster when using the Bitwise Shift Operator technique versus the ToColor and FromColor methods. I was only getting 11-12 FPS with the ToColor and FromColor methods. However, with the Bitwise Shift Operator technique, I saw 16-18 FPS.

If you scroll to the very bottom of the Core concepts article where CustomEffectBase is introduced, you'll see that there is a special note about the example code provided being for explanatory purposes, not optimized for performance.!nokia-imaging-sdk/core-concepts.html;#toc_CustomEffectBase

You can drop this code in to see for yourself.

   for (int x = 0; x < width; ++x, ++index)
       // 16-18 FPS (Frames Per Second)
       uint pixel = sourcePixels[index]; // get the current pixel
       int red = (int)((pixel & 0x00ff0000) >> 16); // red color component
       int green = (int)((pixel & 0x0000ff00) >> 8); // green color component
       int blue = (int)(pixel & 0x000000ff); // blue color component
       targetPixels[index] = (uint)(0xff000000 | (red << 16) | (green << 8) | blue);
       // 11-12 FPS (Frames Per Second)
       Color c = ToColor(sourcePixels[index]);
       targetPixels[index] = FromColor(c);

Rob.Kachmar (talk) 06:56, 2 January 2014 (EET)

Hamishwillee - Subedited introduction

Hi Rob

I subedited the introduction and quick setup section for readability. Please check you think it is improved too :-)

I have also re-written the guidance on using shift vs. To/From colour. The problem with this was that you linked to a comment, and there is no guarantee those won't be deleted. IMO it is better now - we've shown both alternatives but the less efficient approach is shown clearly as your "less preferred one". Please check



hamishwillee (talk) 07:31, 7 January 2014 (EET)

Rob.Kachmar - RE: Subedited introduction

Thanks Hamish, great edits as usual!

Rob.Kachmar (talk) 08:39, 7 January 2014 (EET)

Hamishwillee - How does pixelate here compare to one in ...

... Custom Effects in Nokia Imaging SDK: Pixelate and Color Quantizer

hamishwillee (talk) 08:47, 7 January 2014 (EET)

Rob.Kachmar - RE:How does pixelate here compare to one in ...

The pixelation technique in this QuickStart article performs better than the one in the Other article. Performing some quick tests using a Lumia 822, I produced these results:

   // Pixel Blocks    QuickStart    Other    Other w/o To/FromColor()
   // ============    ==========    =====    ========================
   //   5             13 FPS        6 FPS      9 FPS
   //  15             15 FPS        7 FPS     10 FPS
   //  35             16 FPS        7 FPS     10 FPS
   // 128             17 FPS        7 FPS     11 FPS
   // 480             17 FPS        N/A       N/A
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 5, 0); // 13 FPS
   //_customEffect = new PixelateAlternate(_cameraPreviewImageSource, 5); // 6 FPS >>> 9 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 15, 0); // 15 FPS
   //_customEffect = new PixelateAlternate(_cameraPreviewImageSource, 15); // 7 FPS >>> 10 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 35, 0); // 16 FPS
   //_customEffect = new PixelateAlternate(_cameraPreviewImageSource, 35); // 7 FPS >>> 10 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 128, 0); // 17 FPS
   //_customEffect = new PixelateAlternate(_cameraPreviewImageSource, 128); // 7 FPS >>> 11 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 480, 0); // 17 FPS
   //_customEffect = new PixelateAlternate(_cameraPreviewImageSource, 480); // 7 FPS auto reduced to 128 size

As you can see from the comments, the QuickStart code ran more than twice as fast as the Other code. I noticed the Other code was unnecessarily converting To and From Windows.UI.Color. This technique is not interested in manipulating the target color, just spreading it around. The "Other w/o To/FromColor() column is the output where I removed the unnecessary conversions.

I haven't studied the Other code in extreme detail, but I would imaging the performance issues have something to do with too many excessive calculations. One example is the fact that offsetX and offsetY is calculated in every loop. These variables just need to be calculated once before any of the looping begins, since they don't change. Additionally, sourcePixelRegion.ImageSize.Width/Height properties are being referenced numerous times. It may be more efficient just dropping these values into local variables before the looping beings.

However, the Other code isn't completely inefficient. I did notice that their pixel skipping technique is probably more efficient (haven't confirmed) just reading the code. They are literally skipping blocks of pixels and not even looping over them vs the QuickStart code which is actually looping over every single pixel before determining whether or not it should be skipped. Since the QuickStart is using the recommended ForEachRow method it would at least have to evaluate every row of pixels without skipping initially, but it could skip similarly when traversing the columns. It would also be worth determining if the ForEachRow method should actually be avoided all together with this technique.

When I break Pixelate into its own dedicated article, I'll explore the optimization possibilities and update the techniques in this article if necessary.

Rob.Kachmar (talk) 16:53, 7 January 2014 (EET)

Hamishwillee - Awesome, thank you!

That sounds great. We will then link both articles to your dedicated pixelator :-)

hamishwillee (talk) 03:25, 8 January 2014 (EET)

Hamishwillee - Interpolation filter

FYI: There are some other quite interesting filters here: Implementations of Custom Filter for the Nokia Imaging SDK. In particular the Complex Custom Filter is basically a pixelator, but it uses the top left pixel position to get the colour of each box rather than having to do calculation of center point etc. Without looking too closely at it, I'd assume that would be a little easier calculation for a virtually identical effect.

I think maybe we have enough custom filters to define a new sub category "Nokia Imaging SDK Custom Filters". Thoughts on that?



hamishwillee (talk) 06:07, 8 January 2014 (EET)

SB Dev -

At least the final code given in [Complex Custom Filter] is adding the offsets to get the pixel value of the center pixel. It's iterating over the corner pixels and then adds the offsets to the center to get the value from there.


// get the pixel color in the center of the soon to be pixelated area var pixel = sourcePixelRegion.ImagePixels[DimensionConverter.To1D(x + offsetX, y + offsetY, width)];

In the one case you're iterating over the center pixels and calculating the corner (once per area) before the fill, in the other case you're iterating over the corner pixels and calculating the center (once per area) to get at the desired color. So computationally I'd guess that both are pretty similar (it works without a multiplication though which might make a difference for big images). Calculating the center from the corner might be easier to read for some people ( x + offSetX vs. -1 * (m_ProcessEveryNthRow - 1)).

SB Dev (talk) 15:29, 8 January 2014 (EET)

Rob.Kachmar - RE: Interpolation filter

I definitely agree with the idea for the new sub category "Nokia Imaging SDK Custom Filters".

As for PixelInterpolation, I'm glad you found it. I knew I had seen another pixelating effect out there with interpolation in the name. My previous Wiki searches were not pulling up that article when searching with the key word interpolation, but now it is. (hmmmm) Anyway, I remember coming across it and thinking how very similar the technique is to the PixelateEffect article we were discussing earlier in the comments. I'm not sure which one showed up on the Wiki first, but PixelInterpolation is much more optimized.

As far as performance goes between the QuickStart Pixelate version above and PixelInterpolation, I did find that Pixelate is faster. They are equal with a pixel block of 5, but as soon as you go to a block of 15 or larger, Pixelate starts getting faster. Although, we are talking about 1 and 2 FPS differences, so not terribly significant.

I've updated my performance comparrison of FPS (Frames Per Second) between the effects using a Lumia 822. I also realized I was originally testing with an alternate version of my Pixelate effect where I was also manipulating brightness, ultimately working on a different effect. I corrected it to just be what I have in the article above, and it of course increased the speed.

   // Pixel Blocks    Pixelate    PixelInterpolation    PixelateEffect    PixelateEffect w/o To/FromColor()
   // ============    ========    ==================    ==============    =================================
   //   5             14 FPS      14 FPS                6 FPS              9 FPS
   //  15             16 FPS      15 FPS                7 FPS             10 FPS
   //  35             16 FPS      15 FPS                7 FPS             10 FPS
   // 128             17 FPS      15 FPS                7 FPS             11 FPS
   // 480             17 FPS      14 FPS                N/A               N/A
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 5); // 14 FPS
   //_customEffect = new PixelateEffect(_cameraPreviewImageSource, 5); // 6 FPS >>> 9 FPS
   //_customEffect = new PixelInterpolation(_cameraPreviewImageSource, 5); // 14 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 15); // 16 FPS
   //_customEffect = new PixelateEffect(_cameraPreviewImageSource, 15); // 7 FPS >>> 10 FPS
   //_customEffect = new PixelInterpolation(_cameraPreviewImageSource, 15); // 15 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 35); // 16 FPS
   //_customEffect = new PixelateEffect(_cameraPreviewImageSource, 35); // 7 FPS >>> 10 FPS
   //_customEffect = new PixelInterpolation(_cameraPreviewImageSource, 35); // 15 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 128); // 17 FPS
   //_customEffect = new PixelateEffect(_cameraPreviewImageSource, 128); // 7 FPS >>> 11 FPS
   //_customEffect = new PixelInterpolation(_cameraPreviewImageSource, 128); // 15 FPS
   //_customEffect = new Pixelate(_cameraPreviewImageSource, 480); // 17 FPS
   //_customEffect = new PixelateEffect(_cameraPreviewImageSource, 480); // 7 FPS auto reduced to 128 size
   //_customEffect = new PixelInterpolation(_cameraPreviewImageSource, 480); // 14 FPS

Rob.Kachmar (talk) 20:05, 10 January 2014 (EET)

Hamishwillee - Thanks Rob

Hi Rob

>I definitely agree with the idea for the new sub category "Nokia Imaging SDK Custom Filters". OK, we'll do that as soon as we have a few more in place (ie do as a group).

Thanks for the comparison of the different approaches - and especially for fixing up PixelateEffect to use bit shift rather than To/From so its a fair comparison. As I see it we probably should remove the PixelateEffect from wiki and link to the pixelate instead. PixelInterpolation is slighlty different approach. For that I'd have a note in the article pointing to PixelateEffect and stating that it is a more efficient algorithm with a similar effect. Seem reasonable?

As an aside, we've added a bug against the To/From methods and hopefully this will be fixed in the next update.



hamishwillee (talk) 06:36, 22 January 2014 (EET)

Rob.Kachmar -

Sounds like a good plan.

Rob.Kachmar (talk) 06:58, 22 January 2014 (EET)