Image Sequencer for Windows Phone

Image Sequencer is an example application demonstrating the use of Nokia Imaging SDK’s Image Aligner and GIF Renderer APIs for creating cinemagraph-style animations in animated GIF format. The image sequences are captured using sequence capture functionality of the device camera. User can manipulate the animation by limiting the animated area to a small rectangular section, and by stabilizing the images in order to eliminate camera shake. Animations with still backgrounds and minor repeated movement are commonly called cinemagraphs.

Compatibility

  • Compatible with Windows Phone Store apps using XAML. A separate version is available for Windows Phone Silverlight 8.0.
  • Tested with Nokia Lumia 520, Nokia Lumia 1020, and Nokia Lumia 1520.
  • Developed with Visual Studio 2013 Express for Windows Phone.
  • Compiling the project requires Nokia Imaging SDK.

Design

Application launches into a view where user can select a preset sequence for editing, or start a camera session for capturing a new photo sequence. The view also lists the previously saved GIF animations. User can view a saved animation by tapping the thumbnail.

In preview page user can see a preview of the animation with image stabilization and framing applied in real-time. Four actions are available in the application bar:

Plays / pauses the animation.
Enables / disables the image stabilization.
Enables / disables the animation framing. When the framing is enabled, user can set the animated area by dragging.
Saves the animation to isolated storage as an animated GIF file.

Architecture overview

The application consists of six views.

  • Most of the application functionality is in SequencePreviewPage class. It displays the animation preview and also handles the image stabilization using the Nokia Imaging SDK ImageAligner. For exporting the previewed animation to GIF file it utilizes GifExporter, which in turn uses GifEncoder from Nokia Imaging SDK.
  • SequencesPage is the opening view of the application where example image sequences and saved GIF files are displayed.
  • GifPage displays a saved GIF animation.
  • AboutPage contains the application description and version info and is available in the menu of each view.
  • SequenceCapturePage and VideoPreviewPage both implement the sequence capture functionality. SequenceCapturePage uses the LowLagPhotoSequenceCapture of WinRT MediaCapture API, while VideoPreviewCapturePage is a fallback solution for devices where sequence capture is not supported by the camera hardware. It implements the sequence capture by capturing video preview images in short intervals.

Preview page

Initialization and image stabilization

The animation preview is implemented in SequencePreviewPage class. When the SequencePreviewPage OnNavigatedTo method gets called, it receives a list of image StorageFiles as a parameter. In case of example sequence, an identifier is received. The ID is simply a filename suffix for image resource files stored in the project Assets/Sequences directory. The filename syntax is sequence.[id].[sequence index].jpg. A list of StorageFileImageSources is then created out of the provided files so they can be used by Nokia Imaging SDK APIs.

The list of StorageFileImageSources gets passed to PrepareImageSequence method. PrepareImageSequence method prepares the image sequence playback by creating an aligned copy of the image sequence by using the Image Aligner API. The application can then display either unaligned or aligned bitmaps in real time simply by rendering images either from the list of unaligned image providers, or the list of aligned image providers.

public async Task PrepareImageSequence(List<IImageProvider> imageProviders)
{

    _unalignedImageProviders = imageProviders;
    _onScreenImageProviders = _unalignedImageProviders;
 
    ...
     
    using (ImageAligner imageAligner = new ImageAligner())
    {
        imageAligner.Sources = _unalignedImageProviders;
        imageAligner.ReferenceSource = _unalignedImageProviders[0];
        try
        {
            _alignedImageProviders = await imageAligner.AlignAsync();
            
            ...
        }
        catch (Exception e)
        {
            // If align fails, fail silently but don't enable the align button on UI
        }
    };
}

Animation

DispatcherTimer is used for timing the animation. The timer is created in the SequencePreviewPage constructor and is controlled in Play and Stop methods. When running, it calls AnimationTimer_Tick method on 100 millisecond intervals. AnimationTimer_Tick method calls Render method to render the current frame and advances the index of the displayed frame, wrapping it when necessary.

private void AnimationTimer_Tick(object sender, EventArgs eventArgs)
{
    Render(_animationIndex); 

    if (_animationIndex == (_onScreenImageProviders.Count() - 1))
    {
        _animationIndex = 0;
    }
    else
    {
        _animationIndex++;
    }
}

In Render method the IImageProvider is rendered to the on-screen bitmap.

private void Render(IReadOnlyList<IImageProvider> imageProviders, int animationIndex, bool renderBackground = false)
{
    if (_renderTask == null || _renderTask.IsCompleted)
    {
        _renderTask = DoRender(imageProviders, animationIndex, renderBackground);
    }
}

private async Task DoRender(IReadOnlyList<IImageProvider> imageProviders, int animationIndex, bool renderBackground = false)
{
    if (_onScreenImageProviders[animationIndex] != null)
    {
        int imageWidth = imageProviders == _unalignedImageProviders ? _unalignedImageWidth : _alignedImageWidth;
        int imageHeight = imageProviders == _unalignedImageProviders ? _unalignedImageHeight : _alignedImageHeight;
        if (_foregroundBitmap == null || _foregroundBitmap.PixelWidth != imageWidth || _foregroundBitmap.PixelHeight != imageHeight)
            _foregroundBitmap = new WriteableBitmap(imageWidth, imageHeight);
        using (WriteableBitmapRenderer writeableBitmapRenderer = new WriteableBitmapRenderer(imageProviders[animationIndex], _foregroundBitmap))
        {
            _foregroundBitmap = await writeableBitmapRenderer.RenderAsync();
        }
        if (renderBackground)
        {
            if (_backgroundBitmap == null || _backgroundBitmap.PixelWidth != imageWidth || _backgroundBitmap.PixelHeight != imageHeight)
                _backgroundBitmap = new WriteableBitmap(imageWidth, imageHeight);
            using (WriteableBitmapRenderer writeableBitmapRenderer = new WriteableBitmapRenderer(imageProviders[0], _backgroundBitmap))
            {
                _backgroundBitmap = await writeableBitmapRenderer.RenderAsync();
            }
        }
        await CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
            () =>
            {
                _foregroundBitmap.Invalidate();
                ImageElement.Source = _foregroundBitmap;
                if (renderBackground)
                {
                    _backgroundBitmap.Invalidate();
                    ImageElementBackground.Source = _backgroundBitmap;
                }
            });
    }
}

Setting the animation boundaries

The animated area selection works by having the animation in two layers: background and foreground. In the preview page this works by having two overlapping Image elements, ImageElementBackground and ImageElement. The ImageElementBackground is always displaying the first frame of the sequence. The ImageElement displays the current frame, cropped to the selected area by using the Clip property of Image element. During animation playback only the ImageElement is being refreshed.

The cropped area can be adjusted by user by dragging the canvas. The drag events are handled in ImageElement_ManipulationDelta method. The animated area is indicated by drawing a white rectangle on top of the image elements.

Exporting animated GIF files

When user taps the save icon, the previewed animation is exported as an animated GIF file. The GIF export happens in GifExporter class. It takes a list of IImageProviders as parameter, as well as an optional Rect representing the animated area.

The file is saved to pictures library. This requires the Pictures Library capability set in the package manifest. The saved files are available for viewing and sharing in Photos application, though the Photos image viewer does not play back the animation. The saved files are also accessible via the device file system. Note that we cannot use MediaLibrary API to store the GIF files to camera roll as it would convert the GIF files to still JPEG images.

public static async Task Export(IReadOnlyList<IImageProvider> images, Rect? animatedArea)
{
    // List of aligned images may contain Null items if some images couldn't be aligned 
    List<IImageProvider> sanitizedImages = new List<IImageProvider>();
    foreach (IImageProvider image in images) {
        if (image != null) {
            sanitizedImages.Add(image);
        }
    }

    ImageProviderInfo info = await sanitizedImages[0].GetInfoAsync();
    int w = (int)info.ImageSize.Width;
    int h = (int)info.ImageSize.Height;
    
    IReadOnlyList<IImageProvider> gifRendererSources;
    
    if (animatedArea.HasValue)
    {
        // Ensure the animated area dimensions are smaller than the image dimensions
        double rectW = animatedArea.Value.Width;
        double rectH = animatedArea.Value.Height;
        if ((animatedArea.Value.Width + animatedArea.Value.Left) >= w) 
        {
            rectW = w - animatedArea.Value.Left - 1;
        }
        if ((animatedArea.Value.Top + animatedArea.Value.Height) >= h)
        {
            rectH = h - animatedArea.Value.Top - 1;
        }
        Rect rect = new Rect(animatedArea.Value.Left, animatedArea.Value.Top, rectW, rectH);
        gifRendererSources = CreateFramedAnimation(sanitizedImages, rect, w, h);
    }
    else {
        gifRendererSources = sanitizedImages;
    }
    
    using (GifRenderer gifRenderer = new GifRenderer()) 
    {
        gifRenderer.Duration = 100;
        gifRenderer.NumberOfAnimationLoops = 10000;
        gifRenderer.Sources = gifRendererSources;
        var buffer = await gifRenderer.RenderAsync();
        var filename = "Sequence" + (await GetFileNameRunningNumber()) + ".gif";
        var storageFile = await KnownFolders.SavedPictures.CreateFileAsync(filename, CreationCollisionOption.ReplaceExisting);
        using (var stream = await storageFile.OpenAsync(FileAccessMode.ReadWrite))
        {
             await stream.WriteAsync(buffer);
        }
    }
 }

If the animated area is defined, GifExporter creates a new list of IImageProviders containing images where only the animated area is changing. This is implemented by using a CropFilter to crop the animated area, and blending it on top of the static background using BlendFilter.

private static IReadOnlyList<IImageProvider> CreateFramedAnimation(IReadOnlyList<IImageProvider> images, Rect animationBounds, int w, int h)
{
    List<IImageProvider> framedAnimation = new List<IImageProvider>();
    foreach (IImageProvider frame in images)
    {
        FilterEffect cropFilterEffect = new FilterEffect(frame);
        cropFilterEffect.Filters = new List<IFilter>() { new CropFilter(animationBounds) };
        FilterEffect blendFilterEffect = new FilterEffect(images[0]);
        BlendFilter blendFilter = new BlendFilter();
        blendFilter.ForegroundSource = cropFilterEffect;
        blendFilter.Level = 1.0;
        blendFilter.BlendFunction = BlendFunction.Normal;
        blendFilter.TargetArea = new Rect(
            animationBounds.Left / w,
            animationBounds.Top / h,
            animationBounds.Width / w,
            animationBounds.Height / h
        );
        blendFilterEffect.Filters = new List<IFilter>() { blendFilter };
        framedAnimation.Add(blendFilterEffect);
     }
     return framedAnimation;
 }

Capturing image sequences

Initialization

Sequence capture is implemented in SequenceCapturePage class. A MediaCapture object for controlling the phone back camera is initialized in InitializeMediaCapture method, called in the OnNavigatedTo method of the page.

...

_mediaCapture = new MediaCapture();
var devices = await DeviceInformation.FindAllAsync(DeviceClass.VideoCapture);
var backCamera = devices.FirstOrDefault(x => x.EnclosureLocation != null && x.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Back);

await _mediaCapture.InitializeAsync(new MediaCaptureInitializationSettings
{
    StreamingCaptureMode = StreamingCaptureMode.Video,
    PhotoCaptureSource = PhotoCaptureSource.Auto,
    AudioDeviceId = string.Empty,
    VideoDeviceId = backCamera.Id
});

...

The SequenceCapturePage XAML tree contains a CaptureElement that is used to display the camera viewfinder image. We just need to set the previously initialized MediaCapture object as the source of the CaptureElement, and start the preview:

...

captureElement.Source = _mediaCapture;
await _mediaCapture.StartPreviewAsync();

...

MediaCapture.PrepareLowLagPhotoSequenceCaptureAsync method is then used to initialize photo sequence capture. It returns a LowLagPhotoSequenceCapture object, which can be used to control the sequence capture.

...

var format = ImageEncodingProperties.CreateJpeg();
format.Width = 640;
format.Height = 480;

_variablePhotoSequenceCapture = await _mediaCapture.PrepareLowLagPhotoSequenceCaptureAsync(format);

...

A handler is added for the PhotoCaptured event that is triggered on each captured image. Whenever a shot gets taken, it gets saved to a temporary folder to be used later. In Image Sequencer the amount of captured images is limited to 20 due to memory usage concerns on low end devices.

...

_variablePhotoSequenceCapture.PhotoCaptured += OnPhotoCaptured;

...

public void OnPhotoCaptured(LowLagPhotoSequenceCapture s, PhotoCapturedEventArgs e)
{
     if (_fileIndex < AMOUNT_OF_FRAMES_IN_SEQUENCE)
     {
         if (_saveTask == null)
         {
             _saveTask = Save(e.Frame, _fileIndex++);
         }
         else
         {
             _saveTask = _saveTask.ContinueWith(t => Save(e.Frame, _fileIndex++));
         }
     }
     else
     {
         StopSequenceCapture();
     }
}

private async Task Save(IRandomAccessStream frame, int i) {
     var filename = "ImageSequencer." + i + ".jpg";
     var folder = Windows.Storage.ApplicationData.Current.TemporaryFolder;
     var storageFile = await folder.CreateFileAsync(filename, CreationCollisionOption.ReplaceExisting);
     var stream = await storageFile.OpenAsync(FileAccessMode.ReadWrite);
     await RandomAccessStream.CopyAndCloseAsync(frame, stream);
     _files.Add(storageFile);
}

The photo sequence capture can be started and stopped using the StartAsync and StopAsync methods of the LowLagPhotoSequenceCapture. Use FinishAsync to release the resources used by the photo sequence operation. These methods are used in the event handlers of hardware camera button and the on-screen capture button.

Using video preview images for capturing image sequences

Not all devices support photo sequence capture. You can use the MediaCapture.VideoDeviceController.LowLagPhotoSequence.Supported property to check if sequence capture is supported by the device running your application. For unsupported devices you can implement a fall-back by using video preview to capture sequences of images. This can be done easily using the CameraPreviewImageSource of Nokia Imaging SDK.

The CameraPreviewImageSource is initialized similarly to the LowLagPhotoSequenceCapture:

public async void InitializeAsync()
{
    // Create a camera preview image source (from Imaging SDK)
    _cameraPreviewImageSource = new CameraPreviewImageSource();
    await _cameraPreviewImageSource.InitializeAsync(string.Empty);
    var properties = await _cameraPreviewImageSource.StartPreviewAsync();
 
    // Create a preview bitmap with the correct aspect ratio
    var width = 640.0;
    var height = (width / properties.Width) * properties.Height;
    _writeableBitmap = new WriteableBitmap((int)width, (int)height);
 
    captureElement.Source = _writeableBitmap;
 
    _writeableBitmapRenderer = new WriteableBitmapRenderer();
    _jpegRenderer = new JpegRenderer();
 
    // Attach preview frame delegate
    _cameraPreviewImageSource.PreviewFrameAvailable += OnPreviewFrameAvailable;
}

Instead of using CaptureElement for displaying the viewfinder, a regular Image control is used for displaying the images provided by the CameraPreviewImageSource. If sequence capture is ongoing, the image is saved to JPG file using Nokia Imaging SDK JpegRenderer.

private void OnPreviewFrameAvailable(IImageSize args)
{
    _renderTask = Render();
}

private async Task Render()
{

    if (!_rendering && !_stop)
    {
        _rendering = true;
        
        // Render camera preview frame to screen
        _writeableBitmapRenderer.Source = _cameraPreviewImageSource;
        _writeableBitmapRenderer.WriteableBitmap = _writeableBitmap;
        await _writeableBitmapRenderer.RenderAsync();
        await Windows.ApplicationModel.Core.CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(
            CoreDispatcherPriority.High, () =>
            {
                _writeableBitmap.Invalidate();
            });
        
        // Write camera preview frame to a file if capturing
        if (_capturing)
        {
            if (_sequenceIndex < 20)
            {
                _jpegRenderer.Source = _cameraPreviewImageSource;
                IBuffer jpg = await _jpegRenderer.RenderAsync();
                await Save(jpg, _sequenceIndex++);
            }
            else
            {
                StartStopCapture();
            }
        }
        _rendering = false;
    }
 
    if (_stop)
    {
        _capturing = false;
        _cameraPreviewImageSource.Dispose();
        _writeableBitmapRenderer.Dispose();
        _jpegRenderer.Dispose();
    }
}

Viewing animated GIF files

No standard XAML control exists for displaying animated GIFs, so we need to decode the GIF frames manually by using BitmapDecoder. This is done in GifPage class.

BitmapDecoder provides us access to the animation frames in a GIF file. The GIFs generated by Nokia Imaging SDK uses partial GIF encoding, meaning that the animation frames contain only pixels different from the corresponding pixels in the first frame. Hence, in addition to decoding the frames, we need to overlay each frame on top of the first frame to complete the image. This is done by examining the RGBA values of the pixels. If the value of the alpha channel is zero, we replace the pixel using the corresponding pixel in the first frame.

Note that the overlay procedure is not needed for animated GIFs not using partial encoding.

private async Task LoadImage(String filename)
{      
    var storageFile = await KnownFolders.SavedPictures.GetFileAsync(filename);
    using (var res = await storageFile.OpenAsync(FileAccessMode.Read))
    {
        var bitmapDecoder = await BitmapDecoder.CreateAsync(BitmapDecoder.GifDecoderId, res);
        byte[] firstFrame = null;
        for (uint frameIndex = 0; frameIndex < bitmapDecoder.FrameCount; frameIndex++)
        {
            var frame = await bitmapDecoder.GetFrameAsync(frameIndex);
            var writeableBitmap = new WriteableBitmap((int)bitmapDecoder.OrientedPixelWidth, (int)bitmapDecoder.OrientedPixelHeight);
            var pixelData = await frame.GetPixelDataAsync(
                BitmapPixelFormat.Bgra8,
                BitmapAlphaMode.Straight,
                new BitmapTransform(),
                ExifOrientationMode.RespectExifOrientation,
                ColorManagementMode.DoNotColorManage);
            var bytes = pixelData.DetachPixelData();
            if (firstFrame == null)
            {
                firstFrame = bytes;
            }
            else
            {
                // Blend the frame on top of the first frame
                for (uint i = 0; i < bytes.Count(); i += 4)
                {
                    int alpha = bytes[i + 3];
                    if (alpha == 0 && firstFrame != null)
                    {
                        Array.Copy(firstFrame, i, bytes, i, 4);
                    }
                }
            }
            using (var stream = writeableBitmap.PixelBuffer.AsStream())
            {
                stream.Write(bytes, 0, bytes.Length);
            }
            _frames.Add(writeableBitmap);
        }
    }
    StartAnimation();
}

Downloads

Image Sequencer project v2.0 image-sequencer-2.0.zip

This example application is hosted in GitHub, where you can check the latest activities, report issues, browse source, ask questions, or even contribute to the project yourself.


Last updated 23 June 2014

Back to top

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×