×
Namespaces

Variants
Actions
(Difference between revisions)

Creating a Lens application that uses HLSL effects for filters

From Nokia Developer Wiki
Jump to: navigation, search
r2d2rigo (Talk | contribs)
(R2d2rigo - Added "Introduction" and "Creating and drawing a DirectX texture" sections.)
r2d2rigo (Talk | contribs)
(R2d2rigo - Added the "Obtaining camera preview and updating the texture" section.)
Line 160: Line 160:
 
File:HLSLCamera_SecondScreen_Scaled.png|Texture scaled to show how it covers the background.
 
File:HLSLCamera_SecondScreen_Scaled.png|Texture scaled to show how it covers the background.
 
</gallery>
 
</gallery>
 +
 +
=== Obtaining camera preview and updating the texture ===
 +
 +
First of all, go open your '''WMAppManifest.xml''' file and add the capability '''ID_CAP_ISV_CAMERA''' and the requirement '''ID_REQ_REARCAMERA''' using the visual editor included in VS2012. This allows you to get the privileges to access the camera hardware and restricts your app to devices that have at least a rear camera, respectively.
 +
 +
Instead of using the old '''Microsoft.Devices.PhotoCamera''' class, we are going to take advantage of the functionality added in '''Windows.Phone.Media.Capture.PhotoCaptureDevice'''. One of the more crippling limitations of the old '''PhotoCamera''' was that you had to launch a separate thread and call '''GetPreviewBufferArgb32''' whenever you wanted to update your camera preview. With '''PhotoCaptureDevice''' we can subscribe to the '''PreviewFrameAvailable''' event and get notified automatically when such data is available.
 +
 +
Start by adding a new private member variable of type '''PhotoCaptureDevice''' to your '''MainLoop''' class. We are going to instantiate it in our '''Initialize''' function, but we will need to obtain first the camera's supported preview resolution for creating our texture with the appropiate size:
 +
 +
<code csharp>
 +
protected override async void Initialize()
 +
{
 +
    spriteBatch = new SpriteBatch(GraphicsDevice);
 +
 +
    Size previewSize = PhotoCaptureDevice.GetAvailablePreviewResolutions(CameraSensorLocation.Back)[0];
 +
    Size captureSize = PhotoCaptureDevice.GetAvailableCaptureResolutions(CameraSensorLocation.Back)[0];
 +
 +
    CreateTexture((int)previewSize.Width, (int)previewSize.Height);
 +
    photoDevice = await PhotoCaptureDevice.OpenAsync(CameraSensorLocation.Back, captureSize);
 +
    photoDevice.PreviewFrameAvailable += photoDevice_PreviewFrameAvailable;
 +
 +
    base.Initialize();
 +
}
 +
</code>
 +
 +
The function has been marked as {{Icode|async}} so we can {{Icode|await}} the call that initializes the camera. We obtain the preview size by calling {{Icode|PhotoCaptureDevice.GetAvailablePreviewResolutions}} with {{Icode|CameraSensorLocation.Back}} as its parameter to query the hardware to return all supported preview resolutions by that camera. We index the first element of the array to get the smallest preview size. Note that the call to {{Icode|CreateTexture}} has been appropriately changed to use the new size instead of the old, hardcoded parameters. And at last, {{Icode|PhotoCaptureDevice.OpenAsync}} asynchronously gets the camera device with the properties we want (back facing and specified capture resolution). To make sure we get an updated buffer of the camera's viewpoint, we subscribe to {{Icode|PreviewFrameAvailable}} the following function:
 +
 +
<code csharp>
 +
void photoDevice_PreviewFrameAvailable(ICameraCaptureDevice sender, object args)
 +
{
 +
    int[] data = new int[previewTexture.Width * previewTexture.Height];
 +
    sender.GetPreviewBufferArgb(data);
 +
    previewTexture.SetData<int>(data);
 +
}
 +
</code>
 +
 +
This obtains the ARGB data into an array of {{Icode|int}} values and feeds it to the texture. Since the endianness for DirectX is switched, we don't have to perform any byte swizzling to make the individual colour components fit the texture's format. Run the application and you should see the camera's input drawn on the screen:
 +
 +
<gallery widths="400px" heights="240px">
 +
File:HLSLCamera_CameraPreview.png|Camera preview drawn as a DirectX texture.
 +
</gallery>
 +
 +
 +
  
  
=== Capturing camera input and updating the texture ===
 
  
  

Revision as of 02:58, 13 December 2012

This article covers how to create a Lens application that applies different filters to the photos. These filters are programmed in High Level Shading Language and are executed on the GPU to take advantage of the new DirectX functionality introduced in Windows Phone 8.

Contents

Introduction

Instagram, one of the most popular photo applications for iPhone (and later Android) gave the tradition of applying post-processing filters to pictures an artistic twist by simulating the effect of snapping the photos with an old Polaroid or Lomographic camera. These effects, although very computationally expensive, can be achieved with relative ease thanks to the new DirectX APIs available in Windows Phone 8 that allow us to execute the image processing in the GPU. In this tutorial, we will create an application that allows us to preview the camera input, snap a photo and apply an HLSL post-processing effect to it.

Creating the base project

Open a new instance of Visual Studio and create a project based on the Windows Phone App template. We will be using a standard C#/XAML application and modify it further to accustom our requirements.

Modifying the MainPage

Start by opening the MainPage.xaml file and deleting the Grid control named LayoutRoot and all its children elements. Now create a control of type DrawingSurfaceBackgroundGrid as the page's root and give it a name (we will be calling it DrawingSurface). This is required when your application is going to use full screen mode for advanced graphics rendering through DirectX, so you don't get any performance penalties derived from the XAML composition system. You can find more info about this control in the MSDN article Direct3D with XAML apps for Windows Phone 8.

Adding SharpDX references

Instead of using the default DirectX interoperatiblity though a separate C++ DLL, we are going to use the SharpDX library to make drawing calls from C# code. This library is a wrapper of the underlying DirectX functions that allow them to be used in any .NET language and currently supports Windows Desktop, Windows Metro and Windows Phone 8. Start by heading to the downloads section and get the package of your choice: "binary only" includes the libraries and "full package" has some sample code on how to perform common tasks with SharpDX.

Create a new directory inside your project's folder called Lib. Open it and create two child folders called x86 and ARM. Decompress the package you downloaded and from the Bin folder, copy the contents of Standard-wp8-x86 to your x86 folder and Standard-wp8-ARM to ARM. In your project, right click on References and select Add References..., and in the new window click Browse and navigate to the Lib\x86 folder to select the following assemblies:

  • SharpDX.dll
  • SharpDX.DXGI.dll
  • SharpDX.Direct3D11.dll
  • SharpDX.Toolkit.dll
  • SharpDX.Toolkit.Game.dll
  • ShaprDX.Toolkit.Graphics.dll

Since the assemblies aren't AnyCPU, and currently we only have the x86 ones referenced, we must edit our project manually so the compiler references the correct ones. Close Visual Studio and open your CSPROJ file in your favourite text editor and look for the items named Reference, like this one:

<Reference Include="SharpDX">
<HintPath>Lib\x86\SharpDX.dll</HintPath>
</Reference>

You need to change the x86 part of the path to $(Platform), so it ends like this:

<Reference Include="SharpDX">
<HintPath>Lib\$(Platform)\SharpDX.dll</HintPath>
</Reference>

What we have written is a MSBuild property that gets replaced with the current platform name when building the project, so the correct version of the assemblies is used. Repeat this step for all existing references to SharpDX. When finished, save the changes and reopen the solution in Visual Studio.

Note.pngNote: If Visual Studio can't find the SharpDX references, go to the Build > Configuration Manager... menu and change Active solution platform to x86. Remember to change it again to ARM when deploying to a Windows Phone 8 device.

Application loop and basic drawing

Although we are creating a standard XAML navigation-based application, we are going to leverage some of the DirectX functionality to the framework provided by SharpDX.Toolkit. This is a collection of classes and utilities that mimics a subset of the XNA framework, and is provided as an extension to the core SharpDX libraries. If you have previous experience with XNA, you will find some of the code we are going to write very familiar.

Start by creating a new class and naming it MainLoop. Make it inherit from SharpDX.Toolkit.Game' and add a private field of type SharpDX.Toolkit.GraphicsDeviceManager. Now go to the constructor and initialize this field, passing this as the only parameter. At last, override the virtual function Draw and add the line GraphicsDevice.Clear(GraphicsDevice.BackBuffer, Color.Red); to its body. This should be the result:

public class MainLoop : Game
{
GraphicsDeviceManager deviceManager;
 
public MainLoop()
{
deviceManager = new GraphicsDeviceManager(this);
}
 
protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(GraphicsDevice.BackBuffer, Color.Red);
 
base.Draw(gameTime);
}
}

When created, the GraphicsDeviceManager will fetch the appropriate graphics adapter and initialize a valid graphics device that will allow the application to issue draw calls to the screen through the DirectX runtime. This, in turn, will allow us to tell the GraphicsDevice in the Draw function to draw whatever we want; for now, we will clear the entire screen to red. Finally, to make this loop run independently while the application is open, instantiate it in the MainPage and call its Run method with the DrawingSurfaceBackgroundGrid you created as its parameter:

public partial class MainPage : PhoneApplicationPage
{
MainLoop loop;
 
public MainPage()
{
InitializeComponent();
 
loop = new MainLoop();
loop.Run(this.DrawingSurface);
}
}

Run your application in the emulator or a device and it should display as follows:

Displaying the camera feed onscreen

Now that we have a working DirectX context, we are going to access the camera API to obtain its preview image and drawing it onscreen.

Creating and drawing a DirectX texture

Go to MainLoop.cs and add a public member variable of type SharpDX.Toolkit.Graphics.Texture2D (be careful not to use the one in the SharpDX.Direct3D11 namespace!), and a private one of type SharpDX.Toolkit.Graphics.SpriteBatch. The texture will be drawn onscreen every frame via the SpriteBatch and will hold the camera preview data in the future. Now, create an override for the function Initialize of MainLoop; this function gets called when the DirectX device and adapter have been successfully created, and will initialize a blank version of our texture and the much needed SpriteBatch.

protected override void Initialize()
{
CreateTexture(640, 480);
spriteBatch = new SpriteBatch(GraphicsDevice);
 
base.Initialize();
}

The function CreateTexture is just a shortcut for the creation and initialization of the texture, to make the code cleaner. Here is the code:

private void CreateTexture(int textureWidth, int textureHeight)
{
previewTexture = Texture2D.New(GraphicsDevice, textureWidth, textureHeight, PixelFormat.B8G8R8A8.UNorm);
 
Color[] data = new Color[textureWidth * textureHeight];
for (int i = 0; i < textureWidth * textureHeight; i++)
{
data[i] = Color.White;
}
 
previewTexture.SetData<Color>(data);
}

We just create it by calling Texture2D.New and passing the appropriate arguments. Be careful that the PixelFormat must be B8G8R8A8.UNorm since that's the order the camera will return the colour bytes in, and we will be saving an extra swizzling by declaring it this way. Lastly, the function will initialize an array of Color objects to White and feed it as the initial data to the texture.

Note.pngNote: As of version 2.4.1, SharpDX doesn't support backbuffer orientations other than portrait. We are going to manually rotate and scale the texture when drawing it so it appears in landscape mode.

Now we have to modify the Draw function so the SpriteBatch previously created draws the texture in fullscreen mode:

protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(GraphicsDevice.BackBuffer, Color.Red);
 
float backBufferXCenter = GraphicsDevice.BackBuffer.Width / 2;
float backBufferYCenter = GraphicsDevice.BackBuffer.Height / 2;
 
float textureXCenter = previewTexture.Width / 2;
float textureYCenter = previewTexture.Height / 2;
 
float yScale = (float)GraphicsDevice.BackBuffer.Width / (float)previewTexture.Height;
float xScale = (float)GraphicsDevice.BackBuffer.Height / (float)previewTexture.Width;
 
spriteBatch.Begin();
spriteBatch.Draw(previewTexture, new Vector2(backBufferXCenter, backBufferYCenter), null, Color.White, (float)Math.PI / 2.0f,
new Vector2(textureXCenter, textureYCenter), new Vector2(xScale, yScale), SpriteEffects.None, 0.0f);
spriteBatch.End();
 
base.Draw(gameTime);
}

We calculate the center of both the backbuffer and out texture, so we can properly align the drawing origin to the center of the screen. Next, we obtain the scale in both axis in which the texture must be multiplied so it fits in the entire screen without overflowing. And at last, we apply a rotation of Pi/2 radians (90 degrees) to give it the correct landscape orientation. When executed, the white texture should cover all the red background:

Obtaining camera preview and updating the texture

First of all, go open your WMAppManifest.xml file and add the capability ID_CAP_ISV_CAMERA and the requirement ID_REQ_REARCAMERA using the visual editor included in VS2012. This allows you to get the privileges to access the camera hardware and restricts your app to devices that have at least a rear camera, respectively.

Instead of using the old Microsoft.Devices.PhotoCamera class, we are going to take advantage of the functionality added in Windows.Phone.Media.Capture.PhotoCaptureDevice. One of the more crippling limitations of the old PhotoCamera was that you had to launch a separate thread and call GetPreviewBufferArgb32 whenever you wanted to update your camera preview. With PhotoCaptureDevice we can subscribe to the PreviewFrameAvailable event and get notified automatically when such data is available.

Start by adding a new private member variable of type PhotoCaptureDevice to your MainLoop class. We are going to instantiate it in our Initialize function, but we will need to obtain first the camera's supported preview resolution for creating our texture with the appropiate size:

protected override async void Initialize()
{
spriteBatch = new SpriteBatch(GraphicsDevice);
 
Size previewSize = PhotoCaptureDevice.GetAvailablePreviewResolutions(CameraSensorLocation.Back)[0];
Size captureSize = PhotoCaptureDevice.GetAvailableCaptureResolutions(CameraSensorLocation.Back)[0];
 
CreateTexture((int)previewSize.Width, (int)previewSize.Height);
photoDevice = await PhotoCaptureDevice.OpenAsync(CameraSensorLocation.Back, captureSize);
photoDevice.PreviewFrameAvailable += photoDevice_PreviewFrameAvailable;
 
base.Initialize();
}

The function has been marked as async so we can await the call that initializes the camera. We obtain the preview size by calling PhotoCaptureDevice.GetAvailablePreviewResolutions with CameraSensorLocation.Back as its parameter to query the hardware to return all supported preview resolutions by that camera. We index the first element of the array to get the smallest preview size. Note that the call to CreateTexture has been appropriately changed to use the new size instead of the old, hardcoded parameters. And at last, PhotoCaptureDevice.OpenAsync asynchronously gets the camera device with the properties we want (back facing and specified capture resolution). To make sure we get an updated buffer of the camera's viewpoint, we subscribe to PreviewFrameAvailable the following function:

void photoDevice_PreviewFrameAvailable(ICameraCaptureDevice sender, object args)
{
int[] data = new int[previewTexture.Width * previewTexture.Height];
sender.GetPreviewBufferArgb(data);
previewTexture.SetData<int>(data);
}

This obtains the ARGB data into an array of int values and feeds it to the texture. Since the endianness for DirectX is switched, we don't have to perform any byte swizzling to make the individual colour components fit the texture's format. Run the application and you should see the camera's input drawn on the screen:




Remove Category:Draft when the page is complete or near complete


WP Metro Icon Multimedia.png
WP Metro Icon UI.png
WP Metro Icon DirectX.png
WP Metro Icon WP8.png
Article Metadata
Compatibility
Platform(s):
Windows Phone 8
Article
Created: r2d2rigo (19 Nov 2012)
Last edited: r2d2rigo (13 Dec 2012)
461 page views in the last 30 days.

Was this page helpful?

Your feedback about this content is important. Let us know what you think.

 

Thank you!

We appreciate your feedback.

×