×
Namespaces

Variants
Actions
Revision as of 13:47, 10 May 2012 by meh.at (Talk | contribs)

Multi-Threaded Image Processing using Qt Camera

From Nokia Developer Wiki
Jump to: navigation, search

This article shows how to apply live image processing effects to the camera viewfinder using multi-threading and capture full resolution snapshots.

Article Metadata
Compatibility
Platform(s):
Symbian
Article
Created: (16 May 2012)
Last edited: meh.at (10 May 2012)

Contents

Introduction

Image processing is a resource intensive task. This article describes means how to apply image processing effects to the camera viewfinder using multi-threading to keep the user interface responsive. The following topics are covered in detail:

  • Using the Qt Mobility camera class in Qml.
  • Spawning a worker thread for background image processing.
  • Adding a simple black and white effect.
  • Discussion about hardware acceleration.
  • Conclusion.

The code provided here shows only the most important functional parts. The full code can be downloaded here (TODO).

Using the Qt Mobility Camera in Qml

The Qml camera component provides basic means to view and capture camera images directly from the Qml scripting language. For our purpose the Qml camera is not suitable because we need (i) live viewfinder image data stream and (ii) the final image as a data stream. In this article a stripped-down version of the custom Qml camera component from the Qt Camera Demo is used which uses the Qt Mobility Camera classes.

Project Preparation

First, the Qt Mobility dependency and Symbian capabilities have to be added to the project (*.pro) file:

symbian: {
TARGET.CAPABILITY += LocalServices \ # camera
ReadUserData \ #
WriteUserData \ # writing image file
UserEnvironment # camera
}

On Symbian, depending on the expected memory usage the heap- and stack sizes should be increased as well:

symbian: {
TARGET.EPOCSTACKSIZE = 0x14000
TARGET.EPOCHEAPSIZE = 0x20000 0x8000000
}

Receiving viewfinder frames from the camera

To receive video frames from the camera the QAbstractVideoSurface has to be implemented. The video surface has basically two functions: First, it tells the camera which image formats (for instance ARGB, UYVY, etc.) are supported by our application. Our sample application supports ARGB format only (caution: the Nokia N9 supports only UYVY format, thus either the effect processing has to be changed, or the UYVY data has to be converted to ARGB format before processing):

QList<QVideoFrame::PixelFormat> VideoSurface::supportedPixelFormats(
QAbstractVideoBuffer::HandleType handleType) const
{
Q_UNUSED(handleType);
 
return QList<QVideoFrame::PixelFormat>() << QVideoFrame::Format_ARGB32; //N9: Format_UYVY
}

Second it notifies our application over the FrameObserver interface when new image data is available:

class FrameObserver {
public:
virtual bool updateFrame(const QVideoFrame &frame) = 0;
};

Defining a custom Qml camera view

Next we define the "CustomCamera" class in C++ which communicates with the camera hardware using Qt Mobility camera and shows the live viewfinder image stream in Qml. This class extends QDeclarativeItem which is required for including the camera class as a Qml view and implements the interface FrameObserver to get notifications about new frames arriving from the camera viewfinder. We also define some properties which can be later accessed from Qml:

  • cameraState: information about the camera state, for instance if the camera is loaded properly.
  • availableDevices: a list of available cameras. These are usually the front- and back facing cameras.
  • effectThreshold: a parameter for our live image processing effect.
class CustomCamera : public QDeclarativeItem, public FrameObserver
{
Q_OBJECT
Q_ENUMS(State)
 
// State properties
Q_PROPERTY(State cameraState READ cameraState NOTIFY cameraStateChanged)
 
// Devices properties
Q_PROPERTY(QStringList availableDevices READ availableDevices)
 
// Effect properties
Q_PROPERTY(int effectThreshold READ effectThreshold WRITE effectThreshold)

The "updateFrame" method which receives viewfinder images is implemented from the "FrameObserver". If the worker thread is not busy then the frame is copied for later processing, else it is dropped.

bool CustomCamera::updateFrame(const QVideoFrame &frame)
{
if (!frame.isValid()) {
return false;
}
 
if (m_fipThread->isProcessing()) {
// Discard frame if worker thread is busy.
return true;
}
 
QVideoFrame f = frame;
if (f.map(QAbstractVideoBuffer::ReadOnly)) {
m_fipThread->setNewFrame(&f); // send frame to worker thread
f.unmap(); // ready for next frame from camera
}
 
return true;
}

Next we define the start method to initialize and start the camera:

void CustomCamera::start(const QString &device)
{
destroyResources();
 
m_camera = new QCamera(device.toLatin1(), this);
 
// Make sure the camera is in loaded state.
m_camera->load();
 
m_videoSurface = new VideoSurface(this, m_camera);
m_camera->setViewfinder(m_videoSurface);
 
// Set the image capturing objects.
m_cameraImageCapture = new QCameraImageCapture(m_camera);
m_cameraImageCapture->setCaptureDestination(
QCameraImageCapture::CaptureToBuffer);
 
// Camera API
connect(m_camera, SIGNAL(locked()), this, SIGNAL(locked()));
connect(m_camera, SIGNAL(lockFailed()), this, SIGNAL(lockFailed()));
 
connect(m_camera, SIGNAL(stateChanged(QCamera::State)),
this, SLOT(cameraStateChanged(QCamera::State)));
connect(m_camera, SIGNAL(stateChanged(QCamera::State)),
this, SIGNAL(cameraStateChanged()));
 
// Image capture API
connect(m_cameraImageCapture, SIGNAL(imageCaptured(int, const QImage&)),
this, SIGNAL(imageCaptured(int, const QImage&)));
 
connect(m_cameraImageCapture, SIGNAL(imageAvailable(int, const QVideoFrame&)),
this, SLOT(imageAvailable(int, const QVideoFrame&)));
 
// Set the initial capture mode to image capturing.
m_camera->setCaptureMode(QCamera::CaptureStillImage);
 
// Begin the receiving of view finder frames.
m_camera->start();
}

The capture destination is set to QCameraImageCapture::CaptureToBuffer resulting in an image buffer of the captured image (instead of automatically writing it to a file). This method is available since Qt Mobility 1.2. The captured image buffer is sent through the slot "imageAvailable()". When a full-resolution picture arrives it is copied to the worker thread.

void CustomCamera::imageAvailable(int id, const QVideoFrame &frame)
{
if (frame.map(QAbstractVideoBuffer::ReadOnly))
{
m_fipThread->setFullResolutionFrame(&frame);
frame.unmap();
}
}

The worker thread notifies the "CustomCamera" class when a viewfinder image is processed and tells the Qml view to repaint (update):

void CustomCamera::processedFrameAvailable()
{
update();
}

The "paint" method pulls the latest processed image from the worker thread and draws it on the center of the "CustomCamera" Qml view:

void CustomCamera::paint(QPainter *painter,
const QStyleOptionGraphicsItem *option,
QWidget *widget)
{
// Get processed image from worker thread and draw it.
QImage *ptrImage = m_fipThread->getLatestProcessedImage();
 
if (ptrImage)
{
QPointF upperLeft = boundingRect().center() -
QPointF(ptrImage->width() / 2,
ptrImage->height() / 2);
 
 
// Draw the black borders.
painter->fillRect(0, 0, upperLeft.x(), boundingRect().height(),
Qt::black);
painter->fillRect(upperLeft.x() + ptrImage->width(), 0,
boundingRect().right(), boundingRect().bottom(),
Qt::black);
 
painter->drawImage(QRect(upperLeft.x(), upperLeft.y(),
ptrImage->width(),
ptrImage->height()), *ptrImage);
 
// unlock
m_fipThread->getLatestProcessedImageReady();
}
}

Before we can use our "CustomCamera" class in Qml, it has to be registered somewhere before loading the Qml source code (e.g. in the application’s main method):

void FIPMain::show()
{
qmlRegisterType<CustomCamera>("CustomElements", 1, 0, "CustomCamera");
m_qmlView.setSource(QUrl("qrc:/qml/MainView.qml"));
m_qmlView.showFullScreen();
}

The CustomCamera can now be easily used in Qml:

import CustomElements 1.0
 
Page {
Component.onCompleted: {
camera.start();
}
 
CustomCamera {
id: camera
anchors.fill: parent
}
}

Spawning a worker thread for background image processing

Summary

Add categories below. Remove Category:Draft when the page is complete or near complete

369 page views in the last 30 days.
×