×
Namespaces

Variants
Actions
(Difference between revisions)

Multi-Threaded Image Processing using Qt Camera

From Nokia Developer Wiki
Jump to: navigation, search
meh.at (Talk | contribs)
(Meh.at -)
meh.at (Talk | contribs)
m (Meh.at -)
Line 85: Line 85:
  
 
=== Defining a custom Qml camera view ===
 
=== Defining a custom Qml camera view ===
Next we define the "CustomCamera" class in C++ which communicates with the camera hardware using Qt Mobility camera and shows the live viewfinder image stream in Qml.  This class extends [http://doc.qt.nokia.com/4.7/qdeclarativeitem.html QDeclarativeItem] which is required for including the camera class as a Qml view and implements the interface ''FrameObserver'' to get notifications about new frames arriving from the camera viewfinder.
+
Next we define the ''CustomCamera'' class in C++ which communicates with the camera hardware using Qt Mobility camera and shows the live viewfinder image stream in Qml.  This class extends [http://doc.qt.nokia.com/4.7/qdeclarativeitem.html QDeclarativeItem] which is required for including the camera class as a Qml view and implements the interface ''FrameObserver'' to get notifications about new frames arriving from the camera viewfinder.
 
We also define some properties which can be later accessed from Qml:
 
We also define some properties which can be later accessed from Qml:
 
* cameraState: information about the camera state, for instance if the camera is loaded properly.
 
* cameraState: information about the camera state, for instance if the camera is loaded properly.
Line 107: Line 107:
 
</code>
 
</code>
  
The "updateFrame" method which receives viewfinder images is implemented from the "FrameObserver". If the worker thread is not busy then the frame is copied for later processing, else it is dropped.
+
The ''updateFrame'' method which receives viewfinder images is implemented from the ''FrameObserver''. If the worker thread is not busy then the frame is copied for later processing, else it is dropped.
 
<code cpp>
 
<code cpp>
 
bool CustomCamera::updateFrame(const QVideoFrame &frame)
 
bool CustomCamera::updateFrame(const QVideoFrame &frame)
Line 173: Line 173:
 
</code>
 
</code>
  
The capture destination is set to [http://doc.qt.nokia.com/qtmobility/qcameraimagecapture.html#CaptureDestination-enum QCameraImageCapture::CaptureToBuffer] resulting in an image buffer of the captured image (instead of automatically writing it to a file). This method is available since Qt Mobility 1.2. The captured image buffer is sent through the slot "imageAvailable()". When a full-resolution picture arrives it is copied to the worker thread.
+
The capture destination is set to [http://doc.qt.nokia.com/qtmobility/qcameraimagecapture.html#CaptureDestination-enum QCameraImageCapture::CaptureToBuffer] resulting in an image buffer of the captured image (instead of automatically writing it to a file). This method is available since Qt Mobility 1.2. The captured image buffer is sent through the slot ''imageAvailable()''. When a full-resolution picture arrives it is copied to the worker thread.
 
<code cpp>
 
<code cpp>
 
void CustomCamera::imageAvailable(int id, const QVideoFrame &frame)
 
void CustomCamera::imageAvailable(int id, const QVideoFrame &frame)
Line 184: Line 184:
 
}
 
}
 
</code>
 
</code>
The worker thread notifies the "CustomCamera" class when a viewfinder image is processed and tells the Qml view to repaint (update):
+
The worker thread notifies the ''CustomCamera'' class when a viewfinder image is processed and tells the Qml view to repaint (update):
 
<code cpp>
 
<code cpp>
 
void CustomCamera::processedFrameAvailable()
 
void CustomCamera::processedFrameAvailable()
Line 191: Line 191:
 
}
 
}
 
</code>
 
</code>
The "paint" method pulls the latest processed image from the worker thread and draws it on the center of the "CustomCamera" Qml view:
+
The ''paint'' method pulls the latest processed image from the worker thread and draws it on the center of the ''CustomCamera'' Qml view:
 
<code cpp>
 
<code cpp>
 
void CustomCamera::paint(QPainter *painter,
 
void CustomCamera::paint(QPainter *painter,
Line 223: Line 223:
 
}
 
}
 
</code>
 
</code>
Before we can use our "CustomCamera" class in Qml, it has to be registered somewhere before loading the Qml source code (e.g. in the application’s main method):
+
Before we can use our ''CustomCamera'' class in Qml, it has to be registered somewhere before loading the Qml source code (e.g. in the application’s main method):
 
<code cpp>
 
<code cpp>
 
void FIPMain::show()
 
void FIPMain::show()
Line 232: Line 232:
 
}
 
}
 
</code>
 
</code>
The CustomCamera can now be easily used in Qml:
+
The ''CustomCamera'' can now be easily used in Qml:
 
<code cpp>
 
<code cpp>
 
import CustomElements 1.0
 
import CustomElements 1.0
Line 249: Line 249:
  
 
== Spawning a worker thread for background image processing ==
 
== Spawning a worker thread for background image processing ==
 +
Image processing can be a resource intensive task and should be performed outside the main thread to keep the user interface responsive. First, we define our class FIPThread which is responsible for image processing:
 +
<code cpp>
 +
class FIPThread : public QThread
 +
{
 +
    Q_OBJECT
 +
public:
 +
    // Worker loop
 +
    void run();
 +
 +
    // Is an image currently processed?
 +
    inline bool isProcessing() const {
 +
        return m_stateProcessing;
 +
    }
 +
   
 +
Q_SIGNALS:
 +
    void newFrameReady();
 +
    void fullImageSaved(QString fn);
 +
private:
 +
    enum TMode {
 +
        EMode_Live,
 +
        EMode_Captured
 +
    };
 +
 +
    TMode m_currentMode;
 +
 +
    int m_frameIdx; // current buffer marked as ready
 +
    QImage m_frames[2]; // double buffer
 +
    QImage m_fullResFrame;
 +
 +
    bool m_stateProcessing;
 +
 +
    QMutex m_mutex;
 +
    QWaitCondition m_condition;
 +
 +
    bool m_abort;
 +
    bool m_restart;
 +
 +
    int m_effectThreshold;
 +
};
 +
</code>
 +
 +
''FIPThread'' emits two different signals:
 +
* ''newFrameReady'' is emitted when a viewfinder frame is ready.
 +
* ''fullImageSaved'' is emitted when the captured image has been processed and saved.
 +
 +
The following member variables are defined:
 +
* ''m_currentMode'': ''EMode_Live'' if working on a viewfinder image or ''EMode_Captured'' if working on a full resolution image.
 +
* ''m_frameIdx'', ''m_frames[2]'': two [http://doc.qt.nokia.com/4.7/qimage.html QImage] objects are used for double buffering. One buffer at position ''m_frameIdx'' holds the latest processed image, while the other buffer is used during processing. If ''m_frameIdx == -1'' then no processed image is available.
 +
* ''m_fullResFrame'': holds the full resolution captured image (not processed). The image is automatically freed after processing.
 +
* ''m_stateProcessing'': indicates whether the thread is currently processing an image.
 +
* ''m_effectThreshold'': the effect’s parameter value.
 +
 +
New viewfinder frames are added to the worker thread with the following method:
 +
<code cpp>
 +
void FIPThread::setNewFrame(QVideoFrame *ptrFrame)
 +
{
 +
    // Drop frame if last frame is still being processed or not in live mode
 +
    if (m_stateProcessing || m_currentMode != EMode_Live)
 +
        return;
 +
 +
    QMutexLocker locker(&m_mutex);
 +
 +
    // Select buffer which is not in use at the moment
 +
    if (m_frameIdx < 0) m_frameIdx = 0;
 +
    int bufferIdx = 1 - m_frameIdx;
 +
 +
    if (m_frames[bufferIdx].isNull() || m_frames[bufferIdx].width() != ptrFrame->width() ||
 +
        m_frames[bufferIdx].height() != ptrFrame->height()) {
 +
        m_frames[bufferIdx] = QImage(ptrFrame->width(), ptrFrame->height(), QImage::Format_ARGB32);
 +
    }
 +
 +
    // Copy data to local buffer
 +
    memcpy(m_frames[bufferIdx].bits(), ptrFrame->bits(), ptrFrame->mappedBytes());
 +
 +
    // Start processing
 +
    m_abort = false;
 +
    if (!isRunning()) {
 +
        start(LowPriority);
 +
    } else {
 +
        m_restart = true;
 +
        m_condition.wakeOne();
 +
    }
 +
}
 +
</code>
 +
The ''setNewFrame'' method copies the frame data to the locked double buffer, and starts or restarts the thread. The [http://doc.qt.nokia.com/4.7/qmutexlocker.html QMutexLocker] is used to automatically release the mutex lock when the method is left.
 +
For full-resolution captured images the following method is used which incorporates decoding of the frame data (from usually EXIF Jpeg) to QImage:
 +
<code cpp>
 +
void FIPThread::setFullResolutionFrame(QVideoFrame *ptrFrame)
 +
{
 +
    QMutexLocker locker(&m_mutex);
 +
 +
    // Decode and copy frame data to local buffer.
 +
    // "loadFromData()" consumes a lot of time. To improve performance, the raw data could be copied here
 +
    // and "loadFromData()" be called in "run()" method.
 +
    if (m_fullResFrame.loadFromData(ptrFrame->bits(), ptrFrame->mappedBytes()))
 +
    {
 +
        m_currentMode = EMode_Captured;
 +
 +
        // Start processing
 +
        m_abort = false;
 +
        if (!isRunning()) {
 +
            start(LowPriority);
 +
        } else {
 +
            m_restart = true;
 +
            m_condition.wakeOne();
 +
        }
 +
    }
 +
}
 +
</code>
 +
 +
The image processing is performed in the thread’s ''run'' method:
 +
<code cpp>
 +
void FIPThread::run()
 +
{
 +
    forever
 +
    {
 +
        int effectThreshold;
 +
        TMode currentMode;
 +
        BlackAndWhiteEffect effect;
 +
        int curIdx;
 +
        QImage *ptrImage;
 +
 +
        // We "freeze" the state by copying class variables to local variables.
 +
        m_mutex.lock();
 +
        m_stateProcessing = true;
 +
        effectThreshold = m_effectThreshold;
 +
        currentMode = m_currentMode;
 +
        m_mutex.unlock();
 +
 +
        // In live mode we use double buffering
 +
        if (currentMode == EMode_Live)
 +
        {
 +
            curIdx = 1 - m_frameIdx;
 +
            ptrImage = &m_frames[curIdx];
 +
        }
 +
        else
 +
        {
 +
            curIdx = m_frameIdx;
 +
            ptrImage = &m_fullResFrame;
 +
        }
 +
 +
        // Apply effect directly to the source image (overriding it).
 +
        effect.applyEffect(*ptrImage, *ptrImage, effectThreshold);
 +
 +
        if (currentMode == EMode_Captured)
 +
        {
 +
            // Save image
 +
            QString fn = QDesktopServices::storageLocation(QDesktopServices::PicturesLocation) +
 +
                    QDateTime::currentDateTime().toString("yyyy-MM-dd-hh-mm-ss.jpg");
 +
            if (ptrImage->save(fn))
 +
                emit fullImageSaved(fn);
 +
 +
            // Free memory of full-resolution buffer
 +
            m_fullResFrame = QImage();
 +
        }
 +
        else
 +
        {
 +
            // Signal that a new processed frame is available.
 +
            // There is no guarantee that *this* frame is available with "getLatestProcessedImage()".
 +
            // For this scenario the latest frame is sufficient.
 +
            emit newFrameReady();
 +
        }
 +
 +
        // Now we are ready for the next frame.
 +
        m_mutex.lock();
 +
        m_frameIdx = curIdx;
 +
        m_stateProcessing = false;
 +
 +
        if (m_abort)
 +
        {
 +
            m_mutex.unlock();
 +
            return;
 +
        }
 +
        if (!m_restart)
 +
        {
 +
            // Block the loop and wait for new data
 +
            m_condition.wait(&m_mutex);
 +
        }
 +
        m_restart = false;
 +
        m_mutex.unlock();
 +
    }
 +
}
 +
</code>
 +
 +
First we copy member variables which might change outside the run loop during processing to local variables. A mutex locks to prevent concurrent access to memory during copying. In live mode the buffers are swapped after processing while in capture mode the full resolution image is processed, saved to a file, and memory freed. Finally we check if the thread is about to exit (''m_abort==true'') or more work has to be done (''m_restart==true''). If both, ''m_abort'' and ''m_restart'', evaluate to false then we wait for more work.
 +
 +
To get the latest processed viewfinder image the following method is used:
 +
<code cpp>
 +
QImage * FIPThread::getLatestProcessedImage()
 +
{
 +
    m_mutex.lock();
 +
    if (m_frameIdx == -1 || m_frames[m_frameIdx].isNull())
 +
    {
 +
        m_mutex.unlock();
 +
        return NULL;
 +
    }
 +
    return &m_frames[m_frameIdx];
 +
}
 +
</code>
 +
The mutex is locked to prevent writing to the image buffer during reading. Thus, after reading it has to be released:
 +
<code cpp>
 +
void FIPThread::getLatestProcessedImageReady()
 +
{
 +
    m_mutex.unlock();
 +
}
 +
</code>
 +
 +
Before the worker thread can be released it has to stop processing of eventual remaining work:
 +
<code cpp>
 +
FIPThread::~FIPThread()
 +
{
 +
    // Wait for the worker thread to finish.
 +
    m_mutex.lock();
 +
    m_abort = true;
 +
    m_condition.wakeOne();
 +
    m_mutex.unlock();
 +
    wait();
 +
}
 +
<code>
 +
 +
== Adding a simple black and white effect ==
  
  

Revision as of 14:08, 10 May 2012

This article shows how to apply live image processing effects to the camera viewfinder using multi-threading and capture full resolution snapshots.

Article Metadata
Compatibility
Platform(s):
Symbian
Article
Created: (16 May 2012)
Last edited: meh.at (10 May 2012)

Contents

Introduction

Image processing is a resource intensive task. This article describes means how to apply image processing effects to the camera viewfinder using multi-threading to keep the user interface responsive. The following topics are covered in detail:

  • Using the Qt Mobility camera class in Qml.
  • Spawning a worker thread for background image processing.
  • Adding a simple black and white effect.
  • Discussion about hardware acceleration.
  • Conclusion.

The code provided here shows only the most important functional parts. The full code can be downloaded here (TODO).

Using the Qt Mobility Camera in Qml

The Qml camera component provides basic means to view and capture camera images directly from the Qml scripting language. For our purpose the Qml camera is not suitable because we need (i) live viewfinder image data stream and (ii) the final image as a data stream. In this article a stripped-down version of the custom Qml camera component from the Qt Camera Demo is used which uses the Qt Mobility Camera classes.

Project Preparation

First, the Qt Mobility dependency and Symbian capabilities have to be added to the project (*.pro) file:

symbian: {
TARGET.CAPABILITY += LocalServices \ # camera
ReadUserData \ #
WriteUserData \ # writing image file
UserEnvironment # camera
}

On Symbian, depending on the expected memory usage the heap- and stack sizes should be increased as well:

symbian: {
TARGET.EPOCSTACKSIZE = 0x14000
TARGET.EPOCHEAPSIZE = 0x20000 0x8000000
}

Receiving viewfinder frames from the camera

To receive video frames from the camera the QAbstractVideoSurface has to be implemented. The video surface has basically two functions: First, it tells the camera which image formats (for instance ARGB, UYVY, etc.) are supported by our application. Our sample application supports ARGB format only (caution: the Nokia N9 supports only UYVY format, thus either the effect processing has to be changed, or the UYVY data has to be converted to ARGB format before processing):

QList<QVideoFrame::PixelFormat> VideoSurface::supportedPixelFormats(
QAbstractVideoBuffer::HandleType handleType) const
{
Q_UNUSED(handleType);
 
return QList<QVideoFrame::PixelFormat>() << QVideoFrame::Format_ARGB32; //N9: Format_UYVY
}

Second it notifies our application over the FrameObserver interface when new image data is available:

class FrameObserver {
public:
virtual bool updateFrame(const QVideoFrame &frame) = 0;
};

Defining a custom Qml camera view

Next we define the CustomCamera class in C++ which communicates with the camera hardware using Qt Mobility camera and shows the live viewfinder image stream in Qml. This class extends QDeclarativeItem which is required for including the camera class as a Qml view and implements the interface FrameObserver to get notifications about new frames arriving from the camera viewfinder. We also define some properties which can be later accessed from Qml:

  • cameraState: information about the camera state, for instance if the camera is loaded properly.
  • availableDevices: a list of available cameras. These are usually the front- and back facing cameras.
  • effectThreshold: a parameter for our live image processing effect.
class CustomCamera : public QDeclarativeItem, public FrameObserver
{
Q_OBJECT
Q_ENUMS(State)
 
// State properties
Q_PROPERTY(State cameraState READ cameraState NOTIFY cameraStateChanged)
 
// Devices properties
Q_PROPERTY(QStringList availableDevices READ availableDevices)
 
// Effect properties
Q_PROPERTY(int effectThreshold READ effectThreshold WRITE effectThreshold)

The updateFrame method which receives viewfinder images is implemented from the FrameObserver. If the worker thread is not busy then the frame is copied for later processing, else it is dropped.

bool CustomCamera::updateFrame(const QVideoFrame &frame)
{
if (!frame.isValid()) {
return false;
}
 
if (m_fipThread->isProcessing()) {
// Discard frame if worker thread is busy.
return true;
}
 
QVideoFrame f = frame;
if (f.map(QAbstractVideoBuffer::ReadOnly)) {
m_fipThread->setNewFrame(&f); // send frame to worker thread
f.unmap(); // ready for next frame from camera
}
 
return true;
}

Next we define the start method to initialize and start the camera:

void CustomCamera::start(const QString &device)
{
destroyResources();
 
m_camera = new QCamera(device.toLatin1(), this);
 
// Make sure the camera is in loaded state.
m_camera->load();
 
m_videoSurface = new VideoSurface(this, m_camera);
m_camera->setViewfinder(m_videoSurface);
 
// Set the image capturing objects.
m_cameraImageCapture = new QCameraImageCapture(m_camera);
m_cameraImageCapture->setCaptureDestination(
QCameraImageCapture::CaptureToBuffer);
 
// Camera API
connect(m_camera, SIGNAL(locked()), this, SIGNAL(locked()));
connect(m_camera, SIGNAL(lockFailed()), this, SIGNAL(lockFailed()));
 
connect(m_camera, SIGNAL(stateChanged(QCamera::State)),
this, SLOT(cameraStateChanged(QCamera::State)));
connect(m_camera, SIGNAL(stateChanged(QCamera::State)),
this, SIGNAL(cameraStateChanged()));
 
// Image capture API
connect(m_cameraImageCapture, SIGNAL(imageCaptured(int, const QImage&)),
this, SIGNAL(imageCaptured(int, const QImage&)));
 
connect(m_cameraImageCapture, SIGNAL(imageAvailable(int, const QVideoFrame&)),
this, SLOT(imageAvailable(int, const QVideoFrame&)));
 
// Set the initial capture mode to image capturing.
m_camera->setCaptureMode(QCamera::CaptureStillImage);
 
// Begin the receiving of view finder frames.
m_camera->start();
}

The capture destination is set to QCameraImageCapture::CaptureToBuffer resulting in an image buffer of the captured image (instead of automatically writing it to a file). This method is available since Qt Mobility 1.2. The captured image buffer is sent through the slot imageAvailable(). When a full-resolution picture arrives it is copied to the worker thread.

void CustomCamera::imageAvailable(int id, const QVideoFrame &frame)
{
if (frame.map(QAbstractVideoBuffer::ReadOnly))
{
m_fipThread->setFullResolutionFrame(&frame);
frame.unmap();
}
}

The worker thread notifies the CustomCamera class when a viewfinder image is processed and tells the Qml view to repaint (update):

void CustomCamera::processedFrameAvailable()
{
update();
}

The paint method pulls the latest processed image from the worker thread and draws it on the center of the CustomCamera Qml view:

void CustomCamera::paint(QPainter *painter,
const QStyleOptionGraphicsItem *option,
QWidget *widget)
{
// Get processed image from worker thread and draw it.
QImage *ptrImage = m_fipThread->getLatestProcessedImage();
 
if (ptrImage)
{
QPointF upperLeft = boundingRect().center() -
QPointF(ptrImage->width() / 2,
ptrImage->height() / 2);
 
 
// Draw the black borders.
painter->fillRect(0, 0, upperLeft.x(), boundingRect().height(),
Qt::black);
painter->fillRect(upperLeft.x() + ptrImage->width(), 0,
boundingRect().right(), boundingRect().bottom(),
Qt::black);
 
painter->drawImage(QRect(upperLeft.x(), upperLeft.y(),
ptrImage->width(),
ptrImage->height()), *ptrImage);
 
// unlock
m_fipThread->getLatestProcessedImageReady();
}
}

Before we can use our CustomCamera class in Qml, it has to be registered somewhere before loading the Qml source code (e.g. in the application’s main method):

void FIPMain::show()
{
qmlRegisterType<CustomCamera>("CustomElements", 1, 0, "CustomCamera");
m_qmlView.setSource(QUrl("qrc:/qml/MainView.qml"));
m_qmlView.showFullScreen();
}

The CustomCamera can now be easily used in Qml:

import CustomElements 1.0
 
Page {
Component.onCompleted: {
camera.start();
}
 
CustomCamera {
id: camera
anchors.fill: parent
}
}

Spawning a worker thread for background image processing

Image processing can be a resource intensive task and should be performed outside the main thread to keep the user interface responsive. First, we define our class FIPThread which is responsible for image processing:

class FIPThread : public QThread
{
Q_OBJECT
public:
// Worker loop
void run();
 
// Is an image currently processed?
inline bool isProcessing() const {
return m_stateProcessing;
}
 
Q_SIGNALS:
void newFrameReady();
void fullImageSaved(QString fn);
private:
enum TMode {
EMode_Live,
EMode_Captured
};
 
TMode m_currentMode;
 
int m_frameIdx; // current buffer marked as ready
QImage m_frames[2]; // double buffer
QImage m_fullResFrame;
 
bool m_stateProcessing;
 
QMutex m_mutex;
QWaitCondition m_condition;
 
bool m_abort;
bool m_restart;
 
int m_effectThreshold;
};

FIPThread emits two different signals:

  • newFrameReady is emitted when a viewfinder frame is ready.
  • fullImageSaved is emitted when the captured image has been processed and saved.

The following member variables are defined:

  • m_currentMode: EMode_Live if working on a viewfinder image or EMode_Captured if working on a full resolution image.
  • m_frameIdx, m_frames[2]: two QImage objects are used for double buffering. One buffer at position m_frameIdx holds the latest processed image, while the other buffer is used during processing. If m_frameIdx == -1 then no processed image is available.
  • m_fullResFrame: holds the full resolution captured image (not processed). The image is automatically freed after processing.
  • m_stateProcessing: indicates whether the thread is currently processing an image.
  • m_effectThreshold: the effect’s parameter value.

New viewfinder frames are added to the worker thread with the following method:

void FIPThread::setNewFrame(QVideoFrame *ptrFrame)
{
// Drop frame if last frame is still being processed or not in live mode
if (m_stateProcessing || m_currentMode != EMode_Live)
return;
 
QMutexLocker locker(&m_mutex);
 
// Select buffer which is not in use at the moment
if (m_frameIdx < 0) m_frameIdx = 0;
int bufferIdx = 1 - m_frameIdx;
 
if (m_frames[bufferIdx].isNull() || m_frames[bufferIdx].width() != ptrFrame->width() ||
m_frames[bufferIdx].height() != ptrFrame->height()) {
m_frames[bufferIdx] = QImage(ptrFrame->width(), ptrFrame->height(), QImage::Format_ARGB32);
}
 
// Copy data to local buffer
memcpy(m_frames[bufferIdx].bits(), ptrFrame->bits(), ptrFrame->mappedBytes());
 
// Start processing
m_abort = false;
if (!isRunning()) {
start(LowPriority);
} else {
m_restart = true;
m_condition.wakeOne();
}
}

The setNewFrame method copies the frame data to the locked double buffer, and starts or restarts the thread. The QMutexLocker is used to automatically release the mutex lock when the method is left. For full-resolution captured images the following method is used which incorporates decoding of the frame data (from usually EXIF Jpeg) to QImage:

void FIPThread::setFullResolutionFrame(QVideoFrame *ptrFrame)
{
QMutexLocker locker(&m_mutex);
 
// Decode and copy frame data to local buffer.
// "loadFromData()" consumes a lot of time. To improve performance, the raw data could be copied here
// and "loadFromData()" be called in "run()" method.
if (m_fullResFrame.loadFromData(ptrFrame->bits(), ptrFrame->mappedBytes()))
{
m_currentMode = EMode_Captured;
 
// Start processing
m_abort = false;
if (!isRunning()) {
start(LowPriority);
} else {
m_restart = true;
m_condition.wakeOne();
}
}
}

The image processing is performed in the thread’s run method:

void FIPThread::run()
{
forever
{
int effectThreshold;
TMode currentMode;
BlackAndWhiteEffect effect;
int curIdx;
QImage *ptrImage;
 
// We "freeze" the state by copying class variables to local variables.
m_mutex.lock();
m_stateProcessing = true;
effectThreshold = m_effectThreshold;
currentMode = m_currentMode;
m_mutex.unlock();
 
// In live mode we use double buffering
if (currentMode == EMode_Live)
{
curIdx = 1 - m_frameIdx;
ptrImage = &m_frames[curIdx];
}
else
{
curIdx = m_frameIdx;
ptrImage = &m_fullResFrame;
}
 
// Apply effect directly to the source image (overriding it).
effect.applyEffect(*ptrImage, *ptrImage, effectThreshold);
 
if (currentMode == EMode_Captured)
{
// Save image
QString fn = QDesktopServices::storageLocation(QDesktopServices::PicturesLocation) +
QDateTime::currentDateTime().toString("yyyy-MM-dd-hh-mm-ss.jpg");
if (ptrImage->save(fn))
emit fullImageSaved(fn);
 
// Free memory of full-resolution buffer
m_fullResFrame = QImage();
}
else
{
// Signal that a new processed frame is available.
// There is no guarantee that *this* frame is available with "getLatestProcessedImage()".
// For this scenario the latest frame is sufficient.
emit newFrameReady();
}
 
// Now we are ready for the next frame.
m_mutex.lock();
m_frameIdx = curIdx;
m_stateProcessing = false;
 
if (m_abort)
{
m_mutex.unlock();
return;
}
if (!m_restart)
{
// Block the loop and wait for new data
m_condition.wait(&m_mutex);
}
m_restart = false;
m_mutex.unlock();
}
}

First we copy member variables which might change outside the run loop during processing to local variables. A mutex locks to prevent concurrent access to memory during copying. In live mode the buffers are swapped after processing while in capture mode the full resolution image is processed, saved to a file, and memory freed. Finally we check if the thread is about to exit (m_abort==true) or more work has to be done (m_restart==true). If both, m_abort and m_restart, evaluate to false then we wait for more work.

To get the latest processed viewfinder image the following method is used:

QImage * FIPThread::getLatestProcessedImage()
{
m_mutex.lock();
if (m_frameIdx == -1 || m_frames[m_frameIdx].isNull())
{
m_mutex.unlock();
return NULL;
}
return &m_frames[m_frameIdx];
}

The mutex is locked to prevent writing to the image buffer during reading. Thus, after reading it has to be released:

void FIPThread::getLatestProcessedImageReady()
{
m_mutex.unlock();
}

Before the worker thread can be released it has to stop processing of eventual remaining work:

FIPThread::~FIPThread()
{
// Wait for the worker thread to finish.
m_mutex.lock();
m_abort = true;
m_condition.wakeOne();
m_mutex.unlock();
wait();
}
<code>
 
== Adding a simple black and white effect ==
 
 
== Summary ==
 
 
 
''Add categories below. Remove Category:Draft when the page is complete or near complete''
397 page views in the last 30 days.
×