×

Discussion Board

Page 1 of 2 12 LastLast
Results 1 to 15 of 19
  1. #1
    Regular Contributor
    Join Date
    Jun 2008
    Posts
    59

    AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Hello,

    Just to make life easier to anyone who wants to do AR on N900 i have a few suggestions.

    1.) Use openCV library for any tracking and image manipulation. It is an amazing library that is available in the repo.

    M900:
    $ apt-get install libcv4 libcvaux4 libhighgui4

    Scratchbox:
    $ apt-get install libcv4 libcv-dev opencv-doc libcvaux-dev libcvaux4 python-opencv libhighgui-dev libhighgui4

    2.) For capturing video use openCV library calls or go through the hassle of GStreamer (appsink element) which can be integrated with QT using QGLWidget object. I attached source code of a project GstOpenGL that demonstrates how this can be done. Make sure you have qt 4.6 on your N900 or scratchbox. The GstOoenGL does not use openCV.

    The code has not been cleaned and optimized but should save you time.

    Cheers.
    Klen

  2. #2
    Regular Contributor
    Join Date
    Jun 2008
    Posts
    59

    Qt Camera Application using Gstreamer and OpenGL textures

    Hello,

    GstOpenGL app shows you how to capture a video stream frame by frame using Gstreamer and display it on the QGLWidget component.

    Make sure you have Qt 4.6 to get better performance. The app captures the video stream at 400×240 and in full screen runs at 20 FPS.

    Hope you find this useful.

    BTW: The code has not been cleaned and optimized, but should be useful.

    GstOpenGL_V1.tar

    Klen

  3. #3
    Registered User
    Join Date
    May 2010
    Posts
    7

    Thumbs up Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Hi Klen you really save my life! Great job man!

  4. #4
    Registered User
    Join Date
    May 2010
    Posts
    7

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Own update and correction, I finally found the issue and make the gles1 version working! and see that the original version is "live stream" on rotating box and also live background !

    The issue was that in gles1 you cannot glTexture2D a 400x240 texture, instead use 512x512 and the sub texture it will do the trick.

    For people who want to use gles1 version

    Code:
    const int TextureSizeW = 512;
    const int TextureSizeH = 512;
    int GLWidget::buffer_width=400;
    int GLWidget::buffer_height=240;
    
    void GLWidget::initializeGL ()
    {
        glClearColor(0.1f, 0.1f, 0.2f, 1.0f);
        glEnable(GL_TEXTURE_2D);
        glGenTextures(1, &m_uiTexture);
       // m_uiTexture = bindTexture(QImage(":/qt.png"));
        glBindTexture(GL_TEXTURE_2D, m_uiTexture);
        glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB, TextureSizeW,TextureSizeH, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL );
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
    
        m_fAngle = 0;
        m_fScale = 1;
        createGeometry();
        createBubbles(bubbleNum - bubbles.count());
    
    
    }
    
    void GLWidget::refresh_texture()
    {
    	if(CameraN900::scanning)
    	{
    		CameraN900::refresh_buffer(); //manually call appsink, not through the callback function
    
    		if(CameraN900::buffer!=NULL)
    		{
    			glBindTexture( GL_TEXTURE_2D, m_uiTexture );
    			unsigned char* data=(unsigned char *) GST_BUFFER_DATA (CameraN900::buffer);
    			//glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB, buffer_width,buffer_height, 0, GL_RGB, GL_UNSIGNED_BYTE, data );
    			glTexSubImage2D (GL_TEXTURE_2D, 0, 0, 0, buffer_width,buffer_height, GL_RGB, GL_UNSIGNED_BYTE, data);
    
    		}
    		gst_buffer_unref(CameraN900::buffer);
    	}
    }
    My working paintGL looks as


    Code:
    void GLWidget::paintGL()
    {
        createBubbles(bubbleNum - bubbles.count());
    
        QPainter painter;
        painter.begin(this);
    
        glClearColor(0.1f, 0.1f, 0.2f, 1.0f);
        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    
        glEnable(GL_TEXTURE_2D);
        refresh_texture();
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
    
          drawBackground(&painter);
    //////////////////////////////////////////////////////////////////////////////
    glMatrixMode(GL_PROJECTION);
        glPushMatrix();
        glLoadIdentity();
        glMatrixMode(GL_MODELVIEW);
        glPushMatrix();
        glMatrixMode(GL_TEXTURE);
        glPushMatrix();
    
        //Since OpenGL ES does not support glPush/PopAttrib(GL_ALL_ATTRIB_BITS) 
        //we have to take care of the states ourselves
    
        glEnable(GL_TEXTURE_2D);
        q_glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
        q_glTexParameter(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
        glEnable(GL_LIGHTING);
        glEnable(GL_LIGHT0);
    
        glShadeModel(GL_FLAT);
        glFrontFace(GL_CW);
        glCullFace(GL_FRONT);
        glEnable(GL_CULL_FACE);
        glEnable(GL_DEPTH_TEST);
    
        glMatrixMode(GL_MODELVIEW);
        glLoadIdentity();
    
        q_glRotate(f2vt(m_fAngle), f2vt(0.0), f2vt(1.0), f2vt(0.0));
        q_glRotate(f2vt(m_fAngle), f2vt(1.0), f2vt(0.0), f2vt(0.0));
        q_glRotate(f2vt(m_fAngle), f2vt(0.0), f2vt(0.0), f2vt(1.0));
        q_glScale(f2vt(m_fScale), f2vt(m_fScale),f2vt(m_fScale));
        q_glTranslate(f2vt(0),f2vt(-0.2),f2vt(0));
    
        q_vertexType matDiff[] = {f2vt(0.40), f2vt(1.0), f2vt(0.0), f2vt(1.0)};
        q_glMaterialv(GL_FRONT_AND_BACK, GL_DIFFUSE, matDiff);
    
        if (qtLogo)
            paintQtLogo();
        else
            paintTexturedCube();
        glMatrixMode(GL_MODELVIEW);
        glPopMatrix();
        glMatrixMode(GL_PROJECTION);
        glPopMatrix();
        glMatrixMode(GL_TEXTURE);
        glPopMatrix();
    
        glDisable(GL_LIGHTING);
        glDisable(GL_LIGHT0);
    
        glDisable(GL_DEPTH_TEST);
        glDisable(GL_CULL_FACE);
    	if (m_showBubbles)
    		foreach (Bubble *bubble, bubbles) {
    			bubble->drawBubble(&painter);
    	}
        QString framesPerSecond;
        framesPerSecond.setNum(frames /(time.elapsed() / 1000.0), 'f', 2);
       painter.setPen(Qt::white);
        painter.drawText(20, 40, framesPerSecond + " fps");
       painter.drawRect(0,0,799,479);
       painter.end();
        swapBuffers();
        QMutableListIterator<Bubble*> iter(bubbles);
    
        while (iter.hasNext()) {
            Bubble *bubble = iter.next();
            bubble->move(rect());
        }
        if (!(frames % 100)) {
            time.start();
            frames = 0;
        }
        m_fAngle += 1.0f;
        frames ++;
     }
    and finally the drawBackground function for gles1 using texture as background
    Code:
    void GLWidget::drawBackground(QPainter *painter)
    {
    glMatrixMode(GL_PROJECTION);
    glPushMatrix();
    glLoadIdentity();
    glOrthof(0, 1, 0, 1, -1.0, 1.0);
    glMatrixMode(GL_MODELVIEW);
    glPushMatrix();
    glLoadIdentity();
    glClear( GL_DEPTH_BUFFER_BIT);
    glDisable(GL_BLEND);
    glDisable(GL_ALPHA_TEST);
    glDisable(GL_DEPTH_TEST);
    glDisable(GL_CULL_FACE);
    
    
    
    		q_vertexType afVertices[] = {
    		    f2vt(0.0), f2vt(0.0), f2vt(1.0), f2vt(0.0),f2vt(0.0),f2vt(1.0),f2vt(1.0),f2vt(1.0)
    		};
    		glEnableClientState(GL_VERTEX_ARRAY);
    		glVertexPointer(2,q_vertexTypeEnum,0,afVertices);
    
    		GLfloat tw = GLfloat(buffer_width / TextureSizeW);
    		GLfloat th = GLfloat(buffer_height / TextureSizeH);
    
    		q_vertexType afTexCoord[] = {
    // remember to scale your tex coord according to 400/512  240/512 value
    		    f2vt(0.0f),f2vt(0.469f), f2vt(0.781f),f2vt(0.469f), f2vt(0.0f),f2vt(0.0f), f2vt(0.781f),f2vt(0.0f)
    		
    			//f2vt(0.0f),f2vt(th), f2vt(tw),f2vt(th), f2vt(0.0f),f2vt(0.0f), f2vt(tw),f2vt(0.0f)
    
    		};
    		glEnableClientState(GL_TEXTURE_COORD_ARRAY);
    		glTexCoordPointer(2,q_vertexTypeEnum,0,afTexCoord);
       glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    
    glDisableClientState(GL_TEXTURE_COORD_ARRAY);
    glDisableClientState(GL_VERTEX_ARRAY);
    
    
    glEnable(GL_BLEND);
    glEnable(GL_ALPHA_TEST);
    glEnable(GL_DEPTH_TEST);
    glEnable(GL_CULL_FACE);
    glMatrixMode(GL_MODELVIEW);
    glPopMatrix();
    glMatrixMode(GL_PROJECTION);
    glPopMatrix();
    
    }
    in my case it runs at around 15 fps, already happy about it! all credit goes to Klen!

    Enjoy!
    Previous Post under:=================


    One question for curiosity and confirmation of my understanding:

    I notice in your project you have a drawBackground() and is by default disabled, this is what I need in AR study, unfortunately when I enable it the drawing is correct but frame rate drop to 2 or 3 fps.

    when I tap "show video stream" in app, it update the texture of the rotating cube, but then keep rendering with that frame, it is no "live stream" on my cube, is it the expect result?

    I try to use
    glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB, buffer_width,buffer_height, 0, GL_RGB, GL_UNSIGNED_BYTE, data );

    as in refresh_texture and rend a quad without using QImage, it however does not work so far ( I use opengl es 1 version)


    Have you correct this in similar way, meaning have you get these two effect: "live stream" on your rotating cube or use "rendering background as texture instead of paint QImage"
    thanks for tip.

    ZR_YAO
    Last edited by zr_yao; 2010-06-03 at 22:10.

  5. #5
    Regular Contributor
    Join Date
    Jun 2008
    Posts
    59

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Quote Originally Posted by zr_yao View Post
    I notice in your project you have a drawBackground() and is by default disabled, this is what I need in AR study, unfortunately when I enable it the drawing is correct but frame rate drop to 2 or 3 fps.
    This was just a development step so that I could make sure that the images are correctly decoded into QImage structures in order to eliminate the problem of texturing. This function is very slow because it uses QPainter to paint each frame to the screen instead of using the OpenGL textures. However, when drawing less complicated stuff than full images (i.e. text, squares, small images), like the white border and frame rate text in paintGL() function, this does not affect the perforamnce.

    Quote Originally Posted by zr_yao View Post
    when I tap "show video stream" in app, it update the texture of the rotating cube, but then keep rendering with that frame, it is no "live stream" on my cube, is it the expect result?
    Yes. This is the right response. It was just part of my development. This app is just a snipet app not a finished application.

  6. #6
    Registered User
    Join Date
    Mar 2010
    Posts
    46

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Dear Klen:
    You are so kind to share your excellent work. Thanks!

    May I ask if your N900 already upgraded to PR1.2?
    Since I copy DistanceQT in the tar file to my N900 and also do 'apt-get install libcv4 libcvaux4 libhighgui4', the installation is fine.
    But when I execute DistanceQT in xrem, there is nothing happened.
    Nor application neither error message shows.
    I am wondering if I miss something on N900...

    Thanks anyway, you already save many time for us.

  7. #7
    Regular Contributor
    Join Date
    Jun 2008
    Posts
    59

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Quote Originally Posted by stone.chao View Post
    Dear Klen:
    You are so kind to share your excellent work. Thanks!

    May I ask if your N900 already upgraded to PR1.2?
    Since I copy DistanceQT in the tar file to my N900 and also do 'apt-get install libcv4 libcvaux4 libhighgui4', the installation is fine.
    But when I execute DistanceQT in xrem, there is nothing happened.
    Nor application neither error message shows.
    I am wondering if I miss something on N900...

    Thanks anyway, you already save many time for us.
    To run the app you need QT 4.6. You get it with the new frameware PR1.2. You can flash your device to get the latest frameware if your device does not have one. To check if you have the right QT version do:
    Code:
    Nokia-N900:~# dpkg -l|grep libqt
    ii  libqt4-core                                         4.6.2~git20100401-0maemo1+0m5              Qt 4 core module
    ii  libqt4-dbus                                         4.6.2~git20100401-0maemo1+0m5              Qt 4 D-Bus module
    ii  libqt4-gui                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 GUI module
    ii  libqt4-maemo5                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 Maemo5 module
    ii  libqt4-multimedia                                   4.6.2~git20100401-0maemo1+0m5              Qt 4 Multimedia Libraries
    ii  libqt4-network                                      4.6.2~git20100401-0maemo1+0m5              Qt 4 network module
    ii  libqt4-opengl                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 OpenGL module
    ii  libqt4-phonon                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 Phonon Libraries
    ii  libqt4-sql                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 SQL module
    ii  libqt4-sql-sqlite                                   4.6.2~git20100401-0maemo1+0m5              Qt 4 SQLite plugin
    ii  libqt4-svg                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 SVG module
    ii  libqt4-webkit                                       4.6.2~git20100401-0maemo1+0m5              Qt 4 WebKit module
    ii  libqt4-xml                                          4.6.2~git20100401-0maemo1+0m5              Qt 4 XML module
    ii  libqt4-xmlpatterns                                  4.6.2~git20100401-0maemo1+0m5              Qt 4 XML Patterns module
    Flashing N900:

    1.) get latest version for here
    http://tablets-dev.nokia.com/nokia_N900.php

    2.) Turn off your device and disconnect USB

    3.)
    run the flasher:
    $sudo flasher-3.5 -F <firmware-image> -f -R

    NOTES: More info
    http://wiki.maemo.org/Updating_the_tablet_firmware

  8. #8
    Nokia Developer Moderator
    Join Date
    Oct 2009
    Posts
    4,326

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Quote Originally Posted by Klen View Post
    To check if you have the right QT version do:
    Code:
    dpkg -l "libqt4-*"

  9. #9
    Registered User
    Join Date
    May 2010
    Posts
    7

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Quote Originally Posted by stone.chao View Post
    Since I copy DistanceQT in the tar file to my N900 and also do 'apt-get install libcv4 libcvaux4 libhighgui4', the installation is fine.
    But when I execute DistanceQT in xrem, there is nothing happened.
    .
    Just help to clarify a bit here. Unless Klen has updated the package, as I remember, the original DistanceQt package does not use OpenCV at all, it use Gstreamer for the camera frame capturing, Klen suggest to use OpenCV to "avoid Gstreamer hassle" which I fully agree with, to use opencv you can simple use cvcapture to get the camera frame, it in fact in turn use its own gstreamer library built in the highgui library, so Opencv help to wrap the gstreamer pipeline for you already.

    That says if you like to incooperate your opencv into the DistanceQt, you can strip away all the gstreamer pipeline codes. I would guess the performance should be the same as Klen's version (I have not done any test) but use your own pipeline gives a lot more flexibility.

    A simple way to verify Opencv way is to copy sample/c directory from Opencv to your ARMEL target and do a build_all, you can simply run i.e LKdemo, it works out of box.

  10. #10
    Registered User
    Join Date
    Jun 2010
    Posts
    36

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Hi,

    @zr_yao: Could you briefly elaborate on how you avoided a conflict with GLES2 regarding glTexImage2D() as described in http://talk.maemo.org/showthread.php?t=53328? How would the GLES2 version look like?

    I'm currently trying to plot simple points into the image. To achieve this I inserted
    Code:
    for(int i=0; i<ret_num_corners; i++){
       painter.drawLine(result[i].x,result[i].y,result[i].x,result[i].y);
    }
    into the painter part of GLWidget::paintGL(). I checked the array and the right points are in, but it doesn't show anything. An alternative version using
    Code:
    glBegin (GL_POINTS);
      for (i = 0; i < ret_num_corners; i ++){
         glVertex2f (result[i].x, result[i].y);
      }
    glEnd ();
    doesn't work since Qt can't find GL/gl.h, which it previously recommended by itself. Any ideas how to achieve this rather simple task?

    Again: How difficult is it to port zr_yao's work to GLES2? Any recommendations how to get started with GLES2?

    Thanks in advance.


    Best regards,
    Dorfmeister
    Last edited by Dorfmeister; 2010-06-14 at 00:17.

  11. #11
    Nokia Developer Moderator
    Join Date
    Oct 2009
    Posts
    4,326

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Quote Originally Posted by Dorfmeister View Post
    doesn't work since Qt can't find GL/gl.h, which it previously recommended by itself. Any ideas how to achieve this rather simple task?
    GL/gl.h is OpenGL header, while Maemo is using OpenGL ES2 and emulates OpenGL ES.
    Thus include should be probably GLES2/gl2.h
    Last edited by divanov; 2010-06-14 at 10:52. Reason: fixing a typo

  12. #12
    Nokia Developer Moderator
    Join Date
    Mar 2009
    Posts
    1,024

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Hi,
    you can start to read http://wiki.maemo.org/OpenGL-ES then you can can take a look at these examples: http://www.imgtec.com/powervr/inside...nGLES2xSGX.asp
    If you want a book you could take a look at "OpenGL ES 2.0 Programming Guide". The examples are downloadable from the author website but those doesn't compile in scratchbox.
    You can find a good book review at http://developers.slashdot.org/artic.../10/06/1313224

  13. #13
    Nokia Developer Moderator
    Join Date
    Mar 2009
    Posts
    1,024

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Right it's "GLES2/gl2.h"

  14. #14
    Registered User
    Join Date
    May 2010
    Posts
    7

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Again: How difficult is it to port zr_yao's work to GLES2? Any recommendations how to get started with GLES2?
    I'm a bit confused here, The package in post #1 and #2 use gles2, I add port of gles1 for convenience, why you say port to gles2? if you just mean to write the drawbackground() function in gles2, then it should be trival but I did not do it since again I use gles1.

    In my case my N900 by default use gles2, so I can run i.e. hellogl_es2, not hellogl_es1, by reinstall my own build of qtgl I revert it to gles1 and did my work there, I lose performance in fact, so I suggest you to use DistanceQt (the original package in this thread)directly from gles2. I did porting simply avoid the shader programming that I'm too lazy to learn.

    Hope that helps
    BR/ zr_yao
    Last edited by zr_yao; 2010-06-14 at 12:20.

  15. #15
    Registered User
    Join Date
    Jun 2010
    Posts
    36

    Re: AR applications on N900 (OpenCV, Gstreamer, QGLWidget)

    Oh, dear. I really got confused - thanks for clarifying the matter. I bought the book you recommended and will dive into it soon. However, I'm still confused that my conventional OpenGL approach to plot points didn't work out. If anybody should know right away what mistake I made in my attempt and how to draw points from an array properly (probably drawLine methods), a short hint would be appreciated. Otherwise I'll try to figure it out myself as soon as I find time.


    Best regards,
    Dorfmeister

Page 1 of 2 12 LastLast

Similar Threads

  1. how to list installed applications in N900?
    By trandinhduy in forum Nokia N9
    Replies: 3
    Last Post: 2010-05-10, 14:53
  2. Applications crashing in Cingular-branded 6682. Feedback appreciated.
    By Razor1973 in forum General Development Questions
    Replies: 3
    Last Post: 2006-01-27, 13:11

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •