Miangolarra

From jderobot

Jump to: navigation, search
  • Project Name: Visual memory 3D for mobile robot with stereo couple
  • Authors: Pablo Miangolarra Tejada (pablo.miangolarra [at] gmail [dot] es) and Jose María Cañas Plaza (jmplaza [at] gsyc [dot] es)
  • Academic Year: 2009-2010
  • Degree: Tech
  • Tags: Visual memory, Stereo vision, Pioneer robot, computer vision, 3D reconstruction.
  • Technology: C, C++, JDERobot suite, Glade2, GTK+, openCV, OpenGL.
  • State: Finished
  • Abstract:

My final year project it is focus in the detection of segments in a 3D real or simulated scenario, and the representation of this segments in a simulated environment, with OpenGL.

To extract the 3D segments from the image we use an attention module, appeared in Calvo[1] , that search for edges. Over this edges we apply a hough function for probabilistic segment search, from OpenCV library.

With a couple stereo cameras following a Pin hole model, we are able to obtain the position in three dimensions, of this Segments, relative to a coordinates system set on the base of the robot. Abella[2] started working with segments representation, I have improved the algorithm to obtain complex representations.

I developed a geometrical model for gazebo simulator that will allow to memorize the segments, at the same time the robot explore the simulated world.

But the most original part of this final year project is the powerful, memory of segments 3D, implemented with C++ list, it have the function to extract the position of doors and corridors, by the analysis of this segments.

  • Documentation: 
Master thesis: PDF
Presentation: PDF

Contents

Greatest hits

Visual memory will be the main schema of my final year project, it includes functionality for remote controlling a pioneer robot in a simulated world of Gazebo, the component will build a virtual map of the world.

The next video shows the final component executing in an office simulated environment:

[Short version]

[Extended version]

The next video shows the final component obtaining squares from the simulated world, that it will have to validate afterwards:

Visual memory

Visual memory will be the main schema of my final year project, it includes functionality for remote controlling a pioneer robot, and different artificial vision modes for 3D reconstruction.

Visual memory is an schema that receives the image from two stereo cameras, based in a pinhole model, this schema has five different modes:

  • Pica2, already explained below.
  • Pica1, the user clicks on the master image, the schema calculate the epipolar line, in the slave image and finds the homolog point, aftewards calculate the 3D point in scenario.
  • Obtain points, the schema calculates saliency maps in both images, focused in the edges, afterwards calculates the 3D points.
  • Obtain segments, the schema calculates saliency maps in both images, and apply a hough filter to master image to obtain straight segments in the image, then it check the segments in the slave image, afterwards it obtain 3D segments, finally it is capable to represent them in OpenGL.
  • Cognitive mode, the schema looks for possible squares in the image, and generate hipothesis that is capable to validate afterwards.

This schema, could work with both Gazebo simulator and real images.

[17/12/2009]:

I implement some new features to improve the schema, I've been working in the final thesis where I will explain my project. Some of the new features are:

  • Improved GUI.
  • Validation of hypothesis door by color RGB filter over the image.
  • Better intersection between corridors, allows walls representation as well.
  • New representation of corridors.

The next video shows most of this improvements, you will be able to see the confirmation square where the door look for a door.


[02/12/2009]:

I implemented a cache memory in order to reduce merging time of segments. I added some view of the gazebo map from which I reconstruct the area. The following video shows how the schema reconstructs the whole map, with minimal errors:

[01/12/2009]:

The following video shows how the schema reconstructs the whole map, with minimal errors, applying hypothesis such as two corridor that they come across or objects like big and small doors:

[26/11/2009]:

The following video shows how the schema generates cognitive hipothesis over something that looks like a square, we make the hipothesis of the hidden corner:


I have been working in the projection of the 3D memory over the image, to avoid calculating some of the segments already visited and memorize, in order to reduce the computing time. I am trying to solve the problem of those segments that I am unable to calculate, because I am seeing them in one camera. you will be able to see how the schema shows the projection of the memory over image. I implement a new way to find corridors, for those ones they just have doors in one side of the corridor:


[18/11/2009]:

The following video shows how the schema generates perceptive hypothesis over the extracted segments memorize on the 3D memory, and reconstruct the scene with doors and corridors:

[13/11/2009]:

The next video show how the schema works in gazebo simulator, the schema reconstructs a complex area using a marauder mode over the pioneer robot.

Applying some restriction as obtain only vertical segments, and reconstruct only the closer ones, we get better outcomes, we get an image every 3 seconds:

[12/11/2009]:

The next video show how the schema works in gazebo simulator, the schema reconstructs a complex area using a marauder mode over the pioneer robot.

The schema has a counter, an every 30 seconds try to reconstruct what it is seeing ahead:


The next images shows how it works the function "look at point", in which we indicate to the robot to look to a certain 3D point of the world,

this is an important step for Vergency movements and attention system implementation:




[11/11/2009]:

The next video show the reliability of the memory3D class, after eighteen executions of the algorithm, it merge correctly practically every new segment with the memory:

[10/11/2009]:

In the next image shows how the fusion segment algorithm compare the memorize segments with the new one calculated:


[05/11/2009]:

The next videos show how I have improved the processing time of my schema after solve a bug in the code, I made some other improvement to obtain better 3D segments from the images :



[04/11/2009]:

I add the following functionality to my schema: 

- Memory of 3D segments.
- Allow to move the robot trough the world.
- The geometrical model for changing the reference system is implement with GSL matrix.
- OpenGL represents important objects using global coordinates.

The next video shows all this new improvements:


[29/10/2009]:

We were working in a new technique to obtain the segments, first we obtain small segments, and afterwards we make them grow until they have got the desired size:


[22/10/2009]:

The next image show how the schema finds the segments in the slave image, from the master image segments, as you can see there are some trouble in this step, in which a will have to work more:



The next video shows how the program reconstruct the scene with OpenGL:



[21/10/2009]:

The next video show how the schema build a simple solid, using segments as building primitive, with OpenGL:

[15/10/2009]:

The next videos show how the schema build a simple solid with OpenGL:


[09/10/2009]:

This video show the schema working with real cameras:


This video shows how pica1 mode works with Gazebo simulator images, the 3D point is in the position X: 10250.0000 Y: -5750.0000 Z: 0.0 , in the simulator, our schema give us X. Y: Z:.

[09/10/2009]:

This pictures shows how pica2 mode works with real images. 

- First 3D point is in the position  X: 10800.0 Y: 0.0 Z: 0.0 , in real world, our schema give us X: 10914.777344 Y: -3.388302  Z: 1279.617432. 
- Second 3D point is in the position X: 15800.0 Y: 0.0 Z: 0.0 , in real world, our schema give us X: 13870.358398 Y: -41.809296 Z: 2045.276855.

Further improvements will come ahead, focused on the accuracy of the system.



Geometrical Model

In order to memorize segment in 3D world we need a robust geometrical model, that allow us to change the reference system from the world to the robot, and from the robot to the cameras, having in mind the robot orientation, and the values of pan tilt encoders. We want to build this model using RT matrix, using GSL library.


[29/10/2009]:

The next video show the geometrical model, working properly when the cameras they move, as well as when the robot moves around the scenario:


[27/10/2009]:

I need to have a very fine calibration, in order to implement this model. I modify the schema to allow me to modify camera extrinsics during execution time.

This picture show how the schema projects the cube vertex with the original configuration:


This picture show how the schema projects the cube vertex with the fine configuration parameters:


This picture show how the schema projects the cube vertex applying the geometrical model for camera movement:

[23/10/2009]:

Apart from scene representation with segments, this week I have been revising algebra, with the porpoise of apply RT matrix to change the coordinates reference system, from the camera to the robot, from the robot to the world and vice versa, as well as build a model for the camera movements in pan and tilt.

Based in the information given by the encoders of the robot and the camera pan tilts , of jderobot, we will be able to modify the extrinsic parameters of our camera and memorize 3D points in a global context.

Pica2

Pica2 is a schema that receives the image from two stereo cameras, based in a pinhole model, the user clicks on the same point in both images, the schema gets both pixels, and thanks to the progeo projective geometry library,it calculates the 3D point in both, the simulated and the real world.

[18/09/2009]:


The next pictures show the schema getting the position of three known 3D points in Gazebo.

The point calculated is shown by console.

You could check the accuracy of the system, comparing the points given with those calculated by the schema.

- 1st trial: X: 9750.0000 Y: -6250.0000 Z: 0000.0000 :

- 2nd trial: X: 10250.0000 Y: -5250.0000 Z: 0000.0000 :


- 3rd trial: X: 9750.0000 Y: -4250.0000 Z: 0000.0000:

[17/09/2009]:

I need to use Redo's calibrator as the objective, is to calibrate a couple of stereo cameras set on a pioneer robot:

[14/09/2009]:

This snapshot shows pica2 schema working with simulated3D driver, the point that we are looking for is X: 5000.000 Y: 1000.000 Z:0.0000

[07/09/2009]:

- 1st trial: X: 9250.0000 Y: -6250.0000 Z: 0000.0000 :


Eyeoperator

Eyeoperator works with Optical Flow, it will be useful to remote controlling a pioneer with a camera.

Eyeoperator is a 2007 final year project done by Jose Antonio Santos Cadenas.

This new version of eyeoperator is programmed with Glade2, GTK+ and OpenCV free library.

This application integrates two ways to use it, first one, draw four sensitive areas corresponding to Forward, Reverse, Spin Right and Spin Left movements. Moving our hand in front of the camera, and inside the sensitive areas, we are able to remote controlling our robot. The second one have just a single sensitive area, it calculates the average of the optical flow to determine which movement or combination of movements will do our robot.

[19/05/2009] Next video show eyeoperator schema, working over Gazebo simulator.

[12/05/2009] Finally the application is working in both modes. Next video show it, working over Gazebo simulator.

[08/05/2009] This video show how the application calculates the average optical flow and draw it. I am working in the reliability of the single area movement's recognition.


[07/05/2009]

Colotuner

colortuner gives you the possibility to choose between one of the next color spaces, and then apply a color filter: 

    -RGB
    -HSV
    -YUV

HSV and YUV interfaces completely functional:[25/5/2009]

colortuner with HSV and YUV interfaces:

Cat and mouse

Pioneer prowl with a bump and go algorithm in a Gazebo environment. Once the Pioneer find the other robot, the second robot will try to run away, but the first one will follow it wherever it goes.

Pioneer prowl with a bump and go algorithm in a Gazebo environment. Once the Pioneer find the other robot, using a color filter with simulated camera, our first Pioneer will approach to the second one.

Pioneer robot looking for a green area, after prowl with a bump and go behaviour:

Opencv practical

PracticaOpencv schema, with source video from a webcam. Updated from the last version with bigger displayed images and new color filter option: Get color from a pixel and apply the filter.

PracticaOpencv schema, with source video from a webcam:

Bump and go

This video shows the behaviour of a pioneer robot, in an environment simulated by stage/player, using a simple bump and go algorithm:



  • More information:
Retrieved from "http://jderobot.org/index.php/Miangolarra"
Personal tools