Frivas-phd

From jderobot
Jump to: navigation, search

Title: Human body tracking with RGB-D sensors.

People[edit]

  • Francisco Miguel Rivas Montero (franciscomiguel [dot] rivas [at] urjc [dot] es)
  • José María Cañas Plaza (jmplaza [at] gsyc [dot] es)

Development[edit]

Person detection[edit]

Darknet with YOLO configuration[edit]

The aim of this section is to describe the results obtained by applying several person detection techniques.

Dataset generation[edit]

V0[edit]

Dataset status
Class Number of samples
Person 1707
Person falling 171
Person fall 93
Total 1971

YOLO trained weights[edit]

  • YOLO original weights
  • YOLO trained with VOC dataset
Class Train Samples Test Samples Train images Test images
All classes 19910 12032 8218 4952
  • YOLO trained with VOC dataset using only person class
Class Train Samples Test Samples Train images Test images
Person 6552 4528 8218 4952
  • YOLO trained fusing VOC and our dataset using only person class (V0)
Class Train Samples Test Samples Train images Test images
Person 5938 1993 7133 2464


This dataset will be replaced with the following one:

Class Train Samples Test Samples Train images Test images
From VOC 6552 4528 8218 4952
From Our dataset 1183 787 1183 788
TOTAL 7735 5315 9401 5740



Testing YOLO with our dataset[edit]

Yolo with original weights Trained with VOC dataset Trained with VOC using only person class
True Positives 1562 1581 1507
False Positives 1041 2037 595
False Negatives 146 127 200
Mean IOU 0.770311 0.758627 0.773138
Precision 0.600077 0.436982 0.716936
Recall 0.91452 0.925644 0.882835


Testing YOLO with VOC dataset (testing sample only using person class)[edit]

Yolo with original weights Trained with VOC dataset Trained with VOC using only person class
True Positives 3422 3452 3379
False Positives 1329 1411 1209
False Negatives 1085 1046 1102
Mean IOU 0.757694 0.755414 0.759976
Precision 0.720269 0.70985 0.736486
Recall 0.759263 0.767452 0.754073

Testing YOLO with a fused dataset[edit]

This dataset has been generated fusing dataset and our V0 dataset, generating a new train and test subset with the combined samples. The following evaluations have been performed over the new test sample, so take care while reviewing this results. Some of the samples in the new fused dataset may appear in the old train dataset and vice-versa. This values are only to verify the performance with the new dataset using the previous weights.


Yolo with original weights Trained with VOC dataset Trained with VOC using only person class Trained with VOC and our V0 dataset
VOC True Positives 3422 3452 3379 3311
False Positives 1329 1411 1209 1440
False Negatives 1085 1046 1102 1176
Mean IOU 0.757694 0.755414 0.759976 0.73257
Precision 0.720269 0.70985 0.736486 0.696906
Recall 0.759263 0.767452 0.754073 0.73791
Yolo with original weights Trained with VOC dataset Trained with VOC using only person class Trained with VOC and our V0 dataset
OURS V0 True Positives 670 661 624 788
False Positives 429 846 237 0
False Negatives 124 128 164 0
Mean IOU 0.72197 0.7047 0.729763 0.790136
Precision 0.609645 0.43862 0.724739 1
Recall 0.843829 0.837769 0.791878 1
Yolo with original weights Trained with VOC dataset Trained with VOC using only person class Trained with VOC and our V0 dataset
FUSED True Positives 4021 4001 3903 4099
False Positives 1949 2489 1640 1440
False Negatives 1277 1306 1377 1176
Mean IOU 0.728419 0.722893 0.729226 0.743636
Precision 0.673534 0.616487 0.704131 0.740025
Recall 0.758966 0.75391 0.739205 0.777062


Tiny yolo configuration[edit]

Yolo with original weights Trained with VOC dataset Trained with VOC using only person class Trained with VOC and our V0 dataset
VOC True Positives 2666 1923 1889 3311
False Positives 1690 1231 1300 1440
False Negatives 1886 2622 2674 1176
Mean IOU 0.69198 0.670148 0.664489 0.73257
Precision 0.612029 0.609702 0.592349 0.696906
Recall 0.585677 0.423102 0.413982 0.73791
Yolo with original weights Trained with VOC dataset Trained with VOC using only person class Trained with VOC and our V0 dataset
OURS V0 True Positives 789
False Positives 10
False Negatives 18
Mean IOU 0.745622
Precision 0.987484
Recall 0.977695
Yolo with original weights Trained with VOC dataset Trained with VOC using only person class Trained with VOC and our V0 dataset
FUSED True Positives 3151 2203 2207 2636
False Positives 1900 1481 1465 1170
False Negatives 2206 3146 3150 2707
Mean IOU 0.687934 0.665656 0.665103 0.693964
Precision 0.623837 0.597991 0.601035 0.692591
Recall 0.58202 0.411853 0.411984 0.493356

Training YOLO with depth data[edit]

Trained using the depth information of the V0 dataset. Results for testing sample:

Yolo trained with depth as colormap
True Positives 787
False Positives 0
False Negatives 0
Mean IOU 0.863178
Precision 1
Recall 1


Video:

video


Developing environment[edit]

Sensor[edit]

Microsoft Kinect v2(K4W2)

SDK -> libfreenect2[edit]

Instalation process[edit]

git clone https://github.com/OpenKinect/libfreenect2
cd libfreenect2/depends
./install_ubuntu
../examples/protonect
mkdir build
cd build
cmake ..
make -j4
cd ../bin
sudo ./Protonect

Troubleshoot[edit]

  • OpenCL.

If we get the error: error: CL/cl.hpp not found, you may want to install openCL (opencl-dev) or directly disable openCL: cmake -DENABLE_OPENCL=OFF . make -j4

  • OpenGL deconding is not working:

If everything seems to work fine but the ir and depth images are completely black your gpu acceleration is not working, you may want to fix the problem (maybe due to a dual graphic card, you can try bumblebee) or disable gpu acceleration. To disable the gpu acceleration you have to change the line: OpenGLDepthPacketProcessor *depth_processor = new OpenGLDepthPacketProcessor(0, debug_); (on src/packet_pipeline.cpp:108) to: CpuDepthPacketProcessor *depth_processor = new CpuDepthPacketProcessor();

Thats means that all the decoding will be computed in the cpu using multithreading parallelism.

  • Hide depth_packet_processor window
 static const bool do_debug = false; (on opengl_depth_packet_processor.cpp#L322)

Compatible hardware[edit]

  • Mini Pc

IntelNUC

  • USB controller

NEC D720200 other controllers

Components[edit]

kinect2Server[edit]

video


Resources[edit]

  • Kinect Datasets

http://www0.cs.ucl.ac.uk/staff/M.Firman/RGBDdatasets/

  • Libfreenect

https://github.com/OpenKinect/libfreenect2


Interesting stuff[edit]

TLD[edit]

http://docs.opencv.org/3.0-beta/modules/tracking/doc/tracker_algorithms.html https://www.youtube.com/watch?v=pj-QuE6pdEQ