Robot Vision


Vision technology is a significant for Intelligent robot in cognition of environment. Various sensors such as LRF, camera, UltraSonic and IR sensor are fused on our mobile robot platform. Object detection, Object tracking and Localization are the main part. Recently we are on the project supported by “Development of mobile assisted robot and emotional interaction robot for the elderly ” under the Industrial Source Technology Development Programs of the Ministry of Knowledge Economy (MKE) of Korea.

Definition of intelligent robot (

Robot Platform

Current configuration of mobile robot platform

Robot vision researches performed in our laboratory can be divided into 4 categories:

Object Detection

Pedestrian Detection

The detection of pedestrian has attracted much research in the past decade due to the essential role it plays in intelligent video surveillance and vehicle vision systems. However, the existing algorithms do not meet the requirement of real applications in terms of detection performance. This paper proposes a new robust algorithm for pedestrian detection based on image reconstruction using bidirectional PCA BDPCA). Unlike PCA, since it is a straightforward image projection technique, BDPCA preserves the shape structure of objects and is computationally effective. Due to these advantages, BDPCA is a promising tool for object detection and recognition. The algorithm was tested on two datasets, INRIA and ennFudanPed. Our experiment proved that using BDPCA with vertical edge images was the most suitable for pedestrian detection. The comparison between BDPCA, PCA, and histogram of oriented gradient (HOG) based methods demonstrates superior accuracy and robustness of the proposed algorithm to the others.

BDPCA-based pedestrian classification

BDPCA-based pedestrian classification includes two steps: training and classification.

Process of BDPCA-based pedestrian classification

Experiment Results

The number of row eigenvectors is from three to 64. The number of column eigenvectors is from three to 128. The analysis shows that the algorithm achieves the highest performance with the vertical edge descriptors and when the size of row projector is 64 and that of column projector is 90. Fig. 8 displays the ROC curve of different BDPCA descriptor types. Only the best results from each descriptor type are shown. The vertical edge descriptor eliminates information about clothes colors and keeps only information about pedestrian contours; hence it provides the best performance. In the experimental results, combination of grayscale and vertical edge decreases the performance of using vertical edge only. This result is understandable since gray levels are sensitive to noise, clothes colors and illumination.

ROC curve of different BDPCA descriptors

Related Papers :

Thi-Hai-Binh Nguyen, Hakil Kim, “Novel and Efficient Pedestrian Detection using Bidirectional PCA”, Pattern Recognition, Feb 28, 2012

Back to top

Object Tracking

Mobile robots with a side-view camera have problems as camera jitter, illumination change, object shape variation and occlusion in variety environments. In order to overcome these problems, color histogram and HOG descriptor are fused for efficient representation of an object. Particle filter is used for robust object tracking with on-line learning method IPCA in non-linear environment. The validity of the proposed algorithm is revealed via experiments with DBs acquired in variety environment.

Flowchar of Object Tracking

Original image and 24 HOG descriptors with 8 orientation bin

Object tracking in Day & Night condition

Experiment Results

Performance evaluation in 8 DBs

Related papers :

Hyung-Ho Lee, Xuenan Cui, Seung-Wan Ma, Hyung-Rae Kim, Jae-Hong Lee, and Hakil Kim, “Robust Object Tracking in Mobile Robots using Object Features and On-line Learning based Particle Filter,” Journal of Institute of Control, Robotics and Systems (in Korean), vol. 18, no. 6, pp. 562-570, Jun. 2012.

Back to top

Elevator Recognition

Door recognition by LRF and verification by camera Indicator recognition by NN

The recognition of elevator door is needed for mobile service robots to moving between floors in the building. Using the laser scans by the LRF, we extract line segments and detect candidates as the elevator door. Using the image by the camera, the door candidates are verified and selected as real door of the elevator. The outliers are filtered through the verification process. Then, the door state detection is performed by depth analysis within the door. The proposed method uses extrinsic calibration to fuse the LRF and the camera.

LRF scanning in front of elevator.

Pseudo-code for line splitting algorithm

Experiment Results

Elevator door and indicator recognition

(a) Results of ‘door is’ DB (b) Results of ‘door isn’t’ DB.

Related papers :

Seung-Wan Ma, Xuenan Cui, Hyung-Ho Lee, Hyung-Rae Kim, Jae-Hong Lee, and Hakil Kim, “Robust Elevator Door Recognition using LRF and Camera,” Journal of Institute of Control, Robotics and Systems (in Korean), vol. 18, no. 6, pp. 601-607, Jun. 2012.

Back to top

Localization using Artificial Landmarks

Localization using Fiducial markers has the advantage that errors of global position are not cumulative. In addition, the implementation is simple. ARToolKitPlus used to recognize the fiducial marker.

Procedure of marker detection

Procedure of marker detection

Transformation Matrix

Localization using fiducial marker

Related papers:

Seung-Wan Ma. (2012).Localization using Fiducial Markers and Elevator Door Recognition of Mobile Robot. Unpublished master dissertation, Inha University,Incheon.

Back to top

Demo Video

[2012.01] Object Tracking and Elevator Recognition

Back to top

Localization using Natural Landmarks

Localization using Natural markers is not needed extra works for attaching landmarks on the ceil or walls. So the proposed method is more efficient in various environment like a hall or a lobby. In order to estimate the position of robot, landmark DB has to be prepared. ORB is used for feature extraction and descriptor.

Natural Landmarks

Natural Landmarks

Natural Landmarks

 Experimental Environment (Hightech center in Inha Univ.)

Experiment Results

Results for static images (Error in %)

Results in Diriving corridor

Related papers:

Jaehong Lee, Hyungrae Kim, Xuenan Cui and Hakil Kim, “Localization of Mobile Robot based on Feature Matching using a Single Camera”. Workshop on Image Processing and Image Understanding (IPIU), Feb. 2013.

Back to top