TOWARDS SLAM-AWARE SCENE UNDERSTANDING

Transkript

1 TOWARDS SLAM-AWARE SCENE UNDERSTANDING Bowl Bowl Co ee Mug Soda Can Cap Sudeep Pillai Marine Robotics Group CSAIL, MIT Jan 27, 2016

2 MOTIVATION Our goal: To enable a robot to capture and understand the spatial structure and semantic properties of their immediate environment 2

3 MOTIVATION SLAM-AWARE RECOGNTION MAP REPRESENTATION SLAM-AWARE LEARNING 2 3 Bowl Bowl Co ee Mug 4 Soda Can Cap 1 M Monocular SLAM Supported Object Recognition S. Pillai & J. Leonard (RSS 15) High-Performance and Tunable Stereo Reconstruction S. Pillai, S. Ramalingam & J. Leonard (ICRA 16) Large-scale SLAM-aware Object Learning (in the works) 3

4 MOTIVATION Robots equipped with a single RGB camera need to continuously recognize and localize all potential objects in its immediate environment Single RGB Camera Versatile Robust Avoid spurious detection/mis-classification Multi-view Object Detection Camera & Object localization by leveraging SLAM Scalable Runtime sub-linear in categories identifiable Input RGB Video 4

5 RECENT WORK Leverage SLAM capabilities Semi-dense reconstructions could potentially propose objects Object Detection / Recognition can be better informed with SLAM ORB-SLAM Mur-Artal et al. LSD-SLAM Engel et al. (ECCV 2014) Detection-based Object Labeling in 3D scenes Lai et al. (ICRA 2012) SLAM++ Salas-Moreno et al. (CVPR 2013) 5 Semi-Dense Mapping with ORB-SLAM Mur-Artal et al. (RSS 2015)

6 SLAM-AWARE RECOGNITION 1 M SLAM-aware {, M} Keyframes Map M 6

7 SLAM-AWARE RECOGNITION 2 1 M SLAM-aware {, M} Keyframes Map M 7

8 SLAM-AWARE RECOGNITION M SLAM-aware {, M} Keyframes Map M 8

9 SLAM-AWARE RECOGNITION M SLAM-aware {, M} Keyframes Map M 9

10 SLAM-AWARE RECOGNITION M Keyframes SLAM-aware {, M} Map M ŷ MLE = Most likely argmax y2{1,..., C } p(d o y) 8 o 2 O semantic label 10

11 KEY CONCEPT SLAM-capable robots equipped with a single RGB camera need to continuously recognize and localize all potential objects in its immediate environment Single RGB Camera (Monocular SLAM supports improved recognition) Robust (Reduced false positives via view correspondence from SLAM) Multi-view Object Detection (Proposals from Semi-Dense Maps) Scalable (FLAIR encoding) SLAM-Supported Object Recognition 11

12 MONOCULAR SLAM-SUPPORTED OBJECT RECOGNITION SEMI-DENSE MAPPING Input RGB Video Scale-ambiguous Reconstruction UW-RGBD Dataset (v2) Lai et al. (ICRA 2014) (a) Semi-Dense Depth Filtering SVO: Forster et al. (ICRA 2014) + ORB-SLAM Mur-Artal et al. As a camera-equipped robot moves around in its immediate environment, the camera is localized and simultaneously a semi-dense map is constructed 12

13 MONOCULAR SLAM-SUPPORTED OBJECT RECOGNITION MULTI-VIEW OBJECT PROPOSALS Scale-ambiguous Reconstruction Filtered Reconstruction (a) Semi-Dense Depth Filtering SVO: Forster et al. (ICRA 2014) + ORB-SLAM Mur-Artal et al. (b) Low-density region pruning The semi-dense reconstruction provide spatio-temporally consistent edges, a strong indication of object presence 13

14 MONOCULAR SLAM-SUPPORTED OBJECT RECOGNITION MULTI-VIEW OBJECT PROPOSALS Scale-ambiguous Reconstruction Multi-scale Over-segmentation (a) Semi-Dense Depth Filtering SVO: Forster et al. (ICRA 2014) + ORB-SLAM Mur-Artal et al. (b) Density-based segmentation over 4 spatial scales Object proposals are extracted from the reconstructed scene via a multi-scale density segmentation step, and are further refined for occlusions 14

15 MONOCULAR SLAM-SUPPORTED OBJECT RECOGNITION MULTI-VIEW OBJECT PROPOSALS Multi-scale Over-segmentation View-Consistent Proposals View View (d) View (b) Density-based segmentation over 4 spatial scales View Candidate proposals projection The segmented regions are projected onto each keyframe enabling view-consistent object proposals that perform better than frame-based proposal methods 15

16 MONOCULAR SLAM-SUPPORTED OBJECT RECOGNITION VIEW AGGREGATION Unified Object Prediction SLAM-Supported Object Recognition Bowl View Bowl Co ee Mug Soda Can Cap Fusion of object proposal classification The object proposals in each frame are encoded (via VLAD + FLAIR or R-CNN) and classified, before their evidence is probabilistically fused across all keyframes 16

17 SLAM-SUPPORTED vs. FRAME-BASED OBJECT RECOGNITION SLAM-Supported Recognition (Ours) Frame-based Recognition (Classical approach) CORRECT PREDICTIONS 17 INCORRECT PREDICTIONS

18 SLAM-SUPPORTED vs. FRAME-BASED OBJECT RECOGNITION Object hypotheses are aggregated across views resulting in correct classification SLAM-Supported Recognition (Ours) Frame-based Recognition (Classical approach) CORRECT PREDICTIONS 18 INCORRECT PREDICTIONS

19 SLAM-SUPPORTED vs. FRAME-BASED OBJECT RECOGNITION 19

20 SLAM-SUPPORTED vs. FRAME-BASED OBJECT RECOGNITION PERFORMANCE Recognition performance RGB: Superior to frame-based methods RGB-D: Comparable to state-of-the-art SLAMbased recognition methods Higher is better SINGLE-VIEW RECOGNITION VLAD + FLAIR encoding: Run-time nearly independent of object categories identifiable Ours (RGB) map DetOnly (RGB) Conclusion SLAM-aware / Multi-view reasoning Object proposals via semi-dense reconstructions Scalable recognition (FLAIR encoding) Higher is better MULTI-VIEW RECOGNITION Comparable results to RGB-D recognition 20 map Ours Det3DMRF HMP2D+3D (RGB) (RGB-D) (RGB-D) map - (mean Average Precision)

21 HIGH-PERFORMANCE AND TUNABLE STEREO RECONSTRUCTION S. Pillai, S. Ramalingam & J. Leonard (ICRA 16) 21

22 MOTIVATION Full Scene Reconstruction using Stereo SGBM / ELAS (~ ms) Scene Discretization and Fusion using Octomap (~ 20 ms) 22

23 MOTIVATION NAVIGATION PLANS (SLOW) Discretized Scene Reconstruction (~ 200 ms / 5 Hz) 23

24 MOTIVATION? Navigation-focused / Plan-Aware Scene 5 m/s (11 m/s (22 20 m/s (45 mph) 5 Hz / FPS 5 m/s => 1 m / frame 10 m/s => 2 m / frame 20 m/s => 4 m / frame 100 Hz / FPS 5 m/s => 0.05 m / frame 10 m/s => 0.1 m / frame 20 m/s => 0.2 m / frame

25 PIECE-WISE PLANAR APPROXIMATION VIA TESSELLATION Step 1 & 2: Establish tessellation and piece-wise planar approximation given the sparse key points and their corresponding disparities 25

26 RECURSIVE DEPTH REFINEMENT Step 3 & 4: Determine incorrectly approximated regions in the image (matching cost check), resample, re-tessellate, and re-estimate stereo disparity in these regions 26

27 RECURSIVE DEPTH REFINEMENT VIA TESSELLATION Fast piece-wise planar approximation Extremely efficient ( Hz) 1 Iteration Iterative refinement (any-time performance) Tunable (high-speed/low-accuracy or highaccuracy/low-speed regimes) 2 Iterations Surprisingly accurate for reconstruction Depth prior 4 Iterations 2 iterations: 90 % within 3px disparity 80 Hz 27

28 FAST YET SURPRISINGLY POWERFUL Depth prior determined via Delaunay triangulation of sparse support points. Disparities for the sparse support points are obtained via sparse-stereo matching 28

29 SEMI-DENSE DISPARITY ESTIMATION Interpolated semi-dense disparities estimated from the piecewise planar tessellation 29

30

31 CONSISTENT STEREO RECONSTRUCTIONS AT HIGH-SPEEDS High-performance and tunable stereo algorithm Fastest semi-dense stereo algorithm to-date Piece-wise planar approximation Any-time performance Adaptive algorithm that enables for low-speed / high-quality or high-speed / low-quality stereo reconstructions Compelling reconstructions with fused VO for high-speed navigation Locally-consistent stereo reconstructions using our high-performance stereo algorithm coupled with stereo visual odometry 31

32 SLAM-AWARE LEARNING Wide/Narrow Baseline Stereo RGB/Depth Camera Laser Rangefinder Simulated Turtlebot (Gazebo) in an environment

33 SLAM-AWARE LEARNING Simulated Robot Exploration Large-scale data collection Life-long object learning / autonomy Object Detector evolution Objects as landmarks Train in simulation, adapt to real-world Coupled recognition and SLAM Scalable scene description Viewpoint and Lighting Invariant Human interpretable Simulated Turtlebot (Gazebo) exploring in an environment. SLAM provides view correspondences

34 SLAM-AWARE OBJECT LEARNING 2 3 Tag detections are registered using deadreckoned (odometry) pose 1 M 4 Single object instance / Multiple-views Dead-reckoned pose trajectory of robot

35 THANKS! Contact: Sudeep Pillai Webpage: 35