Scan registration benchmarks

Here we provide a more complete set of statistics from the results presented in the ICRA'15 paper Beyond Points: Evaluating Recent 3D Scan-Matching Algorithms by Magnusson, Vaskevicius, Stoyanov, Pathak, and Birk. For more detail about the benchmark protocol that the results are based on, please refer to Comparing ICP variants on real-world data sets by Pomerleau et al. and the associated web site.

The plots below summarise the results of the local scan-matching methods P2D-NDT and D2D-NDT in comparison with point-to-plane ICP.

These are cumulative probability plots, showing the proportion of outcomes below any given error. The final translation error (in metres) is shown on the horizontal axis, and the vertical axis shows the cumulative probability. Percentiles for 5, 25, 50, 75, and 95 are shown with dashed horizontal lines. The plots also include a suggested success threshold (10 cm).

Additional plots will be added later.

Complete benchmark

Here are the results for the complete benchmark protocol, including all scan pairs and pose perturbations.

Your browser does not support SVG Your browser does not support SVG Your browser does not support SVG

Sensitivity to overlap ratio

The following plots relate to the sensitivity to the amount of overlap between the two pairs.

Large overlap only

Here are the results for all scan pairs with more than 75% overlap, but including all pose perturbations.

Your browser does not support SVG Your browser does not support SVG Your browser does not support SVG

Small overlap only

Here are the results for all scan pairs with less than 50% overlap, but including all pose perturbations.

Your browser does not support SVG Your browser does not support SVG Your browser does not support SVG

Sensititivy to pose estimate

Easy poses only

Here are the results for the easy pose perturbations only, but including all scan pairs.

Your browser does not support SVG Your browser does not support SVG Your browser does not support SVG

Hard poses only

Here are the results for the hard pose perturbations only, but including all scan pairs.

Your browser does not support SVG Your browser does not support SVG Your browser does not support SVG

All results, tabulated

In the table below, the accuracy results are categorised by initial pose and overlap ratio, respectively. Entries where both translation and rotation error are below the thresholds 0.10 m and 0.05 rad are shaded. Entries where translation or rotation is below the threshold are lightly shaded. Your browser does not support SVG

References

  1. F. Pomerleau, F. Colas, R. Siegwart, and S. Magnenat, Comparing ICP variants on real-world data sets, Autonomous Robots, Volume 34, Issue 3, pp 133-148, Feb. 2013.
  2. M. Magnusson, N. Vaskevicius, T. Stoyanov, K. Pathak, and A. Birk, Beyond Points: Evaluating Recent 3D Scan-Matching Algorithms, Proc. ICRA, 2015.