Testing Global Accuracy and Georeferenced Accuracy in 3D Mapping with the Elios 3 and GeoSLAM Connect

The focus of this white paper is on global accuracy and georeferenced accuracy as tested in a large industrial warehouse. View the results of testing we did for system precision and local accuracy as tested in an office space in this article.

Note: The first two sections of this article (the Introduction and Why We Performed These Tests) are the same in this article and the one mentioned above. If you've read the other white paper, we recommend you begin reading at the Defining Our Terms section below.

test results OVERVIEW

Introduction

Over the last few years, LiDAR data has quickly become one of the most reliable foundations for creating precise and accurate 3D models.

Sectors like mining, construction, and infrastructure are using these models to conduct routine inspections, make safety determinations, track the change of assets over time, and support project planning. 

The types of outputs professionals in these industries are getting from 3D models made with LiDAR data include detailed digital twins, accurate 2D and 3D measurements, the ability to pinpoint the locations of defects within assets, the ability to export data to common 3D point cloud file extensions like *.e57, *.las, *.laz, and *.ply, and the ability to merge multiple georeferenced 3D models to track changes in assets over time.

Regardless of the industry or output, the quality of the model is key to its usefulness. If the data is not precise and accurate—terms that have specific definitions in 3D modeling, which will be covered in a separate section below—then it may not represent the real world well enough to offer valuable insights.

This article covers findings from tests performed by experts at GeoSLAM and the Flyability product team that highlight findings from 3D models created using the Elios 3 with GeoSLAM Connect to identify global accuracy and georeferenced accuracy as compared to the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems in the market.

Why We Performed These Tests 

Flyability’s Elios 3 comes with Ouster’s OS0-32 LiDAR sensor and the ability to perform SLAM (simultaneous localization and mapping), which means that it can create 3D models in real time, while in flight. 

After the flight, Elios 3 users can process the LiDAR data they collected with GeoSLAM Connect to create precise, accurate 3D models. 

The 3D Live Model and the post-processed model have distinct uses, and should not be seen as the same kinds of 3D models. While the 3D Live Model can be used during a mission for navigation, route planning, and verifying scan coverage, the post-processed model you make with GeoSLAM Connect can provide an accurate point cloud.

Processing Elios 3’s LiDAR data with GeoSLAM Connect adds the ability to create precise, accurate 3D mapping to Flyability’s field-tested collision tolerance technology for work in confined spaces, allowing inaccessible environments to be mapped effectively.

But potential Elios 3 users may have questions about what happens when Flyability’s indoor drone technology is fitted with a LiDAR payload and that data is post-processed, such as: 

• Will the drone’s vibrations or environmental factors like dust or moisture impact the precision of the resulting 3D models? 
• What will the resulting 3D models look like, and how useful will they be?

To address these questions, we performed a thorough analysis of global accuracy and georeferenced accuracy with the Elios 3’s point clouds when processed with GeoSLAM Connect. All the tests were conducted in a manner that would be both representative and repeatable. 

Keep reading to learn the results.

Defining Our Terms: Global Accuracy, Georeferenced Accuracy, and Drift in 3D Mapping 

Global accuracy in 3D modeling relates to the distance between two points in a point cloud, where the object cannot be viewed from a single position (e.g., the distance between two rooms). 

Georeferenced accuracy is global accuracy plus inaccuracies caused by the alignment method. Alignment methods include target based registration and Iterative Closest Point (ICP).

Drift is a term used in 3D modeling to describe global accuracy. Global accuracy cannot easily be expressed in absolute values (e.g., +-2cm/.79 in) because, in the absence of ground control points or GNSS, the absolute error usually grows along with the size of the asset, or the distance that is measured. 

For example, the error on a 30m (98.4 feet) measurement is likely to be smaller than the error on a 300m (984 feet) distance measurement because a mobile scanner moving through an asset will accumulate errors on top of previous errors the longer it travels and records data. This accumulation of errors over distance is described by the word drift, representing a percentage of the traveled distance during data collection—for example, a 1% drift on a 300m (984 feet) distance corresponds to a 3m (9.84 feet) error as compared to reality.

Global Accuracy and Georeferenced Accuracy Assessments with the Elios 3

To evaluate the global accuracy and georeferenced accuracy of the Elios 3’s point clouds processed with GeoSLAM Connect, GeoSLAM 3D mapping experts and the Flyability product team carried out tests within an industrial factory.

Keep reading to get the details of the tests, and to learn the results.

Establishing a Control

When assessing the accuracy of any system, a second measurement system must be used to provide the benchmark value (i.e., the control) and this system must be of greater accuracy than the system being tested. 

To test the accuracy of a mobile mapping solution like the Elios 3, the industry standard is to use either a Total Station (TPS) or a Terrestrial Laser Scanner (TLS) as a control, because the accuracy for both of these approaches exceeds that of a mobile mapping solution.

The reason the TPS and TLS approaches obtain greater accuracy than a mobile mapping solution is because they each capture data from a single, stationary position, with multiple positions registered together using point matching algorithms.

In comparison, a mobile mapping solution like the Elios 3 moves continuously as it collects data, capturing data at multiple positions as the drone moves through the environment it is mapping.

The global accuracy and georeferenced accuracy tests we performed relied on data collected by a Terrestrial Laser Scanner (TLS) as their control. It is to be noted that the control took more than six hours of time to acquire the data, due to the size of the asset. In comparison, the Elios 3 took 8.5 minutes to acquire the data.

Test Environment 

To evaluate the global accuracy and georeferenced accuracy of the Elios 3’s point clouds processed with GeoSLAM Connect, the LiDAR data was captured in a factory named the Blue Factory, located in Fribourg, Switzerland.

The factory consists of 12 rooms of varying sizes separated by several doorways, presenting a representative industrial environment suitable for the Elios 3. 

A room from the test environment

Figure 1. An overview of the entire test environment

15x Retroreflective Diamond Grade 3M (9.84 feet) targets (200mm x 200mm / 7.87 in x 7.87 in) were placed evenly around the test environment, as shown in the next two pictures below:

Collecting the Data

Three scans were carried out with the Elios 3 to capture LiDAR data for testing, all following the same approximate flight path to ensure consistency between the results. 

All the scans started and ended in the same location, both for consistency and to ensure that the data capture loop was closed, as recommended. Best practice for SLAM data capture was maintained by performing loops within the capture and entering doorways sideways to ensure good visualization when moving into new environments.

The three scans had an average flight time of 8 minutes and 30 seconds with 108 million points captured per scan over the ~450 meters (1,476 feet) flight path. 

Figure 2. The path the drone followed in the test environment

Watch this video to see an accelerated recording of the drone's flight path:

Data Processing & Centroid Extraction

To ensure that the test was representative of what an end user can expect from the system, the processing was carried out using the standard Flyability processing parameters found in GeoSLAM Connect 2.1.1.  The data was not reprocessed by any other means, neither was it decimated nor filtered.

As a post-processing step, an extraction tool* was run to identify the 15x targets in both the Elios 3 data and the TLS data. Once the targets were identified, the tool extracted the centroids of the targets to provide 15x centroids for both the Elios 3 and the TLS data. The centroids of the TLS data were used as Control Points and the Elios 3 centroids were used for comparison. 

*Note: This extraction tool will be available in GeoSLAM Connect 2.3.0.

Assessing Global Accuracy—Distance Measurements

To assess the global accuracy of the Elios 3, distance measurements were carried out and the Elios 3 data was compared against the TLS control data.  

The steps carried out to complete the process were:

1. Distance measurements. The distance between a dispersed array of pairs of centroids was measured for both the control TLS data and the Elios 3 scans .
2. Find residuals. Residuals were found between the point pair distance of TLS data and the Elios 3 point pair distances.
3. Find RMSE. The RMSE (root mean square error) of the residuals was calculated for each distance from the residuals of the 3 scans.
4. Find RMSE as a percentage of length. The RMSE of the residuals as a percentage of distance between centroids was computed.
5. Find the mean error. Finally, the average of the RMSE (MAE) of the residuals as a percentage of distance between centroids was computed.

The centroid pairs used to find the residuals and in turn the RMSE, RMSE as a percentage of length, and the MAE are shown in Figure 3 below.

Figure 3. Point pairs used for global accuracy assessment

Global Accuracy Results

The results from the global accuracy assessment of the Elios 3 data when processed using GeoSLAM Connect provided a global accuracy RMSE of 35mm (1.38 inches), or a drift of 0.16% (mean error with regard to the distance being measured).

Assessing Georeferenced Accuracy—Cloud-to-Cloud Alignment around Take-Off

In the previous section we discussed the global accuracy of the system. In this section we will assess the first registration method—cloud-to-cloud alignment around the take-off location.

Georeferenced accuracy assesses the global accuracy of the system as well as the accuracy of the georeferencing technique used.

To simulate a common use case for Elios 3 scans, the Elios 3 point cloud was aligned to the reference model around the take-off location. Doing this simulates the procedure that might be followed during a mission in an inaccessible area, in which Elios 3 is used to complete an existing georeferenced model and the control can only be placed at the flight’s starting location. 

Only having control in one section of the scan environment causes any inaccuracies in the registration process to propagate throughout the scan and will result in increased inaccuracies as compared to using targets or ground control points (GCPs) across the whole scan.

Figure 4. TLS reference model with the 15m (49.2 feet) section of the Elios 3 scan used for cloud-to-cloud alignment overlaid

To assess the cloud-to-cloud alignment accuracy of the Elios 3 using GeoSLAM Connect, the following workflow was implemented:

1. A 15 meter (49.2 feet) section of the scan around the take-off location was used to perform cloud-to-cloud registration.
2. The 15 meter section of the TLS data was used as the reference and the 15m section of the Elios 3 data was aligned to the reference using cloud-to-cloud alignment. This alignment was then applied on the entire Elios 3 point cloud.
3. The reference centroids from the TLS data were recorded and compared to the aligned Elios 3 centroids. 
4. The residuals between the reference centroids and the aligned centroids was calculated. 
5. The RMSE for both dXY and dXYZ for the 3x scans was computed for each reference point. 
6. The average RMSE values for both dXY and dXYZ were output. 

Results

The results from assessing cloud-to-cloud alignment and global accuracy were split to show the RMSE for dXY and dXYZ. The results show that only using a section of the scan for registration and not the whole scan will increase inaccuracies. 

The dXY cloud-to-cloud results can be seen here:

The dXYZ cloud-to-cloud results can be seen here:

Assessing Georeferenced Registered Accuracy—Target Based Alignment with Four Model-Wide GCPs

To simulate an alternate use case of referencing the Elios 3 point cloud to a set of ground control points (GCPs), target based alignment was undertaken. The alignment performed was a rigid transformation. 

Assessing the target based accuracy will include both the accuracy of the system and the accuracy of the target based registration process. 

This approach simulates a use case in which Elios 3 is used to scan an environment that comprises known features or control points in various locations beyond only the take-off location.

To undertake the alignment process, the following methodology was applied:  

1. Four targets were chosen around the periphery of the scan—one target each from the North, South, East, and West of the dataset. Target control points 3,6,11, and 14 were used (see Figure 5 below).  
2. The centroids of the four TLS targets were used as reference points. 
3. The centroids of the same four targets from the Elios 3 scans were aligned to the four reference points.
4. The dXY and dXYZ residuals between the TLS centroids and the Elios 3 centroids were calculated for all 15x targets. 
5. The RMSE per target for all the scans was computed. 
6. RMSE were computed for each scan and an overall average RMSE value for target based alignment accuracy was calculated for both dXY and dXYZ.

Figure 5. Four targets around the periphery of the scan

Results

Due to the control being located around the point cloud (NSEW), not only at the flight start location, there were tighter constraints applied to the registration process.  

These constraints helped produce improved accuracy of the registration process in comparison to the cloud-to-cloud alignment process, in which only a 15 meter (49.2 feet) section of the scan around the start location was used to align the point clouds. In this scenario, a user has the ability to place control at varying positions around the scan. 

Using target based registration from the four points around the periphery of the scan led to a 28% improvement in accuracy of the registered point cloud, as compared to the cloud-to-cloud referencing process using only a 15 meter section around the flight start location.  

It should be noted that an even more accurate model would be obtained if the GCPs were used in the SLAM processing. This capability is planned to be integrated for future releases of GeoSLAM Connect. 

The results from assessing the target based alignment were split to show both the RMSE for dXY and dXYZ. The tables below show that the RMSE for dXY were lower than the dXYZ results, as expected.

The dXY target based results can be seen here:

The dXYZ target based results can be seen here:

Conclusion

The test results show that the Elios 3’s point clouds processed with GeoSLAM Connect produce high accuracy and present an appropriate tool to meet survey requirements, with a global system accuracy of 35mm (1.38 inches).

In addition to the system accuracy, the georeferencing capabilities of the Elios 3 were assessed simulating on site practices.  

The Cloud-to-Cloud and Target Based Alignment assessments show how the Elios 3 can easily be implemented in both map and georeference-inaccessible environments and achieve georeferenced accuracies up to 60mm (2.36 inches).