Testing Local Accuracy and System Precision in 3D Mapping with the Elios 3 and GeoSLAM Connect

Results suggest that the precision and local accuracy of Elios 3’s point clouds when processed with GeoSLAM Connect compare well against a traditional TLS and reach a similar level of accuracy to the accuracy obtained with the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems in the market.

The focus of this article is on system precision and local accuracy as tested in an office space. View the results of testing we did for global accuracy and georeferenced accuracy in a large warehouse in this article.

Note: The first two sections of this article (the Introduction and Why We Performed These Tests) are the same in this article and the one mentioned above. If you've read the other white paper, we recommend you begin reading at the Defining Our Terms section below.

test results OVERVIEW

  • What was tested. System precision and local accuracy of 3D models made with LiDAR data collected by Flyability’s Elios 3 processed with GeoSLAM Connect.
  • Who did the testing. GeoSLAM 3D mapping experts and members of the Flyability product team.
  • What tests were conducted. Local accuracy was tested with a Plane to-Plane analysis. System precision was tested with a Range Noise evaluation.
  • Reference model. The reference model used for the tests was made with a TLS (Terrestrial Laser Scans) Riegl  VZ-400, and the registration process was undertaken using RiScan Pro V2.14.1.
  • Test results—local accuracy. All the comparisons fell within +/- 16mm (.63 inches), and the Mean Absolute Normal Distance between the Elios 3 and the Reference Model was 8mm (.31 inches).
  • Test results—precision analysis. The standard deviation of all the planes fell within 15mm (.59 inches) and the Mean Standard Deviation between the Elios 3 and the Reference Model was 8mm to 1-sigma.

Introduction

Over the last few years, LiDAR data has quickly become one of the most reliable foundations for creating precise and accurate 3D models.

Sectors like mining, construction, and infrastructure are using these models to conduct routine inspections, make safety determinations, track the change of assets over time, and support project planning. 

The types of outputs professionals in these industries are getting from 3D models made with LiDAR data include detailed digital twins, accurate 2D and 3D measurements, the ability to pinpoint the locations of defects within assets, the ability to export data to common 3D point cloud file extensions like *.e57, *.las, *.laz, and *.ply, and the ability to merge multiple georeferenced 3D models to track changes in assets over time.

Regardless of the industry or output, the quality of the model is key to its usefulness. If the data is not precise and accurate—terms that have specific definitions in 3D modeling, which will be covered in a separate section below—then it may not represent the real world well enough to offer valuable insights.

This article covers findings from tests performed by experts at GeoSLAM and the Flyability product team that highlight findings from 3D models created using the Elios 3 with GeoSLAM Connect to identify system precision and local accuracy as compared to the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems in the market.

Why We Performed These Tests

Flyability’s Elios 3 comes with Ouster’s OS0-32 LiDAR sensor and the ability to perform SLAM (simultaneous localization and mapping), which means that it can create 3D models in real time, while in flight. 

After the flight, Elios 3 users can process the LiDAR data they collected with GeoSLAM Connect to create precise, accurate 3D models. 

The 3D Live Model and the post-processed model have distinct uses, and should not be seen as the same kinds of 3D models. While the 3D Live Model can be used during a mission for navigation, route planning, and verifying scan coverage, the post-processed model you make with GeoSLAM Connect can provide an accurate point cloud.

Processing Elios 3’s LiDAR data with GeoSLAM Connect adds the ability to create precise, accurate 3D mapping to Flyability’s field-tested collision tolerance technology for work in confined spaces, allowing inaccessible environments to be mapped effectively.

But potential Elios 3 users may have questions about what happens when Flyability’s indoor drone technology is fitted with a LiDAR payload and that data is post-processed, such as: 

  • Will the drone’s vibrations or environmental factors like dust or moisture impact the precision of the resulting 3D models? 
  • What will the resulting 3D models look like, and how useful will they be?

To address these questions, we performed a thorough analysis of system precision and local accuracy with the Elios 3’s point clouds when processed with GeoSLAM Connect. All the tests were conducted in a manner that would be both representative and repeatable. 

Keep reading to learn the results.

Defining Our Terms: Precision and Local Accuracy in 3D Mapping 

When discussing 3D models made with LiDAR data, the terms precision and accuracy have specific definitions.

Accuracy in 3D modeling is most universally defined as the degree of conformity of a measured quantity to its actual (benchmark) value.

For example, if you measure the distance of 100mm (3.9 inches) in your point cloud, but the actual or known distance is 500mm (19.7 inches), then your measurement is not accurate.

Accuracy is important in a 3D model to ensure that the model represents reality. This includes considerations like getting the shapes of the corners right, and avoiding having walls appear on top of each other.

Local accuracy in 3D modeling relates to the distance between two points in a point cloud when the object can be viewed from a single position (e.g., the dimensions of a single room). As noted above, this article contains test data only for local accuracy with the Elios 3 and GeoSLAM Connect. 

Precision in 3D modeling is defined as the degree to which further measurements show the same result.

For example,  if you measure the distance between two points on a 3D model five different times and get 100mm (3.9 inches) every time, then your measurements on the model are precise. Precision may also be called noise or repeatability.

The amount of precision you need in a 3D model will generally be determined by the industry in which you plan to use it.

For stockpile measurements used in mining, precision of a few centimeters can be enough, since the statistical error evens itself out. Other applications, such as construction, may require even greater precision.

It should be noted that precision sets a lower limit for the accuracy of some measurements, in particular when measurements are made by picking features on a point cloud. The lower the precision, the higher the noise in the point cloud, and the harder it is to pick the point corresponding exactly to the right feature. 

precision-accuracy-definitionsAccurate measurements fall in the bullseye. Precise measurements are tightly clustered.

System Precision and Local Accuracy Assessments with the Elios 3

To evaluate the precision and local accuracy of the Elios 3 with GeoSLAM Connect, GeoSLAM 3D mapping experts and members of the Flyability product team carried out:

  • A Plane to-Plane analysis to test local accuracy 
  • A Range Noise evaluation to test system precision

Establishing a Control

When assessing the accuracy of any system, a second measurement system must be used to provide the benchmark value (control), and this system must be of greater accuracy than the system being tested. 

To test the accuracy of a mobile mapping solution like the Elios 3, the industry standard is to use either a Total Station (TPS) or a Terrestrial Laser Scanner (TLS) as a control, because the accuracy for both of these approaches exceeds that of a mobile mapping solution like the Elios 3.

The reason the TPS and TLS approaches obtain greater accuracy than a mobile mapping solution is because they each capture data from a single, stationary position, with multiple positions registered together using point matching algorithms.

In comparison, a mobile mapping solution like the Elios 3 moves continuously as it collects data, capturing data at multiple positions as the drone moves through the environment it is mapping.

Collecting the Data

GeoSLAM experts and Flyability product team members captured data from an indoor planar surface environment using both the Elios 3 and an industry-standard TLS.

The test team used a Riegl VZ-400 TLS as control for these tests, defining the accuracy of the control from a single position at a set confidence level. 

The Riegl VZ-400 manufacturer states an accuracy of 5mm (.2 inches) at 1-sigma, meaning 68% of all measurements have to be within a range of 5mm (.2 inches). From the Riegl point cloud, the reference model was created to act as the known ground control.  

Aligning the Elios 3 Point Cloud to the Reference Model

To compare the Elios 3 point cloud to the TLS reference model effectively, the Elios 3 model was first aligned to the reference model.

The alignment of the Elios 3 model changed the position and orientation of the point cloud data, bringing it into the coordinate system of the TLS reference model. 

GeoSLAM used PolyWorks|Inspector MRS2019 IR3 software to align the Elios 3’s point cloud with the TLS reference model. PolyWorks is a 3D analysis and quality control software solution used to assess product accuracy.

Here are the steps followed to execute the alignment:

  1. Manual alignment. In PolyWorks, a manual alignment was used to provide an initial rough alignment of the comparison point cloud (i.e., the Elios 3 point cloud) to the reference model (i.e., the TLS reference model).
  2. Computed the transformation matrix. Once complete, the Best-Fit* Automatic Alignment function was used to create a computed  transformation matrix between the point cloud and the reference model. 
  3. Apply the transformation to the point cloud. After the rigid transformation matrix was calculated, it was applied to the point cloud to align the comparison Elios 3 data to the reference model.  

*The Best-Fit operation is a surface-based alignment tool that iteratively transforms the position and orientation of the comparison point cloud data to minimize the deviation of the point cloud with respect to the reference model.

Assessing Local Accuracy—Plane-to-Plane Comparison

A Plane-to-Plane comparison was carried out by fitting planes to both Elios 3 data and the surveyed reference model. The Normal Distance between the planes was then evaluated. 

Normal Distance was calculated by finding the difference between the extracted plane in the Elios 3 point cloud and the corresponding plane in the reference model.

Normal Distance was calculated using an automated workflow in PolyWorks MRS2019 IR3. This assessment indicated that the local accuracy of the point cloud and any variations across the point cloud were identified. 

Assessing System Precision—Range Noise Analysis

To assess the precision of the Elios 3, Range Noise Analysis was carried out to assess the precision of the Elios 3’s point cloud.

Range Noise is the difference between each range reading (point) and the mean range value within the selected area. The areas chosen to assess the Range Noise were the planar surfaces extracted for the Plane-to Plane comparison. 

The Range Noise is presented as a Standard Deviation from the mean point of the plane. Hence, Standard Deviation is the measure of the system precision and is given to 1-sigma. The Standard Deviation was computed using PolyWorks MRS2019 IR3. 

Test Environment 

To evaluate the accuracy and precision of the Elios 3’s point clouds when processed with GeoSLAM Connect, the data was captured in a standard office environment with 6 planar surfaces, approximately 1 meter (3.3 feet) square, located at frequent intervals around the scan. 

Locations for the planar surfaces can be seen below in Figures 2 and 3.

precision-accuracy-test-environment-1

Figure 2. Location of planar test surfaces 1 and 3

precision-accuracy-test-environment-2

Figure 3. Location of planar test surfaces 2, 4, and 5

The test team placed laser scanning reference spheres of 145mm (5.7 inches) diameter around the environment to register the Terrestrial Laser Scans in order to create the reference model. They then created a reference model (Figure 4) using the LAZ output from RiSCAN Pro.  

precision-accuracy-test-environment-3Figure 4. Riegl point cloud of the test area

After creating the reference model, a drone pilot flew the Elios 3 following recommended mapping flight guidelines, starting and ending the flight in the same location. The scan trajectory (i.e., the flight path) can be seen in Figure 5 below. 

The pilot flew one entire loop of the office and an additional small loop where the two corridors meet. After the flight, the LiDAR data collected by the Elios 3 was processed using GeoSLAM Connect v2.1.0, filtered to remove outliers, and exported in the LAZ file format (Figure 6). 

precision-accuracy-flight pathFigure 5. Elios 3 accuracy test flight path.     Figure 6. Elios 3 point cloud of the test                                                                                                   area processed by GeoSLAM Connect

System Precision and Local Accuracy Test Results

Here are the test results for local accuracy and system precision.

Assessing Local Accuracy

Experts assessed the local accuracy of the Elios 3’s LiDAR data using  Plane-to-Plane analysis. 

The Normal Distances between the planes in the reference model and the planes from the model made by Elios 3 data processed with GeoSLAM Connect appear below in Table 1. 

The results show that all the comparisons fall within +/- 16mm (.63 inches), and the Mean Absolute Normal Distance between the Elios 3/GeoSLAM Connect model and the reference model was 8mm (.32 inches).

Name 

Normal Distance

Plane 1 

6 mm (.24 inches)

Plane 2 

0 mm

Plane 3 

-16 mm (-.63 inches)

Plane 4 

-10 mm (-.39 inches)

Plane 5 

-13 mm (-.51 inches)

Plane 6 

-3 mm (-.19 inches)

Mean—Absolute Normal Distance 

8 mm (.32 inches)

Table 1. Local plane-to-plane accuracy

Assessing System Precision

The results of Range Noise Analysis calculated using the Standard Deviation of the comparison planes in the Elios 3 data are shown below in Table 2. 

The results of the Elios 3 precision analysis show the standard deviation of all the planes falls within 15mm (.59 inches) and the Mean Standard Deviation between the Elios 3/GeoSLAM Connect model and the reference model was 8mm (.32 inches) to 1-sigma. 

Name 

Standard Deviation

Test 1 

7 mm (.28 inches)

Test 2 

7 mm (.28 inches)

Test 3 

8 mm (.32 inches)

Test 4 

6 mm (.24 inches)

Test 5 

6 mm (.24 inches)

Test 6 

15 mm (.59 inches)

Mean Standard Deviation 

8 mm (.32 inches)

Table 2. Range noise precision

Conclusion

Accuracy tests were carried out in a standard office environment with planar surfaces using a mobile mapping system, the Elios 3 with GeoSLAM Connect. 

The data was compared against a reference model created by an industry standard Terrestrial Laser Scanner (TLS), the Riegl VZ-400. The Elios 3 point cloud data was processed with GeoSLAM Connect v2.1.0. and the Riegl reference data was processed with RiScan Pro 2.14.1. Alignment and Accuracy computations were calculated with PolyWorks MRS2019 IR3.

Plane-to-Plane analysis showing local accuracy output a mean Normal Distance of 8mm (.32 inches) between the planes in the reference model and the Elios 3/GeoSLAM Connect model.

When assessing system precision, the Range Noise results showed a mean Standard Deviation of 8mm (.32 inches) to 1-sigma.

The results suggest that the Elios 3 with GeoSLAM Connect compares well against a traditional TLS and with the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems in the market.

SEE ALSO

Articles

Introduction Over the last few years, LiDAR data has quickly become one of the most reliable foundations for creating precise and accurate 3D models.

Articles

Our webinar with GeoSLAM had so many questions that we couldn't answer all of them live. Here are the answers to those questions we didn't get to during […]

Articles

Results suggest that the global accuracy of Elios 3’s point clouds when processed with GeoSLAM Connect compare well against a traditional TLS and reach a […]

Articles LiDAR

LiDAR data collected by drone in confined spaces is helping inspectors revolutionize the way they work—here are seven key benefits they're realizing.

Articles Nuclear

The Idaho Environmental Coalition under contract with the Department of Energy exposed the Elios 3 to 10,000 R/h of radiation. Here’s what happened.

Articles Nuclear

The Idaho Environmental Coalition (IEC) under contract with the Department of Energy needed a way to create 3D maps inside a radioactive waste storage […]

Articles Mining

The ability to 3D map confined spaces with the Elios 3 is changing the way inspectors work—here are several examples from the field illustrating real uses.

Articles

The first time I saw the Elios 3 fly was in the deepest mine in the world.

Articles Cement

Holcim Ltd. is the global leader in building materials, from cement, concrete, and aggregates, to roofing materials.With the global headquarters in Zug, […]

Articles Intrinsically Safe Oil & Gas

The ATEX directives are a pair of directives from the European Committee for Standardization created to protect people from ATmosphères […]

Articles

Remote visual inspections are leading the way toward safer, more efficient, and more cost-effective inspections. In addition to being required by […]

Articles Intrinsically Safe Oil & Gas

If the term “intrinsically safe” (IS) is new to you, it’s important to understand how it relates to devices used in hazardous locations and classified […]

Articles Nuclear

For years, robots have been helping nuclear personnel meet ALARA requirements by reducing exposure to radiation—here's how indoor drones can help.

Articles Power Generation Nuclear

Professionals in the power generation industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Pharmaceuticals

Professionals in the pharmaceuticals industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Chemicals

Professionals in the chemicals industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Oil & Gas

Professionals in the oil and gas industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Chemicals

Drones present several benefits as a tool for remote visual data collection in internal inspections for the chemicals industry—here's our list of the top […]

Articles Pharmaceuticals

Drones present several benefits as a tool for remote visual data collection in internal inspections for the pharmaceutical industry—here's our list of the […]

Articles Oil & Gas

Drones present several benefits as a tool for remote visual data collection in internal inspections for the oil and gas industry—here's our list of the […]