Testing Global Accuracy and Georeferenced Accuracy in 3D Mapping with the Elios 3 and GeoSLAM Connect

Results suggest that the global accuracy of Elios 3’s point clouds when processed with GeoSLAM Connect compare well against a traditional TLS and reach a similar level of accuracy to the accuracy obtained with the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems in the market.

The focus of this article is on global accuracy and georeferenced accuracy as tested in a large industrial warehouse. View the results of testing we did for system precision and local accuracy as tested in an office space in this article.

Note: The first two sections of this article (the Introduction and Why We Performed These Tests) are the same in this article and the one mentioned above. If you've read the other white paper, we recommend you begin reading at the Defining Our Terms section below.

test results OVERVIEW

  • What was tested. System global accuracy and georeferenced accuracy of 3D models of the LiDAR data collected by Flyability’s Elios 3 processed with GeoSLAM Connect.
  • Who did the testing. GeoSLAM 3D mapping experts and members of the Flyability product team.
  • What tests were conducted. Three scans were captured with Elios 3 of the same large underground facility of 70 meters x 40 meters and processed with GeoSLAM Connect. 15 targets were set-up in the facility and a centroid extraction script was used for accuracy evaluations. 
  • Reference Model. The Reference Model used for the tests was made with a TLS (Terrestrial Laser Scans) Riegl  VZ-400, and the registration process was undertaken using RiScan Pro V2.14.1.
  • Test results—global accuracy. The RMSE of target-to-target distances in the three scans vs. the reference model was 35mm with an MAE as a percentage of distance of 0.16%.
  • Test results—global georeferenced accuracy (cloud-to-cloud alignment). The average RMSE of targets in the three scans vs. the reference model was 110mm, when the scans were aligned using only one corner of the reference model.
  • Test results—global georeferenced accuracy (target based alignment). The average RMSE of targets in the three scans vs. the reference model was 65mm, when the scans were aligned with 4 of the targets.

Introduction

Over the last few years, LiDAR data has quickly become one of the most reliable foundations for creating precise and accurate 3D models.

Sectors like mining, construction, and infrastructure are using these models to conduct routine inspections, make safety determinations, track the change of assets over time, and support project planning. 

The types of outputs professionals in these industries are getting from 3D models made with LiDAR data include detailed digital twins, accurate 2D and 3D measurements, the ability to pinpoint the locations of defects within assets, the ability to export data to common 3D point cloud file extensions like *.e57, *.las, *.laz, and *.ply, and the ability to merge multiple georeferenced 3D models to track changes in assets over time.

Regardless of the industry or output, the quality of the model is key to its usefulness. If the data is not precise and accurate—terms that have specific definitions in 3D modeling, which will be covered in a separate section below—then it may not represent the real world well enough to offer valuable insights.

This article covers findings from tests performed by experts at GeoSLAM and the Flyability product team that highlight findings from 3D models created using the Elios 3 with GeoSLAM Connect to identify global accuracy and georeferenced accuracy as compared to the ZEB Revo and ZEB Horizon, which are leading mobile mapping systems in the market.

Why We Performed These Tests 

Flyability’s Elios 3 comes with Ouster’s OS0-32 LiDAR sensor and the ability to perform SLAM (simultaneous localization and mapping), which means that it can create 3D models in real time, while in flight. 

After the flight, Elios 3 users can process the LiDAR data they collected with GeoSLAM Connect to create precise, accurate 3D models. 

The 3D Live Model and the post-processed model have distinct uses, and should not be seen as the same kinds of 3D models. While the 3D Live Model can be used during a mission for navigation, route planning, and verifying scan coverage, the post-processed model you make with GeoSLAM Connect can provide an accurate point cloud.

Processing Elios 3’s LiDAR data with GeoSLAM Connect adds the ability to create precise, accurate 3D mapping to Flyability’s field-tested collision tolerance technology for work in confined spaces, allowing inaccessible environments to be mapped effectively.

But potential Elios 3 users may have questions about what happens when Flyability’s indoor drone technology is fitted with a LiDAR payload and that data is post-processed, such as: 

  • Will the drone’s vibrations or environmental factors like dust or moisture impact the precision of the resulting 3D models? 
  • What will the resulting 3D models look like, and how useful will they be?

To address these questions, we performed a thorough analysis of global accuracy and georeferenced accuracy with the Elios 3’s point clouds when processed with GeoSLAM Connect. All the tests were conducted in a manner that would be both representative and repeatable. 

Keep reading to learn the results.

Defining Our Terms: Global Accuracy, Georeferenced Accuracy, and Drift in 3D Mapping 

Global accuracy in 3D modeling relates to the distance between two points in a point cloud, where the object cannot be viewed from a single position (e.g., the distance between two rooms). 

Georeferenced accuracy is global accuracy plus inaccuracies caused by the alignment method. Alignment methods include target based registration and Iterative Closest Point (ICP).

Drift is a term used in 3D modeling to describe global accuracy. Global accuracy cannot easily be expressed in absolute values (e.g., +-2cm/.79 in) because, in the absence of ground control points or GNSS, the absolute error usually grows along with the size of the asset, or the distance that is measured. 

For example, the error on a 30m (98.4 feet) measurement is likely to be smaller than the error on a 300m (984 feet) distance measurement because a mobile scanner moving through an asset will accumulate errors on top of previous errors the longer it travels and records data. This accumulation of errors over distance is described by the word drift, representing a percentage of the traveled distance during data collection—for example, a 1% drift on a 300m (984 feet) distance corresponds to a 3m (9.84 feet) error as compared to reality.

Global Accuracy and Georeferenced Accuracy Assessments with the Elios 3

To evaluate the global accuracy and georeferenced accuracy of the Elios 3’s point clouds processed with GeoSLAM Connect, GeoSLAM 3D mapping experts and the Flyability product team carried out tests within an industrial factory.

Keep reading to get the details of the tests, and to learn the results.

Establishing a Control

When assessing the accuracy of any system, a second measurement system must be used to provide the benchmark value (i.e., the control) and this system must be of greater accuracy than the system being tested. 

To test the accuracy of a mobile mapping solution like the Elios 3, the industry standard is to use either a Total Station (TPS) or a Terrestrial Laser Scanner (TLS) as a control, because the accuracy for both of these approaches exceeds that of a mobile mapping solution.

The reason the TPS and TLS approaches obtain greater accuracy than a mobile mapping solution is because they each capture data from a single, stationary position, with multiple positions registered together using point matching algorithms.

In comparison, a mobile mapping solution like the Elios 3 moves continuously as it collects data, capturing data at multiple positions as the drone moves through the environment it is mapping.

The global accuracy and georeferenced accuracy tests we performed relied on data collected by a Terrestrial Laser Scanner (TLS) as their control. It is to be noted that the control took more than six hours of time to acquire the data, due to the size of the asset. In comparison, the Elios 3 took 8.5 minutes to acquire the data.

Test Environment 

To evaluate the global accuracy and georeferenced accuracy of the Elios 3’s point clouds processed with GeoSLAM Connect, the LiDAR data was captured in a factory named the Blue Factory, located in Fribourg, Switzerland.

The factory consists of 12 rooms of varying sizes separated by several doorways, presenting a representative industrial environment suitable for the Elios 3. 

global-accuracy-test-environment

A room from the test environment

global-accuracy-2

Figure 1. An overview of the entire test environment

15x Retroreflective Diamond Grade 3M (9.84 feet) targets (200mm x 200mm / 7.87 in x 7.87 in) were placed evenly around the test environment, as shown in the next two pictures below:

global-accuracy-3

global-accuracy-5

Collecting the Data

Three scans were carried out with the Elios 3 to capture LiDAR data for testing, all following the same approximate flight path to ensure consistency between the results. 

All the scans started and ended in the same location, both for consistency and to ensure that the data capture loop was closed, as recommended. Best practice for SLAM data capture was maintained by performing loops within the capture and entering doorways sideways to ensure good visualization when moving into new environments.

The three scans had an average flight time of 8 minutes and 30 seconds with 108 million points captured per scan over the ~450 meters (1,476 feet) flight path. 

Flight_PathFigure 2. The path the drone followed in the test environment

Watch this video to see an accelerated recording of the drone's flight path:

HubSpot Video

Data Processing & Centroid Extraction

To ensure that the test was representative of what an end user can expect from the system, the processing was carried out using the standard Flyability processing parameters found in GeoSLAM Connect 2.1.1.  The data was not reprocessed by any other means, neither was it decimated nor filtered.

As a post-processing step, an extraction tool* was run to identify the 15x targets in both the Elios 3 data and the TLS data. Once the targets were identified, the tool extracted the centroids of the targets to provide 15x centroids for both the Elios 3 and the TLS data. The centroids of the TLS data were used as Control Points and the Elios 3 centroids were used for comparison. 

*Note: This extraction tool will be available in GeoSLAM Connect 2.3.0.

Assessing Global Accuracy—Distance Measurements

To assess the global accuracy of the Elios 3, distance measurements were carried out and the Elios 3 data was compared against the TLS control data.  

The steps carried out to complete the process were:

  1. Distance measurements. The distance between a dispersed array of pairs of centroids was measured for both the control TLS data and the Elios 3 scans .
  2. Find residuals. Residuals were found between the point pair distance of TLS data and the Elios 3 point pair distances.
  3. Find RMSE. The RMSE (root mean square error) of the residuals was calculated for each distance from the residuals of the 3 scans.
  4. Find RMSE as a percentage of length. The RMSE of the residuals as a percentage of distance between centroids was computed.
  5. Find the mean error. Finally, the average of the RMSE (MAE) of the residuals as a percentage of distance between centroids was computed.

The centroid pairs used to find the residuals and in turn the RMSE, RMSE as a percentage of length, and the MAE are shown in Figure 3 below.

global-accuracy-7Figure 3. Point pairs used for global accuracy assessment

Global Accuracy Results

The results from the global accuracy assessment of the Elios 3 data when processed using GeoSLAM Connect provided a global accuracy RMSE of 35mm (1.38 inches), or a drift of 0.16% (mean error with regard to the distance being measured).

 

 

Residual dXYZ

     

Tie Line

Scan 1

Scan 2

Scan 3

RMSE

Length (VZ400)

RMSE % Length

TP01-TP08

0.0005

0.0119

0.0326

0.0200

52.60

0.0381

TP02-TP07

0.0066

0.0071

0.0060

0.0066

41.11

0.0160

TP04-TP08

0.0284

0.0183

0.0164

0.0217

57.15

0.0379

TP05-TP14

0.0351

0.0292

0.0079

0.0268

19.69

0.1359

TP07-TP09

0.0661

0.0496

0.0992

0.0746

26.73

0.2789

TP11-TP12

0.0259

0.0540

0.0571

0.0478

19.88

0.2403

TP14-TP15

0.0558

0.0517

0.0483

0.0520

13.34

0.3898

 

 

 

RMSE

0.0356

MAE (%)

0.1624

 

Assessing Georeferenced Accuracy—Cloud-to-Cloud Alignment around Take-Off

In the previous section we discussed the global accuracy of the system. In this section we will assess the first registration method—cloud-to-cloud alignment around the take-off location.

Georeferenced accuracy assesses the global accuracy of the system as well as the accuracy of the georeferencing technique used.

To simulate a common use case for Elios 3 scans, the Elios 3 point cloud was aligned to the reference model around the take-off location. Doing this simulates the procedure that might be followed during a mission in an inaccessible area, in which Elios 3 is used to complete an existing georeferenced model and the control can only be placed at the flight’s starting location. 

Only having control in one section of the scan environment causes any inaccuracies in the registration process to propagate throughout the scan and will result in increased inaccuracies as compared to using targets or ground control points (GCPs) across the whole scan.

global-accuracy-8

Figure 4. TLS reference model with the 15m (49.2 feet) section of the Elios 3 scan used for cloud-to-cloud alignment overlaid

To assess the cloud-to-cloud alignment accuracy of the Elios 3 using GeoSLAM Connect, the following workflow was implemented:

  1. A 15 meter (49.2 feet) section of the scan around the take-off location was used to perform cloud-to-cloud registration.
  2. The 15 meter section of the TLS data was used as the reference and the 15m section of the Elios 3 data was aligned to the reference using cloud-to-cloud alignment. This alignment was then applied on the entire Elios 3 point cloud.
  3. The reference centroids from the TLS data were recorded and compared to the aligned Elios 3 centroids. 
  4. The residuals between the reference centroids and the aligned centroids was calculated. 
  5. The RMSE for both dXY and dXYZ for the 3x scans was computed for each reference point. 
  6. The average RMSE values for both dXY and dXYZ were output. 

Results

The results from assessing cloud-to-cloud alignment and global accuracy were split to show the RMSE for dXY and dXYZ. The results show that only using a section of the scan for registration and not the whole scan will increase inaccuracies. 

The dXY cloud-to-cloud results can be seen here:

 

 

dXY Results

     

Target

Scan 1

Scan 2

Scan 3

RMSE

 

 

TP001

0.0213

0.0344

0.0444

0.0347

 

 

TP002

0.0338

0.0154

0.0315

0.028

 

 

TP003

0.0081

0.0059

0.0060

0.0068

 

 

TP004

0.0195

0.0386

0.0372

0.0329

Distance

Drift

TP005

0.0310

0.0310

0.0617

0.0418

45

0.09%

TP006

0.0182

0.0751

0.0488

0.0528

56

0.09%

TP007

0.0186

0.0214

0.0256

0.0221

50

.04%

TP008

 

0.0258

0.0317

0.0373

0.0319

68

.05%

TP009

0.0666

0.1057

0.1100

0.0961

88

.11%

TP010

0.0658

0.0969

0.1526

0.1110

73

.15%

TP011

0.0867

0.0210

0.0560

0.0608

69

.09%

TP012

0.0541

0.0558

0.0901

0.0687

57

.12%

TP013

0.0282

0.0385

0.0539

0.0416

50

.08%

TP014

0.0425

0.0640

0.0591

0.0560

35

.16%

TP015

0.0159

0.0098

0.0071

0.0115

30

.04%

 

 

 

MEAN RMSE

0.0543

DRIFT

0.09%

 

The dXYZ cloud-to-cloud results can be seen here:

 

 

dXYZ Results

     

Target

Scan 1

Scan 2

Scan 3

RMSE

 

 

TP001

0.0266

0.0395

0.0488

0.0394

 

 

TP002

0.0526

0.0228

0.0438

0.0416

 

 

TP003

0.0148

0.0191

0.0064

0.0145

 

 

TP004

0.0260

0.0396

0.0396

0.0357

Distance

Drift

TP005

0.0627

0.0460

0.0640

0.0582

45

0.13%

TP006

0.0212

0.0769

0.0490

0.0541

56

0.10%

TP007

0.0393

0.0215

0.0367

0.0335

50

.07%

TP008

 

0.1368

0.1256

0.2081

0.1610

68

.24%

TP009

0.1475

0.2231

0.2247

0.2017

88

.23%

TP010

0.1825

0.1440

0.2170

0.1836

73

.25%

TP011

0.1786

0.1172

0.1609

0.1544

69

.22%

TP012

0.1648

0.1382

0.1975

0.1686

57

.30%

TP013

0.0951

0.0938

0.1092

0.0996

50

.20%

TP014

0.0847

0.0736

0.0923

0.0839

35

.24%

TP015

0.0277

0.0139

0.0433

0.0307

30

.10%

 

 

 

MEAN RMSE

0.1104

DRIFT

0.19%

Assessing Georeferenced Registered Accuracy—Target Based Alignment with Four Model-Wide GCPs

To simulate an alternate use case of referencing the Elios 3 point cloud to a set of ground control points (GCPs), target based alignment was undertaken. The alignment performed was a rigid transformation. 

Assessing the target based accuracy will include both the accuracy of the system and the accuracy of the target based registration process. 

This approach simulates a use case in which Elios 3 is used to scan an environment that comprises known features or control points in various locations beyond only the take-off location.

To undertake the alignment process, the following methodology was applied:  

  1. Four targets were chosen around the periphery of the scan—one target each from the North, South, East, and West of the dataset. Target control points 3,6,11, and 14 were used (see Figure 5 below).  
  2. The centroids of the four TLS targets were used as reference points. 
  3. The centroids of the same four targets from the Elios 3 scans were aligned to the four reference points.
  4. The dXY and dXYZ residuals between the TLS centroids and the Elios 3 centroids were calculated for all 15x targets. 
  5. The RMSE per target for all the scans was computed. 
  6. RMSE were computed for each scan and an overall average RMSE value for target based alignment accuracy was calculated for both dXY and dXYZ.

global-accuracy-9

Figure 5. Four targets around the periphery of the scan

Results

Due to the control being located around the point cloud (NSEW), not only at the flight start location, there were tighter constraints applied to the registration process.  

These constraints helped produce improved accuracy of the registration process in comparison to the cloud-to-cloud alignment process, in which only a 15 meter (49.2 feet) section of the scan around the start location was used to align the point clouds. In this scenario, a user has the ability to place control at varying positions around the scan. 

Using target based registration from the four points around the periphery of the scan led to a 28% improvement in accuracy of the registered point cloud, as compared to the cloud-to-cloud referencing process using only a 15 meter section around the flight start location.  

It should be noted that an even more accurate model would be obtained if the GCPs were used in the SLAM processing. This capability is planned to be integrated for future releases of GeoSLAM Connect. 

The results from assessing the target based alignment were split to show both the RMSE for dXY and dXYZ. The tables below show that the RMSE for dXY were lower than the dXYZ results, as expected.

The dXY target based results can be seen here:

 

 

dXY Results

 

Target

Scan 1

Scan 2

Scan 3

RMSE

TP001

0.0742

0.0523

0.0549

0.0612

TP002

0.0959

0.0293

0.0326

0.0609

TP003

0.0757

0.0215

0.0167

0.0465

TP004

0.0612

0.0158

0.0258

0.0394

TP005

0.0392

0.0142

0.0525

0.0387

TP006

0.0307

0.0537

0.0452

0.0442

TP007

0.0194

0.0079

0.0236

0.0182

TP008

 

0.0163

0.0560

0.0155

0.0348

TP009

0.0447

0.1105

0.1294

0.1016

TP010

0.0608

0.1199

0.1327

0.1091

TP011

0.0578

0.0248

0.0583

0.0495

TP012

0.0056

0.0682

0.0684

0.0559

TP013

0.0227

0.0349

0.0334

0.0308

TP014

0.0583

0.0621

0.0425

0.0550

TP015

0.0598

0.0122

0.0149

0.0363

 

 

 

MEAN RMSE

0.0573

 

The dXYZ target based results can be seen here:

 

 

dXYZ Results

 

Target

Scan 1

Scan 2

Scan 3

RMSE

TP001

0.0820

0.0523

0.0562

0.0649

TP002

0.0962

0.0304

0.0460

0.0640

TP003

0.0908

0.0216

0.0185

0.0549

TP004

0.0827

0.0305

0.0280

0.0534

TP005

0.0727

0.0302

0.0599

0.0571

TP006

0.0308

0.0538

0.0454

0.0444

TP007

0.0234

0.0258

0.0299

0.0265

TP008

 

0.0480

0.0992

0.1391

0.1025

TP009

0.0593

0.1365

0.1392

0.1176

TP010

0.0610

0.1202

0.1333

0.1094

TP011

0.0579

0.0251

0.0598

0.0502

TP012

0.0212

0.0742

0.0685

0.0596

TP013

0.0228

0.0437

0.0357

0.0351

TP014

0.0583

0.0697

0.0454

0.0587

TP015

0.0764

0.0925

0.0570

0.0767

 

 

 

MEAN RMSE

0.0698

 

 

 

MAE

0.0650

Conclusion

The test results show that the Elios 3’s point clouds processed with GeoSLAM Connect produce high accuracy and present an appropriate tool to meet survey requirements, with a global system accuracy of 35mm (1.38 inches).

In addition to the system accuracy, the georeferencing capabilities of the Elios 3 were assessed simulating on site practices.  

The Cloud-to-Cloud and Target Based Alignment assessments show how the Elios 3 can easily be implemented in both map and georeference-inaccessible environments and achieve georeferenced accuracies up to 60mm (2.36 inches).

 

SEE ALSO

Articles

What does the landscape look like for drones in South African mining? Read this interview with Hendrik Grobler, Head of the Department of Mining […]

Articles

Our webinar with GeoSLAM had so many questions that we couldn't answer all of them live. Here are the answers to those questions we didn't get to during […]

Articles LiDAR

Results suggest that the precision and local accuracy of Elios 3’s point clouds when processed with GeoSLAM Connect compare well against a traditional TLS […]

Articles LiDAR

LiDAR data collected by drone in confined spaces is helping inspectors revolutionize the way they work—here are seven key benefits they're realizing.

Articles Power Generation

The Idaho Environmental Coalition under contract with the Department of Energy exposed the Elios 3 to 10,000 R/h of radiation. Here’s what happened.

Articles Power Generation

The Idaho Environmental Coalition (IEC) under contract with the Department of Energy needed a way to create 3D maps inside a radioactive waste storage […]

Articles Mining

The ability to 3D map confined spaces with the Elios 3 is changing the way inspectors work—here are several examples from the field illustrating real uses.

Articles

The first time I saw the Elios 3 fly was in the deepest mine in the world. 

Articles Cement

Holcim Ltd. is the global leader in building materials, from cement, concrete, and aggregates, to roofing materials.With the global headquarters in Zug, […]

Articles Intrinsically Safe Oil & Gas

The ATEX directives are a pair of directives from the European Committee for Standardization created to protect people from ATmosphères […]

Articles

Remote visual inspections are leading the way toward safer, more efficient, and more cost-effective inspections. In addition to being required by […]

Articles Intrinsically Safe Oil & Gas

If the term “intrinsically safe” (IS) is new to you, it’s important to understand how it relates to devices used in hazardous locations and classified […]

Articles Power Generation

For years, robots have been helping nuclear personnel meet ALARA requirements by reducing exposure to radiation—here's how indoor drones can help.

Articles Power Generation

Professionals in the power generation industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Pharmaceuticals

Professionals in the pharmaceuticals industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Chemicals

Professionals in the chemicals industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Oil & Gas

Professionals in the oil and gas industry are using indoor drones in a variety of ways—read this article to get the details.

Articles Chemicals

Drones present several benefits as a tool for remote visual data collection in internal inspections for the chemicals industry—here's our list of the top […]

Articles Pharmaceuticals

Drones present several benefits as a tool for remote visual data collection in internal inspections for the pharmaceutical industry—here's our list of the […]

Articles Oil & Gas

Drones present several benefits as a tool for remote visual data collection in internal inspections for the oil and gas industry—here's our list of the […]