How is vision AI improving European transport safety ?

Within the Horizon 2020 framework – a EU research and innovation program with a budget of nearly €80 billion – the European Union has been funding and developing a research project for road bridges and railways inspection in Europe called Drones4Safety. Nine members have been involved in the research consortium, including universities, research institutes and companies like Alteia.

 

The project consists of a mix of hardware and software development with the goal of operating autonomous drone systems for accurate and frequent inspections of critical infrastructure. This new autonomous inspection method will be made available to rail and bridge operators so that they can remotely monitor the condition of their infrastructure over time and alert them if a defect is detected. From a data perspective, the project leverages satellite and open-source maps to identify areas of interest, and includes the development of artificial intelligence (AI) algorithms to automate the process of image analysis.

 

The European Commission launched this project following the tragedy that happened in Genoa (Italy) on 14 august 2018, when a 200-meter section of one of the longest concrete bridges in the world collapsed, killing 43 people. Morandi Bridge was known to be stronger and lighter than any other bridge of its era and was a symbol of Italian engineering. According to the Guardian, the collapsing of the bridge might be due to carelessness and structural problems.

 

Following this event and to avoid future tragedies, the EU launched the Drones4Safety initiative. The idea was to gather industrial, research and academic experts in infrastructure inspection, artificial intelligence, connectivity, and drone technology to help increase the safety of the European civil transport system by being able to inspect the whole European electrical railway system and a big portion of the European road bridges with drones in a repetitive and autonomous manner.

 

Within the project scope, Alteia’s role was to provide its software platform to visualize, process and analyze the data collected by drones, and develop from scratch AI-based algorithms to detect certain types of defects on bridges and rails.

 

In this article, you will gain a better understanding of the capabilities the team has developed for infrastructure inspection and how they fit into this project.

A cloud-based platform for data visualization, contextualization, and processing at scale

 

At Alteia, we are building a software platform to ingest, process and analyze data collected by any visual data source, including drones. The process is as follows:

 

• Create a digital twin of any infrastructure (system representation)
• Configure automated data ingestion pipelines in order to aggregate, contextualize data and update the current state of the system
• Visualize, analyze and predict the state of the system over time
  (notably to detect anomalies, defects or risks)
• Create reports with actionable information that integrate easily into a business workflow
• Organize the set of actions to be taken and allocate it to the relevant teams

How does it work in the context of Drones4Safety?

 

A swarm of drones is deployed over a critical infrastructure to collect a set of raw images. Data collection is very fast because several drones can be operated in parallel.

 

The collected images are uploaded and ingested into the Alteia platform in real-time via SDK. 

 

The images are then processed to create a digital twin of the inspected structure in 3D using integrated photogrammetry tools. The digital twin is an additional support to understand the position and orientation of the images taken, and allows reprojecting the detected anomalies to their physical location on the infrastructure.

 

After building the digital twin, AI based algorithms for equipment and defect detection are launched. Images are segmented automatically and if some anomalies are  found they are automatically highlighted with an annotation (cf image below). 

 

Alteia provides feedback loop workflows for incremental learning. A feedback loop, or closed-loop learning, describes the process of leveraging the output of an AI system and corresponding end-user actions to retrain and improve models over time. The AI-generated outputs (predictions or recommendations) are compared against the final decision (for example, to perform work or not) and provide feedback to the model, allowing it to learn from its mistakes. 

 

Thanks to the previous contextualization step, each defect becomes a vector with GPS coordinates that can be exported to 3rd party tools. All defects are then merged into a file, in which duplicates are removed. This is the data input for the following step.  

 

Drones4Safety’s partner EUCENTRE then performs a lifecycle analysis from that data input. They leverage the information contained into the generated file to make a structural analysis and estimate the residual life of the structure.

Building AI algorithms for detecting defects on railways and bridges

To increase the efficiency of the inspection process and to ensure its scalability, Alteia’s role was to develop AI based algorithms trained to detect different classes of equipment and related defects. These analytics are now available on the platform. How did we get there?

 

1. Supervised learning for developing algorithms from scratch

To build the defect detection algorithms, Alteia managed the development of the AI models.

 

The first step was to prepare a training data set. Given the nature of what we were looking for, the data was not available in large quantities. In this context, synthetic data is a viable solution. To find anomalies, we had to use gaming tools to build numerical bridges and simulate defects with well-defined rules. Once you have created a virtual environment, you can add new textures, contrast or brightness conditions to enhance and maximize your training data set. And with synthetic data, you always have pixel-perfect labels, which is especially important for segmentation applications.

 

Using a synthetic dataset and fitting the models, we obtained a recall rate of more than 80%. The “recall” or “sensitivity” is the proportion of relevant features found among all relevant features. In the case of searching for rare anomalies on bridges and rails, taking the risk of losing anomalies could be dramatic, so it is better to maximize defect detection and thus have the highest recall rate possible.

 

 2. Defects on railways

To develop our algorithms for bridges and railroads, we have first trained a segmentation model for automated equipment detection. For railroads, we focused on several ground and air parts:

 

• Tracks: consists of two parallel steel rails, anchored perpendicular to members called ties (sleepers) of timber, concrete, steel, or plastic to maintain a consistent distance apart, or rail gauge
• Ballasts: the rails and ties are usually placed on a foundation made of compressed earth on top of which is placed a bed of ballast to distribute the load from the ties and to prevent the track from buckling as the ground settles over time under the weight of the vehicles passing above
• Ties/sleepers: sleepers are elements anchored to rails to maintain a consistent distance apart between them or rail gauge
• Catenary masts and mast equipment
• Cables
• Signaling equipment

 

Then we have performed image classification at equipment level:

 

• the problem is rescoped to be managed at the equipment level rather than on the full image
• the model classifies each equipment in a binary manner (ok/ko) according to the anomaly classes defined for that specific equipment

For example on sleepers, cracks or splits can occur. In that case they have to be replaced with new wooden ties or concrete substitutes.

 

Cracks can also appear on rails as a separation of metal extending partially, but not completely, through the rail section.

 

Regarding catenary masts, the surface of the pole can be covered by rust, commonly caused by degradation. Mast equipment can also be broken when one or more discs of the insulator string connecting the mast with the tie or with the bracket are broken. 

 

Finally, the cables may be broken, missing or stranded in the event that one of the cable wires is broken.

 

3. Defects on bridges

For bridges, we trained our models to identify three main types of anomalies shared between three types of components :

 

• Cracks
• Spalling
• Deformations

Cracks

 

The first common defect on bridges is any visible fracture or crack on the concrete structure. Concrete cracks can occur due to several factors, like drying shrinkage, thermal expansion, chemical reactions, corrosion of the reinforcement and applied external loads (for example shear or flexural cracks).  

 

Spalling

 

Concrete spalling and concrete delamination are generally recognized as the most serious and troublesome kinds of bridge deck deterioration. It happens when a portion of concrete is detached from the main structural element, leaving the steel reinforcement exposed to the environment. Generally, spalling results primarily from corrosion forces that radiate from the embedded steel. The forces normally produce cracks in the deck surface, which spall and ultimately expose the reinforcement. The spalling reduces the riding quality of the bridge and may have an effect on the structural integrity of the structure.

 

Deformations

 

Deformations can be observed in steel reinforcement, strand anchors and guardrails. The deformations can be due to large displacements or deflection of the structure. They can be triggered by performing time series analysis and 3D comparison with design files or previous acquisitions.

 

Summary

 

Drones4Safety (D4S) is a research project aimed at ensuring European transportation safety using a combination of autonomous drones and image processing accelerated by artificial intelligence.

 

Alteia’s platform, Aether, was adopted to manage the data for the Drones4safety project, and was used to develop the workflow for data ingestion, processing and analysis.

 

Alteia’s data science team worked on developing AI models for equipment identification and fault detection. Using a mix of real and synthetic data, the team achieved excellent operational performance on the models.

 

Finally, Alteia delivered a dedicated user interface to validate or invalidate the predictions made by the models, and as such generate reports that are fully reviewed and validated by inspectors – which contributes also to the continuous learning of the models.

Click here if you want to know more about Drones4Safety.

 

If you have a business case in mind, get in touch with our project team to define the outline of your project and eventually build a POC with us. You can also visit our website.

 

Sources :

 

• https://www.researchgate.net/publication/328533148_Analysis_on_the_Causes_of_Cracks_in_Bridges
• https://theconstructor.org/practical-guide/spalling-concrete-causes-repair/26027/
• https://drive.google.com/file/d/1CDLr_hMYmb4Ad67uXogScQP2oxC-kUrc/view?ts=641d5912
• https://www.sciencedirect.com/science/article/abs/pii/004316489190003D
• https://www.rta.org/assets/docs/ComprehensiveRail/sec2/separating%20shelling%20or%20spalling.pdf
• https://drones4safety.eu/