Point cloud classification and machine learning : an introduction to practical uses in vision AI

This article is about three fundamentals of visual AI: point clouds, point cloud classification, and machine learning applied to point cloud data.

You’ll learn about the basics of these topics:

• How LiDAR scanning technology generates point clouds.
• How supervised, unsupervised, and semi-supervised machine learning work.
• How machine learning automates the process of point cloud classification and analysis.
• How organizations can create 3D shapes from a point cloud and then use the shapes in reconstructing a digital twin in a building information system (BIM).

The purpose of this article, and others to follow, is to provide a practical understanding of these terms and the ideas and processes behind them. The goal is to help you understand both the potential and the limitations of real-world applications of visual AI.

We at Alteia know that visual AI can deliver great value to many businesses today. And we believe it will soon play a huge role in nearly every facet of the business. However, without at least a cursory understanding of the fundamentals, it is easy to misjudge the potential of visual AI. As a result, you may think it is smarter and more capable than it is, with the result that you could form unrealistic expectations for its uses. Or you may think visual AI is less advanced than it is, leading you to underestimate its current potential.

1. What is a point cloud?

A point cloud is a digital representation of 3-dimensional reality. It’s formed by scanning a 3-dimensional scene or object with thousands of beams of laser light, using a technology called LiDAR. LiDAR is an acronym for light detection and ranging.

A high-resolution LiDAR scanner may project millions of laser beams across objects at which it’s aimed. It captures the reflected light and measures the distance between the LiDAR sensor and points on the surface of the object. Then it assigns an X, Y, and Z coordinate to each point.

When you view a point cloud in an appropriate software application, you see an image that looks truly 3D. Depending on the software you use, you may be able to rotate and manipulate the image as if it were floating in three-dimensional space. A point cloud is also called a digital twin because it digitally represents in exact scale the surfaces of the original object.

Many organizations use point clouds for high-value applications. In engineering and construction, companies use them to measure the precise dimensions of large-scale projects. By comparing a succession of point clouds, they can measure the speed of progress on their project. The point cloud in Figure 1 is a graphically enhanced LiDAR scan of an open-pit mine. The colors represent the depth of the steps in the pit.

Figure 1. 3D point cloud of an open-pit mine. This representation shows the mine’s steps or benches with an adjusted color scale, where each color indicates a designated depth.

2. How to generate a point cloud

You can generate point clouds by using either of 2 technologies or methods:

• Laser scanners: These are survey-grade systems that typically include a variety of sensors and cameras to create realistic 3D digital renditions of objects and their environment.
• Photogrammetry: This is the art, science, and technology of recording accurate information about physical objects and the environment. It records, measures, and interprets photographic images created from the frequencies of light that objects reflect in various circumstances. Besides using visible and invisible light frequencies, photogrammetry can also record and interpret patterns of radiant electromagnetic energy.

In general, laser scanners produce more accurate data than photogrammetry. Let’s look at each in more detail.

Laser scanners

The most important laser scanner technology is a LiDAR sensor. It uses rapid laser pulses to gather hundreds of thousands of extremely accurate measurements per second. Unenhanced LiDAR images are monochromatic.

Most laser scanners also include 2 other sensors in addition to LiDAR. They include:

• A camera with red, green, and blue (RGB) sensors. RGB images add color to point clouds.
• An inertial measurement unit (IMU). This is the professional-grade cousin of the accelerometer in your smartphone. It measures the acceleration forces acting on the scanner. Its purpose is to determine the scanner’s position in space and to monitor its movement.

Laser scanners come in a variety of forms, each designed for a specific range of applications. You would use a terrestrial laser scanner (TLS) to generate point clouds of extreme accuracy for specialized applications such as these:

• Measuring beam deflection in mechanical engineering. Beam deflection is the amount by which a structural beam deforms from its original shape under a load.
• Analyzing the flatness of floors or other surfaces, or measuring volumes of materials.
• Capturing a single object with great precision, such as a car or other complex machine.

You can use a mobile laser scanner to capture point clouds as you walk. Such devices are ideal for documenting buildings because they generate fast and accurate scans at a low cost per square meter. Mobile laser scanners are accurate to within 4mm. Their speed makes them a preferred technology for mapping active sites such as factories, which often allow only a brief time to complete a scan.

The market for laser scanners offers devices for a variety of specialized applications. Specialty scanners can capture diverse objects, including large-scale ones such as roads or railways. They can generate wide-area topographic maps. You can use a different scanner for each of several tasks, then join the separate point clouds into one.

Photogrammetry

Unlike laser scanning, photogrammetry is more a methodology than a set of specific tools. It uses cameras to capture a space from different viewpoints. You then process the images with specialized software to reconstruct the space in 3D.

If you use drones to capture a point cloud of a building, chances are good that the technology they use will be photogrammetry rather than laser scanners. That’s because today’s laser scanners are likely to be too heavy for use in small drones.

Point cloud density and dataset coverage

Point cloud density refers to the number of coordinates collected per unit area. When point cloud data is processed and turned into a 3D digital model, these coordinates act like pixels. Point clouds may offer varying levels of density, depending on the type of sensor used and its distance from an object. This impacts also its coverage.

The tiny LiDAR sensor in a Model 13 iPhone is suitable for use in a room or other small environment. It can help create scenes for use in the metaverse. In contrast, you can mount bigger and more accurate LiDAR sensors on aircraft, robots, or stationary locations on the ground. Bigger sensors can scan huge areas with accuracy of a centimeter or less.

3. What is point cloud classification?

Point clouds can contain a wealth of spatial information that expresses the shape, size, position, and direction of an object. But point clouds do not distinguish among various objects within scanned areas. If you use your iPhone to scan a room that contains a table, the point cloud does not “know” the difference between points that define the room and shape of the table within it. The cloud treats all points as if they are part of the same object.

To “see” and “understand” the differences between the table and the room, the point cloud needs help from a process called classification. Point cloud classification is the task of assigning predefined category labels to various groups of points within a point cloud. To put it another way, classification defines or interprets subsets of points in a cloud. It determines which points belong to which objects. In the example of a table in a room, classification defines appropriate points as table and others as room.

With the objects in a point cloud properly classified, we can digitally manipulate them in ways that provide useful insights. We can add or subtract objects from our view. Let’s look at a high-value commercial application of classification in the Power & Utilities industry.

Electric utilities use LiDAR point clouds to spot where the growth of vegetation puts power lines at risk. Without proper control of such growth, utilities risk wildfires and service outages. Point clouds offer an efficient and accurate way to measure the distances between power lines and nearby trees. Utilities can identify areas of high risk and then direct tree-trimming crews to cut back foliage. Figure 2 shows a point cloud made for LiDAR scans by aircraft flying over electrical power lines in a wooded area.

Figure 2. Point cloud classification enables power utilities to measure the risks of vegetation encroachment. Through point cloud classification, electrical utilities can distinguish among buildings, vegetation, ground, poles, and conductors. The analysis tells them where vegetation is growing too close for the safe operation of power lines.

4. What is machine learning, and how does it apply to point cloud classification?

Machine learning is a form of AI that uses data and algorithms to imitate the way humans learn. Through a process of training, ML gradually improves its accuracy. In conventional data processing, software developers use programming language to communicate instructions to a computer. Computer programs must be semantically and syntactically exact. That’s because most computers today cannot infer a programmer’s intent. They follow code by rote, exactly as the programmer wrote it.

In contrast, ML provides a way for computers to accomplish tasks without being programmed to do them. It enables computers to “learn” to perform tasks through repetition. Systems that use ML can learn to perform tasks progressively better through repetition.

Machine learning is well suited for 3 kinds of tasks

ML beats other technologies in 3 areas:

• It’s well suited for situations where traditional programming requires extensive fine-tuning or creation of long lists of rules.
• It’s also useful in environments that change often, such as the position of promotional display racks in a retail store. It’s easier for an ML system to adapt to new data than for a programmer to modify code every time the environment changes.
• It’s a powerful tool for discovering unexpected insights in large datasets.

Machine learning occurs in 3 ways

Computers learn to classify point clouds (or to perform nearly any other ML process) through 3 methods:

• Supervised learning
• Unsupervised learning
• Semi-supervised learning

Supervised learning

With supervised learning, a computer uses labeled data to train its ML algorithms. The goal is for the computer to see so many variations of a pattern that it can recognize the pattern even when the computer comes across an image it’s never seen.

In point cloud classification, an ML system learns from its exposure to previous manually classified point clouds. The previous point clouds serve as training data. The ML system then uses this knowledge to classify new point clouds. The resulting capability is called a point cloud classifier. 

Unsupervised learning

In unsupervised learning, an ML system learns without first seeing training data. The method can be useful when you want an ML system to discover hidden patterns in data.

For example, ML systems monitor credit-card transactions in real-time, seeking to flag transactions that may be fraudulent. They can find fraudulent patterns in transactions they’ve never seen before.

Semi-supervised learning

As its name suggests, semi-supervised learning is a hybrid of supervised and unsupervised methods. Semi-supervised learning is an approach to machine learning that combines a small amount of labeled data with a large amount of unlabeled data during training.

Here’s an example: A model can use a semi-supervised learning process to predict the translation of a sentence from its context, based on other texts and related translations.

5. How does machine learning enable point cloud classification?

Machine learning enables point cloud classification through a series of 5 steps that are common to all ML:

• Gathering data
• Cleaning data
• Defining the model and its parameters
• Training and testing the model and predicting the output

For point cloud classification, the training process uses a supervised learning model. Defining the parameters is an essential element of supervised ML. It has 2 goals:

• To prepare the dataset that will be used as inputs for the ML algorithms
• To improve the performance over time by adjusting them

The parameters are important because it turns data inputs into elements a model can understand. For Alteia’s point cloud classification model, the model has been trained on datasets for various industrial assets where experts provided or proof-checked the labeling.

6. How does machine learning enable ‘reconstruction’ in a building information management system (BIM)?

In the Oil & Gas industry, point clouds enable operators to create digital twins of complex structures and equipment in remote locations. The capability enables experts in one or more central locations to monitor the operations of wells, pipelines, plants, and offshore oil rigs more efficiently. The Power & Utilities industry and Engineering & Construction often face similar challenges with other kinds of structures.

Here’s how the application works:

• A sensor captures data for an asset.
• Software uses the data to create a digital twin point cloud of the scanned asset.
• Algorithms decompose the point cloud into classes. This means algorithms find identifiable classes within the point cloud and classify them as various kinds of objects. The classes may include pipes, electrical lines, poles, valves, transformers, trees, ground, buildings, and the like.
• Once the objects in the point cloud have been correctly classified, they are vectorized. This means algorithms further transform them into vector data models.

A vector data model is a mathematical representation of the physical characteristics of an object. It defines objects such as valves, transformers, and pipes as sets of lines, or collections of vertices and polygons.

1. You can then load the vector files into a building information system (BIM). In the BIM, you can visualize the vector files along with other information to document the details of your assets.
2. Remote operators can then view details from within a BIM. They can  monitor operations for safety, compliance, productivity, and more.

7. What can point cloud classification and machine learning do for your organization?

Point cloud classification has high-value applications in all 8 of the industry segments Alteia serves today:

• Mining & aggregates
• Agriculture & forestry
• Oil & gas
• Power & utilities
• Manufacturing
• Retailing
• Engineering & construction
• Defense & security

Next we’ll deep dive into examples of how point cloud classification serves several sectors.

Forestry

Point cloud classification enables tree counting and tree-density assessments. Coupled with time-series analytics, it can also predict tree growth.

Civil engineering & construction

In the construction industry, point clouds can verify that complex structures comply with project specifications. Use of point clouds can find small discrepancies between a 3D site plan and a digital twin of the actual site.

Mining & aggregates

In underground mines, companies also use point clouds to map the pitch-dark interiors of drifts and stopes. They can observe their infrastructure and safety conditions in the horizontal tunnels used for haulage, ventilation, or exploration.

In the vertical areas where miners extract ore, point clouds can help monitor operations to prevent collapses. Point clouds can also plot the electrical utilities and air shafts that run through them. With such information, miners can plan production and comply with safety specs.

Many sectors other use point clouds and classification for other uses, including these:

• Guidance of autonomous vehicles
• Medical diagnostics and planning of surgical procedures
• Damage assessment for insurance coverage of natural disasters
• Environmental monitoring and sustainability
• Identification of archeological sites where ruins are buried

3D scans, point clouds, and machine learning are among the foundational technologies that enable visual AI. The technologies are fascinating in themselves. But their biggest appeal lies in their ability to help organizations in many industry segments solve expensive problems.

Dig deeper

For more on the technologies and uses of 3D scanners, point cloud classification, and machine learning, see any of the following resources:

“What Is Machine Learning and Why Is It Important?” Ed Burns. 2021. TechTarget. [Online article.]

“An Object-Based Bidirectional Method for Integrated Building Extraction and Change Detection between Multimodal Point Clouds.” Dai, C., Zhang, Z., & Lin, D. 2020. Remote Sensing, 12(10). [Link is to a listing in MDPI.com. Access to the full article is at no charge.]

“Lidar point classification—Help.” Esri. 2019. ArcGIS Desktop. [Online article]

“What is lidar data?—Help.” ArcGIS Desktop. Esri. 2019. [Online article.]

“Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images.” Holzer, S., Rusu, R. B., Dixon, M., Gedikli, S., & Navab, N. 2012. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2684–2689. [Link is to an abstract in IEEE Xplore at IEEE.org. Registration and fee required for full viewing.]

“Real-Time Table Plane Detection Using Accelerometer Information And Organized Point Cloud Data From Kinect Sensor.” Hung, L. Van, Michiel, V., Hai, V., Thuy, N. T., Lan, L. T., Hai, T. T. T., Hiep, L. Q., Peter, V., & Philips, W. 2017. Journal of Computer Science and Cybernetics, 32(3), 243–258. [Link is to an abstract.]

“PointGuard: Provably Robust 3D Point Cloud Classification.” Jia, J., & Gong, N. Z. 2021. In The Institute of Electrical and Electronics Engineers, Inc. (IEEE) Conference Proceedings. The Institute of Electrical and Electronics Engineers, Inc. (IEEE). [Link is to an abstract in IEEE Xplore at IEEE.org. Registration and fee required for full viewing.]

“Point Cloud to BIM: BIM for Renovation and Retrofitting.” Rana, A. 2015. [Online article.]

“ Classification in Machine Learning: What it is & Classification Models.” Simplilearn. (2021). [Online article.]

“What are point clouds? 5 easy facts that explain point clouds.” Thompson, C. 2019. Vercator. [Online article]

“Integrating Normal Vector Features into an Atrous Convolution Residual Network for LiDAR Point Cloud Classification.” Zhang, C., Xu, S., Jiang, T., Liu, J., Liu, Z., Luo, A., & Ma, Y. 2021. Remote Sensing, 13(17), 3427. [Link is to a listing in MDPI.com. Access to the full article is at no charge.]