AI Visual Inspection For Defect Detection in Manufacturing

Artificial intelligence in manufacturing is a trendy term. When describing AI-based defect detection solutions, it’s often about visual inspection technology based on deep learning and computer vision.

What Is Deep Learning in a Visual Inspection?

Deep learning is an aspect of machine learning technology powered by artificial neural networks. The operating principle of deep learning technology is teaching machines to learn by example. By providing a neural network with labeled examples of specific data types, it’s possible to extract common patterns between those examples and then transform them into a math equation. This helps to classify future pieces of information.

With visual inspection technology, integrating deep learning algorithms allows differentiating parts, anomalies, and characters, which imitate a human visual inspection while running a computerized system. 

So, what does it mean exactly? Let’s use an example:

If you were to create a visual inspection software for automotive manufacturing, you should develop a deep learning-based algorithm and train it with examples of defects it must detect. With enough data, the neural network will eventually detect defects without any additional instructions.

Deep learning-based visual inspection systems are good at detecting defects that are complex in nature. They address complex surfaces and cosmetic flaws and generalize and conceptualize the parts’ surfaces.

How to Integrate AI Visual Inspection System

1. State the Problem

Visual inspection development often starts with a business and technical analysis. The goal here is to determine what kind of defects the system should detect.

Other important questions to ask include:

• What is the visual inspection system environment?
• Should the inspection be real-time or deferred? 
• How thoroughly should the visual inspection system detect defects, and should it distinguish them by type?
• Is there any existing software that integrates the visual inspection feature, or does it require a development from scratch?
• How should the system notify the user(s) about detected defects?
• Should the visual inspection system record defects detection statistics?
• And the key question: Does data for deep learning model development exist, including images of “good” and “bad” products and the different types of defects?

Data science engineers choose the optimal technical solution and flow to proceed based on the answers they receive.

2. Gather and Prepare Data

Data science engineers must gather and prepare data required to train a future model before deep learning model development starts. For manufacturing processes, it’s important to implement IoT data analytics. When discussing visual inspection models, the data is often video records, where images processed by a visual inspection model include video frames. There are several options for data gathering, but the most common are:

1. Taking an existing video record provided by a client
2. Taking open-source video records applicable for defined purposes
3. Gathering data from scratch according to deep learning model requirements

The most important parameters here are the video record’s quality. Higher quality data will lead to more accurate results. 

Once we gather the data, we prepare it for modeling, clean it, check for anomalies, and ensure its relevance.

3. Develop Deep Learning Model

The selection of a deep learning model development approach depends on the complexity of a task, required delivery time, and budget limitations. There are several approaches:

Using a deep learning model development service (e.g: Google Cloud ML Engine, Amazon ML, etc.)

This type of approach makes sense when requirements for defect detection features are in line with templates provided by a given service. These services can save both time and budget as there is no need to develop models from scratch. You have to upload data and set model options according to the relevant tasks. 

What’s the catch? These types of models are not customizable. Models’ capabilities are limited to options provided by a given service.

Using Pre-trained Models

A pre-trained model is an already created deep learning model that accomplishes tasks similar to what we want to perform. We do not have to build a model from scratch as it uses a trained model based on our data.

A pre-trained model may not 100% comply with all of our tasks, but it offers significant time and cost savings. Using models previously trained on large datasets lets us customize these solutions according to our problem. 

Deep Learning Model Development from Scratch

This method is ideal for complex and secure visual inspection systems. The approach may be time and effort-intensive, but the results are worth it. 

When developing custom visual inspection models, data scientists use one or several computer vision algorithms. These include image classification, object detection, and instance segmentation.

Many factors influence the choice of a deep learning algorithm(s). These include:

• Business goals
• Size of objects/defects 
• Lighting conditions
• Number of products to inspect
• Types of defects
• Resolution of images

An example of defect categories:

source: NECAM

Let’s say that we’re developing a visual inspection model for quality assessment in buildings. The main focus is to detect defects on the walls. An extensive dataset is necessary to obtain accurate visual inspection results, as the defect categories might be incredibly diverse, from peeling paint and mold to wall cracks. The optimal approach here would be to develop an instance segmentation-based model from scratch. A pre-trained model approach is also viable in some cases.

Another example is a visual inspection for pharmaceutical manufacturing, where you want to differentiate air bubbles from particles in products like highly viscous parental solutions. The presence of bubbles is the only defect category here, so the required dataset will not be as extensive as in the example above. The optimal deep learning model development approach might be to use a model development service over developing one from scratch.

4. Train and Evaluate

The next step after developing the visual inspection model is to train it. In this stage, data scientists validate and evaluate the performance and result accuracy of the model. A test dataset is useful here. A visual inspection system may be a set of video records that are either outdated or similar to ones we want to process after deployment.

5. Deploy and Improve

When deploying a visual inspection model, it’s important to consider how software and hardware system architectures correspond to a model capacity.

Software 

The structure of visual inspection-powered software bases itself on the combination of web solutions for data transmission and a Python framework for neural network processing. 

The key parameter here is data storage. There are three common ways to store data: on a local server, a cloud streaming service, or serverless architecture. 

A visual inspection system involves the storage of video records. The choice of a data storage solution often depends on a deep learning model functionality. For example, if a visual inspection system uses a large dataset, the optimal selection may be a cloud streaming service.

Hardware

Depending on the industry and automation processes, devices required to integrate visual inspection system may include:

• Camera: The key camera option is real-time video streaming. Some examples include IP and CCTV.
• Gateway: Both dedicated hardware appliances and software programs work well for a visual inspection system.
• CPU / GPU: If real-time results are necessary, a GPU would be the better choice than a CPU, as the former boasts a faster processing speed when it comes to image-based deep learning models. It is possible to optimize a CPU for operating the visual inspection model, but not for training. An example of an optimal GPU might be the Jetson Nano. 
• Photometer (optional): Depending on the lighting conditions of the visual inspection system environment, photometers may be required.
• Colorimeter (optional): When detecting color and luminance in light sources, imaging colorimeters have consistently high spatial resolutions, allowing for detailed visual inspections. 
• Thermographic camera (optional): In case of automated inspection of steam/water pipelines and facilities it is a good idea to have thermographic camera data. Thermographic camera data provides valuable information for heat/steam/water leakage detection. Thermal camera data is also useful for heat insulation inspection.
• Drones (optional): Nowadays it is hard to imagine automated inspection of hard-to-reach areas without drones: building internals, gas pipelines, tanker visual inspection, rocket/shuttle inspection. Drones may be equipped with high resolution cameras that can do real-time defect detection.

Deep learning models are open to improvement after deployment. A deep learning approach can increase the accuracy of the neural network through the iterative gathering of new data and model re-training. The result is a “smarter” visual inspection model that learns by increasing data during operation.

Visual Inspection Use Cases

Healthcare

In the fight against COVID-19, most airports and border crossings can now check passengers for signs of the disease.

Baidu, the large Chinese tech company, developed a large-scale visual inspection system based on AI. The system consists of computer vision-based cameras and infrared sensors that predict the temperatures of passengers. The technology, operational in Beijing’s Qinghe Railway Station, can screen up to 200 people per minute. The AI algorithm detects anyone who has a temperature above 37.3 degrees.

Another real-life case is the deep learning-based system developed by the Alibaba company. The system can detect the coronavirus in chest CT scans with 96% accuracy. With access to data from 5,000 COVID-19 cases, the system performs the test in 20 seconds. Moreover, it can differentiate between ordinary viral pneumonia and coronavirus.

Airlines

According to Boeing, 70% of the $2.6 trillion aerospace services market is dedicated to quality and maintenance. In 2018, Airbus introduced a new automated, drone-based aircraft inspection system that accelerates and facilitates visual inspections. This development reduces aircraft downtime while simultaneously increasing the quality of inspection reports.

Automotive

Toyota recently agreed to a $1.3 billion settlement due to a defect that caused cars to accelerate even when drivers attempted to slow down, resulting in 6 deaths in the U.S. Using the cognitive capabilities of visual inspection systems like Cognex ViDi, automotive manufacturers can analyze and identify quality issues much more accurately and resolve them before they occur.

Computer Equipment Manufacturing

The demand for smaller circuit board designs is growing. Fujitsu Laboratories has been spearheading the development of AI-enabled recognition systems for the electronics industry. They report significant progress in quality, cost, and delivery.

Textile

The implementation of automated visual inspection and a deep learning approach can now detect texture, weaving, stitching, and color matching issues.

For example, Datacolor’s AI system can consider historical data of past visual inspections to create custom tolerances that match more closely to the samples.

We will conclude with a quotation from the general manager we mentioned earlier: “It makes no difference to me whether the suggested technology is the best, but I do care how well it’s going to solve my problems.”

Solar Panels

Solar panels are known to suffer from dust and microcracks. Automated inspection of solar panels during manufacturing and before and after installation is a good idea to prevent shipment of malfunctioning solar panels and quick detection of damaged panels on your solar farm. For example, DJI Enterprise uses drones for solar panels inspection.

Pipeline Inspection

Gas and oil pipelines are known to have a huge length. The latest data from 2014 gives a total of slightly less than 2,175,000 miles (3,500,000 km) of pipeline in 120 countries of the world. Gas and oil leakages may lead to massive harm to nature by chemical pollution, explosions, and conflagrations.

Satellite and drone inspection with the help of computer vision techniques is a good tool for early detection and localization of a gas/oil leakage. Recently, DroneDeploy reported that they mapped about 180 miles of pipelines.

AI Visual Inspection: Key Takeaways

1. Concept: Al visual inspection bases itself on traditional computer vision methods and human vision.
2. Choice: Deep learning model development approach depends on the task, delivery time, and budget limits.
3. Algorithm: Deep learning algorithms detect defects by imitating a human analysis while running a computerized system.
4. Architecture: Software & hardware should correspond to deep learning model capacity.
5. Main question: When initiating a visual inspection, the main question is “What defects should the system detect?”
6. Improvements: After deployment, deep learning model becomes “smarter” through data accumulation.