While machine and deep learning – both forms of artificial intelligence – are increasingly prevalent in day-to-day life, from self-driving cars to spam filters and personalised video recommendations, they remain relatively unchartered territory in mining.

Motion Metrics, however, is doing some pioneering work in that area with the launch of its loader tooth monitoring system LoaderMetrics. Using a rugged thermal camera and a sophisticated computer algorithm, the system identifies teeth that have broken and fallen into the bucket, which can end up mixed in with ore. The system alerts the machine operator with an audible alarm and a visual indicator on an in-cab display. The operator can then ensure that the heavy steel tooth is not unwittingly dumped into the crusher plant with the ore, and avoid a colossal – and highly expensive – jam.

“When you lose a tooth, you’re stuffing something very large and hard into your crusher,” explained Derek Cooper, head of business development at Motion Metrics. Loader teeth can weigh up to 100 kilograms and jammed crushers bring operations to a standstill. An even worse scenario, he added, is if the tooth makes its way further downstream, through the primary crusher and mangles the mine’s secondary crushing systems. “That just rips everything apart, from conveyor belts to gear boxes. It’s nasty.”

The downtime needed for the removal process for a jammed tooth in a crusher is considerable and results in huge financial losses. “For less precious commodity mines, you are looking at $100,000 per incident,” Cooper said. “If you are looking at a precious metal mine, it could be millions of dollars. [Tooth monitoring] pays off in one incident.”

In order to remove a jammed loader tooth from a crusher, all other material in the crusher plant has to be painstakingly scooped out. Mine operators crouched in the belly of the emptied crusher have to use a cutting torch to cut, soften and free the jammed tooth. Reacting to extreme pressure and heat, the tooth can eject at a high speed and at great risk to mining personnel.

The shovel tooth monitoring system grew out of a PhD thesis in robotics and intelligent systems by now-Motion Metrics CEO Shahram Tafazoli at the University of British Columbia. Tafazoli had been informed by an industry insider about the catastrophic effect a broken tooth has on mine operations and safety.

Tafazoli would later found Motion Metrics to provide advanced monitoring solutions to mines worldwide. The company’s flagship technology, ShovelMetrics, rolled out in 2000, addressed the same tooth breakage issue in shovels using custom-developed conventional image processing methods.

Developing and adapting the preceding shovel monitoring technology for loaders involved overcoming a tough set of technological challenges. The shovel technology relies on an optical camera and an LED light mounted on the highest point of the shovel, up to 15 metres above the ground. This positioning affords a clear, unobstructed view of the shovel bucket below, allowing the camera to monitor for teeth for at least 20 to 30 seconds of the 45-second loading cycle.

Related: Motion Metrics' handheld fragmentation analysis improves throughput at Pinto Valley

Necessity breeds invention

Loaders, on the other hand, are more squat and do not have a boom. The only place a camera can be installed is between the front wheels, with the lens looking up at the bucket. As a result, there is a far smaller window for data-gathering; Cooper estimated it at two to four seconds. Moreover, when positioned so close to the ground and looking up, the camera and LED lights are more acutely exposed to the perils of the harsh mining environment, such as debris, rocks, dirt, mud and rain.

As a result, the team developing LoaderMetrics realized early in the research and development process that the technology needed an algorithm that processed images more effectively than before. “We couldn’t analyze the images fast, often and accurate enough with the conventional method,” said Saeed Karimifard, LoaderMetrics product manager. “This is where deep and machine learning come in.”

Motion Metrics put together a team consisting of machine learning experts plus electrical, mechanical and computer engineers over a two-year period to develop the AI-based technology, which was later led by Karimifard and rolled out in 2016. Their main challenge was to improve the system’s image-grabbing frequency and accuracy in missing tooth detection, and to design a resilient camera that could survive unforgiving conditions.

In order to train the algorithm to identify a missing tooth and trigger an alert in real time, the team had to teach the algorithm to focus data-gathering efforts on the part of the image containing the bucket and teeth – in the same way a human instinctively knows to look on the table, not the floor or ceiling, when asked to find a glass of water. They also had to teach it how to recognize whether the tooth was missing or not from the camera pixels.

To do so, the team designed an artificial neural network (ANN). They labelled thousands of images of the bucket area and teeth, creating a vast data set that was fed into the network. The algorithm interpreted this data set to learn how to detect those regions in an image and determine the status of the tooth on its own.

“Human reinforcement is key to this type of machine learning,” said Cooper. “So, when we have a whole bunch of humans labelling images, that’s essentially training the computer algorithm to know to look for the tooth in a certain area, and to know that a tooth looks a certain way.”

ANNs are what set deep learning apart from other types of machine learning. They are a multi-layered, interconnected group of nodes, inspired by the vast network of neurons in the human brain. Each layer of data takes the output of the previous one as its input and creates a more abstract representation of it. This teaches the computer to recognize abstract patterns and to correct its predictions based on all the samples it has seen.

“It’s like when you teach a child the difference between a cat and a dog,” explained Karimifard. “By repetitively showing them the animals and clarifying which it is, you are training them. They are going to improve performance and accuracy through time.” He added that the fact that the neural networks become more and more adept at correctly identifying the image is a “key advantage” of deep learning.

In order to combat the camera’s shortened visibility time and vulnerable position, and to eliminate the need for the LED light, the team opted for a thermal camera, instead of an optical one, and designed a strong metal mesh to guard the camera glass from falling rocks without obstructing its view.

They also invented an automated lens-cleaning solution made from a combination of high-pressure washer fluid and compressed air that can withstand temperatures ranging from -40 C to 40 C.

While other tooth detection technologies exist, some of which involve embedding an RFID tag inside the teeth, Cooper said he believes Motion Metrics’ approach is “easier, more cost-efficient, more reliable and accurate.”

He added that while machine learning technologies are still novel for the mining industry, he said he believes the technology has huge potential for asset management techniques and operational improvements in mining. (Motion Metrics recently applied the deep learning techniques to the latest version of their shovel technology.)

“We are moving dirt and rocks in the mining industry,” he explained. “But in reality, we are moving data. Every single bucket scoop is giving us valuable data and operational insight and that’s what we are hoping to gather.”