In a meeting room in BGC Engineering’s Toronto Bay Street offices, Matthew Lato, a geotechnical engineer with the company, put on a Microsoft HoloLens. Slowly, he moved his head slightly up, then down, then to the side, then pinched the air with his fingers. It seemed as if he was performing mime but in reality, he was clicking the menu of the tiny yet powerful computer wrapped around his head and loading 3D holographic models onto the headset.

The HoloLens technology “is basically like a cellphone on steroids but instead of looking at you, it sees what you’re looking at,” he said before turning his head toward the long table in the room and addressing his conversation back to the device. “Computer, zoom in.”

A couple of minutes later, he passed the headset to me. With the smart glasses on my head, I could suddenly see a 3D map of a mine site on the table. I could walk around it and examine it from different angles, zoom in for close ups, or out to see the map expand beyond the initial horizon. I could also still see Lato, the table and chairs, windows and everything else in the room.

Because the HoloLens can be scaled up to a one-to-one ratio, a few minutes later, I was immersed in a virtual reclaimed mine site with a lake, trees and sunshine. I could step onto the boardwalk and listen to birds chirping. “All the trees are the right species and aged 20 years after planting. It’s September 15, 2037 at two o’clock in the afternoon,” said Lato. “The sun is where it would be at that time of day.”

The next big disruptive technology

Launched in 2016, HoloLens combines both virtual reality (VR), which blocks everything from view except the 3D images, and augmented reality (AR), which allows the users to view a 3D image while still seeing the physical environment around them, making it a mixed reality (MR) device. It can also pull up additional information from the database, including charts that can be displayed on the model. The HoloLens also has such innovations as spatial anchoring, which fixes the 3D image on a certain spot. If 10 other people had been in the room each wearing headsets, they would all have been able to see the same model on the table as I saw, each from the perspective of their position relative to the model.  At a cost of about $5,000 a headset, the HoloLens is considered by many to be a game changer, primarily because of its MR capabilities. While the HoloLens might be to holographic computing what the smartphone was to the cellphone, all the emerging VR, AR and MR technologies have the potential to be as disruptive as smartphones if not more – and to transform industries such as mining.

Like smartphones, these are technologies that open up the human imagination to an almost infinite number of applications once people get past the wow factor. In fact, the biggest challenge the technology faces, said Lato, is convincing people that it is not just a cool gimmick. It has very real and practical benefits and uses that can improve designs, communication and collaboration, efficiencies and safety. But with holographic computing, seeing is believing. “They’re usually convinced pretty quickly with a demonstration,” said Lato. BGC has only been making the technology available to clients since this summer. The company already has five committed to using its MR immersive landscapes and interactive 3D engineering map application that use data from topographical maps, soil and rock profiles and engineering plans. The most recent client to sign up for BGC’s visualization service is Barrick Gold’s joint venture with Goldcorp at the Pueblo Viejo mining project in the Dominican Republic, and Rio Tinto’s IOC Railway in Canada is currently testing the technology.

“Mixed Reality is going to become the dominant way people interact with information, because it is less abstract, and more natural and intuitive,” said John Howard, a holographic computing specialist who partnered with BGC to develop the applications.

BGC’s vice-president, Bill Burton, added: “In mine development, sometimes we’re talking to experts in their own fields who are not used to working with traditional maps. This makes it accessible to them.”

From technical input to community feedback, Burton said he believes the 3D maps and landscapes will result in greater insights and communication as well as improved designs. In fact, while the maps are currently created by inputting designs and data into the software, in the near future he said he believes the original designs will be made using virtual reality.

Mining imagination

But geological engineers are not the only ones who are imagining the possibilities. Last year, Goldcorp launched its own AR and VR research and development initiative as part of its ever-expanding commitment towards innovation. Luis Canepari, vice-president of technology at Goldcorp, and his team are looking into applications like equipping shovel operators with HoloLenses loaded with models of the different grades where they are digging to help them improve their precision and efficiency. They are also considering building 3D models of Goldcorp’s mines that could include high definition imagery enhanced with data from real-time sensor at their underground mines. “Imagine the productivity and safety improvements if our engineers and geologist could visualize the mine real time just by putting on a HoloLens from the safety of their office,” said Canepari.

The technology also allows for collaboration among people in locations across the world, all viewing the same images through the Cloud and their HoloLenses headsets. “Within a few years, I believe our workers will be able to get support from their supervisors or from subject matter experts located even at different sites and collaborate in real time,” said Canepari.

The HoloLens hard drive has some limitations simply due to size. It cannot create very complex, large 3D models, such as LiDAR scan data or other large mining and construction models. But Ottawa-based Arvizio has developed an MR software platform that optimizes point cloud models, with billions of points covering many square kilometres, for display on mixed reality devices. “MR Studio allows you to visualize LiDAR point clouds and overlay the LiDAR data on CAD models for comparison,” said Jonathan Reeves, the company’s CEO. “Project teams get an early read if things are not going quite right during the design process, visualize problem areas and collaborate on solutions throughout the project lifecycle.” The technology also allows for collaboration among people both locally and remotely, all viewing the same images via the internet and/or their mixed reality headsets.

Arvizio’s MR Studio allows the visualization of LiDAR point clouds and can overlay the LiDAR data on CAD models. Courtesy of Arvizio

 Virtually transforming reality

VR technology is also staking out territory in the mining sector. Within a year of introducing its own proprietary VR technology, Cementation, which designs and builds underground mines as well as the processing and material handling facilities above ground, has now adopted it for most of its projects. Its system is completely immersive and life-sized. Wearing a headset, users can travel across an entire mine site or facility while standing in a small.. They can navigate up and down stairs and cover kilometres simply by clicking on a handheld device that “tele-transports” them in Star Trek style to anywhere they want in the VR model.

When designing a conveyer belt, Cementation had its client’s maintenance team participate in VR sessions to test out the design. Inside the VR model, the team squatted, reached, crawled and cautiously leaned over edges to look downward as they would have in the real space, said Clark Whiting, vice-president at Cementation AG. Their insights resulted in key changes, including enhanced lighting, widening of the walkway, replacement of a full enclosure that reduced accessibility with a three-quarter hood – and, since the maintenance team found they could reach everything from one side of the walkway, it also led to the cost-cutting elimination of the walkway on the other side.

“We use it to get good solid feedback from clients,” said Whiting. “After we’ve completed the design and build, many are asking us if they can use the model themselves for such things as training or maintenance.”

Cementation’s system currently lacks AR’s ability to pull up additional information, but is currently working on enhancements to integrate the capability.

 “For example, if you are looking at a piece of equipment while wearing the VR headset, you can bring up equipment specifications from the database, such as the maintenance history, so it gives you more than just a picture image,” said Whiting.

The company is also working on several other applications, including a prototype for use in mucking with a camera on the cab of the equipment that streams the live video to the operator who can sit safely in another room while seeing everything in VR through a headset.

The company is also working on a VR maintenance history application that will allow maintenance teams to see what was previously done on the equipment, as well as a training and planning application for mine rescue teams with simulated flash fires. They have even been asked to try to simulate extreme heat by cranking up the heat in the room. “Your emotions change in such conditions when it’s hard to breathe and you’re not able to think clearly. You panic,” said Whiting.

Training companies like Immersive Technologies have also seized VR opportunities in the last couple of years. “The main benefit of VR is its sense of presence to replicate high-risk events, provide hazard awareness training and a way of experiencing a workplace environment without the cost of travelling to site,” said Cian Dobson, the company’s visual database department manager.

Dream big

The day might come when miners face a problem and ask themselves “is there an app for that?” And if there is not, they will be able to make one, working with video game designers, the brains behind holographic computing. Much as with smartphones, just about anybody will be able to imagine new applications. “When someone puts the headset on, they start thinking of ways they can use it given their expertise,” said Whiting. “We’re always writing notes down as fast as we can because everybody has an idea of how this can be used.”

Lato has had a similar experience when people try out BGC’s HoloLens. “It’s just the beginning. I think it will change the way we think and work,” he said.