Exploring the Cognitive Gap and the Potential of XR Technologies
03 - Digital Ecosystem
With the evolution of computing as an essential part of design and manufacturing engineering, sequential and simultaneous engineering methods of product design and development have been replaced by concurrent engineering approaches. By embracing concurrent engineering, manufacturers can avoid costly errors, improve cost and quality, and accelerate time-to-market.
However, as Porter and Heppelmann noted in the December 2017 issue of the Harvard Business Review, “there is a fundamental disconnect between the wealth of digital data available to us and the physical world in which we apply it”. The authors continue to note that, “the rich data we now have to inform our decisions and actions remain trapped on 2D pages and screens”.
Closing the gap between the digital and physical world presents an opportunity to exploit the power of digital product development and digital manufacturing as a competitive advantage. This post discusses how Theorem Solutions has taken a unique approach to make this opportunity accessible, demonstrates how we are helping enlightened manufacturers to embrace the next wave of collaboration, and presents the technologies that are at the forefront of this immersive revolution.
04 - Closing the Cognitive Gap
Humans access and process information through each of our five senses; sight, hearing, smell, taste and touch. However, the ability to process this information varies between the senses, and the majority (an estimated 80% to 90%) of human information is accessed through sight.
Our ability to consume and process information is limited by the demand for our mental capacity, in what is referred to as ‘cognitive load’. Cognitive load is impacted by the mental effort required to process any given type of information. For instance, in reading this paper, mental effort is required to translate letters into words, and the words must be further interpreted into meaning.
More importantly, in the context of the disconnect between digital and physical worlds, cognitive load also depends significantly on ‘cognitive distance’, that is, the gap between the form in which information is presented, and the context in which it is applied. For instance, when following navigation instructions from a dash-mounted in-vehicle satellite navigation system, the driver must consume the information presented on the screen, retain that information in working memory, translate the instructions onto the physical world in front of them, and then finally act on the instructions, all while still maintaining control of the vehicle. The resulting cognitive journey creates a significant cognitive load. This is why we are seeing Head-Up-Displays (HUDs) now being used to overlay the relevant data and guidance where it is needed – by placing information in context and superimposing that information onto the physical world by way of a projection on the vehicle’s windshield, this reduces cognitive distance and therefore minimizes cognitive load.
Psychologists take this concept even further, with research showing that humans can only attend to one ‘cognitive task’ at a time. The research shows that task switching (aka multitasking) is “expensive”, taking more time to complete, and more errors made when you switch between tasks. If the tasks are complex, then the time and error penalties, or ‘cognitive cost’, will increase. In fact, the same research concluded that task switching could add up to a loss of 40% of your productivity.
Consider therefore the scenario of today’s engineer, constrained to viewing scaled down versions of 3D models, through a 2D screen, while attempting to consider the impact of changes in the context of full-sized products in the real world.
Given the increasing pressures on manufacturers to do more, to do it faster and to do it better, the advantages of closing the cognitive distance, and the potential to improve productivity is a compelling reason to take up the challenge in an engineering context.
05 - How has technology been responding to close 'Cognitive Distance'?
Virtual Reality (VR) is an interactive computer-generated experience taking place within an entirely simulated environment, incorporating auditory and visual feedback, but can also include other types of sensory feedback like haptics. The computer-generated simulation of a three-dimensional image or environment can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a head-mounted display or gloves fitted with sensors. In a literal sense, VR can make it possible to ‘experience’ anything, anywhere, anytime. It is the most immersive type of reality technology.
In an industrial context, Commercial VR has been evolving for around 30 years. Developed by specialist niche software companies, to serve large product development and manufacturing businesses, the immersive experience can be created through complex CAVE environments where projectors are directed onto three to six of the walls of a room-sized cube. This produces full-size renderings, giving the user a feeling of being immersed in a 3D world that can be interacted with through 3D glasses and a hand controller or glove. CAVEs are, however, extremely expensive, costing anything around $1M+.
PowerWalls have attempted to reduce the required investment to around $100k (possibly more if a 4K resolution is sought). Like a CAVE, a PowerWall creates a display through projection or more recently through a matrix display. Again, like in a CAVE, 3D glasses are used to create the illusion of three dimensions. However, despite the resolution, inevitably the sensation is one of working on a 2D screen, albeit at full-scale.
It is also worth noting that just presenting the data in 3D does not make an immersive system. It is only through motion capture tracking technologies that the system becomes actually immersive. The markers are seen by the system and the image on the screen is then rendered to that person’s point of view. This means that the tracked user is properly immersed. However, it can only track one person. Accordingly, for everyone else watching, the view is slightly offset, unless they are standing directly behind the tracked person. So, your commercial CAVE or Powerwall is really only truly immersive for one person at a time; when you consider this fact, these seem even more expensive!
Commercial VR in the form of CAVEs and PowerWalls has been closely aligned to the Product Lifecycle Management (PLM) software developers, such as Siemens, Dassault Systèmes and PTC, presenting another high-cost point of entry for product development companies attempting to leverage VR. Similarly, the preparation of data for use in these Commercial VR systems is a difficult and time-consuming process, resulting in larger businesses deploying dedicated data-preparation teams to support the demand.
Interestingly, Virtual Reality has seen a parallel development path in the consumer world; Consumer VR has been evolving for around 20 years. Unlike Commercial VR, the technology companies behind Consumer VR are global tech giants, such as Samsung, HTC, Google and Oculus (Facebook). The objective has been wide-scale availability and low cost. Most significantly, however, unlike Commercial VR, and despite multiple hardware giants playing in this field, the software behind Consumer VR has come from the gaming industry, and it is dominated by two massive players in the gaming world; Unity and Unreal. Consumer VR is a totally immersive technology, with the user seeing a completely simulated environment through consumer head-mounted displays. The upside is that the users can be ‘transported’ to a full-sized simulation of any environment, and then interact with that environment through linked handsets. 
The simulation also allows multiple users to interact within the simulated world. A fantastic example of this was portrayed in the 2018 movie “Ready Player One”.
Consumer VR has further evolved into two aligned technologies, Augmented Reality and Mixed Reality. Using the same graphical engines as Consumer VR, but also now including other development approaches such as Apple’s ARKit and Google’s ARCore, Augmented Reality (AR) provides the ability to overlay and view digital data onto images of the real world, using traditional devices such as smartphones and tablets. While dominated today by gaming applications, AR is seeing a rapid uptake in applications such as training, where digital assembly or service instructions can be presented by anchoring digital images onto real world ‘locators’. A number of training solutions developers are helping manufacturers to exploit this opportunity. The disadvantage can be a cognitive distance being created by having to place the screen to one side while working, or the hindrance of holding a screen while trying to work with the images displayed. Head-mounted devices, such as VUZIX, or the Google Glass, have presented some hands-free options to address this problem. Despite restricting the user to a relatively ‘flat’ viewing experience (through a screen), it is a substantial step forward in terms of contextual relevance when viewing data. AR is seeing significant growth, and it is predicted by industry observers to ultimately be the winner over Consumer VR.
Mixed Reality (MR) is a relative newcomer in this space. Until recently the MR space has been dominated by the Microsoft HoloLens, but recently challenged by a competitive offering from Magic Leap (with high profile investment from Google, Alibaba, and AT&T), both providing a Head-Mounted-Display (HMD) that blends digital [holographic] data with the physical world. This digital and physical blend is the closest step yet towards the provision of true spatial context for the user, placing full-sized 3D digital data ‘into’ the user’s world, and removing the need to ‘view through’ a screen. With the distinct advantage of being able to see other physical users in the same space, and even to present virtual ‘avatars’ of users into the room, MR is the closest solution yet to removing the cognitive gap. Just like Consumer VR and AR, MR is enabled by the same graphical gaming engines.
While the advantage of bringing engineers closer to their designs may be obvious, the scale of investment required for Commercial VR- and the complications of applying it- will present a barrier for many product development businesses. Consumer VR and the related Augmented and Mixed Reality technologies present an opportunity to overcome the financial barriers to entry. These technologies, collectively known as ‘XR’, also represent the potential to provide engineers with better solutions to interact with full-sized digital representations of their designs in a spatially relevant context. Unfortunately for application in the engineering field, the Unity and Unreal gaming ‘engines’ are not set up to support product development approaches, such as the management of a BoM (Bill of Materials) structure – we will return to that problem later in this post.
06 - Product development today - Globally distributed Engineering
Rarely are products completely developed in a single location with the luxury of co-located development teams. The reality of product development today involves collaboration across international borders. 
The engineering skills gap was first highlighted over a decade ago, and it has since been the topic of much debate. Recent findings continue to paint a gloomy picture, predicting that by 2020, the global gap between supply and incremental demand for qualified scientists, engineers and technicians may be as high as 74%4. In a 2015 survey by the Confederation of British Industry (CBI), 44% of engineering, science and high-tech firms reported difficulties in finding experienced recruits with the right STEM skills. Inevitably, therefore, businesses have been forced to locate skills in other locations, setting up their own offshore facilities, or working with Engineering Services Providers (ESPs), in locations such as Eastern Europe, India, Korea and China. As engineering moves from traditional in-house and supply chain sources, manufacturer’s partnerships with ESPs are evolving from a tactical response to additional capacity, progressing instead to strategic partnerships delivering higher value throughout the entire product realization lifecycle.
Technology may be a solution to closing Cognitive Distance, but what about improving collaboration when engineering teams can no longer work within the same walls? The realities of globally distributed product development mean that the engineering teams and their customers are increasingly facing the challenge of collaboration at a distance.
07 - The Theorem Solutions approach: Theorem XR - the next frontier of collaboration
The problem statement, therefore, is one of reducing the cognitive distance, bringing engineers closer to their designs, while simultaneously addressing the realities of collaboration at a distance. All of which must be focused on the objective of improving the quality and productivity of engineering and manufacturing processes.
Theorem Solutions has taken up the challenge of ‘providing enabling technologies for engineering and manufacturing businesses to improve their design, engineering and manufacturing processes by utilizing existing digital assets to collaborate within innovative Virtual, Augmented and Mixed Reality Experiences’. By combining the strengths of humans and machines, we believe that XR will dramatically increase value creation. Our goal has been to produce a technology agnostic collaboration platform for businesses of all sizes – ‘XR for Everybody’.
At the center of the Theorem Solutions approach therefore we are leveraging over 25 years of product Visualization experience, placing a technology agnostic ‘Visualization Pipeline’ at the core of immersive, contextual and spatially relevant XR collaborative workspaces. Recognizing the significant investments that some businesses have already made in Commercial VR, this approach also supports and improves any existing Commercial VR or photorealistic rendering processes. 
The Visualization Pipeline is a server-based technology that enables fast, efficient, flexible, and automated processing of CAD and Visualization data, while maintaining the associated metadata (retaining the essential knowledge embedded in the original source). It has been designed to optimise the output for all Visualization workflows, supporting CAVEs and PowerWalls, but also supporting photorealistic rendering (such as Alias, VRED and 3DExcite), existing lightweight viewing technologies (such as JT, Creo View and 3D XML), and preparing data for use within XR-based experiences (powered by Unity and Unreal). For collaboration to become a reality, data must be made available to all, irrespective of their toolset of choice.
The Visualization Pipeline can automatically process data in the background, or on-demand with ‘save as’ and ‘drag and drop’ functionality, removing the need for dedicated data processing teams.
However, preparing data for Visualization in the engineering context is only the first step towards the exploitation of the potential that XR represents. Theorem Solutions has invested significantly to understand how XR can be applied, not just in terms of the long term potential, but also how the technology can be exploited today to bring real business benefits to product development and manufacturing businesses. Working with early adopters in the Automotive and Aerospace industries, four key use-cases have been identified that represent an immediate benefit; we have created four ‘XR Experiences’ that get the most from the XR toolset, and further extended these experiences with a suite of targeted tools to deliver the greatest gain:
- Visualization Experience
- Design Review Experience
- Factory Layout Experience
- Training Guides Experience
- Visual Digital Twin Experience
Each use-case focused experience builds on the foundational Visualization Experience which empowers engineers to interact with data in a 3D spatial context, visualizing their models, at full scale. The Visualization Experience provides the essential tools to view, navigate, explode, section, and measure, in addition to interrogating materials and other information (including data pulled from external databases).
Design reviews are a critical part of the product development lifecycle. Across the globe, engineers meet regularly, and at key program milestones to undertake review activities such as:
• Release status and maturity (and identifying/reviewing gaps therein).
• Discussing known issues / anticipated issues.
• Detecting new issues (e.g. geometry clash issues, or manufacturing/service access problems).
• Resolving integration (interface) problems, including methodologies such as zone management/block management.
• Discussing proposed changes.
• Bringing together the necessary business functions to move decision making as far upstream in the development process as possible (avoiding costly changes late in the process).
Critically, digital design reviews enable manufacturers to:
• Reduce the expense of producing physical prototypes to aid review/issue detection.
• Enable the review process to include distributed teams.
• Improve the quality of downstream quality events by informing and sign-posting the review of focal areas, and better preparing for the effective comparison between physical output and design intent.
The Design Review Experience leverages the Visualization Experience, but further enables a group of engineers, local or in remote locations, to interactively review a design through the immersive experience of XR technologies, closing the cognitive distance for all involved, and providing a unique collaborative workspace. Importantly, the review process can be recorded in part or full, capturing comments and issues for wider cascade and use by other members of the extended team (including full management reports). The Design Review Experience can bring distributed experts and practitioners together to review, debate and resolve issues in real-time, improving access, saving time, and avoiding costly travel. The Factory Layout Experience, again leveraging the enabling technologies of the Visualization Experience, provides a rich Visualization experience designed to aid production engineers in the optimization of production facilities. Using an immersive VR experience or anchoring the data in the real world through AR and MR technologies, engineers can work at full scale to create and move the components of production facilities, assessing the efficiency of workflow, human and ergonomic design. Different layout strategies can be explored and saved for wider distribution and review by the extended team.
The Factory Layout Experience focuses on the needs of the planner, facilitating the optimization of equipment and plant layout, and ancillaries such as component bins and stillages, thereby maximizing productivity and efficiency, and reducing inventory. Similarly, engineers can factor in access for maintenance, cleaning, and line-side material handling. All of which can be explored and validated before committing to physical resources; reducing the potential for expensive and time-consuming late changes.
Using scanned data in a VR environment or using full-scale models viewed in the real world through AR and MR devices, production engineers can work around the realities of ever-changing factory facilities; planning around infrastructures such as building pillars, electrical and heating installations, and air ducting.
Production operatives are often hampered by late access to physical assets and typically cannot contribute their unique insight into production engineering until late in the new product introduction process. Working in totally immersive or real-world environments, staff can leverage XR to provide upstream feedback, improving the final production design, and once again reducing the potential for expensive and time-consuming late changes.
The Training Guides Experience harnesses the power of Virtual, Augmented, or Mixed Reality to enable manufacturers to prepare their teams ahead of the availability of physical assets. Working in totally immersive or real-world environments, production staff can operate and handle virtual tools and equipment, leveraging XR to rehearse build processes and undertake training in readiness for the real thing.
The Training Guides Experience is aimed at operators or trainers to teach and foster familiarity with new production processes, new build techniques or assembly sequences. It can also be used by experienced operators to address complex build processes and consider scenarios to overcome problems or optimize existing approaches.
Supported by the powerful viewing tools within the Visualization Experience, and recording interactions for wider cascade and action, the Training Guides Experience enables a new level of understanding about digital manufacturing, interacting at full scale within familiar workspaces, optimizing the production processes, and enabling the repeated practice of assembly activities to reduce manufactured cost.
It’s common sense that product development and manufacturing engineers should work together to develop a product that performs flawlessly and is manufactured (and assembled) in the most effective way. Both the Factory Layout Experience and Training Guides Experiences take advantage of the same collaboration functions introduced in the Design Review Experience. The power of the Digital Realities toolset enables distributed groups of engineers to interact through the immersive experience of XR technologies, closing the cognitive distance for all involved, and providing a unique collaborative workspace.
08 - Theorem XR for Everyone
Theorem Solutions has been working hard to overcome the limitations of the Consumer VR (and XR) technologies such that they can be leveraged by product development and manufacturing engineers today.
Focusing on the near-term gain from Virtual, Augmented and Mixed Reality technologies, we have crafted a suite of XR Experiences that bring out-of-the-box, task-orientated applications to aid modern manufacturers in the pursuit of more products, accelerated time-to-market and superior performance.
The Theorem XR Experiences, tightly integrated with the core Visualization Pipeline, engage the problem of cognitive distance, bringing engineers closer to their designs, while simultaneously addressing the realities of collaboration at a distance. Our device agnostic solutions and methodologies maintain a razor-sharp focus on the objective of improving the quality and productivity of engineering and manufacturing processes through enabling technologies.