Datapages, Inc.Print this page

Augmented Reality Field Performance Support System for the Enterprise

Carl Byers and Andrew Woo
NGRAIN, Vancouver, BC, Canada


With its ability to bridge the gap between the digital and physical worlds and to combine geometric data with abstract data, augmented reality (AR) is uniquely capable of helping people make better decisions, faster. AR for field performance support is an often cited example of this capability, but one that has not been implemented in an enterprise-level setting.

In this presentation, NGRAIN will introduce an Augmented Reality Field Performance Support System (AR-FPSS) model that not only provides robust field support features, such as context-specific work-orders, AR maintenance and task information, support resources, and integration to real-time sensor-equipped machines for live data feeds, but also captures the field activities and updates the changes back in the enterprise management systems in real time. An AR-FPSS that can close the information loop in the enterprise data and visual analytics ecosystem has the potential to greatly reduce the complexity of that ecosystem and to put the power of big data in the hands of those that need it most, when they need it.

The state-of-the-art in field based maintenance performance support is paper manuals, paper drawings, and remote terminals for accessing work orders or parts ordering. In limited cases, these resources are supplemented with Interactive Electronic Technical Manuals (IETMs), computer based versions of traditional manuals.

Guidance and support for field based maintainers has changed little in decades with an accepted inefficiency of 40-60% maintainer downtime, higher in more remote environments. This downtime results from searching for procedures, looking for parts, obtaining guidance from others, and waiting for parts. Additional downtime results when items with long-lead times for delivery are not planned into a maintenance procedure or when lack of experience results in incorrect maintenance requiring re-work.

In addition, current enterprise maintenance management environments rely on manual work order report input following a maintenance procedure. Delays in capturing, assessing, and disseminating this information ripple across the enterprise, as information that could be valuable to others as part of maintenance and logistics planning is not communicated in a timely manner.

The proposed initiative, Augmented Reality Field Performance Support System (AR-FPSS), is designed to revolutionize the field-based maintenance process, decreasing maintainer downtime, and reducing the delays updating the enterprise maintenance management system with field-based data. The AR-FPSS is a tablet based product that contains context-specific work-orders, augmented reality (AR) maintenance information and task guidance, and integrated maintenance and operational resources. The system uses the tablet's camera to recognize and track the equipment under maintenance, and overlays graphical and textual augmentations on the reality of the live camera image to guide and assist the maintainer. The AR-FPSS tablet is connected wirelessly to the enterprise maintenance management system to obtain real-time inventory status, order parts, and confirm part delivery. In addition, the AR-FPSS captures the maintenance activities for real-time or curated insertion into the enterprise management system.

The AR-FPSS is an iPad tablet AR product incorporating equipment maintenance information that is dynamically updated from enterprise systems with new information or equipment types.

To use the full functionality of the AR-FPSS operationally, the user simply points the iPad primary video camera at the piece of equipment under consideration (i.e., equipment for which there is content pre-loaded on the iPad). The iPad AR-FPSS application will recognize the equipment through a number of methods, including 3D object recognition, QR code markers, or RFID/bar-codes. The user observes the video camera image on the tablet display and, once the system automatically detects the real equipment and engages active inertial tracking, he or she can begin to interact with the AR experience through the iPad touchscreen.

The information available to the user changes depending on whether the AR object tracking is engaged with the real equipment present, or whether the system is 'off-line' without the equipment present. The maintainer is able to use a number of system modes for assistance. Each mode provides different access to augmented guidance and related resources and the user can move freely between modes.

Technical challenges of AR registration, tracking, performance, functionality, and usability have been addressed:

  • NGRAIN has developed reliable methods of rapid and consistent marker-based registration and tracking for real equipment that yields high performance and stability in the target field operational environment.
  • NGRAIN has optimized its volumetric rendering algorithms and data structure for the iPad in order to support a blend of functional augmentations, including basic graphic overlays, 3D geometry manipulation, and other text and graphics, without latency.
  • NGRAIN has optimized the user interface, considering display size, touch based usability, and user environments, to ensure a usable and smooth user experience.

  • Key technical issues of markerless tracking and guided maintenance will continue to be addressed:

  • AR-FPSS currently requires fiducial markers for tracking, reducing broader applicability. On-going investigation will focus on automatic object based recognition and tracking without markers. NGRAIN is uniquely placed to address this issue because the AR-FPSS contains an NGRAIN 3D equipment model that can be combined with image recognition to detect and track the equipment and align augmentations. Initial tests have already been conducted.
  • While not restricting current product utility, automatic guided maintenance techniques based on context and activity will further increase the effectiveness of the field maintainer.
  • AAPG Search and Discovery Article #90206 © AAPG Hedberg Conference, Interpretation Visualization in the Petroleum Industry, Houston, Texas, June 1-4, 2014