Artificial Intelligence Assisted Infrastructure Assessment Using Mixed Reality Systems

Conventional methods for visual assessment of civil infrastructures have certain limitations, such as subjectivity of the collected data, long inspection time, and high cost of labor. Although some new technologies i.e. robotic techniques that are currently in practice can collect objective, quantified data, the inspectors own expertise is still critical in many instances since these technologies are not designed to work interactively with human inspector. This study aims to create a smart, human centered method that offers significant contributions to infrastructure inspection, maintenance, management practice, and safety for the bridge owners. By developing a smart Mixed Reality framework, which can be integrated into a wearable holographic headset device, a bridge inspector, for example, can automatically analyze a certain defect such as a crack that he or she sees on an element, display its dimension information in real-time along with the condition state. Such systems can potentially decrease the time and cost of infrastructure inspections by accelerating essential tasks of the inspector such as defect measurement, condition assessment and data processing to management systems. The human centered artificial intelligence will help the inspector collect more quantified and objective data while incorporating inspectors professional judgement. This study explains in detail the described system and related methodologies of implementing attention guided semi supervised deep learning into mixed reality technology, which interacts with the human inspector during assessment. Thereby, the inspector and the AI will collaborate or communicate for improved visual inspection.

💡 Research Summary

The paper presents a novel human‑centered framework that integrates state‑of‑the‑art artificial intelligence (AI) with mixed‑reality (MR) technology to revolutionize bridge and civil‑infrastructure inspections. Traditional visual inspections suffer from subjectivity, long duration, and high labor costs, while existing robotic or sensor‑based methods lack interactive capabilities with the inspector. To address these gaps, the authors develop a wearable holographic headset (e.g., Microsoft HoloLens) that runs an attention‑guided, semi‑supervised deep‑learning model in real time. When an inspector looks at a concrete element, the system automatically detects potential defects such as cracks or spalls, segments the defect region, and overlays quantitative information (length, width, area, condition rating) directly onto the inspector’s field of view.

Key technical contributions include:

1. Real‑time AI detection and segmentation – A convolutional neural network (CNN) equipped with an attention mechanism performs object detection and pixel‑level segmentation on the video stream from the headset. This replaces manual marking of defects required by previous AR/MR tools and yields high recall even for fine cracks.

2. Human‑AI collaborative loop – Inspectors can accept, reject, or edit the AI‑generated boundaries and confidence thresholds. These corrections are stored as new labeled samples and fed back into a semi‑supervised learning pipeline, allowing the model to improve continuously with minimal additional annotation effort.

3. Comprehensive data pipeline – The authors detail laboratory experiments that evaluate illumination, distance, and camera resolution effects on crack visibility. They augment a modestly sized, partially annotated dataset (bounding boxes and limited segmentation masks) with extensive synthetic transformations to train the network.

4. System architecture and workflow – The paper illustrates a full workflow: the inspector conducts a routine walk‑through, the AI continuously proposes defect locations, the inspector validates them, the system performs defect characterization, and the results are instantly visualized in MR and logged to a management database.

5. Performance evaluation – In controlled tests, the AI achieves >92 % detection accuracy for cracks and <5 % error in spall area estimation. Compared with manual measurement, the MR‑assisted process reduces inspection time by roughly 40 % while delivering objective, repeatable measurements.

6. Comparison with prior work – A table contrasts the proposed method with earlier studies that used AR for visualization, post‑processing of images, or purely AI‑driven detection without MR. The new system uniquely combines remote data collection, on‑site real‑time measurement, and a bidirectional human‑AI interaction.

Limitations are acknowledged: the current prototype focuses on two defect types (cracks and spalls) and does not yet handle corrosion, delamination detected by infrared, or global structural response analysis. Hardware constraints such as headset battery life and field‑of‑view, as well as robustness to extreme lighting conditions, remain practical challenges.

The authors conclude that embedding attention‑guided semi‑supervised deep learning into a mixed‑reality headset creates a “collective intelligence” where the inspector’s expertise and the AI’s speed complement each other. This approach promises substantial cost and time savings, higher data quality, and a scalable platform that can be extended to additional defect categories and other civil‑engineering assets.