ELLISON: An Advanced Multimodal Deep Fusion Framework for Attention Lapse Detection in Industrial Human-Robot Collaboration

Accurate evaluation of human attention in Human-Robot Collaboration (HRC) is essential to ensure intuitive and safe interactions. Although recent research has made progress in predicting human attention in social contexts, accurately estimating attention in industrial settings remains a challenge, particularly in industrial settings where it is crucial for error prevention, productivity optimization, and maintaining a secure work environment. In this paper, we present a multimodal deep fusion framework, ELLISON, designed to predict human attention lapses during HRC activities in manufacturing assembly tasks. First, we introduce a multimodal attention tracking pipeline featuring dual backbone Transformer Encoder models, which is trained to identify areas of interest related to human attention incorporating operator’s head pose and gaze features. Secondly, a deep fusion method is applied to consolidate the outputs of both modalities using a learned weighted strategy to provide a unified estimate of human attention throughout the tracking process. Experimental results demonstrate the effectiveness of our approach in tracking human attention in identifying instances where workers paid close attention or became distracted while working alongside robots, during collaborative tasks. The proposed method has potential applications in enhancing safety measures, optimizing workflows, and improving efficiency of HRC in industrial environments.