Using Virtual Reality and Electroencephalography to Investigate the Effect of Eye-Gaze in Joint Attention

Abstract

This thesis presents the use of virtual reality combined with EEG techniques to investigate joint attention. When two individuals can successfully exchange information with each other, it can aid the development and strength of that relationship. However, some individuals struggle to engage in this exchange and as a result, this can lead to difficulties in developing social and romantic relationships.

Until recently, most of the experiments that have investigated social cognition have used either (i) on-screen computer experiments or, (ii) observation experiments. These methods are limited in either (i) ecological validity or, (ii) internal validity. One solution that was used to combat this limitation is
by combining virtual reality and electroencephalogram (EEG) research methods to create a ’Neuro-VR’ approach. This facilitated (i) the control the ecological validity by using a virtual human that participants interacted and collaborated with and, (ii) maintained internal validity by displaying a controlled and consistent environment and behaviour from the virtual human.

Studies one and two refined the gaze sequence and made changes to develop the paradigm. When participant-facing research could resume, I ran the refined paradigm with the Neuro-VR research method. The paradigm consisted of two levels: Collaboration; which dictated whether the virtual human would present informative gaze shifts to the target puzzle piece (Collaborative) or the virtual
human will provide gaze shifts that are not informative (Non-Collaborative). The second level is Gaze Type; where the virtual human would engage in eye-contact with the participant (Eye) or not (No Eye).

We found that an immersive environment such as a virtual reality head-mounted display had an effect on Collaboration and Gaze Type when compared to a 2D screen. On a neural level, this resulted in significant alpha-band decreases that were predominantly observed in the effect of Collaboration and significant theta-band increases within Gaze Type comparisons. Overall, the findings suggest an effect of gaze on joint attention where direct eye-contact may facilitate faster cognitive processing (observed through faster response times) that possibly indicates an influence of social gaze. Further investigations
using dynamic stimuli should be used to explore the effect.

Date of Award	May 2023
Original language	English
Supervisor	Tim S. Meese (Supervisor), Klaus Kessler (Supervisor), Ulysses Bernardet (Supervisor) & Johanna Zumer (Supervisor)