Connectivity analysis of functional brain networks in using multi-modal human-computer interaction

To develop next generation multi-modal computer-aided design systems, it is important to evaluate the relationships between the user dependent factors and the combined performance of man and machine. The purpose of this research is to investigate if users' cognitive activity would increase with the use of multi-modal input, speech and gestures by analysing EEG signals. Experiments are conducted, using traditional (keyboard and mouse) and multi-modal (speech and gesture) inputs. We used Normalized transfer entropy as a connectivity measure to find the information flow patterns. We constructed binary and weighted Functional Brain Networks to explore distinct and varied brain regions quantitatively. We found significant differences in cognitive activity between the traditional and multi-modal inputs. Our statistical analysis results state that the user's cognitive activity increase when a multi-modal input is used. The findings have implications for the development of multi-modal interfaces for 3D modelling.