A survey on psycho-physiological analysis & measurement methods in multimodal systems

Muhammad Zeeshan Baig*, Manolya Kavakli

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Psycho-physiological analysis has gained greater attention in the last few decades in various fields including multimodal systems. Researchers use psychophysiological feedback devices such as skin conductance (SC), Electroencephalography (EEG) and Electrocardiography (ECG) to detect the affective states of the users during task performance. Psycho-physiological feedback has been successful in detection of the cognitive states of users in human-computer interaction (HCI). Recently, in game studies, psycho-physiological feedback has been used to capture the user experience and the effect of interaction on human psychology. This paper reviews several psycho-physiological, cognitive, and affective assessment studies and focuses on the use of psychophysiological signals in estimating the user’s cognitive and emotional states in multimodal systems. In this paper, we review the measurement techniques and methods that have been used to record psycho-physiological signals as well as the cognitive and emotional states in a variety of conditions. The aim of this review is to conduct a detailed study to identify, describe and analyze the key psycho-physiological parameters that relate to different mental and emotional states in order to provide an insight into key approaches. Furthermore, the advantages and limitations of these approaches are also highlighted in this paper. The findings state that the classification accuracy of >90% has been achieved in classifying emotions with EEG signals. A strong correlation between self-reported data, HCI experience, and psychophysiological data has been observed in a wide range of domains including games, human-robot interaction, mobile interaction, and simulations. An increase in β and γ-band activity have been observed in high intense games and simulations.

Original languageEnglish
Article number37
Number of pages20
JournalMultimodal Technologies and Interaction
Issue number2
Publication statusPublished - 28 May 2019

Bibliographical note

Publisher Copyright:
© 2019 by the authors. Licensee MDPI, Basel, Switzerland.


  • Cognitive activity
  • EEG
  • Emotion recognition
  • Gaming systems
  • Mental load
  • Multimodal interaction
  • Psycho-physiological analysis


Dive into the research topics of 'A survey on psycho-physiological analysis & measurement methods in multimodal systems'. Together they form a unique fingerprint.

Cite this