TY - JOUR
T1 - Are we capturing individual differences? Evaluating the test-retest reliability of experimental tasks used to measure social cognitive abilities.
AU - Pennington, Charlotte Rebecca
AU - Birch-Hurst, Kayley
AU - Ploszajski, Matthew
AU - Clark, Kait
AU - Hedge, Craig
AU - Shaw, Daniel Joel
PY - 2025/1/13
Y1 - 2025/1/13
N2 - Background: Social cognitive skills are crucial for positive interpersonal relationships, health, and wellbeing and encompass both automatic and reflexive processes. To assess this myriad of skills, researchers have developed numerous experimental tasks that measure automatic imitation, emotion recognition, empathy, perspective taking, and intergroup bias and have used these to reveal important individual differences in social cognition. However, the very reason these tasks produce robust experimental effects – low between-participant variability – can make their use as correlational tools problematic. Aims: To perform an evaluation of test-retest reliability for common experimental tasks that measure social cognition. Method: One-hundred and fifty participants completed the race-Implicit Association Test (r-IAT), Stimulus-Response Compatibility (SRC) task, Emotional Go/No-Go (eGNG) task, Dot Perspective-Taking (DPT) task, and State Affective Empathy (SAE) task, as well as the Interpersonal Reactivity Index (IRI) and indices of Explicit Bias (EB) across two sessions within 3 weeks. Results: Estimates of test-retest reliability varied considerably between tasks and their indices: the eGNG task had good reliability (ICC = 0.63-0.69); the SAE task had moderate-to-good reliability (ICC = 0.56-0.77); the r-IAT had moderate reliability (ICC = 0.49); the DPT task had poor-to-good reliability (ICC = 0.24-0.60); and the SRC task had poor reliability (ICC = 0.09-0.29). The IRI had good-to-excellent reliability (ICC = 0.76-0.83) and EB had good reliability (ICC = 0.70-0.77). Conclusions: Experimental tasks of social cognition are used routinely to assess individual differences but their suitability for this is rarely evaluated. Researchers investigating individual differences must assess the test-retest reliability of their measures.
AB - Background: Social cognitive skills are crucial for positive interpersonal relationships, health, and wellbeing and encompass both automatic and reflexive processes. To assess this myriad of skills, researchers have developed numerous experimental tasks that measure automatic imitation, emotion recognition, empathy, perspective taking, and intergroup bias and have used these to reveal important individual differences in social cognition. However, the very reason these tasks produce robust experimental effects – low between-participant variability – can make their use as correlational tools problematic. Aims: To perform an evaluation of test-retest reliability for common experimental tasks that measure social cognition. Method: One-hundred and fifty participants completed the race-Implicit Association Test (r-IAT), Stimulus-Response Compatibility (SRC) task, Emotional Go/No-Go (eGNG) task, Dot Perspective-Taking (DPT) task, and State Affective Empathy (SAE) task, as well as the Interpersonal Reactivity Index (IRI) and indices of Explicit Bias (EB) across two sessions within 3 weeks. Results: Estimates of test-retest reliability varied considerably between tasks and their indices: the eGNG task had good reliability (ICC = 0.63-0.69); the SAE task had moderate-to-good reliability (ICC = 0.56-0.77); the r-IAT had moderate reliability (ICC = 0.49); the DPT task had poor-to-good reliability (ICC = 0.24-0.60); and the SRC task had poor reliability (ICC = 0.09-0.29). The IRI had good-to-excellent reliability (ICC = 0.76-0.83) and EB had good reliability (ICC = 0.70-0.77). Conclusions: Experimental tasks of social cognition are used routinely to assess individual differences but their suitability for this is rarely evaluated. Researchers investigating individual differences must assess the test-retest reliability of their measures.
KW - test-retest
KW - social cognition
KW - reliability
KW - social behaviour
KW - task reliability
KW - individual differences
U2 - 10.31234/osf.io/azq23
DO - 10.31234/osf.io/azq23
M3 - Article
SN - 1554-3528
JO - Behavior Research Methods
JF - Behavior Research Methods
ER -