EgoCap and EgoFormer: First-person image captioning with context fusion

Zhuangzhuang Dai, Vu Tran, Andrew Markham, Niki Trigoni, M. Arif Rahman, L.N.S. Wijayasingha, John Stankovic, Chen Li

Research output: Contribution to journalArticlepeer-review


First-person captioning is significant because it provides veracious descriptions of egocentric scenes in a unique perspective. Also, there is a need to caption the scene, a.k.a. life-logging, for patients, travellers, and emergency responders in an egocentric narrative. Ego-captioning is indeed non-trivial since (1) Ego-images can be noisy due to motion and angles; (2) Describing a scene in a first-person narrative involves drastically different semantics; (3) Empirical implications have to be made on top of visual appearance because the cameraperson is often outside the field of view. We note we humans make good sense out of casual footage thanks to our contextual awareness in judging when and where the event unfolds, and whom the cameraperson is interacting with. This inspires the infusion of such "contexts" for situation-aware captioning. We create EgoCap which contains 2.1K ego-images, over 10K ego-captions, and 6.3K contextual labels, to close the gap of lacking ego-captioning datasets. We propose EgoFormer, a dual-encoder transformer-based network which fuses both contextual and visual features. The context encoder is pre-trained on ImageNet before fine tuning with context classification tasks. Similar to visual attention, we exploit stacked multi-head attention layers in the captioning decoder to reinforce attention to the context features. The EgoFormer has realized state-of-the-art performance on EgoCap achieving a CIDEr score of 125.52. The EgoCap dataset and EgoFormer are publicly available at
Original languageEnglish
Pages (from-to)50-56
Number of pages7
JournalPattern Recognition Letters
Early online date20 Mar 2024
Publication statusPublished - May 2024

Bibliographical note

Copyright © 2024 Elsevier B.V. This manuscript version is made available under the CC-BY-NC-ND 4.0 license


  • image captioning
  • storytelling
  • dataset


Dive into the research topics of 'EgoCap and EgoFormer: First-person image captioning with context fusion'. Together they form a unique fingerprint.

Cite this