Variational Recurrent Sequence-to-Sequence Retrieval for Stepwise Illustration

Vishwash Batra*, Aparajita Haldar, Yulan He, Hakan Ferhatosmanoglu, George Vogiatzis, Tanaya Guha

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We address and formalise the task of sequence-to-sequence (seq2seq) cross-modal retrieval. Given a sequence of text passages as query, the goal is to retrieve a sequence of images that best describes and aligns with the query. This new task extends the traditional cross-modal retrieval, where each image-text pair is treated independently ignoring broader context. We propose a novel variational recurrent seq2seq (VRSS) retrieval model for this seq2seq task. Unlike most cross-modal methods, we generate an image vector corresponding to the latent topic obtained from combining the text semantics and context. This synthetic image embedding point associated with every text embedding point can then be employed for either image generation or image retrieval as desired. We evaluate the model for the application of stepwise illustration of recipes, where a sequence of relevant images are retrieved to best match the steps described in the text. To this end, we build and release a new Stepwise Recipe dataset for research purposes, containing 10K recipes (sequences of image-text pairs) having a total of 67K image-text pairs. To our knowledge, it is the first publicly available dataset to offer rich semantic descriptions in a focused category such as food or recipes. Our model is shown to outperform several competitive and relevant baselines in the experiments. We also provide qualitative analysis of how semantically meaningful the results produced by our model are through human evaluation and comparison with relevant existing methods.

Original languageEnglish
Title of host publicationAdvances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Proceedings
EditorsJoemon M. Jose, Emine Yilmaz, João Magalhães, Flávio Martins, Pablo Castells, Nicola Ferro, Mário J. Silva
PublisherSpringer
Pages50-64
Number of pages15
ISBN (Print)9783030454388
DOIs
Publication statusE-pub ahead of print - 8 Apr 2020
Event42nd European Conference on IR Research, ECIR 2020 - Lisbon, Portugal
Duration: 14 Apr 202017 Apr 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12035 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference42nd European Conference on IR Research, ECIR 2020
CountryPortugal
CityLisbon
Period14/04/2017/04/20

Keywords

  • Multimodal datasets
  • Semantics
  • Sequence retrieval

Fingerprint Dive into the research topics of 'Variational Recurrent Sequence-to-Sequence Retrieval for Stepwise Illustration'. Together they form a unique fingerprint.

  • Cite this

    Batra, V., Haldar, A., He, Y., Ferhatosmanoglu, H., Vogiatzis, G., & Guha, T. (2020). Variational Recurrent Sequence-to-Sequence Retrieval for Stepwise Illustration. In J. M. Jose, E. Yilmaz, J. Magalhães, F. Martins, P. Castells, N. Ferro, & M. J. Silva (Eds.), Advances in Information Retrieval - 42nd European Conference on IR Research, ECIR 2020, Proceedings (pp. 50-64). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12035 LNCS). Springer. https://doi.org/10.1007/978-3-030-45439-5_4