How to read paintings: Semantic art understanding with multi-modal retrieval

Noa Garcia*, George Vogiatzis

*Corresponding author for this work

Research output: Chapter in Book/Published conference outputConference publication


Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage the Text2Art challenge, a multi-modal retrieval task where relevant paintings are retrieved according to an artistic text, and vice versa. We also propose several models for encoding visual and textual artistic representations into a common semantic space. Our best approach is able to find the correct image within the top 10 ranked images in the 45.5% of the test samples. Moreover, our models show remarkable levels of art understanding when compared against human evaluation.

Original languageEnglish
Title of host publicationComputer Vision – ECCV 2018 Workshops, Proceedings
EditorsStefan Roth, Laura Leal-Taixé
Number of pages16
ISBN (Electronic)978-3-030-11012-3
ISBN (Print)9783030110116
Publication statusPublished - 29 Jan 2019
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: 8 Sept 201814 Sept 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11130 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference15th European Conference on Computer Vision, ECCV 2018


  • Art analysis
  • Image-text retrieval
  • Multi-modal retrieval
  • Semantic art understanding


Dive into the research topics of 'How to read paintings: Semantic art understanding with multi-modal retrieval'. Together they form a unique fingerprint.

Cite this