The mosaic test

benchmarking colour-based image retrieval systems using image mosaics

William Plant, Jo Lumsden, Ian Nabney

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Evaluation and benchmarking in content-based image retrieval has always been a somewhat neglected research area, making it difficult to judge the efficacy of many presented approaches. In this paper we investigate the issue of benchmarking for colour-based image retrieval systems, which enable users to retrieve images from a database based on lowlevel colour content alone. We argue that current image retrieval evaluation methods are not suited to benchmarking colour-based image retrieval systems, due in main to not allowing users to reflect upon the suitability of retrieved images within the context of a creative project and their reliance on highly subjective ground-truths. As a solution to these issues, the research presented here introduces the Mosaic Test for evaluating colour-based image retrieval systems, in which test-users are asked to create an image mosaic of a predetermined target image, using the colour-based image
retrieval system that is being evaluated. We report on our findings from a user study which suggests that the Mosaic Test overcomes the major drawbacks associated with existing image retrieval evaluation methods, by enabling users to reflect upon image selections and automatically measuring image relevance in a way that correlates with the perception of many human assessors. We therefore propose that the Mosaic Test be adopted as a standardised benchmark for evaluating and comparing colour-based image retrieval systems.
Original languageEnglish
Title of host publicationProceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR)
PublisherCEUR-WS.org
Pages23-26
Number of pages4
Publication statusPublished - 2011
Event1st European Workshop on Human-Computer Interaction and Information Retrieval - Newcastle, United Kingdom
Duration: 4 Jul 2011 → …

Publication series

NameCEUR workshop proceedings
Volume763
ISSN (Electronic)1613-0073

Conference

Conference1st European Workshop on Human-Computer Interaction and Information Retrieval
CountryUnited Kingdom
CityNewcastle
Period4/07/11 → …

Fingerprint

Image retrieval
Benchmarking
Color

Keywords

  • benchmarking
  • content-based image retrieval
  • image databases
  • image mosaic
  • performance evaluation

Cite this

Plant, W., Lumsden, J., & Nabney, I. (2011). The mosaic test: benchmarking colour-based image retrieval systems using image mosaics. In Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR) (pp. 23-26). (CEUR workshop proceedings; Vol. 763). CEUR-WS.org.
Plant, William ; Lumsden, Jo ; Nabney, Ian. / The mosaic test : benchmarking colour-based image retrieval systems using image mosaics. Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR). CEUR-WS.org, 2011. pp. 23-26 (CEUR workshop proceedings).
@inproceedings{e71794800e7d4e3895217e1f0585d75b,
title = "The mosaic test: benchmarking colour-based image retrieval systems using image mosaics",
abstract = "Evaluation and benchmarking in content-based image retrieval has always been a somewhat neglected research area, making it difficult to judge the efficacy of many presented approaches. In this paper we investigate the issue of benchmarking for colour-based image retrieval systems, which enable users to retrieve images from a database based on lowlevel colour content alone. We argue that current image retrieval evaluation methods are not suited to benchmarking colour-based image retrieval systems, due in main to not allowing users to reflect upon the suitability of retrieved images within the context of a creative project and their reliance on highly subjective ground-truths. As a solution to these issues, the research presented here introduces the Mosaic Test for evaluating colour-based image retrieval systems, in which test-users are asked to create an image mosaic of a predetermined target image, using the colour-based imageretrieval system that is being evaluated. We report on our findings from a user study which suggests that the Mosaic Test overcomes the major drawbacks associated with existing image retrieval evaluation methods, by enabling users to reflect upon image selections and automatically measuring image relevance in a way that correlates with the perception of many human assessors. We therefore propose that the Mosaic Test be adopted as a standardised benchmark for evaluating and comparing colour-based image retrieval systems.",
keywords = "benchmarking, content-based image retrieval, image databases, image mosaic, performance evaluation",
author = "William Plant and Jo Lumsden and Ian Nabney",
year = "2011",
language = "English",
series = "CEUR workshop proceedings",
publisher = "CEUR-WS.org",
pages = "23--26",
booktitle = "Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR)",

}

Plant, W, Lumsden, J & Nabney, I 2011, The mosaic test: benchmarking colour-based image retrieval systems using image mosaics. in Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR). CEUR workshop proceedings, vol. 763, CEUR-WS.org, pp. 23-26, 1st European Workshop on Human-Computer Interaction and Information Retrieval, Newcastle, United Kingdom, 4/07/11.

The mosaic test : benchmarking colour-based image retrieval systems using image mosaics. / Plant, William; Lumsden, Jo; Nabney, Ian.

Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR). CEUR-WS.org, 2011. p. 23-26 (CEUR workshop proceedings; Vol. 763).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - The mosaic test

T2 - benchmarking colour-based image retrieval systems using image mosaics

AU - Plant, William

AU - Lumsden, Jo

AU - Nabney, Ian

PY - 2011

Y1 - 2011

N2 - Evaluation and benchmarking in content-based image retrieval has always been a somewhat neglected research area, making it difficult to judge the efficacy of many presented approaches. In this paper we investigate the issue of benchmarking for colour-based image retrieval systems, which enable users to retrieve images from a database based on lowlevel colour content alone. We argue that current image retrieval evaluation methods are not suited to benchmarking colour-based image retrieval systems, due in main to not allowing users to reflect upon the suitability of retrieved images within the context of a creative project and their reliance on highly subjective ground-truths. As a solution to these issues, the research presented here introduces the Mosaic Test for evaluating colour-based image retrieval systems, in which test-users are asked to create an image mosaic of a predetermined target image, using the colour-based imageretrieval system that is being evaluated. We report on our findings from a user study which suggests that the Mosaic Test overcomes the major drawbacks associated with existing image retrieval evaluation methods, by enabling users to reflect upon image selections and automatically measuring image relevance in a way that correlates with the perception of many human assessors. We therefore propose that the Mosaic Test be adopted as a standardised benchmark for evaluating and comparing colour-based image retrieval systems.

AB - Evaluation and benchmarking in content-based image retrieval has always been a somewhat neglected research area, making it difficult to judge the efficacy of many presented approaches. In this paper we investigate the issue of benchmarking for colour-based image retrieval systems, which enable users to retrieve images from a database based on lowlevel colour content alone. We argue that current image retrieval evaluation methods are not suited to benchmarking colour-based image retrieval systems, due in main to not allowing users to reflect upon the suitability of retrieved images within the context of a creative project and their reliance on highly subjective ground-truths. As a solution to these issues, the research presented here introduces the Mosaic Test for evaluating colour-based image retrieval systems, in which test-users are asked to create an image mosaic of a predetermined target image, using the colour-based imageretrieval system that is being evaluated. We report on our findings from a user study which suggests that the Mosaic Test overcomes the major drawbacks associated with existing image retrieval evaluation methods, by enabling users to reflect upon image selections and automatically measuring image relevance in a way that correlates with the perception of many human assessors. We therefore propose that the Mosaic Test be adopted as a standardised benchmark for evaluating and comparing colour-based image retrieval systems.

KW - benchmarking

KW - content-based image retrieval

KW - image databases

KW - image mosaic

KW - performance evaluation

UR - http://www.scopus.com/inward/record.url?scp=84891908499&partnerID=8YFLogxK

M3 - Conference contribution

T3 - CEUR workshop proceedings

SP - 23

EP - 26

BT - Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR)

PB - CEUR-WS.org

ER -

Plant W, Lumsden J, Nabney I. The mosaic test: benchmarking colour-based image retrieval systems using image mosaics. In Proceedings of 1st European Workshop on Human-Computer Interaction and Information Retrieval (EuroHCIR). CEUR-WS.org. 2011. p. 23-26. (CEUR workshop proceedings).