A Deep Learning Approach to Automatic Caption Generation for News Images

Vishwash Batra, Yulan He, George Vogiatzis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Automatic caption generation of images has gained significant interest. It gives rise to a lot of interesting image-related applications.
For example, it could help in image/video retrieval and management of vast amount of multimedia data available on the Internet. It
could also help in development of tools that can aid visually impaired individuals in accessing multimedia content. In this paper, we
particularly focus on news images and propose a methodology for automatically generating captions for news paper articles consisting
of a text paragraph and an image. We propose several deep neural network architectures built upon Recurrent Neural Networks. Results
on a BBC News dataset show that our proposed approach outperforms a traditional method based on Latent Dirichlet Allocation using
both automatic evaluation based on BLEU scores and human evaluation.
LanguageEnglish
Title of host publicationThe 11th International Conference on Language Resources and Evaluation (LREC)
Publication statusPublished - 1 Jan 2019
EventThe 11th International Conference on Language Resources and Evaluation (LREC) - Miyazaki, Japan
Duration: 7 May 201812 May 2018
http://lrec2018.lrec-conf.org/en/

Conference

ConferenceThe 11th International Conference on Language Resources and Evaluation (LREC)
CountryJapan
CityMiyazaki
Period7/05/1812/05/18
Internet address

Fingerprint

Recurrent neural networks
Network architecture
Internet
Deep neural networks
Deep learning

Bibliographical note

The LREC 2018 Proceedings are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License

Keywords

  • Recurrent Neural Networks
  • Image caption generation
  • Deep learning
  • Order Embedding

Cite this

Batra, V., He, Y., & Vogiatzis, G. (2019). A Deep Learning Approach to Automatic Caption Generation for News Images. In The 11th International Conference on Language Resources and Evaluation (LREC)
Batra, Vishwash ; He, Yulan ; Vogiatzis, George. / A Deep Learning Approach to Automatic Caption Generation for News Images. The 11th International Conference on Language Resources and Evaluation (LREC). 2019.
@inproceedings{c5067f5679054fd2a1bc0bbe61bd026b,
title = "A Deep Learning Approach to Automatic Caption Generation for News Images",
abstract = "Automatic caption generation of images has gained significant interest. It gives rise to a lot of interesting image-related applications.For example, it could help in image/video retrieval and management of vast amount of multimedia data available on the Internet. Itcould also help in development of tools that can aid visually impaired individuals in accessing multimedia content. In this paper, weparticularly focus on news images and propose a methodology for automatically generating captions for news paper articles consistingof a text paragraph and an image. We propose several deep neural network architectures built upon Recurrent Neural Networks. Resultson a BBC News dataset show that our proposed approach outperforms a traditional method based on Latent Dirichlet Allocation usingboth automatic evaluation based on BLEU scores and human evaluation.",
keywords = "Recurrent Neural Networks, Image caption generation, Deep learning, Order Embedding",
author = "Vishwash Batra and Yulan He and George Vogiatzis",
note = "The LREC 2018 Proceedings are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License",
year = "2019",
month = "1",
day = "1",
language = "English",
isbn = "979-10-95546-00-9",
booktitle = "The 11th International Conference on Language Resources and Evaluation (LREC)",

}

Batra, V, He, Y & Vogiatzis, G 2019, A Deep Learning Approach to Automatic Caption Generation for News Images. in The 11th International Conference on Language Resources and Evaluation (LREC). The 11th International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan, 7/05/18.

A Deep Learning Approach to Automatic Caption Generation for News Images. / Batra, Vishwash; He, Yulan; Vogiatzis, George.

The 11th International Conference on Language Resources and Evaluation (LREC). 2019.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A Deep Learning Approach to Automatic Caption Generation for News Images

AU - Batra, Vishwash

AU - He, Yulan

AU - Vogiatzis, George

N1 - The LREC 2018 Proceedings are licensed under a Creative Commons Attribution-NonCommercial 4.0 International License

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Automatic caption generation of images has gained significant interest. It gives rise to a lot of interesting image-related applications.For example, it could help in image/video retrieval and management of vast amount of multimedia data available on the Internet. Itcould also help in development of tools that can aid visually impaired individuals in accessing multimedia content. In this paper, weparticularly focus on news images and propose a methodology for automatically generating captions for news paper articles consistingof a text paragraph and an image. We propose several deep neural network architectures built upon Recurrent Neural Networks. Resultson a BBC News dataset show that our proposed approach outperforms a traditional method based on Latent Dirichlet Allocation usingboth automatic evaluation based on BLEU scores and human evaluation.

AB - Automatic caption generation of images has gained significant interest. It gives rise to a lot of interesting image-related applications.For example, it could help in image/video retrieval and management of vast amount of multimedia data available on the Internet. Itcould also help in development of tools that can aid visually impaired individuals in accessing multimedia content. In this paper, weparticularly focus on news images and propose a methodology for automatically generating captions for news paper articles consistingof a text paragraph and an image. We propose several deep neural network architectures built upon Recurrent Neural Networks. Resultson a BBC News dataset show that our proposed approach outperforms a traditional method based on Latent Dirichlet Allocation usingboth automatic evaluation based on BLEU scores and human evaluation.

KW - Recurrent Neural Networks

KW - Image caption generation

KW - Deep learning

KW - Order Embedding

UR - http://www.lrec-conf.org/proceedings/lrec2018/index.html

M3 - Conference contribution

SN - 979-10-95546-00-9

BT - The 11th International Conference on Language Resources and Evaluation (LREC)

ER -

Batra V, He Y, Vogiatzis G. A Deep Learning Approach to Automatic Caption Generation for News Images. In The 11th International Conference on Language Resources and Evaluation (LREC). 2019