Extractive summarization of documents with images based on multi-modal RNN

Jingqiang Chen; Hai Zhuge

doi:10.1016/j.future.2019.04.045

Extractive summarization of documents with images based on multi-modal RNN

Jingqiang Chen, Hai Zhuge^*

^*Corresponding author for this work

Nanjing University of Posts and Telecommunications

Research output: Contribution to journal › Article › peer-review

35 Citations (SciVal)

105 Downloads (Pure)

Abstract

Rapid growth of multi-modal documents containing images on the Internet expresses strong demand on multi-modal summarization. The challenge is to create a computing method that can uniformly process text and image. Deep learning provides basic models for meeting this challenge. This paper treats extractive multi-modal summarization as a classification problem and proposes a sentence–image classification method based on the multi-modal RNN model. Our method encodes words and sentences with the hierarchical RNN models and encodes the ordered image set with the CNN model and the RNN model, and then calculates the selection probability of sentences and the sentence–image alignment probability through a logistic classifier taking text coverage, text redundancy, image set coverage, and image set redundancy as features. Two methods are proposed to compute the image set redundancy feature by combining the important scores of sentences and the hidden sentence–image alignment. Experiments on the extended DailyMail corpora constructed by collecting images and captions from the Web show that our method outperforms 11 baseline text summarization methods and that adopting the two image-related features in the classification method can improve text summarization. Our method is able to mine the hidden sentence–image alignments and to create informative well-aligned multi-modal summaries.

Original language	English
Pages (from-to)	186-196
Number of pages	11
Journal	Future Generation Computer Systems
Volume	99
Early online date	25 Apr 2019
DOIs	https://doi.org/10.1016/j.future.2019.04.045
Publication status	Published - 1 Oct 2019

Bibliographical note

© 2019, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/

Funding: National Natural Science Foundation of China (No. 61806101, No. 61876048, No. 61602256, No. 61876091), and the Open Foundation of Key Laboratory of Intelligent Information Processing, ICT, CAS, China (IIP2019-2).

Funding

The research was sponsored by the National Natural Science Foundation of China (No. 61806101 , No. 61876048 , No. 61602256 , No. 61876091 ), and the Open Foundation of Key Laboratory of Intelligent Information Processing, ICT, CAS, China ( IIP2019-2 ). Professor Hai Zhuge is the corresponding author of this paper. Jingqiang Chen is a Lecturer in the School of Computer Science, Nanjing University of Posts and Telecommunications, China. He received the Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences, in 2014. His current research interests include Semantic Link Network and text summarization. Hai Zhuge is a Distinguished Scientist of the ACM and a Fellow of British Computer Society. He has made a systematic contribution to semantics and knowledge modeling through lasting fundamental research on the Semantic Link Network and the Resource Space Model based on multi-dimensional methodology. He is leading research toward Cyber--Physical Society through methodological, theoretical and technical innovation. He gave 17 keynotes at international conferences and invited lectures in universities of many countries as a Distinguished Speaker of the ACM. As a chair in computer science, he leads the International Research Network on Cyber–Physical–Social Intelligence consisting of Aston University, Guangzhou University, KLIIP at Institute of Computing Technology in Chinese Academy of Sciences, and University of Chinese Academy of Sciences. He was a Distinguished Visiting Fellow of Royal Academy of Engineering. He is the author of four monographs: Cyber–Physical–Social Intelligence on Human–Machine–Nature Symbiosis (Springer, 2019), Multi-Dimensional Summarization in Cyber–Physical Society (Morgan Kaufmann, 2016), The Knowledge Grid: Toward Cyber–Physical Society (World Scientific, 2012, 2nd edition), and The Web Resource Space Model (Springer, 2008). He was an associate editor of the Future Generation Computer Systems. He is serving as an associate editor of the IEEE Intelligent Systems. Homepage: http://www.knowledgegrid.net/ h.zhuge .

Keywords

Document summarization
Extractive summarization
Multi-modal summarization
RNN
Summarization

Access to Document

10.1016/j.future.2019.04.045

Extractive summarization of documents with images based on multi-modal RNN
© 2019, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/
Accepted author manuscript, 879 KBLicence: CC BY-NC-ND 3.0

Cite this

@article{2d4af9bec97e4be9b031b7d5e6402de9,

title = "Extractive summarization of documents with images based on multi-modal RNN",

abstract = "Rapid growth of multi-modal documents containing images on the Internet expresses strong demand on multi-modal summarization. The challenge is to create a computing method that can uniformly process text and image. Deep learning provides basic models for meeting this challenge. This paper treats extractive multi-modal summarization as a classification problem and proposes a sentence–image classification method based on the multi-modal RNN model. Our method encodes words and sentences with the hierarchical RNN models and encodes the ordered image set with the CNN model and the RNN model, and then calculates the selection probability of sentences and the sentence–image alignment probability through a logistic classifier taking text coverage, text redundancy, image set coverage, and image set redundancy as features. Two methods are proposed to compute the image set redundancy feature by combining the important scores of sentences and the hidden sentence–image alignment. Experiments on the extended DailyMail corpora constructed by collecting images and captions from the Web show that our method outperforms 11 baseline text summarization methods and that adopting the two image-related features in the classification method can improve text summarization. Our method is able to mine the hidden sentence–image alignments and to create informative well-aligned multi-modal summaries.",

keywords = "Document summarization, Extractive summarization, Multi-modal summarization, RNN, Summarization",

author = "Jingqiang Chen and Hai Zhuge",

note = "{\textcopyright} 2019, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ Funding: National Natural Science Foundation of China (No. 61806101, No. 61876048, No. 61602256, No. 61876091), and the Open Foundation of Key Laboratory of Intelligent Information Processing, ICT, CAS, China (IIP2019-2). ",

year = "2019",

month = oct,

day = "1",

doi = "10.1016/j.future.2019.04.045",

language = "English",

volume = "99",

pages = "186--196",

journal = "Future Generation Computer Systems",

issn = "0167-739X",

publisher = "Elsevier",

}

TY - JOUR

T1 - Extractive summarization of documents with images based on multi-modal RNN

AU - Chen, Jingqiang

AU - Zhuge, Hai

N1 - © 2019, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ Funding: National Natural Science Foundation of China (No. 61806101, No. 61876048, No. 61602256, No. 61876091), and the Open Foundation of Key Laboratory of Intelligent Information Processing, ICT, CAS, China (IIP2019-2).

PY - 2019/10/1

Y1 - 2019/10/1

N2 - Rapid growth of multi-modal documents containing images on the Internet expresses strong demand on multi-modal summarization. The challenge is to create a computing method that can uniformly process text and image. Deep learning provides basic models for meeting this challenge. This paper treats extractive multi-modal summarization as a classification problem and proposes a sentence–image classification method based on the multi-modal RNN model. Our method encodes words and sentences with the hierarchical RNN models and encodes the ordered image set with the CNN model and the RNN model, and then calculates the selection probability of sentences and the sentence–image alignment probability through a logistic classifier taking text coverage, text redundancy, image set coverage, and image set redundancy as features. Two methods are proposed to compute the image set redundancy feature by combining the important scores of sentences and the hidden sentence–image alignment. Experiments on the extended DailyMail corpora constructed by collecting images and captions from the Web show that our method outperforms 11 baseline text summarization methods and that adopting the two image-related features in the classification method can improve text summarization. Our method is able to mine the hidden sentence–image alignments and to create informative well-aligned multi-modal summaries.

AB - Rapid growth of multi-modal documents containing images on the Internet expresses strong demand on multi-modal summarization. The challenge is to create a computing method that can uniformly process text and image. Deep learning provides basic models for meeting this challenge. This paper treats extractive multi-modal summarization as a classification problem and proposes a sentence–image classification method based on the multi-modal RNN model. Our method encodes words and sentences with the hierarchical RNN models and encodes the ordered image set with the CNN model and the RNN model, and then calculates the selection probability of sentences and the sentence–image alignment probability through a logistic classifier taking text coverage, text redundancy, image set coverage, and image set redundancy as features. Two methods are proposed to compute the image set redundancy feature by combining the important scores of sentences and the hidden sentence–image alignment. Experiments on the extended DailyMail corpora constructed by collecting images and captions from the Web show that our method outperforms 11 baseline text summarization methods and that adopting the two image-related features in the classification method can improve text summarization. Our method is able to mine the hidden sentence–image alignments and to create informative well-aligned multi-modal summaries.

KW - Document summarization

KW - Extractive summarization

KW - Multi-modal summarization

KW - RNN

KW - Summarization

UR - http://www.scopus.com/inward/record.url?scp=85064813348&partnerID=8YFLogxK

UR - https://www.sciencedirect.com/science/article/pii/S0167739X18326876?via%3Dihub

U2 - 10.1016/j.future.2019.04.045

DO - 10.1016/j.future.2019.04.045

M3 - Article

AN - SCOPUS:85064813348

SN - 0167-739X

VL - 99

SP - 186

EP - 196

JO - Future Generation Computer Systems

JF - Future Generation Computer Systems

ER -

Extractive summarization of documents with images based on multi-modal RNN

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this