Multimodal CNN Pedestrian Classification: A Study on Combining LIDAR and Camera Data

Gledson Melotti, Cristiano Premebida, Nuno M. M. Da S. Goncalves, Urbano J. C. Nunes, Diego R. Faria

Research output: Chapter in Book/Report/Conference proceedingConference publication

Abstract

This paper presents a study on pedestrian classification based on deep learning using data from a monocular camera and a 3D LIDAR sensor, separately and in combination. Early and late multi-modal sensor fusion approaches are revisited and compared in terms of classification performance. The problem of pedestrian classification finds applications in advanced driver assistance system (ADAS) and autonomous driving, and it has regained particular attention recently because, among other reasons, safety involving self-driving vehicles. Convolutional Neural Networks (CNN) is used in this work as classifier in distinct situations: having a single sensor data as input, and by combining data from both sensors in the CNN input layer. Range (distance) and intensity (reflectance) data from LIDAR are considered as separate channels, where data from the LIDAR sensor is feed to the CNN in the form of dense maps, as the result of sensor coordinate transformation and spatial filtering; this allows a direct implementation of the same CNN-based approach on both sensors data. In terms of late-fusion, the outputs from individual CNNs are combined by means of learning and non-learning approaches. Pedestrian classification is evaluated on a ‘binary classification’ dataset created from the KITTI Vision Benchmark Suite, and results are shown for each sensor-modality individually, and for the fusion strategies.
Original languageEnglish
Title of host publication2018 21st International Conference on Intelligent Transportation Systems (ITSC)
PublisherIEEE
Pages3138-3143
ISBN (Electronic)978-1-7281-0323-5
ISBN (Print)978-1-7281-0321-1
DOIs
Publication statusPublished - 10 Dec 2018
Event2018 IEEE International Conference on Intelligent Transportation Systems (ITSC) - Maui, HI, USA
Duration: 4 Nov 20187 Nov 2018

Publication series

Name2018 21st International Conference on Intelligent Transportation Systems (ITSC)
PublisherIEEE
ISSN (Print)2153-0009
ISSN (Electronic)2153-0017

Conference

Conference2018 IEEE International Conference on Intelligent Transportation Systems (ITSC)
Period4/11/187/11/18

Fingerprint Dive into the research topics of 'Multimodal CNN Pedestrian Classification: A Study on Combining LIDAR and Camera Data'. Together they form a unique fingerprint.

  • Cite this

    Melotti, G., Premebida, C., Goncalves, N. M. M. D. S., Nunes, U. J. C., & Faria, D. R. (2018). Multimodal CNN Pedestrian Classification: A Study on Combining LIDAR and Camera Data. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC) (pp. 3138-3143). (2018 21st International Conference on Intelligent Transportation Systems (ITSC)). IEEE. https://doi.org/10.1109/ITSC.2018.8569666