Social media is one of the uses of crowdsourcing to gather vast amounts of information. The applications of incident detection using social media data are commonly focus on text analysis. Due to the ability of social media to capture variant data types such as text, voice, image, or video, the development of incident detection based on multimodal data is preferred. The use of multimodal data on incident detection is expected to improve the accuracy of prediction. This research aims to detect emergency incidents based on multimodal data streams from social media using deep learning methods. We compare several deep learning architectures that implement some neural network variants, namely Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM). We crawled data from Twitter API and labeled into three incident categories i.e. flood, traffic jam, and wildfire. The CNN and C-LSTM are used for text prediction, and the best performance obtained by C-LSTM and achieved 99.09% in the accuracy. The compared CNN models for image prediction are AlexNet, VGG16, VGG19, and SqueezeNet. The best performance obtained by VGG16 with data augmentation and achieved 99.08% in the accuracy. The incident detection results of multimodal data are obtained from the highest confidence level of text or image.

Original languageEnglish
Title of host publication7th International Conference on ICT for Smart Society
Subtitle of host publicationAIoT for Smart Society, ICISS 2020 - Proceeding
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665404228
Publication statusPublished - 19 Nov 2020
Event7th International Conference on ICT for Smart Society, ICISS 2020 - Virtual, Bandung, Indonesia
Duration: 19 Nov 202020 Nov 2020

Publication series

Name7th International Conference on ICT for Smart Society: AIoT for Smart Society, ICISS 2020 - Proceeding


Conference7th International Conference on ICT for Smart Society, ICISS 2020
CityVirtual, Bandung


  • Convolutional Neural Network
  • Deep learning
  • Incident detection
  • Long Short-Term Memory
  • Social media data


Dive into the research topics of 'Incident Detection based on Multimodal data from Social Media using Deep Learning Methods'. Together they form a unique fingerprint.

Cite this