Abstract

The Indonesian language is spoken by almost 200 million people and is the 10th most spoken language in the world, but it is under-represented in NLP (Natural Language Processing) research. A sparsity of language resources has hampered previous work on Indonesian. The Transformer is a new architecture rapidly becoming dominant for NLP, surpassing alternatives like convolutional and recurrent neural networks. T5 (Text-to-Text Transfer Transformer) is a Transformer model that converts all text-based language problems to text-to-text format for English. The multilingual variant is mT5 (multilingual T5), which has shown promising results on many NLP tasks across languages. However, the size of this multilingual model is a drawback for its application in real production applications, which sometimes require only one language. In this study, the mT5 model was adapted for only one language, Indonesian, resulting in a pre-trained T5 model that was specific only for Indonesian with a smaller size. For performance comparison, we fine-tuned this model and the mT5 model to the Sentiment Analysis (SA), Question Generation (QG), and Question Answering (QA) tasks with the exact mechanism and dataset. Fine-tuned model based on our model achieved 77.18% accuracy on SA, 8% higher than the mT5based model, and obtained nearly the same score as the mT5based model on QG and QA. The results confirm that it is possible to produce a smaller pre-trained model that maintains comparable yields while reducing the model size by up to 58%. In addition, the resulting model requires less memory, loads faster, and inference times faster.

Original languageEnglish
Title of host publicationProceeding - IEEE 9th Information Technology International Seminar, ITIS 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9798350306835
DOIs
Publication statusPublished - 2023
Event9th IEEE Information Technology International Seminar, ITIS 2023 - Batu Malang, Indonesia
Duration: 18 Oct 202320 Oct 2023

Publication series

NameProceeding - IEEE 9th Information Technology International Seminar, ITIS 2023

Conference

Conference9th IEEE Information Technology International Seminar, ITIS 2023
Country/TerritoryIndonesia
CityBatu Malang
Period18/10/2320/10/23

Keywords

  • Model Compression
  • Pre-trained Model
  • Question Answering
  • Question Generation
  • Sentiment Analysis
  • Transformer

Fingerprint

Dive into the research topics of 'Adaptation of Multilingual T5 Transformer for Indonesian Language'. Together they form a unique fingerprint.

Cite this