Gesture recognition for Indonesian Sign Language Systems (ISLS) using multimodal sensor leap motion and myo armband controllers based-on naïve bayes classifier

Khamid*, Adhi Dharma Wibawa, Surya Sumpeno

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

17 Citations (Scopus)

Abstract

Indonesian Sign Language System (ISLS) has been used widely by Indonesian for translating the sign language of disabled people to many applications, including education or entertainment. ISLS consists of static and dynamic gestures in representing words or sentences. However, individual variations in performing sign language have been a big challenge especially for developing automatic translation. The accuracy of recognizing the signs will decrease linearly with the increase of variations of gestures. This research is targeted to solve these issues by implementing the multimodal methods: Leap motion and Myo armband controllers (EMG electrodes). By combining these two data and implementing Naïve Bayes classifier, we hypothesized that the accuracy of gesture recognition system for ISLS then can be increased significantly. The data streams captured from hand-poses were based on time-domain series method which will warrant the generated data synchronized accurately. The selected features for leap motion data would be based on fingers positions, angles, and elevations, while for the Myo armband would be based on electrical signal generated by eight channels of EMG electrodes relevant to the activities of linked finger's and forearm muscles. This study will investigate the accuracy of gesture recognition by using either single modal or multimodal for translating Indonesian sign language. For multimodal strategy, both features datasets were merged into a single dataset which was then used for generating a model for each hand gesture. The result showed that there was a significant improvement on its accuracy, from 91% for single modal using leap motion to 98% for multi-modal (combined with Myo armband). The confusion matrix of multimodal method also showed better performance than the single-modality. Finally, we concluded that the implementation of multi-modal controllers for ISLS's gesture recognition showed better accuracy and performance compared of single modality of using only leap motion controller.

Original languageEnglish
Title of host publicationProceedings - 2017 International Conference on Soft Computing, Intelligent System and Information Technology
Subtitle of host publicationBuilding Intelligence Through IOT and Big Data, ICSIIT 2017
EditorsHenry Novianus Palit, Leo Willyanto Santoso
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-6
Number of pages6
ISBN (Electronic)9781467398992
DOIs
Publication statusPublished - 2 Jul 2017
Event5th International Conference on Soft Computing, Intelligent System and Information Technology, ICSIIT 2017 - Denpasar, Bali, Indonesia
Duration: 26 Sept 201729 Sept 2017

Publication series

NameProceedings - 2017 International Conference on Soft Computing, Intelligent System and Information Technology: Building Intelligence Through IOT and Big Data, ICSIIT 2017
Volume2018-January

Conference

Conference5th International Conference on Soft Computing, Intelligent System and Information Technology, ICSIIT 2017
Country/TerritoryIndonesia
CityDenpasar, Bali
Period26/09/1729/09/17

Keywords

  • Indonesian sign language systems
  • Myo armband
  • Naïve Bayes
  • dynamic hand gestures
  • leap motion
  • sign language

Fingerprint

Dive into the research topics of 'Gesture recognition for Indonesian Sign Language Systems (ISLS) using multimodal sensor leap motion and myo armband controllers based-on naïve bayes classifier'. Together they form a unique fingerprint.

Cite this