Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired

Indri Aulia Ulfa; Hendra Kusuma; Totok Mujiono

doi:10.1109/ISITIA71267.2026.11642239

Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired

Indri Aulia Ulfa
, Hendra Kusuma
, Totok Mujiono

Laboratory of Microelectronics and Embedded Systems

Department of Electrical Engineering
Institut Teknologi Sepuluh Nopember

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

People with visual impairments face difficulties in accessing visual information, including accurately identifying medications. This study develops assistive technology in the form of an artificial intelligence-based medication recognition system to help users independently recognize and understand medication information. The system detects medication packaging in real-time using the YOLOv8 model with a precision of 0.998 and a recall of 0.997, resulting in an F1-score of approximately 0.998, then converts the data to TensorFlow Lite and presents the results in text and speech via Text-to-Speech (TTS). In addition, the system offers a voice-based Q&A feature using the OpenAI API and Firebase Realtime Database. It also includes a medication reminder feature that allows users to set a schedule for taking their medication via voice commands, which are processed into automatic alarms accompanied by notifications, vibrations, and voice prompts.

Original language	English
Title of host publication	27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	940-945
Number of pages	6
ISBN (Electronic)	9798319521576
DOIs	https://doi.org/10.1109/ISITIA71267.2026.11642239
Publication status	Published - 2026
Event	27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026 - Hybrid, Bali, Indonesia Duration: 22 Jul 2026 → 24 Jul 2026

Conference

Conference	27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026
Country/Territory	Indonesia
City	Hybrid, Bali
Period	22/07/26 → 24/07/26

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 7 Affordable and Clean Energy

Keywords

Firebase Database
OpenAI API
YOLOv8
assistive technology
visually impaired

Access to Document

10.1109/ISITIA71267.2026.11642239

Cite this

Ulfa, I. A., Kusuma, H., & Mujiono, T. (2026). Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired. In 27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026 (pp. 940-945). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISITIA71267.2026.11642239

@inproceedings{c6e247f2753540d382324a31fc9efa79,

title = "Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired",

abstract = "People with visual impairments face difficulties in accessing visual information, including accurately identifying medications. This study develops assistive technology in the form of an artificial intelligence-based medication recognition system to help users independently recognize and understand medication information. The system detects medication packaging in real-time using the YOLOv8 model with a precision of 0.998 and a recall of 0.997, resulting in an F1-score of approximately 0.998, then converts the data to TensorFlow Lite and presents the results in text and speech via Text-to-Speech (TTS). In addition, the system offers a voice-based Q\&A feature using the OpenAI API and Firebase Realtime Database. It also includes a medication reminder feature that allows users to set a schedule for taking their medication via voice commands, which are processed into automatic alarms accompanied by notifications, vibrations, and voice prompts.",

keywords = "Firebase Database, OpenAI API, YOLOv8, assistive technology, visually impaired",

author = "Ulfa, \{Indri Aulia\} and Hendra Kusuma and Totok Mujiono",

note = "Publisher Copyright: {\textcopyright} 2026 IEEE.; 27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026 ; Conference date: 22-07-2026 Through 24-07-2026",

year = "2026",

doi = "10.1109/ISITIA71267.2026.11642239",

language = "English",

pages = "940--945",

booktitle = "27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

}

Ulfa, IA, Kusuma, H & Mujiono, T 2026, Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired. in 27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026. Institute of Electrical and Electronics Engineers Inc., pp. 940-945, 27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026, Hybrid, Bali, Indonesia, 22/07/26. https://doi.org/10.1109/ISITIA71267.2026.11642239

Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired. / Ulfa, Indri Aulia; Kusuma, Hendra ; Mujiono, Totok.
27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026. Institute of Electrical and Electronics Engineers Inc., 2026. p. 940-945.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired

AU - Ulfa, Indri Aulia

AU - Kusuma, Hendra

AU - Mujiono, Totok

PY - 2026

Y1 - 2026

N2 - People with visual impairments face difficulties in accessing visual information, including accurately identifying medications. This study develops assistive technology in the form of an artificial intelligence-based medication recognition system to help users independently recognize and understand medication information. The system detects medication packaging in real-time using the YOLOv8 model with a precision of 0.998 and a recall of 0.997, resulting in an F1-score of approximately 0.998, then converts the data to TensorFlow Lite and presents the results in text and speech via Text-to-Speech (TTS). In addition, the system offers a voice-based Q&A feature using the OpenAI API and Firebase Realtime Database. It also includes a medication reminder feature that allows users to set a schedule for taking their medication via voice commands, which are processed into automatic alarms accompanied by notifications, vibrations, and voice prompts.

AB - People with visual impairments face difficulties in accessing visual information, including accurately identifying medications. This study develops assistive technology in the form of an artificial intelligence-based medication recognition system to help users independently recognize and understand medication information. The system detects medication packaging in real-time using the YOLOv8 model with a precision of 0.998 and a recall of 0.997, resulting in an F1-score of approximately 0.998, then converts the data to TensorFlow Lite and presents the results in text and speech via Text-to-Speech (TTS). In addition, the system offers a voice-based Q&A feature using the OpenAI API and Firebase Realtime Database. It also includes a medication reminder feature that allows users to set a schedule for taking their medication via voice commands, which are processed into automatic alarms accompanied by notifications, vibrations, and voice prompts.

KW - Firebase Database

KW - OpenAI API

KW - YOLOv8

KW - assistive technology

KW - visually impaired

UR - https://www.scopus.com/pages/publications/105048317721

U2 - 10.1109/ISITIA71267.2026.11642239

DO - 10.1109/ISITIA71267.2026.11642239

M3 - Conference contribution

AN - SCOPUS:105048317721

SP - 940

EP - 945

BT - 27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 27th International Seminar on Intelligent Technology and Its Applications, ISITIA 2026

Y2 - 22 July 2026 through 24 July 2026

ER -

Development of a Visual-Based Medication Packaging Recognition System Supported by Generative AI for the Visually Impaired

Abstract

Conference

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this