A Gated Vision–language Architecture for Accurate Offline Medicine Identification and Information Interpretation Using Generative AI

This study presents a gated vision-language architecture for offline medicine identification and explanation in resource-constrained settings. The framework combines YOLO-based object detection, OCR with fuzzy matching for packaging recognition, and ResNet-based visual embeddings with cosine similarity for pill identification. Verified medicine attributes are retrieved from a local registry and passed to a locally deployed Phi-3 model used only as a controlled explanation layer. An offline Indonesian text-to-speech module supports elderly and visually impaired users. Experimental results show 0.905 packaging identification accuracy, 0.83 Top-1 pill recognition accuracy, and a mean cosine similarity of 0.87 for correct pill matches. Under the adaptive dual-path protocol, the system identified 350 of 385 cases, corresponding to 0.91 end-to-end accuracy. Explanation quality on 350 recognized cases reached 97.1% factual consistency and 96.9% completeness, with no hallucination observed within the evaluated sample. End-to-end latency was 16.72 s for packaging and 15.86 s for pill inputs.