---

Table of Contents

1. YAKSOK - A Medicine Identification App for the Visually Impaired
2. Problems with Existing Solutions
   1. The Limits of OCR
   2. A Frustrating User Experience
3. Design
4. Core Features
   1. Medicine Identification
   2. Easy-to-Understand Medication Guides
   3. Expiration Date Detection
   4. Photo Description
5. On-Device AI
6. Demo
7. Closing Thoughts

---

An on-device AI medicine identification app built for the visually impaired.
👉 한국어 버전

---

YAKSOK - A Medicine Identification App for the Visually Impaired

From identifying medicine and expiration dates to easy-to-understand AI medication guides, meet YAKSOK.
👉 Homepage
👉 App Store
👉 Play Store

YAKSOK was developed as part of the 3rd Seoul Youth Planning Volunteer Corps, a joint project between the Seoul Volunteer Center and the National Evidence-based Healthcare Collaborating Agency (NECA).

How do visually impaired people identify and take their medicine? As of 2026, out of roughly 40,000 types of medicine in Korea, only 0.1% carry Braille labeling. Even then, the Braille only spells out the drug’s name, leaving dosage instructions and expiration dates completely inaccessible.

This risk is compounding into a society-wide problem alongside rapid population aging. From older adults with declining eyesight to people taking ten or more medications a day, information that is overwhelmingly text-based is cutting off access to medicine for those who need it most and threatening their health and safety.

A survey of 41 visually impaired people confirmed exactly this difficulty. Most respondents reported significant trouble with identifying medicine, understanding dosage instructions, and checking expiration dates.

Even more concerning: more than half of respondents said they had either taken the wrong medicine or given up on taking medicine altogether because of these problems.

Problems with Existing Solutions

So how are visually impaired people currently coping with these difficulties?

According to the survey, the most common responses were “storing medicine in a specific spot (65.9%)” and “getting help from family or a personal care assistant (61%).”

In other words, people either rely on the personal habit of always putting medicine back in the same place, or find it very difficult to tell medicines apart on their own without help. Many try smartphone apps, rubber bands or tape markers, or relying on touch and memory, but none of these have become a safe, fundamental solution.

One notable finding was that using a smartphone’s text-reading feature (41.5%) was also a fairly common response. That raised a question: if a phone can read text aloud, it should also be able to read expiration dates — so why was checking the expiration date the most commonly cited pain point? To find out, we ran our own technical verification.

The Limits of OCR

We tested OCR (Optical Character Recognition) against 161 real medicine OCR samples, and 20% could not be recognized at all. Most failures came from glare on plastic packaging or embossed dates with unclear edges, which meant overcoming this technical limitation would require far more precise recognition technology.

A Frustrating User Experience

An 80% recognition rate might sound “good enough” at first glance. But even setting recognition accuracy aside, existing OCR apps have a UX that’s genuinely difficult for visually impaired users. There are three reasons why.

First, taking the photo itself is hard. Where the camera is pointed, whether the subject is at the right distance, whether it’s in focus — all of this information depends on sight. For a visually impaired person, the very first step of taking a photo is already a major barrier.

Second, these apps don’t read only the information you want. Existing OCR apps read everything printed in the photo, start to finish. The user only wants the expiration date, but they have to sift through a flood of unnecessary spoken information to find it, which is exhausting.

Third, the position of the expiration date and barcode isn’t standardized. Every medicine box prints this information in a different spot, with no Braille or tactile marking to guide the way. Users are left to photograph all six sides of the box one by one, listening to voice feedback each time — repeating the process up to six times.

Together, the difficulty of taking photos, the noise of unnecessary readouts, and the lack of a standard label position create an extreme accessibility barrier for visually impaired users — making a dedicated solution urgently needed.

Design

YAKSOK’s design started from a single principle: “make it easy to see.” Among visually impaired users, many have low vision rather than total blindness, and small text or a dim screen is also a major barrier for older adults and people on multiple medications. So instead of visual flair, we made readability the first design priority.

The design draws inspiration from simple, typography-driven magazines, letting the text itself become the center of the screen. We stripped away decoration and used font size and weight alone to convey information hierarchy, and we support variable fonts for low-vision users.

Colors were defined with a contrast ratio meeting or exceeding WCAG AA (4.5:1), and the app supports both light and dark mode. Since people take medicine in all kinds of environments — from spaces with heavy glare to dark rooms — it was important that text stay legible no matter the lighting.

For screens that are hard to navigate with a screen reader, we added a voice guide button, so onboarding can be completed just by pressing one button and listening.

To make sure screen reader users never lose their bearings through sound alone, we focused on three things.

Preventing TTS and screen reader audio from overlapping

• In our previous project Guita, we avoided overlapping audio by simply disabling TTS whenever the screen reader was on.
• This time, TTS voice guidance is a core feature, so instead we stop any playing TTS the moment screen reader focus moves.

Keeping accessibility labels as concise as possible

• Screen reader users must listen to information one piece at a time, so longer labels mean a longer wait before reaching the information they actually want.
• We kept only the essential information to minimize navigation fatigue.

Fixing focus order on every page transition

• On navigating to a new page, focus always moves in the order title → back button.
• This tells the user what screen they’re on first, then lets them predict exactly where to go to return.

Core Features

On top of this refined interface, we built features that all trace back to a single question: “How can someone take their medicine safely and comfortably, without needing anyone else’s help?” From that question came YAKSOK’s four core features:

1. Medicine identification via barcode scan
2. Fast, accurate expiration date detection
3. Easy-to-understand medication guides
4. Photo description

Medicine Identification

Easy-to-Understand Medication Guides

Expiration Date Detection

Photo Description

On-Device AI

To support both Android and iOS, the app was built with Flutter. For the on-device AI model, we chose Gemma4 E2B, a multimodal model that handles both images and text. Since features like expiration date detection and photo description depend directly on image input, multimodal support was the deciding factor in model selection.

Our own testing showed that running the AI model smoothly requires at least 6GB of memory. So on devices with less than 6GB, the app falls back to OCR and the Gemini API instead of on-device AI.

The model file is also fairly large at about 2.5GB, and requiring users to keep the app open until the download finishes would be a real barrier. To solve this, we implemented background downloads that continue even after leaving the app, along with resumable downloads so a dropped connection doesn’t force users to start over.

Adopting on-device AI gave us the following benefits:

1. Offline support - AI features work without an internet connection.
2. Lower cost - No AI token costs, so the feature can be offered to users for free.
3. Privacy - Photos taken by the user never leave the device.

Demo

Closing Thoughts

The biggest lesson from building YAKSOK is that teaching people how to use an app matters just as much as building the app itself. Not every visually impaired person is comfortable using a screen reader, so learning a brand-new app without anyone’s help was far from easy. That’s why we made onboarding accessible with a single voice guide button, and why we also held demo sessions at a shelter for the visually impaired to walk people through YAKSOK in person.

This project was carried out together with the Seoul Volunteer Center and NECA (National Evidence-based Healthcare Collaborating Agency). NECA supported our demo sessions with volunteers and donated 90 bottles of hand sanitizer, and also published a blog post promoting the app. Thanks to everyone who took part, this app was able to reach the people who actually need it.

At the demo sessions, what visually impaired participants struggled with most wasn’t operating the app itself, but finding the non-standardized barcode and expiration date positions on each medicine box. So during the sessions, we guided people by attaching tactile stickers at the barcode and expiration date locations to make them easier to find. No matter how well the app recognizes things, if you can’t tell where to point the camera in the first place, you’re stuck at the very first step. This isn’t something any single app can solve on its own — it made us hope that a social standard for label placement will eventually emerge.

It was also great to get hands-on experience with on-device AI in this project. The technology is still constrained by device limitations today, but it’s steadily gaining traction, and as devices and models keep improving, we expect more users will be able to safely get AI’s help even while offline.

We hope YAKSOK can be of some help to visually impaired people, older adults, and people on multiple medications in taking their medicine safely. (Since YAKSOK can only identify medicines approved for sale in Korea, it is currently available only on the Korean app stores.)

YAKSOK - A Medicine Identification App for the Visually Impaired
👉 Homepage
👉 App Store
👉 Play Store