The Problem

People with Alzheimer’s often lose track of a conversation within minutes: appointment times, who visited, what was said. Notebooks and reminder apps don’t help much, since they require the user to remember to write things down and remember to look them up later. We wanted a tool that needed neither: something that listens to a conversation and lets the person or their caregiver ask about it afterward in plain language.

This created two problems at once. Usability had to work for seniors with little tech experience, so the interface needed large text, few buttons, and nothing to configure. Accuracy had to be higher than usual, since a confident wrong answer, like an invented appointment time, is worse than no answer in a memory-care setting.

Note: this was built as a science fair prototype and tested with volunteer users, not deployed to real patients. You can also think of it as an automatic note-taker you can talk to that doesn’t interfere with your conversation, though that wasn’t the original intent.

Initial Approach

The app had three core responsibilities:

1. Listen: a Flask and Socket.IO backend streams microphone audio through the Google Web Speech API, transcribing live and pushing updates to the browser over websocket.
2. Understand: OpenAI GPT models turn the conversation into short, readable notes shown on screen during the session.
3. Recall: a separate page lets the user ask a question and get an answer based on what was actually said.

The interface stayed deliberately simple: a large transcript panel, a few clearly labelled buttons, nothing to configure. In testing with volunteer users, this version retrieved key details from past conversations with 94% accuracy.

Idea Evolution

The original award-winning version proved the concept, but its recall logic was severely limited. It kept the full conversation in memory, and for every question, sent the entire history to the model to search through. That works in a demo, but not as a product: data is lost on restart, it eventually exceeds the model’s input limit, and it gets slower, costlier, and less accurate the more someone talks.

After the competition, I rebuilt the recall engine as a Retrieval-Augmented Generation (RAG) pipeline:

This keeps answers more accurate and keeps cost roughly constant no matter how large the history gets.

Key Challenge: Reliable Speech Recall

The central challenge in the rebuild was turning live speech into something the app could answer questions about accurately later.

Listening and storing. Natural speech comes in bursts with pauses, and elderly or hesitant speakers pause a lot. Storing whatever the speech recognizer returned produced short, broken fragments that meant nothing on their own. To fix this, a two-timer system was used to group speech into complete units:

Two timers do two jobs. The short one (~1.5s) marks the end of a phrase so the audio gets transcribed in small clips, which the speech API handles more accurately, and a failed clip only loses one phrase instead of a whole thought. The long one waits for real silence in the room; once it’s quiet long enough, the thought is done and the phrases since the last pause get joined into one stored unit.

One timer alone doesn’t work: only-long makes clips too big for the API, only-short never knows when a thought ends. Together they give short, reliable clips stitched into one complete, searchable thought.

Recalling. The original in-memory approach broke down as history grew. Rebuilding recall as a retrieval step, shown in the diagram in Idea Evolution, meant only the most relevant stored pieces are pulled for any given question, keeping answers accurate regardless of how much has been said.

Outcome

Beyond the numbers, it delivered a working real-time app usable by non-technical seniors. After the competition, the recall system was rebuilt into an embedding-based RAG pipeline that stays accurate and affordable as history grows, with added error handling and session summaries.

Possible next steps:

• Per-user accounts with private histories.
• Extending the idea into an in-meeting notetaker or general conversation recall tool.

Personal Takeaways