Interaction Systems ; How Voice Becomes a Real Conversation

The interaction layer is where a model stops feeling like software and starts feeling present. The point is not only to transcribe speech correctly, but to keep the loop alive while a person is still mid-thought: hear continuously, form partial understanding early, carry the active turn forward, and shape a final reply before the moment goes cold.

That loop is the actual product. Users do not experience speech recognition, model routing, and synthesis as separate features. They only feel whether the system stays with them through a turn or disappears between steps. If the loop breaks, even a strong final answer feels delayed. If the loop stays alive, the whole exchange feels attentive.

Voice interaction is not one feature sitting on top of a model. It is a full stack of decisions about timing, continuity, and response formation. The reason some systems feel alive and others feel mechanical is usually not the model alone. It is whether the interaction layer can keep the conversational thread active from the first spoken syllable to the final reply.

This is the layer that translates human rhythm into machine rhythm. It decides how quickly sound enters the system, when partial understanding becomes useful, how much context should be active before the model begins deeper work, and how the final response should return without feeling detached from the original turn.

Overview

The interaction layer sits between raw speech and finished language. Its job is to convert a continuous human act into something the model can work with, without losing the rhythm of the act itself. That means the system has to listen while the user is speaking, begin understanding before the sentence is over, and keep enough state alive that the final reply still belongs to the same moment.

In practice, this means handling more than transcription. It means preserving turn state, managing interruptions, deciding when enough meaning has accumulated to start deeper work, and making sure the response pipeline begins before the audio stream has fully cooled down.

How Speech Enters the System

The client should only do the minimal, reliable part: capture microphone audio, keep it in a stable format, and stream it continuously to one public backend. That backend receives small chunks, appends them into an active per-user stream, and decides when enough audio has arrived to begin useful inference.

This matters because the first stage is not about producing a perfect transcript instantly. It is about keeping the backend close enough to the turn that the rest of the stack can move with the user rather than after them.

The important thing here is that the backend starts to know something before it knows everything. That early knowledge is what makes the system feel quick. It can begin deciding whether this is a search request, a simple local answer, a follow-up to the prior turn, or something that will need retrieval or tool use later in the exchange.

How the Backend Holds the Turn

A voice interaction is not just audio in and audio out. The backend also has to hold the active shape of the turn: who the user is, which chat this belongs to, what recent state is already stored in Convex, what the assistant was doing before, and whether the system is still in the middle of a live exchange. Without this, every utterance becomes isolated and the assistant loses continuity even when the transcription is correct.

This is also the closest thing to “remembering” during a live interaction. It is not long-term autobiographical memory. It is short-lived, structured context: active turn state, recent messages, current task, unresolved references, and whatever partial transcript has already accumulated. That is what allows the model to respond as if it has stayed in the room.

How the Response Is Formed

Once enough signal exists, the backend stops being a listener and starts becoming a composer. It already has partial speech, recent context, and a sense of the user’s intent. From there it can decide whether the turn is simple enough for direct generation, whether additional tools or search are needed, and whether the final answer should come back as text, voice, or both.

The key is that the answer should not begin only after every upstream task has perfectly finished. The best systems let response formation begin as soon as the structure of the turn is clear enough. By the time the user stops speaking, the backend should already be partway through forming the reply.

This is where the interaction starts to feel intelligent rather than merely reactive. The system is not waiting passively for a completed transcript and then beginning from zero. It is building an answer through the turn, so the final response feels connected to the same conversational breath.

Architecture

The frontend should not know about STT URLs, TTS URLs, cloud provider keys, routing rules, or orchestration logic. It should sign the user in, create the app records it needs, and speak to one public backend. That backend can then manage everything else behind the curtain: stream intake, identity, Convex state, model calls, tools, and final response shaping.

This is not only cleaner operationally. It also protects the interaction itself. When the client has too many direct responsibilities, every branch and retry leaks into user-facing hesitation. A single public backend keeps the rhythm coherent because only one system is responsible for holding the turn from beginning to end.

Why It Reacts Faster

A fast interaction system is not one that calls every internal stage more often. It is one that begins the right stages earlier. Small audio chunks help because they move speech to the backend quickly. Buffered transcription helps because it gives the backend stable partial understanding instead of noisy fragments. Context loading helps because recent state is already ready when the model needs it. Streaming replies help because the answer can begin before the entire pipeline has closed.

In other words, responsiveness comes from overlap. Listening, understanding, routing, and response formation should happen in a staggered way rather than in four hard phases.

The Minimal Spec Under It

The timing details still matter, but mainly as support. At 16 kHz, 320 samples equal 20 ms and 512 samples equal 32 ms. Either can work as a transport chunk. If the system already uses 512-sample chunks, that is not a problem. The important thing is that the STT service should buffer enough audio to infer well, usually on a 0.5 to 1.0 second window, rather than transcribing every tiny chunk in isolation.

That gives the backend enough phonetic context to produce stable partial text, while still keeping the stream responsive enough to feel live. The principle is simple: send tiny, infer bigger, stream continuously.

---

Final Take

Interaction systems are the difference between a model that merely answers and a system that feels like it is staying with you. They hold the turn, preserve context, begin understanding early, and shape the response while the conversation is still warm. The architecture matters because it protects that rhythm from breaking apart across too many disconnected services.

If the rhythm holds, the user feels presence. If the rhythm breaks, even an accurate model feels distant. The real product is not only the final answer. It is how the answer arrives.