How to Stream TTS Audio to a Real-Time AI Avatar

The voice an avatar appears to speak is easy to mistake for a playback detail. In a real-time product, it is a system boundary.

Your application or agent stack decides what to say and produces the speech audio through its own TTS layer. Spatius receives that avatar speech audio, converts it into real-time motion data, and AvatarKit renders the avatar locally. It does not run the ASR, LLM, TTS, user-turn policy, or tool logic for your product. The Spatius developer docs map is explicit about that boundary.

Once the boundary is clear, the integration problem becomes manageable: send the right audio, in the right format, at the right time, and clearly mark when an avatar speech turn has ended. If you are still deciding where that runtime should live, start with Spatius’ integration-path guide.

Key takeaways

Send the TTS output the avatar should speak—not microphone input and not audio captured back from a live playback stream.
Normalize the source to mono 16-bit PCM (s16le) at a sample rate configured for the session. Spatius does not resample audio automatically.
Send chunks as the TTS provider produces them. Do not sleep between chunks to imitate 1× playback timing.
Mark the final chunk of a speech turn with the SDK’s end-of-input flag. This closes the avatar’s audio input; it does not make turn-taking decisions for your product.
If the only source is paced, playback-speed audio, add a per-turn pre-buffer and accept the extra startup delay as an application-level trade-off.

Separate the speech pipeline from the avatar pipeline

For a voice-enabled SaaS product, several systems may be involved. Treating all of them as “the avatar” hides the ownership you need to test and operate.

Stage	Typical owner	Responsibility
User speech	Your product and ASR provider	Capturing and transcribing the user, if the experience accepts voice input
Reasoning and tools	Your application, agent framework, or backend	Context, permissions, knowledge retrieval, tool calls, and response policy
Speech generation	Your TTS provider or application	Producing the audio the avatar should speak
Motion generation	Spatius Motion Server	Converting avatar speech audio into motion data
Local presentation	AvatarKit in the client	Rendering the avatar from the returned motion data

The input to Spatius is the fourth-row boundary: the audio your avatar will speak. In a conventional voice-agent flow, that is usually TTS output after the application has already decided on the response. It is not the user’s microphone audio by default. Spatius’ audio guidance describes the distinction directly.

For the broader streaming-TTS context, see Stream’s overview of how text-to-speech systems work, Async’s engineering write-up on a streaming TTS system, and Chrome for Developers’ AudioWorklet design pattern. These resources describe general audio-pipeline choices; Spatius-specific input behavior remains defined by its audio documentation.

This separation also makes debugging clearer. A poor response is an application, model, retrieval, or tool-policy problem. An invalid or poorly timed audio stream is an integration problem. They should not be debugged as the same issue.

Use source audio, not a playback capture

The timing rule is simple: hand avatar speech audio to the avatar pipeline when it is generated, as described in Spatius’ audio concepts.

That can be counterintuitive if your application already plays audio through RTC, a browser audio element, or a WebSocket stream. Those channels often deliver audio at the speed a listener hears it. They are designed for playback, not as a source feed for audio-to-motion inference.

Motion Server can generate motion from valid audio that arrives slowly, but the client needs enough prepared audio and motion segments to keep presentation continuous. If it starts playing the first ready segment before the next one is ready, the avatar can stall. The Spatius audio guidance therefore recommends sending TTS output at its generation speed rather than pacing it to wall-clock playback time.

The wider delivery trade-off is familiar in streaming systems: Spotify Engineering writes about smoother streaming, Mux explains how to recover from rebuffering, and Ably discusses backpressure in real-time streams. They are useful background when you assess source timing and buffer behavior around the avatar runtime.

Source pattern	Use it as avatar speech input?	Why
A PCM chunk emitted by your TTS provider	Yes	It is source audio and can be forwarded as it is produced
A completed reply replayed quickly after normalization	Usually	It is still a source asset, not a live playback capture
Browser microphone audio	Not by default	It is user input, not normally the avatar’s speech
RTC or WebSocket audio arriving at 1× speed	Avoid as a direct source	Its pacing can leave too little ready audio and motion for smooth playback

If your system mixes several paths, use the Spatius integration-path guide to confirm which layer is actually responsible for sending avatar speech audio in the shape you selected.

Do not solve the pacing problem by inserting sleeps based on a chunk’s duration. Preserve a direct path from TTS source output to the avatar input whenever possible; this is the source-timing distinction in the Direct Mode model.

Define one audio contract at the boundary

Before connecting a TTS provider, write down an audio contract shared by the service producing speech and the client or backend sending it to Spatius. The contract belongs with the selected integration path, not with an arbitrary browser-playback implementation.

Motion Server accepts mono 16-bit PCM (s16le). The configured sample rate must be one of 8000, 16000, 22050, 24000, 32000, 44100, or 48000 Hz. Spatius does not automatically resample input, so a source that does not match your session configuration must be converted before it is sent. See the audio format reference for the current supported values.

For general audio-engineering background, ForaSoft’s explanation of frames, packets, and audio chunking, Chrome’s note that AudioWorklet is available by default, and its AudioWorklet design-pattern article are useful companions. They do not alter the PCM and sample-rate contract published by Spatius.

Contract question	Decision to document
What enters the avatar path?	TTS output for the avatar’s approved reply
Which format?	Mono PCM16 / `s16le`
Which sample rate?	One supported rate, configured consistently for that session
Where does conversion happen?	A clearly owned service or client-side adapter before the avatar send
What closes a speech turn?	The final source chunk is sent with the platform’s end-of-input flag
What happens when a response is cancelled?	Your application clears or replaces pending output under its own turn policy

This is not a recommendation to choose one universal sample rate. The right choice depends on the audio your TTS provider produces and the rest of your pipeline. The important part is that the value is deliberate, configured, and verified in an integration test—not inherited accidentally from a browser default.

If your browser is responsible for converting or buffering audio, MDN’s Web Audio API overview and AudioWorklet reference are useful implementation references. They do not change the Spatius input contract: normalize the audio before it reaches the avatar path.

Stream at generation speed, then close the final chunk cleanly

A reliable implementation treats the output as a sequence of source chunks with a distinct end-of-input event.

Your application starts a response and receives TTS audio.
It normalizes each chunk to the session’s PCM contract.
It forwards chunks as they are generated rather than waiting for listener-paced playback.
When the response is complete, it sends the actual final chunk with end: true or the equivalent for the SDK in use.

The final flag matters. In Direct Mode, an end-of-input flag marks the end of avatar speech audio for that conversation round. The avatar continues playing remaining animation and then returns to idle; a later audio send begins a new round. Direct Mode’s conversation guidance explains this lifecycle.

For related stream-lifecycle patterns, see Ably’s article on continuity in a distributed real-time system, its analysis of resumable AI token streaming, and Async’s streaming-TTS architecture write-up. These are useful ways to think about a clean end-of-turn boundary without turning it into a claim about Spatius behavior.

One practical application pattern is to retain one normalized chunk in a small pending buffer. When a following chunk arrives, send the pending chunk as non-final and retain the new one. When the TTS provider signals completion, send the retained chunk as final. This avoids declaring a chunk final before you know there is no next chunk.

onTtsChunk(chunk):
  pcm = normalizeToConfiguredPcm(chunk)
  if pendingChunk exists:
    sendToAvatar(pendingChunk, end=false)
  pendingChunk = pcm

onTtsCompleted():
  if pendingChunk exists:
    sendToAvatar(pendingChunk, end=true)
  clear pendingChunk

Use SDK-specific calls and error handling from the relevant AvatarKit Web SDK reference. The point of the pattern is the boundary: the application owns TTS completion, while the avatar integration receives a correctly terminated speech input. The client state and events guide is the right companion when you map that input boundary to visible client states.

If you only have paced audio, pre-buffer honestly

Sometimes the source you inherit is not generation-speed TTS. For example, a third-party service may expose only a stream paced for listening. Forwarding each chunk immediately can then cause the playback-stall pattern described above.

The documented workaround is a pre-buffer at the start of each avatar speech turn:

Accumulate paced audio locally before sending it to the avatar path.
Start sending only after the buffer covers Motion Server’s startup window.
Continue filling the local buffer while you send from it.
Reset the buffer when the speech turn is interrupted or a new one begins.

Spatius suggests starting around 3.5 seconds of audio and using 4 seconds as a more conservative value, then tuning in your application if stalls persist. That time is not an SDK guarantee or a universal latency target; it is a trade-off between a longer start and a more stable supply of synchronized audio and motion. Read the full paced-audio guidance before selecting a production threshold.

If you can access the original TTS output instead, that is usually the cleaner architecture. A pre-buffer is a fallback for a constrained source, not a reason to turn normal TTS output into playback-speed data. Re-check the current audio guidance when changing this threshold or source type.

For a broader view of that compromise, compare Spotify Engineering’s discussion of streaming smoothness, Mux’s explanation of rebuffering recovery, and Stream’s guide to TTS latency and deployment. They make the product trade-off—longer startup versus a less interrupted experience—easier to reason about.

Keep avatar input boundaries separate from user-turn policy

It is tempting to let a single “end” event mean everything: the TTS service has finished, the avatar has finished, the user can talk, and the agent is ready for a new instruction. Those events are related, but they are not identical.

The final audio flag tells the avatar pipeline that no more audio belongs to the current avatar speech input. Your product still owns questions such as:

When should a user be allowed to interrupt?
What should happen to an unfinished agent response?
Does a cancellation stop tool work, TTS generation, or only presentation?
When should the UI return input focus to the user?
Which state should be shown while an external tool or human teammate is taking over?

Design those policies in the application or agent layer, then give the avatar a clear source-audio boundary. For the product decision rather than the transport detail, see When Should Users Be Able to Interrupt an AI Avatar?.

For the broader system boundary around an existing agent, read How to Add an AI Avatar to an Existing SaaS AI Agent. For a constrained first rollout, How to Pilot an AI Avatar in Your SaaS Product provides the product-level frame.

For general guidance on interruption and transport around real-time AI, Ably’s discussion of a dedicated AI streaming SDK, Daily’s advice on browser performance for real-time media, and WebRTC.ventures’ overview of transport for voice-AI architectures offer useful external context. They do not make user-turn policy an avatar-layer responsibility.

Test the stream as a product flow, not only as an API call

An integration can connect successfully and still feel broken to users. Test the transition from one speech turn to the next with real source conditions.

Scenario	What to verify	A useful failure signal
Normal streamed reply	Chunks are sent at generation speed and the final chunk is marked	The avatar completes the reply and reaches idle without an unexplained stall
Format mismatch	The input is converted before it reaches the avatar path	An unsupported or mismatched source never silently passes through
Slow, paced source	A pre-buffer is applied per turn if source TTS is unavailable	Playback is an explicit trade-off, not a surprise in production
User interruption	Your app discards or replaces its own pending speech according to policy	Stale audio is not presented as the next reply
TTS failure before completion	Pending state is cleared and the UI reflects the application’s recovery path	The experience does not wait forever for a final chunk that will never arrive

For Direct Mode specifically, include connection-failure behavior in the test plan. The current documentation says that if its WebSocket connection fails within 15 seconds, the SDK enters an audio-only fallback: audio continues while animation is unavailable. That is a narrow connection fallback, not a substitute for your product’s broader error, retry, or human-handoff design.

For test design beyond this SDK-specific fallback, Mux’s lesson on sample timing in media processing, Ably’s overview of WebSockets and real-time data streaming, and Chrome’s explanation of Web Audio autoplay behavior are helpful background. They support testing timing, reconnection, and browser conditions separately from your avatar integration contract.

Release checklist

The avatar’s speech source is TTS output, not default microphone input.
Each session uses one documented mono PCM16 format and supported sample rate.
Resampling happens before the avatar send and is covered by tests.
Generated chunks are forwarded without playback-speed sleeps.
The actual final chunk is marked as end-of-input.
Pending audio is reset when the application cancels or replaces a turn.
The application—not the avatar layer—owns turn-taking, interruption, permissions, tools, and handoffs.
The team has tested normal streaming, paced-source buffering, format mismatch, cancellation, and connection fallback.

For a concrete client-side state checklist around loading, connection, playback, and returning to idle, use the current Spatius Client Lifecycle guide alongside this audio-specific review.

For the broader browser-audio implementation layer, Chrome’s AudioWorklet design pattern, Stream’s TTS pipeline overview, and Async’s article on streaming TTS are relevant external references. They are especially useful when your client owns conversion, buffering, or playback controls around the Spatius input boundary.

Frequently asked questions

Does Spatius take user microphone audio as the avatar input?

Not by default. In a voice-agent experience, user microphone audio usually goes to your ASR and agent flow. The audio sent to Spatius is the TTS output that the avatar should speak. Spatius’ audio documentation describes that distinction.

Can I send an MP3 or another compressed format directly?

The documented Motion Server input is mono 16-bit PCM (s16le) at one of the supported sample rates. Convert a compressed source before sending it, and configure the session to match. Spatius does not resample it automatically.

Why is forwarding an RTC audio stream risky?

RTC audio often arrives at listener-paced, 1× speed. That can leave AvatarKit without the next synchronized audio and motion segment before it consumes the current one. When possible, forward the original TTS output instead. If paced audio is all you have, introduce a per-turn pre-buffer using the documented paced-source approach.

Does the final end-of-input flag determine when the user can speak again?

No. It marks the end of the avatar’s input audio for that response; the Direct Mode conversation lifecycle describes the avatar-side boundary. Your product or agent framework remains responsible for user turns, interruption policy, focus management, and any actions that follow.

For a wider explanation of separating a stream’s transport lifecycle from the product session, see Ably on stream continuity, Daily on media-session performance, and Chrome’s AudioWorklet background-processing guide. These are general engineering references, not alternate Direct Mode documentation.

Stream the speech your product has already decided to present

The strongest TTS-to-avatar integrations are intentionally unremarkable: application logic decides the response, TTS produces source audio, the avatar layer receives a valid and clearly bounded stream, and the client renders the result locally.

That separation makes each layer easier to replace, test, and explain. If you are evaluating a real-time avatar layer for a SaaS product, request a Spatius demo.

For teams continuing their research, Ably’s article on resumable AI streams, ForaSoft’s guide to audio frames and packets, and Spotify Engineering’s discussion of smoother media delivery are useful third-party reading alongside the product-specific sources in this guide.