AWS Nova Sonic Voice Agent

Phone-based service workflows need natural turn-taking, grounded answers, and reliable actions without forcing callers through rigid menu trees or disconnected speech services.

• Connected Twilio Media Streams to Nova Sonic so caller audio and model speech could move bidirectionally during one live session.
• Added Bedrock retrieval for approved knowledge and tool schemas for workflow actions during the call.
• Kept telephony events, model events, and tool results separated so sessions could be inspected and extended without coupling every concern.

• Twilio initiates or receives the phone call and opens a media WebSocket.
• The Fastify service converts telephony audio into the stream format expected by Nova Sonic.
• Nova Sonic manages speech understanding, response generation, and spoken output in one bidirectional session.
• Knowledge retrieval and tool handlers provide grounded context and return structured results mid-call.

• Audio codec and sample-rate handling preserve intelligibility across the telephony boundary.
• Session and event separation make disconnects, tool failures, and call state easier to diagnose.
• Human escalation remains an explicit path for workflows that require an operator.

• Natural turn-taking depends on network and provider latency, so interruption handling must remain explicit.
• Production rollout would require broader call-quality, escalation, and failure-recovery evaluation.

The prototype demonstrated a complete voice-agent loop from dial and live conversation through retrieval and tool execution, rather than a browser-only speech demo.