# Voice orb

> Voice mode for your assistant: ink in water that the voice moves, with the talk button, the microphone and the reply wired to your own backend.

Most voice-mode orbs are a looping video, and wiring one to a real assistant is left to you. This is the whole piece. The orb is a real fluid solved on the GPU: while you talk, your voice's frequency bands push ink in from the rim; while it answers, each syllable blooms ink out from the centre; while it thinks, two streams wind into a spiral. VoiceTalk puts one button under it that morphs into what pressing it does next (Talk, Send, Thinking, Interrupt), and it handles the microphone, recording, the end of speech, playing the reply, interrupting it and every error. You give it one async function, or a realtime connection, and it fits whatever speech stack you already run.

## Install

```bash
npx shadcn@latest add https://hairlineui.com/r/voice-orb.json
```

Needs React 19, Tailwind CSS v4 and shadcn/ui theme variables (any style, light and dark). Icons from lucide-react.

- npm: `lucide-react`
- Hairline UI components it needs (the CLI installs them too): `text-morph`
- Docs and live preview: https://hairlineui.com/ui/voice-orb
- Registry item (source): https://hairlineui.com/r/voice-orb.json

## Usage

```tsx
import { VoiceTalk } from "@/components/voice-talk";

<VoiceTalk onTurn={(audio, { signal }) => fetch("/api/voice", { method: "POST", body: audio, signal })} />
```

## With your own speech stack, or a realtime call

Record-then-reply: one route takes the audio and answers with audio. Realtime: hand the microphone to a WebRTC connection and return the remote track. The same orb and button work for both.

```tsx
// app/api/voice/route.ts: your speech-to-text, your model, your voice.
export async function POST(req: Request) {
  const text = await transcribe(await req.blob());
  const answer = await reply(text);
  return new Response(await speak(answer), { headers: { "content-type": "audio/mpeg" } });
}

// A turn-based voice mode.
"use client";
import { VoiceTalk } from "@/components/voice-talk";

export function VoiceMode() {
  return (
    <VoiceTalk
      onTurn={async (audio, { signal }) => {
        const res = await fetch("/api/voice", { method: "POST", body: audio, signal });
        if (!res.ok) throw new Error("The assistant didn't answer.");
        return res; // played as it arrives, metered for the orb
      }}
    />
  );
}

// Realtime over WebRTC (OpenAI Realtime shown; LiveKit and others look the same).
export function RealtimeVoice() {
  return (
    <VoiceTalk
      connect={async (mic, { signal, setState }) => {
        const pc = new RTCPeerConnection();
        mic.getTracks().forEach((track) => pc.addTrack(track, mic));
        const remote = new Promise<MediaStream>((resolve) => (pc.ontrack = (e) => resolve(e.streams[0])));
        const events = pc.createDataChannel("oai-events");
        events.onmessage = (e) => {
          const { type } = JSON.parse(e.data);
          if (type === "response.created") setState("thinking");
          if (type === "response.done") setState(null); // back to reading the audio
        };
        await pc.setLocalDescription(await pc.createOffer());
        const answer = await fetch("/api/realtime-session", { method: "POST", body: pc.localDescription!.sdp, signal });
        await pc.setRemoteDescription({ type: "answer", sdp: await answer.text() });
        return { output: await remote, close: () => pc.close() };
      }}
    />
  );
}
```

## Props

| Prop | Type | Default | Description |
| --- | --- | --- | --- |
| `onTurn` | `(audio: Blob, { signal, mimeType, duration }) => Promise<VoiceReply>` | – | Record-then-reply. Return a Response, Blob, ArrayBuffer, URL, <audio> element, MediaStream or nothing. Throw to show an error. |
| `connect` | `(mic: MediaStream, { signal, setState, end }) => Promise<{ output, close? }>` | – | Realtime instead: return the voice coming back (a MediaStream or an <audio> element). setState overrides what the audio says; end hangs up. |
| `mode` | `"tap" \| "hold"` | `"tap"` | Tap to talk (sends when you stop, or on Send), or hold to talk and let go to send. Space and Enter work too. |
| `continuous` | `boolean` | `false` | After each reply, listen again, hands-free. |
| `silence` | `number` | `900` | Quiet after speech, in ms, that ends a turn. |
| `maxTurn` | `number` | `30000` | Longest turn in ms; it is sent when reached. |
| `waitForSpeech` | `number` | `8000` | How long to wait for any speech before giving up, in ms. |
| `getMicrophone` | `() => Promise<MediaStream>` | – | Your own microphone stream (a device picker, an SDK). Default: getUserMedia with echo cancellation, released after each turn. |
| `onPhaseChange / onError` | `(phase) => void / (error: VoiceError) => void` | – | Follow the session: idle, starting, listening, thinking, speaking, error. Errors carry a kind. |
| `session` | `VoiceSession` | – | A session from useVoiceSession, to start or stop it from elsewhere (a keyboard shortcut, a menu). |
| `size / colors / label` | `number / [string, string, string] / string` | `200 / chart-1, chart-3, chart-2 / "Voice assistant"` | The orb: diameter in px, three ink colours low notes to high (theme variables work; if the theme only has greys it uses blue, teal and coral of its own), its name for screen readers. |
| `labels` | `Partial<VoiceTalkLabels>` | – | Every word on the button and in the errors, for your copy or language. |
| `VoiceOrb` | `{ state, getLevel, getSpectrum, level, size, colors }` | – | The orb alone, for a UI of your own: drive it with useVoiceSession or useVoiceAudio(stream \| element). |

## States

- **idle**: The button says Talk (or Hold to talk); leftover ink drifts and clears in the slowly turning water.
- **listening**: The microphone is recording and the button says Send. Eight jets on the rim, one per frequency band, push ink inward as the voice gets louder in that band. 900ms of quiet after speech sends the turn.
- **thinking**: onTurn is running and the button says Thinking (pressing aborts it). The water turns and two thin streams wind into a spiral.
- **speaking**: The reply plays and the button says Interrupt. Each band blooms ink out from the centre; the vessel swells slightly with loudness.
- **error**: The button turns into the fix (Allow the microphone, Try again) and the reason folds open under it.
- **between states**: Each state is a spring weight (about 0.8s), so one motion hands the ink to the next; labels morph, End slides in beside the button.
- **reduced motion**: No loop: each state is shown as one still, settled picture; the controls change in place.

## What it handles

A voice button looks simple until it ships: permission prompts, knowing when someone has stopped talking, a reply that has to be interruptible, a request to cancel, and an orb that should follow the real sound. This handles all of it and leaves the speech stack to you.

- **One function to plug in**: onTurn gets the recorded audio and an AbortSignal; return a fetch Response, a Blob, a URL, an <audio> element or a stream, and it plays.
- **Or a realtime call**: connect gets the open microphone and returns the voice coming back; listening, thinking and speaking are read from the audio, or set from your provider's events.
- **Knows when you stopped**: Speech is detected from the microphone level; a short quiet after it sends the turn, with a longest-turn limit and a give-up if nobody speaks.
- **Interrupt and cancel**: Pressing during the reply stops it and listens; pressing while it thinks aborts the request through the signal; End lets go of everything.
- **Every failure explained**: A blocked microphone, no microphone, an insecure page, a failed request or reply: the button becomes the fix and the reason folds open under it.
- **A real fluid, cheap**: Stable fluids on the GPU with sharp ink; sound read inside the frame loop, never React state; sleeps off screen; a still under reduced motion.

## Replaces

MediaRecorder boilerplate, voice activity detection, AnalyserNode visualizer, looping orb video, Lottie voice animation, push-to-talk button, framer-motion

## Where it fits

- **Voice mode**: Talk to your assistant: record, send to your speech-to-text, model and voice, play the answer.
- **Realtime agents**: OpenAI Realtime, LiveKit or ElevenLabs over WebRTC: hand it the call, it shows who is talking.
- **Phone and support bots**: A caller-facing agent listening, working and answering, interruptible mid-sentence.
- **Dictation**: Hold to talk, let go to send, with the microphone shown hearing you.

## Notes

- The talk button is a real button whose name is what it does next; End appears beside it during a session; a polite status region announces listening, thinking, speaking and errors, never the sound. The orb is role="img", named with its state.
- Browsers only open the microphone and play audio after a press, so start() runs inside the click; the reply player is unlocked there too, which Safari needs.
- Microphones it opens are closed after each turn so the recording light goes off; a stream you pass through getMicrophone stays yours.
- Reply URLs play as they stream in; Web Audio can only meter them if they are same-origin or sent with CORS.
- The fluid runs on WebGL 2 with half-float textures (a 128 grid for the water, up to 384 for the ink), only on screen in a visible tab. Without WebGL 2, a CSS picture in the same colours stands in.
- The page must be allowed to use the microphone: https, and a Permissions-Policy that permits it on that page.

License: MIT (https://hairlineui.com/license#components)
