# Voice and speech-to-text providers

In companion mode an agent speaks its replies out loud and can listen to the user's voice. AgentParley ships
fish.audio, ElevenLabs, Deepgram and OpenAI-compatible voices; a plugin can add any other vendor — for speaking,
for listening, or both. Users add it under **AI Providers → Add provider** like a native vendor, then pick its
voices on a companion.

These are [model providers](plugin-model-providers.md): the same `manifest.modelProvider`, the same
`tools.fail(kind, message)`, the same "no agent powers" rule. Two handler groups:

| Group | Handlers | Rule |
| --- | --- | --- |
| Voice (text-to-speech) | `speak` + `voices` | both or neither |
| Speech-to-text | `transcribe` (+ optional `startLiveTranscription`, which needs `transcribe`) | on its own, or alongside anything else |

The reference implementation is **OpenAI Voice** — see the [walkthrough](example-openai-voice.md).

```ts
// infrastructure/plugins/openai-voice/index.ts
globalThis.manifest = {
  egress: ["api.openai.com"],
  modelProvider: { kindId: "openai-voice", displayName: "OpenAI Voice" },
  settings: [
    {
      key: "apiKey",
      title: "API key",
      description: "An OpenAI API key with access to /v1/audio/speech.",
      type: "secret",
      required: true,
    },
  ],
};
```

## `speak` — say one group of sentences

```ts
request: {
  voiceId: string;    // the voice the user picked (one of yours, from `voices`)
  text: string;       // one short group of sentences, ≤ ~300 characters, with the cues already taken out
  model: string;      // the text-to-speech model id the user saved
  cues: { kind: "emotion" | "tone" | "effect" | "pause"; name: string; intensity: number; wordIndex: number }[];
  spans: { name: "whisper" | "emphasis" | "soft" | "faster" | "slower" | "stress" | "forceful";
           startWordIndex: number; endWordIndex: number }[];
}
returns: {
  audioBase64: string;                                  // raw PCM: signed 16-bit little-endian, 24 kHz, mono, no header
  words?: { text: string; start: number; end: number }[]; // optional per-word timing, seconds from the first sample
} | ModelProviderFailure
```

- **Cues arrive parsed.** The agent writes bracketed stage directions such as `[happy]`, `[whispering]`,
  `[laughing]`, `[pause]`. You never parse them: `text` has them removed and `cues` lists each one with its `kind`,
  its lowercase `name` (the modifier removed), an `intensity` (0.35 for "slightly", 0.6 by default, 0.85 for "very",
  1.0 for "extremely") and its `wordIndex` (how many words of `text` come before it; `words.length` when it is last).
  For `[slightly sad] I lost the match. [sighing] Next time. [whispering] Don't tell anyone.` you get:

  ```ts
  {
    text: "I lost the match. Next time. Don't tell anyone.",
    cues: [
      { kind: "emotion", name: "sad", intensity: 0.35, wordIndex: 0 },
      { kind: "effect", name: "sighing", intensity: 0.6, wordIndex: 4 },
      { kind: "tone", name: "whispering", intensity: 0.6, wordIndex: 6 },
    ],
  }
  ```

  If your vendor understands cues, translate them into its own form; if not, ignore `cues` and speak `text`.

- **Scoped tags arrive parsed too.** The agent can wrap words in `<whisper>…</whisper>` (also `emphasis`, `soft`,
  `faster`, `slower`, `stress`, `forceful`). `text` has the tags removed and `spans` says which words each covered:
  for `<whisper>Go on.</whisper> Bye.` you get `text: "Go on. Bye."` and
  `spans: [{ name: "whisper", startWordIndex: 0, endWordIndex: 2 }]` (the words from start up to, not including, end).
  Spans nest and never cross. Render them in your vendor's own form, or ignore them.
- A bracket word that is not on the platform's cue list works only at the start of a sentence; elsewhere it stays in
  `text`. A plugin that can't speak such brackets should strip them, as OpenAI Voice does.

- **Audio format is fixed**: raw 24 kHz mono 16-bit PCM, base64. Fetch binary from your vendor with
  `host.fetch({ …, responseEncoding: "base64" })`:

  ```ts
  // infrastructure/plugins/openai-voice/index.ts
  async speak(request, tools) {
    const apiKey = await host.settings.getSecret("apiKey");
    const response = await host.fetch({
      method: "POST",
      url: "https://api.openai.com/v1/audio/speech",
      headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
      body: JSON.stringify({
        model: "tts-1",
        voice: request.voiceId,
        input: request.text,
        response_format: "pcm",
      }),
      responseEncoding: "base64",
    });
    if (response.status !== 200) return mapFailure(response.status, tools);
    return { audioBase64: response.body };
  },
  ```

- **Word timings make the face land on the word.** If your vendor returns per-word timestamps, pass them as
  `words`; lip-sync and gestures then line up exactly. Without them the platform estimates from the audio itself.
  At most 2,000 entries; each needs `0 ≤ start ≤ end` (invalid entries are dropped, the audio still plays).
- **Buffered, not streamed.** The user hears nothing until your handler returns the whole group, so keep it quick:
  about **15 s** per call, and the response must fit the transport (~4 MB, about 30 s of audio).

### Telling the agent what your voice does — `voiceDirection`

The platform already teaches every companion the cue words and that the face and body act on them. Add what is
specific to your voice with `voiceDirection` in `manifest.modelProvider` (needs `speak` and `voices`): which cues it
performs, which it ignores, how it handles pauses, any limit on stacking, or that it reads none.

```ts
modelProvider: {
  kindId: "my-voice",
  voiceDirection: {
    default: "- Your voice performs [whispering] and [laughing]; it ignores emotion words.",
    models: { "my-voice-v2": "- This model also performs emotion words." },
  },
},
```

A plain string works too. Each text is at most 4,000 characters; `models` keys are model identifiers (matched ignoring
case, at most 20), and `default` covers any other model. Without `voiceDirection`, the platform adds a generic line saying the
voice comes from a plugin and performs what it can, with pauses anywhere and at most two cues together. The text is the same
every turn, so keep it static.

## `voices` — list voices to pick from

```ts
request: {
  search: string | null;      // what the user typed in the picker
  mine: boolean;              // only voices this account owns on the vendor (vs. the public library)
  cursor: string | null;      // null for page 1; otherwise your own nextCursor from last time
  voiceIds: string[] | null;  // look up specific ids (e.g. to show an already-picked voice's name)
  model: string;              // the TTS model the voices are for
}
returns: {
  voices: { id: string; name: string; description?: string; languages?: string[];
            sampleUrl?: string; coverImageUrl?: string }[];
  nextCursor: string | null;  // null = no more pages
} | ModelProviderFailure
```

`nextCursor` is opaque to the platform — a page number, an offset, a vendor token. `sampleUrl` and `coverImageUrl`
are loaded by the user's browser directly. About 10 s per call. There is no manual voice-id entry: a voice can only
be picked from what `voices` returns.

```ts
// infrastructure/plugins/openai-voice/index.ts
voices(request) {
  const search = request.search?.trim().toLowerCase();
  const voices = search ? VOICES.filter((voice) => voice.name.toLowerCase().includes(search)) : VOICES;
  return { voices, nextCursor: null };
},
```

## `transcribe` — turn the user's speech into text

```ts
request: {
  audioBase64: string;       // the whole recorded clip, base64
  mediaType: string;         // its MIME type — today always "audio/wav" (16 kHz, mono, 16-bit)
  model: string;             // the speech-to-text model id the user saved
  language: string | null;   // the user's language code, or null for auto-detect
}
returns: { text: string; durationSeconds?: number } | ModelProviderFailure
```

- **Recorded unless you also register `startLiveTranscription`.** The platform waits until the user stops talking, then
  calls you once with the whole clip. To show words while the user is still speaking, add the optional
  `startLiveTranscription` handler (below); `transcribe` stays as the fallback.
- `durationSeconds` is what the user is billed for (rounded up); omit it and a flat 1 second is billed.
- About **20 s** per call.
- Vendors that want a file upload need `multipart/form-data`. The sandbox has no `Buffer` or `TextEncoder`, so
  build the body as bytes and send it with `bodyEncoding: "base64"` — OpenAI Voice shows how:

```ts
// infrastructure/plugins/openai-voice/index.ts
async transcribe(request, tools) {
  const apiKey = await host.settings.getSecret("apiKey");
  const boundary = "agentparley-openai-voice-boundary";
  const extension = extensionFor(request.mediaType);
  const preamble = asciiBytes(
    `--${boundary}\r\n` +
      `Content-Disposition: form-data; name="model"\r\n\r\n` +
      `whisper-1\r\n` +
      `--${boundary}\r\n` +
      `Content-Disposition: form-data; name="file"; filename="audio.${extension}"\r\n` +
      `Content-Type: ${request.mediaType}\r\n\r\n`,
  );
  const postamble = asciiBytes(`\r\n--${boundary}--\r\n`);
  const body = base64Encode([...preamble, ...base64Decode(request.audioBase64), ...postamble]);

  const response = await host.fetch({
    method: "POST",
    url: "https://api.openai.com/v1/audio/transcriptions",
    headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": `multipart/form-data; boundary=${boundary}` },
    body,
    bodyEncoding: "base64",
  });
  if (response.status !== 200) return mapFailure(response.status, tools);

  const parsed = JSON.parse(response.body);
  return { text: typeof parsed.text === "string" ? parsed.text : "" };
},
```


## `startLiveTranscription` — live words while the user talks (optional)

The browser connects straight to your vendor; you only describe how. Platform calls this **once per turn, when the
microphone opens**, and reuses the answer until its `expiresAt`. Usually you mint a short-lived browser token from your
key and return it in `protocols` or the URL.

```ts
request: { model: string; language?: string | null }
returns: LiveTranscriptionSessionDescriptor | ModelProviderFailure
// url (wss://), protocols?, expiresAt?, sampleRate (8000|16000|24000|48000),
// audio: { frame: "binary" } | { frame: "json", template, field },
// start?, words: { when: { path, equals }, text, isFinal? }, endOfTurn?, error?, finalize?, close?
```

- `words.when` selects the vendor messages that carry transcript text and `words.text` is the dot-path to the text;
  `words.isFinal` (optional) is the path to a boolean saying the text is final. `endOfTurn.when` selects the vendor's
  "user stopped talking" message. Paths read own properties only and may not contain `__proto__`, `prototype` or
  `constructor`; at most 100 characters and 8 parts.
- `audio: { frame: "json" }` sends each ~80 ms frame as a copy of `template` with the base64 PCM written at `field`.
- **No secret value may appear anywhere in the descriptor** (secrets of 8 or more characters are checked) — Platform
  refuses it, logs the secret's name for you, and the turn falls back to recording. `url` must be `wss://` to a host
  on your egress list; `expiresAt`, if given, must be in the future; the JSON is at most 8 KB. The call gets 10 s.
- Requires `transcribe` too. `infrastructure/plugins/hello-live-stt` is a working example against
  `test-utils/fake-live-stt.mjs`.

## Failures

Return `tools.fail(kind, message)`:

- `keyRejected`, `noCredit`, `modelNotFound`, `requestRejected` → the account's red provider banner.
- `rateLimited`, `unavailable` → no banner; treated as a transient failure.

A thrown error or timeout counts as `unavailable`.

## How a user sets it up

1. Install your plugin.
2. **AI Providers → Add provider → "*Your name* · from plugin"**, enter the key.
3. Add a **text-to-speech model** and/or a **speech-to-text model** under it, with the model identifier your
   vendor uses — it reaches you as `request.model`.
4. On a companion, open the voice picker, choose your provider, and pick a voice. For listening, choose your
   speech-to-text model as the account default or for that companion. The mic button appears once a
   speech-to-text model is available.

Companion characters can suggest your voices by your `kindId` — see
[Companion packs](plugin-companion-packs.md#character--characterjson). For plugin providers, include `model` in the
suggestion.