Building

Voice and speech-to-text providers

Give companions a new voice vendor, or a new way to hear the user.

In companion mode an agent speaks its replies out loud and can listen to the user's voice. AgentParley ships fish.audio, ElevenLabs, Deepgram and OpenAI-compatible voices; a plugin can add any other vendor — for speaking, for listening, or both. Users add it under AI Providers → Add provider like a native vendor, then pick its voices on a companion.

These are model providers: the same manifest.modelProvider, the same tools.fail(kind, message), the same "no agent powers" rule. Two handler groups:

Group	Handlers	Rule
Voice (text-to-speech)	`speak` + `voices`	both or neither
Speech-to-text	`transcribe` (+ optional `startLiveTranscription`, which needs `transcribe`)	on its own, or alongside anything else

The reference implementation is OpenAI Voice — see the walkthrough.

// infrastructure/plugins/openai-voice/index.ts
globalThis.manifest = {
  egress: ["api.openai.com"],
  modelProvider: { kindId: "openai-voice", displayName: "OpenAI Voice" },
  settings: [
    {
      key: "apiKey",
      title: "API key",
      description: "An OpenAI API key with access to /v1/audio/speech.",
      type: "secret",
      required: true,
    },
  ],
};

`speak` — say one group of sentences

request: {
  voiceId: string;    // the voice the user picked (one of yours, from `voices`)
  text: string;       // one short group of sentences, ≤ ~300 characters, with the cues already taken out
  model: string;      // the text-to-speech model id the user saved
  cues: { kind: "emotion" | "tone" | "effect" | "pause"; name: string; intensity: number; wordIndex: number }[];
  spans: { name: "whisper" | "emphasis" | "soft" | "faster" | "slower" | "stress" | "forceful";
           startWordIndex: number; endWordIndex: number }[];
}
returns: {
  audioBase64: string;                                  // raw PCM: signed 16-bit little-endian, 24 kHz, mono, no header
  words?: { text: string; start: number; end: number }[]; // optional per-word timing, seconds from the first sample
} | ModelProviderFailure

Cues arrive parsed. The agent writes bracketed stage directions such as [happy], [whispering], [laughing], [pause]. You never parse them: text has them removed and cues lists each one with its kind, its lowercase name (the modifier removed), an intensity (0.35 for "slightly", 0.6 by default, 0.85 for "very", 1.0 for "extremely") and its wordIndex (how many words of text come before it; words.length when it is last). For [slightly sad] I lost the match. [sighing] Next time. [whispering] Don't tell anyone. you get:
```
{
  text: "I lost the match. Next time. Don't tell anyone.",
  cues: [
    { kind: "emotion", name: "sad", intensity: 0.35, wordIndex: 0 },
    { kind: "effect", name: "sighing", intensity: 0.6, wordIndex: 4 },
    { kind: "tone", name: "whispering", intensity: 0.6, wordIndex: 6 },
  ],
}
```
If your vendor understands cues, translate them into its own form; if not, ignore cues and speak text.
Scoped tags arrive parsed too. The agent can wrap words in <whisper>…</whisper> (also emphasis, soft, faster, slower, stress, forceful). text has the tags removed and spans says which words each covered: for <whisper>Go on.</whisper> Bye. you get text: "Go on. Bye." and spans: [{ name: "whisper", startWordIndex: 0, endWordIndex: 2 }] (the words from start up to, not including, end). Spans nest and never cross. Render them in your vendor's own form, or ignore them.
A bracket word that is not on the platform's cue list works only at the start of a sentence; elsewhere it stays in text. A plugin that can't speak such brackets should strip them, as OpenAI Voice does.

Audio format is fixed: raw 24 kHz mono 16-bit PCM, base64. Fetch binary from your vendor with host.fetch({ …, responseEncoding: "base64" }):

// infrastructure/plugins/openai-voice/index.ts
async speak(request, tools) {
  const apiKey = await host.settings.getSecret("apiKey");
  const response = await host.fetch({
    method: "POST",
    url: "https://api.openai.com/v1/audio/speech",
    headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "tts-1",
      voice: request.voiceId,
      input: request.text,
      response_format: "pcm",
    }),
    responseEncoding: "base64",
  });
  if (response.status !== 200) return mapFailure(response.status, tools);
  return { audioBase64: response.body };
},

Word timings make the face land on the word. If your vendor returns per-word timestamps, pass them as words; lip-sync and gestures then line up exactly. Without them the platform estimates from the audio itself. At most 2,000 entries; each needs 0 ≤ start ≤ end (invalid entries are dropped, the audio still plays).
Buffered, not streamed. The user hears nothing until your handler returns the whole group, so keep it quick: about 15 s per call, and the response must fit the transport (~4 MB, about 30 s of audio).

Telling the agent what your voice does — `voiceDirection`

The platform already teaches every companion the cue words and that the face and body act on them. Add what is specific to your voice with voiceDirection in manifest.modelProvider (needs speak and voices): which cues it performs, which it ignores, how it handles pauses, any limit on stacking, or that it reads none.

modelProvider: {
  kindId: "my-voice",
  voiceDirection: {
    default: "- Your voice performs [whispering] and [laughing]; it ignores emotion words.",
    models: { "my-voice-v2": "- This model also performs emotion words." },
  },
},

A plain string works too. Each text is at most 4,000 characters; models keys are model identifiers (matched ignoring case, at most 20), and default covers any other model. Without voiceDirection, the platform adds a generic line saying the voice comes from a plugin and performs what it can, with pauses anywhere and at most two cues together. The text is the same every turn, so keep it static.

`voices` — list voices to pick from

request: {
  search: string | null;      // what the user typed in the picker
  mine: boolean;              // only voices this account owns on the vendor (vs. the public library)
  cursor: string | null;      // null for page 1; otherwise your own nextCursor from last time
  voiceIds: string[] | null;  // look up specific ids (e.g. to show an already-picked voice's name)
  model: string;              // the TTS model the voices are for
}
returns: {
  voices: { id: string; name: string; description?: string; languages?: string[];
            sampleUrl?: string; coverImageUrl?: string }[];
  nextCursor: string | null;  // null = no more pages
} | ModelProviderFailure

nextCursor is opaque to the platform — a page number, an offset, a vendor token. sampleUrl and coverImageUrl are loaded by the user's browser directly. About 10 s per call. There is no manual voice-id entry: a voice can only be picked from what voices returns.

// infrastructure/plugins/openai-voice/index.ts
voices(request) {
  const search = request.search?.trim().toLowerCase();
  const voices = search ? VOICES.filter((voice) => voice.name.toLowerCase().includes(search)) : VOICES;
  return { voices, nextCursor: null };
},

`transcribe` — turn the user's speech into text

request: {
  audioBase64: string;       // the whole recorded clip, base64
  mediaType: string;         // its MIME type — today always "audio/wav" (16 kHz, mono, 16-bit)
  model: string;             // the speech-to-text model id the user saved
  language: string | null;   // the user's language code, or null for auto-detect
}
returns: { text: string; durationSeconds?: number } | ModelProviderFailure

Recorded unless you also register startLiveTranscription. The platform waits until the user stops talking, then calls you once with the whole clip. To show words while the user is still speaking, add the optional startLiveTranscription handler (below); transcribe stays as the fallback.
durationSeconds is what the user is billed for (rounded up); omit it and a flat 1 second is billed.
About 20 s per call.
Vendors that want a file upload need multipart/form-data. The sandbox has no Buffer or TextEncoder, so build the body as bytes and send it with bodyEncoding: "base64" — OpenAI Voice shows how:

// infrastructure/plugins/openai-voice/index.ts
async transcribe(request, tools) {
  const apiKey = await host.settings.getSecret("apiKey");
  const boundary = "agentparley-openai-voice-boundary";
  const extension = extensionFor(request.mediaType);
  const preamble = asciiBytes(
    `--${boundary}\r\n` +
      `Content-Disposition: form-data; name="model"\r\n\r\n` +
      `whisper-1\r\n` +
      `--${boundary}\r\n` +
      `Content-Disposition: form-data; name="file"; filename="audio.${extension}"\r\n` +
      `Content-Type: ${request.mediaType}\r\n\r\n`,
  );
  const postamble = asciiBytes(`\r\n--${boundary}--\r\n`);
  const body = base64Encode([...preamble, ...base64Decode(request.audioBase64), ...postamble]);

  const response = await host.fetch({
    method: "POST",
    url: "https://api.openai.com/v1/audio/transcriptions",
    headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": `multipart/form-data; boundary=${boundary}` },
    body,
    bodyEncoding: "base64",
  });
  if (response.status !== 200) return mapFailure(response.status, tools);

  const parsed = JSON.parse(response.body);
  return { text: typeof parsed.text === "string" ? parsed.text : "" };
},

`startLiveTranscription` — live words while the user talks (optional)

The browser connects straight to your vendor; you only describe how. Platform calls this once per turn, when the microphone opens, and reuses the answer until its expiresAt. Usually you mint a short-lived browser token from your key and return it in protocols or the URL.

request: { model: string; language?: string | null }
returns: LiveTranscriptionSessionDescriptor | ModelProviderFailure
// url (wss://), protocols?, expiresAt?, sampleRate (8000|16000|24000|48000),
// audio: { frame: "binary" } | { frame: "json", template, field },
// start?, words: { when: { path, equals }, text, isFinal? }, endOfTurn?, error?, finalize?, close?

words.when selects the vendor messages that carry transcript text and words.text is the dot-path to the text; words.isFinal (optional) is the path to a boolean saying the text is final. endOfTurn.when selects the vendor's "user stopped talking" message. Paths read own properties only and may not contain __proto__, prototype or constructor; at most 100 characters and 8 parts.
audio: { frame: "json" } sends each ~80 ms frame as a copy of template with the base64 PCM written at field.
No secret value may appear anywhere in the descriptor (secrets of 8 or more characters are checked) — Platform refuses it, logs the secret's name for you, and the turn falls back to recording. url must be wss:// to a host on your egress list; expiresAt, if given, must be in the future; the JSON is at most 8 KB. The call gets 10 s.
Requires transcribe too. infrastructure/plugins/hello-live-stt is a working example against test-utils/fake-live-stt.mjs.

Failures

Return tools.fail(kind, message):

keyRejected, noCredit, modelNotFound, requestRejected → the account's red provider banner.
rateLimited, unavailable → no banner; treated as a transient failure.

A thrown error or timeout counts as unavailable.

How a user sets it up

Install your plugin.
AI Providers → Add provider → "Your name · from plugin", enter the key.
Add a text-to-speech model and/or a speech-to-text model under it, with the model identifier your vendor uses — it reaches you as request.model.
On a companion, open the voice picker, choose your provider, and pick a voice. For listening, choose your speech-to-text model as the account default or for that companion. The mic button appears once a speech-to-text model is available.

Companion characters can suggest your voices by your kindId — see Companion packs. For plugin providers, include model in the suggestion.

Voice and speech-to-text providers

speak — say one group of sentences

Telling the agent what your voice does — voiceDirection

voices — list voices to pick from

transcribe — turn the user's speech into text

startLiveTranscription — live words while the user talks (optional)