Building

Companion packs

Ship 3D and 2.5D (Live2D) characters with their own motions, expressions, thinking sounds and behaviour. Data only, no code.

View as Markdown

This is the full reference. To make your first companion, start with Create a 3D companion or Create a 2D companion.

Companion mode gives an agent a face and a voice: a character in the browser that listens, looks thoughtful while the agent works, and answers out loud with lip-sync and gestures timed to the words — while still being the same agent, with its tools, files and crew.

A companion pack is a plugin that ships the looks and the characters for that, as data. No code runs.

Every look is drawn by the Unity renderer, the only one; a pack ships data and never a renderer.

Voices come from voice providers — see Voice and speech-to-text. A character only suggests voices.

A complete pack

readme.md
companion-models/sample-seed/model.json
companion-models/sample-seed/seed-san.vrm
companion-models/sample-seed/thumbnail.png
companion-models/sample-seed/background.jpg
companion-models/sample-seed/idle-neutral.vrma
companion-models/sample-seed/talk-idle.vrma
companion-models/sample-seed/listen.vrma
companion-models/sample-seed/think.vrma
companion-models/sample-seed/wave.vrma
companion-models/sample-seed/laugh.vrma
companion-characters/sample-seed-character/character.json
companion-characters/sample-rose-character/character.json

(infrastructure/plugins/sample-companion-pack — walkthrough: Sample Companion Pack)

An item pack looks the same, with companion-items/<key>/item.json, its .glb model and an optional thumbnail per item (infrastructure/plugins/sample-companion-items).

Keys are unique across the whole package. A character can't reuse its model's key — use fox-model and fox.

Every binary file must be referenced by one of the JSON files, or the publish is refused naming the file.

Model — model.json

A model is a look: the avatar plus everything that says how it moves and how its face is driven. The motions live inside the look — there is no separate motion-set capability.

{
  "name": "Sample Seed-san",
  "description": "VirtualCast's sample VRM avatar Seed-san, with a small set of its own motions.",
  "avatar": "seed-san.vrm",
  "thumbnail": "thumbnail.png",
  "motions": [
    { "id": "file:idle-neutral.vrma", "roles": ["idle"], "name": "Idle",
      "description": "A long, unhurried neutral standing idle loop." },
    { "id": "file:talk-idle.vrma", "roles": ["talkIdle"], "name": "Talk idle",
      "description": "Idle loop with light gestures while speaking." },
    { "id": "file:wave.vrma", "roles": ["gesture"], "name": "Wave",
      "description": "Waves a hand — greeting or farewell.", "cues": ["waving"], "strokeSeconds": 0.9 }
  ]
}

The whole manifest, as the platform stores it, may be at most 64 KB. An unknown key anywhere in model.json refuses the publish.

Framing

By default the stage guesses each preset from where it thinks her head is. A look can say exactly what it wants instead:

"framing": {
  "defaultPreset": "fullBody",
  "face": { "bottom": 0.73, "top": 1.0 },
  "upperBody": { "bottom": 0.45, "top": 1.0, "centerX": 0.5 }
}

• defaultPreset (face, upperBody or fullBody) is where a new companion starts; without it, Upper body.
• face, upperBody, fullBody are windows, each optional. A preset without a window keeps the stage's guess: the stage measures her head (a 2.5D look: from its Head/Face hit area, else from the parts that move as far as her eyes when ParamAngleX turns, else the top 13% of the canvas), then shows about 1.5 head heights below the head for Face and 4.5 for Upper body, and the whole model plus a 4% margin for Full body. If the head can't be found, every preset shows the whole model.
• A window is bottom and top, as fractions of the model's height (0 = its bottom edge, 1 = its top edge), and an optional centerX (0 = left edge, 1 = right edge). Without centerX the crop is centred on her head. For a 3D look the model is its rest-pose box (hat and hair included), measured at its front; for a 2.5D look it is the canvas the artist drew in.
• A window guarantees height only. At zoom 1 the stage's height shows exactly bottom to top, on any stage shape. The width is whatever the stage allows: a wide stage shows more space beside her, a narrow one crops her sides around centerX. centerX moves the crop, it cannot widen it.
• Values may go a little past 0 and 1 ("bottom": -0.04 leaves air under her feet), between -1 and 2, and top must be at least 0.02 above bottom. An unknown key or an out-of-range value refuses the publish with the reason.

A user who uploads their own look gets the same thing from the Use this view for Face button in the companion's Frame mode.

For a 3D look, framing can also carry a camera that says where the camera rests, how far a viewer can turn it and how it follows her:

"framing": { "camera": { "defaultYawDegrees": 20, "yawRange": [-45, 45], "isDriftEnabled": true, "followSeconds": 0.6 } }

A 2.5D look with a camera is refused.

Pacing and the walk motion

With pacing on, she now and then takes a step to the side on her own: while idle every 12 to 30 seconds, and sometimes between sentences of a long reply. She turns three-quarters toward the direction she walks (60 degrees), plays her walk loop on her legs while she travels, squares up to the viewer before the last step and stops to face the user when they start talking or typing.

"pacing": { "isEnabled": true, "speed": 0.35, "stepDistance": 0.25 },
"motions": [
  { "id": "file:walk.vrma", "roles": ["walk"], "name": "Walk", "description": "An in-place walking loop for stepping across the floor." }
]

• Distances are in body heights, so a tall and a short model pace alike.
• The walk clip must be an in-place loop (no travel in the clip; the stage moves her). The stage measures how far the left foot travels in the loop and plays it faster or slower (0.5x to 2x) to match speed; a clip it can't measure plays at 1x.
• Without a walk motion she never paces, whatever pacing says: pacing alone does nothing. A user with reduced motion on never sees her pace either.
• Without a pacing block the defaults above apply; she paces if the look has a walk motion.
• She only steps when there is floor to step on. If the current framing (a close-up Face view, say) leaves less visible floor than one step, the step is skipped.
• A character turns it off or tunes it with behaviour.behaviours.pacing ({ "enabled": false } or { "strength": 0.75 }).
• A [pace:left], [pace:right] or [pace:home] tag in a reply makes her step on cue, and [camera:30] (a whole number of degrees within -75 to 75), [camera:front] or [camera:reset] turns the camera. On a 3D look the agent's briefing teaches the camera tags, and the pace tags when the look has a walk motion. A plugin that edits the reply can write them too. An invalid argument drops the tag.
• Pacing is for 3D looks; a 2.5D look with pacing is refused.

Outfits

An outfit is a named set of the model's parts to show. She wears one outfit at a time, by design: "what is she wearing" is a single answer for the agent and the user. (Independent toggles such as "glasses on or off" are left out on purpose; make a second outfit instead.)

"outfits": [
  { "key": "default", "name": "Casual",  "meshes": ["Tshirts", "Bottoms", "Shoes"], "shapes": [{ "mesh": "Body", "shape": "Hip1", "weight": 1 }] },
  { "key": "dress",   "name": "Dress",   "meshes": ["Dress", "Dress_Shoes"] },
  { "key": "swim",    "name": "Swimwear", "meshes": ["SwimB_L", "Bottoms_L002"], "materials": ["Mat_Swim"] }
]

• The union rule. A part that any outfit lists is hidden unless the worn outfit lists it; a part no outfit lists (body, face, hair, a tail) always shows. So an outfit only names its own clothes.
• Shapes follow the same rule. A (mesh, shape) pair that any outfit names is 0 unless the worn outfit sets it; a pair no outfit names is never touched. The mesh must be one of the avatar's and the shape one of its shape keys.
• The first outfit is the default, and what a companion wears until someone changes it. At most 20 outfits per look.
• Write mesh names the way three.js spells them (the same rule as bone maps): Bottoms_L.002 becomes Bottoms_L002. Material names are written as they are. The publish is refused when the avatar has no part of that name, and tells you the spelling when it differs.
• A VRoid export draws the top, skirt and shoes as one Body mesh split by material: list the clothing materials in materials instead. VRoid has no skin under its clothes, so hiding a garment leaves a hole; give such an outfit its own mesh in the file, or keep the garments in all of them.
• A material shared between a garment and the body hides both. That is the author's to avoid.
• Outfits are for 3D looks. A Live2D look with outfits is refused.

Hair, cloth and what a garment covers

All of this is 3D only and needs no code, but the model has to carry the right things. What the stage does with what you leave out:

cloth is a list of up to 4 pieces:

"cloth": [
  { "chains": ["Skirt_F1", "Skirt_F2", "Skirt_R1", "Skirt_B1", "Skirt_B2", "Skirt_L1"], "isClosedLoop": true, "stiffness": 0.5, "drag": 0.4, "gravity": 1 }
]

• "cloth": [] on the look means no cloth at all. A skirt-less look needs nothing.
• An outfit can carry its own cloth (same rules). A list, even an empty one, replaces the look's cloth while that outfit is worn; an outfit without cloth uses the look's, or the detected skirt. Yuzuki gives Casual, Loungewear and Swimwear "cloth": [] so only Dress uses the detected skirt.
• Cloth on a 2.5D look is refused.
• An outfit change shows at once; the covered skin hides a moment later (the stage waits at most a quarter of a second for it, and remembers the result on that device).

Background

background is a picture that sits behind her on the Companion tab, blurred more the closer the framing (Face most, Full body least), so she stays the focus. It belongs to her stage, not the page, and it is not drawn in previews. Use a soft, out-of-focus room picture (1920×1080, a JPEG under 1 MB is plenty). It keeps its proportions: on a stage of another shape it is scaled to cover and cropped at the edges, never stretched. It works for 2.5D looks too.

Each motion:

isBodyAnimated and partBones are worked out from the clip itself — writing them refuses the publish.

Roles:

Posture roles are optional (a look with none just does not shift), never chosen by the agent, and a look's own posture clips replace the standard set's. walk is optional too: without it she does not pace.

An extra-parts role needs a .glb clip (or an avatar animation) that moves at least one named bone the avatar has. A body clip on a humanoid avatar must map humanoid bones: a .vrma always does; a .glb or .fbx needs a rig.

Cues tie motions to the tone of what the agent says. When the agent speaks aloud it can mark lines with stage directions such as [happy] or [laughing]; a motion whose cues include that word becomes the natural pick. Allowed words are the emotions (happy, sad, excited, calm, curious, surprised, proud, grateful, confused, … — about fifty), the sound effects (laughing, chuckling, sighing, gasping, crying, yawning, …) and the stage actions waving (her greeting), stretching, yawning, humming and looking-around — a motion cued with one of the last four is played when she is idle for a long while, so a look can bring its own idle habits. An unknown cue refuses the publish, naming it.

2.5D looks (Live2D)

Set avatar to a .zip of a Live2D Cubism 5 model: one .model3.json, its .moc3, textures, motions, expressions and physics. Publish opens the archive and refuses a missing file, a hostile path or a broken .moc3. A 2.5D look is a painted character whose layers bend and slide, and it can only play its own motions — so list them as avatar:<path>.motion3.json (the path inside the zip, leaving out the model's own folder if you zipped the folder) with roles idle, talkIdle, listen, think, gesture, reaction, postureShift and leanIn. Extra-parts roles, walk and rig do not apply, and a 2.5D look with cloth, pacing or a framing camera is refused. The face map names the model's own parameters (ParamMouthOpenY, ParamEyeLSmile, ParamCheek, …); anything it leaves out uses the standard Cubism parameters. Working example: infrastructure/plugins/hello-live2d. A 2.5D look takes a background like a 3D one, but no outfits and no items: the artwork is one painted piece.

What the 2.5D look should supply, and what happens when it doesn't:

• A head hit area. Name a hit area in the .model3.json with "head" or "face" in its name. The stage uses its drawable to find her head for framing, clicks and the backdrop shift. Without one, the stage measures the head when the model loads: the parts that move at least 60% as far as her eyes do when ParamAngleX turns. A model without ParamAngleX gets a guessed head (the top 13% of the canvas) and no backdrop shift. Declared framing windows replace the guess for the buttons.
• ParamAngleX and ParamAngleY. They carry the head turn, nods and the shift of the picture against the backdrop.
• The look is drawn on its own layer and lit with the backdrop's tint, rim light, soft shadow and glow; the artist's colours are not changed, and there are no per-model settings for it.
• A model must fit the budget the stage keeps for it: on a desktop 448 MB and four 4096-pixel textures, on a phone 192 MB and two.

Face map: smiling eyes and closed lips

Besides emotions, mouth shapes, blink, smile and brows, a face map can drive:

• eyeSmile (left and right lists, like smile) — the squint of a smile. It follows her happiness at full strength, even while she speaks, so her eyes keep smiling as she talks; the smile mouth dial is capped while she speaks so it doesn't fight the lip-sync.
• Two more visemes: pp (closed lips, for "p", "b", "m") and ss (teeth together, for "s", "z"), next to aa, ih, ou, ee and oh.

When the face map has any mouth shape, the VRM's own aa…oh expressions are not driven at the same time, so the two never fight. Auto-fill in the look editor suggests these names, and a pack author can start from them:

Calm and lively talk loops

A talkIdle motion can carry emotion cues ("cues": ["calm"], ["happy"]). A reply that is mostly calm (or sad) prefers a talkIdle cued with a calm word; a happy, excited reply prefers one cued with a happy word. A look with only one talkIdle just plays it. Gestures also run a little faster and bigger in a lively reply.

Looks are followed live. A companion wearing your look points at it — nothing is copied. When a user's installation moves to a new version of your plugin, every companion wearing the look moves the new way, and open Companion tabs update. If a version drops a look, companions wearing it show "Model missing" with one-click replacements until a version brings it back. If the user uninstalls your plugin, they do too.

Item — item.json

An item is a small prop model with an instruction for where it sits. Any 3D companion with a humanoid look can wear any installed item, whatever pack its look came from.

{
  "name": "Mug",
  "description": "A warm mug of coffee.",
  "model": "mug.glb",
  "thumbnail": "thumbnail.png",
  "attach": { "bone": "rightHand", "position": [-0.03, 0, 0.03], "rotation": [0, 0, -90], "scale": 1 }
}

• The reference body is AvatarSample_A (the VRoid Studio sample, freely downloadable from VRoid Hub). Author the position, rotation and scale against it (the bone frames are the VRM's normalized humanoid, arms in a T-pose, model facing +Z, so her right side is -X). The browser scales the item to each body by the length of the body part it sits on: scale × (her segment ÷ the reference segment), and the position by the same factor. The reference segments, in metres:

  Bone	Segment	Length
  head	head height	0.240
  spine	spine to neck	0.330
  hips	hip width	0.308
  leftUpperArm, rightUpperArm	upper arm	0.240
  leftLowerArm, rightLowerArm	forearm	0.234
  leftHand, rightHand	wrist to middle finger base	0.057
  leftUpperLeg, rightUpperLeg	thigh	0.364
  leftLowerLeg, rightLowerLeg	shin	0.406
  leftFoot, rightFoot	ankle to toes	0.111

• Items are rigid props. They move with the bone; there is no skinning and no animation inside an item.

• An item in a hand makes that hand curl around it.

• One item per body part. Wearing a second item on the same bone replaces the first.

• Worn budgets, enforced together (a save or a [hold:…] that would break one is refused): at most 8 items at a time, 12 MB of item models in total, and 16 million texture pixels in total.

• Items are not drawn on 2.5D looks.

Character — character.json

{
  "name": "Sample Seed",
  "vibe": "A test character with a plugin look that brings its own motions.",
  "look": "sample-seed",
  "soul": "You are Sample Seed, a cheerful test companion. You keep answers short and friendly.",
  "instructions": "You talk with the person out loud. Keep replies to a sentence or two and avoid lists.",
  "voice": {
    "description": "A bright, friendly young voice.",
    "suggestions": [
      { "provider": "fish-audio", "voiceId": "8ef4a238714b45718ce04243307c57a7", "voiceName": "Sample voice" },
      { "provider": "elevenlabs", "voiceId": "21m00Tcm4TlvDq8ikWAM", "voiceName": "Rachel", "models": ["eleven_v3"] }
    ]
  }
}

A character appears in the Companions gallery as an equal of the built-in ones, tagged with your plugin's name. Talk creates a new agent from it; Talk as… puts the face on an existing agent (optionally adopting the suggested soul). Each character becomes at most one companion per account.

The file is read strictly: an unknown field (a typo like "sugestions") refuses the publish. The old top-level model and motions are gone — a character names its one look, and a look's motions live in its model.json.

How the voice is picked: the companion takes the first suggestion whose provider the user has set up. None matching means the user's default voice; with no voice set up at all, captions only, with a notice offering to add a provider. For ElevenLabs, fish.audio and Deepgram a voice id works on any of their models; for OpenAI-compatible and plugin providers a voice id belongs to certain models, so list them in models, preferred first — the suggestion then applies only when the user has one of them.

Soul and instructions are suggestions: the personality always belongs to the agent. The agent's soul also steers which gestures the companion picks.

How the agent changes her wardrobe

The agent changes what she wears by putting tags in what she says aloud, the same way it writes [happy]:

• A tag happens where it is written: "[outfit:swim] Ready for the beach! [hold:mug]" keeps the old outfit until the first word, switches, and adds the mug at the end.
• The change is permanent. It is saved when the reply is saved, so a reload, another tab and the next reply all see it. The agent's briefing lists the outfits she has and the items installed, and what she wears now.
• Item tags are the item's key (the folder name). When two installed item packs share a key, the tag is key-pluginslug (for example coffee-cozy-props); the long form always works, so older history keeps matching. Tags are case-insensitive. A tag that names nothing she has, or that would break a limit, is ignored.
• A plugin can write these tags too — see assistantMessageOutbound — but the companionPerformance hook never sees them.
• The picture follows the voice: the outfit and items change on screen as her voice reaches the tag. Replaying an old reply does not change her clothes again.

When the user changes her wardrobe

The agent is told when the user changes what she wears in the Companion tab. A run of clicks produces one platform notice, about 4 seconds after the last click, worded like "The user just changed what you are wearing in the Companion tab. Before: Casual, nothing extra. Now: Swimwear, with Mug (right hand)." — "Before" is what she wore when the run began, so clicking back to where she started produces no notice. The agent can react out loud if it fits the moment. Only changes made in the Companion tab are reported, never the agent's own tags.

Preparing a VRM

A VRM straight out of an authoring tool can be tens of megabytes. infrastructure/scripts/dev/companion-model-shrink.mjs compresses any VRM for the web while keeping its humanoid bones, expressions, spring bones (hair and tail physics) and MToon toon shading; it turns PNG textures into WebP and meshopt-compresses the geometry:

npm i --no-save @gltf-transform/core@4 @gltf-transform/extensions@4 @gltf-transform/functions@4 meshoptimizer@1 sharp
node infrastructure/scripts/dev/companion-model-shrink.mjs in.vrm out.vrm [maxTextureSize]

maxTextureSize (pixels, optional) caps the longest side of every texture; leave it out and no texture is resized (2048 takes an 88 MB model with 4K textures down to about 6 MB). The script refuses a model whose expressions use material-colour or texture-transform binds, or whose first-person annotations or thumbnail point at meshes or images, and writes nothing in that case.

Bone maps (rig)

A body clip in an FBX file, or in a GLB without VRM humanoid data, needs a bone map from humanoid bone names to the file's node names:

"rig": {
  "version": 1,
  "bones": {
    "hips": "mixamorigHips", "spine": "mixamorigSpine", "head": "mixamorigHead",
    "leftUpperArm": "mixamorigLeftArm", "leftLowerArm": "mixamorigLeftForeArm", "leftHand": "mixamorigLeftHand",
    "rightUpperArm": "mixamorigRightArm", "rightLowerArm": "mixamorigRightForeArm", "rightHand": "mixamorigRightHand",
    "leftUpperLeg": "mixamorigLeftUpLeg", "leftLowerLeg": "mixamorigLeftLeg", "leftFoot": "mixamorigLeftFoot",
    "rightUpperLeg": "mixamorigRightUpLeg", "rightLowerLeg": "mixamorigRightLeg", "rightFoot": "mixamorigRightFoot"
  }
}

version must be 1; those 15 bones are required.

Write every bone name the way three.js spells it. Browsers load these files with three.js, which rewrites node names: whitespace becomes _, and [ ] . : / are removed. Names in your JSON must already be in that form — the publish refuses a raw name and tells you the correct spelling:

A file where two nodes would end up with the same name after that rewrite is refused.

Limits