Realtime interaction

Use MOSS-Live to hold realtime voice conversations over one WebSocket, with camera or screen sharing for visual understanding.

1. Create an API Key

   Create a key in API Key management and use it only in server-side environment variables.

   MOSS_API_KEY=<your_api_key_here>

2. Connect and authenticate

   Connect with a server-side WebSocket client. Specify the model in the query parameter and pass the API Key in the handshake request header.

   FieldTypeRequiredDescription

   model: query stringRequired

   Model ID. Always moss-live-1.0; put it in the query parameters of the WebSocket URL.

   Authorization: HTTP headerRequired

   Bearer <API Key>; only in the WebSocket handshake request header.

   wss://api.mosi.cn/v1/realtime?model=moss-live-1.0
   Authorization: Bearer $MOSS_API_KEY

3. Configure the session

   After connecting, send attach_session. Once you receive session_attached and save session_id, configure the persona and business rules with session.update. Wait for session.updated and check data.error; an empty string means the configuration succeeded, and then you can enable media.

   FieldTypeRequiredDescription

   persona.prompt: stringOptional

   Replaces the default persona, up to 8000 characters.

   persona.name / aliases: string / arrayOptional

   Assistant name and aliases, used to recognize how users address the assistant and to correct transcription.

   instructions: stringOptional

   Business rules that do not change the persona; sending only this field keeps the current persona.

   {
     "action": "session.update",
     "session": {
       "persona": {
         "name": "Ann",
         "aliases": [
           "Annie"
         ],
         "prompt": "You are Ann, a friendly assistant with a warm and concise tone."
       },
       "instructions": "Reply with one short sentence in Chinese."
     }
   }

4. Send realtime voice and video

   Audio and frames share one WebSocket. After the session is configured, enable the inputs you need first.

   // Turn on the microphone
   {"action":"mic_state","enabled":true}
   
   // When frames are needed, enable tagged-v1 first
   {"action":"media.mode","binary_protocol":"tagged-v1"}
   
   // Turn on the camera; for screen sharing, change source to screen
   {"action":"video.state","source":"camera","enabled":true}

   Send audio continuously at realtime speed as 16 kHz mono PCM S16LE, and keep sending silent frames after the user stops speaking. With tagged-v1 enabled, prefix each audio message with 0x00.

   Send each frame as a separate JPEG. With tagged-v1 enabled, prefix camera frames with 0x01 and screen frames with 0x02.

   // ws is an open connection; pcmBytes / jpegBytes are captured Buffers.
   ws.send(Buffer.concat([Buffer.from([0x00]), pcmBytes]));  // audio
   ws.send(Buffer.concat([Buffer.from([0x01]), jpegBytes])); // camera
   ws.send(Buffer.concat([Buffer.from([0x02]), jpegBytes])); // screen

   Choose the source to focus on: camera, screen, or auto. auto prefers the camera when both sources have input.

   {
     "action": "video.stream_focus",
     "source": "screen"
   }

5. Receive and display answers

   Display the user transcript from input.transcript.done and the answer text from output.transcript.done. Play the 48 kHz mono PCM S16LE audio in order and keep its mapping to response.audio.context. Send playback.chunk.completed after each audio chunk has actually finished playing; send playback.completed after receiving response.audio.done and finishing playback of the whole turn. Do not wait for the answer text to finish before acknowledging playback. After receiving response.audio.stop, stop the corresponding playback and send playback.stopped with the duration actually played of the current unacknowledged audio chunk. When an answer is interrupted, you receive response.interrupted; when data.status of response.ended is completed, aborted, or failed, the turn has ended.

   {
     "action": "playback.chunk.completed",
     "chunk_id": "<chunk_id>",
     "run_id": "<run_id>",
     "turn_id": "<turn_id>"
   }

   {
     "action": "playback.completed",
     "run_id": "<run_id>",
     "turn_id": "<turn_id>"
   }

   {
     "action": "playback.stopped",
     "run_id": "<run_id>",
     "turn_id": "<turn_id>",
     "played_audio_ms": 1200
   }

6. End the conversation

   To stop the screen, change source to screen and stop sending JPEG frames from that source. Stopping frames alone does not end the session or stop billing, and audio still uses the 0x00 prefix. Billing runs from session_attached until the connection closes and is settled asynchronously afterwards. No charges are deducted in real time during the call, and the call is not interrupted because the balance changes.

   {
     "action": "video.state",
     "source": "camera",
     "enabled": false
   }

   ws.close();
   stream?.getTracks().forEach(track => track.stop());
   // Clear the audio queue and release capture and playback resources.

7. Verify success

   You receive input.transcript.done, output.transcript.done, and audio, complete the actual playback acknowledgements, and finally receive response.ended with data.status=completed, with no error along the way.

Next steps