Skip to main content
Stream analysis from a live camera and microphone, or from a microphone alone: connect to Interhuman over WebSocket, capture media, and send binary chunks as they are recorded. In this codealong, you will:
  1. Choose a model and connect to wss://api.interhuman.ai/v2/stream/analyze
  2. Get the camera and microphone, or the microphone alone
  3. Send the recording to Interhuman and read typed server events
Wire the steps together in your app. Use JavaScript in the browser (getUserMedia + MediaRecorder) or Python on the desktop (ffmpeg for capture).

Choose video or audio

The endpoint serves two Inter-2 models. Pick one per session: Inter-2 requires a video track: media with no video is reported with ih5001 and not analyzed. Inter-2 Audio requires an audio track instead, and ignores any video. Both send the same event types and need the same interhumanai.stream scope. The code below sets the model in one constant, MODEL, and branches on it where the two paths differ.
You’ll need an API key. Follow the API key guide for details. In the browser, use a short-lived client token minted from that key on your server instead of the key itself.
Have an existing WS /v1/stream/analyze integration? See Migrate from V1 to V2. A new integration, including any use of Inter-2 Audio, needs no migration.

1) Connect to the WebSocket

Open a TLS WebSocket to the stream endpoint, naming the model with the model query parameter. On connect, send session config as a text frame (UTF-8 JSON), then listen for text replies and parse JSON. Branch on type. The analysis events are signal.detected, signal.updated, signal.ended, engagement.updated, conversation_quality.updated, and error. The connection also sends session and coverage events (session.ready, session.updated, session.closing, session.ended, coverage.degraded, coverage.dropped), listed in Inter-2 and Inter-2 Audio Streaming Analyze. Without the query parameter, the session opens on inter-2. If no backend is serving the model you named, the server refuses the session after the handshake with an ih1003 error envelope and close code 1013; retry later. The Python example sends Inter-2 Audio raw PCM rather than a container: its session config adds audio_format, which declares that every binary frame is signed 16-bit little-endian mono PCM at 16000 Hz (24000 Hz is the other accepted rate). audio_format is accepted on inter-2-audio sessions only and is fixed once the first audio frame arrives. The example waits for session.updated, which confirms the config was applied, before it sends any media. The JavaScript example sends a WebM stream on both models, so it doesn’t declare audio_format.
Reference: Inter-2 and Inter-2 Audio Streaming Analyze

2) Get the camera and microphone

Request the camera only for Inter-2. An Inter-2 Audio session needs the microphone alone, so the user is never asked for camera permission.
Device names and indexes vary by machine. To list them, run ffmpeg -f avfoundation -list_devices true -i "" on macOS or v4l2-ctl --list-devices on Linux. On macOS, the terminal also needs camera and microphone permission (microphone only for Inter-2 Audio). On Linux, the pulse input needs an ffmpeg build with PulseAudio support.

3) Send the recording to Interhuman

Send media as binary WebSocket frames. Start recording after the WebSocket is open and session config is sent.
  • A container stream (both models): send the recording as one continuous media stream, split into frames. The first frame carries the container’s header, and each later frame continues the same stream: WebM clusters from MediaRecorder, or fragmented-MP4 fragments from ffmpeg. Don’t send each segment as a separate file. For Inter-2 Audio, record the microphone alone as audio/webm.
  • Raw PCM (Inter-2 Audio only, after audio_format is confirmed): send the samples as they arrive. Frames can be any size and cut at any byte boundary, so there is no header or fragment structure to keep intact. The Python example sends 100 ms of samples per frame.
When the user stops, stop recording and ask the server to end the session with a session.close text frame. Don’t close the socket yourself. The server finishes analyzing the media it already accepted, sends the remaining envelopes (including signal.ended for signals still active), sends session.ended, and then closes the connection. If you close the socket early, those last envelopes are lost. See End the session cleanly.

4) Read server envelopes

Every server message shares the same outer shape: type, timestamp, correlation_id, and data. Narrow on type before reading fields inside data. This section covers the analysis events; the session and coverage events are in Inter-2 and Inter-2 Audio Streaming Analyze.

signal.detected

Each signal.detected envelope reports one signal becoming active. The example comes from an Inter-2 session; on an Inter-2 Audio session, the rationale cites what was heard, such as tone, pace, or pauses, and signal_type is one of the six signals that model reports. It always carries data.signal_type and data.start. It may also carry data.probability and data.rationale, so treat those two as optional. It carries no end time.

signal.updated and signal.ended

While a signal stays active, signal.updated reports a change to its probability or rationale, with the same data fields as signal.detected. On signal.updated, data.start is when the new state applies, not when the signal first became active; keep the start from signal.detected to measure the signal’s full span. When the signal is no longer active, signal.ended reports its data.signal_type and data.end:

engagement.updated

engagement.updated is sent when engagement is first established and again each time the level changes. data.state is engaged, neutral, or disengaged, and it holds until the next engagement.updated.

conversation_quality.updated (when opted in)

When your session config include lists conversation_quality_overall and/or conversation_quality_timeline, you may receive conversation_quality.updated with data.overall and/or data.timeline for the window that was just processed. See Conversation quality.

error

Errors use the same envelope with type: "error" and structured fields under data (for example code, message, link, and segment when applicable). See Error handling.

How to interpret it quickly

  • signal.detected, signal.updated, signal.ended: the lifecycle of one social signal, from the moment it becomes active until it ends. A rationale, when present, explains the detection.
  • engagement.updated: the attention level from start onward, until the next engagement.updated.
  • Times: start and end are in seconds, cumulative across the whole session rather than relative to each segment or frame.
  • conversation_quality.updated: optional overall and per-window quality metrics when requested.

Next steps