- Choose a model and connect to
wss://api.interhuman.ai/v2/stream/analyze - Get the camera and microphone, or the microphone alone
- Send the recording to Interhuman and read typed server events
getUserMedia + MediaRecorder) or Python on the desktop (ffmpeg for capture).
Choose video or audio
The endpoint serves two Inter-2 models. Pick one per session:
Inter-2 requires a video track: media with no video is reported with
ih5001 and not analyzed. Inter-2 Audio requires an audio track instead, and ignores any video. Both send the same event types and need the same interhumanai.stream scope.
The code below sets the model in one constant, MODEL, and branches on it where the two paths differ.
You’ll need an API key. Follow the API key guide for details. In the browser, use a short-lived client token minted from that key on your server instead of the key itself.
1) Connect to the WebSocket
Open a TLS WebSocket to the stream endpoint, naming the model with themodel query parameter. On connect, send session config as a text frame (UTF-8 JSON), then listen for text replies and parse JSON. Branch on type. The analysis events are signal.detected, signal.updated, signal.ended, engagement.updated, conversation_quality.updated, and error. The connection also sends session and coverage events (session.ready, session.updated, session.closing, session.ended, coverage.degraded, coverage.dropped), listed in Inter-2 and Inter-2 Audio Streaming Analyze.
Without the query parameter, the session opens on inter-2. If no backend is serving the model you named, the server refuses the session after the handshake with an ih1003 error envelope and close code 1013; retry later.
The Python example sends Inter-2 Audio raw PCM rather than a container: its session config adds audio_format, which declares that every binary frame is signed 16-bit little-endian mono PCM at 16000 Hz (24000 Hz is the other accepted rate). audio_format is accepted on inter-2-audio sessions only and is fixed once the first audio frame arrives. The example waits for session.updated, which confirms the config was applied, before it sends any media. The JavaScript example sends a WebM stream on both models, so it doesn’t declare audio_format.
2) Get the camera and microphone
Request the camera only for Inter-2. An Inter-2 Audio session needs the microphone alone, so the user is never asked for camera permission.ffmpeg -f avfoundation -list_devices true -i "" on macOS or v4l2-ctl --list-devices on Linux. On macOS, the terminal also needs camera and microphone permission (microphone only for Inter-2 Audio). On Linux, the pulse input needs an ffmpeg build with PulseAudio support.
3) Send the recording to Interhuman
Send media as binary WebSocket frames. Start recording after the WebSocket is open and session config is sent.- A container stream (both models): send the recording as one continuous media stream, split into frames. The first frame carries the container’s header, and each later frame continues the same stream: WebM clusters from
MediaRecorder, or fragmented-MP4 fragments fromffmpeg. Don’t send each segment as a separate file. For Inter-2 Audio, record the microphone alone asaudio/webm. - Raw PCM (Inter-2 Audio only, after
audio_formatis confirmed): send the samples as they arrive. Frames can be any size and cut at any byte boundary, so there is no header or fragment structure to keep intact. The Python example sends 100 ms of samples per frame.
session.close text frame. Don’t close the socket yourself. The server finishes analyzing the media it already accepted, sends the remaining envelopes (including signal.ended for signals still active), sends session.ended, and then closes the connection. If you close the socket early, those last envelopes are lost. See End the session cleanly.
4) Read server envelopes
Every server message shares the same outer shape:type, timestamp, correlation_id, and data. Narrow on type before reading fields inside data. This section covers the analysis events; the session and coverage events are in Inter-2 and Inter-2 Audio Streaming Analyze.
signal.detected
signal.detected envelope reports one signal becoming active. The example comes from an Inter-2 session; on an Inter-2 Audio session, the rationale cites what was heard, such as tone, pace, or pauses, and signal_type is one of the six signals that model reports. It always carries data.signal_type and data.start. It may also carry data.probability and data.rationale, so treat those two as optional. It carries no end time.
signal.updated and signal.ended
While a signal stays active, signal.updated reports a change to its probability or rationale, with the same data fields as signal.detected. On signal.updated, data.start is when the new state applies, not when the signal first became active; keep the start from signal.detected to measure the signal’s full span. When the signal is no longer active, signal.ended reports its data.signal_type and data.end:
engagement.updated
engagement.updated is sent when engagement is first established and again each time the level changes. data.state is engaged, neutral, or disengaged, and it holds until the next engagement.updated.
conversation_quality.updated (when opted in)
When your session config include lists conversation_quality_overall and/or conversation_quality_timeline, you may receive conversation_quality.updated with data.overall and/or data.timeline for the window that was just processed. See Conversation quality.
error
Errors use the same envelope with type: "error" and structured fields under data (for example code, message, link, and segment when applicable). See Error handling.
How to interpret it quickly
signal.detected,signal.updated,signal.ended: the lifecycle of one social signal, from the moment it becomes active until it ends. Arationale, when present, explains the detection.engagement.updated: the attention level fromstartonward, until the nextengagement.updated.- Times:
startandendare in seconds, cumulative across the whole session rather than relative to each segment or frame. conversation_quality.updated: optional overall and per-window quality metrics when requested.
Next steps
- Inter-2 and Inter-2 Audio Streaming Analyze — full AsyncAPI channel, headers, and message schemas.
- Migrate from V1 to V2 — for integrations on
WS /v1/stream/analyze. - Authentication — API key usage; browser WebSockets use the subprotocol as above.
- Error handling — structured error codes and recovery.
- Video Analysis and Audio Analysis — analyze a recorded file with a
POST /v2/upload/analyzejob. - Agent Skills — installable skills that wrap upload and stream calls.
- Social signals and Conversation quality — meaning of outputs.