> ## Documentation Index
> Fetch the complete documentation index at: https://docs.interhuman.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Build with the stream endpoint

> The architecture you need around wss://api.interhuman.ai/v1/stream/analyze: mint client tokens server-side, stream directly from the browser, and send segments that arrive intact.

The [stream analysis quickstart](/getting-started/stream-analyze-quickstart) shows how to call `wss://api.interhuman.ai/v1/stream/analyze`. This guide covers what the quickstart deliberately leaves out: how to put that call inside a real application — one where the camera runs in a user's browser and your API key stays secret.

If you follow only three rules from this page, make them these:

1. **Never ship your API key to the browser.** Mint a short-lived, capped **client token** server-side and hand that to the client instead.
2. **Let the browser stream directly to Interhuman with the client token.** The token is designed for exactly this, and the API enforces its caps.
3. **Send media segments exactly as the recorder produces them.** Never re-slice, reassemble, or reinterpret the media container in between.

<Note>
  This guide is written for humans and for AI coding agents. If you are prompting an agent to build an Interhuman integration, include this page (or paste the three rules above) — the naive implementation that hardcodes the API key in client code works in a demo and leaks your credential in production.
</Note>

## Why you can't ship the API key

The stream endpoint authenticates with your API key (`ih_live_...`) as a Bearer credential on the WebSocket handshake. If that key appears anywhere in client code, it ships to the user — it's in your JS bundle, visible in the network tab, extractable from anyone's devtools. That is a leaked production credential.

<Warning>
  Framework-specific version of the same leak: any environment variable prefixed `NEXT_PUBLIC_` (Next.js), `VITE_` (Vite), or `REACT_APP_` (CRA) is bundled into the browser build. Never put your Interhuman API key in one of these.
</Warning>

The key must stay server-side. What the browser gets instead is a **client token**: a short-lived credential your backend mints from [`POST /v1/client_tokens`](/api-reference/client-tokens), scoped and capped so that leaking it costs you a few minutes of bounded usage, not your account.

## The architecture

```mermaid theme={null}
flowchart LR
    A["Browser<br/>(records media)"] -- "1 - request session" --> B["Your backend<br/>(holds the API key)"]
    B -- "2 - POST /v1/client_tokens" --> C["Interhuman"]
    C -- "capped client token" --> B
    B -- "client token" --> A
    A -- "3 - wss /v1/stream/analyze<br/>+ client token" --> C
    C -- "analysis events (JSON)" --> A
```

Two parts, one job each:

* **A token route on your backend** calls `POST /v1/client_tokens` with your API key and returns the minted token to the browser. This is stateless, single-request work — a serverless function is fine here.
* **The browser** connects directly to `wss://api.interhuman.ai/v1/stream/analyze` with the client token, streams recorded segments up as binary frames, and receives analysis events back. The caps embedded in the token (duration, bytes, concurrency, video budget, allowed origins) are enforced by the API itself.

No media ever touches your servers, and there is no socket for you to host.

## Mint a client token

Use the [TypeScript SDK](/sdk-reference/typescript-sdk) (`@interhumanai/sdk`) in your token route, or call the endpoint directly:

<CodeGroup>
  ```typescript TypeScript SDK icon="square-js" theme={null}
  import { AuthClient } from "@interhumanai/sdk";

  const auth = new AuthClient();

  export async function mintStreamToken() {
    const token = await auth.createClientToken({
      apiKey: process.env.INTERHUMAN_API_KEY,   // never leaves the server
      scopes: ["interhumanai.stream"],           // default
      expiresIn: 300,                            // seconds; clamped to 60–3600
      maxDurationSeconds: 600,                   // max wall-clock session length
      maxVideoSeconds: 600,                      // total video budget for this token
      allowedOrigins: ["https://app.example.com"],
    });
    return token.access_token;                   // safe to hand to the browser
  }
  ```

  ```bash HTTP icon="terminal" theme={null}
  curl -X POST https://api.interhuman.ai/v1/client_tokens \
    -H "Content-Type: application/json" \
    -d '{
      "api_key": "ih_live_...",
      "scopes": ["interhumanai.stream"],
      "expires_in": 300,
      "max_duration_seconds": 600,
      "max_video_seconds": 600,
      "allowed_origins": ["https://app.example.com"]
    }'
  ```
</CodeGroup>

Every cap is optional, but set them deliberately — they are your blast radius if a token leaks:

| Cap                    | Meaning                                                                                     | Default                  |
| ---------------------- | ------------------------------------------------------------------------------------------- | ------------------------ |
| `expires_in`           | Token time-to-live in seconds                                                               | 300 (clamped to 60–3600) |
| `max_duration_seconds` | Max wall-clock duration of a session opened with the token                                  | unset                    |
| `max_bytes`            | Max cumulative video bytes a session may send                                               | unset                    |
| `max_concurrent`       | Max simultaneous sessions on one token                                                      | 1                        |
| `max_video_seconds`    | Total seconds of video the token may process — spans upload, stream, and real-time combined | unset                    |
| `allowed_origins`      | Browser `Origin` allow-list; connections from other origins are rejected                    | unset                    |

You can also **revoke** a token early (`POST /v1/client_tokens/revoke`, or `auth.revokeClientToken(...)` in the SDK) — new requests are rejected immediately and any live session is torn down on its next chunk. Useful when a token outlives the thing it was minted for, like a user logging out mid-session.

## Connect from the browser

The SDK's `StreamClient` handles the connection, authentication, typed events, and graceful shutdown:

```typescript theme={null}
import { StreamClient, StaticTokenProvider } from "@interhumanai/sdk";

const clientToken = await fetch("/api/stream/session").then((r) => r.text());

const stream = new StreamClient({
  tokenProvider: new StaticTokenProvider(clientToken),
});

stream.on("signal.detected", (e) => console.log(e.data.signal_type));
stream.on("engagement.updated", (e) => console.log(e.data));

await stream.connect();
await stream.waitForSessionReady();
stream.updateConfig({ include: ["conversation_quality_overall"] });
// now start recording and call stream.sendVideo(chunk) per segment
```

Without the SDK: browsers cannot set an `Authorization` header on a `WebSocket`, so pass the token via the subprotocol pair the endpoint accepts:

```javascript theme={null}
const ws = new WebSocket("wss://api.interhuman.ai/v1/stream/analyze", [
  "access_token",
  clientToken,
]);
```

## Send media the way the recorder produces it

This is where integrations lose the most time. The producer of the media — `MediaRecorder` in the browser — already emits self-contained, valid WebM segments when you record with a timeslice:

```javascript theme={null}
const recorder = new MediaRecorder(mediaStream, {
  mimeType: "video/webm;codecs=vp8,opus",
  videoBitsPerSecond: 1_000_000, // 1 Mbps is plenty for analysis
});

recorder.addEventListener("dataavailable", async (event) => {
  if (!event.data || event.data.size === 0) return;
  stream.sendVideo(await event.data.arrayBuffer()); // one segment = one binary frame
});

recorder.start(3000); // emit a self-contained segment every 3 seconds
```

Each 3-second segment is small (roughly 400 KB at 1 Mbps) and structurally complete. Send it the moment it is produced, and nothing downstream ever has to reverse-engineer the container structure.

What does **not** work, in increasing order of subtlety:

* **One recording, one giant frame.** The endpoint enforces a **32 MB maximum WebSocket message size**; a 3-minute recording at typical bitrates exceeds it, and you get an `ih6002` ("message too large") error or the upstream resets the socket mid-send (close code `1006`, no close frame). Worse, large single messages fail unreliably even *under* the cap. Small and many beats big and one.
* **Re-slicing the media in transit.** Parsing the WebM/EBML structure yourself and cutting your own chunk boundaries is fragile: dropped trailing metadata or a mis-cut boundary produces a truncated container and an `ih5004` ("malformed segment") error. Container parsers that pass tests against synthetic ffmpeg files still mis-cut real live-muxed browser output.
* **Concatenating timesliced segments later.** `MediaRecorder` only writes the duration into the header on `stop()`, so stitching timeslice segments into one blob yields a broken file. Streaming each segment live and independently is fine; concatenation is not.

## End the session cleanly

Analysis events stream in throughout the recording, so most results have already arrived by the time the user stops. Accumulate events client-side as they come — there is no end-of-session summary to wait for.

When the user stops recording, request a graceful shutdown instead of closing the socket:

```javascript theme={null}
recorder.stop();
stream.requestClose(); // sends session.close
stream.on("session.ended", () => {
  mediaStream.getTracks().forEach((t) => t.stop());
});
```

The server acknowledges with `session.closing` (its `data.max_drain_seconds` is the longest you should wait), finishes analyzing the video it already accepted — emitting the remaining envelopes, plus `signal.ended` for still-active signals — then sends `session.ended` and closes with code 1000. Listen for `session.ended` (or `close`) rather than closing the socket yourself; closing early discards trailing analysis.

## Debugging: symptom → cause

| Symptom                                                                      | Likely cause                                                                                                     | Fix                                                                                  |
| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `401` on the WebSocket upgrade                                               | Client token expired or revoked, or the subprotocol isn't the `access_token, <token>` pair                       | Mint a fresh token; check the `new WebSocket(url, ["access_token", token])` form     |
| Connection rejected despite a valid token                                    | Page `Origin` missing from the token's `allowed_origins`                                                         | Mint the token with the right origins for each environment                           |
| Processing refused mid-session                                               | Token's `max_video_seconds` budget exhausted (it spans upload, stream, and real-time)                            | Mint a new token; size the budget to your session length                             |
| `ih6002`, or socket drops mid-send with close code `1006` and no close frame | A WebSocket message is too large — over the 32 MB cap, or large enough to be unreliable                          | Send small timesliced segments (1–3 s each), one per frame                           |
| `ih5004` — "malformed segment"                                               | The media container was re-sliced, reassembled, or truncated between the recorder and Interhuman                 | Send recorder-produced segments verbatim; never rebuild the stream from parsed parts |
| Two sessions/sockets open per recording in development                       | React StrictMode double-invokes effects and state updaters                                                       | Guard session start with a synchronous ref check, not state                          |
| Segments from test files analyze fine; real recordings fail                  | Pipeline only tested against synthetic (ffmpeg) files, which are structured differently from live browser output | Always test with real `MediaRecorder` output from a real browser                     |

## Next steps

* [Client tokens API](/api-reference/client-tokens) — full request/response schema, scopes, and caps.
* [TypeScript SDK reference](/sdk-reference/typescript-sdk) — `AuthClient`, `StreamClient`, typed events, and upload support.
* [Stream analysis quickstart](/getting-started/stream-analyze-quickstart) — the endpoint call itself: session config, media frames, and event envelopes.
* [Stream & analyze API reference](/api-reference/stream-analyze) — full AsyncAPI channel, headers, and message schemas.
* [Error handling](/api-reference/error-handling) — structured error codes, including `ih5004` and `ih6002`.