Streaming responses

Streaming is available via the Responses API on models that declare this route. The minimal request is the curl below: note stream: true and the max_output_tokens cap — without it the stream can run unexpectedly long and cost more than planned.

curl https://api.normahub.cc/v1/responses \
  -H "Authorization: Bearer $NORMAHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "input": "Explain the result step by step",
    "max_output_tokens": 512,
    "stream": true
  }'

The response arrives as text/event-stream. Read events sequentially and extract the content from data: lines. Russian version of this guide: Потоковые ответы.

The classic beginner mistake is treating a network chunk as an event. One chunk can hold several SSE events or half a line: the client must accumulate a buffer, decode UTF-8 in streaming mode, split events on the blank line, and keep the tail until the next read. Parse data events separately from control markers, and treat completion as its own state.

const response = await fetch("https://api.normahub.cc/v1/responses", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.NORMAHUB_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ model: "MODEL_ID", input: "Hello", stream: true }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
let done = false;

while (!done) {
  const { value, done: finished } = await reader.read();
  done = finished;
  buffer += decoder.decode(value ?? new Uint8Array(), { stream: !done });
  const parts = buffer.split("\n\n");
  buffer = parts.pop() ?? "";
  for (const event of parts) {
    if (!event.startsWith("data:")) continue; // control markers handled separately
    const payload = event.slice(5).trim();
    if (payload === "[DONE]") { done = true; break; }
    process.stdout.write(payload);
  }
}
reader.releaseLock();

When the user cancels, close the reader explicitly and record the state as “cancelled by user”, not “error”: these are different branches for analytics and billing.

Do not treat a TCP break as a successful finish. Wait for the final stream event, close the reader, and save the X-Request-Id. The gateway finalizes the usage and cost record when the streaming operation closes, so the request total exists even if the user closed the tab halfway.

Minimum test set before production: a short answer, a long answer, a slow answer, a user cancel, a network break, and an error after the first event. For each one, record what the user sees and what ends up stored in usage. Retry after a partially received answer only with an explicit deduplication strategy.

In the current NormaHub version streaming is not enabled for /v1/chat/completions and /v1/messages. For them send stream: false or use the Responses API.

Set separate connect and read timeouts. After a partially received answer, do not auto-retry without an explicit deduplication strategy. Errors that arrive as an event after the stream started are covered in the errors guide, and stream pricing in the token cost guide.