Skip to main content

Streaming a model response

A model that takes twenty seconds to answer gives you nothing to show for nineteen of them. stream: forwards each piece of a streaming action's output to a realtime channel as it arrives, so a UI can render tokens while the action is still running.

name: ChatApp
interfaces:
  # Subscribers attach here.
  live:
    ws: "on"
    actions:
      - name: Echo
        input: a|body|

  # The browser posts here.
  chat:
    output: http
    method: POST
    respond: early                    # 202 immediately, keep working
    stream:
      from: AskModel                  # which action's output to forward
      to: channel
      channel: a|body::session_id|    # where subscribers watch
    actions:
      - name: AskModel
        http:
          url: https://api.deepseek.com/chat/completions
          method: POST
          stream: true                # read the upstream stream
          idle_timeout: 30s
          headers:
            authorization: "Bearer a|ap_var::DEEPSEEK_KEY|"
          body: |
            {
              "model": "deepseek-chat",
              "stream": true,
              "messages": [{ "role": "user", "content": a|body::prompt->double_quote| }]
            }

Two stream keys, doing different jobs. stream: true on the http: action tells the engine to read the upstream response as it arrives. stream: on the interface says where those pieces go.

What a subscriber receives​

Frames arrive on the channel like any other push, with the stream envelope under data:

{ "ap_channel": "sess-42", "data": { "seq": 0, "delta": "Once upon", "done": false } }
{ "ap_channel": "sess-42", "data": { "seq": 1, "delta": " a time", "done": false } }
{ "ap_channel": "sess-42", "data": { "seq": 2, "delta": "", "done": true } }
  • seq increases by one per frame, so a client can tell if it missed something.
  • delta is a run of characters, not a single token — see batching below.
  • done marks the end. A client should treat this as the signal to stop, not silence.
  • error appears on the final frame when the action failed: the upstream returned an error, the stream was cut off before the model finished, it went idle past idle_timeout, or the run hit its time limit. The text sent before the failure has already arrived, so show it as incomplete rather than as the answer.
  • dropped appears on the final frame only if chunks were lost because a subscriber could not keep up.
  • retry appears, with "done": false, when an action with retry starts another attempt. Each attempt streams from the start, so discard the text received before it.
{ "ap_channel": "sess-42", "data": { "seq": 3, "delta": "", "done": true,
  "error": "the event stream ended before the model finished: no [DONE], finish_reason or message_stop arrived, so the answer is cut off (15 characters received)" } }

Before engine 3.16.2 the final frame was the same "done": true on success and on failure, so a client could not tell a cut-off answer from a complete one.

Concatenating every delta gives exactly the text the action itself produced, so the action's own body is unchanged and any assert or transform on it still works.

Batching​

Deltas are not published one per token. Each publish fans out to every node holding a subscriber, so a five-hundred-token answer would mean five hundred fan-outs. The forwarder buffers instead and flushes about every 50ms or 200 characters, whichever comes first — roughly twenty frames a second, which reads as smooth. Tune with flush_ms and flush_chars if you need to.

A subscriber that cannot keep up has frames dropped rather than being allowed to stall the model call, and the count is reported on the terminal frame.

Answering early​

respond: early is separate from streaming and useful on its own. The caller gets:

{ "accepted": true, "request_id": "0199..." }

with a 202, and the run continues in the background. Reach for it when the work outlives what the caller — or a proxy in front of you — will wait for.

It works the same on managed AirPipe and self-hosted (engine 3.10.1 and later on managed).

It changes the error contract. The status code is committed before the work runs, so http_code_on_error and any assert after that point can no longer change what the caller sees; a later failure has to reach them over the channel instead (the final frame's error). The engine also logs it as a warning, respond: early run finished with a failure, with the request_id the caller was given and the failed action's error. A config's on_error handler fires for it too, and its run_uuid is that same request_id. A run still going after 15 minutes is stopped and fails with a 504, which on_error and, on managed, your hosted metrics see like any other failure (engine 3.16.4 and later). Detached runs are also capped, per organisation and per node, and a request over either limit is refused with 429 and Retry-After: 1 rather than queued. On managed AirPipe the per-organisation limit follows your plan, from 8 concurrent runs on Free up to 128 (see limits); self-hosted sets it with AIRPIPE__MAX_DETACHED_RUNS (default 64).

An early run is billed exactly as it would be in-line: the trigger when the run is accepted, and its duration when it finishes. A request refused with 429 is not billed.

Streaming without respond: early works too: the caller holds the request, watches tokens arrive on the channel, and gets the complete body when the action finishes.

Subscribing​

Clients subscribe exactly as they would to any channel — see WebSocket channels for the control frames, and Realtime auth for restricting who may join one. Per-organisation bounds are in Realtime limits.