Batching

One turn, one request#

A context queues each question and flushes the queue on the next microtask. Everything created before the current synchronous turn ends goes out together:

const t = await semantic(ticket).batch({
  urgency: score("A human needs to act on this urgently."),
  frustration: score("The person writing this sounds frustrated or angry."),
  security: is("This describes a security or access-control problem."),
  team: choose(["billing", "technical", "security"]),
})
// 1 request, 4 answers

batch() is the explicit form: it names the results and gives you a typed object. Promise.all on the same context batches identically:

const s = semantic(ticket)
const [urgency, frustration] = await Promise.all([
  s.score("A human needs to act on this urgently."),
  s.score("The person writing this sounds frustrated or angry."),
])
// also 1 request

What cannot batch#

Sequential awaits cannot share a request. The second question does not exist until the first has resolved:

const s = semantic(ticket)
const urgency = await s.score("...")       // request 1
const frustration = await s.score("...")   // request 2

When this happens outside production, the runtime warns once per context:

[semantic] This context issued a second provider request. Questions awaited one at a time cannot share a request. Use `batch({...})` or `Promise.all([...])` to send them together — batching is dramatically cheaper and no slower.

Turn it off with warnUnbatched: false, or route it to a hook with observability.onUnbatched. See Configuration.

What it saves#

Measured against the live API on a short support ticket with seven questions, median of three runs (2026-09):

5.0× cheaper and 3.2× faster for the same seven answers. The saving grows with the size of the state, because a sequential call re-sends the whole state every time. On a long document it approaches a full N×.

How it works#

1. is(), score() and choose() each push a planned question onto the context and return a promise.
2. The first push schedules a flush with queueMicrotask.
3. When the flush runs, pending questions are grouped by provider and sent as one SemanticRequest per provider.
4. Every promise settles from the same response.

Two details follow from this:

• Timeouts. When questions with different timeoutMs share a request, the largest applies to the request.
• Providers. A question with its own provider option joins a separate request to that provider, in the same turn.

Batching inside definitions#

A schema resolves every field in one request, however many there are. A defined metric or rule exposes .question() so it can join a batch with ad-hoc questions:

const t = await semantic(ticket).batch({
  churn: churnRisk.question(),          // a defineMetric
  safe: safeToAutoReply.question(),     // a defineRule
  team: choose(["billing", "technical"]),
})

Batching across items#

Batching is about one state and many questions. Many states are a different problem: each item is its own request, because a request carries one state. Collections explains which operations cost one request and which cost N, and the RAG gate scenario shows how to fold many items into one state when they are small enough.

Proving it in tests#

The mock provider counts requests. This assertion is the one that catches a refactor that broke batching:

const provider = createMockSemanticProvider({ urgently: 0.9, frustrated: 0.7 })
const semantic = createSemantic({ provider })

await semantic(ticket).batch({ urgency: score("..."), frustration: score("...") })

assert.equal(provider.requestCount, 1)