Guides

Observability

Every evaluation emits one event with latency, request count, token usage, cache status and the definition it came from. Wire it to whatever you already use.

`onEvaluation`#

One event per operation, after it completes:

lib/semantic.ts

export const semantic = createSemantic({
  provider: jev(),
  observability: {
    onEvaluation(e) {
      metrics.histogram("semantic.latency_ms", e.latencyMs, { op: e.operation, metric: e.metric?.name })
      metrics.counter("semantic.requests", e.requestCount)
      metrics.counter("semantic.tokens", e.usage?.inputTokens ?? 0)
      if (e.cached) metrics.counter("semantic.cache_hits")
    },
  },
})

The hook must never break the call it observes: exceptions thrown inside it are swallowed.

`EvaluationEvent`#

Field	Type	Default	Description
operation	"is" \| "score" \| "choose" \| "batch" \| "evaluate" \| "rank" \| "filter" \| "find" \| "compare"	—	What was called. A batch of one is reported as `evaluate`.
keys	string[]	—	The question keys in the request.
provider	string	—	Provider name.
model	string	—	Pinned model.
latencyMs	number	—	Wall time for the whole operation, including retries and sample rounds.
cached	boolean	—	`true` when every question was served from the cache.
batched	boolean	—	More than one question shared the request.
questionCount	number	—	Questions in the operation. For collections, the number of items.
requestCount	number	—	Requests actually issued. Greater than one when `samples` was in play, or for N-request collections.
samples	number	—	The largest sample count among the questions.
usage	{ inputTokens?, outputTokens? }	—	Summed over requests, when the provider reports it.
metric	{ name, version? }	—	The definition every question came from, when they all came from the same one.

`onUnbatched`#

Fires when a context issues a second request that could have been batched. Without the hook, the runtime prints a warning outside production; with it, nothing is printed and you decide:

observability: {
  onUnbatched({ context, flushCount }) {
    logger.warn("semantic.unbatched", { context, flushCount, stack: new Error().stack })
  },
}

Set warnUnbatched: false to disable both. See Batching.

Naming through definitions#

A definition stamps its identity on every event it produces. A batch whose questions all come from the same definition carries that identity; a mixed batch is reported without one rather than mislabelled.

onEvaluation(e) {
  // e.metric → { name: "ticket-triage", version: "4" }
  logger.info("semantic.eval", { definition: `${e.metric?.name}@${e.metric?.version}`, ms: e.latencyMs })
}

This is what makes results comparable over time: a dashboard keyed on name@version never mixes evaluations from two wordings.

Estimating cost#

usage.inputTokens is the number to price. A Jev request returns no generated text, so output tokens are zero. The examples in the library repository use a constant price per million input tokens and sum usage across events; the same three lines work in production:

let inputTokens = 0
observability: { onEvaluation: (e) => { inputTokens += e.usage?.inputTokens ?? 0 } }
// later
const usd = (inputTokens / 1_000_000) * PRICE_PER_MILLION_INPUT_TOKENS

Tracing#

jev() accepts a fetch option. Pass an instrumented fetch to get a span per HTTP request with the headers and timing your tracer already understands:

jev({ fetch: tracedFetch })

Logging the decision#

Observability tells you what a call cost. For why a decision was made, ask for the raw signal with detailed: true and store it next to the record: the probability, the confidence if measured, the distribution for a choice. See Uncertainty.

onEvaluation#

EvaluationEvent#

onUnbatched#