# Observability

> Every evaluation emits one event with latency, request count, token usage, cache status and the definition it came from. Wire it to whatever you already use.

Section: Guides · HTML: https://jevascript.org/docs/observability · Markdown: https://jevascript.org/docs/observability.md

## `onEvaluation`

One event per operation, after it completes:

```ts
// lib/semantic.ts
export const semantic = createSemantic({
  provider: jev(),
  observability: {
    onEvaluation(e) {
      metrics.histogram("semantic.latency_ms", e.latencyMs, { op: e.operation, metric: e.metric?.name })
      metrics.counter("semantic.requests", e.requestCount)
      metrics.counter("semantic.tokens", e.usage?.inputTokens ?? 0)
      if (e.cached) metrics.counter("semantic.cache_hits")
    },
  },
})
```

The hook must never break the call it observes: exceptions thrown inside it are swallowed.

### `EvaluationEvent`

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `operation` | `"is" \| "score" \| "choose" \| "batch" \| "evaluate" \| "rank" \| "filter" \| "find" \| "compare"` | — | What was called. A batch of one is reported as `evaluate`. |
| `keys` | `string[]` | — | The question keys in the request. |
| `provider` | `string` | — | Provider name. |
| `model` | `string` | — | Pinned model. |
| `latencyMs` | `number` | — | Wall time for the whole operation, including retries and sample rounds. |
| `cached` | `boolean` | — | `true` when every question was served from the cache. |
| `batched` | `boolean` | — | More than one question shared the request. |
| `questionCount` | `number` | — | Questions in the operation. For collections, the number of items. |
| `requestCount` | `number` | — | Requests actually issued. Greater than one when `samples` was in play, or for N-request collections. |
| `samples` | `number` | — | The largest sample count among the questions. |
| `usage` | `{ inputTokens?, outputTokens? }` | — | Summed over requests, when the provider reports it. |
| `metric` | `{ name, version? }` | — | The definition every question came from, when they all came from the same one. |

## `onUnbatched`

Fires when a context issues a second request that could have been batched. Without the hook, the runtime prints a warning outside production; with it, nothing is printed and you decide:

```ts
observability: {
  onUnbatched({ context, flushCount }) {
    logger.warn("semantic.unbatched", { context, flushCount, stack: new Error().stack })
  },
}
```

Set `warnUnbatched: false` to disable both. See [Batching](https://jevascript.org/docs/batching).

## Naming through definitions

A [definition](https://jevascript.org/docs/definitions) stamps its identity on every event it produces. A batch whose questions all come from the same definition carries that identity; a mixed batch is reported without one rather than mislabelled.

```ts
onEvaluation(e) {
  // e.metric → { name: "ticket-triage", version: "4" }
  logger.info("semantic.eval", { definition: `${e.metric?.name}@${e.metric?.version}`, ms: e.latencyMs })
}
```

This is what makes results comparable over time: a dashboard keyed on `name@version` never mixes evaluations from two wordings.

## Estimating cost

`usage.inputTokens` is the number to price. A Jev request returns no generated text, so output tokens are zero. The examples in the library repository use a constant price per million input tokens and sum `usage` across events; the same three lines work in production:

```ts
let inputTokens = 0
observability: { onEvaluation: (e) => { inputTokens += e.usage?.inputTokens ?? 0 } }
// later
const usd = (inputTokens / 1_000_000) * PRICE_PER_MILLION_INPUT_TOKENS
```

## Tracing

`jev()` accepts a `fetch` option. Pass an instrumented `fetch` to get a span per HTTP request with the headers and timing your tracer already understands:

```ts
jev({ fetch: tracedFetch })
```

## Logging the decision

Observability tells you what a call cost. For *why* a decision was made, ask for the raw signal with `detailed: true` and store it next to the record: the probability, the confidence if measured, the distribution for a choice. See [Uncertainty](https://jevascript.org/docs/uncertainty).