# Testing

> Run everything around a semantic call without a model or a key. The mock provider answers from a table and counts requests.

Section: Guides · HTML: https://jevascript.org/docs/testing · Markdown: https://jevascript.org/docs/testing.md

Application logic around a semantic call is ordinary code, and it should be tested like ordinary code: offline, fast, deterministic. `jevascript/testing` provides a provider that answers from a lookup table.

## The mock provider

```ts
import { createMockSemanticProvider } from "jevascript/testing"

const provider = createMockSemanticProvider({
  urgently: 0.9,          // a truth question mentioning "urgently" → probability 0.9
  "may leave": false,     // → 0.05
  "own this ticket": "payments",   // a choice mentioning "own this ticket" → "payments"
})
```

**Keys match as substrings of the question's instructions**, case-insensitive. A rule like `urgently` answers `score("A human needs to act on this urgently.")` however it was framed, so a stub survives rewording. When several keys match, the first in object order wins. When none match, `fallback` is used: `0.5` for truth questions, the first option for choices, level `0` for rubrics.

### Values and what they become

| Value | Truth question | Choice question | Level question |
|---|---|---|---|
| `true` / `false` | `0.95` / `0.05` | first option | level 0 |
| number | the probability | first option | that rubric position (0-indexed) |
| string | `0.5` | that option, if it exists | level 0 |
| a `SemanticAnswer` object | as given | as given | as given |

### Options

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `fallback` | `MockValue` | — | Answer when no rule matches. |
| `jitter` | `number` | `0` | Adds ±jitter noise to truth probabilities, so sampling-based confidence can be exercised. |
| `model` | `string` | `"mock-1"` | The model string reported in events and cache keys. |
| `capabilities` | `Partial<ProviderCapabilities>` | — | Override the defaults (255 options, 2–10 levels) to test `UnsupportedByProviderError` paths. |

### What it records

```ts
provider.requestCount   // requests actually issued
provider.calls          // [{ state, questions }] per request
provider.reset()        // clear both
```

## Swapping the provider

Prefer a `createSemantic()` instance in the application and reconfigure it in tests. The global instance is never touched:

```ts
// lib/semantic.ts
export const semantic = createSemantic({ provider: jev(), defaults: { timeoutMs: 5_000 } })
```

```ts
// triage.test.ts
import { semantic } from "../lib/semantic"

const provider = createMockSemanticProvider({ urgently: 0.9 })
semantic.configure({ provider, warnUnbatched: false, observability: {} })
```

With the module-level `semantic`, use `configureSemantic({ provider })` and `resetSemantic()` between tests.

## A complete test

The service under test here is the one from the [ticket triage](https://jevascript.org/docs/scenarios/ticket-triage) scenario.

  **node:test**

```ts
// tickets.test.ts
import { strict as assert } from "node:assert"
import { describe, it } from "node:test"
import { ProviderError } from "jevascript"
import { createMockSemanticProvider } from "jevascript/testing"
import { semantic } from "../lib/semantic"
import { TicketService } from "../tickets"

function app(rules: Record<string, number | boolean | string>) {
  const provider = createMockSemanticProvider(rules)
  semantic.configure({ provider, warnUnbatched: false, observability: {} })
  const jobs: { name: string }[] = []
  return { service: new TicketService(memoryDb(jobs)), provider, jobs }
}

describe("TicketService.intake", () => {
  it("pages on-call for a blocking enterprise ticket", async () => {
    const { service, provider, jobs } = app({
      "unable to take money": true,
      urgently: 0.95,
      "own this ticket": "payments",
    })

    const record = await service.intake({ customerId: "enterprise-1", subject: "Charges failing", body: "All declines." })

    assert.equal(record.priority, "critical")
    assert.equal(record.queue, "payments")
    assert.deepEqual(jobs.map((j) => j.name), ["page-oncall"])
    assert.equal(provider.requestCount, 1, "the whole schema is one request")
  })

  it("never pages on-call for a free plan, however bad it looks", async () => {
    const { service, jobs } = app({ "unable to take money": true, urgently: 0.95 })
    const record = await service.intake({ customerId: "free-1", subject: "down", body: "everything broken" })

    assert.equal(record.priority, "high")
    assert.deepEqual(jobs, [])
  })

  it("keeps working when the provider is down", async () => {
    const provider = createMockSemanticProvider({})
    semantic.configure({
      provider: { ...provider, async evaluate() { throw new ProviderError("upstream 503", 503) } },
    })
    const service = new TicketService(memoryDb([]))

    const record = await service.intake({ customerId: "enterprise-1", subject: "API is down", body: "outage" })

    assert.equal(record.gradedBy, "fallback")
  })
})
```
  
  **Vitest**

```ts
// tickets.test.ts
import { describe, expect, it } from "vitest"
import { ProviderError } from "jevascript"
import { createMockSemanticProvider } from "jevascript/testing"
import { semantic } from "../lib/semantic"
import { TicketService } from "../tickets"

function app(rules: Record<string, number | boolean | string>) {
  const provider = createMockSemanticProvider(rules)
  semantic.configure({ provider, warnUnbatched: false, observability: {} })
  const jobs: { name: string }[] = []
  return { service: new TicketService(memoryDb(jobs)), provider, jobs }
}

describe("TicketService.intake", () => {
  it("pages on-call for a blocking enterprise ticket", async () => {
    const { service, provider, jobs } = app({
      "unable to take money": true,
      urgently: 0.95,
      "own this ticket": "payments",
    })

    const record = await service.intake({ customerId: "enterprise-1", subject: "Charges failing", body: "All declines." })

    expect(record.priority).toBe("critical")
    expect(record.queue).toBe("payments")
    expect(jobs.map((j) => j.name)).toEqual(["page-oncall"])
    expect(provider.requestCount).toBe(1)
  })

  it("never pages on-call for a free plan, however bad it looks", async () => {
    const { service, jobs } = app({ "unable to take money": true, urgently: 0.95 })
    const record = await service.intake({ customerId: "free-1", subject: "down", body: "everything broken" })

    expect(record.priority).toBe("high")
    expect(jobs).toEqual([])
  })

  it("keeps working when the provider is down", async () => {
    const provider = createMockSemanticProvider({})
    semantic.configure({
      provider: { ...provider, async evaluate() { throw new ProviderError("upstream 503", 503) } },
    })
    const service = new TicketService(memoryDb([]))

    const record = await service.intake({ customerId: "enterprise-1", subject: "API is down", body: "outage" })

    expect(record.gradedBy).toBe("fallback")
  })
})
```
  

## What to assert

- **Decisions, not probabilities.** Assert on `priority`, on which jobs were enqueued, on what was refused. The stub decides the probabilities; your code decides the rest.
- **`requestCount`.** The assertion that proves batching survived a refactor.
- **Deterministic gates.** A test where the model says "yes" and the plan says "no" is the most valuable test in the suite.
- **Degradation.** Throw `ProviderError` from a wrapped provider and assert the fallback path ran.
- **Low confidence.** Use `jitter` with `samples` to make a truth answer unstable, then assert the `fallback` was used or `LowConfidenceError` was thrown.

```ts
const provider = createMockSemanticProvider({ fraudulent: 0.5 }, { jitter: 0.3 })
const semantic = createSemantic({ provider })

await assert.rejects(
  semantic(order).score("This order is fraudulent.", { asProposition: true, minConfidence: 0.95 }),
  LowConfidenceError,
)
```

## Testing definitions

A `defineSchema` evaluator is a function; call it with the mock configured and check the typed result. Its `definition` property is public, so wording can be snapshot-tested to catch accidental changes without a version bump:

```ts
expect(triage.version).toBe("4")
expect(triage.definition.output).toMatchSnapshot()
```

## Live tests

Keep a small, separate suite that runs against the real provider with `JEV_API_KEY` set, and skip it when the variable is absent. Assert on bands and orderings, not exact numbers; providers are not deterministic. The example scripts in the library repository are written this way.