Guides

Testing

Run everything around a semantic call without a model or a key. The mock provider answers from a table and counts requests.

Application logic around a semantic call is ordinary code, and it should be tested like ordinary code: offline, fast, deterministic. jevascript/testing provides a provider that answers from a lookup table.

The mock provider#

import { createMockSemanticProvider } from "jevascript/testing"

const provider = createMockSemanticProvider({
  urgently: 0.9,          // a truth question mentioning "urgently" → probability 0.9
  "may leave": false,     // → 0.05
  "own this ticket": "payments",   // a choice mentioning "own this ticket" → "payments"
})

Keys match as substrings of the question's instructions, case-insensitive. A rule like urgently answers score("A human needs to act on this urgently.") however it was framed, so a stub survives rewording. When several keys match, the first in object order wins. When none match, fallback is used: 0.5 for truth questions, the first option for choices, level 0 for rubrics.

Values and what they become#

Value	Truth question	Choice question	Level question
`true` / `false`	`0.95` / `0.05`	first option	level 0
number	the probability	first option	that rubric position (0-indexed)
string	`0.5`	that option, if it exists	level 0
a `SemanticAnswer` object	as given	as given	as given

Options#

Option	Type	Default	Description
fallback	MockValue	—	Answer when no rule matches.
jitter	number	0	Adds ±jitter noise to truth probabilities, so sampling-based confidence can be exercised.
model	string	"mock-1"	The model string reported in events and cache keys.
capabilities	Partial<ProviderCapabilities>	—	Override the defaults (255 options, 2–10 levels) to test `UnsupportedByProviderError` paths.

What it records#

provider.requestCount   // requests actually issued
provider.calls          // [{ state, questions }] per request
provider.reset()        // clear both

Swapping the provider#

Prefer a createSemantic() instance in the application and reconfigure it in tests. The global instance is never touched:

lib/semantic.ts

export const semantic = createSemantic({ provider: jev(), defaults: { timeoutMs: 5_000 } })

triage.test.ts

import { semantic } from "../lib/semantic"

const provider = createMockSemanticProvider({ urgently: 0.9 })
semantic.configure({ provider, warnUnbatched: false, observability: {} })

With the module-level semantic, use configureSemantic({ provider }) and resetSemantic() between tests.

A complete test#

The service under test here is the one from the ticket triage scenario.

tickets.test.ts

import { strict as assert } from "node:assert"
import { describe, it } from "node:test"
import { ProviderError } from "jevascript"
import { createMockSemanticProvider } from "jevascript/testing"
import { semantic } from "../lib/semantic"
import { TicketService } from "../tickets"

function app(rules: Record<string, number | boolean | string>) {
  const provider = createMockSemanticProvider(rules)
  semantic.configure({ provider, warnUnbatched: false, observability: {} })
  const jobs: { name: string }[] = []
  return { service: new TicketService(memoryDb(jobs)), provider, jobs }
}

describe("TicketService.intake", () => {
  it("pages on-call for a blocking enterprise ticket", async () => {
    const { service, provider, jobs } = app({
      "unable to take money": true,
      urgently: 0.95,
      "own this ticket": "payments",
    })

    const record = await service.intake({ customerId: "enterprise-1", subject: "Charges failing", body: "All declines." })

    assert.equal(record.priority, "critical")
    assert.equal(record.queue, "payments")
    assert.deepEqual(jobs.map((j) => j.name), ["page-oncall"])
    assert.equal(provider.requestCount, 1, "the whole schema is one request")
  })

  it("never pages on-call for a free plan, however bad it looks", async () => {
    const { service, jobs } = app({ "unable to take money": true, urgently: 0.95 })
    const record = await service.intake({ customerId: "free-1", subject: "down", body: "everything broken" })

    assert.equal(record.priority, "high")
    assert.deepEqual(jobs, [])
  })

  it("keeps working when the provider is down", async () => {
    const provider = createMockSemanticProvider({})
    semantic.configure({
      provider: { ...provider, async evaluate() { throw new ProviderError("upstream 503", 503) } },
    })
    const service = new TicketService(memoryDb([]))

    const record = await service.intake({ customerId: "enterprise-1", subject: "API is down", body: "outage" })

    assert.equal(record.gradedBy, "fallback")
  })
})

tickets.test.ts

import { describe, expect, it } from "vitest"
import { ProviderError } from "jevascript"
import { createMockSemanticProvider } from "jevascript/testing"
import { semantic } from "../lib/semantic"
import { TicketService } from "../tickets"

function app(rules: Record<string, number | boolean | string>) {
  const provider = createMockSemanticProvider(rules)
  semantic.configure({ provider, warnUnbatched: false, observability: {} })
  const jobs: { name: string }[] = []
  return { service: new TicketService(memoryDb(jobs)), provider, jobs }
}

describe("TicketService.intake", () => {
  it("pages on-call for a blocking enterprise ticket", async () => {
    const { service, provider, jobs } = app({
      "unable to take money": true,
      urgently: 0.95,
      "own this ticket": "payments",
    })

    const record = await service.intake({ customerId: "enterprise-1", subject: "Charges failing", body: "All declines." })

    expect(record.priority).toBe("critical")
    expect(record.queue).toBe("payments")
    expect(jobs.map((j) => j.name)).toEqual(["page-oncall"])
    expect(provider.requestCount).toBe(1)
  })

  it("never pages on-call for a free plan, however bad it looks", async () => {
    const { service, jobs } = app({ "unable to take money": true, urgently: 0.95 })
    const record = await service.intake({ customerId: "free-1", subject: "down", body: "everything broken" })

    expect(record.priority).toBe("high")
    expect(jobs).toEqual([])
  })

  it("keeps working when the provider is down", async () => {
    const provider = createMockSemanticProvider({})
    semantic.configure({
      provider: { ...provider, async evaluate() { throw new ProviderError("upstream 503", 503) } },
    })
    const service = new TicketService(memoryDb([]))

    const record = await service.intake({ customerId: "enterprise-1", subject: "API is down", body: "outage" })

    expect(record.gradedBy).toBe("fallback")
  })
})

What to assert#

Decisions, not probabilities. Assert on priority, on which jobs were enqueued, on what was refused. The stub decides the probabilities; your code decides the rest.
requestCount. The assertion that proves batching survived a refactor.
Deterministic gates. A test where the model says "yes" and the plan says "no" is the most valuable test in the suite.
Degradation. Throw ProviderError from a wrapped provider and assert the fallback path ran.
Low confidence. Use jitter with samples to make a truth answer unstable, then assert the fallback was used or LowConfidenceError was thrown.

const provider = createMockSemanticProvider({ fraudulent: 0.5 }, { jitter: 0.3 })
const semantic = createSemantic({ provider })

await assert.rejects(
  semantic(order).score("This order is fraudulent.", { asProposition: true, minConfidence: 0.95 }),
  LowConfidenceError,
)

Testing definitions#

A defineSchema evaluator is a function; call it with the mock configured and check the typed result. Its definition property is public, so wording can be snapshot-tested to catch accidental changes without a version bump:

expect(triage.version).toBe("4")
expect(triage.definition.output).toMatchSnapshot()

Live tests#

Keep a small, separate suite that runs against the real provider with JEV_API_KEY set, and skip it when the variable is absent. Assert on bands and orderings, not exact numbers; providers are not deterministic. The example scripts in the library repository are written this way.