Caching

Caching does two jobs. It saves a request when the same question is asked about the same state again. And it stops a re-evaluated record from drifting across a threshold, which is the more important one.

Turn it on#

Per call, per definition, or for a whole instance:

await semantic(doc).score("...", { cache: "1h" })

const triage = defineSchema({ /* ... */, defaults: { cache: "10m" } })

const semantic = createSemantic({ defaults: { cache: "10m" } })

Unset means no caching. A call option overrides the instance default.

TTL formats#

Decimals are accepted ("1.5h"). Anything else throws at call time with a message that lists the valid forms. parseTtl() is exported if you want the same parser.

What the key is#

The key is a hash of five things, with object keys sorted so ordering does not matter:

• the provider name
• the provider model
• the state, as sent
• the question, including trueWhen, falseWhen, options and levels
• the number of samples

So changing the wording of a question, bumping the model, or adding a field to the state is a new key. Nothing stale is ever served for a changed question. The consequence: a state that includes a timestamp or a request id never hits the cache. Strip volatile fields before wrapping.

cacheKeyFor(provider, state, question, samples) is exported if you need the key outside the runtime.

The in-memory store#

MemoryCache is the default: an LRU with per-entry TTL, capped at 5 000 entries. Every createSemantic() instance gets its own; the module-level instance gets one that resetSemantic() replaces.

import { MemoryCache } from "jevascript"

createSemantic({ cacheStore: new MemoryCache(50_000) })

A shared store#

Implement two methods. Both may be sync or async.

interface SemanticCache {
  get(key: string): Promise<SemanticAnswer | undefined> | SemanticAnswer | undefined
  set(key: string, value: SemanticAnswer, ttlMs: number): Promise<void> | void
}

With Redis:

import type { SemanticCache, SemanticAnswer } from "jevascript"
import { redis } from "./redis"

export const cache: SemanticCache = {
  async get(key) {
    const raw = await redis.get(`sem:${key}`)
    return raw ? (JSON.parse(raw) as SemanticAnswer) : undefined
  },
  async set(key, value, ttlMs) {
    await redis.set(`sem:${key}`, JSON.stringify(value), "PX", ttlMs)
  },
}

export const semantic = createSemantic({
  provider: jev(),
  cacheStore: cache,
  defaults: { cache: "30m" },
})

Cached values are the raw provider answers, so a cache shared between processes serves the same probabilities everywhere and thresholds are applied at read time. A store may also be swapped later with semantic.configure({ cacheStore }).

What gets cached#

• Each question is cached individually, even inside a batch. A batch where three of four questions hit only sends the fourth.
• Answers from sampling are cached as the aggregated answer, keyed with the sample count, so a sampled and an unsampled question never share an entry.
• Collection operations cache per item.
• A cached: true observability event means every question in that operation was served from the store. See Observability.

When to cache#

• Always where a record can be re-evaluated for the same input: retries, webhook replays, idempotent handlers, list views that re-render.
• Long for judgements about immutable content: a document, a listing, a historical message.
• Not at all when the state changes on every call, or when you are measuring the model itself.