Goldset

## What it does Goldset locks an AI app's behavior so a pull request can't quietly change it. Three runners cover it: a golden-dataset check, an LLM judge that scores the current output, and structural assertions on shape. It ships as a GitHub Action, so the check runs in CI on every PR and comments the delta against the base branch. A regression blocks the merge instead of slipping through review. ## Why this approach Snapshot tests assume deterministic output, which LLM responses aren't, so a byte-for-byte diff fails on every harmless rewording. The alternative — a human eyeballing output diffs — works fine until you pass about twenty examples, then nobody reads them carefully. An LLM judge scoring against golden examples scales past that point without pretending the output is deterministic. ## How it's built A GitHub Action that loads your golden dataset, runs the app's current output through a judge model, and posts the delta-vs-base scores back to the PR as a comment. The judge talks to a pluggable provider — OpenAI or Anthropic — so you pick the model that scores your domain best. The judge prompt and the pass threshold are yours to configure. The dataset is plain files in the repo, so it versions alongside the code it guards. ## Known limitations - The judge is a model, so it has its own bias and can drift between provider updates. - No multi-turn examples yet. Each golden case is a single exchange. - Cost scales linearly with the dataset; every example is a judge call per run. - A vague judge prompt gives noisy scores, and writing a sharp one takes iteration. ## Where it runs - GitHub: github.com/ykstorm/goldset - npm: @ykstormsorg/goldset