Goldset

A GitHub Action that blocks a merge when an AI app's answers get worse.

  • TypeScript
  • GitHub Actions
Codenpm

The problem

Snapshot tests assume the output never changes, and model output does, so a byte-for-byte diff fails on every harmless rewording. Reading output diffs by hand works for a handful of examples and then stops getting read carefully.

How it works

Goldset scores the current output against a golden dataset with a judge model and posts the delta on the PR.

  • It runs as a GitHub Action on every pull request.
  • The golden dataset is plain files in the repo, so it is versioned with the code it guards.
  • A judge model scores the app's current answers against the golden ones. The judge prompt and the pass threshold are yours to set.
  • Structural checks cover the shape of the output, separately from the judge's score.
  • The result is posted on the PR as the change against the base branch. A drop past your threshold fails the check and blocks the merge.

What it does not do

  • The judge is a model, with its own bias, and it can shift when the provider updates it.
  • Each golden case is a single exchange. Multi-turn conversations are not covered yet.
  • Every case costs one judge call per run, so cost grows with the dataset.

Links