Quickdraw
Measures how an LLM streams on your network and your prompts.
- TypeScript
- Node.js
- GitHub Actions
The problem
Claims that a model is fast come from someone else's network and someone else's prompts. The numbers that matter are the ones your own app will hit.
How it works
Quickdraw records time to first token, tokens per second, latency and cost for your prompts, in CI so the numbers become a trend.
- A command-line tool sends your prompts to a model and times the stream as it arrives.
- Cost comes from a table of public per-token prices.
- Results are written to CSV and JSON so they can be diffed and charted.
- Run on a schedule in CI, each run adds a point to the history instead of being read once and forgotten.
What it does not do
- It sends one request at a time, so it shows nothing about behaviour under concurrent load.
- Cold and warm starts are not separated, so a slow first call can skew a run.
- Cost uses list prices and ignores any discount you have negotiated.
- The numbers describe your network and prompts, not a general ranking of models.