Looking for a Jev benchmark you can trust? The honest answer is that there are many small ones, measuring different things. This post collects every public test we could verify, explains what each one actually measured, and adds our own run on four public datasets, with the raw responses published so you can check our work.
How to read a Jev benchmark
Before comparing numbers, check three things about any test:
- Who wrote the reference answers? Some benchmarks compare Jev with human labels; others compare it with what a larger model said. Agreement with another model is not the same as being right.
- Could the data be in Jev’s training set? Public datasets such as OpenBookQA have been around for years. High scores there say less about new data than a freshly generated test does.
- Was the answer even in the input? Jev always picks one of your options. A test where the deciding fact is missing measures overconfidence, not understanding.
Two terms come up often. Accuracy is the share of answers that match the reference. Calibration asks whether the stated probabilities are honest: of all the answers given with 90% confidence, about 90% should be right. It is usually summarised as ECE (expected calibration error), where 0 is perfect.
What TypeSafe reports
TypeSafe’s launch post reports Jev as up to 193.6× faster and 444.6× cheaper than frontier LLMs on its own “workflow evals”, with 70–500 ms end-to-end latency. Read those as best cases: the comparison models are large frontier LLMs, and the reference answers came from other models rather than people. On that suite Jev agreed with the reference about 68% of the time. TypeSafe also claims 0% type errors, which is a guarantee about the answer’s format, not its correctness.
TypeSafe’s own documentation is refreshingly candid about weak spots in jev-1.13: literal reading, maths and counting, dates, indirection, long irrelevant input, and adversarial text. Its cookbook reports that asking 13 questions in one call was 12.2× cheaper and 10× faster than 13 separate calls, with the same answers.
Independent tests
| Test | What it measured | Result | Caveat |
|---|---|---|---|
| jev-ood-calibration | 3 public benchmarks + 900 synthetic tickets | OpenBookQA 94.2%, CommonsenseQA 88.1%, HellaSwag 86.1%; ticket queue 89.0%, angry 91.7% | Public sets may be in training data |
| Same study, “priority” | A label set by a rule that isn’t in the text | 44.7% accuracy at an average confidence of 0.74 | Deliberately unanswerable |
| jev-spam-eval | ~9,900 real emails: ham, spam, phishing | 98.6% with added context | No calibration reported |
| jev-phishing-bench | 2,000 hard phishing emails | 62.6%; ECE 0.154 vs Claude Haiku 4.5’s 0.097 | A deliberately difficult set |
| Laya’s benchmark page | Laya vs published Jev numbers | Banking77: Jev 0.870, Laya 0.425; AG News: Jev 0.910, Laya 0.953 | Jev figures not measured by Laya’s authors |
Across these, two results hold up. First, the format is reliable: the calibration study logged zero invalid responses in 4,621 calls. Second, calibration depends on question type: yes/no answers came out underconfident, while choice and score answers were overconfident. The same study found TypeSafe’s separate confidence field was never a better signal than the top probability.
Our Jev benchmark
We wanted numbers we could stand behind, so we ran our own test on 5 tasks built from four public datasets, with the same sampled items sent to Jev and to Laya, an open-weight alternative. Every raw response is published: summary.json, Jev responses, Laya responses. The head-to-head is written up in Jev vs Laya; here we focus on Jev.
Accuracy
| Task | Items | Jev accuracy | Laya accuracy |
|---|---|---|---|
| Banking77 intent (77 options) | 100 | 79% 70%–86% | 35% 26%–45% |
| AG News topic (4 options) | 32 | 78% 61%–89% | 97% 85%–99% |
| Emotion (6 options) | 200 | — | 56% 49%–62% |
| SST-2 sentiment (yes/no) | 200 | — | 48% 41%–55% |
| SST-2 sentiment as a choice (2 options) | 200 | — | 93% 89%–96% |
Findings are finalised when the run completes.
Calibration
| Task | Jev avg. confidence | Jev ECE | Laya avg. confidence | Laya ECE |
|---|---|---|---|---|
| Banking77 intent (77 options) | 89% | 0.103 | 90% | 0.557 |
| AG News topic (4 options) | 97% | 0.210 | 89% | 0.077 |
| Emotion (6 options) | — | — | 82% | 0.265 |
| SST-2 sentiment (yes/no) | — | — | 97% | 0.501 |
| SST-2 sentiment as a choice (2 options) | — | — | 89% | 0.088 |
Findings are finalised when the run completes.
Speed and cost
| Task | Jev median / p95 | Laya median / p95 | Jev tokens per call | Jev cost per 1,000 |
|---|---|---|---|---|
| Banking77 intent (77 options) | 1.7 s / 3.9 s | 199 ms / 230 ms | 1687 | $0.071 |
| AG News topic (4 options) | 1.8 s / 3.8 s | 80 ms / 183 ms | 427 | $0.018 |
| Emotion (6 options) | — / — | 89 ms / 130 ms | — | — |
| SST-2 sentiment (yes/no) | — / — | 117 ms / 601 ms | — | — |
| SST-2 sentiment as a choice (2 options) | — / — | 56 ms / 70 ms | — | — |
Findings are finalised when the run completes.
What it means for your app
Findings are finalised when the run completes.
Method and limits
- Data. A fixed random sample (seed 20260926) from the test splits of Banking77 (77 intents, CC BY 4.0), AG News (4 topics), and Emotion (6 emotions), and the SST-2 validation split, asked once as a yes/no question and once as a two-option choice.
- Questions. One question per call, identical for both models, with a short description for every option. The exact wording is in the summary file.
- Jev. Model
typesafe-ai/jevthrough Vercel AI Gateway with the AI SDK, zero data retention on, one call at a time. Latency is the successful attempt only; waits caused by rate limits are excluded. - Scoring. Accuracy against the dataset label, with Wilson 95% intervals; ECE over 10 equal-width confidence bins, using the probability of the chosen answer.
- Limits. Small samples; public datasets that may be in either model’s training data; one wording per task. Treat differences of a few points as noise.
Frequently asked questions
How accurate is Jev?
It depends heavily on the task. Independent tests range from about 45% on a question whose answer wasn't in the text, to 62.6% on a hard phishing set, to 94.2% on OpenBookQA. Test on labelled examples from your own traffic before relying on any single figure.
Is Jev well calibrated?
Partly. On public benchmarks its probabilities needed little correction, but on unseen tasks independent tests found choice and score answers tend to be overconfident while yes/no answers tend to be underconfident. Set thresholds per question from your own data.
How fast is Jev?
TypeSafe reports 70–500 ms end to end. Our calls through Vercel AI Gateway had a median around 1–2 seconds including network overhead, and rate limits added waits when we sent requests in parallel.
Where can I download the raw benchmark data?
Our raw Jev and Laya responses and the summary are published at jevai.tools/data/benchmarks/2026-09/. The scripts that produced them are described in the Method section.
Sources
- TypeSafe AI Introducing System One Models & Jev (workflow evals) ↗
- TypeSafe docs Jev 1.13 jaggedness ↗
- GitHub scienthoon/jev-ood-calibration ↗
- GitHub SamuelSacco/jev-exploration (claims ledger) ↗
- GitHub bitnovus/jev-spam-eval ↗
- GitHub anisselbd/jev-phishing-bench ↗
- GitHub Laya BENCHMARKS.md ↗
- TypeSafe docs Cookbook: parallel questions ↗
Third-party figures are quoted as their authors published them (checked 26 September 2026); we did not re-run those studies. Our own numbers come from live calls on 26 September 2026. This site is not affiliated with TypeSafe AI or Convai Innovations.