From transformers to decision models

The problem

A lot of everyday "AI" work is really classification. Route this ticket. Is this claim covered? Does this message need a human? You want one answer from a fixed list, and a number that says how sure the model is. A chatbot gives you a paragraph you then have to parse.

There is a cheap trick for this. A language model already computes a probability for every possible next token. If the prompt lists lettered options and ends at Answer:, the probabilities of the tokens A, B and C are sitting right there after one forward pass. No training, no new weights, nothing generated. TypeSafe AI calls this kind of typed, fast answer "System One".

Cheap is not the same as good. I wanted to know what the shortcut costs, and to pull apart which errors belong to the base model, which come from squeezing a decision into a single next-token read, and which come from the option order or the inference stack.

How it works

I wrote myNameisJevToo, a Python library and a local HTTP server. It loads a model with MLX on Apple silicon, or talks to a running llama.cpp server, and answers three question types: choice (one of a list), noul (yes or no) and score (a level on an ordered scale).

1. Render the task.The state, the instructions and the options, each written as a letter and a label, then a blank line and Answer:. The rubric text for each option goes in too, because that is where the actual criterion lives.
2. Check the letters are single tokens." A", " B" and so on must each be one token in the model's tokeniser, so every option can be read from the same distribution. Labels A to I were checked; coverage was 100%, with none of 1,698 labels missing.
3. Run one forward pass.The model processes the whole prompt once. At the last position it produces a probability for every token in its vocabulary.
4. Read and renormalise.Take the probability of each option letter, then divide by their sum. The biggest share is the answer and the share is its confidence. Whatever went to other tokens is set aside, but kept as a number.
5. Serve it.POST /v1/systemone takes a state and named questions and returns typed answers in TypeSafe AI's published request and answer shape, with optional per-question diagnostics.

<state>

<instructions>

A. <label> - <what it means>
B. <label> - <what it means>
C. <label> - <what it means>

Answer:

For options that are not single tokens, the library prefills the state once and scores each option against a cloned cache. Against a full re-prefill that path was 4.2 times faster and bit-identical.

Two numbers for every answer

The renormalising step hides something. In the README example the letters get 0.04, 0.26 and 0.06. B's share is 0.26 ÷ 0.36, about 72%. But only 36% of all the probability landed on a valid letter; the other 64% went elsewhere in the vocabulary. So the library keeps both: the option mass (how much probability landed on valid letters at all) and the share (given a valid option, which one wins).

In the real runs the gap is far wider than in the example. I took the median option mass over the 111 hard items for each configuration from the committed per-item records. The 27B models put only 4% to 8% of their probability on the letters, and their shares were still the best calibrated. The 2B model put most of its mass on the letters and was the worst calibrated. Option mass is worth logging, but it is not confidence.

What I measured

JevBench is an open benchmark for decision models, with its own scoring formulas. Its public items are 48 easy, 72 standard and 111 hard. The judge tier and the private halves are not public, so these are scores on the public items only, with the tier weights renormalised over three tiers. Every configuration saw the same 231 items and was scored with JevBench's own formulas.

I ran five configurations: the 2B MiniCPM5 model, a ternary 27B model (Ternary-Bonsai-2-27B), and one 27B model, Qwen3.8-27B, in three quantisations across two inference stacks. The 2B ran on a Mac mini M4 (16 GB); the 27B legs ran on a Mac Studio M3 Ultra (96 GB).

• 82/111hard items right in one pass (Qwen3.8-27B, GGUF Q8_0)
• 82.0intelligence axis on the public items, with 86.1 calibration
• 23/29of its hard misses answered correctly with a thinking budget
• 70.7%of its hard accuracy kept with the state removed

The three-axis score leaves out cost, so it is not a JevBench Score, and it can't be lined up against the vendors' published board numbers, which come from their own harnesses on the full board.

No precision ladder. Exact paired McNemar tests on the hard items can't tell the three Qwen3.8-27B quantisations apart. With 111 items, one item is 0.9 points of accuracy and 0.57 intelligence points.

Calibration follows the quantisation scheme

Calibration asks whether a 70% answer is right about 70% of the time. JevBench derives it from the hard tier's expected calibration error. My own earlier notes said calibration was the precision-sensitive axis, because GGUF Q4_K_M scored worse than the 8-bit build. Adding the MLX 4-bit leg showed that was wrong: a 4-bit MLX build calibrated as well as 8-bit GGUF.

The state-blind control

An open audit of Jev, the jev-calibration-audit, found that much multiple-choice accuracy can be recovered from the option list alone. So I ran the same check on every configuration: replace the state with the placeholder [state withheld], keep the question and the options, and score again.

The 27B models do read the state: they lose 16 to 22 points without it. But a correct answer from this readout is not proof that the model used the evidence.

Option order can be the answer. Two-option questions showed a sharper version of this on the 2B model. Across 74 public yes/no items it scored 54.1% in one order and 52.7% reversed, which looks stable until you notice that a model that always picks the first slot also scores about 50% either way. The repo's own refund test makes it visible: the state says "I need this refunded today", and the model picked whichever word came first, with or without the state (try Refund above). Two-option reads need an order flip before you trust them.

A thinking budget

Where the hard items fail says something. On Q8_0 the weakest families were temporal_numeric (6 of 15) and probability (6 of 10): arithmetic over dates and amounts, and probability estimates. A one-pass readout has to land on the answer at the letter position, with no room to work anything out.

So I took the 29 hard items Q8_0 got wrong and re-ran the identical prompts, this time letting the same model generate greedily with up to 4,000 tokens of thinking. It answered 23 of the 29 correctly.

That moves the hard tier from 82/111 (73.9%) to at most 105/111 (94.6%), and the intelligence axis from 82.0 to about 95.0. About 95.0 is an upper bound, not a measured score. Only the misses were re-tried, so it assumes reasoning never breaks one of the 82 answers the readout already had right, and a reasoning model can talk itself out of a right answer. Three of the 29 re-tries stopped at the token cap mid-reasoning, so 23 is also a floor on what was recoverable. The real number needs all 111 hard items run with reasoning, and I haven't done that.

The trade is time. The recovered answers took from about 15 seconds to just over three minutes each on the Mac Studio, against a one-pass median of 589 ms.

Latency tracks context, not model size

Ternary-Bonsai's weights are 6.7 GB against Q8_0's 27 GB, yet their latency matched: a median of 600 ms against 589 ms, and a 95th percentile of 8.7 s against 8.5 s. The long tail is prefill. Hard-item states run to about 3,700 tokens, and the model has to read all of them before the one cheap read at the end. Skipping generation removes decoding time, not reading time.

Limits

• Public items only: 231 of them, and no judge tier. These are not full-board JevBench scores, and the vendor figures on JevBench's board are not comparable to them.
• The thinking-budget figure re-tried only the 29 misses. A full reasoned run of all 111 hard items was not done, so about 95.0 stays an upper bound.
• Single runs, one seed, two Apple machines. McNemar tests handle pairing on the hard items but are no substitute for replication.
• The GGUF and MLX legs differ in both quantiser and library, so the calibration result can't be pinned on one of them.
• The binary order test showed position-picking on the 2B model; I did not repeat the order flip on the 27B legs.
• A suspected min_p artefact in llama.cpp was checked and ruled out, but that probe's raw output was not archived. Xing4.0-29B-A4B could not be run at all: llama.cpp does not know its architecture.
• Results are task-specific and include confident errors (try Escalation above). The write-up is a technical report from LintLabs Research, not a peer-reviewed paper.

Credits

Jev and the System One framing come from TypeSafe AI, and the JevBench benchmark and scoring formulas come from its open repository. The jev-calibration-audit motivated the state-blind control. MiniCPM5 is OpenBMB's; Qwen3.8 is Qwen's; Ternary-Bonsai's PQ2_0 build ran on PrismML's llama.cpp fork. This project is independent and not affiliated with any of them, and the repo redistributes no task text and no model weights.

Read more

• Evaluation report: method, all five legs, the controls, threats to validity.
• myNameisJevToo on GitHub: the library, server, benchmark scripts and a write-up of the pitfalls (double BOS tokens, scaffolds, position priors).
• Committed results: per-item probabilities for every leg and the 29 reasoning re-tries, so each number here can be checked.