Today, we are releasing Karpo Decision 31B, a natively multimodal decision model built on Gemma 4 31B. It reaches state-of-the-art results on several authoritative benchmarks and has a clear advantage in intelligence, cost and speed over Jev 1.14 and other open and closed models. Given a state (a web page, a ticket, a document) and a set of typed questions, Karpo Decision 31B answers many questions in one forward pass, which suits high-frequency, latency-sensitive scenarios such as judgement, ticket handling, safety review, and retrieval and ranking. The weights are open on Hugging Face, and you are welcome to have a try.
73.9 Decision Index Full score, ranked #1 on the board among 300 models.
87.2 JevBench Capability Score, ranked #3 among 243 Jev-class open-weight models on the board.
95 ms median latency, one request at a time on a single RTX PRO 6000, across all 140,217 public requests.

The Decision Index leaderboard (Full score): Karpo Decision 31B ranks first among 300 models.
Besides the 31B model, we have also open-sourced five more model sizes, 0.8B, 2B, 4B, 9B and 12B, forming the Karpo Decision model family. The chart below shows each size's score on DI public (37 benchmarks), with the other board models of the same size class in gray.
Fast and accurate, with probabilities you can trust
A personal agent works out what a user means from a vague request, makes recommendations, searches and decides for the user, and operates the browser on the user's behalf. Every step starts with a judgement. When recognising intent, it has to judge what a vague request is really asking for and whether to act or to ask first; when recommending, whether a product meets the user's requirements; when searching, how relevant a result is to what the user needs; when deciding, whether the task is done and whether a tool should be called; when operating the browser, which element on the page to click. The speed and accuracy of these judgements directly determine how fast the agent responds and whether the task gets done. Karpo Decision 31B is built for this kind of judgement. It is fast and accurate, and it takes multimodal input:
#1 on the board. Decision Index Full score 73.9, ranked #1 among the 300 models on the board; Jev 1.14's Full score is 60.2.
Fast inference. Served on one RTX PRO 6000, it has a median latency of 95 ms on the Decision Index suite; Jev 1.14's median latency on the board is 196 ms.
Many questions per request. A request can ask any number of questions, and each is answered independently; the more questions, the less time each one takes.
Multimodal input. Multimodal input is supported natively: a request can carry images directly, which the vision encoder reads and the model judges together with the text.
Long context. A single sequence is at most 32k tokens by default, and a context window of up to 256k is supported.
Valid answers, probabilities you can trust. The answer is always one of the options, and every answer comes with a calibrated probability. Our ECE (expected calibration error) on the JevBench public items is 0.023.
Open and deployable. The weights and the inference code are public, one RTX PRO 6000 is enough to deploy it, and the interface is compatible with the System One API.
| Benchmark | Karpo Decision 31BOpen weights | Gemini Decision 1Decisions API | TypeSafe Jev 1.14 | GPT-6 LunaOpenAI Decisions API | ||
|---|---|---|---|---|---|---|
| Key benchmarks | JevBench | Public split | 92.2% | 92.2% | 90.0% | 90.9% |
| Amazon ESCI | Macro-F1 | 64.7% | 58.7% | 56.7% | 54.7% | |
| New Yorker Caption | Text-only | 90.5% | 79.7% | 72.0% | 68.4% | |
| Mind2Web Multimodal | 90.0% | 86.2% | 86.8% | 81.8% | ||
| PubMedQA | 79.2% | 79.8% | 78.2% | 74.4% | ||
| Global-MMLU-Lite | Non-English | 85.2% | 86.0% | 85.0% | 79.6% | |
| When2Call | 90.4% | 79.9% | 82.0% | 70.4% | ||
| MMAU | — | 71.8% | — | — | ||
| Video-MME | No subtitles | — | 69.1% | — | — | |
| Calibration | Top-1 ECE | Lower is better | 0.023 | 0.0250 | 0.0494 | 0.0721 |
Karpo Decision 31B figures are our own runs; ESCI, New Yorker and When2Call are independently reproduced by the Jev Decision Index: huggingface.co/spaces/multimodalart/jev-decision-index
On the three kinds of judgement a personal agent makes most often, Karpo Decision 31B is ahead of Gemini Decision 1:
Agent action judgements: When2Call tests whether to call a tool, ask a follow-up question or hold back; Karpo Decision 31B scores 90.4% and Gemini Decision 1 79.9%. Mind2Web tests which element on a web page to click next: 90.0% against 86.2%. These two decide whether an agent places the wrong order or clicks the wrong place.
Reading between the lines: New Yorker Caption asks which of five captions belongs to the cartoon; 90.5% against 79.7%. What users say to an agent is rarely a complete instruction, and what they want has to be judged from tone and context.
Fine-grained business classification: Amazon ESCI asks whether a product is an exact match for a search query, a substitute, a complement or irrelevant; macro-F1 is 64.7% against 58.7%. When an agent picks things for a user, the hard part is telling substitutes from complements.
Karpo Decision 31B answers three question types:
from huggingface_hub import snapshot_download
from karpo_decision.decider import Decider
decider = Decider.from_pretrained(snapshot_download("MachinePulseAI/karpo-decision-31B"), device="cuda")
response = decider.decide({
"state": "Order 1182: the customer asks to return a blender bought 45 days ago. Policy: returns within 30 days.",
"questions": {
"eligible": {
"type": "noul",
"instructions": "Is the return eligible under the policy?"
},
"action": {
"type": "choice", "instructions": "What should the agent do?",
"criteria": {
"refund": "Issue a full refund", "decline": "Politely decline the return",
"escalate": "Escalate to a supervisor"
}
},
"urgency": {
"type": "score", "instructions": "How urgent is this ticket?",
"criteria": ["Low", "Medium", "High"]
},
},
})
print(response["answers"])Model design
Readout head: option probabilities read out directly
Each question's options are labelled with option codes (A, B, … Z, AA, …), followed by a readout marker. The readout head scores 255 option codes at the marker and normalises over that question's own options only. Every question in the request is answered in the same forward pass, with no decoding.
Three implementation details:
Option codes are verified single tokens. We pick option codes from the tokenizer and check that each one is the same single token in each context it appears in. The label the model sees on an option is the same token it is read out from.
The head starts from the LM head. Each row of the readout head is initialised from the base model's LM-head row for that option code's token, with the base model's logit softcap kept. The model recognises the labels from step zero. Geuter et al. likewise initialise a decision head from the LM-head rows of the answer letters [2].
Option order is randomised in training. LLMs prefer particular option labels when they answer multiple-choice questions [3], so in training, wherever the order of a choice question's options carries no meaning, the options are shuffled at random and the question is trained on several times.
Joint multi-question: every question answered in one forward pass
A request is one state plus any number of questions, encoded as a single sequence: first a list of all the questions, then the state, then each question's options and readout marker.
The page is read once. In a browser agent a page is often over 10k tokens, and a question is a few dozen. Asking each question separately would re-read the page every time.
The model knows what it will be asked before it reads. The question list comes first, so the state is read with the questions in view.
No silent truncation. Requests over 32k tokens are split into segments, each with the full shared content and the full question list. If a single question cannot fit, the request fails with an explicit error.
Branch attention: shared computation, independent answers
The sequence has two parts: a shared block (system turn, question list, state) and one branch per question (the question, its options, its readout marker). A branch sees the shared block and itself, never another branch; the shared block sees only itself. Each branch's position ids restart at the end of the shared block, so every question sees the state exactly as it would if it were asked alone. In the global layers, attention is bidirectional within what a token may see; the sliding-window layers stay causal.
Asked together this way, the questions share one computation of the state, yet each question, just as when asked alone, sees no other question's options. Retrieval and ranking need this most, because both judge many candidates one by one.
Most of the engineering went into the training-time implementation:
Planned in advance. Each packed batch is planned once: every branch's queries, the shared-block keys and values it may read, and its own keys and values. All layers and activation recomputation share that plan.
Exact gradients. When branches read the shared block's keys and values, the backward pass accumulates each branch's gradient back into the shared block exactly. Training and inference run the same differentiable forward pass.
Sliding-window layers use FlashAttention 4's varlen kernel: one call for the whole batch, with separate query and key boundaries, and each branch sees the last 1,023 tokens of the shared block. On a 30k-token branch layout on B300, forward plus backward drops from 14.7 ms to 8.5 ms.
Global layers have head dim 512, beyond FlashAttention's 256. We use cuDNN's new attention kernels, precompiled per length bucket, with front padding to keep gradients exact. On 16k-token batches, forward plus backward is about 4× faster than memory-efficient attention.
Numerics. Outputs and gradients match a reference implementation within bf16 rounding.
Decision models overfit easily
In decision tasks, the question types, option formats and answer distributions are fairly fixed. Having seen enough of them in training, a model can score high on public benchmarks through familiarity with these formats, and still answer poorly in a new domain.
The Decision Index board shows this (private-test scores are converted to the scale of the public score). mammoth-1, tied for the highest public score at 75.3, scores only 66.0 on the private tests of the same skills, 72.1 on the private tests from new domains, and 68.4 Full. From the public tests to the private same-skill tests, the ten models with the highest public scores drop 6.8 points on average. Karpo Decision 31B drops 4.3 points, the least of the ten; it scores 78.6 on the private new-domain tests and 73.9 Full.
The same thing has happened in our own training. Below are four checkpoints along one training run; the public scores are from our internal batched runner:
Stage | DI public (37 benchmarks) | Internal evaluation set |
|---|---|---|
Early | 73.15 | 72.1% |
Middle | 73.94 | 71.3% |
Late | 74.25 | 69.8% |
Final | 75.09 | 70.7% |
The public score rose the whole way, while unseen-domain accuracy fell by up to 2.3 points and the model grew more confident overall. The Decision Index now ranks models only on private tests: 60% same-skill tasks and 40% tasks from new domains. Chasing the public score alone costs rank.
Training with RSI
How a round works
Karpo Decision 31B is trained with RSI (recursive self-improvement), a loop that runs in rounds: each round trains a batch of candidates, scores each one, fuses a selection into one set of weights, and compares the result with the current model, replacing it only on a win.
Every decision in this loop is made, as far as possible, on questions the model did not see in training:
Scoring. Every candidate is scored on two kinds of questions. One is the public suite, which measures the same skills; the other is the unseen-domain evaluation set (see its section below), none of which was used in training. The two are weighted 6:4, the same ratio as in the Full score.
Fusion. Each round fuses several checkpoints into one set of weights; see Model Fusion below.
Promotion. The merged model replaces the current model only if it beats it on the weighted score.
In the round that produced this release, the candidates came from two training regimes (full fine-tuning and LoRA) and several training stages. The fusion of five checkpoints won on the weighted score, and Karpo Decision 31B is the first model promoted this way.
Since the release, we have added two rules to the loop:
The acceptance rule is fixed before a run starts.
When two recipes are compared, each arm runs two random seeds where possible; a gap within run-to-run noise counts as a tie, and nothing is replaced.
No later candidate has passed this bar, including further training on the released weights, adjusted fusion weights and changed training recipes.
Model Fusion: merging checkpoints from different stages and training regimes
Model Fusion is the RSI loop's fusion step: a model soup [7], a uniform average of the weights of five checkpoints. Three come from the full fine-tuning run of the earlier table and two from a separate LoRA [8] run, with the low-rank updates folded into the weights before averaging. Early checkpoints generalise better and late ones are more specialised; averaging across stages keeps strengths of both, as reported for an early and a late checkpoint of one reasoning fine-tune [9]. The LoRA run is there for diversity: LoRA is reported to learn less and forget less than full fine-tuning [10], and checkpoints from separate runs differ more than checkpoints along one run, which helps weight averaging [11]. The merged model is still an ordinary model: its architecture, parameter count and inference cost are unchanged.
Model | DI public (37 benchmarks) | Internal evaluation set | Inference cost |
|---|---|---|---|
Best single checkpoint | 75.09 | 70.7% | 1× |
Model Fusion (five checkpoints) | 75.07 | 72.3% | 1× |
Public scores in this section are from our internal batched runner. The Model Fusion row is the released model; the official kit gives it 75.0.
Same public score, +1.6 points on unseen domains.
On the public suite, it matches the final checkpoint and is 0.8–2.5 points above the other four that went into it.
Among the seven candidates we compared, none beats the merged model on knowledge accuracy or safety.
Calibration: probabilities that mean what they say
A decision model has to pick the right answer, and it also has to know how sure it is. With calibrated probabilities, an agent can act automatically above a threshold and hand off to a human below it.
RLCD. Karpo Decision 31B is trained with RLCD (Reinforcement Learning for Calibrated Decisions [12]): the reward is a strictly proper scoring rule on the model's full probability distribution, so the highest reward comes only from reporting true confidence. Where labels disagree, the target keeps the distribution rather than collapsing it to one answer. A right-or-wrong reward is not enough: in LLMs that reason before they output a decision, reinforcement learning with such a reward raises accuracy but leaves them extremely overconfident; an added loss on the decision probability itself removes that extreme overconfidence [13].
Per-type temperatures. After training, each question type gets its own temperature, fitted on a held-out set.
Base-rate shift. Calibration holds for the base rates it was measured on. When base rates in deployment move, confidence drifts and a fixed threshold stops meaning the same thing [14]. The release applies no base-rate correction. A label-free option for callers is EM prior correction, with the model's average prediction on reference data standing in for its prior [14]; we have not yet measured it on Karpo Decision 31B.
Expected calibration error on the JevBench public items is 0.023 in our run. On the JevBench board, Karpo Decision 31B scores 94.1 on JevBench's Calibration axis, second among 261 ranked models; this is a separate measure where higher is better (see Detailed Results).
The unseen-domain evaluation set: 10 categories, 283 subcategories
We build and maintain this evaluation set ourselves. It is organised by domain into 10 categories and 283 subcategories, and none of it was used in training. Only a part of it was used to choose the released model, and the "internal evaluation set" scores in this post's tables were measured on that part; the rest was built after the release and took no part in the selection, so it serves as a blind test.
Over the whole set, Karpo Decision 31B's mean accuracy is 65.5% and Jev 1.14's is 60.8%, and Karpo Decision 31B is ahead in all 10 categories. Broken down, it is 72.4% against 67.6% on the part used for selection and 61.7% against 57.2% on the part that took no part in it; the lead largely holds on the blind test.
Lessons learned
Overfitting of a decision model during training has three known sources:
Models that tie on familiar tests can behave very differently on new domains [4].
The way a model is selected also changes the result [5].
Runs that differ only in their random seed can also spread widely out of distribution [6].
We ran into all three.
A gap between single runs is often just the random seed. The same recipe with a different random seed differed by 1.4 points in accuracy on the internal evaluation set (72.05% against 70.62%). When we compare two recipes now, each arm runs two random seeds where possible, and a gap no larger than this counts as a tie.
Scores on an evaluation set used for selection run high. We chose the released model with a part of the internal evaluation set, so the scores on that part carry a selection bias. After the release we built a new batch of datasets that took no part in the selection, as a blind test (see the previous section).
A change that only raises the public score should first be suspected of overfitting. One example is reading every request a second time with its options reversed and averaging the two results. On earlier checkpoints this raised the public score by 0.07–0.16 and changed nothing on unseen domains, while median latency rose from 94 ms to 162 ms. The released model reads once.
More elaborate merging methods were no better. TIES [16], DARE [17] and layer-wise scaling of the fine-tuning update [18] did not clearly beat the uniform average. An evolutionary search over per-layer-group merge coefficients [15] raised the score on the few hundred questions the search used but not on held-out questions: the search itself had overfitted.
The gain from tuning fusion weights is within noise. We tried six combinations: different weights for the five members, or members replaced or added. On unseen domains they score 72.4–72.7%, against 72.3% for the uniform average. Four of them were run on the public suite and score 74.80–74.99, against 75.07 for the uniform average. Every gap is within seed-to-seed noise, so we treat them as ties and keep the uniform average.
No further training after fusion. Further training on the released weights gave a lower public score than the release in each of several attempts.
Serving Optimization
Decision requests are latency-sensitive. Across all public Decision Index requests (one RTX PRO 6000, bf16, one request at a time), median latency is 95 ms and p80 245 ms. About 80% of requests are under 1k tokens, with a median of 83 ms. Latency grows with length: on the sample below, requests over 8k tokens, the class a 10k-token page falls in, average 6.1 s in bf16 and 3.6 s on the fastest path. The model fits on one GPU and speaks the System One API.
One forward pass, no decoding. Every question's probabilities come out of one forward pass; output tokens are zero. The last global attention layer runs only at the readout markers, not over the whole sequence.
Shared KV across questions. The shared block's keys and values are computed once per request and reused by every branch; each question adds only its own branch. The more questions per request, the cheaper each one gets: for 1k–4k-token requests, one question takes 371 ms at the median and 16–63 questions take 653 ms.
Variable-length packing, no padding. Sequences are packed end to end into one forward pass with varlen attention, invisible to each other. 50 of 60 layers only look at the last 1,024 tokens, so attention cost and KV memory on long requests come mostly from the 10 global layers. On B200/B300, the sliding-window layers can switch to FlashAttention 4's varlen kernel and the global layers to cuDNN's new attention kernels.
FP8 and CUDA graphs. The model's linear projections run in FP8, and single-question requests replay CUDA graphs captured at startup, removing per-kernel launch overhead. Peak memory drops from 80 GB to about 60 GB; the full public suite scores 75.0, the same as in bf16.
A Triton kernel for the global layers. Gemma 4's global layers use head dim 512, which no fused attention kernel on RTX PRO 6000 (sm_120) supports, so they fell back to memory-efficient attention and dominated long requests (half of GPU time at 32k tokens). We wrote a Triton flash-attention forward for them: global attention is 1.8–2× faster, and a 32k-token request drops from 10.4 s to 8.0 s. Answers move only by bf16 noise (99.8% same top answer).
On a 2,000-request stratified sample of the public suite, one RTX PRO 6000, one request at a time:
Path | Median | Mean | p80 | p95 | Throughput |
|---|---|---|---|---|---|
bf16 | 94 ms | 278 ms | 239 ms | 603 ms | 4,577 tok/s |
FP8 + CUDA graphs | 74 ms | 201 ms | 152 ms | 382 ms | 6,344 tok/s |
FP8 + CUDA graphs + Triton global attention | 73 ms | 179 ms | 148 ms | 362 ms | 7,100 tok/s |
Three ways of running the model (inference paths), each adding to the one before
Time to answer one request (latency)
milliseconds; lower is better
- bf16
- FP8 + CUDA graph
- FP8 + CUDA graph + Triton global attention
- Medianhalf are faster
- 947473
- Mean
- 278201179
- p8080% are faster
- 239152148
- p9595% are faster
- 603382362
Tokens per second (throughput)
higher is better
- bf16
- FP8 + CUDA graph
- FP8 + CUDA graph + Triton global attention
- Throughputtok/s
- 4,5776,3447,100
Many questions in one request
median milliseconds, all requests of 1k to 4k tokens, bf16
- 1 question
- 371
- 16 to 63 questions
- 653in total
- Requests over 8k tokens gain the most: mean latency falls from 6.1 s in bf16 to 3.6 s on the fastest path.
Half of all requests are answered in under a tenth of a second on one GPU.
Detailed Results
Headline numbers
Measure | Karpo Decision 31B | Reference | Notes |
|---|---|---|---|
Decision Index 0.4.1 Full | 73.9 | Rank #1 on the board, 300 models | Scored by the Decision Index maintainers on private tests; November 12 board |
Decision Index 0.3 public (37 benchmarks) | 75.0 | Board best 75.3 | Official kit, one RTX PRO 6000 |
Knowledge (AA-Omniscience public) | 56.0% | KnowLine 53.2%, Sieve 48.6% | Highest among models that answer every question |
Safety (WildGuardTest) | 93.4% | Five other models: 90.9%–92.9% | Mean of three yes/no judgements |
JevBench Capability, public and sealed items | 87.2 | Rank #3 on the board, 243 Jev-class open-weight models; Jev 1.14 75.0 | Scored by the JevBench maintainers; November 12 board |
Internal evaluation set | 72.3% | KnowLine 71.5%, Sieve 70.4%, GLiDE 69.1% | Same items, same scoring |
Latency, median / p80 | 95 / 245 ms | Decision Index board limit 1,000 ms | All public-suite requests; one RTX PRO 6000, one request at a time |
Decision Index results
On the November 12 board, the three models that follow Karpo Decision 31B by Full score are GLiDE 27B, REAX 1.3 and Logits Sigma-1 Duo. Every number in the table below is from the board:
Karpo Decision 31B | GLiDE 27B | REAX 1.3 | Logits Sigma-1 Duo | |
|---|---|---|---|---|
DI Full (official) | 73.9 | 73.4 | 72.6 | 72.3 |
DI public (37 benchmarks) | 75.0 | 72.5 | 73.7 | 72.9 |
Median latency | 94 ms | 122 ms | 246 ms | 368 ms |
REAX 1.3 and Logits Sigma-1 Duo are each an ensemble of two models, 57B parameters in total; the board labels such entries "inference technique". Karpo Decision 31B's median latency on the board is 94 ms; the 95 ms elsewhere in this post is our own measurement over all public-suite requests.
On the Decision Index board, Jev 1.14 scores 60.2 Full and 60.7 on the public suite.
The last three rows of the table below were measured by us on the same items for every model. KnowLine 31B Gen1, GLiDE 1.2 and Sieve 31B v3 were the Full top three on the November 4 board, and we ran them with their published weights and serving code; Gemini Decision 1 is not on the Decision Index board, and we measured it through its hosted API. We have not yet measured the top three of the November 12 board on these three rows.
Karpo Decision 31B | KnowLine 31B Gen1 | GLiDE 1.2 | Sieve 31B v3 | Gemini Decision 1 | |
|---|---|---|---|---|---|
DI public (37 benchmarks) | 75.0 | 70.1 | 72.0 | 71.7 | 60.9§ |
Internal evaluation set | 72.3% | 71.5% | 69.1% | 70.4% | 70.4% |
Knowledge accuracy | 56.0% | 53.2% | 41.8%† | 48.6% | 28.9%† |
WildGuardTest | 93.4% | 90.9% | 92.7% | 92.8% | 86.8%‡ |
† GLiDE 1.2 answers "I don't know" on about 48% of these items and Gemini Decision 1 on about 64%, which lowers their accuracy; on the items they answer, both are right about 80% of the time.
‡ Google's input filter blocked 38 of the 1,725 requests, which count as wrong; on the rest, Gemini Decision 1's accuracy is 88.7%.
§ Gemini Decision 1 is not on the Decision Index board; its public score is from our run of the official kit through its hosted API. Google's input filter blocked 69 requests, which count as wrong. Two forms its API does not accept (an empty-string state, and question keys that do not conform to its rules; 4,798 requests in all) were sent in an equivalent form. The other three models' public scores are from the November 4 board.
- Karpo Decision 31B
- the models it is compared with
From the official board, 12 November
Decision Index Full
official score, private tests
- Karpo Decision 31B
- 73.9
- GLiDE 27B
- 73.4
- REAX 1.3
- 72.6
- Logits Sigma-1 Duo
- 72.3
Decision Index public suite
score, 37 benchmarks
- Karpo Decision 31B
- 75.0
- GLiDE 27B
- 72.5
- REAX 1.3
- 73.7
- Logits Sigma-1 Duo
- 72.9
Median latency
milliseconds; lower is better
- Karpo Decision 31B
- 94
- GLiDE 27B
- 122
- REAX 1.3
- 246
- Logits Sigma-1 Duo
- 368
Measured by us on the same questions for every model
Internal evaluation set
unseen domains; accuracy, %
- Karpo Decision 31B
- 72.3
- KnowLine 31B Gen1
- 71.5
- GLiDE 1.2
- 69.1
- Sieve 31B v3
- 70.4
- Gemini Decision 1
- 70.4
Knowledge accuracy
AA-Omniscience public questions, %
- Karpo Decision 31B
- 56.0
- KnowLine 31B Gen1
- 53.2
- GLiDE 1.2
- 41.8†
- Sieve 31B v3
- 48.6
- Gemini Decision 1
- 28.9†
Safety judgement
WildGuardTest, %
- Karpo Decision 31B
- 93.4
- KnowLine 31B Gen1
- 90.9
- GLiDE 1.2
- 92.7
- Sieve 31B v3
- 92.8
- Gemini Decision 1
- 86.8‡
- † GLiDE 1.2 chose "don't know" on about 48% of the questions and Gemini Decision 1 on about 64%, which lowers their accuracy; on the questions they did answer, both are at about 80%.
- ‡ Google's input filter blocked 38 of the 1,725 requests, which are counted as wrong; on the remaining requests Gemini Decision 1's accuracy is 88.7%.
Within a row, every model is scored on the same questions in the same way.
JevBench results
At our request, the JevBench maintainers ran the released weights and server code, unchanged, on November 10. The run used the October 27 draw of public and sealed items, one request at a time on one RTX PRO 6000, with no network access. On November 12, JevBench v1.8.10 equated that draw to the board's scale with four anchor systems, lowering Intelligence by 2.9 points and Calibration by 1.9, and the result was ranked on the board.
Capability, the mean of Intelligence and Calibration, is 87.2, ranked #3 among 243 Jev-class open-weight models, those within twice Jev's cost and median latency. The Composite Score, the harmonic mean of all four axes, is 77.4, official rank #7 of 261 ranked models.
Axis | Score |
|---|---|
Intelligence, chance-corrected correctness | 80.2 |
Calibration, stated confidence against outcomes | 94.1 |
Speed, from latency per decision | 88.3 |
Cost, from estimated cost per 1,000 decisions | 57.9 |
Sealed and public. Intelligence is 85.67 on the draw's sealed items and 80.45 on its public ones (raw scores from the maintainers' run, before equating), so the method's penalty for a public score well above the sealed one does not apply.
Latency and cost. On the board, measured median latency is 82 ms, timed on the GPU with no network. For a self-hosted model the method scores Speed on twice that plus 150 ms, which is 314 ms. Cost is the lowest axis: the method estimates $0.0252 per 1,000 decisions from GPU time at its assumed throughput for a dense 31B model, not from a measurement.
On the JevBench board, from November 12
JevBench scores on four axes
0 to 100, higher is better
- Intelligence
- 80.2
- Calibration
- 94.1
- Speed
- 88.3
- Cost
- 57.9
The maintainers' run, raw scores before equating
Intelligence by item set
0 to 100; sealed items are never published
- Sealed items
- 85.67
- Public items
- 80.45
Capability 87.2, #3 of 243 Jev-class open-weight models; Composite 77.4, official #7 of 261 ranked models.
On the JevBench public items, in our own run, Karpo Decision 31B answers 213 of 231 correctly; Jev 1.13's published result is 200. Both are near-perfect on the easy and original tiers; the whole gap is in the 111 hard items.
Acknowledgements
Our prompt layout and attention rule follow the public work of the decision-model community: the question-list-first layout of FlexDM and jev-brain, and the shared-block and branch rule with bidirectional global layers of Sieve-31B-v3. Thanks to the Decision Index maintainers for an open, reproducible benchmark, and to the JevBench maintainers for their run.
Try it
Weights: MachinePulseAI/karpo-decision-31B
Full Decision Index run: MachinePulseAI/karpo-decision-31B-decision-index
References
Gemma Team. Gemma 4 Technical Report. arXiv:2607.02770, 2026.
Geuter, J., Cheng, A., Alvarez-Melis, D. Releasing the Inner Jev: A Simple Recipe for Open Decision Models. arXiv:2610.17205, 2026.
Zheng, C. et al. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024.
D'Amour, A. et al. Underspecification Presents Challenges for Credibility in Modern Machine Learning. JMLR 23(226), 1–61, 2022.
Gulrajani, I., Lopez-Paz, D. In Search of Lost Domain Generalization. ICLR 2021.
McCoy, R. T., Min, J., Linzen, T. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. BlackboxNLP 2020.
Wortsman, M. et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022.
Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
Dang, X. et al. Weight Ensembling Improves Reasoning in Language Models. COLM 2025.
Biderman, D. et al. LoRA Learns Less and Forgets Less. TMLR 2024.
Ramé, A. et al. Diverse Weight Averaging for Out-of-Distribution Generalization. NeurIPS 2022.
Almeida, D. Introducing System One Models & Jev. TypeSafe AI blog, September 2026 (origin of the term RLCD).
Yaldiz, D. N. et al. Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning. arXiv:2601.13284, 2026.
Iqbal, A. R., Nasim, M. F., Saleem, A. Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors. arXiv:2610.26325, 2026.
Akiba, T. et al. Evolutionary Optimization of Model Merging Recipes. Nature Machine Intelligence 7, 195–204, 2025.
Yadav, P. et al. TIES-Merging: Resolving Interference When Merging Models. NeurIPS 2023.
Yu, L. et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML 2024.
Wang, K. et al. LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging. ICLR 2025.