BlogResearch19 min read

Karpo Decision 31B: A Fast, Accurate Multimodal Decision Model

Today, we are releasing Karpo Decision 31B, a decision model built on Gemma 4 31B: given a state and typed questions, it answers them all in one inference. Decision Index Full 73.9, first on the board; JevBench Capability 87.2, third among 243 Jev-class open-weight models; median latency 95 ms.

Karpo Research

On this page
  1. 1Fast and accurate, with probabilities you can trust
  2. 2Model design
  3. 3Decision models overfit easily
  4. 4Training with RSI
  5. 5The unseen-domain evaluation set: 10 categories, 283 subcategories
  6. 6Lessons learned
  7. 7Serving Optimization
  8. 8Detailed Results
  9. 9Acknowledgements
  10. Try it
  11. References

Today, we are releasing Karpo Decision 31B, a natively multimodal decision model built on Gemma 4 31B. It reaches state-of-the-art results on several authoritative benchmarks and has a clear advantage in intelligence, cost and speed over Jev 1.14 and other open and closed models. Given a state (a web page, a ticket, a document) and a set of typed questions, Karpo Decision 31B answers many questions in one forward pass, which suits high-frequency, latency-sensitive scenarios such as judgement, ticket handling, safety review, and retrieval and ranking. The weights are open on Hugging Face, and you are welcome to have a try.

  • 73.9 Decision Index Full score, ranked #1 on the board among 300 models.

  • 87.2 JevBench Capability Score, ranked #3 among 243 Jev-class open-weight models on the board.

  • 95 ms median latency, one request at a time on a single RTX PRO 6000, across all 140,217 public requests.

The top five of the Decision Index by Full score, and Jev

The Decision Index leaderboard (Full score): Karpo Decision 31B ranks first among 300 models.

Besides the 31B model, we have also open-sourced five more model sizes, 0.8B, 2B, 4B, 9B and 12B, forming the Karpo Decision model family. The chart below shows each size's score on DI public (37 benchmarks), with the other board models of the same size class in gray.

KARPO DECISION FAMILYScore vs. SizeDecision Index 0.3 public suite, board of November 12, 2026 · parameters as servedKarpo Decision, 0.8B to 31BOther board models in the same size class80706050401B2B4B8B16B32Bparameters · log scaleJev 1.14 · 60.7closed model,size undisclosed58.8PKU-KCL-Meteor-0.8B54.7Valen 0.8B54.7Adjudex-0.8B v554.3GLiDE 0.8B53.8StartLux 0.8B48.8OneJev1.6-0.8B47.8Crivo-1B47.8Saif 0.8B Lightning44.5KnowLine 0.8B Gen142.8crystal-0.8b63.3joey-161.9Valen 2B61.0PKU-KCL-Meteor-2B60.6GLiDE 2B59.2StartLux 2B53.5Sieve 2B v352.2KnowLine 2B Gen151.3Aurelis-2B51.1d1-2B49.0d1-Omni-2B69.0Valen 4B67.8StartLux 4B67.0Senova-4B v466.8KnowLine 4B Gen5.565.0Cerno 4B v1.264.8Jeffan-4B-201063.4Perplexity Flash v1.372.2elephant-170.6Valen 9B70.5Jeffan-9B-121170.0StartLux 9B69.3Adjudex-9B v568.8GLiDE 9B66.8Cerno 9B v1.265.7KnowLine 9B Gen165.3OneJev1.6-9B55.2AJev lora751.4muria50.7Winnow-12B49.7decisio 0.4.048.3Hopper 12B75.3mammoth-174.7Valen 27B73.4Jeffan-27B-121173.1Jeffan-31b-171072.9StartLux 27B72.7Nimble v4.1 MaxKarpo Decision 0.8B53.4Karpo Decision 2B58.9Karpo Decision 4B67.6Karpo Decision 9B69.9Karpo Decision 12B69.8Karpo Decision 31B75.0
Karpo Decision 0.8B–12B: our batched runner; 31B: official kit run. Gray: board entries with declared parameter counts, every one above Karpo Decision in its size class and the top five below it; size classes 0.5–1.3B, 1.3–3.2B, 3.2–6.5B, 6.5–10.5B, 10.5–20B, 20–40B.

Fast and accurate, with probabilities you can trust

A personal agent works out what a user means from a vague request, makes recommendations, searches and decides for the user, and operates the browser on the user's behalf. Every step starts with a judgement. When recognising intent, it has to judge what a vague request is really asking for and whether to act or to ask first; when recommending, whether a product meets the user's requirements; when searching, how relevant a result is to what the user needs; when deciding, whether the task is done and whether a tool should be called; when operating the browser, which element on the page to click. The speed and accuracy of these judgements directly determine how fast the agent responds and whether the task gets done. Karpo Decision 31B is built for this kind of judgement. It is fast and accurate, and it takes multimodal input:

  • #1 on the board. Decision Index Full score 73.9, ranked #1 among the 300 models on the board; Jev 1.14's Full score is 60.2.

  • Fast inference. Served on one RTX PRO 6000, it has a median latency of 95 ms on the Decision Index suite; Jev 1.14's median latency on the board is 196 ms.

  • Many questions per request. A request can ask any number of questions, and each is answered independently; the more questions, the less time each one takes.

  • Multimodal input. Multimodal input is supported natively: a request can carry images directly, which the vision encoder reads and the model judges together with the text.

  • Long context. A single sequence is at most 32k tokens by default, and a context window of up to 256k is supported.

  • Valid answers, probabilities you can trust. The answer is always one of the options, and every answer comes with a calibrated probability. Our ECE (expected calibration error) on the JevBench public items is 0.023.

  • Open and deployable. The weights and the inference code are public, one RTX PRO 6000 is enough to deploy it, and the interface is compatible with the System One API.

BenchmarkKarpo Decision 31BOpen weightsGemini Decision 1Decisions APITypeSafe Jev 1.14GPT-6 LunaOpenAI Decisions API
Key
benchmarks
JevBenchPublic split92.2%92.2%90.0%90.9%
Amazon ESCIMacro-F164.7%58.7%56.7%54.7%
New Yorker CaptionText-only90.5%79.7%72.0%68.4%
Mind2Web Multimodal90.0%86.2%86.8%81.8%
PubMedQA79.2%79.8%78.2%74.4%
Global-MMLU-LiteNon-English85.2%86.0%85.0%79.6%
When2Call90.4%79.9%82.0%70.4%
MMAU—71.8%——
Video-MMENo subtitles—69.1%——
CalibrationTop-1 ECELower is better0.0230.02500.04940.0721
Gemini Decision 1, TypeSafe Jev 1.14 and GPT-6 Luna figures are quoted from Google's Gemini Decisions API blog post (November 11, 2026): https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/
Karpo Decision 31B figures are our own runs; ESCI, New Yorker and When2Call are independently reproduced by the Jev Decision Index: huggingface.co/spaces/multimodalart/jev-decision-index

On the three kinds of judgement a personal agent makes most often, Karpo Decision 31B is ahead of Gemini Decision 1:

  • Agent action judgements: When2Call tests whether to call a tool, ask a follow-up question or hold back; Karpo Decision 31B scores 90.4% and Gemini Decision 1 79.9%. Mind2Web tests which element on a web page to click next: 90.0% against 86.2%. These two decide whether an agent places the wrong order or clicks the wrong place.

  • Reading between the lines: New Yorker Caption asks which of five captions belongs to the cartoon; 90.5% against 79.7%. What users say to an agent is rarely a complete instruction, and what they want has to be judged from tone and context.

  • Fine-grained business classification: Amazon ESCI asks whether a product is an exact match for a search query, a substitute, a complement or irrelevant; macro-F1 is 64.7% against 58.7%. When an agent picks things for a user, the hard part is telling substitutes from complements.

The pagethe stateLog inSign upCancelThe questionwith its options“Which button shouldthe agent click?”(A)Log in(B)Sign up(C)CancelA generative model (LLM)generates tokens one by one; the answer is parsed out of the textone forward pass per tokenmodel…ClickLoginthinkinganswerparse the text“Login”is that (A) Log in?Slowone forward passper tokenExpensivetens to hundredsof tokensUnconstrainedthe answer may notbe one of the optionsNo usable confidencenothing to threshold onA decision modelreturns a probability for every option; it generates no textprobability (illustrative)modelone forward pass(A)Log in0.82(B)Sign up0.13(C)Cancel0.051 forward passthe model runs once0 generated tokensnothing is decodedAlways validthe answer is alwaysone of the optionsProbabilisticevery option has a probabilityto rank or thresholdA generative model writes an answer; a decision model picks among the options and says how sure it is.
One decision, two ways. A generative model writes its answer token by token, and the answer has to be parsed out of the text; a decision model returns a probability for every option from one forward pass. Probabilities are illustrative.

Karpo Decision 31B answers three question types:

Python
from huggingface_hub import snapshot_download
from karpo_decision.decider import Decider

decider = Decider.from_pretrained(snapshot_download("MachinePulseAI/karpo-decision-31B"), device="cuda")
response = decider.decide({
    "state": "Order 1182: the customer asks to return a blender bought 45 days ago. Policy: returns within 30 days.",
    "questions": {
        "eligible": {
            "type": "noul",
            "instructions": "Is the return eligible under the policy?"
        },
        "action": {
            "type": "choice", "instructions": "What should the agent do?",
            "criteria": {
                "refund": "Issue a full refund", "decline": "Politely decline the return",
                "escalate": "Escalate to a supervisor"
            }
        },
        "urgency": {
            "type": "score", "instructions": "How urgent is this ticket?",
            "criteria": ["Low", "Medium", "High"]
        },
    },
})
print(response["answers"])

Model design

Readout head: option probabilities read out directly

Each question's options are labelled with option codes (A, B, … Z, AA, …), followed by a readout marker. The readout head scores 255 option codes at the marker and normalises over that question's own options only. Every question in the request is answered in the same forward pass, with no decoding.

The tokens of one questionthe question, its options, then the position where the answer is readWhich button next?(A) Log in(B) Sign up(C) CancelAnswer: (readout markernothing is generated after itthe questionits options, labelled with option codesh255 option codeslabels for the options: A to Z, AA, AB, …this question uses A, B, CEach code is a single token,verified both after “Answer: (”and after a newline plus “(”.The readout heada score for each option code, then a probabilityOne score peroption codez=Whhthe hidden state at the readout markerWone row per option code;each row starts from the base model’s LM-headrow for that option code’s tokenThe scores turnedinto probabilitiesp(k)=exp(zk/T)Σjexp(zj/T)kone of this question’s options (here A, B, C)jruns over this question’s own options onlyTa temperature per question typeThe outputa probability per optionp(k)(A) Log in0.82(B) Sign up0.13(C) Cancel0.0501Illustrative values;the three sum to 1.The model never writes the answer; it reads one score per option label and turns the scores into probabilities.
At the readout marker the head gives each of the 255 option codes a score, then turns the scores of the question's own options into probabilities (a softmax with a temperature per question type). The base model's logit softcap, which the head keeps, is not drawn.

Three implementation details:

  • Option codes are verified single tokens. We pick option codes from the tokenizer and check that each one is the same single token in each context it appears in. The label the model sees on an option is the same token it is read out from.

  • The head starts from the LM head. Each row of the readout head is initialised from the base model's LM-head row for that option code's token, with the base model's logit softcap kept. The model recognises the labels from step zero. Geuter et al. likewise initialise a decision head from the LM-head rows of the answer letters [2].

  • Option order is randomised in training. LLMs prefer particular option labels when they answer multiple-choice questions [3], so in training, wherever the order of a choice question's options carries no meaning, the options are shuffled at random and the question is trained on several times.

Joint multi-question: every question answered in one forward pass

A request is one state plus any number of questions, encoded as a single sequence: first a list of all the questions, then the state, then each question's options and readout marker.

  • The page is read once. In a browser agent a page is often over 10k tokens, and a question is a few dozen. Asking each question separately would re-read the page every time.

  • The model knows what it will be asked before it reads. The question list comes first, so the state is read with the questions in view.

  • No silent truncation. Requests over 32k tokens are split into segments, each with the full shared content and the full question list. If a single question cannot fit, the request fails with an explicit error.

Branch attention: shared computation, independent answers

The sequence has two parts: a shared block (system turn, question list, state) and one branch per question (the question, its options, its readout marker). A branch sees the shared block and itself, never another branch; the shared block sees only itself. Each branch's position ids restart at the end of the shared block, so every question sees the state exactly as it would if it were asked alone. In the global layers, attention is bidirectional within what a token may see; the sliding-window layers stay causal.

One request is one sequencea shared block, followed by one branch per questionevery branch starts counting positionswhere the shared block ends (position ids restart)system turnquestion listthe page (the state)often 10k+ tokensquestion 1, its options,its readout markerquestion 2, its options,its readout markerShared blockdrawn below as 6 tokensBranch 1drawn as 2 tokensBranch 2drawn as 2 tokensThe 10 global attention layersa coloured cell: the row’s token can read the column’s tokenthe tokens it can readthe token that readsThe shared blockreads itself, in bothdirections, and neverreads a branchA branchreads the shared blockand itself, in bothdirections, and neverreads another branchThe 50 sliding-window layersthe same matrix; a window of 1,024 tokens, drawn here as 4the tokens it can readthe token that readsThe shared blockreads only earliertokens within thewindow (causal)A branchreads the last 1,023tokens of the sharedblock and its ownearlier tokenscan read a token ofthe shared blockbranch 1branch 2cannot readThe page is read once, however many questions there are; no question sees another question’s options.
The shared block is computed once; a branch reads the shared block and itself, never another branch. Global layers read in both directions within what a token may see; sliding-window layers read only earlier tokens (causally). Sizes are schematic.

Asked together this way, the questions share one computation of the state, yet each question, just as when asked alone, sees no other question's options. Retrieval and ranking need this most, because both judge many candidates one by one.

Most of the engineering went into the training-time implementation:

  • Planned in advance. Each packed batch is planned once: every branch's queries, the shared-block keys and values it may read, and its own keys and values. All layers and activation recomputation share that plan.

  • Exact gradients. When branches read the shared block's keys and values, the backward pass accumulates each branch's gradient back into the shared block exactly. Training and inference run the same differentiable forward pass.

  • Sliding-window layers use FlashAttention 4's varlen kernel: one call for the whole batch, with separate query and key boundaries, and each branch sees the last 1,023 tokens of the shared block. On a 30k-token branch layout on B300, forward plus backward drops from 14.7 ms to 8.5 ms.

  • Global layers have head dim 512, beyond FlashAttention's 256. We use cuDNN's new attention kernels, precompiled per length bucket, with front padding to keep gradients exact. On 16k-token batches, forward plus backward is about 4× faster than memory-efficient attention.

  • Numerics. Outputs and gradients match a reference implementation within bf16 rounding.

Decision models overfit easily

In decision tasks, the question types, option formats and answer distributions are fairly fixed. Having seen enough of them in training, a model can score high on public benchmarks through familiarity with these formats, and still answer poorly in a new domain.

The Decision Index board shows this (private-test scores are converted to the scale of the public score). mammoth-1, tied for the highest public score at 75.3, scores only 66.0 on the private tests of the same skills, 72.1 on the private tests from new domains, and 68.4 Full. From the public tests to the private same-skill tests, the ten models with the highest public scores drop 6.8 points on average. Karpo Decision 31B drops 4.3 points, the least of the ten; it scores 78.6 on the private new-domain tests and 73.9 Full.

The same thing has happened in our own training. Below are four checkpoints along one training run; the public scores are from our internal batched runner:

Stage

DI public (37 benchmarks)

Internal evaluation set

Early

73.15

72.1%

Middle

73.94

71.3%

Late

74.25

69.8%

Final

75.09

70.7%

The public score rose the whole way, while unseen-domain accuracy fell by up to 2.3 points and the model grew more confident overall. The Decision Index now ranks models only on private tests: 60% same-skill tasks and 40% tasks from new domains. Chasing the public score alone costs rank.

Training with RSI

How a round works

Karpo Decision 31B is trained with RSI (recursive self-improvement), a loop that runs in rounds: each round trains a batch of candidates, scores each one, fuses a selection into one set of weights, and compares the result with the current model, replacing it only on a win.

Every decision in this loop is made, as far as possible, on questions the model did not see in training:

  • Scoring. Every candidate is scored on two kinds of questions. One is the public suite, which measures the same skills; the other is the unseen-domain evaluation set (see its section below), none of which was used in training. The two are weighted 6:4, the same ratio as in the Full score.

  • Fusion. Each round fuses several checkpoints into one set of weights; see Model Fusion below.

  • Promotion. The merged model replaces the current model only if it beats it on the weighted score.

In the round that produced this release, the candidates came from two training regimes (full fine-tuning and LoRA) and several training stages. The fusion of five checkpoints won on the weighted score, and Karpo Decision 31B is the first model promoted this way.

Since the release, we have added two rules to the loop:

  • The acceptance rule is fixed before a run starts.

  • When two recipes are compared, each arm runs two random seeds where possible; a gap within run-to-run noise counts as a tie, and nothing is replaced.

No later candidate has passed this bar, including further training on the released weights, adjusted fusion weights and changed training recipes.

A loop of four steps (RSI)the team runs the loop, round by round; RSI (recursive self-improvement) is the team’s name for itnext round1Train candidatestwo training regimes,full fine-tuning and LoRA,and several training lengths2Score each candidateon the public suite andon unseen domains3Fuse a selectioninto one set of weights:the fused candidate4Compare with the current modelweighting same-skill results (from thepublic suite) 60% and new-domainresults (from unseen domains) 40%,as the Decision Index Full score doesThe two outcomes of step 4the fused candidate is promoted only if it comes out ahead of the current modelAheadpromoted, and the next round has to beat itfused candidatecurrent modelNot aheadthe current model staysfused candidatecurrent modelKarpo Decision 31B is the first model promoted this way; no round since has replaced it.A new model replaces the current one only if it beats it.
One round of RSI: train candidates, score each, fuse a selection, compare with the current model.

Model Fusion: merging checkpoints from different stages and training regimes

Model Fusion is the RSI loop's fusion step: a model soup [7], a uniform average of the weights of five checkpoints. Three come from the full fine-tuning run of the earlier table and two from a separate LoRA [8] run, with the low-rank updates folded into the weights before averaging. Early checkpoints generalise better and late ones are more specialised; averaging across stages keeps strengths of both, as reported for an early and a late checkpoint of one reasoning fine-tune [9]. The LoRA run is there for diversity: LoRA is reported to learn less and forget less than full fine-tuning [10], and checkpoints from separate runs differ more than checkpoints along one run, which helps weight averaging [11]. The merged model is still an ordinary model: its architecture, parameter count and inference cost are unchanged.

Model

DI public (37 benchmarks)

Internal evaluation set

Inference cost

Best single checkpoint

75.09

70.7%

1×

Model Fusion (five checkpoints)

75.07

72.3%

1×

Public scores in this section are from our internal batched runner. The Model Fusion row is the released model; the official kit gives it 75.0.

Model Fusionfive saved copies of the model(checkpoints) become onefull fine-tuning runLoRA runθ1θ2θ3θ4θ5averaged,weight by weightModel Fusionone modelsame architecture, same parameter count,same inference costθfused=θ1+⋯+θ55θ: the weightsof a modelLoRA trains small (low-rank) updatesinstead of all the weights; they are foldedinto the weights before averaging.Public suite against unseen domainsfour checkpoints along the full fine-tuning run and Model Fusion;up and to the right is betteraccuracy on our internal evaluation set, %72717073.073.574.074.575.075.5Decision Index public suite score (37 benchmarks)EarlyMiddleLateFinaltraining orderModel Fusion+1.6 pointson unseen domains,same public scoreThe average matches the public score of the final checkpoint and the unseen-domain accuracy of the early one.
Left: five checkpoints, three from the full fine-tuning run and two from a separate LoRA run, are averaged weight by weight. Right: public score against unseen-domain accuracy for four checkpoints along the full fine-tuning run, three of which are in the average, and for the five-checkpoint average. Public scores are from our internal batched runner; the two LoRA checkpoints are not plotted.
  • Same public score, +1.6 points on unseen domains.

  • On the public suite, it matches the final checkpoint and is 0.8–2.5 points above the other four that went into it.

  • Among the seven candidates we compared, none beats the merged model on knowledge accuracy or safety.

Calibration: probabilities that mean what they say

A decision model has to pick the right answer, and it also has to know how sure it is. With calibrated probabilities, an agent can act automatically above a threshold and hand off to a human below it.

  • RLCD. Karpo Decision 31B is trained with RLCD (Reinforcement Learning for Calibrated Decisions [12]): the reward is a strictly proper scoring rule on the model's full probability distribution, so the highest reward comes only from reporting true confidence. Where labels disagree, the target keeps the distribution rather than collapsing it to one answer. A right-or-wrong reward is not enough: in LLMs that reason before they output a decision, reinforcement learning with such a reward raises accuracy but leaves them extremely overconfident; an added loss on the decision probability itself removes that extreme overconfidence [13].

  • Per-type temperatures. After training, each question type gets its own temperature, fitted on a held-out set.

  • Base-rate shift. Calibration holds for the base rates it was measured on. When base rates in deployment move, confidence drifts and a fixed threshold stops meaning the same thing [14]. The release applies no base-rate correction. A label-free option for callers is EM prior correction, with the model's average prediction on reference data standing in for its prior [14]; we have not yet measured it on Karpo Decision 31B.

Expected calibration error on the JevBench public items is 0.023 in our run. On the JevBench board, Karpo Decision 31B scores 94.1 on JevBench's Calibration axis, second among 261 ranked models; this is a separate measure where higher is better (see Detailed Results).

What calibrated meansprobabilities that mean what they say; the plot is a schematic, not measured datahow often the answer is right, %002020404060608080100100stated confidence, %underconfidentoverconfidentcalibratedon the diagonal the two agreesays 80%, right 8 times in 10Measured for Karpo Decision 31B on the public itemsof the JevBench benchmark (our own run)an average gap of 2.3 percentage points between confidence and accuracy(expected calibration error 0.023)How an agent uses itthe example is a yes/no question; P(yes) is the probability the model gives to yesJevBench’s yes/no ruleon P(yes), from 0 to 1counts as noat 0.20 or belowabstentionthe agent can observe again or hand off to a personcounts as yesat 0.80 or above00.200.801JevBench maintainers’ runshare of yes/no answers92.0%8.0%0%100% of yes/no answersdecisive: counts as no or as yesin between: abstentionA calibrated “80% sure” is right about 8 times in 10,so an agent can act on confident answers and ask when the model is unsure.
Top: for a calibrated model, stated confidence matches how often the answer is right; the plot is a schematic. Bottom: JevBench's yes/no rule, under which an answer with P(yes) between 0.20 and 0.80 counts as an abstention.

The unseen-domain evaluation set: 10 categories, 283 subcategories

We build and maintain this evaluation set ourselves. It is organised by domain into 10 categories and 283 subcategories, and none of it was used in training. Only a part of it was used to choose the released model, and the "internal evaluation set" scores in this post's tables were measured on that part; the rest was built after the release and took no part in the selection, so it serves as a blind test.

Over the whole set, Karpo Decision 31B's mean accuracy is 65.5% and Jev 1.14's is 60.8%, and Karpo Decision 31B is ahead in all 10 categories. Broken down, it is 72.4% against 67.6% on the part used for selection and 61.7% against 57.2% on the part that took no part in it; the lead largely holds on the blind test.

Unseen domains by categoryaccuracy, %, averaged over each category’s evaluation sets; the scale runs from 30 at the centre to 80 at the outer ringKarpo Decision 31BJev 1.14406080Lab Science47.5Clinical Medicine62.0Research & Education71.3Courts & Tribunals51.4Compliance & Examination72.4Safety Regulation71.0Disasters & Accidents66.7Software & Security62.5Games & Sports72.5Media & Language66.3Karpo Decision 31B is ahead of Jev 1.14 in all ten categories of unseen domains.
The 10 categories of the unseen-domain evaluation set: the accuracy of Karpo Decision 31B (blue) and Jev 1.14 (orange) in each category (the mean of the accuracies of the datasets in that category). Under each category name is Karpo Decision 31B's accuracy. The scale runs from 30% at the centre to 80% at the outer ring, 10 points per ring. Both models were measured by us on the same items with the same scoring: Karpo Decision 31B with the released weights, Jev 1.14 through its hosted API (November 12).

Lessons learned

Overfitting of a decision model during training has three known sources:

  • Models that tie on familiar tests can behave very differently on new domains [4].

  • The way a model is selected also changes the result [5].

  • Runs that differ only in their random seed can also spread widely out of distribution [6].

We ran into all three.

  1. A gap between single runs is often just the random seed. The same recipe with a different random seed differed by 1.4 points in accuracy on the internal evaluation set (72.05% against 70.62%). When we compare two recipes now, each arm runs two random seeds where possible, and a gap no larger than this counts as a tie.

  2. Scores on an evaluation set used for selection run high. We chose the released model with a part of the internal evaluation set, so the scores on that part carry a selection bias. After the release we built a new batch of datasets that took no part in the selection, as a blind test (see the previous section).

  3. A change that only raises the public score should first be suspected of overfitting. One example is reading every request a second time with its options reversed and averaging the two results. On earlier checkpoints this raised the public score by 0.07–0.16 and changed nothing on unseen domains, while median latency rose from 94 ms to 162 ms. The released model reads once.

  4. More elaborate merging methods were no better. TIES [16], DARE [17] and layer-wise scaling of the fine-tuning update [18] did not clearly beat the uniform average. An evolutionary search over per-layer-group merge coefficients [15] raised the score on the few hundred questions the search used but not on held-out questions: the search itself had overfitted.

  5. The gain from tuning fusion weights is within noise. We tried six combinations: different weights for the five members, or members replaced or added. On unseen domains they score 72.4–72.7%, against 72.3% for the uniform average. Four of them were run on the public suite and score 74.80–74.99, against 75.07 for the uniform average. Every gap is within seed-to-seed noise, so we treat them as ties and keep the uniform average.

  6. No further training after fusion. Further training on the released weights gave a lower public score than the release in each of several attempts.

Serving Optimization

Decision requests are latency-sensitive. Across all public Decision Index requests (one RTX PRO 6000, bf16, one request at a time), median latency is 95 ms and p80 245 ms. About 80% of requests are under 1k tokens, with a median of 83 ms. Latency grows with length: on the sample below, requests over 8k tokens, the class a 10k-token page falls in, average 6.1 s in bf16 and 3.6 s on the fastest path. The model fits on one GPU and speaks the System One API.

  • One forward pass, no decoding. Every question's probabilities come out of one forward pass; output tokens are zero. The last global attention layer runs only at the readout markers, not over the whole sequence.

  • Shared KV across questions. The shared block's keys and values are computed once per request and reused by every branch; each question adds only its own branch. The more questions per request, the cheaper each one gets: for 1k–4k-token requests, one question takes 371 ms at the median and 16–63 questions take 653 ms.

  • Variable-length packing, no padding. Sequences are packed end to end into one forward pass with varlen attention, invisible to each other. 50 of 60 layers only look at the last 1,024 tokens, so attention cost and KV memory on long requests come mostly from the 10 global layers. On B200/B300, the sliding-window layers can switch to FlashAttention 4's varlen kernel and the global layers to cuDNN's new attention kernels.

  • FP8 and CUDA graphs. The model's linear projections run in FP8, and single-question requests replay CUDA graphs captured at startup, removing per-kernel launch overhead. Peak memory drops from 80 GB to about 60 GB; the full public suite scores 75.0, the same as in bf16.

  • A Triton kernel for the global layers. Gemma 4's global layers use head dim 512, which no fused attention kernel on RTX PRO 6000 (sm_120) supports, so they fell back to memory-efficient attention and dominated long requests (half of GPU time at 32k tokens). We wrote a Triton flash-attention forward for them: global attention is 1.8–2× faster, and a 32k-token request drops from 10.4 s to 8.0 s. Answers move only by bf16 noise (99.8% same top answer).

On a 2,000-request stratified sample of the public suite, one RTX PRO 6000, one request at a time:

Path

Median

Mean

p80

p95

Throughput

bf16

94 ms

278 ms

239 ms

603 ms

4,577 tok/s

FP8 + CUDA graphs

74 ms

201 ms

152 ms

382 ms

6,344 tok/s

FP8 + CUDA graphs + Triton global attention

73 ms

179 ms

148 ms

362 ms

7,100 tok/s

Three ways of running the model (inference paths), each adding to the one before

Time to answer one request (latency)

milliseconds; lower is better

  • bf16
  • FP8 + CUDA graph
  • FP8 + CUDA graph + Triton global attention
Medianhalf are faster
947473
Mean
278201179
p8080% are faster
239152148
p9595% are faster
603382362

Tokens per second (throughput)

higher is better

  • bf16
  • FP8 + CUDA graph
  • FP8 + CUDA graph + Triton global attention
Throughputtok/s
4,5776,3447,100

Many questions in one request

median milliseconds, all requests of 1k to 4k tokens, bf16

1 question
371
16 to 63 questions
653in total
  • Requests over 8k tokens gain the most: mean latency falls from 6.1 s in bf16 to 3.6 s on the fastest path.

Half of all requests are answered in under a tenth of a second on one GPU.

Serving latency. Latency and throughput of three serving paths on a 2,000-request stratified sample of the public suite (one RTX PRO 6000, one request at a time). One question against 16–63 questions in one request: median latency over all public-suite requests of 1k–4k tokens, in bf16.

Detailed Results

Headline numbers

Measure

Karpo Decision 31B

Reference

Notes

Decision Index 0.4.1 Full

73.9

Rank #1 on the board, 300 models

Scored by the Decision Index maintainers on private tests; November 12 board

Decision Index 0.3 public (37 benchmarks)

75.0

Board best 75.3

Official kit, one RTX PRO 6000

Knowledge (AA-Omniscience public)

56.0%

KnowLine 53.2%, Sieve 48.6%

Highest among models that answer every question

Safety (WildGuardTest)

93.4%

Five other models: 90.9%–92.9%

Mean of three yes/no judgements

JevBench Capability, public and sealed items

87.2

Rank #3 on the board, 243 Jev-class open-weight models; Jev 1.14 75.0

Scored by the JevBench maintainers; November 12 board

Internal evaluation set

72.3%

KnowLine 71.5%, Sieve 70.4%, GLiDE 69.1%

Same items, same scoring

Latency, median / p80

95 / 245 ms

Decision Index board limit 1,000 ms

All public-suite requests; one RTX PRO 6000, one request at a time

Decision Index results

On the November 12 board, the three models that follow Karpo Decision 31B by Full score are GLiDE 27B, REAX 1.3 and Logits Sigma-1 Duo. Every number in the table below is from the board:

Karpo Decision 31B

GLiDE 27B

REAX 1.3

Logits Sigma-1 Duo

DI Full (official)

73.9

73.4

72.6

72.3

DI public (37 benchmarks)

75.0

72.5

73.7

72.9

Median latency

94 ms

122 ms

246 ms

368 ms

REAX 1.3 and Logits Sigma-1 Duo are each an ensemble of two models, 57B parameters in total; the board labels such entries "inference technique". Karpo Decision 31B's median latency on the board is 94 ms; the 95 ms elsewhere in this post is our own measurement over all public-suite requests.

On the Decision Index board, Jev 1.14 scores 60.2 Full and 60.7 on the public suite.

The last three rows of the table below were measured by us on the same items for every model. KnowLine 31B Gen1, GLiDE 1.2 and Sieve 31B v3 were the Full top three on the November 4 board, and we ran them with their published weights and serving code; Gemini Decision 1 is not on the Decision Index board, and we measured it through its hosted API. We have not yet measured the top three of the November 12 board on these three rows.

Karpo Decision 31B

KnowLine 31B Gen1

GLiDE 1.2

Sieve 31B v3

Gemini Decision 1

DI public (37 benchmarks)

75.0

70.1

72.0

71.7

60.9§

Internal evaluation set

72.3%

71.5%

69.1%

70.4%

70.4%

Knowledge accuracy

56.0%

53.2%

41.8%†

48.6%

28.9%†

WildGuardTest

93.4%

90.9%

92.7%

92.8%

86.8%‡

† GLiDE 1.2 answers "I don't know" on about 48% of these items and Gemini Decision 1 on about 64%, which lowers their accuracy; on the items they answer, both are right about 80% of the time.

‡ Google's input filter blocked 38 of the 1,725 requests, which count as wrong; on the rest, Gemini Decision 1's accuracy is 88.7%.

§ Gemini Decision 1 is not on the Decision Index board; its public score is from our run of the official kit through its hosted API. Google's input filter blocked 69 requests, which count as wrong. Two forms its API does not accept (an empty-string state, and question keys that do not conform to its rules; 4,798 requests in all) were sent in an equivalent form. The other three models' public scores are from the November 4 board.

  • Karpo Decision 31B
  • the models it is compared with

From the official board, 12 November

Decision Index Full

official score, private tests

Karpo Decision 31B
73.9
GLiDE 27B
73.4
REAX 1.3
72.6
Logits Sigma-1 Duo
72.3

Decision Index public suite

score, 37 benchmarks

Karpo Decision 31B
75.0
GLiDE 27B
72.5
REAX 1.3
73.7
Logits Sigma-1 Duo
72.9

Median latency

milliseconds; lower is better

Karpo Decision 31B
94
GLiDE 27B
122
REAX 1.3
246
Logits Sigma-1 Duo
368

Measured by us on the same questions for every model

Internal evaluation set

unseen domains; accuracy, %

Karpo Decision 31B
72.3
KnowLine 31B Gen1
71.5
GLiDE 1.2
69.1
Sieve 31B v3
70.4
Gemini Decision 1
70.4

Knowledge accuracy

AA-Omniscience public questions, %

Karpo Decision 31B
56.0
KnowLine 31B Gen1
53.2
GLiDE 1.2
41.8†
Sieve 31B v3
48.6
Gemini Decision 1
28.9†

Safety judgement

WildGuardTest, %

Karpo Decision 31B
93.4
KnowLine 31B Gen1
90.9
GLiDE 1.2
92.7
Sieve 31B v3
92.8
Gemini Decision 1
86.8‡
  • † GLiDE 1.2 chose "don't know" on about 48% of the questions and Gemini Decision 1 on about 64%, which lowers their accuracy; on the questions they did answer, both are at about 80%.
  • ‡ Google's input filter blocked 38 of the 1,725 requests, which are counted as wrong; on the remaining requests Gemini Decision 1's accuracy is 88.7%.

Within a row, every model is scored on the same questions in the same way.

Against the Full top three and Gemini Decision 1. Top row: Karpo Decision 31B and the three models that follow it by Full score on the November 12 board, from the board. Bottom row: measured by us on the same items for every model; KnowLine 31B Gen1, GLiDE 1.2 and Sieve 31B v3 with their published weights and serving code, Gemini Decision 1 through its hosted API.

JevBench results

At our request, the JevBench maintainers ran the released weights and server code, unchanged, on November 10. The run used the October 27 draw of public and sealed items, one request at a time on one RTX PRO 6000, with no network access. On November 12, JevBench v1.8.10 equated that draw to the board's scale with four anchor systems, lowering Intelligence by 2.9 points and Calibration by 1.9, and the result was ranked on the board.

Capability, the mean of Intelligence and Calibration, is 87.2, ranked #3 among 243 Jev-class open-weight models, those within twice Jev's cost and median latency. The Composite Score, the harmonic mean of all four axes, is 77.4, official rank #7 of 261 ranked models.

Axis

Score

Intelligence, chance-corrected correctness

80.2

Calibration, stated confidence against outcomes

94.1

Speed, from latency per decision

88.3

Cost, from estimated cost per 1,000 decisions

57.9

  • Sealed and public. Intelligence is 85.67 on the draw's sealed items and 80.45 on its public ones (raw scores from the maintainers' run, before equating), so the method's penalty for a public score well above the sealed one does not apply.

  • Latency and cost. On the board, measured median latency is 82 ms, timed on the GPU with no network. For a self-hosted model the method scores Speed on twice that plus 150 ms, which is 314 ms. Cost is the lowest axis: the method estimates $0.0252 per 1,000 decisions from GPU time at its assumed throughput for a dense 31B model, not from a measurement.

On the JevBench board, from November 12

JevBench scores on four axes

0 to 100, higher is better

Intelligence
80.2
Calibration
94.1
Speed
88.3
Cost
57.9

The maintainers' run, raw scores before equating

Intelligence by item set

0 to 100; sealed items are never published

Sealed items
85.67
Public items
80.45

Capability 87.2, #3 of 243 Jev-class open-weight models; Composite 77.4, official #7 of 261 ranked models.

JevBench results. Top: the four axes on the JevBench board, from November 12. Bottom: Intelligence on the sealed and on the public items in the maintainers' run, raw scores before the draw was equated to the board's scale.

On the JevBench public items, in our own run, Karpo Decision 31B answers 213 of 231 correctly; Jev 1.13's published result is 200. Both are near-perfect on the easy and original tiers; the whole gap is in the 111 hard items.

Acknowledgements

Our prompt layout and attention rule follow the public work of the decision-model community: the question-list-first layout of FlexDM and jev-brain, and the shared-block and branch rule with bidirectional global layers of Sieve-31B-v3. Thanks to the Decision Index maintainers for an open, reproducible benchmark, and to the JevBench maintainers for their run.

Try it

References

  1. Gemma Team. Gemma 4 Technical Report. arXiv:2607.02770, 2026.

  2. Geuter, J., Cheng, A., Alvarez-Melis, D. Releasing the Inner Jev: A Simple Recipe for Open Decision Models. arXiv:2610.17205, 2026.

  3. Zheng, C. et al. Large Language Models Are Not Robust Multiple Choice Selectors. ICLR 2024.

  4. D'Amour, A. et al. Underspecification Presents Challenges for Credibility in Modern Machine Learning. JMLR 23(226), 1–61, 2022.

  5. Gulrajani, I., Lopez-Paz, D. In Search of Lost Domain Generalization. ICLR 2021.

  6. McCoy, R. T., Min, J., Linzen, T. BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance. BlackboxNLP 2020.

  7. Wortsman, M. et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022.

  8. Hu, E. J. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.

  9. Dang, X. et al. Weight Ensembling Improves Reasoning in Language Models. COLM 2025.

  10. Biderman, D. et al. LoRA Learns Less and Forgets Less. TMLR 2024.

  11. Ramé, A. et al. Diverse Weight Averaging for Out-of-Distribution Generalization. NeurIPS 2022.

  12. Almeida, D. Introducing System One Models & Jev. TypeSafe AI blog, September 2026 (origin of the term RLCD).

  13. Yaldiz, D. N. et al. Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning. arXiv:2601.13284, 2026.

  14. Iqbal, A. R., Nasim, M. F., Saleem, A. Typed Decision Models under Prevalence Shift: Decision Training, Calibration, and Implicit Priors. arXiv:2610.26325, 2026.

  15. Akiba, T. et al. Evolutionary Optimization of Model Merging Recipes. Nature Machine Intelligence 7, 195–204, 2025.

  16. Yadav, P. et al. TIES-Merging: Resolving Interference When Merging Models. NeurIPS 2023.

  17. Yu, L. et al. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. ICML 2024.

  18. Wang, K. et al. LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging. ICLR 2025.