mumubench 1.0: decision models, scored on a coding agent's decisions
On mubench's main board, every item is a request mu sent to a decision model, word for word, in a real coding session on an open-source repository. Each answer is scored by the cost of its consequences, and every model is compared with mu running without a decision model.
Real coding sessions
3,193 requests from 86 coding tasks on open-source repositories.
Without a decision model, mu's mistakes have a cost. A score is how much of that cost a model removes: 100 removes all of it, 0 is no better, below 0 is worse. It isn't accuracy; that's in decision accuracy below.
| Model | Overall | Permission approval | Malicious-instruction detection† | Context compression | On-demand loading and reminders | Classification and routing | Progress and completion | Memory and lessons | Multi-agent collaboration |
|---|---|---|---|---|---|---|---|---|---|
| mude 1.0 | 64.9 | 52.6 | — | 64.6* | −2.7* | 83.6 | 86.6 | 58.7 | 19.8* |
| Jev 1.14 | 63.1 | 53.7 | — | 76.7* | 16.8* | 87.5 | 61.6 | 75.1 | 4.6* |
| Jev 1.13 | 61.5 | 52.7 | — | 70.7* | 25.1* | 85.8 | 61.6 | 72.2 | −0.8* |
| d3-nano (original) | 32.0 | 5.9 | — | −19.1* | −118.4* | 51.7 | 70.1 | 40.5 | 26.0* |
| Items (tasks) | 3,193 (86) | 449 (86) | 19 (5) | 74 (15) | 300 (86) | 547 (86) | 1,143 (86) | 437 (82) | 224 (9) |
| Weight in overall | 100.0% | 23.7% | 0.0% | 5.2% | 3.0% | 31.5% | 18.0% | 9.0% | 9.6% |
- *Few items or an unstable score: fewer than 150 items, items from fewer than 20 tasks, or mu already right more than 95% of the time with no decision model.
- †Malicious-instruction detection isn't scored yet.
Sub-board: constructed scenarios
1,655 scenarios written for mubench and scored the same way. Their mix of decisions differs from real sessions, so the two boards are scored separately and never added together.
| Model | Overall | Permission approval | Malicious-instruction detection† | Context compression | On-demand loading and reminders | Classification and routing | Progress and completion | Memory and lessons | Multi-agent collaboration |
|---|---|---|---|---|---|---|---|---|---|
| mude 1.0 | 81.8 | 74.5 | — | 83.5* | 64.2 | 93.5 | 82.1 | 85.1 | 81.8 |
| Jev 1.14 | 80.5 | 74.8 | — | 58.5* | 83.3 | 94.2 | 80.8 | 83.0 | 69.4 |
| Jev 1.13 | 76.4 | 65.2 | — | 60.2* | 88.7 | 93.3 | 71.2 | 81.8 | 73.7 |
| d3-nano (original) | 56.7 | 51.2 | — | 37.5* | 42.4 | 77.6 | 54.2 | 36.2 | 67.6 |
| Items | 1,655 | 242 | 0 | 139 | 193 | 310 | 341 | 276 | 154 |
| Weight in overall | 100.0% | 28.7% | 0.0% | 4.7% | 6.3% | 23.1% | 18.6% | 9.5% | 9.1% |
Decision accuracy
The share of checks a model gets right (an item can have several checks), with mu running without a decision model as the first row. Here every check counts the same, whatever its mistakes cost, so accuracy and score can rank models differently.
Main board: real coding sessions
| Model | Overall | Permission approval | Malicious-instruction detection† | Context compression | On-demand loading and reminders | Classification and routing | Progress and completion | Memory and lessons | Multi-agent collaboration |
|---|---|---|---|---|---|---|---|---|---|
| No decision model | 78.0% | 60.6% | — | 96.3% | 96.4% | 61.3% | 86.6% | 82.9% | 43.6% |
| mude 1.0 | 91.5% | 96.6% | — | 98.0%* | 98.0%* | 83.1% | 94.7% | 93.4% | 70.7%* |
| Jev 1.14 | 90.1% | 81.8% | — | 98.7%* | 98.2%* | 85.2% | 92.5% | 92.1% | 62.8%* |
| Jev 1.13 | 89.4% | 80.7% | — | 98.5%* | 98.3%* | 84.1% | 92.3% | 91.5% | 59.0%* |
| d3-nano (original) | 83.5% | 66.0% | — | 87.7%* | 94.3%* | 75.3% | 92.9% | 80.5% | 60.9%* |
Sub-board: constructed scenarios
| Model | Overall | Permission approval | Malicious-instruction detection† | Context compression | On-demand loading and reminders | Classification and routing | Progress and completion | Memory and lessons | Multi-agent collaboration |
|---|---|---|---|---|---|---|---|---|---|
| No decision model | 44.5% | 54.9% | — | 35.7% | 47.0% | 31.2% | 49.8% | 57.9% | 49.5% |
| mude 1.0 | 92.1% | 93.8% | — | 93.8%* | 91.5% | 91.0% | 90.9% | 93.8% | 91.2% |
| Jev 1.14 | 91.8% | 95.8% | — | 88.0%* | 93.0% | 95.4% | 89.8% | 91.3% | 81.7% |
| Jev 1.13 | 91.1% | 94.8% | — | 88.9%* | 93.6% | 94.2% | 84.7% | 90.4% | 84.6% |
| d3-nano (original) | 80.1% | 74.7% | — | 76.7%* | 88.4% | 82.1% | 72.1% | 77.6% | 82.8% |
- *Few items or an unstable score: fewer than 150 items, items from fewer than 20 tasks, or mu already right more than 95% of the time with no decision model.
- †Malicious-instruction detection isn't shown here either: none of the main board's 19 such requests contains an attack, and every model, as well as mu with no decision model, gets them all right. The sub-board has none yet.
Scored by consequences
How the score is computed
Capability score
100 × Σ w·(f − r) / Σ w·fSummed over the capability's decision points: f is the average cost per check with no decision model, r the same with the model, and w the decision point's weight (3 for safety gates, 2 for decisions made every turn, 1 for the rest). 100 means all of that cost is removed, 0 no better than no decision model, below 0 worse.Overall scoremubench's pooled score over every decision point. It equals the mean of the capability scores, each weighted by its share of the cost with no decision model (the "Weight in overall" row); the costs and weights were fixed before any model was scored. Not equal weights: item counts differ about 60-fold between capabilities, and equal weights would let the smallest, noisiest ones swing the result.
95% intervalsThe thin lines on the overall charts. A bootstrap with 1,000 draws, resampled within each decision point: by task on the main board, since requests from one session aren't independent, and by scenario on the sub-board.
— (not scored)The capability has no items, or mu loses nothing there without a decision model, so the score is undefined.
Eight capabilities
Permission approvalWhether an action may go ahead: is a flagged command what the user asked for, can a call be approved for the user, does it break a constraint the user set.tool.risktool.approvaltool.constraint
Malicious-instruction detectionWhether outside content, such as a web page, a search result or an MCP reply, carries instructions aimed at the AI, and whether an MCP tool's description slips in commands that have nothing to do with the tool.tool.injectiontool.description
Context compressionWhen the context is nearly full, which old tool output to keep; whether an old result is still worth its tokens; which parts of a tool's output can stay out of the context.context.compactcontext.forgettool.admission
On-demand loading and remindersWhich packs and skills a task needs; whether and when to tell the model about diagnostics or outside events; whether to keep the prompt cache warm.capability.disclosureskills.disclosurediagnostics.deliverynotify.routingcache.warming
Classification and routingWhat kind of message this is and how much effort it deserves; whether a mid-run message changes course now or waits; also review triage, finding files and the browser's next step.input.preflightinput.interjectiontask.framejudge.itemsreview.triagefiles.locatebrowser.step
Progress and completionWhether the turn is done, whether the agent should keep going, whether the goal is met, whether the work has drifted or should be rewound, and whether output breaks a rule.turn.completionturn.continuegoal.metturn.driftturn.rewindoutput.driftboard.read
Memory and lessonsWhat is worth remembering, whether it is already known, which lesson applies now, whether the agent followed it, and whether it can become a skill.memory.capturememory.worthmemory.mergememory.recallmemory.appliedmemory.outcomeskill.capture
Multi-agent collaborationWhat to pass between agents in a hive, to whom, and how it relates to earlier findings; whether a subagent's changes stay in scope; which role and thinking level a subtask gets.hive.publishhive.deliverhive.relateswarm.patchswarm.routing