Accuracy, efficiency
and the cost of failure.
Leon, Hermes and OpenClaw · the same 36 selected tasks, six domains and three independent attempts per agent. Native rubric scores and recorded telemetry, with no new benchmark runs.
Overall native score
Average the three attempts per task, then six tasks per domain, then six equally weighted domains. Higher is better.
Runtime completion is separate
Finished without an agent exception. A completed turn can score zero; a timed-out turn can retain valid partial or full rubric credit.
Scores by domain
Every domain has full matched coverage: six tasks × three attempts × three agents.
Score variability and uncertainty
The intervals below resample the selected tasks within each domain, paired across agents. They describe local task variability; they do not prove population representativeness or a stable ranking over all possible tasks.
4,000 bootstrap samples · seed 20261013 · three-attempt task means retained. Per-task attempt medians and ranges are available in the task explorer.
Latency, tokens and inference volume
Candidate telemetry is separated from image generation, judging and grader recovery.
Task and lifecycle timing
Pre-environment wait is measured from trial creation to environment setup; it includes admission/orchestration and is not a pure scheduler queue metric. Native execution time includes startup and ends at completion or the published deadline.
Model response latency
Each cell shows p50 / p95 and sample count. A tool-only response may have no text event. Assistant-event spacing includes thinking and tools; it is not isolated model latency. Tool-duration coverage differs across native exports.
Candidate token usage and requests
Token totals include delegated/auxiliary inference passing the observed transport. Cached input is a subset of input; reasoning output is a subset of output. They must not be added again. Missing usage means totals are observed lower bounds, not invented zeros.
Image generation, judging and recovery usage
Returned models, service tiers and request errors
Correctness has several layers
A valid tool name and schema do not prove that the chosen action or target was correct. Successful execution can still produce an incorrect artifact.
Tool schema validity
Valid calls / calls with a checkable tool and argument schema. Coverage is reported separately.
Explicit tool execution status
Native structured result flags only. Legitimate probes may fail. Unknown results are excluded from the success denominator.
Semantic tool accuracy
Native action, routing, factual and artifact criteria remain available for all attempts. A universal semantic per-call accuracy rate was not independently labeled.
Resource consumption
Each accepted attempt had 2 CPUs and 4,096 MB RAM. Per-container peak memory/disk were not captured. Cleanup receipts and evidence hashes are preserved.
Critical Safety criteria
These means use only the six API-leakage attempts per agent where each criterion exists, not all tool calls. Safety gates can force the native overall score to zero. Other Safety tasks and every native submetric are available in attempt details.
Additional metrics from the original plan
Exact inference retry counts, a universal false-completion-claim rate, isolated decoding throughput, and peak container memory/disk are unknown. Repeated tool signatures are observable, but repetition alone is not wasted work or an error. No extra model review was run to fill those gaps.
What failed, and what to improve first
These findings apply to the frozen versions and recorded presets. Genuine native failures retain their scores; known broken environments were replaced consistently.
Zero scores and runtime exceptions
Every selected task and accepted attempt
Choose a task to inspect all nine runs, score variation, native criteria and per-run metrics. No superseded results enter the ranking.
Attempt explorer
The recovered metrics agreement
Recovered from our original October 11 session agreement and the follow-up that kept additional model review out of the default checks.
Original agreement messages
Frozen conditions
How scoring works
- 36 fixed tasks, six per domain.
- Three independent attempts per agent/task.
- Task score = arithmetic mean of its three attempts.
- Domain score = mean of its six task scores.
- Overall score = mean of the six domain scores.
- Native partial scores and genuine deadline outcomes retained.
- Superseded and withdrawn attempts preserved outside ranking.
Concurrency and cleanup
- Up to nine independent attempts for one task cohort.
- Fresh mutable workspace and profile for each attempt.
- Shared immutable images/dependencies reused across the cohort.
- Native deadlines; no arbitrary 30-minute cutoff.
- Export and grade before removing containers.
- Task-owned resources retired after their final consumer.
- No global Docker prune; large records retained in place.
Repairs, exclusions and limitations
Frozen evidence and reproduction files
The archive inventories original artifacts in place and snapshots mutable metadata. Verify checksums before reproducing or extending the campaign. The HTML contains only derived telemetry and numeric criteria; raw transcripts and credentials are not embedded.
Companion evidence links require the archive folder. Charts, filtering and the JSON download work offline in this single HTML file.
Evidence fingerprint: