Behind the builder / 13 October 2026
How do you benchmark a harness?
A Kaggle idea puts a useful question on the table: how do we measure what an AI harness contributes when the language model can be swapped?
The model is not the whole machine
The 13 October close captures a concept for evaluating the R4 harness through a Kaggle task. It is planning, not an entered competition, an executed benchmark or a leaderboard result.
The question is useful because UBIKITOS is model-agnostic. A WorkMachine pairs a compatible LLM with the harness that manages work around it. Evaluating that combination requires more than asking whether the model can produce a plausible answer.
Hold the comparison still
A meaningful future comparison would keep the task, model and resource budget comparable while changing the surrounding workflow. It could then examine whether evidence checks, bounded tools and recovery improve a measurable outcome.
Accuracy alone would not tell the whole story. Cost, failed attempts, permission violations and the ability to reconstruct a result could matter too. These are proposed evaluation criteria, not reported measurements from a completed trial.
Run the route before racing it
The dated review still places the native R4 canary and independent acceptance ahead of a benchmark claim. Research activity also remains unverified; growing ledgers do not establish a successful result.
First make the authorized execution path work and preserve its evidence. Then define a comparison that can fail. That sequence gives a future benchmark a chance to reveal something useful about the harness instead of merely decorating it.