What makes an AI comparison fair?

Start with the conditions

Suppose two AI workflows attempt the same task. One has clearer tool descriptions, extra attempts, or a less disruptive failure to recover from. A higher score may say something about those conditions rather than the method being evaluated.

Our development work has reinforced a simple lesson: the evaluation setup deserves as much scrutiny as the result. A comparison becomes useful only when we can explain what stayed the same and what deliberately changed.

Make recovery part of the task

Business workflows are rarely one clean question followed by one clean answer. They involve tools, changing state, incomplete responses, and occasional failures. A system may need to work out what already happened before it can safely continue.

For a fictional order-update task, both candidates should face equivalent information, permissions, and interruptions. If a tool fails, the failure should have comparable meaning for both. Otherwise we risk rewarding an easier environment rather than better recovery.

Keep development separate from confirmation

Development runs are useful for finding defects in a test and improving the system. Once those results shape our choices, they are no longer a clean test of a previously fixed hypothesis.

A separate, untouched evaluation helps answer a different question: does the improvement survive beyond the examples that guided development? The tasks, scoring rules, and permitted resources should be settled before interpreting that result.

Publish only what the result supports

A promising development signal is a reason to investigate. It is not automatically a general capability claim, a production guarantee, or a customer outcome. The result needs a clear scope and a recoverable record.

UBIKITOS is model-agnostic by design. That makes fair comparison especially valuable: we want to understand which parts of a governed workflow help, under which conditions, without confusing a preferred model or a favorable setup with evidence of improvement. This note describes our evaluation principles, not a benchmark result.

Decide which question the comparison answers

An evaluation can compare complete products, isolate a particular mechanism, or explore failure patterns during development. These are different questions. A product comparison may reasonably include the whole workflow, including prompts and tools. A mechanism comparison needs tighter control over other differences. A development run may deliberately change conditions to diagnose a problem. The interpretation should match the design. A result from one kind of comparison should not quietly become a stronger claim from another simply because the score looks encouraging.

Writing the question before the run helps keep that relationship stable. What would count as success? What would count as a regression? Which cases are included, and why? What resources may each side use? What result would make the team change its view? These choices do not eliminate judgement, but they make the judgement inspectable. They also reduce the temptation to select a favourable explanation after seeing the numbers. The evaluation becomes a way to learn, rather than a way to find a statistic that supports an existing preference.

Look inside the average

An overall success rate can conceal a mixed result. One approach may recover from interruptions better while making more unsupported claims in research tasks. Another may be faster on straightforward requests but weaker when a workflow has several dependent steps. The distribution matters because users do not experience an average in the abstract. They experience the task in front of them. A useful report therefore preserves meaningful task families and failure types alongside the total, especially when the intended product use emphasises one of them.

Small groups need careful interpretation too. A clean result across a few cases can be an excellent reason to investigate further without supporting a broad performance promise. The sample size, task selection, and uncertainty should remain attached to the finding. That does not make the result uninteresting. It makes the next experiment easier to design. The team can focus on the family that showed a signal, repair the family that regressed, and avoid treating a single aggregate as a verdict on every capability the system might have.

Count resources and human help

The resources used to obtain a result are part of its practical meaning. More tokens, more attempts, extra context, or human corrections can improve performance. Those contributions may be worthwhile, but they should be visible. Otherwise, a comparison can suggest that a workflow is more efficient when it is actually spending more to achieve the outcome. Efficiency and quality are related questions, not synonyms. The right tradeoff depends on the task and on what the user values in the intended setting.

Human intervention deserves the same attention. If one side receives clarifications or repairs that the other does not, the result may partly measure that assistance. Even when intervention is allowed, it should be accounted for consistently. A founder assessing autonomy needs to include the effort around the output: preparing the request, correcting assumptions, reviewing the result, and recovering from failure. An agent can produce an answer quickly while leaving substantial work with the person. The evaluation should help reveal that burden instead of hiding it outside the scoring window.

Make the result recoverable

A fair design is less useful if the exact result cannot be recovered later. Preserve the evaluated configuration, the task set, the scoring method, and the outputs needed to inspect the verdict. A short public summary can then remain connected to a more complete internal record. This helps when a later result appears to conflict with an earlier one. The team can compare conditions instead of relying on memory or assuming that the newer number supersedes everything that came before it.

The same care supports model-agnostic development. A system should be able to compare different models or workflows without changing the interpretation rules to favour a preferred candidate. The outcome may be that different approaches suit different jobs. It may be that a promising mechanism needs more work. Either answer can guide product decisions. The value of a fair comparison is not that it guarantees an exciting winner. It is that the next engineering or deployment choice can be made with a clearer understanding of what the result actually establishes.

A final practical question is whether the finding changes a decision. Does it suggest a narrower supported use, a new test, or a candidate worth further development? A score that cannot be connected to any next action may be less useful than a smaller result that identifies a concrete weakness. Evaluation should help the team choose where to spend effort and what to tell users. Keeping the question, conditions, and result together makes that choice more defensible. It also allows the team to change its mind when new evidence arrives without having to protect a claim that was broader than the original test.