Consensus scoring
The panelScore is the core signal that drives convergence. This page explains how the orchestrator computes it, why the choice of evaluator matters, and how to tune agreementLimit for different question types.
The basic algorithm
After every round, the orchestrator asks an independent evaluator model to score every pair of responses on a 0 to 100 agreement scale. The orchestrator takes the mean of all those pairwise scores as the round's panelScore.
For a 3-model panel, that's 3 choose 2 = 3 pairwise scores. For a 5-model panel, that's 10. For an 8-model panel, 28. The orchestrator averages across all pairs.
panelScore = mean([
score(modelA_response, modelB_response),
score(modelA_response, modelC_response),
score(modelB_response, modelC_response),
… // all N choose 2 pairs
])
converged = panelScore >= agreementLimit. The debate ends when this is true. It also ends when the orchestrator hits maxRounds.
What "agreement" means to the evaluator
The evaluator is given a system prompt that says, in effect:
You are an independent consensus evaluator. Compare these two responses. Score 0 to 100 how much they agree on the substantive answer. 100 means they reach the same conclusion with the same reasoning. 0 means they disagree on the core claim. Do not penalize stylistic differences; only count substantive agreement.
The score is therefore a measure of substantive consensus, not surface-level text overlap. Two models that reach the same recommendation via different reasoning will score high. Two models that reach opposing recommendations with similar confidence will score low.
The score is a single number from a single evaluator per pair. No averaging happens across evaluators. This is a deliberate trade-off. It is cheap, one model call per pair per round, and consistent, one evaluator and one rubric. If you need higher-stakes scoring for a high-stakes decision, run the same debate twice with two different evaluator models and average.
What you see in the response
The panelConsensus field on each round:
{
"panelScore": 82,
"perModel": [
{ "modelId": "openai/gpt-5.5", "score": 80 },
{ "modelId": "anthropic/claude-opus-4.8", "score": 84 },
{ "modelId": "google/gemini-3.1-pro-preview", "score": 82 }
],
"pairwise": [
{ "a": "openai/gpt-5.5", "b": "anthropic/claude-opus-4.8", "score": 78 },
{ "a": "openai/gpt-5.5", "b": "google/gemini-3.1-pro-preview", "score": 84 },
{ "a": "anthropic/claude-opus-4.8", "b": "google/gemini-3.1-pro-preview", "score": 84 }
],
"rationale": "All three converge on the recommendation; minor disagreement on the cost-benefit framing.",
"isConverged": true
}
The orchestrator computes perModel as the mean of that model's pairwise scores against the others. So a model that's the outlier will have a lower perModel score than the rest.
What "isConverged" means
The orchestrator sets isConverged: true when both of the following are true:
panelScore >= 85, AND- No pairwise score is below 30 (i.e. no outright contradiction in the panel)
This is stricter than hitting the agreementLimit. A panel that scores 80 because every model said a weak "yeah, probably" is not converged. A panel that scores 75 but has one model that scored 8 against another is not converged either.
If you want to override this and force convergence at any panelScore >= agreementLimit, set agreementLimit higher, say 90, so the soft convergence and the strict check align.
Picking the right agreementLimit
The threshold is the main tuning control. It defaults to 75, but the right value depends on the question.
| Threshold | Use when |
|---|---|
| 60 to 70 | Exploratory questions. You want the panel to surface disagreement, not paper over it. Use it for genuinely contested questions. |
| 70 to 80 | General use. This is the default. It works for most decisions and recommendations. |
| 80 to 90 | High-stakes recommendations. The panel has to agree, not nod along. Use for board memos, hiring decisions, legal reviews. |
| 90+ | Critical recommendations where convergence at a lower level would claim consensus too early. Use sparingly. Debates that fail to converge are still useful. The verdict does not claim consensus. |
A debate that doesn't converge is not a failure. It signals that the question is harder than you thought or that you chose the panel poorly. The chairman still writes a verdict. That verdict does not claim consensus.
Cost and latency
The pairwise evaluation is the second-largest cost in the platform, after the model turns themselves. For an N-model debate that runs to M rounds, the evaluator cost is M * N*(N-1)/2 model calls. For 3 models and 3 rounds, that's 9 evaluator calls. It is cheap because the evaluator is a small model, but it is not free.
The evaluator defaults to deepseek/deepseek-v4-pro, the same model as the chairman. Override via the assembly's formatMeta.evaluatorModelId if you need a specific evaluator, for example for cost or capability reasons.
Per-claim confidence vs panel score
Two different signals:
panelScore. How much the panel agrees. One number per round.per-claim confidence. How confident the chairman is in each individual claim it makes. One number per claim in the verdict.
They often correlate but are not the same. A panel can converge on a wrong answer. Everyone agrees, but the chairman scores that answer's claims low. A panel can fail to converge but still produce a high-confidence verdict. The chairman can still pick a side even if the panel didn't.
For trust calibration, look at both. Low panelScore with high per-claim confidence means the chairman was decisive despite disagreement. High panelScore with low per-claim confidence means the panel agreed but the chairman is hedging.