Part 1, Understanding AI agents
Part 3, Introducing an agent
Principle 5
Challenge
Scope and limitations of this guide
This guide provides general information on organising work with AI agents, not advice tailored to a particular organisation, system or decision. It is not legal or regulatory advice.
Its examples, checklists and templates do not establish that a proposed use is safe, lawful or suitable. You need to assess and test the proposed use in your own circumstances. Following the guide does not by itself meet the requirements of any law or standard.
The guide is provided in good faith, using information available at the time of publication or revision. AI systems, provider terms and legal requirements can change. Before acting, check the current requirements that apply to your organisation and country, confirm what the system can do and test the proposed use. You are responsible for how you apply the guide and should obtain professional advice where appropriate.
Reading, downloading or using the guide does not by itself create an adviser–client relationship with Tom Clowry or Considered Ideas Ltd.
To the fullest extent permitted by law, Tom Clowry and Considered Ideas Ltd accept no liability for loss or damage arising from use of, or reliance on, this guide. Nothing in this notice excludes or limits liability for death or personal injury caused by negligence, fraud or fraudulent misrepresentation, or any other liability that cannot lawfully be excluded or limited.
What would make you reconsider the answer?
If two agents agree, what evidence supports their conclusion?
Decide which conclusion or assumption needs testing, and what you would do differently if it proved wrong.
- What to decide
- The assumption to test and the evidence that would change the decision.
- Key point
- Agreement is not enough to establish that a conclusion is justified.
Why it matters#
Review and challenge#
Establishes whether the work is reliable enough to use.
In the opening hours example, the service lead checks that relevant records have been included, the calculations are correct and the assumptions are appropriate.
Examines whether its assumptions, interpretation or proposed approach should change.
For example, the team tests whether the current records understate demand for later appointments.
The same assessment can involve both.
Why two agents agree
Two agents can agree because the evidence is persuasive. They can also agree because both started with the same assumption or missed the same information.
Agreement alone does not tell you which has happened.
Background#
From political philosophy
Why independent judgements matter#
The jury theorem, named after the Marquis de Condorcet, holds that a majority vote becomes more likely to be right as a group grows, provided that, on a choice between two options, each member has the same better-than-even chance of being right and each judges independently.
Each one adds evidence.
Their errors differ, so a majority is more likely to be right as the group grows.
Many votes add little.
They share the same errors, so agreement can simply repeat one mistake.
Models that share training data, assumptions or sources can make related errors. Their agreement should not be treated as independent confirmation. Principle 5 therefore asks what evidence supports a conclusion, rather than how many agents agree.
Source Condorcet, Marquis de (1785), Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix.
In practice#
6 steps
01Decide how much scrutiny the decision deservesIt comes down to the cost of a mistake and whether another perspective would alter the choice.
How much scrutiny a decision deserves comes down to the potential cost of a mistake and whether another perspective would meaningfully alter the choice. If neither justifies further analysis, it may be better to decide and move on.
02Choose a check that could resolve the doubtCheck a claim against its source, or examine a recommendation.
Choose a check that could resolve the specific doubt.
For a factual claimCheck it against its source.
For a recommendationConsider another explanation of the evidence or examine whether the proposal is practical.
03Choose how to examine the workOne analysis and a check, or several perspectives.
For a narrow question, one analysis followed by a check may be sufficient. A complex proposal needs examination from several perspectives. For example, a cost comparison also needs scrutiny from someone who understands how the work is done.
Focused work→
One clear questionCheck the resultMultiple perspectives→
Different professional viewsCompare trade-offsFresh analysis→
Separate assessmentsExamine assumptionsThree ways to examine an analysis If you ask an agent for a separate assessment, withhold the preferred conclusion so that the agent is not instructed to justify it.
04Add perspectives when they could change the decisionSeparate analyses can still rely on the same incomplete evidence.
Separate analyses can reveal different assumptions, but they can still rely on the same incomplete evidence. Use them when another assessment could help resolve a question.
When comparing separate analyses, those responsible for the decision should examine the evidence and the reasons for any disagreement. Agreement between agents does not by itself establish that a conclusion is justified.
05Identify what would change your viewAn instruction to be critical gives an agent little direction.
An instruction to be critical gives an agent little direction. A more useful Brief The instructions given to the agent, including the purpose, permitted information and actions, checks and arrangements for seeking advice. asks:
Which assumption the recommendation depends on
What evidence could contradict it
What should be done differently if that evidence appears
How a credible alternative compares
Ask the agent to assess it using the same criteria.
06Consider what kind of disagreement it isCounting how many agents favour each option will not resolve it.
When analyses disagree, consider what kind of disagreement it is. A dispute about a cost estimate can be resolved by better evidence. A disagreement about how much extra cost is justified by better access requires someone to weigh those priorities.
Counting how many agents favour each option will not resolve it.
See also · 1.3 Several agents working together
Examples#
The opening hours example
When two analyses disagree#
A local service is open from 9 am to 5 pm on weekdays. Some service users have asked for evening appointments, and the team must decide whether longer hours would improve access. An agent examines twelve months of request records and compares the costs of the options. See chapter 1.5.
Favours later opening because it could improve access.
Asks the agent for a separate assessment, without the preferred conclusion.
Questions whether the service could cover the extra hours.
Notes that both lack evidence about who would attend. Another recommendation based on the current records would not supply it.
Approves a short survey of service users who have agreed to be contacted. The first brief did not allow the agent to contact anyone, so this needs new permission.
Contacting people is an action rather than analysis, so it needs its own permission and checks.
The opening hours example
The survey#
A local service is open from 9 am to 5 pm on weekdays. Some service users have asked for evening appointments, and the team must decide whether longer hours would improve access. An agent examines twelve months of request records and compares the costs of the options. See chapter 1.5.
Approves the survey wording and arrangements for handling replies, including what personal information is needed, who can see it and how long it will be retained.
Sends the survey and one reminder, collects the replies and adds them to the comparison. Under the approved handling arrangements, it passes any reply that raises a personal difficulty to the service lead.
Considers how to hear from people who cannot currently access the service, since a survey of existing users will miss them.
In other work
A software fault#
Risks#
From the research
Agreement between models may add less evidence than expected#
A preprint published in 2026 examined nine AI model The part of an agent that reads, reasons and writes, trained on large amounts of text and other material. from seven model families on benchmark judgement tasks. Their errors were correlated, reducing the panel’s effective information to about two independent votes. In these evaluations, the panel offered no meaningful improvement over the best individual model. The finding illustrates why agreement between models should not be treated as independent confirmation.
Source Kohli, G. (2026), Nine judges, two effective votes, arXiv:2605.29800 (preprint).
Exercise#
Decide what would change the answer#
Choose a recommendation and identify one plausible finding that would make you reconsider it. Decide whether to investigate by checking a source, requesting another analysis, speaking to someone or testing the proposal in practice.
If further analysis would not resolve the remaining question, explain what evidence you need to gather next.
Record the assumption you need to test and how you will test it. Explain what evidence is needed before a decision can be made, and record any disagreement for those responsible for the decision to consider.
Further reading · The Right Amount of Doubt · clowryai.com/thinking
© 2026 Considered Ideas Ltd. Licensed under the Creative Commons Attribution 4.0 International licence (CC BY 4.0). This guide provides general information and is not legal or regulatory advice. The guide’s home page gives the suggested credit.
Scope and limitations of this guide
This guide provides general information on organising work with AI agents, not advice tailored to a particular organisation, system or decision. It is not legal or regulatory advice.
Its examples, checklists and templates do not establish that a proposed use is safe, lawful or suitable. You need to assess and test the proposed use in your own circumstances. Following the guide does not by itself meet the requirements of any law or standard.
The guide is provided in good faith, using information available at the time of publication or revision. AI systems, provider terms and legal requirements can change. Before acting, check the current requirements that apply to your organisation and country, confirm what the system can do and test the proposed use. You are responsible for how you apply the guide and should obtain professional advice where appropriate.
Reading, downloading or using the guide does not by itself create an adviser–client relationship with Tom Clowry or Considered Ideas Ltd.
To the fullest extent permitted by law, Tom Clowry and Considered Ideas Ltd accept no liability for loss or damage arising from use of, or reliance on, this guide. Nothing in this notice excludes or limits liability for death or personal injury caused by negligence, fraud or fraudulent misrepresentation, or any other liability that cannot lawfully be excluded or limited.

