As AI agents take on more decisions, they will increasingly encounter situations where achieving an objective is not the same as doing what is morally right.
An agent's objectives, permissions and constraints embody assumptions about what is desirable, permissible and fair.
But on what moral basis should its decisions be made?
Moral philosophy offers three broad approaches:
- Consequentialism judges actions by their consequences, considering the overall benefits and harms.
- Deontology emphasises moral duties and principles, such as honesty and fairness, which cannot always be overridden by better outcomes.
- Virtue ethics asks what good character and practical wisdom require in the circumstances.
Even when the facts are agreed, these approaches can reach different conclusions.
This introduces moral uncertainty. Uncertainty about which account of right and wrong should govern a decision.
How should these competing approaches be weighed? The difficulty is that there is no agreed common measure of moral value. What weight should moral duties carry against better overall outcomes? Even establishing a basis for comparison involves moral assumptions.
An AI agent can make decisions with moral consequences without itself being a moral agent.
Delegating decisions to AI does not eliminate the moral judgements involved. It places them within the design and governance of the system, and makes ongoing human oversight essential.
The question is not whether AI agents can make moral decisions, but whose moral principles govern those decisions, how they are justified, and who is responsible for them.
Related reading: Autonomy expands accountability.