GoKawiilA computer-science experiment captures, with unusual clarity, the difference between designing artificial-intelligence systems that are ever-more powerful according to a fixed benchmark and developing tools that genuinely support human judgement1. Researchers have created a collaborative chess game in which each team comprises pairs, partnering a strong AI with a weaker, human-like one. A coin toss decides, before each move, which partner will play. Neither knows in advance which will go next.
The result was striking. Despite being weaker at conventional chess, AI tools designed to make moves that the human-like partner could build on consistently beat teams led by Leela, a superhuman chess AI1. Being powerful was not enough: compatibility with a partner was more important.
The intelligence illusion: why AI isn’t as smart as it is made out to be
This result reframes what interpretability should mean in the context of AI adoption. Rather than asking whether a human can understand an AI system’s output, we should check whether they can act on it productively. In radiology, for example, that might mean an AI tool suggesting not just a diagnosis, but also highlighting the region of a chest X-ray that prompted the diagnosis, so that the physician can assess whether the system’s focus matches their own analysis.
Yet individual interpretability is only the first step. Chess has a fixed objective: checkmate. The goal never changes. In most professions now adopting AI, the situation is more complex. In health care, law and education, for example, practices are continuously adjusted by the professionals working in these fields. What counts as good patient care, sound legal interpretation or a fulfilling education cannot be specified and optimized only once.
This gap between fixed objectives and evolving professional values has consequences for AI design. The chess experiment looks at whether the next player can comprehend and act on an AI’s output. In professional practice, however, it is equally important that the field retains the capacity to interrogate and refine the values that structure the work. An AI system can be interpretable for each user, yet narrow the range of questions the community asks.
AI is programmed to hijack human empathy — we must resist that
Consider a diagnostic AI tool trained to frame clinical uncertainty in terms of probability scores. In some cases, that works well: a 70% likelihood of bacterial infection based on the test results, for example. But this approach collapses when the judgement has no concrete statistical basis2. For instance, when a physician suspects domestic abuse, the decision of whether to record their concern cannot be reduced to a probability. It rests on their judgement about whether documentation would help to protect the person or risk worsening the family situation. Such decisions are interpretive and ethical, rooted in contested values. When every uncertainty is forced into a probabilistic frame, value-laden professional judgement does not just get distorted, it becomes invisible.
The impact of AI tools on such collective decisions matters more than is currently acknowledged. When practitioners grapple with how uncertainties should be expressed, they are also deciding what counts as professional judgements and which values should guide their work.