GoKawiilArtificial intelligence is fast becoming embedded in hospitals and healthcare clinics. Yet, with the benefits of AI tools come fresh patient-safety risks — and questions about who is responsible when things go wrong.
Conventionally, responsibilities in health care are clearly delineated. Clinicians must provide a set standard of treatment. Institutions must organize safe treatment processes. Manufacturers must deliver non-defective medical devices. Regulators and bodies involved with licensing and credentials can intervene if any of these parties fall short of expected standards. And courts can assess which party is liable if patients are harmed.
An incoming wave of medical AI tools is set to blur these lines.
Current AI tools are being used mainly as assistants, backed up by human checks, much as with other medical devices. For instance, when AI models trigger an alert that a patient is at risk of sepsis, a clinician must review and confirm the physiological rationale before any treatment.
Medical AI can transform medicine — but only if we carefully track the data it touches
But in the next few years, AI-driven clinical tools are expected to advance from synthesizing data to acting on it, with less and less human input. They will make diagnoses, devise treatment plans and make patient-management decisions that clinicians play little or no part in. Whereas the outputs of many existing tools are understandable — sepsis models, for instance, work by combining defined physiological variables according to a transparent formula1 — more-advanced tools can involve black-box deep-learning processes. Clinicians will no longer be able to fully follow the tools’ reasoning, even for algorithms that provide some explanations for their judgements2.
Such tools land between accountability rules. Their black-box reasoning makes it hard to determine when they are defective. And when decisions are made jointly by humans and an opaque AI model, it becomes difficult to assess responsibility when people are harmed while receiving health care. Existing medical-liability frameworks do not address this crossover point3.
This legal uncertainty means that people who are harmed could fall into liability gaps in which no one has clearly broken a rule3. Hospitals are likely to avoid AI for potentially useful functions because of concerns about legal risks — concerns that are repeatedly cited by physicians as a barrier to safe AI adoption4–6 and that are seen by some as grounds not to use black-box tools already available. And AI vendors might shirk their obligation to monitor safety once a tool reaches the market, confident that it will be hard to attribute blame when things go wrong.
Clear liability frameworks are needed.
Here we outline how to achieve that, by defining seven levels of AI capabilities on the basis of three factors: autonomy, automation and operational scope. With aircraft and self-driving cars, regulators already use graded levels to specify exactly what tasks systems must perform, how independently, and when humans are expected to take over7–9. A similar set of levels for medical AI systems would help regulators, policymakers, governments, courts and professional health-care bodies to plan for the incoming tools.
... continue reading