METHODOLOGY How we investigated Amsterdam’s attempt to build a ‘fair’ fraud detection model
For the past four years, Lighthouse has investigated welfare fraud detection algorithms deployed in five European countries. Our investigations have found evidence that these systems discriminated against vulnerable groups with oftentimes steep consequences for people’s lives.
Governments and companies deploying these systems often show little regard for the biases they perpetrate against vulnerable groups. The city of Rotterdam told us that it had never run code designed to test whether its model was disproportionately flagging vulnerable groups. France’s social security agency, CNAF, confirmed to us that it had never audited its model for bias.
In January of 2023, three months before publishing our investigation into Rotterdam’s risk scoring algorithm, we sent a public records request to the city of Amsterdam, one of Europe’s most progressive capitals. Among other things, we asked for documents, code and data relating to a similar system Amsterdam had been developing. Given that we had fought over a year to obtain these types of material in other investigations, we were surprised when the city immediately complied with our request.
The materials disclosed by the city were related to a machine learning model it was developing in order to predict which of the city’s residents were most likely to have submitted an incorrect application for welfare. At the time of our public records request, the model was still in development. The overall development goal for the model, according to the city’s internal documentation, was to have fewer welfare applicants investigated, but a higher share of those investigated rejected. Internal documents also emphasized two other aims: avoid bias against vulnerable groups, and outperform human caseworkers.
After reading through the documentation, it quickly became clear to us why the city had been so forthcoming. The city had gone to significant lengths in order to develop a model that was transparent and treated vulnerable groups fairly.
We wanted to investigate whether the city of Amsterdam had succeeded in developing a fair model. Over the course of eight months, we ran a series of our own tests on the model and data containing real world outcomes for people flagged as suspicious by the model. When it came to the outcome data, the city ran our own tests locally and returned aggregate results in order to comply with European data protection laws.
The past five years have seen a number of regulatory efforts to reign in harmful uses of AI. The EU AI Act, passed in 2024, will require AI systems deemed “high risk” to be registered with the European Commission. In New York City, employers have been prohibited from screening employees using AI without conducting bias audits since July 2023. Meanwhile, the private sector, academics and multilateral institutions have produced a number of responsible AI frameworks. In taking on this investigation, we wanted to look ahead and understand the thorny reality of building a fair AI tool that makes consequential decisions about people’s lives.
The code and data underlying our analysis can be found on our Github.
How the model works The City of Amsterdam’s model aimed to identify applications for further investigation. Staff members check for applications that contain mistakes and redirect any concerning applications to an investigator. The model is designed to replicate this first screening. A flagged application is not automatically rejected. However, investigators are empowered to request a beneficiary’s bank statements, summon them for meetings, and even visit their homes. Past reporting, including our own, has shown how these investigations can be a stressful or even traumatising experience. The model uses a type of algorithm called an Explainable Boosting Machine (EBM). EBMs prioritize explainability — the degree to which an AI algorithm is understandable instead of being a confusing “black box” of abstract mathematical processes. Amsterdam’s EBM model predicts ‘investigation worthiness’ based on variables that measure welfare applicants’ behavior and characteristics, features. Model predictions are based on 15 features. None of these features explicitly referred to an applicant’s gender or racial background, as well as other demographic characteristics protected by anti-discrimination law. But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies. The model calculates a risk score for each application using these features. The risk scores suggest whether the applicant merits further investigation or not. Applicants with a risk score above 0.56 were redirected for further investigation.
... continue reading