Socio-economy & New Tech

    Artificial Intelligence

Joint Research Initiative


Fairness in AI: ending bias and discrimination

There is a popular phrase in computer science that says: garbage in, garbage out. It means that when we feed machines poor-quality or biased data; it reflects on the output. In perhaps the most notorious case of AI prejudice, a report by investigative journalism organization ProPublica claimed that an AI tool used by the US justice system (the COMPAS recidivism algorithm) discriminated against black prisoners. Even though these accusations are based upon a very specific and narrow interpretation of fairness, they give substance to a worry that has been brewing among computer scientists for years: that the algorithms continuously used to profile us in contemporary society may be inherently discriminatory. In fact, tech giants such as Google and Microsoft are taking the problem very seriously and have both taken steps to investigate. “Fairness-aware machine learning is a pressing issue and a very hot topic at the moment”, confirms professor Toon Calders, an expert in data mining at the University of Antwerp. Given the increasingly important role machine-learning plays in the insurance industry, the researcher has undertaken an ambitious 3-year collaborative project with practitioners from AXA to tackle the problem within the operational context of insurance. Called the AXA Joint Research Initiative (JRI) on measuring fairness of predictive models, the research program has two specific objectives: to develop measures of fairness that are useful in the context of AXA insurance and to use them to assess fairness of existing decision procedures and models generated by state-of-the-art machine learning methods.
Insurance companies increasingly rely on decision-making procedures that are, at least partially, automated, and often based on data mining (the process of finding patterns within large data sets, to transform it into useful information). They are not the only ones, as prof. Toon Calders points out. Banks have profiles to divide up people according to credit risk, web corporations profile users according to their interests and preferences based on web activity and visitation patterns, etc. “You would think that these methods of profiling, based on facts, would be fairer than a person’s judgement, wouldn’t you? It turns out, it is not necessarily the case”. One major reason for this, he explains, is that bias can be embedded in the data itself. In other words, if the information the algorithms learn from contains biases, the machine is likely going to reproduce them. For example: “in the historical data that it is fed, the machine might find that the words robbery and violence are more often associated with black people than with white people. Similarly, words like intelligence or diligence might be more often affiliated with men than women”. So how do you rectify this disturbance?

Garbage in, garbage out: how to ensure fairness-aware machine learning?

When measuring fairness, a natural preliminary question to ask is how to define it. Building upon previous research (Friedler et al. 2016), prof. Calders works on the assumption that giving an exact and generic definition of fairness is an impossible task, and that consequently, measures of fairness should be situation dependent. “We consider that constructing a definition of what is fairness in AXA’s operational context is part of the project itself, he explains. Once this specific definition is constructed, the project will pursue its second objective: find methods to assess the level of fairness of AXA’s existing decision procedures. “One simple approach could be to assume that men and women should have equal access to low insurance premiums. Then we would just have to compare percentages. However, this approach does not work in this case, because a correlation between gender and accidents has been proven. In other words, it wouldn’t be fair.” The approach that will be adopted is slightly different. “A better way is to look at people with a similar level of premium, say high, and see how many of them were involved in actual accidents. If your prediction is correct, you would expect to see high probability in this group. Now, if you split these high premiums into genders, and see that the number of accidents is much higher for men than women, you will be able to tell that the system is biased”. For ethnicity, the problem becomes yet more complicated. Indeed, gender is a characteristic that is stored in insurance data sets, ethnicity, on the other hand, isn’t. “That’s a crucial challenge we want to deal with in this project. How do we assess whether an insurance discriminates on ethnicity, if we, ourselves, can’t make out the difference in the input data? For all we know, the algorithms could be discriminating on names, schools, neighbourhoods, but that is much harder to examine. Our solution is to create some kind of synthetic population, with artificial profiles, and then to run the algorithms as if they were real”.

Using this approach, the project ultimately aims is to obtain a set of compatible and effective measures that reflect the type of fairness that AXA wants, while at the same time, obeying increasingly severe data protection laws. In particular, the European Union has one of the strongest anti-discrimination legislations. In fact, the recent General Data Protection Regulation (GDPR) explicitly mentions profiling as an activity in which decisions should not be based on personal data and suitable measures should be in place to safeguard the data subject’s rights and freedoms and legitimate interests.

The present JRI effort was precisely initiated in anticipation of the enforcement of such regulations. In the near future, companies will be increasingly asked to answer for their decisions made by algorithmic systems. In this context, it will primordial to have mechanisms in place to continually screen decision prediction and make sure they are not biased. “Historically, research on fairness and machine-learning has stayed very academic. Here, the JRI offers the opportunity to confront the issue with real-life scenarios and cases. This is a big motivational boost for me, says prof. Calders. Not only does it help the research align with reality, but it also makes sure it will have an impact”.



University of Antwerp





ORCID Open Researcher and Contributor ID, a unique and persistent identifier to researchers