Every day, algorithms make quiet decisions that shape people’s lives. It could be who gets called back for a job interview, who is approved for a loan, or who is deemed likely to become a repeat offender. The introduction of artificial intelligence (AI) into these processes has often resulted in mirroring existing societal prejudices rather than supporting more effective and fair decision-making solutions.
“In job recruitment, for example, AI tools have been shown to systematically disadvantage women or ethnic minorities because the model learned from historically biased hiring patterns,” said Jing Li, a Research Scientist at the A*STAR Centre for Frontier AI Research (A*STAR CFAR).
To check whether models are treating different groups fairly, one would need to compare decision outcomes against demographic information like gender and race. However, such sensitive data is typically limited in real-world datasets due to critical privacy measures.
“You know bias might exist, but you can’t directly measure or correct it because you don’t know who belongs to which group,” said Xiuju Fu, a Senior Principal Scientist at the A*STAR Institute of High Performance Computing (IHPC). “It is precisely because this demographic data is unavailable that we need a new approach that can protect vulnerable groups without ever identifying who they are.”
Li, Fu, A*STAR CFAR Director Ivor Tsang and team turned to a framework inspired by the philosophy of John Rawls. Rawls proposed that, when societal group memberships are unknown, the natural response is to design a system that prioritises the most disadvantaged.
Translated into machine learning, the question shifts to asking whether the worst-off group is adequately protected. Such an approach also only requires a statistical estimate of the size of the most vulnerable group, rather than needing individual-level demographic data.
The researchers implemented a method called intrinsic reweighting (IRW), which assigns each training sample a certain importance, or weight, based on their expected impact on the worst-performing group. “We compute the weight using gradient information, measuring how each sample’s learning signal aligns with what the model needs to improve fairness. This gives a more accurate picture of each sample’s contribution to how the model learns,” explained Li.
When tested across four standard fairness benchmarks, the IRW framework consistently matched or outperformed existing fairness methods. The resulting models even showed comparable results to those that had access to demographic labels during training.
“Integrating this approach into standard AI development would make it much more accessible to those who are not fairness researchers themselves,” said Tsang. To advance the IRW framework towards adoption, the researchers are focusing on reducing its computational demands and improving scalability. They also hope to work more closely with policymakers to derive more accurate measures of worst-off cases within real-world contexts and privacy measures.
The A*STAR-affiliated researchers contributing to this research are from the A*STAR Centre for Frontier AI Research (A*STAR CFAR) and the A*STAR Institute of High Performance Computing (A*STAR IHPC).
