A new machine learning technique can reduce algorithmic bias and improve fairness for vulnerable populations even when their identities remain completely unknown to the system. Researchers at Singapore's A*STAR institutes have created a method called intrinsic reweighting (IRW) that protects disadvantaged groups during AI model training without requiring access to sensitive personal information like race, gender, or ethnicity.

Why Does AI Keep Repeating Historical Bias?

Algorithms now make consequential decisions that shape people's lives every single day. These systems determine who gets called back for a job interview, who qualifies for a loan, and even who is flagged as a potential repeat offender. The problem is that artificial intelligence often mirrors the prejudices already embedded in historical data rather than correcting them.

"In job recruitment, for example, AI tools have been shown to systematically disadvantage women or ethnic minorities because the model learned from historically biased hiring patterns," said Jing Li, a Research Scientist at the A*STAR Centre for Frontier AI Research.

Jing Li, Research Scientist at A*STAR Centre for Frontier AI Research

To check whether AI models treat different groups fairly, researchers typically need to compare decision outcomes against demographic information. However, this sensitive data is rarely available in real-world datasets due to privacy protections and regulations. This creates a catch-22: organizations know bias might exist, but they cannot directly measure or correct it without the demographic labels they're not allowed to collect.

How Can You Fix Bias You Can't Measure?

The A*STAR team, led by Research Scientist Jing Li, Senior Principal Scientist Xiuju Fu, and A*STAR CFAR Director Ivor Tsang, turned to an unexpected source for inspiration: the political philosophy of John Rawls. Rawls proposed that when group memberships are unknown, the fairest approach is to design systems that prioritize protecting the most disadvantaged. The researchers translated this principle into machine learning by asking a simple question: is the worst-off group adequately protected ?

The intrinsic reweighting method works by assigning each training sample a weight based on its expected impact on the worst-performing group. Rather than needing individual demographic labels, the approach only requires a statistical estimate of how large the most vulnerable group is. This fundamental shift makes the technique practical for real-world deployment where sensitive demographic data is restricted.

"We compute the weight using gradient information, measuring how each sample's learning signal aligns with what the model needs to improve fairness. This gives a more accurate picture of each sample's contribution to how the model learns," explained Jing Li.

Jing Li, Research Scientist at A*STAR Centre for Frontier AI Research

Steps to Implement Fair AI Training Without Demographic Data

• Estimate Vulnerable Group Size: Begin by obtaining a statistical estimate of the proportion of the population that belongs to the most disadvantaged group, without identifying individuals.
• Apply Gradient-Based Weighting: Use gradient information from the training process to assign importance weights to each data sample based on how it affects the worst-performing group's outcomes.
• Train with Reweighted Samples: Feed the weighted training data into your standard machine learning pipeline, allowing the model to learn with built-in fairness protections.
• Validate Across Benchmarks: Test the resulting model on standard fairness evaluation datasets to confirm it matches or exceeds the performance of traditional fairness methods.

When tested across four standard fairness benchmarks, the intrinsic reweighting framework consistently matched or outperformed existing fairness methods. Remarkably, the resulting models showed comparable results to those that had direct access to demographic labels during training. This is significant because it means organizations can achieve fairness without the privacy risks and regulatory complications of collecting sensitive personal data.

"You know bias might exist, but you can't directly measure or correct it because you don't know who belongs to which group. It is precisely because this demographic data is unavailable that we need a new approach that can protect vulnerable groups without ever identifying who they are," said Xiuju Fu, a Senior Principal Scientist at the A*STAR Institute of High Performance Computing.

Xiuju Fu, Senior Principal Scientist at A*STAR Institute of High Performance Computing

The research team is now focused on making the approach more practical for widespread adoption. Their immediate priorities include reducing the computational demands of the intrinsic reweighting method and improving its scalability so it can handle larger datasets and more complex models. They are also planning to work more closely with policymakers to develop more accurate measures of worst-off cases within real-world contexts while respecting privacy regulations.

"Integrating this approach into standard AI development would make it much more accessible to those who are not fairness researchers themselves," noted Ivor Tsang, Director of A*STAR Centre for Frontier AI Research.

Ivor Tsang, Director at A*STAR Centre for Frontier AI Research

The implications of this work extend across industries where algorithmic decision-making affects people's opportunities and outcomes. From hiring systems to lending platforms to criminal justice applications, the ability to reduce bias without collecting sensitive demographic data addresses a fundamental tension in modern AI deployment: the need for fairness versus the imperative to protect privacy. By proving that fairness protections can work without demographic labels, this research opens a path forward for organizations struggling to build trustworthy AI systems within privacy-conscious regulatory environments.