A new approach combines large language models (LLMs) with human oversight to analyze customer feedback at scale, helping public sector organizations like tax agencies detect emerging service quality gaps that may signal underlying bias or unfair treatment across different demographic groups. Rather than waiting for complaints to pile up or relying on slow manual reviews, this methodology flags patterns in how different populations experience government services, offering a data-driven foundation for building more equitable systems.

Why Public Sector Organizations Are Struggling to Spot Service Disparities?

Public trust in government services depends on fair, consistent treatment. Yet many tax administrations and other public agencies still rely on outdated methods to understand customer concerns. In Canada, the overall Client Satisfaction Index score for tax services sits at 63 out of 100, reflecting moderate satisfaction with room for improvement. In the United States, citizen satisfaction with federal government services reached 69.7 out of 100 in 2024, the highest level since 2017, but still leaving significant gaps.

The core problem is that traditional feedback analysis methods fall short. Manual processes are slow, resource-intensive, and prone to human error. They struggle to capture the complexity of concerns across diverse populations, especially when feedback arrives in multiple languages. As feedback volumes grow, nuanced patterns, such as disparities in service delivery between demographic segments, often get missed entirely.

How Can AI Help Detect Hidden Service Gaps?

Researchers at a public sector organization developed a novel methodology that integrates three key components:

• Fine-tuned Language Models: Large language models optimized specifically for analyzing customer feedback in organizational contexts, reducing computational demands while maintaining accuracy.
• Statistical Analysis: Quantitative techniques that identify significant differences in topic frequency across demographic groups, revealing where service experiences diverge.
• Human-in-the-Loop Oversight: Tax officers and domain experts review and validate AI findings, preventing false conclusions and ensuring outputs align with real-world service realities.

The approach focuses on detecting emerging topics in customer feedback, then examining whether certain topics appear more frequently in feedback from specific demographic groups. Significant differences can signal potential disparities in how services are experienced, which may indicate systemic issues or unintended biases in service delivery.

The research team validated their methodology using similarity analysis and evaluation surveys conducted by tax officers with direct expertise in service feedback operations. The results demonstrated improved alignment with expert assessments compared to baseline models, suggesting the approach captures real patterns that human experts recognize as meaningful.

What Makes This Approach Different From Traditional Bias Detection?

This methodology does not directly detect bias in the traditional sense. Instead, it identifies emerging trends and disparities in how different demographic groups experience services. The logic is straightforward: if one group consistently reports problems with appointment scheduling while another reports issues with documentation clarity, those patterns suggest the organization may be serving populations unequally, even if no intentional discrimination exists.

This indirect approach has practical advantages. Rather than requiring organizations to define bias in advance, the system discovers what customers actually care about and flags when those concerns cluster by demographic group. This data-driven foundation supports fairer, more responsive decision-making without requiring organizations to make assumptions about where problems might exist.

The integration of human expertise is critical. LLMs can sometimes generate plausible-sounding but inaccurate outputs, a phenomenon researchers call "fabrication." By embedding tax officers and service experts into the analysis loop, the methodology mitigates this risk and ensures that AI findings reflect genuine patterns in customer experience rather than statistical artifacts.

Steps to Implement AI-Driven Service Equity Analysis in Your Organization

• Audit Your Feedback Data: Gather and organize customer feedback across channels, ensuring demographic information is captured consistently and that feedback spans diverse customer segments and time periods.
• Fine-Tune Models for Your Context: Rather than using generic AI tools, customize language models to your organization's specific vocabulary, service types, and operational context to improve accuracy and relevance.
• Establish Expert Review Processes: Assign domain experts, frontline staff, and service leaders to validate AI findings, ensuring recommendations reflect real service gaps rather than statistical noise.
• Monitor Disparities Over Time: Use the system to track whether topic patterns shift across demographic groups, allowing your organization to measure progress toward more equitable service delivery.

The research highlights a broader shift in how public sector organizations approach fairness. Rather than treating bias detection as a one-time compliance exercise, this methodology treats it as an ongoing process of listening to customer feedback and responding to emerging patterns. As feedback volumes increase and service demands grow more complex, automated systems paired with human judgment offer a scalable way to maintain equity and public trust.

For tax administrations and other government agencies, the stakes are high. Public satisfaction depends not just on efficient service, but on fair treatment across all demographic groups. By combining advanced AI tools with statistical rigor and human expertise, organizations can move beyond reactive complaint handling to proactive identification of service gaps, supporting the development of more equitable systems that strengthen public confidence in government institutions.