Alignment researchers have historically focused on training AI models in English, leaving billions of speakers of other languages with fewer safeguards against harmful outputs. A new dataset released by Mozilla Data Collective and PROXIMA aims to change that by making alignment training data available in Urdu, one of the world's most widely spoken languages.

Why Does AI Alignment Matter for Non-English Languages?

When AI companies train large language models, or LLMs (AI systems that understand and generate human language), they use a technique called RLHF, which stands for reinforcement learning from human feedback. RLHF works by showing the model pairs of responses to the same question, one marked as "good" and one marked as "bad," so the model learns to prefer helpful, harmless outputs. The problem is that most RLHF training data has been created in English, meaning models trained on non-English languages often lack the same safety guardrails.

Urdu, spoken by roughly 230 million people worldwide, had no publicly available RLHF dataset until now. This gap meant that developers building Urdu-language AI models had to either skip alignment training entirely or rely on lower-quality workarounds. The new dataset changes that equation by providing a foundation for building safer, more reliable AI systems in a language spoken across Pakistan, India, and diaspora communities globally.

What Exactly Is in This New Dataset?

The Helpful and Harmless RLHF Urdu dataset contains 11,225 preference pairs of multi-turn conversations, each showing a preferred response and a rejected response to the same user query. The dataset was created by translating Anthropic's original Helpful and Harmless RLHF dataset, a foundational resource in AI alignment research, into Urdu using a specialized translation model called TranslateGemma27B, a 27-billion-parameter instruction-tuned system designed for high-quality multilingual translation.

The translation process preserved the original structure of each conversation, maintaining the distinction between user and assistant roles while ensuring the Urdu text reads naturally and fluently. The dataset is formatted as JSONL (a machine-readable text format), totals 44.4 megabytes, and includes scoring information for both the preferred and rejected responses, giving model trainers granular feedback signals.

How to Use This Dataset for AI Model Training

• Reward Model Training: Developers can use the preference pairs to train reward models, specialized AI systems that learn to score responses on a helpfulness and harmlessness scale, which then guide the training of larger language models.
• Direct Fine-Tuning: The dataset supports RLHF-based fine-tuning, a process where an already-trained language model is adjusted using the preference data to improve its alignment with human values and safety standards.
• Constitutional AI Alignment: The dataset extends Anthropic's Constitutional AI framework, a method that uses a set of principles (a "constitution") to guide AI behavior, to Urdu-language models, enabling developers to build systems that follow consistent ethical guidelines.

One model has already been trained using this dataset: mahwizzzz/Qalb-1.0-8B-DPO, an 8-billion-parameter text generation model that uses a technique called DPO (direct preference optimization) to incorporate the alignment training.

What Are the Licensing and Practical Constraints?

The dataset is released under a Creative Commons CC-BY-NC-SA-4.0 license, which means it is free to use for non-commercial research and development, but commercial use requires written permission from the stewards. This licensing choice reflects a deliberate effort to keep alignment research open and accessible to academic institutions and non-profit organizations while maintaining some control over commercial applications.

The license also includes a specific restriction: the dataset cannot be used to train models intended to bypass or contradict the original helpfulness and harmlessness alignment goals. This safeguard ensures that the resource is used to improve AI safety, not to circumvent it. Any derivative works must be released under the same license, creating a chain of accountability for downstream uses.

Why Does This Matter for the Future of AI Safety?

The release of this Urdu dataset represents a significant step toward democratizing AI alignment research. For years, alignment work has been concentrated in English-speaking institutions and companies, creating an implicit bias in which languages and cultures receive the most rigorous safety training. By extending Anthropic's framework to Urdu, researchers are beginning to address what experts call the "alignment gap" in low-resource languages.

This matters because AI models trained without proper alignment can produce harmful outputs, spread misinformation, or reinforce biases at scale. When billions of people use AI systems in their native languages, those systems should be held to the same safety standards as English-language models. The Urdu dataset is a proof of concept that this is possible, and it opens the door for similar efforts in other languages spoken by large populations but underrepresented in AI research.

The work also highlights a broader trend in AI development: as language models become more global, the infrastructure for training them safely must become global too. Researchers and organizations are beginning to recognize that alignment is not a luxury feature for wealthy markets, but a foundational requirement for responsible AI deployment everywhere.