Researchers have developed a new method to identify which large language model (LLM) works best for generating synthetic training data, even when human test sets don't exist. The technique, called RoSE (Round-robin Synthetic data Evaluation), solves a critical problem in natural language processing (NLP): selecting the right AI model to create artificial text for training smaller, specialized systems, particularly for languages with limited human-labeled data.

Why Is Picking the Right LLM Generator So Difficult?

Large language models like GPT-4 and Llama have become powerful tools for generating synthetic text that can train smaller, more efficient downstream models. However, different LLMs produce vastly different quality outputs, and figuring out which one works best has been a major headache for researchers. The traditional approach requires human experts to manually evaluate generated text, which is expensive, time-consuming, and often impossible in low-resource language settings where labeled data is scarce.

Existing evaluation methods fall into two categories: intrinsic metrics, which measure properties of the generated text itself without training a model, and extrinsic metrics, which require human-labeled test data to validate results. The problem is that intrinsic metrics don't reliably predict how well synthetic data will actually perform when used to train real models. Prior research shows these correlations are weak or inconsistent, even for English text.

How Does RoSE Work Without Human Test Data?

RoSE takes a clever cross-evaluation approach. Instead of relying on human annotations, the method trains a small model on synthetic text generated by one LLM candidate, then tests that model on synthetic examples created by all the other candidate LLMs. The performance across these cross-evaluations reveals which generator produces the most generalizable and useful training signal. This process repeats multiple times, and the LLM with the highest average performance is selected as the best generator.

The intuition behind RoSE is straightforward: synthetic data carries the signature of its generator, and different LLMs produce data with varying quality and coverage. By having models trained on one LLM's output evaluate on another's, researchers can identify which generator provides the most robust training foundation without needing expensive human validation.

What Results Did the Research Show?

Researchers tested RoSE across six different LLMs, eleven typologically diverse languages (including Welsh, Romanian, and Azerbaijani), and three natural language processing tasks: sentiment analysis, topic classification, and intent detection. The results were compelling. RoSE identified the optimal LLM generator more consistently than any other proxy metric tested. When a small model was trained on data selected by RoSE and evaluated on human test data, it achieved an average performance gap of only 0.76 percentage points compared to the optimal human-performance-based selection. The second-best proxy metric had a gap of 2.52 percentage points.

Additionally, RoSE was the only proxy metric that consistently showed a positive correlation between classifier performance and human evaluation results. It ranked best across 9 of 11 languages tested and second-best in the remaining 2.

Steps to Implement RoSE for LLM Selection

• Gather Candidate LLMs: Assemble a set of LLMs you want to evaluate as synthetic data generators, such as GPT-4, Llama, or other open-weight models of varying sizes and families.
• Generate Synthetic Text: Use each candidate LLM to generate synthetic text for your specific task and language combination, leveraging a small amount of human examples (approximately 10 per label) to guide generation.
• Train and Cross-Evaluate: Train a smaller downstream model on synthetic data from one LLM generator, then evaluate its performance on synthetic test sets created by all other candidate generators.
• Calculate Mean Performance: Compute the average performance across all cross-evaluations to determine each LLM's RoSE score, repeating this process 10 times for statistical reliability.
• Select the Best Generator: Choose the LLM with the highest mean RoSE score as your synthetic data generator for downstream model training.

The research team noted that RoSE remains effective even when comparing as few as three LLMs, and it performs strongly when comparing models of similar parameter size. The method also works well regardless of how many LLMs are included in the comparison.

Why Does This Matter for Natural Language Processing?

This breakthrough addresses a real bottleneck in NLP development. For many low-resource languages, human-labeled test sets either don't exist or contain only a handful of examples. Without a reliable way to select the best LLM generator, researchers have been forced to rely on intrinsic heuristics that don't correlate well with actual performance. RoSE provides a practical, cost-effective alternative that works without human validation data.

The implications extend beyond academic research. Organizations building NLP systems for underrepresented languages can now confidently select which LLM to use for generating training data, accelerating development timelines and reducing costs. The method is particularly valuable for sentiment analysis, topic classification, and intent detection tasks, which are foundational to chatbots, customer service systems, and voice assistants.

The research team has released all code, data, and results publicly, making RoSE accessible to the broader NLP community. This openness should accelerate adoption and enable researchers worldwide to apply the method to their own language-task combinations.