Researchers have discovered a significant gap between what privacy-protected synthetic text promises and what it actually delivers in practice. A new benchmark called ContinuousBench shows that even state-of-the-art methods for creating synthetic data from sensitive information fail to preserve the valuable knowledge contained in the original data, raising questions about whether this approach can truly unlock restricted datasets for AI training.

What Is Differentially Private Synthetic Text, and Why Does It Matter?

As publicly available text becomes depleted as a training resource for improving AI models, much of the remaining high-value data sits locked away in sensitive or proprietary sources like clinical notes and user data. Differentially private (DP) text synthesis offers a potential solution: it creates synthetic versions of sensitive data that preserve privacy while allowing researchers to train AI models on the information. The core question, however, is whether this synthetic data actually transfers the knowledge and capabilities that access to the original restricted corpus would have provided.

The problem is that existing evaluations of DP synthetic text have relied on tasks that are nearly solvable without any special training data. This means strong performance on standard benchmarks does not prove that DP synthesis can genuinely substitute for access to the original sensitive data. Researchers needed a more rigorous way to test whether synthetic data actually preserves the valuable facts and skills learnable from restricted sources.

How Does ContinuousBench Test Synthetic Data Quality?

To address these limitations, researchers introduced ContinuousBench, a benchmark specifically designed to measure whether DP synthetic text can preserve capability gains from sensitive corpora. The benchmark works by automatically releasing new training data every quarter, paired with question-and-answer tasks that are impossible to solve without access to that specific corpus.

The benchmark includes two tracks to test different types of knowledge. The first, called Geminon, uses procedurally-generated data about fictional creatures. The second, called News, draws from newly crawled public news articles. By using fictional knowledge and post-cutoff news, the benchmark ensures that models cannot rely on knowledge already learned during their initial training.

ContinuousBench also eliminates a common source of confusion in synthetic data research: the distillation effect. When a stronger AI model generates synthetic data for a weaker model to learn from, improvements might come from the strength of the teacher model rather than from the quality of the synthetic data itself. To prevent this, ContinuousBench requires that identical base models be used both for generating the synthetic data and for measuring downstream task improvement.

What Did the Benchmark Reveal About Current Methods?

The results were sobering for the field. Although standard benchmarks are nearly saturated and show little room for improvement, ContinuousBench revealed a stark difference between synthetic and real data. Non-private synthesis, which does not add privacy protections, transferred substantial knowledge from the original corpus to trained models. In contrast, state-of-the-art DP synthesis methods generally failed to do so, even at epsilon equals 100, a privacy parameter that represents relatively weak privacy protections.

This finding suggests that the privacy protections added to make synthetic data safe for sharing come at a significant cost to the utility of that data. The gap between what DP synthesis can preserve and what researchers hoped it could preserve remains substantial, even when privacy budgets are relatively generous.

Why Do Existing Benchmarks Miss This Problem?

The research identifies three key reasons why earlier evaluations failed to catch this issue. Understanding these limitations helps explain why the field has been overly optimistic about DP synthetic data:

• Knowledge Already Present: Existing benchmarks often test knowledge that modern pretrained language models already possess from their initial training on public data, making it impossible to tell whether synthetic data actually transferred new information or simply elicited knowledge already learned.
• Surface-Level Matching: Common evaluation metrics like MAUVE score and next-token prediction accuracy can improve through style matching alone, without requiring recovery of specific facts or skills from the sensitive corpus, conflating general linguistic competence with genuine information transfer.
• Teacher-Student Confusion: When synthetic data is generated by a stronger model and consumed by a weaker one, improvements may come from ordinary distillation from a more capable teacher rather than from the private corpus itself, making it impossible to assess whether sensitive data unlocked via DP synthesis can improve the best models available today.

How Can Researchers Improve Synthetic Data Methods?

ContinuousBench is now publicly available to help researchers measure and accelerate progress on DP synthetic data. The benchmark's design ensures that future evaluations will test whether DP synthesis methods can genuinely preserve the high-value facts and skills learnable from sensitive corpora, rather than simply generating stylistically similar text.

By establishing a continuously regenerated benchmark with quarterly releases of new, never-before-seen training corpora, the research community now has a rigorous framework for testing whether DP synthesis can truly unlock sensitive datasets. The benchmark's focus on short-answer factual question-answering tasks, similar to widely used knowledge benchmarks, ensures that improvements reflect genuine capability gains rather than superficial pattern matching.

The implications are significant for enterprises and researchers who hoped to use DP synthetic data as a way to share and train on sensitive information without compromising privacy. While the approach remains promising in theory, current methods fall short of delivering on that promise. The gap between privacy protection and data utility remains a fundamental challenge that the field must address before DP synthesis can serve as a reliable substitute for direct access to sensitive corpora.