Current AI evaluation systems collapse individual human judgment into a single consensus score, potentially missing the diverse perspectives that should guide AI alignment. A new research framework called PersonaJudge demonstrates that large language models can be trained to simulate how specific individual evaluators would judge AI outputs, achieving up to 9.9 percentage point improvements over standard consensus-based approaches.

What's Wrong With How We Currently Evaluate AI?

When AI companies train systems like ChatGPT or Claude, they rely on human feedback to guide the model's behavior. This process, known as Reinforcement Learning from Human Feedback (RLHF), typically works by collecting judgments from multiple human evaluators and then averaging their preferences into a single "correct" answer. The assumption is that consensus represents truth. But this approach has a fundamental flaw: it treats disagreement as noise rather than signal.

In open-ended evaluation tasks, where reasonable people might apply different criteria to the same output, averaging away individual perspectives can obscure meaningful differences in how people judge quality, helpfulness, or safety. A researcher studying medical AI might prioritize accuracy, while a clinician might prioritize clarity for patient communication. A consensus score would split the difference, satisfying neither.

This matters because alignment researchers increasingly recognize that a single shared standard may not exist for many AI evaluation tasks. When annotators apply different criteria to the same outputs, disagreement is often meaningful rather than a sign of poor evaluation design.

How Can AI Systems Learn Individual Evaluator Preferences?

PersonaJudge proposes a different approach: instead of training AI judges to match crowd consensus, train them to simulate how specific individual evaluators would judge new cases. The framework uses three types of data about each evaluator:

• Categorical Judgments: The evaluator's final decision on whether one AI output is better than another.
• Reasoning Traces: Retrospective explanations of why the evaluator made their decision, capturing the thought process behind the judgment.
• Interface Telemetry: Data about how evaluators inspect tasks, including which parts of the output they examined and how long they spent on each section.

Researchers tested this framework with data from 32 trained annotators who made 4,200 preference judgments across multiple evaluation tasks. The results revealed important insights about what actually helps AI systems learn individual preferences.

Which Data Signals Matter Most for Personalized Evaluation?

The study found that retrospective reasoning traces provided the largest performance gains, improving simulation accuracy by up to 9.9 percentage points over a baseline model that only saw final judgments. When evaluators explained their reasoning, AI systems could better understand the principles and priorities that guided their decisions.

Surprisingly, interface telemetry often hurt performance despite being cheaper to collect. Knowing how long an evaluator spent on a particular section or which parts they clicked on did not reliably predict their judgment. This suggests that the process of decision-making, as captured in reasoning traces, matters more than the mechanics of how evaluators interact with the interface.

The research also identified a stable predictor of how difficult an evaluator would be to simulate: their tendency to use neutral judgments. Evaluators who frequently selected "neutral" or "unclear" options were harder to simulate across different tasks, with this trait showing a correlation of 0.728 across tasks. This neutral-usage tendency proved to be a cross-task-stable property, meaning it predicted simulatability even when evaluators moved between different evaluation domains.

Why Does This Matter for AI Alignment?

The implications extend beyond evaluation methodology. Current alignment approaches like Constitutional AI and RLHF both aim to learn a single, static preference function that applies universally. But if individual evaluators genuinely have different criteria, then training AI systems against averaged preferences may be training them against a target that no actual human would endorse.

PersonaJudge opens the door to per-person reward modeling, where AI systems could be trained to satisfy different stakeholder preferences rather than collapsing them into consensus. This could support evaluation auditing, where researchers verify that AI systems fairly represent the perspectives of different groups, and fairness analysis of whose viewpoints evaluation pipelines underrepresent.

The research also highlights a broader challenge in AI evaluation: current LLM-as-Judge pipelines, which use large language models to replace human evaluators, typically target aggregate signals and miss group- and person-level differences. Even when an AI judge matches average human ratings, it may fail to faithfully simulate any individual evaluator's perspective.

Steps to Implement Individual-Aware AI Evaluation

• Collect Reasoning Traces: Ask evaluators to explain their judgments after making them, capturing the principles and priorities that guided their decisions rather than relying solely on final labels.
• Build Evaluator Profiles: Identify stable traits like neutral-usage tendency that predict how difficult an evaluator will be to simulate, allowing teams to focus data collection efforts on the most informative evaluators.
• Test Personalized Reward Models: Train separate reward models for different evaluators or stakeholder groups rather than averaging preferences, then evaluate whether AI systems trained on these diverse signals better serve their intended users.
• Audit Representation: Analyze whose perspectives your evaluation pipeline represents and whose it underrepresents, using individual-aware methods to ensure fairness across different evaluator groups.

The PersonaJudge framework represents a shift in how alignment researchers think about human feedback. Rather than treating human judgment as a noisy signal to be aggregated into consensus, it treats individual evaluators as having legitimate, learnable preferences that deserve representation in AI training pipelines. As AI systems become more consequential, ensuring they can learn and respect diverse human values may be as important as ensuring they learn any single value at all.