A new class of AI models is challenging the assumption that smarter AI requires more thinking time. Instead of generating responses token-by-token like traditional language models, these "System One" models produce structured answers in a single computational pass, achieving response times around 70 milliseconds instead of several seconds. This shift raises a fundamental question: as AI labs race to build reasoning models that think longer, are they missing a more practical path forward ?

What Makes These Fast Models Different From Traditional AI?

Conventional large language models (LLMs), which are AI systems trained on vast amounts of text to generate human-like responses, work autoregressively. This means they generate one token, or word fragment, at a time, requiring a separate computational pass for each piece of output. When you ask an LLM a question, it produces the first word, then the second, then the third, building the response sequentially. This flexibility allows LLMs to handle nearly any task a computer can perform, but it comes at a cost: speed and unpredictability.

The emerging alternative, exemplified by models like Jev, constrains output to structured formats like multiple-choice selections or formatted data objects. Because the model only needs to produce a single token or a predetermined structure per query, it can generate answers in parallel rather than sequentially. A user provides a prompt along with predefined choices, and the model selects from those options in one forward pass, rather than typing out an entire JSON response character by character.

How Do These Models Achieve Such Speed?

The speed advantage stems from a fundamental architectural difference. Traditional LLMs must run a new generation pass for each token they produce, which accumulates latency across long responses. Structured-output models bypass this bottleneck by prefilling responses with a template and generating only the variable portion. Since LLMs already process all input tokens in parallel during the initial phase, multiple queries can be batched into a single forward pass, enabling dramatic speed improvements.

The performance gains are substantial. Response times for structured-output models range from around 70 milliseconds at the fastest to 500 milliseconds at the slowest, compared to several seconds for traditional LLMs. To illustrate the practical impact, one researcher noted that these models can even play real-time video games like Doom by processing game state descriptions and selecting actions from predefined choices fast enough to maintain playable frame rates.

What Are the Trade-Offs Between Speed and Intelligence?

The speed advantage comes with a significant limitation: these models cannot leverage test-time compute, the practice of allocating additional computational resources during inference to improve reasoning quality. This architectural constraint likely caps the intelligence ceiling of fast structured-output models at the level of non-reasoning LLMs, meaning they may not match the problem-solving depth of frontier models designed for extended reasoning.

Additionally, while proponents claim these models are immune to hallucinations, the reality is more nuanced. A model can still select an incorrect choice from user-provided options, which technically differs from inventing false information but produces the same practical error. This limitation mirrors what already exists with traditional LLMs using structured output, so it does not represent a meaningful reliability improvement.

How to Implement Fast Structured Output in AI Systems

• Prefill with Templates: Design your structured output by prefilling the response with a template structure and generating only the variable portion, reducing the number of tokens the model must produce per query.
• Constrain Choices: Provide users with predefined options rather than allowing free-form generation, enabling the model to select from a limited set in a single forward pass.
• Batch Multiple Queries: Combine multiple structured-output requests into a single computational pass using standard inference batching, allowing the model to process many queries simultaneously.
• Optimize for Latency-Sensitive Tasks: Deploy these models for applications where response time matters more than reasoning depth, such as real-time decision-making, interactive systems, or high-throughput APIs.

Is This a Genuinely New Technology or an Existing Technique?

The technical novelty of fast structured-output models remains debatable. Researchers have observed that existing LLMs can already achieve similar speed improvements through grammar-constrained decoding, a technique that restricts token generation to valid structured-output formats. The key insight is that if you only need to generate one token per query from a limited set of choices, you do not strictly need a specialized model; you can simply constrain a standard LLM's output.

In fact, developers have already begun experimenting with this approach following recent announcements in this space, and early results suggest the technique works reasonably well. This raises the possibility that specialized structured-output models may lack a substantial technical advantage, and that their claimed innovations, such as "Reinforcement Learning for Calibrated Decisions," may not represent a fundamentally new scaling axis for AI.

However, models specifically fine-tuned for structured output may still outperform generic LLMs retrofitted with constrained decoding. The ability to optimize training data and model parameters exclusively for structured tasks could provide a meaningful edge, even if the underlying technique is not revolutionary.

What Does This Mean for the Future of AI Inference?

The emergence of fast structured-output models highlights a divergence in how AI labs are approaching the inference problem. While some organizations invest heavily in reasoning models that allocate more compute at test time, others are exploring whether constraining output format and sacrificing reasoning depth might unlock new use cases. As one observer noted, fast software does not just enable existing tasks to run quicker; it enables entirely new kinds of tasks that were previously impractical.

The question of whether major AI labs will release their own optimized versions of fast structured-output models remains open. If they do, products built on top of these systems could look fundamentally different from today's chatbot-centric AI applications, potentially enabling real-time decision-making systems, interactive tools, and embedded intelligence at scales currently impractical with autoregressive generation.