Perplexity's engineering team revealed that embedding models, which power search and retrieval systems, require fundamentally different serving infrastructure than the chat-based AI systems most people interact with daily. On September 4, 2026, the company published "Fast Embeddings on GPUs," a technical breakdown of the serving infrastructure behind their embedding and ranking models used across their Search, Computer, and API Platform products, including their own pplx-embed model.

The core insight for builders and engineers is counterintuitive: Perplexity doesn't run a separate embedding-serving stack at all. Instead, they reuse their large language model (LLM) inference stack almost entirely intact, because the compute-heavy part of a forward pass,the dense layers that perform matrix multiplications,is identical whether the system is generating a token or producing an embedding vector.

Why Do Embedding Models Need Different Infrastructure Than Chat AI?

The practical reason embedding serving differs from traditional LLM (large language model) chat serving comes down to two opposing traffic patterns that pull infrastructure in opposite directions. Batch embedding handles bulk indexing or reindexing of large document collections, prioritizing throughput over latency. Online embedding handles per-query lookups at request time, the kind that fires every time a user types a search, prioritizing latency over throughput.

A generic inference server tuned for one of these shapes tends to underperform on the other. This explains why a retrieval-augmented generation (RAG) pipeline,a system that retrieves relevant documents before generating answers,might work smoothly during a bulk reindex but feel sluggish on live queries, or vice versa. The two workloads have opposite optimization targets even though they run the same model.

How Does Perplexity's Three-Layer Architecture Handle Both Workloads?

Perplexity's serving stack splits cleanly by responsibility, with each layer built in a different programming language chosen for what it does best:

• Ivy (Rust HTTP gateway): Handles all CPU-side request work including JSON parsing, tokenization, input templating, and batch splitting before translating requests downstream. Ivy load-balances by splitting large-batch requests across replicas to avoid imbalance when one caller sends a disproportionately large batch, and runs Perplexity's own in-house unigram tokenizer, which the team says "drastically improves latencies" over off-the-shelf tokenizers.
• Tulip (Rust gRPC scheduler): Sits between Ivy and the model engine, built on tokio and tonic libraries, and owns request scheduling and batching. Its scheduler is deliberately simple,first-come-first-served,because latency is dominated by total token count, not sequence count, meaning once a batch hits roughly 512 tokens on a sub-1 billion parameter model, the GPU is already saturated.
• ROSE (Python model engine): Implements the actual model forward passes, kernels, and layers. Originally built for LLM serving, it was extended to embeddings by sharing nearly all of that code, with the difference being ragged and unpadded attention instead of paged key-value cache attention.

The architectural decision to reuse LLM inference kernels represents a real cost-saving pattern worth copying for teams building their own infrastructure. If an organization already operates an LLM inference stack, extending it to serve an embedding model is closer to swapping an attention backend than building a parallel system.

What GPU Optimizations Make the Biggest Performance Difference?

Two GPU-level optimizations emerged as the most transferable lessons from Perplexity's work. The first is whole-model CUDA graph capture, which addresses a fundamental CPU bottleneck. Launching kernels one at a time means the CPU must issue a launch instruction for every single operation in a forward pass, and at small batch sizes this CPU-side launch overhead can dominate total latency more than the GPU compute itself.

A CUDA graph captures an entire forward pass as one launchable unit, so the CPU issues a single "replay this graph" call instead of hundreds of individual kernel launches. Perplexity found the inflection point where GPU cost starts to outweigh CPU launch overhead comes at thousands of tokens or tens of sequences for these small models, which is precisely why they capture the whole model as one graph rather than partial subgraphs.

Perplexity also uses lazy graph capture, where the first call for a given input shape runs eager (uncaptured) as a warmup pass, and the second call captures the graph and replays it from then on. This spreads a multi-minute capture cost across hours of serving instead of paying it all up front at startup, a deliberate tradeoff of slightly worse performance on a cold start in exchange for much faster fleet-wide startup and scaling.

The second optimization is a Rust-side async abstraction called LazyTensor. Instead of a step() call blocking until a GPU result is ready, LazyTensor tracks a pending result via page-locked host memory, an asynchronous memory copy call, and a CUDA event, then returns a future-like handle immediately. The practical effect is that Tulip can kick off the next batch's GPU work while still waiting on the CPU-side copy of the previous batch's result, overlapping CPU and GPU work to improve both latency and throughput simultaneously.

How Fast Does Perplexity Respond to Search Queries Now?

Beyond the infrastructure details, Perplexity's embedding serving speed has real implications for how quickly the platform can surface citations in search results. According to a 2026 State of AEO (Answer Engine Optimization) Report from Trustpoint Xposure, Perplexity retrieves from live web sources at query time and shows the fastest response to certified AEO implementation among major AI search platforms.

Schema markup and genuine editorial placements in recognized publications produce measurable Perplexity citation improvements within days to weeks of correct deployment, making Perplexity the platform most directly responsive to the schema markup and editorial coverage components of search optimization methodology. This speed advantage stems directly from the infrastructure Perplexity published on September 4, which enables the platform to embed and rank documents quickly enough to surface fresh web results in real time.

For builders and companies optimizing for AI search visibility, the technical lesson is clear: embedding serving is not a solved problem that can be handled by generic inference servers. The infrastructure choices Perplexity made,reusing LLM kernels, capturing GPU graphs, and overlapping CPU and GPU work,represent the kinds of optimizations that separate platforms that feel responsive from platforms that feel sluggish. As more companies build retrieval systems and compete for visibility across AI search platforms, these infrastructure patterns will likely become industry standard.