The silicon powering artificial intelligence on your phone, car, and smartwatch was built for a job that's rapidly becoming outdated. For the past decade, edge AI chips (processors that run AI directly on devices rather than in the cloud) were engineered to do one thing extremely well: recognize images through convolutional neural networks. But as more complex AI models move from data centers onto consumer devices, those chips are hitting a wall. The problem isn't raw computing power; it's memory, data movement, and how the hardware actually executes modern AI workloads.

Why Peak Computing Power Is No Longer the Whole Story?

For years, the tech industry measured edge AI performance using a single metric: TOPS, or trillion operations per second. A chip with higher TOPS was assumed to be faster. But that assumption breaks down when running vision large language models (LLMs), which are AI systems that combine image understanding with reasoning capabilities. These models fuse perception, semantics, and reasoning in a single pipeline, allowing devices to understand scenes, answer questions about what they see, and determine what to do next.

The bottleneck isn't arithmetic throughput; it's memory traffic and utilization. When a vision LLM runs on a device, three critical problems emerge. First, modern transformer-based AI systems contain billions of parameters (the numerical weights that make the model work), and multimodal systems add visual front ends that convert images or video into tokens for downstream reasoning. This creates a massive weight footprint, substantial activations, and growing key-value state, all of which demand more memory capacity and memory bandwidth. Second, the underlying attention mechanism (a core part of how transformers work) grows roughly quadratically with context, meaning longer prompts and richer visual context can quickly overwhelm an edge device's memory subsystem. Third, vision LLMs combine visual encoders, transformer layers, feed-forward blocks, normalization, and vector operations, all with different shapes and reuse patterns, creating workload irregularity that existing hardware wasn't designed to handle.

"Peak TOPS is becoming a weaker proxy for delivered edge performance. A design that looks strong on synthetic benchmarks may still perform poorly on actual Vision LLM graphs if it cannot maintain locality and utilization as the workload shifts from stage to stage," explained researchers at Expedera, a company focused on edge AI optimization.

Expedera Research Team

How Should Hardware Designers Rethink Edge AI Chips?

• Shift from Layer-by-Layer Execution: Traditional edge AI chips process neural networks one complete layer at a time, which causes activations to spill into external memory more often. New architectures should use packet-based processing, where small, dependency-aware fragments of a neural network move vertically through the graph, allowing intermediate data to be consumed and retired earlier, reducing costly external memory movement.
• Support Hardware-Software Co-Design: Efficient edge deployment requires optimization across three layers: model architecture (using hybrid or distilled variants), system-level scheduling (quantization and tiling methods like FlashAttention), and dedicated hardware support. Hardware and software can no longer be treated as separate deliverables; they must be designed together from the ground up.
• Evaluate Against Real Workloads: Peak TOPS and TOPS per watt should be complemented by workload-specific measures such as sustained utilization, external memory transactions, and tail latency on actual vision LLM graphs. Hardware should be tested against a portfolio that includes legacy convolutional neural networks, transformer-based LLMs, diffusion pipelines, and newer multimodal models.
• Enable Flexible Routing for Heterogeneous Work: Vision LLMs combine multiple computational personalities into a single inference path, starting with visual encoding, moving into multimodal reasoning with attention and feed-forward layers, and ending with output generation. Hardware should route work through specialized Feed Forward, Attention, and Vector blocks rather than forcing every stage to use the same execution model.

What Does This Mean for the Broader Edge AI Market?

The shift toward hybrid AI, where inference increasingly moves to devices while complex processing remains in the cloud, is creating structural demand for more efficient, purpose-built edge processors. According to CEVA, a semiconductor intellectual property company, demand for highly efficient ultra-low power solutions is growing across wearables, automotive, industrial, and smart home applications. As more products require local sensing, inference, and real-time decision-making capabilities, the amount of AI content per device is increasing.

"We are seeing a structural shift towards hybrid AI, where inference is increasingly moving to the device while more complex processing remains in the cloud or across connected systems. This right AI model, right place, right time approach enables real-time on-device decision-making while maintaining the flexibility to scale compute as needed," stated CEVA in its latest earnings guidance.

CEVA Leadership

The rise of agentic AI, where AI systems autonomously select and execute tools to accomplish tasks, is further accelerating this shift. Small language models designed for on-device deployment are becoming increasingly capable. Models like Hugging Face's SmolLM3 (3 billion parameters), Alibaba's Qwen3-4B (4 billion parameters), Microsoft's Phi-3-mini (3.8 billion parameters), Google DeepMind's Gemma-4-E2B (2.3 billion effective parameters), and Mistral-7B (7.25 billion parameters) all support structured tool calling, enabling agentic workflows entirely at the edge without relying on cloud infrastructure. These models can run on constrained hardware such as edge devices or machines with limited graphics processing unit (GPU) memory, making them practical for real-world deployments where latency, privacy, and cost matter.

The challenge for hardware makers is clear: the next generation of edge AI chips must be designed around the actual behavior of vision LLMs and agentic AI systems, not the image classification tasks of the past. This requires rethinking the unit of execution, improving memory efficiency, and treating hardware and software as a unified system. Companies that make this transition will power the next wave of AI-enabled devices; those that don't will find their chips increasingly mismatched to the workloads they're supposed to handle.