Andes Technology has fundamentally reshaped how neural processing units handle artificial intelligence at the edge by upgrading its AnDLA I370 accelerator to support transformer-based models, vision-language models, and small language models on power-constrained devices. The company released version 2.0 of its deep learning accelerator alongside version 1.2.0 of its AndesAIRE neural network software development kit, marking a significant evolution from the original I370 announced in June 2025.

Why Are Transformer Models So Difficult for Edge Devices?

The original I370 was designed primarily for conventional computer vision, audio processing, and recurrent neural networks, which are relatively straightforward to optimize on specialized chips. Transformer models, by contrast, present a fundamentally different engineering challenge. These models, which power modern large language models and vision systems, require handling complex mathematical operations like normalization and attention mechanisms that don't map neatly onto traditional matrix-multiplication arrays.

The new I370 v2.0 addresses this by including native division and square root operations, operator fusion, and what Andes calls a Programmable Tensor Operator Set Architecture. This means more of the transformer computation stays on the accelerator itself rather than bouncing back to the main processor, reducing data movement and latency. The hardware can now be configured with between 32 and 2,048 multiply-accumulate units, with an AXI interface that scales to 512 bits for improved memory utilization.

What Makes Software Changes Just as Important as Hardware Upgrades?

The software improvements may ultimately matter more than the additional computing units. AndesAIRE NN SDK v1.2.0 now supports attention mechanisms, separates the prefill and decode phases of small language model inference, and manages KV-cache placement and memory scheduling. These capabilities are critical because language models require careful management of intermediate computation results, called the KV-cache, which can consume enormous amounts of memory in constrained environments.

The updated software stack includes Linux kernel-mode drivers, user-space runtime support, and real-time operating system compatibility. Task preemption is available in the RTOS environment, allowing higher-priority inference jobs to interrupt less urgent work. Andes has also added Model Advisor and Hardware Advisor tools to help developers balance cost and performance when selecting hardware configurations.

How to Deploy Transformer Models on Edge Devices

• Model Compression: Use word-level knowledge distillation to compress larger teacher models into smaller models suitable for deployment on the accelerator, reducing memory footprint without sacrificing accuracy.
• Framework Compatibility: Leverage support for current LiteRT and LiteRT for Microcontrollers frameworks, which allow developers to use familiar tools without rewriting code or switching development environments.
• Hardware Configuration: Select appropriate multiply-accumulate unit counts and memory interfaces using the Hardware Advisor tool, tailoring the chip design to specific application requirements like power budgets and latency constraints.
• Benchmark Validation: Test implementations against the MLPerf Tiny v1.4 benchmark suite to ensure performance meets real-world requirements before deployment.

Andes is positioning the updated platform for smart cameras, industrial automation, robotics, wearables, and edge gateways. These are highly fragmented markets where local inference can reduce network traffic and latency while keeping sensitive sensor data on the device rather than sending it to cloud servers.

"Bringing transformer models and VLMs to cost-sensitive edge devices is arguably one of the toughest engineering problems in the current AI landscape," said Charlie Su, CTO and president of Andes Technology.

Charlie Su, CTO and President at Andes Technology

Why Is RISC-V Becoming the Foundation for Edge AI?

The broader significance of Andes' work lies in its use of RISC-V, an open-source processor instruction set architecture. RISC-V is unlikely to displace established GPU platforms in data centers, but it offers something more valuable for edge AI: flexibility. A smart camera, hearing device, industrial robot, vehicle controller, and wearable assistant all use neural networks, but they have little else in common. They operate under different latency requirements, memory constraints, safety expectations, operating systems, and power budgets.

A fixed accelerator paired with a fixed processor configuration will leave performance, power, or silicon area on the table in at least some applications. RISC-V gives system-on-chip (SoC) developers greater scope to tune the processor and surrounding system to the workload. Andes is extending that flexibility beyond the CPU by offering a configurable accelerator, programmable tensor operations, selectable MAC resources, adjustable memory interfaces, and a software stack intended to map models onto the resulting hardware.

This approach differs fundamentally from simply pairing a standard CPU with someone else's neural processing unit. Andes is building a licensable edge-AI subsystem in which the control processor, accelerator, runtime, model-conversion tools, and operating system support can be developed together. That integration reduces the engineering burden on customers and gives designers more freedom to produce domain-specific silicon without assembling the entire hardware and software chain themselves.

What Real-World Applications Are Already Using These Chips?

Beyond Andes' work, the broader market is moving toward server-based neural processing units for more demanding applications. Rebellions, another AI chip company, announced that it will supply server-type NPUs to the CCTV monitoring centers of South Gyeongsang Province and Ulsan Metropolitan City in South Korea. The project is part of a domestic NPU-based public AI CCTV transition initiative promoted by the Ministry of Science and Information and Communications Technology and the National Information Society Agency.

Rebellions' ATOM-Max server runs vision-language models of 30 billion parameters or larger at monitoring centers, precisely processing CCTV footage. Models of this size require large memory capacity and high computing performance. Small edge chips alone have limitations in stably achieving precise video analysis and minimizing false positives and false negatives in large-scale computing environments, making server-type NPUs essential for this use case.

The server-type NPUs supplied by Rebellions can handle large-scale simultaneous processing when sudden situations surge and operate independently even within closed networks separated from the external internet. They also provide the flexibility to quickly replace and deploy new AI models. Rebellions officially supports vLLM, a standard AI operating tool widely used by developers, allowing public institutions to adopt the NPUs immediately without changing their existing development environment.

The evolution of neural processing units from simple convolutional accelerators to flexible, transformer-capable systems reflects the changing landscape of AI deployment. As transformer models become the dominant architecture for both language and vision tasks, the ability to run these models efficiently at the edge, whether on wearables or in surveillance centers, is becoming a critical competitive advantage. Andes deserves credit for moving with the workload, though the company still must demonstrate that the architecture can execute useful models at commercially attractive power, area, and latency in high-volume production.