A new approach called iPack uses vision-language models to teach robots how to pack groceries the way humans do, preventing damage by understanding product fragility and weight without requiring any training on new items. Researchers have developed a system that combines the semantic reasoning abilities of large language models (LLMs) with computer vision to solve a problem that has largely been ignored in warehouse automation: how to pack diverse items safely and efficiently.

Why Has Grocery Packing Been Overlooked in Robotics?

While robots have become increasingly common in warehouses and manufacturing facilities, most focus on picking items from shelves or optimizing how many objects fit into a container. Grocery packing, however, involves a fundamentally different challenge. Heavy objects should never rest on top of fragile ones, frozen items shouldn't sit on moisture-sensitive products, and the variety of items in a typical grocery store makes it nearly impossible to write rules that cover every scenario.

Humans solve this intuitively. When you pack your own groceries, you automatically know that eggs go on top, canned goods go on the bottom, and bread shouldn't be crushed under anything. But translating that common sense into instructions a robot can follow has remained largely unexplored until now. Previous work in grocery store automation focused on navigation and picking, leaving the crucial final step of packing almost entirely unaddressed.

How Does iPack Combine Vision and Language Models?

The iPack system works by dividing the packing process into three major steps. First, it uses vision foundation models (VFMs) to detect and classify objects in a camera image, estimating their weight, size, and position. Then it queries a large language model to generate packing constraints that mimic human strategies. Finally, it uses a mathematical optimization framework called mixed-integer linear programming (MILP) to compute the optimal packing arrangement, which a robotic arm then executes.

The key innovation is that iPack operates in an "open-vocabulary" manner, meaning it doesn't require retraining when encountering new items. Because the system relies on the general knowledge embedded in pretrained language models, it can reason about novel products it has never seen before, adapting its packing strategy based on semantic understanding rather than memorized rules.

Steps to Implement Vision-Language Models in Robotics Tasks

• Perception Phase: Deploy vision foundation models to detect objects, classify them, and estimate physical properties like weight and fragility from visual input alone.
• Semantic Reasoning: Query a large language model to generate context-aware constraints, such as "eggs are fragile" or "frozen items should not contact warm surfaces," without manual rule specification.
• Spatial Optimization: Use mathematical optimization frameworks to compute an arrangement that satisfies all constraints while maximizing space utilization and minimizing damage risk.
• Robotic Execution: Translate the optimized packing plan into robot manipulator commands, executing the sequence in the physical world.

The researchers also introduced a new metric called the Packing Consistency Score (PCS) to measure how closely a robot's packing strategy matches human preferences. This data-driven approach allows the system to be evaluated not just on whether items fit, but on whether they're packed in a way that feels natural and safe to humans.

What Makes This Different From Previous Packing Systems?

Earlier bin packing methods, particularly those used in warehouses, focused almost exclusively on two goals: maximizing how much space is used and ensuring objects don't topple over. These systems treat items as geometric shapes to be arranged efficiently, ignoring the semantic properties that matter in real-world grocery packing.

Some prior work on grocery packing did attempt to capture human preferences, but it required collecting training data. Researchers would have humans pack groceries in virtual reality, then train a machine learning model on those sequences. iPack eliminates this requirement entirely. Because it leverages the general knowledge already embedded in large language models, it can reason about packing constraints for products it has never encountered, making it immediately applicable to new grocery items without any additional training.

The modular design of iPack also means that as better vision and language models become available, they can be seamlessly integrated into the system. Researchers don't need to retrain or redesign the entire pipeline; they can simply swap in improved foundation models as they emerge.

What Are the Real-World Implications?

The researchers extensively evaluated iPack in both simulation and on real robots, demonstrating its applicability across different scenarios. The system can also be extended to related logistics tasks, such as selecting the smallest suitable container from a set of options or incorporating constraints for packing stability and robot reachability.

By releasing the code and evaluation dataset publicly on GitHub, the team has made it possible for other researchers and companies to integrate iPack into custom scenarios. This open-source approach could accelerate adoption of vision-language models in real-world logistics, moving beyond the controlled environments of traditional warehouses into grocery stores, fulfillment centers, and other settings where product integrity matters.

The success of iPack demonstrates a broader trend: vision-language models are moving beyond chatbots and image captioning into practical robotics applications where semantic reasoning directly impacts physical outcomes. As these models continue to improve, we can expect to see similar applications emerge in other domains where human intuition about object properties and relationships is difficult to encode manually.