When it comes to automation in online grocery warehouses, robotic arms play a pivotal role in ensuring efficient order fulfilment. Yet, these arms face a key challenge: moving swiftly without dropping the items they transport. A common solution is to slow down, but that approach hampers productivity. Kareem Elsawah, a graduate student in the University of Toronto’s Master of Science in Applied Computing (MScAC) program, partnered with Ocado Technology to tackle this problem. Under the supervision of Kevin George from Ocado and Professor Igor Gilitschenski from the University of Toronto, Elsawah explored how to maximize speed while maintaining a low drop rate.
The result? An innovative system that uses machine learning and model predictive control (MPC) to optimize robotic arm motions. This research not only advances robotic manipulation but also has real-world implications for e-commerce logistics, where speed and precision are paramount.
Predicting When Items Will Drop
The core of the challenge lies in predicting when an item will fall. Robotic arms handle a wide variety of objects, from lightweight bags of chips to heavy cans of soup, each presenting unique demands. Elsawah’s research tackles this variability by leveraging a wealth of production data, including joint states, suction pressure, and visual embeddings generated by the CLIP image and text understanding neural network.
This problem is framed as a survival analysis task, commonly used in medical research to predict time-to-event outcomes. In this case, the “event” is the item dropping. The machine learning model uses data from both past and planned states of the robot to predict how long an item can be held securely. This predictive capability allows the robotic system to anticipate drops before they occur, enabling proactive adjustments to the arm’s motion.
“This project at Ocado Technology gave me valuable insights into applied research. I learned to balance theoretical innovation with practical factors like robustness, monitorability, maintainability, and scalability,” said Elsawah. “Working in robotics taught me to think through multiple layers of abstraction — from mechanical and electrical components at the lowest level, through drivers and ML models, up to software, containers, and cluster management. These ideas enriched my approach and helped develop a solution that meets the demands of real-world production environments.”
Optimizing Motions with Model Predictive Control
Prediction alone is not enough; the robotic arm also needs to adjust its movements to maximize speed without exceeding the limits defined by the drop predictions. Here, Elsawah employed model predictive control (MPC), a control strategy known for its effectiveness when paired with a reliable world model.
The world model in this system combines classical kinematics with the learnt time-to-drop predictions. This dual approach allows the MPC to dynamically adjust the arm’s trajectory, optimizing for speed while minimizing the risk of dropping items. By separating the prediction and optimization components, the system becomes modular and adaptable, enabling faster development and fine-tuning.
In simulations, this approach reduced drop rates by up to 50 per cent at high speeds. When implemented on physical robotic arms, the system showed a statistically significant reduction in forces applied to items during motion (p < 0.01). These results demonstrate the potential of combining predictive models with advanced control strategies to achieve both precision and efficiency.
Paving the Way for Smarter Robots Everywhere
Ocado Technology, known for its expertise in robotic manipulation, provided the ideal environment for testing this innovative approach.
“Kareem was pivotal to our product by diving deep into a research idea based on a novel machine learning approach to control our robotic arms, experiment various implementations, and finally integrating it into our product,” said Guillaume Crabe, senior robotics software developer at Ocado. “Only an MScAC internship would allow us to transform research ideas into viable products thanks to the extended time we are able to partner with the student.”
This research aligns with broader trends in automation and e-commerce. As online grocery platforms grow, so does the demand for efficient and versatile robotic systems. The insights gained from this project could inform the design of next-generation robotic manipulators, enabling them to handle an even wider range of tasks with speed and reliability.
However, the implications of Elsawah’s work extend beyond warehouses. Robotic arms are used in industries ranging from manufacturing to health care, where precision and speed are equally critical. By demonstrating the feasibility of combining survival analysis and MPC, this research sets the stage for advancements in diverse applications, from surgical robots to autonomous delivery systems.
A Glimpse into the Future
As automation becomes an integral part of modern life, the ability to move fast without compromising accuracy will define the next generation of robotic systems. Kareem Elsawah’s research, supported by Kindred AI and the University of Toronto, offers a glimpse into this future. By combining predictive modelling and advanced control strategies, this work not only enhances robotic performance but also exemplifies the power of interdisciplinary innovation.
For online grocery platforms and beyond, the message is clear: hold tight and move fast. With smarter robots, the possibilities are limitless.
