A new study by researchers at Columbia Engineering has revealed that robots can now teach themselves about their own bodies and how they move simply by observing their actions through a camera.
This advancement allows robots to not only plan their movements but also adapt to physical damage, enabling more resilience in real-world applications.
The study, led by Yuhang Hu, a doctoral student at Columbia University’s Creative Machines Lab, and directed by Professor Hod Lipson, presents a significant breakthrough in robotic ‘self-awareness’.
By mimicking the process humans use to learn new skills, such as watching their reflection to learn to dance, robots can now use video footage to build an understanding of their own kinematics. This self-awareness allows robots to adapt to physical changes or damage, without the need for constant human intervention or reprogramming.
Traditionally, robots learn to move through simulations that require detailed engineering to create. Once robots are able to navigate virtual environments, they are released into the real world to continue their learning. However, building an effective simulation is often an intricate and time-consuming process. In this study, the researchers demonstrated a method for robots to autonomously create simulations of their own movements by simply observing themselves with a regular 2D camera.
This method utilises three brain-like AI systems known as deep neural networks, which analyse 2D video to infer 3D motion. With this ability, robots can understand their own movements, identify changes in their structure—such as a bent arm—and adjust their actions to recover from the damage. This advancement saves engineers considerable effort in creating simulations and allows the robots to adapt as they experience wear and tear over time.
The researchers believe the implications of this technology are significant. In practical terms, robots could become more resilient in everyday tasks. For instance, a robot vacuum or personal assistant bot that sustains a minor damage, such as a bent arm, could continue working by adjusting its movements, rather than needing repair.
This adaptability would reduce the need for constant reprogramming and make robots more reliable in homes and workplaces.
Similarly, in industrial settings, such as car factories, a robot arm that becomes misaligned could adjust its movements and continue performing tasks like welding, reducing downtime and production costs. This could lead to more resilient manufacturing processes and less dependency on human intervention.
The need for such adaptability becomes especially crucial as robots take on more critical functions in industries like manufacturing and healthcare. Researchers emphasise that, in order for robots to be truly useful, they must be capable of taking care of themselves, learning to adjust their performance, and recovering from damage without constant human oversight.
The new study builds on a series of projects led by the Columbia team over the past two decades, which have focused on teaching robots to self-model using various sensors. In 2006, the team demonstrated that robots could create simple stick-figure models of themselves based on observations. Over time, this capability has evolved, with robots using multiple cameras to create more detailed simulations.
The breakthrough in this latest study allows robots to generate comprehensive kinematic models using just a short video clip from a single camera, a process the researchers refer to as “Kinematic Self-Awareness.”
Yuhang Hu, a doctoral student at the Creative Machines Lab at Columbia University, said: “Like humans learning to dance by watching their mirror reflection, robots now use raw video to build kinematic self-awareness.
“Our goal is a robot that understands its own body, adapts to damage, and learns new skills without constant human programming.”
Join more than 11,000 industry leaders at Robotics and Automation Exhibition on 25-26 March 2025, at the NEC Birmingham to explore cutting-edge technologies, connect with peers and discover the latest innovations shaping the future of warehousing, manufacturing, engineering and logistics. Register for free now to secure your place at this premier event!
