Researchers from New York University (NYU), in collaboration with Meta AI, have developed an intelligent robot that can follow verbal instructions to pick up designated objects and relocate them within unfamiliar home environments. Published on the preprint server arXiv, the study demonstrates a promising fusion of advanced robotics and AI.
The robot leverages visual language models (VLMs) - AI systems adept at recognizing objects from descriptive prompts. "VLMs have progressed a great deal in recent years and have become very good at recognizing objects based on language," the researchers explain. The mobile robot, called OK-Robot, additionally wields mechanical skills for dexterous grasping and navigation.
During trials across 10 volunteer households, the team issued OK-Robot 170 tasks like "move the pink bottle on the shelf to the trash can." Without any environment-specific training, OK-Robot succeeded 58% of the time. Decluttering spaces beforehand boosted success rates to 82%.
The zero-shot transfer performance exhibits VLMs effectively guiding skilled robots in unfamiliar real-world settings. "Our work could be the first step toward advanced VLM-based robots," the researchers state.
Tweaks to the system and more advanced platforms would likely further enhance reliability down the road. But the prototype already demonstrates AI's potential to direct robotic helpers via intuitive voice commands in messy home environments.
Such voice-controlled robot assistants could someday aid the elderly or disabled with everyday household tasks. And the self-directed visual intelligence showcases promising scalability beyond constrained industrial settings reliant on extensive supervision and environmental instrumentation.
"Robots skills have improved - they can grasp things without breaking them, carry them to locations and set them down," the team notes. "But little has been done to combine VLMs with skilled robots." This breakthrough melds complementary AI and robotic capabilities towards that long-standing goal.
