Dobb-E

What is Dobb-E?

Dobb·E is an open-source framework designed to teach robots how to do household chores by watching and learning from demonstrations. It is a way to help robots learn by imitation. It’s a pretty neat solution for the challenges in home robotics, especially because it offers an affordable way to collect the necessary demonstrations. How? With a tool called ‘The Stick.’ This tool is pretty straightforward to make – it uses a $25 Reacher-grabber stick, some 3D printed parts, and an iPhone. Dobb·E uses The Stick to gather data from the Homes of New York (HoNY) dataset. This dataset is quite extensive, featuring 13 hours of interactions recorded in 22 different homes across New York City. The framework then trains a special model called Home Pretrained Representations (HPR). This model is built on the ResNet-34 architecture and uses self-supervised learning objectives. The goal here is to enable robots to tackle new tasks in all sorts of different environments.

Who created Dobb-E?

Dobb-E was developed by a team of researchers: Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, and Lerrel Pinto. They launched it as an open-source project on November 28, 2023. Their aim is to really change how household robots learn and perform tasks, using imitation learning. They’ve developed tools like ‘The Stick’ specifically for collecting data, and the ‘Home Pretrained Representations’ model is key for getting robot policies started in new environments.

What is Dobb-E used for?

Dobb-E is pretty versatile for home robotics. Here’s a breakdown of what it’s used for:

• Teaching robots household tasks: It’s all about teaching robots to do chores by showing them how.
• Improving home robotics: It tackles the current limitations in home robotics by offering a cost-effective and user-friendly way to collect demonstrations.
• Data collection with The Stick: It uses a handy tool called The Stick to gather data from the Homes of New York (HoNY) dataset.
• Training the HPR model: It trains a representation learning model called Home Pretrained Representations (HPR).
• High success rates: It’s achieved an impressive 81% average success rate in completing new tasks within just 15 minutes.
• Open access: You can easily get the pre-trained models, code, and documentation from GitHub.
• Initializing robot policies: It helps start a robot’s policy for performing new tasks in unfamiliar environments.
• Deploying HPR: Home Pretrained Representations (HPR) are used to set up policies for new tasks.
• Fast learning: It demonstrates the ability to learn new tasks in as little as 20 minutes.
• Advancing home robotics: It makes collecting robot demonstrations much easier, pushing progress in the field.
• Collecting demonstrations: It uses The Stick tool specifically for collecting data for robots.
• Accessing resources: Pre-trained models, code, and documentation are available on GitHub.
• Research insights: You can find details about its methods and results in the paper ‘On Bringing Robots Home.’
• HoNY dataset use: The Stick tool is used to collect data from the Homes of New York (HoNY) dataset.
• Stick construction: The Stick itself is built using a $25 Reacher-grabber stick, 3D printed parts, and an iPhone for data collection.
• HPR model details: The HPR model is based on the ResNet-34 architecture and uses self-supervised learning objectives.
• Quick task learning: It shows it can learn a new task in just 20 minutes.

How to use Dobb-E?

Want to teach robots household tasks using imitation learning with Dobb·E? Here’s how you can get started:

1. Get the Model: You can find the Dobb·E model on Huggingface. Alternatively, you can use PyTorch Image Models (TIMM) and get it with just a few lines of code.
2. Collect Demonstrations with The Stick: Grab The Stick – the tool made from a $25 Reacher-grabber stick, some 3D printed parts, and an iPhone. Use it to collect data from the Homes of New York (HoNY) dataset. This dataset includes RGB and depth videos, plus action annotations for the gripper’s position and how wide it opens.
3. Train the Home Pretrained Representations (HPR) Model: Dobb·E uses the data you collect to train the HPR model. This model is built on the ResNet-34 architecture and was trained using the HoNY dataset with self-supervised learning objectives.
4. Initialize Robot Policy: When you’re ready to deploy, the HPR model helps initialize a robot policy. This means it sets up the robot to perform new tasks in different environments. The core of the policy uses the pretrained ResNet-34 model, followed by two linear layers.
5. Access Resources: Dobb·E makes it easy to access pre-trained models, code, and documentation through GitHub. Plus, if you want to dive deeper into how Dobb-E works and its results, check out the paper titled “On Bringing Robots Home.”