Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents

by   Jacopo Tani, et al.

As robotics matures and increases in complexity, it is more necessary than ever that robot autonomy research be reproducible. Compared to other sciences, there are specific challenges to benchmarking autonomy, such as the complexity of the software stacks, the variability of the hardware and the reliance on data-driven techniques, amongst others. In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research/development processes. We first provide the overall conceptual objectives to achieve this goal and then a concrete instance that we have built: the DUCKIENet. One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible. When evaluating agents, careful definition of interfaces allows users to choose among local versus remote evaluation using simulation, logs, or remote automated hardware setups. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.


page 1

page 4

page 6

page 7


Accessible Interfaces for the Development and Deployment of Robotic Platforms

Accessibility is one of the most important features in the design of rob...

DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation

We present DeepClaw as a reconfigurable benchmark of robotic hardware an...

BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots

We introduce BenchBot, a novel software suite for benchmarking the perfo...

Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research

Autonomous fabric manipulation is a longstanding challenge in robotics, ...

Stanford Pupper: A Low-Cost Agile Quadruped Robot for Benchmarking and Education

We present Stanford Pupper, an easily-replicated open source quadruped r...

A Development Cycle for Automated Self-Exploration of Robot Behaviors

In this paper we introduce Q-Rock, a development cycle for the automated...

Accuracy evaluation of a Low-Cost Differential Global Positioning System for mobile robotics

Differential GPS, commonly referred as DGPS, is a well-known and very ac...

Please sign up or login with your details

Forgot password? Click here to reset