Silvio Savarese

research

∙ 08/16/2023

Enhancing Performance on Seen and Unseen Dialogue Scenarios using Retrieval-Augmented End-to-End Task-Oriented System

End-to-end task-oriented dialogue (TOD) systems have achieved promising ...

0 Jianguo Zhang, et al. ∙

research

∙ 08/11/2023

BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents

The massive successes of large language models (LLMs) encourage the emer...

0 Zhiwei Liu, et al. ∙

research

∙ 08/04/2023

Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

Recent months have seen the emergence of a powerful new trend in which l...

0 Weiran Yao, et al. ∙

research

∙ 07/19/2023

DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI

Despite advancements in conversational AI, language models encounter cha...

0 Jianguo Zhang, et al. ∙

research

∙ 07/18/2023

REX: Rapid Exploration and eXploitation for AI Agents

In this paper, we propose an enhanced approach for Rapid Exploration and...

0 Rithesh Murthy, et al. ∙

research

∙ 05/18/2023

UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

Achieving machine autonomy and human control often represent divergent o...

0 Can Qin, et al. ∙

research

∙ 05/03/2023

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

Large language models (LLMs) have demonstrated remarkable abilities in r...

3 Erik Nijkamp, et al. ∙

research

∙ 03/31/2023

Procedure-Aware Pretraining for Instructional Video Understanding

Our goal is to learn a video representation that is useful for downstrea...

0 Honglu Zhou, et al. ∙

research

∙ 03/16/2023

HIVE: Harnessing Human Feedback for Instructional Visual Editing

Incorporating human feedback has been shown to be crucial to align text ...

1 Shu Zhang, et al. ∙

research

∙ 01/30/2023

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

The cost of vision-and-language pre-training has become increasingly pro...

0 Junnan Li, et al. ∙

research

∙ 11/22/2022

Best-k Search Algorithm for Neural Text Generation

Modern natural language generation paradigms require a good decoding str...

0 Jiacheng Xu, et al. ∙

research

∙ 11/17/2022

Online Distribution Shift Detection via Recency Prediction

When deploying modern machine learning-enabled robotic systems in high-s...

0 Rachel Luo, et al. ∙

research

∙ 10/17/2022

Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

Visual question answering (VQA) is a hallmark of vision and language rea...

0 Anthony Meng Huat Tiong, et al. ∙

research

∙ 09/15/2022

LAVIS: A Library for Language-Vision Intelligence

We introduce LAVIS, an open-source deep learning library for LAnguage-VI...

0 Dongxu Li, et al. ∙

research

∙ 08/22/2022

Minkowski Tracker: A Sparse Spatio-Temporal R-CNN for Joint Object Detection and Tracking

Recent research in multi-task learning reveals the benefit of solving re...

0 JunYoung Gwak, et al. ∙

research

∙ 07/05/2022

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

Program synthesis or code generation aims to generate a program that sat...

0 Hung Le, et al. ∙

research

∙ 06/07/2022

Masked Unsupervised Self-training for Zero-shot Image Classification

State-of-the-art computer vision models are mostly trained with supervis...

0 Junnan Li, et al. ∙

research

∙ 06/01/2022

OmniXAI: A Library for Explainable AI

We introduce OmniXAI, an open-source Python library of eXplainable AI (X...

0 Wenzhuo Yang, et al. ∙

research

∙ 03/25/2022

A Conversational Paradigm for Program Synthesis

Program synthesis strives to generate a computer program as a solution t...

11 Erik Nijkamp, et al. ∙

research

∙ 03/15/2022

Long Document Summarization with Top-down and Bottom-up Inference

Text summarization aims to condense long documents and retain key inform...

11 Bo Pang, et al. ∙

research

∙ 12/09/2021

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

In mobile manipulation (MM), robots can both navigate within and interac...

9 Josiah Wong, et al. ∙

research

∙ 09/28/2021

Sample-Efficient Safety Assurances using Conformal Prediction

When deploying machine learning models in high-stakes robotics applicati...

0 Rachel Luo, et al. ∙

research

∙ 09/20/2021

Merlion: A Machine Learning Library for Time Series

We introduce Merlion, an open-source machine learning library for time s...

78 Aadyot Bhatnagar, et al. ∙

research

∙ 09/02/2021

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

We study the problem of learning a range of vision-based manipulation ta...

1 Suraj Nair, et al. ∙

research

∙ 08/06/2021

BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities i...

10 Sanjana Srivastava, et al. ∙

research

∙ 08/06/2021

iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks

Recent research in embodied AI has been boosted by the use of simulation...

18 Chengshu Li, et al. ∙

research

∙ 06/26/2021

Discovering Generalizable Skills via Automated Generation of Diverse Tasks

The learning efficiency and generalization ability of an intelligent age...

5 Kuan Fang, et al. ∙

research

∙ 06/16/2021

JRDB-Act: A Large-scale Multi-modal Dataset for Spatio-temporal Action, Social Group and Activity Detection

The availability of large-scale video action understanding datasets has ...

0 Mahsa Ehsanpour, et al. ∙

research

∙ 03/29/2021

LASER: Learning a Latent Action Space for Efficient Reinforcement Learning

The process of learning a manipulation task depends strongly on the acti...

0 Arthur Allshire, et al. ∙

research

∙ 03/23/2021

Neural Architecture Search From Fréchet Task Distance

We formulate a Fréchet-type asymmetric distance between tasks based on F...

0 Cat P. Le, et al. ∙

research

∙ 02/22/2021

Localized Calibration: Metrics and Recalibration

Probabilistic classifiers output confidence scores along with their pred...

1 Rachel Luo, et al. ∙

research

∙ 02/03/2021

Embodied Intelligence via Learning and Evolution

The intertwined processes of learning and evolution in complex environme...

10 Agrim Gupta, et al. ∙

research

∙ 12/12/2020

Learning Multi-Arm Manipulation Through Collaborative Teleoperation

Imitation Learning (IL) is a powerful paradigm to teach robots to perfor...

7 Albert Tung, et al. ∙

research

∙ 12/12/2020

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Imitation Learning is a promising paradigm for learning complex robot ma...

15 Ajay Mandlekar, et al. ∙

research

∙ 12/09/2020

Topological Planning with Transformers for Vision-and-Language Navigation

Conventional approaches to vision-and-language navigation (VLN) are trai...

1 Kevin Chen, et al. ∙

research

∙ 12/07/2020

Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search

Searching for objects in indoor organized environments such as homes or ...

68 Andrey Kurenkov, et al. ∙

research

∙ 12/05/2020

iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes

We present iGibson, a novel simulation environment to develop robotic so...

9 Bokui Shen, et al. ∙

research

∙ 11/17/2020

Deep Affordance Foresight: Planning Through What Can Be Done in the Future

Planning in realistic environments requires searching in large planning ...

0 Danfei Xu, et al. ∙

research

∙ 11/13/2020

Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation

Vision-based robotics often separates the control loop into one module f...

0 Bryan Chen, et al. ∙

research

∙ 10/25/2020

Multimodal Sensor Fusion with Differentiable Filters

Leveraging multimodal information with recursive Bayesian filters improv...

0 Michelle A. Lee, et al. ∙

research

∙ 10/16/2020

Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning

Navigating fluently around pedestrians is a necessary capability for mob...

0 Claudia Pérez-D'Arpino, et al. ∙

research

∙ 08/21/2020

Privacy Preserving Recalibration under Domain Shift

Classifiers deployed in high-stakes real-world applications must output ...

4 Rachel Luo, et al. ∙

research

∙ 08/18/2020

ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

Many Reinforcement Learning (RL) approaches use joint control signals (p...

2 Fei Xia, et al. ∙

research

∙ 08/13/2020

Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter

When searching for objects in cluttered environments, it is often necess...

4 Andrey Kurenkov, et al. ∙

research

∙ 08/08/2020

How Trustworthy are the Existing Performance Evaluations for Basic Vision Tasks?

Performance evaluation is indispensable to the advancement of machine vi...

3 Hamid Rezatofighi, et al. ∙

research

∙ 07/14/2020

Goal-Aware Prediction: Learning to Model What Matters

Learned dynamics models combined with both planning and policy learning ...

13 Suraj Nair, et al. ∙

research

∙ 07/01/2020

Adaptive Procedural Task Generation for Hard-Exploration Problems

We introduce Adaptive Procedural Task Generation (APT-Gen), an approach ...

18 Kuan Fang, et al. ∙

research

∙ 06/22/2020

Generative Sparse Detection Networks for 3D Single-shot Object Detection

3D object detection has been widely studied due to its potential applica...

7 JunYoung Gwak, et al. ∙

research

∙ 03/20/2020

Probabilistic Visual Navigation with Bidirectional Image Prediction

Humans can robustly follow a visual trajectory defined by a sequence of ...

0 Noriaki Hirose, et al. ∙

research

∙ 03/13/2020

Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations

Imitation learning is an effective and safe technique to train robot pol...

3 Ajay Mandlekar, et al. ∙

Silvio Savarese

Featured Co-authors