b'Yi Tay'

research

∙ 05/15/2023

Symbol tuning improves in-context learning in language models

We present symbol tuning - finetuning language models on in-context inpu...

0 Jerry Wei, et al. ∙

research

∙ 04/18/2023

UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining

Pretrained multilingual large language models have typically used heuris...

0 Hyung Won Chung, et al. ∙

research

∙ 03/17/2023

CoLT5: Faster Long-Range Transformers with Conditional Computation

Many natural language processing tasks benefit from long inputs, but pro...

0 Joshua Ainslie, et al. ∙

research

∙ 03/07/2023

Larger language models do in-context learning differently

We study how in-context learning (ICL) in language models is affected by...

0 Jerry Wei, et al. ∙

research

∙ 01/31/2023

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

We study the design decisions of publicly available instruction tuning m...

0 Shayne Longpre, et al. ∙

research

∙ 12/19/2022

DSI++: Updating Transformer Memory with New Documents

Differentiable Search Indices (DSIs) encode a corpus of documents in the...

0 Sanket Vaibhav Mehta, et al. ∙

research

∙ 12/16/2022

Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification

With the devastating outbreak of COVID-19, vaccines are one of the cruci...

0 Jai Gupta, et al. ∙

research

∙ 12/09/2022

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

Training large, deep neural networks to convergence can be prohibitively...

0 Aran Komatsuzaki, et al. ∙

research

∙ 11/03/2022

Inverse scaling can become U-shaped

Although scaling language models improves performance on a range of task...

0 Jason Wei, et al. ∙

research

∙ 10/20/2022

Transcending Scaling Laws with 0.1

Scaling language models improves performance but comes with significant ...

2 Yi Tay, et al. ∙

research

∙ 10/06/2022

Language Models are Multilingual Chain-of-Thought Reasoners

We evaluate the reasoning abilities of large language models in multilin...

0 Freda Shi, et al. ∙

research

∙ 10/04/2022

Recitation-Augmented Language Models

We propose a new paradigm to help Large Language Models (LLMs) generate ...

0 Zhiqing Sun, et al. ∙

research

∙ 07/21/2022

Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

There have been a lot of interest in the scaling properties of Transform...

0 Yi Tay, et al. ∙

research

∙ 04/25/2022

ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference

State-of-the-art neural models typically encode document-query pairs usi...

3 Kai Hui, et al. ∙

research

∙ 04/05/2022

PaLM: Scaling Language Modeling with Pathways

Large language models have been shown to achieve remarkable performance ...

6 Aakanksha Chowdhery, et al. ∙

research

∙ 02/22/2022

A New Generation of Perspective API: Efficient Multilingual Character-level Transformers

On the world wide web, toxic content detectors are a crucial line of def...

0 Alyssa Lees, et al. ∙

research

∙ 11/25/2021

PolyViT: Co-training Vision Transformers on Images, Videos and Audio

Can we train a single transformer model capable of processing multiple m...

5 Valerii Likhosherstov, et al. ∙

research

∙ 10/25/2021

The Efficiency Misnomer

Model efficiency is a critical aspect of developing and deploying machin...

5 Mostafa Dehghani, et al. ∙

research

∙ 10/18/2021

SCENIC: A JAX Library for Computer Vision Research and Beyond

Scenic is an open-source JAX library with a focus on Transformer-based m...

31 Mostafa Dehghani, et al. ∙

research

∙ 10/16/2021

Sharpness-Aware Minimization Improves Language Model Generalization

The allure of superhuman-level capabilities has led to considerable inte...

1 Dara Bahri, et al. ∙

research

∙ 10/16/2021

Improving Compositional Generalization with Self-Training for Data-to-Text Generation

Data-to-text generation focuses on generating fluent natural language re...

4 Sanket Vaibhav Mehta, et al. ∙

research

∙ 09/30/2021

Born Again Neural Rankers

We introduce Born Again neural Rankers (BAR) in the Learning to Rank (LT...

0 Zhen Qin, et al. ∙

research

∙ 09/22/2021

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

There remain many open questions pertaining to the scaling behaviour of ...

3 Yi Tay, et al. ∙

research

∙ 07/14/2021

The Benchmark Lottery

The world of empirical machine learning (ML) strongly relies on benchmar...

8 Mostafa Dehghani, et al. ∙

research

∙ 06/29/2021

SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption

Self-supervised contrastive representation learning has proved incredibl...

7 Dara Bahri, et al. ∙

research

∙ 06/23/2021

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

State-of-the-art models in natural language processing rely on separate ...

4 Yi Tay, et al. ∙

research

∙ 05/12/2021

How Reliable are Model Diagnostics?

In the pursuit of a deeper understanding of a model's behaviour, there i...

6 Vamsi Aribandi, et al. ∙

research

∙ 05/07/2021

Are Pre-trained Convolutions Better than Pre-trained Transformers?

In the era of pre-trained language models, Transformers are the de facto...

11 Yi Tay, et al. ∙

research

∙ 05/05/2021

Rethinking Search: Making Experts out of Dilettantes

When experiencing an information need, users want to engage with an expe...

6 Donald Metzler, et al. ∙

research

∙ 03/01/2021

OmniNet: Omnidirectional Representations from Transformers

This paper proposes Omnidirectional Representations from Transformers (O...

19 Yi Tay, et al. ∙

research

∙ 02/23/2021

Do Transformer Modifications Transfer Across Implementations and Applications?

The research community has proposed copious modifications to the Transfo...

10 Sharan Narang, et al. ∙

research

∙ 02/17/2021

Switch Spaces: Learning Product Spaces with Sparse Gating

Learning embedding spaces of suitable geometry is critical for represent...

6 Shuai Zhang, et al. ∙

research

∙ 02/17/2021

Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters

Recent works have demonstrated reasonable success of representation lear...

6 Aston Zhang, et al. ∙

research

∙ 02/09/2021

Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection

Detecting out-of-distribution (OOD) examples is critical in many applica...

5 Dara Bahri, et al. ∙

research

∙ 12/01/2020

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

There are two major classes of natural language grammars – the dependenc...

2 Yikang Shen, et al. ∙

research

∙ 11/08/2020

Long Range Arena: A Benchmark for Efficient Transformers

Transformers do not scale very well to long sequence lengths largely bec...

5 Yi Tay, et al. ∙

research

∙ 10/19/2020

Surprise: Result List Truncation via Extreme Value Theory

Work in information retrieval has largely been centered around ranking a...

1 Dara Bahri, et al. ∙

research

∙ 10/06/2020

Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder

This paper demonstrates a fatal vulnerability in natural language infere...

21 Alvin Chan, et al. ∙

research

∙ 09/14/2020

Efficient Transformers: A Survey

Transformer model architectures have garnered immense interest lately du...

33 Yi Tay, et al. ∙

research

∙ 08/17/2020

Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study

Large generative language models such as GPT-2 are well-known for their ...

0 Dara Bahri, et al. ∙

research

∙ 07/12/2020

HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections

Achieving state-of-the-art performance on natural language understanding...

0 Yi Tay, et al. ∙

research

∙ 05/02/2020

Synthesizer: Rethinking Self-Attention in Transformer Models

The dot product self-attention is known to be central and indispensable ...

0 Yi Tay, et al. ∙

research

∙ 04/26/2020

Choppy: Cut Transformer For Ranked List Truncation

Work in information retrieval has traditionally focused on ranking and r...

2 Dara Bahri, et al. ∙

research

∙ 04/13/2020

Reverse Engineering Configurations of Neural Text Generation Models

This paper seeks to develop a deeper understanding of the fundamental pr...

0 Yi Tay, et al. ∙

research

∙ 02/26/2020

Sparse Sinkhorn Attention

We propose Sparse Sinkhorn Attention, a new efficient and sparse method ...

0 Yi Tay, et al. ∙

research

∙ 01/20/2020

Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching

Transformer has been successfully applied to many natural language proce...

0 Shuohang Wang, et al. ∙

research

∙ 12/21/2019

Jacobian Adversarially Regularized Networks for Robustness

Adversarial examples are crafted with imperceptible perturbations with t...

15 Alvin Chan, et al. ∙

research

∙ 12/11/2019

What it Thinks is Important is Important: Robustness Transfers through Input Gradients

Adversarial perturbations are imperceptible changes to input pixels that...

27 Alvin Chan, et al. ∙

research

∙ 08/27/2019

Interactive Machine Comprehension with Information Seeking Agents

Existing machine reading comprehension (MRC) models do not scale effecti...

5 Xingdi Yuan, et al. ∙

research

∙ 06/11/2019

Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

Many state-of-the-art neural models for NLP are heavily parameterized an...

0 Yi Tay, et al. ∙

Yi Tay

Featured Co-authors