Differentially Private Adapters for Parameter Efficient Acoustic Modeling

05/19/2023

∙

In this work, we devise a parameter-efficient solution to bring differential privacy (DP) guarantees into adaptation of a cross-lingual speech classifier. We investigate a new frozen pre-trained adaptation framework for DP-preserving speech modeling without full model fine-tuning. First, we introduce a noisy teacher-student ensemble into a conventional adaptation scheme leveraging a frozen pre-trained acoustic model and attain superior performance than DP-based stochastic gradient descent (DPSGD). Next, we insert residual adapters (RA) between layers of the frozen pre-trained acoustic model. The RAs reduce training cost and time significantly with a negligible performance drop. Evaluated on the open-access Multilingual Spoken Words (MLSW) dataset, our solution reduces the number of trainable parameters by 97.5 only a 4 classifier while preserving DP guarantees.

READ FULL TEXT

Differentially Private Adapters for Parameter Efficient Acoustic Modeling

Differentially Private Bias-Term only Fine-tuning of Foundation Models

Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers

An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition

PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification

DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classification

Parameter-Efficient Learning for Text-to-Speech Accent Adaptation

Differentially Private Adapters for Parameter Efficient Acoustic Modeling

Related Research

Differentially Private Bias-Term only Fine-tuning of Foundation Models

Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers

An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition

PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification

DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classification

Parameter-Efficient Learning for Text-to-Speech Accent Adaptation