Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers

by   Xilong Wang, et al.

For machine learning with tabular data, Table Transformer (TabTransformer) is a state-of-the-art neural network model, while Differential Privacy (DP) is an essential component to ensure data privacy. In this paper, we explore the benefits of combining these two aspects together in the scenario of transfer learning – differentially private pre-training and fine-tuning of TabTransformers with a variety of parameter-efficient fine-tuning (PEFT) methods, including Adapter, LoRA, and Prompt Tuning. Our extensive experiments on the ACSIncome dataset show that these PEFT methods outperform traditional approaches in terms of the accuracy of the downstream task and the number of trainable parameters, thus achieving an improved trade-off among parameter efficiency, privacy, and accuracy. Our code is available at github.com/IBM/DP-TabTransformer.


page 1

page 2

page 3

page 4


Differentially Private Bias-Term only Fine-tuning of Foundation Models

We study the problem of differentially private (DP) fine-tuning of large...

Selective Pre-training for Private Fine-tuning

Suppose we want to train text prediction models in email clients or word...

Differentially Private Sharpness-Aware Training

Training deep learning models with differential privacy (DP) results in ...

Differentially Private Adapters for Parameter Efficient Acoustic Modeling

In this work, we devise a parameter-efficient solution to bring differen...

Mixed Differential Privacy in Computer Vision

We introduce AdaMix, an adaptive differentially private algorithm for tr...

Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy

Large convolutional neural networks (CNN) can be difficult to train in t...

Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classification

Scalability is a significant challenge when it comes to applying differe...

Please sign up or login with your details

Forgot password? Click here to reset