Parameter-Efficient Fine-Tuning

1. Simple, but efficient, comparable to full model fine-tuning
2. Can learn downstream tasks sequentially, which helps to deploy efficiently
3. For each downstream task, only very small number of parameters should be stored

• For BERT backbone, 0.1% parameters are modified (retrained)

Add Adapter module to each layer of pretrain models

Use prefixs, which can be learned and specific to given tasks, to mimic prompts

• Construct a prompt for each task, concatenate the prompt with data to input to Large Model
• Only add prompt (tokens) at input layer, no MLP A Simplified version of Prefix Tuning

• Adapter Tuning adds Adapter layer in Transformer, which makes the model more deeper and increases the inference time
• Prompt-based methods, such as Prefix Tuning and Prompt Tuning, are hard to train. In addition, the prompt token occupy the input space so as to decrease the number of available tokens