LoRA: Low-Rank Adaptation of Large Language Models
Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, Chen
2021 ICLR 2022
TL;DR
Fine-tune giant models by only training small low-rank update matrices instead of all the original weights. Cuts trainable parameters by 10,000x with no quality loss.