LoRA, step by step
Five small interactive demos that build from a single matrix to a tiny attention model. Each one lets you change the numbers and watch the result. Best explored in order.
W and learn a small correction ΔW = B·A, where B and A are thin matrices of rank r.The matrices themselves
A frozen 6×6 matrix, a target matrix, and a trainable adapter. Edit B and A by hand or train them, and see the calculations, loss, and rank.
A tiny language model (one word of context)
The matrix becomes a next-word table. Choose the dictionary, what the original model says after each word, and what fine-tuning should teach it.
Adding context (two words)
Each row is now a pair of words, so the same word can lead to different next words depending on what came before. Includes “echo the previous word”.
Rearrange a phrase
Write an original phrase and a target phrase made from the same words. The model memorizes the first, and a LoRA adapter teaches it the second.
LoRA on attention
A tiny transformer-style model that decides how much attention to pay to each of the last three words. Put adapters on query, key, value, and output, and watch the attention bars change.
These are teaching toys: the matrices are tiny and the models are far simpler than a real LLM, but the mechanism of freezing weights and training a low-rank correction is the same.