LoRA in miniature

Fine-tuning changes a weight matrix W. LoRA freezes W and learns a small correction ΔW = (α/r)·B·A, where B is d×r and A is r×d. Edit cells (dashed outline) by hand, or press Train to let gradient descent adjust A and B toward the target matrix T.

LoRA rank r = 1 (caps the rank of ΔW) Alpha α = 2 Learning rate = 0.02 What fine-tuning “wants” (target T)

Loss (mean sq. error vs T): –

Trainable params: –

rank(ΔW), computed: –

rank(T − W): –

1 · The small trainable pieces

× α/r =

B starts at all zeros (standard LoRA init), so ΔW = 0 and the model initially behaves exactly like the original.

2 · Frozen weights + correction vs. target

err

Orange = positive, blue = negative. When err is all near zero, the adapted weights W′ match T. A rank-2 target can’t be matched by r=1 (try it), because ΔW can only have rank r.

3 · Effect on an input

Edit the input vector x and compare outputs of the original model, the LoRA-adapted model, and the target.

4 · The math, with your current numbers

Rank vs. r. The rank of a matrix is its number of linearly independent columns (or rows). The slider r is only the inner size of B and A, so rank(ΔW) ≤ r, with equality only if B’s columns and A’s rows are independent. Try r=2 and make both columns of B identical: rank(ΔW) drops to 1. The target can only be matched exactly when rank(ΔW) can reach rank(T − W).

Building one cell of ΔW

row column

The loss

→

Here E is shown at its true scale (the “err” grid above is magnified ×3).

One training step: the gradients

Why bother?

Here the full matrix has 36 numbers and LoRA at r=1 trains 12. At real scale the gap is huge: a 4096×4096 layer has 16,777,216 weights, but rank 8 trains only 2·4096·8 = 65,536 (about 0.4%). The original W is never changed, so you can swap small adapters in and out.