1
00:00:01,500 --> 00:00:04,442
Here are two fine-tuning runs.

2
00:00:04,734 --> 00:00:10,108
One is LoRA+, which trains its two factors
at different learning rates.

3
00:00:10,389 --> 00:00:15,063
The other is plain LoRA, with its scale,
alpha, turned up.

4
00:00:15,254 --> 00:00:22,093
Train both with Adam, with epsilon, weight
decay and clipping switched off,

5
00:00:22,143 --> 00:00:24,239
and watch their weights.

6
00:00:27,247 --> 00:00:28,469
They're identical.

7
00:00:29,244 --> 00:00:30,244
Not close.

8
00:00:30,440 --> 00:00:32,759
Identical, step after step.

9
00:00:33,642 --> 00:00:38,296
That isn't luck, and it isn't the only
coincidence of its kind.

10
00:00:38,569 --> 00:00:42,842
Parameter-efficient fine-tuning now has
hundreds of methods;

11
00:00:42,892 --> 00:00:46,402
this atlas lists three hundred and eighty-
two.

12
00:00:46,832 --> 00:00:51,359
We usually compare them with two numbers:
how many parameters they train,

13
00:00:51,409 --> 00:00:53,119
and how well they score.

14
00:00:53,676 --> 00:00:56,820
Those numbers can't see a coincidence like
this one.

15
00:00:57,177 --> 00:01:00,479
To see it, we have to compare methods by
their shape.

16
00:01:10,883 --> 00:01:17,071
The mathematician Alexander Grothendieck
described two ways to open a hard nut.

17
00:01:17,316 --> 00:01:22,009
You can take a hammer and chisel and
strike, point after point,

18
00:01:22,059 --> 00:01:24,013
until the shell cracks.

19
00:01:24,411 --> 00:01:31,215
Or you can soak it, and let time pass,
until the pressure of a hand is enough.

20
00:01:31,708 --> 00:01:36,448
On a larger scale, he called this second
way the rising sea.

21
00:01:36,822 --> 00:01:42,762
So slowly that nothing seems to happen,
the water rises around the problem,

22
00:01:42,812 --> 00:01:47,151
until it dissolves into a theory that
reaches far beyond it.

23
00:01:47,740 --> 00:01:50,650
Here are four nuts from fine-tuning.

24
00:01:51,088 --> 00:01:52,246
You've already met one.

25
00:01:54,361 --> 00:01:59,995
No exact initialisation lets LoRA reach an
update of rank above twice its own.

26
00:02:00,475 --> 00:02:04,290
LoRA+ is LoRA with alpha rescaled.

27
00:02:04,950 --> 00:02:07,895
GaLore is one-sided LoRA with a moving
frame.

28
00:02:07,945 --> 00:02:12,583
And exactly orthogonal methods never
change a weight's singular values,

29
00:02:12,633 --> 00:02:14,771
however often you merge them.

30
00:02:14,965 --> 00:02:19,949
Each sounds like a separate fact with its
own clever proof,

31
00:02:19,999 --> 00:02:23,270
and most were first proved that way.

32
00:02:23,879 --> 00:02:25,350
We won't crack them again.

33
00:02:25,599 --> 00:02:28,704
We'll raise the sea, and watch them
dissolve.

34
00:02:36,504 --> 00:02:38,630
Start with the pretrained model.

35
00:02:38,955 --> 00:02:44,142
All of its weights together are a single
point in an enormous space.

36
00:02:44,265 --> 00:02:46,648
Call it θ₀.

37
00:02:46,836 --> 00:02:49,317
Fine-tuning moves that point.

38
00:02:49,714 --> 00:02:52,310
Full fine-tuning can move it anywhere.

39
00:02:53,186 --> 00:02:57,954
A parameter-efficient method trains only a
few numbers.

40
00:02:58,048 --> 00:03:00,630
So think of it as a map, rho,

41
00:03:00,680 --> 00:03:07,597
from a small space of trainable parameters
into weight space, with one rule:

42
00:03:08,062 --> 00:03:12,078
the starting parameters land exactly on
θ₀.

43
00:03:15,952 --> 00:03:20,709
Maps like this form what mathematicians
call a slice category.

44
00:03:20,919 --> 00:03:26,239
An arrow from one method to another means
the second can simulate the first;

45
00:03:26,586 --> 00:03:31,631
in particular, wherever the first can take
θ₀, the second can follow.

46
00:03:32,499 --> 00:03:36,879
At one end sits the frozen model, which
can't move at all.

47
00:03:37,094 --> 00:03:42,153
At the other sits full fine-tuning, which
can simulate everything.

48
00:03:42,539 --> 00:03:47,129
Every method that starts exactly at θ₀
lives in between.

49
00:03:52,233 --> 00:03:56,156
So instead of asking how many parameters a
method has,

50
00:03:56,206 --> 00:03:59,417
we can ask three questions about its map.

51
00:04:00,021 --> 00:04:01,397
What is its image: where can it go?

52
00:04:01,447 --> 00:04:07,243
What are its fibres: which parameter
settings give the same weights,

53
00:04:07,293 --> 00:04:09,626
and how does training move through them?

54
00:04:10,167 --> 00:04:13,402
And what happens when the base point
itself moves?

55
00:04:14,341 --> 00:04:16,791
Its image is what it can say.

56
00:04:17,060 --> 00:04:19,366
Its fibres are how it learns.

57
00:04:19,929 --> 00:04:22,187
Base change is how it travels.

58
00:04:30,259 --> 00:04:32,791
Let's look at images.

59
00:04:32,979 --> 00:04:36,559
LoRA adds a low-rank matrix to each
weight,

60
00:04:36,609 --> 00:04:41,727
and the matrices of rank at most r don't
form a flat space.

61
00:04:41,967 --> 00:04:47,902
They form a cone, and for standard LoRA
its tip sits right at θ₀.

62
00:04:48,300 --> 00:04:54,549
OFT rotates the weights, so it moves θ₀
along an orbit.

63
00:04:54,861 --> 00:04:58,162
(IA)³ rescales neurons; in this picture,

64
00:04:58,546 --> 00:05:02,322
it slides along a line through the origin.

65
00:05:02,604 --> 00:05:09,597
And BitFit, which trains only the biases,
moves parallel to a few coordinate axes.

66
00:05:10,477 --> 00:05:12,879
Images also expose disguises.

67
00:05:12,929 --> 00:05:18,765
DoRA, which splits each weight into a
magnitude and a direction,

68
00:05:18,815 --> 00:05:25,326
reaches exactly the same weights as LoRA
followed by one gain per output neuron.

69
00:05:25,376 --> 00:05:31,792
Beyond those gains, anything DoRA adds, it
adds in training, not in reach.

70
00:05:34,712 --> 00:05:37,862
Where you start on an image matters.

71
00:05:38,097 --> 00:05:42,870
At a smooth point, the directions you can
move in, to first order,

72
00:05:42,920 --> 00:05:47,125
form a flat plane as large as the image
itself.

73
00:05:47,400 --> 00:05:49,735
At the tip of a cone, they don't.

74
00:05:50,654 --> 00:05:56,406
Zero-initialised LoRA, the usual default,
starts exactly at the tip.

75
00:05:56,970 --> 00:06:03,500
With B at zero, nudging A changes nothing,
so at first only B can move the weights,

76
00:06:03,550 --> 00:06:06,035
and fewer directions are open.

77
00:06:06,918 --> 00:06:12,150
Split initialisations, which carve a low-
rank piece out of the frozen weight,

78
00:06:12,200 --> 00:06:14,617
start at a smooth point instead.

79
00:06:15,333 --> 00:06:19,836
The same lens finds tips where you might
not expect them: HRA,

80
00:06:19,886 --> 00:06:22,235
a method built from reflections,

81
00:06:22,285 --> 00:06:26,671
also starts at one at Hugging Face's
default initialisation,

82
00:06:26,721 --> 00:06:31,189
when r is even and the frozen weight has
full column rank.

83
00:06:33,498 --> 00:06:35,219
Now the first nut.

84
00:06:35,425 --> 00:06:39,844
Split initialisations start with a nonzero
adapter,

85
00:06:40,047 --> 00:06:43,029
and subtract that piece from the frozen
weights,

86
00:06:43,167 --> 00:06:46,464
so the model still begins at θ₀.

87
00:06:46,754 --> 00:06:47,853
In the picture,

88
00:06:47,903 --> 00:06:53,251
that's sliding the whole cone until it
passes through θ₀ again.

89
00:06:53,670 --> 00:06:55,852
Slide it every possible way,

90
00:06:55,902 --> 00:07:02,924
and together the copies cover θ₀ plus all
differences of two points on the cone.

91
00:07:03,201 --> 00:07:08,756
The difference of two matrices of rank at
most r has rank at most two r.

92
00:07:09,005 --> 00:07:12,831
So no exact initialisation, zero or split,

93
00:07:12,881 --> 00:07:17,278
lets LoRA reach an update of rank above
two r.

94
00:07:17,795 --> 00:07:20,172
Practitioners already knew this ceiling:

95
00:07:20,243 --> 00:07:25,227
converting a split adapter back into plain
LoRA takes rank two r.

96
00:07:25,475 --> 00:07:28,290
The picture adds that two r is really
needed,

97
00:07:28,340 --> 00:07:32,141
and that the same few lines work for any
additive method.

98
00:07:37,591 --> 00:07:40,157
Rank has a picture of its own.

99
00:07:40,843 --> 00:07:44,791
Draw an adapter as boxes joined by wires.

100
00:07:45,095 --> 00:07:49,269
Cut the diagram anywhere between input and
output,

101
00:07:49,355 --> 00:07:54,593
and the rank can't exceed the product of
the sizes of the wires you cut.

102
00:07:54,922 --> 00:08:00,059
For LoRA, the narrowest cut is the single
wire of size r.

103
00:08:00,427 --> 00:08:05,175
LoHa multiplies two low-rank factors entry
by entry,

104
00:08:05,367 --> 00:08:08,709
and its narrowest cut is r times r.

105
00:08:09,155 --> 00:08:11,693
But rank is a cut, and dimension,

106
00:08:11,743 --> 00:08:18,082
the number of independent directions a
method can actually move in, is a volume.

107
00:08:18,593 --> 00:08:25,693
At the same budget as LoRA of rank two r,
LoHa reaches rank up to r squared,

108
00:08:25,948 --> 00:08:29,749
but on large layers it gives up directions
to get there:

109
00:08:29,927 --> 00:08:34,225
a trade of dimension for rank when r is
three or more,

110
00:08:34,275 --> 00:08:37,529
and a pure loss when r is two or less.

111
00:08:45,931 --> 00:08:48,326
Images say where a method can go.

112
00:08:48,802 --> 00:08:50,984
Fibres say how it gets there.

113
00:08:51,470 --> 00:08:54,318
Take a LoRA update, B times A,

114
00:08:54,368 --> 00:08:59,832
and slip any invertible r-by-r matrix g
into the middle: B times g-inverse,

115
00:09:00,273 --> 00:09:01,692
times g times A.

116
00:09:01,742 --> 00:09:03,634
The factors change.

117
00:09:03,738 --> 00:09:05,801
Their product doesn't.

118
00:09:05,923 --> 00:09:10,888
All of these settings give the same
weights.

119
00:09:11,256 --> 00:09:14,780
When the product has full rank, they make
up the whole fibre,

120
00:09:15,044 --> 00:09:18,287
and the group of all such g is LoRA's
gauge.

121
00:09:18,820 --> 00:09:23,958
Training moves through the fibres too, and
the symmetry leaves a fingerprint.

122
00:09:24,570 --> 00:09:28,236
Under gradient flow, one quantity never
changes:

123
00:09:28,453 --> 00:09:34,246
the balance between the two factors, BᵀB −
AAᵀ,

124
00:09:34,344 --> 00:09:36,839
each divided by its own learning rate.

125
00:09:36,969 --> 00:09:41,935
It's a conserved charge, known for
factorised networks since 2018,

126
00:09:42,123 --> 00:09:44,217
in the spirit of Noether's theorem.

127
00:09:44,630 --> 00:09:50,576
With plain SGD steps, it drifts, but only
at second order in the step size.

128
00:09:50,974 --> 00:09:53,597
Under Adam, it isn't conserved at all.

129
00:09:55,943 --> 00:09:57,511
Here's the surprise.

130
00:09:57,838 --> 00:10:03,528
The gauge is a whole group, but the usual
optimizers respect only part of it.

131
00:10:04,236 --> 00:10:07,929
Gradient descent respects the rotations
and reflections.

132
00:10:08,143 --> 00:10:11,888
Adam respects far less: only the signed
permutations,

133
00:10:11,938 --> 00:10:15,068
which shuffle the r channels and flip
their signs.

134
00:10:15,687 --> 00:10:20,134
Start Adam from two initialisations
related by a generic rotation,

135
00:10:20,184 --> 00:10:22,112
and the runs drift apart.

136
00:10:22,594 --> 00:10:26,239
Relate them by a signed permutation, and
they stay identical.

137
00:10:26,632 --> 00:10:30,443
The optimizer sees more of where you start
than the image does.

138
00:10:35,040 --> 00:10:38,896
One more symmetry hides in the
hyperparameters.

139
00:10:39,076 --> 00:10:44,340
Rescale a block's starting size and its
learning rate, adjust alpha to match,

140
00:10:44,390 --> 00:10:49,802
and under Adam, with epsilon, decay and
clipping off,

141
00:10:49,995 --> 00:10:52,758
the weights follow exactly the same path.

142
00:10:53,344 --> 00:10:56,044
These rescalings form a torus.

143
00:10:56,594 --> 00:11:02,198
LoRA+, which gives B a learning rate
lambda times larger than A's,

144
00:11:02,248 --> 00:11:04,929
is just a point on that torus.

145
00:11:05,462 --> 00:11:09,558
That's the coincidence from the start, and
our second nut:

146
00:11:09,811 --> 00:11:11,808
from the usual zero start,

147
00:11:11,923 --> 00:11:18,606
LoRA+ and plain LoRA with alpha times
lambda take the same path, exactly.

148
00:11:19,210 --> 00:11:25,678
Schulman and colleagues found this for
LoRA and Adam in 2025, epsilon included.

149
00:11:25,928 --> 00:11:27,756
The torus carries it to SGD,

150
00:11:27,806 --> 00:11:31,813
and to any method whose update is a
product of its blocks.

151
00:11:42,668 --> 00:11:47,279
Some methods add no adapter at all: they
train the weights directly,

152
00:11:47,329 --> 00:11:49,037
but compress the gradient.

153
00:11:49,273 --> 00:11:55,272
GaLore projects each gradient onto a small
subspace and runs Adam there.

154
00:11:55,572 --> 00:12:01,378
Backpropagation has the shape of a lens: a
forward map paired with a backward one.

155
00:12:01,650 --> 00:12:02,798
Through that lens,

156
00:12:02,848 --> 00:12:07,827
a linear adapter and a compressed gradient
are two views of the same thing.

157
00:12:08,296 --> 00:12:13,181
So the third nut: without weight decay or
gradient clipping,

158
00:12:13,317 --> 00:12:18,548
GaLore is exactly one-sided LoRA, with
GaLore's projector as the frozen factor,

159
00:12:18,598 --> 00:12:23,338
merged and restarted every period, its
optimizer state carried over.

160
00:12:23,607 --> 00:12:27,858
Torroba-Hennigen and colleagues proved
this for GaLore in 2025.

161
00:12:28,113 --> 00:12:32,506
The lens gives it a short proof by
induction, for every linear adapter.

162
00:12:43,429 --> 00:12:46,660
So far θ₀ has stayed put.

163
00:12:46,979 --> 00:12:48,859
But plenty of things move it.

164
00:12:49,235 --> 00:12:54,644
Quantise the model to four bits, and the
frozen weights snap to a grid.

165
00:12:55,157 --> 00:13:00,377
QLoRA is ordinary LoRA pointed at the
snapped weights, not the real ones.

166
00:13:00,427 --> 00:13:06,920
So it starts off target by the
quantisation error, which is generically
full rank,

167
00:13:07,246 --> 00:13:12,646
and it can only get back to the original
point if that error has rank at most r.

168
00:13:14,516 --> 00:13:17,369
Merging moves θ₀ too.

169
00:13:17,992 --> 00:13:21,634
Train an adapter, merge it into the
weights,

170
00:13:21,824 --> 00:13:25,471
and start a fresh one from the new point.

171
00:13:25,521 --> 00:13:31,720
Every cycle re-points the method, and the
reach grows: after K cycles,

172
00:13:31,770 --> 00:13:37,476
ReLoRA reaches exactly the updates of rank
at most K times r.

173
00:13:39,133 --> 00:13:41,401
Now the last nut.

174
00:13:41,652 --> 00:13:46,060
A method that rotates the weights from the
input side, exactly,

175
00:13:46,110 --> 00:13:49,749
never changes the inner products between
neurons.

176
00:13:50,105 --> 00:13:52,942
That is what OFT was designed for.

177
00:13:52,992 --> 00:13:56,872
So it never changes a single singular
value,

178
00:13:57,037 --> 00:14:01,592
and in exact arithmetic you can merge as
often as you like:

179
00:14:02,115 --> 00:14:05,641
a rotation times a rotation is still a
rotation.

180
00:14:06,022 --> 00:14:09,524
One caution that matters in practice.

181
00:14:09,991 --> 00:14:20,049
The default OFT in recent versions of
Hugging Face's PEFT library uses an
approximation that isn't exactly
orthogonal.

182
00:14:20,228 --> 00:14:24,417
Each merge shrinks the weights a little,
and in the paper's test,

183
00:14:24,494 --> 00:14:28,962
two hundred merges move the neuron inner
products by about a fifth.

184
00:14:40,929 --> 00:14:44,682
Do this for every method, and you get an
atlas.

185
00:14:44,962 --> 00:14:50,251
The paper reads seven coordinates off each
method's formula: where it acts;

186
00:14:50,588 --> 00:14:57,232
the shape of its image; where it starts,
at a tip, a smooth point, or off target;

187
00:14:57,550 --> 00:15:03,646
its gauge; which changes of basis it
respects; whether it can be merged;

188
00:15:03,784 --> 00:15:06,433
and what merge-and-restart can reach.

189
00:15:06,951 --> 00:15:12,281
Three hundred and eighty-two methods from
a hundred and thirty-eight papers.

190
00:15:12,565 --> 00:15:16,771
Four hundred and three arrows saying which
method simulates which,

191
00:15:16,920 --> 00:15:18,625
each with an explicit map,

192
00:15:19,020 --> 00:15:23,717
and four hundred and eighty-five
obstructions saying which can't.

193
00:15:25,248 --> 00:15:27,960
A theory should also risk being wrong,

194
00:15:28,189 --> 00:15:32,292
so the atlas makes predictions a training
run can test.

195
00:15:32,554 --> 00:15:38,236
Under Adam, a fair LoRA+ baseline should
multiply alpha by lambda.

196
00:15:38,395 --> 00:15:43,638
LoRA's charge should drift by a second-
order amount per step of SGD,

197
00:15:43,688 --> 00:15:47,543
a cheap diagnostic of what your optimizer
adds.

198
00:15:47,949 --> 00:15:50,684
And it marks its own limits.

199
00:15:51,159 --> 00:15:54,310
Reaching a weight isn't the same as
learning it.

200
00:15:54,510 --> 00:15:58,368
Most results are about images, in exact
arithmetic.

201
00:15:58,789 --> 00:16:02,558
And words like gauge and Noether are
analogies;

202
00:16:02,873 --> 00:16:05,777
the paper says exactly where each one
stops.

203
00:16:06,700 --> 00:16:12,006
So the next time a new method appears, you
don't have to reach for the hammer.

204
00:16:12,420 --> 00:16:13,947
Ask three questions.

205
00:16:14,495 --> 00:16:16,069
What can its map reach?

206
00:16:16,644 --> 00:16:19,688
How do its fibres meet the optimizer?

207
00:16:20,024 --> 00:16:22,691
And what happens when the base point
moves?

208
00:16:23,470 --> 00:16:26,519
Often, the answer is already under the
water.

209
00:16:27,521 --> 00:16:29,979
The proofs, the full catalogue,

210
00:16:30,029 --> 00:16:35,096
and twenty interactive figures are in the
paper and on the companion site.

211
00:16:35,146 --> 00:16:36,624
The link is below.

212
00:16:36,993 --> 00:16:38,132
Thanks for watching.