A Categorical Atlas of Parameter-Efficient Fine-Tuning

Plays from YouTube

1. 0:00Two runs, one path
2. 1:10The nut and the sea
3. 2:30I · Fine-tuning moves a point
4. 4:24II · The image: what a method can say
5. 8:40III · Fibres: how a method learns
6. 11:36IV · The lens: GaLore
7. 12:37V · Base change: how a method travels
8. 14:35The atlas: seven coordinates, 382 methods
9. 16:06Three questions

Abstract

Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. This site compares them as maps. A pretrained checkpoint \(\theta_0\) is a base point in weight space \(\Theta\), and a weight-space PEFT method is a pointed smooth map \(\rho:(Q,q_0)\to(\Theta,\theta_0)\) from a small parameter space. These maps form the pointed slice category \(\mathsf{Meth}(\Theta,\theta_0)\), where the frozen model is initial and full fine-tuning terminal, so every method pointed at \(\theta_0\) sits in the interval between them.

Model referees (see How this atlas was made) checked all 41 numbered results against their proofs, the literature and the Hugging Face PEFT source. They repaired 27, a second review narrowed many of those, and only the narrowed forms appear. Some results are new, such as the HRA apex; many are known and re-derived with credit.

Three invariants

Image germ · what a method can say

Near \(\theta_0\), the weights a method reaches form a linear space, a cone, a group orbit, or a torus saturation or meet of these. Its first-order image \(S_1\), expressive dimension \(d(M)\), min-cut rank bound and base-point type are read off this germ. Zero-init LoRA starts at the apex of the cone \(\theta_0+\mathcal M_{\le r}\), where first steps miss directions it reaches later; a PiSSA-style split starts at a smooth point (Exposés II.A, II.B and II.C).

Fibres and metric · how it learns

Many parameter values give the same weight; for LoRA these fibres are \(GL_r\) gauge orbits. The optimizer's rates push forward to a cometric \(K\) on weight space, so isomorphic methods can train differently. With per-block rates, the compact half of LoRA's gauge acts by isometries and the non-compact half yields charges conserved under gradient flow. Through the backpropagation lens, GaLore without weight decay or full-gradient clipping is one-sided LoRA with a moving frozen factor and carried-over optimizer state (Prop IV.2; Exposés III and IV).

Base change · how it travels

Splitting initialisations, quantisation, restarts and merging move the base point. QLoRA is LoRA pointed at the quantised weights \(\kappa\theta_0\), with defect \(e=\kappa\theta_0-\theta_0\), and its image misses \(\theta_0\) whenever \(\operatorname{rank}e\gt r\), the generic case (Prop V.2). Merge-and-restart with a fixed shape reaches the orbit of the monoid one cycle generates; for ReLoRA, every update of rank at most \(Kr\) after \(K\) cycles (Thm V.3; Exposé V).

Adapters, prefixes and ReFT extend the network; seven local rewrite rules with side conditions certify when they merge back (Exposé VI).

Things you can check in your browser

1. LoRA+ is \(\alpha\) in disguise. For zero-init LoRA under Adam with \(\varepsilon=0\), no weight decay or clipping, one schedule for both groups and the same data order and dropout masks, LoRA+ with ratio \(\lambda\) follows, in exact arithmetic, the same weight trajectory as LoRA with \(\alpha\) times \(\lambda\). Thm III.10 · Cor III.11 · orbit explorer
2. No initialisation buys rank above \(2r\). No exact pointing of \(\mathrm{LoRA}_r\), zero-init or split, reaches an update of rank above \(2r\). For \(r\lt\min(m,n)\), its full-rank pointings form three strata, classified as a set by \(\mathrm{Gr}(r,n)\sqcup\mathrm{Gr}(r,m)\sqcup\mathcal M_r\). Thm II.1 · Thm II.2 · apex and smooth point
3. Two apexes. Zero-init LoRA misses \(r(n-r)\) first-order directions. HRA without Gram–Schmidt, at HF PEFT's default paired initialisation, is an apex too when \(W_0\) has full column rank and \(r\) is even with \(2\le r\le n-2\). Thm II.2 · Thm II.7 · apex · Gram lab
4. Orthogonal methods keep the spectrum. If every merged weight is \(W_0R\) with \(R\) exactly orthogonal (exact-Cayley OFT, HRA, and GOFT and ETHER as in their papers), the neuron Gram matrix \(WW^\top\) and the singular values never change in exact arithmetic, through any number of merges and restarts. HF PEFT's default Cayley–Neumann OFT is only approximately orthogonal. Cor II.6 · Thm V.3 · Gram lab
5. Rank is a cut, dimension is a volume. At equal budget, with \(2r\le\min(m,n)\), \(r^2\le\min(m,n)\) and \(2r^2\lt m+n-1\), \(\mathrm{LoHa}_{r,r}\) reaches at least \((m+n-1)-2r^2\) fewer dimensions than \(\mathrm{LoRA}_{2r}\), buying rank up to \(r^2\) for \(r\ge3\) and strictly dominated for \(r\le2\). GraLoRA keeps LoRA's dimension and lifts the maximum rank to \(\min(kr,m,n)\), but caps each block at rank \(r/k\); for \(k\ge2\) and \(kr\le\min(m,n)\) the images are incomparable. Prop II.12 · rank is not volume
6. Kronecker sums cut the other way. With equal square splits of a \(d\times d\) weight, a generic rank-one update has the largest Kronecker rank, \(d\), while the identity has Kronecker rank one, so zero-init \(\mathrm{LoRA}_r\) and unconstrained \(k\)-term Kronecker sums, for \(r,k\lt d\), have incomparable images. Thm II.11 · cut the tensor
7. A conserved charge. With constant per-block rates, LoRA's charge \(\Phi=B^\top B/\eta_B-AA^\top/\eta_A\) is conserved by every continuous-time update \(\dot q=-\Lambda D\rho^\top\gamma\), gradient or not, including minibatches, global-norm clipping and DoRA's detached norm. Discrete steps make the charge drift, with relative drift of second order in the step size; Adam and momentum are not covered. Thm III.5 · Noether hyperbolas
8. Adam sees a finite shadow of the gauge. On LoRA's full-rank locus, with channel-uniform hyperparameters, the largest part of \(GL_r\) an update rule respects is \(O(r)\) for SGD, the \(2^rr!\) signed permutations for Adam and AdamW, and all of \(GL_r\) for undamped scaled GD. Thm III.12 · gauge lab
9. The activation decides the merge. (IA)\(^3\)'s feed-forward scale always folds into \(W_{\rm down}\). It folds into \(W_{\rm up}\) under SwiGLU for every scale and under ReLU for nonnegative ones, while under GELU only entries \(0\) and \(1\) fold. Thm VI.3 · merge game

How to read the labels

Notation

Papers that store \(W^\top\) swap input and output; we translate every method into this convention.

Readers new to category theory or Lie groups can start with the appendix, which collects the background the site assumes.

The thesis

Fine-tuning papers usually compare methods by their number of trainable parameters. That number does not say what a method can reach, how it trains, or what merging does. Those are properties of a map.

A pretrained checkpoint \(\theta_0\) is a point in weight space \(\Theta\), and fine-tuning moves it. A parameter-efficient method chooses where the point may move, through a pointed map

\[\rho:(Q,q_0)\longrightarrow(\Theta,\theta_0)\]

from a small space of trainable parameters into weight space, which sends the initialisation \(q_0\) to the checkpoint. LoRA is the map \((B,A)\mapsto W_0+sBA\) on each adapted matrix. Orthogonal fine-tuning (OFT) is \(Q\mapsto W_0\,C(Q)\), where \(C\) is the Cayley map of a skew matrix. Full fine-tuning is the identity of \(\Theta\), and the frozen model is the inclusion of the single point \(\theta_0\). Figure 1 shows where each catalogued method sits relative to these two.

Following Grothendieck, we study a morphism over a base, here the pretrained point. Three invariants of \(\rho\) do most of the work, with one further layer for architectures.

Each result in the list of checkable results carries the scope of its proof. For example, LoRA+ with ratio \(\lambda\) follows exactly the trajectory of plain LoRA with \(\alpha\) multiplied by \(\lambda\), for zero-init \(B\) under Adam with \(\varepsilon=0\), no weight decay or gradient clipping, one learning-rate schedule for both groups and the same data order and dropout masks, in exact arithmetic (Thm III.10, Cor III.11).

Where the category theory works

Categorical language is cheap, so we name the seven places where it does real work.

1. The pointed slice gives the interval [frozen, full FT] and makes “simulation” precise (Def I.2, Obs I.4).
2. Isomorphism invariants finer than the image, \(S_1\) and the singular locus, classify LoRA's full-rank initialisations as a set (Obs I.6, Thm II.2).
3. A categorical product produces LoRA-XS. Intersections of images, which are not products, organise others such as HRA (Prop I.8, Thm II.7).
4. Base change along maps that change the function, such as quantisers, creates pointing defects (Prop V.2).
5. The lens 2-functor \(\mathbf{Para}(\mathsf R)\) separates tangent-side from cotangent-side methods. Its degree-one collapse shows that GaLore under Adam, without weight decay or global gradient clipping, is period by period one-sided LoRA on a moving base with its optimizer state carried over (Prop IV.2, after Torroba-Hennigen et al., 2025).
6. Naturality. Gradient flow is natural exactly where cometrics agree, which for LoRA's linear gauge with block-scalar rates means along \(O(r)\) where \(A\) has full row rank and \(r\lt n\). A rule combining LoRA factors is well defined exactly when it is constant on the fibres of \((B,A)\mapsto BA\) (Thm III.3, Prop V.4).
7. Symmetric monoidal structure bounds rank by a min-cut of a tensor-network presentation (Thm II.10).

The rest is real algebraic geometry, invariant theory, Lie generation and gradient-flow ODEs, and we say so where it happens.

More: constructions we leave out

Some constructions are true here and do no work, so we leave them out. They include coends for the rank filtration, a saturation monad, sheaf language for locality, the Grothendieck construction \(\int\mathsf{Meth}\to\mathbf{PNet}\), and coKleisli 2-cells for mixtures.

Pretrained models and methods

A pretrained network is a function with a parameter slot and a chosen value in it. Its natural home is the bicategory of parametrised maps.

More: four objects and one arrow

On one \(m\times n\) box, LoRA (Hu et al., 2021) is \(\rho(B,A)=W_0+sBA\) on \(Q=\mathbb R^{m\times r}\times\mathbb R^{r\times n}\), pointed at \((0,A_0)\). It is additive, with budget \(r(m+n)\). OFT (Qiu et al., 2023) with the exact Cayley map is \(\rho(Q)=W_0\,C(Q)\) for skew \(Q\) (block-diagonal in OFT), pointed at \(Q=0\). It is of action type, with \(\gamma=C\) landing in \(SO(n)\) on the input side. The frozen model is \((\mathrm{pt},*,\theta_0)\), and full fine-tuning is \((\Theta,\theta_0,\mathrm{id})\).

For an arrow, take LoRA-FA (Zhang et al., 2023), \(B\mapsto W_0+sBA_0\) with \(A_0\) frozen. The pointed map \(h(B)=(B,A_0)\) satisfies \(\rho_{\rm LoRA}\circ h=\rho_{\rm FA}\), so LoRA pointed at \((0,A_0)\) simulates LoRA-FA.

What Para adds

More: why the next exposés work in weight space

Reach in \(\Theta\) is a property of the image, and the cometric \(K=D\rho\,\Lambda\,D\rho^\top\) that shapes training is built from \(\rho\) and the optimizer's rates. The network enters training through the gradient it hands to the method, and enters expressivity through \(\Phi_f\) once we ask about functions instead of weights. So Exposés II and III work in \(\Theta\) and flag the places where \(f\) matters.

Frozen and full fine-tuning

(expand)

(a) The only pointed map \(\mathrm{pt}\to Q\) is \(*\mapsto q_0\), and it lies over \(\Theta\) because \(\rho(q_0)=\theta_0\). A morphism into the terminal object must satisfy \(\mathrm{id}\circ h=\rho_M\).

(b) Products in a pointed slice are pullbacks in \(\mathbf{Diff}_*\). The forgetful functor \(\mathbf{Diff}_*\to\mathbf{Diff}\) creates limits, since \(\mathbf{Diff}_*\) is the coslice \(\mathrm{pt}/\mathbf{Diff}\), and the induced map out of a pointed test object is pointed because its components are. Smooth maps landing in an embedded submanifold are smooth into it, which gives the sufficient condition. Transversality at \((q_0,q_0')\) means \(d\rho_M(T_{q_0})+d\rho_N(T_{q_0'})=T_{\theta_0}\Theta\), and the left side has dimension at most \(\lvert M\rvert+\lvert N\rvert\). A point \(\theta\in\mathrm{Im}\,M\cap\mathrm{Im}\,N\) has preimages \(q,q'\), and \((q,q')\) lies in the fibre product. Testing a product in \(\mathsf{Meth}\) against pointed two-point \(0\)-manifolds shows that its points are exactly the pairs \((q,q')\) over a common \(\theta\).

For \(\mathrm{LoRA}_1\times\text{Frozen}\), suppose a product \(P\) exists, with canonical map \(\iota:P\to\mathbb R^2\), and let \(p^*\) be its unique point over the origin \((0,0)\). The forgetful functor \(\mathbf{Diff}_*\to\mathbf{Diff}\) is a right adjoint (its left adjoint adds a disjoint base point), so \(P\) is also the pullback in \(\mathbf{Diff}\), and unpointed smooth curves into the cross lift to \(P\). The axis curves \(t\mapsto(t,0)\) and \(t\mapsto(0,t)\) lift to curves through \(p^*\), so \(d\iota_{p^*}\) hits \((1,0)\) and \((0,1)\) and has rank \(2\). Then \(\iota\) is a submersion near \(p^*\), and its image contains an open subset of \(\mathbb R^2\), which the cross does not. The argument covers every pointing of \(\mathrm{LoRA}_1\), including the standard one \((0,a_0)\) with \(a_0\ne0\), whose base point is a smooth point of the cross.

(c) Functoriality is \(\rho_N\circ h=\rho_M\). For the converse, choose (by the axiom of choice) for each \(\theta\in\mathrm{Im}\,M\) a preimage \(s(\theta)\) under \(\rho_N\) with \(s(\theta_0)=q_0'\), and put \(h=s\circ\rho_M\). In the example the only candidate is the cube root, which is not differentiable at \(0\).∎

Every weight-space method that starts at \(\theta_0\) lies in [frozen, full FT]. Adapters, prefixes and ReFT are extensions outside \(\mathsf{Meth}\) (Exposé VI). Merging into full-precision weights is the canonical morphism to the terminal object, so in exact arithmetic it is definitional; quantised bases and combinations of adapters raise real questions (Prop V.2, Prop V.4). The product, when it exists, is the universal method simulated by both factors, with image \(\mathrm{Im}\,M\cap\mathrm{Im}\,N\). Prop I.8(a) computes one, LoRA-XS, while Prop I.8(b) and Thm II.7 compute intersections of images only.

More: defaults that do not start at the checkpoint

Some defaults move the weights before training starts. Hugging Face PEFT's default FourierFT draws its spectrum from a standard normal, and VB-LoRA's default vector bank and logits are random too. LoRA with init_lora_weights=False, LoRA-XS with its small random \(R\), and every method on a quantised base also start away from \(\theta_0\). Each lives over its own starting weight or carries a pointing defect, which Exposé V measures.

More: lifting obstructions beyond first order

The gap between \(\mathbf{Set}\) and \(\mathbf{Diff}\) in (c) is a smooth lifting obstruction. Obs I.6 below detects its first-order instances. In the cube-root example, \(S_1(M)=\mathbb R\not\subseteq S_1(N)=0\). Higher-order obstructions need finer invariants. The maps \(\rho_M(t)=t^3\) and \(\rho_N(u)=u^5\) have equal images and \(S_1=0\) on both sides, yet the only lift, \(t^{3/5}\), is not \(C^1\).

Invariants of a method

The budget \(\lvert M\rvert\) counts parameters. Two other numbers say more, the dimension of what a method reaches and of what it reaches to first order at the start, which is what the first gradient steps use. For LoRA with \(r=16\) on a \(4096\times4096\) matrix they are \(130{,}816\) and, from zero initialisation, \(65{,}536\), against a budget of \(131{,}072\) (Thm II.2, Prop II.12).

More: cones in this site, and the word “gauge”

Apex and vertex differ in general. At a crossing of two smooth branches the image is singular, yet the deficit can be \(0\). In every case below with a positive deficit, the image germ is (diffeomorphic to) a cone with vertex \(\theta_0\). The exception is DoRA, which inherits LoRA's deficit through its torus factor.

“Gauge” here means the symmetry group of a parametrisation, as in the definition above. There is no connection, and away from the full-rank stratum the fibres jump. The word borrows the physicist's picture of redundant coordinates that leave every observation unchanged. Only the group and its action enter the theorems.

If \(N\) simulates \(M\), whatever \(M\) can do at its first step, \(N\) can do too.

(expand)

Apply the chain rule \(d\rho_N\circ dh=d\rho_M\) at \(q_0\), using \(h(q_0)=q_0'\). If \(h\) is an isomorphism, \(dh_q\) is invertible for every \(q\), so \(\operatorname{rank}d\rho_M(q)=\operatorname{rank}d\rho_N(h(q))\). Hence \(d(M)=d(N)\) (equivalently \(\mathrm{Im}\,M=\mathrm{Im}\,N\)), and since \(dh_{q_0}\) is surjective, \(S_1(M)=S_1(N)\). So the deficits agree.∎

\(S_1\) is what linearised (lazy, NTK) dynamics can use at initialisation, since the lazy kernel \(J_f\,d\rho\,\Lambda\,d\rho^\top J_f^\top\) has range \(J_f(S_1)\). The kernel also depends on \(\Lambda\) and the network Jacobian \(J_f\), and for random-\(A_0\) zero-init LoRA it is close to full fine-tuning's, so a large first-order deficit need not cost much on data. \(S_1\) separates initialisations the image cannot, namely zero-init \(\mathrm T_A\) against \(\mathrm T_B\), and the choice of \(\mathrm{row}\,A_0\) (Thm II.2).

Sums

Weight space is a vector space, so running two methods side by side and adding their updates gives a method again.

(expand)

\([B_1\ B_2][A_1;A_2]=B_1A_1+B_2A_2\), and \(h\) is a linear bijection. The coherence isomorphisms of \(\oplus\) are those of the cartesian structure on \(\mathbf{Diff}_*\), and they lie over \(\Theta\) because \(+\) is associative and commutative. The pointing claim is \(\dim(U+V)=r+s-\dim(U\cap V)\) for \(U=\mathrm{row}\,A_0\) and \(V=\mathrm{row}\,A_0'\).

For the failure of strong monoidality, recall that any pointed isomorphism preserves \(S_1\) (Obs I.6). If \(F(r)\) is a standard \(\mathrm T_A\) pointing and \(F(r)\oplus F(r)\cong F(2r)\), then \(S_1(F(r)\oplus F(r))=\{XA_0+X'A_0\}\) has dimension \(mr\), while a standard \(F(2r)\) has \(\dim S_1=2mr\). A standard \(F(2r)\) needs \(2r\lt\min(m,n)\) (Thm II.2). The degenerate cases are direct.∎

[Num: \(\dim S_1=28\) for a standard \(\mathrm{LoRA}_4\) at \(m=7\), against \(14\) for \(\mathrm{LoRA}_2\oplus\mathrm{LoRA}_2\) with the same \(A_0\).]

RoSA (LoRA \(\oplus\) a fixed-mask sparse method), GraLoRA (\(\oplus\) of \(k^2\) block-embedded LoRAs) and ReLoRA, read as \(\mathrm{LoRA}^{\oplus K}\) unrolled in time apart from its full-rank warm start, are \(\oplus\)-words with Minkowski-sum images. A sum of \(k\) rank-\(r\) LoRAs with residual \(\theta_0\) is isomorphic in \(\mathsf{Meth}\) to one LoRA of rank \(kr\). Under \(\alpha/r\) scaling the isomorphism rescales the \(A\) blocks, so at equal \(\alpha\) and learning rate the two train differently. Hugging Face PEFT's cat merge is this rescaled concatenation applied to trained adapters, exact for unit weights and plain \(\alpha/r\) sources.

More: what is and is not a sum, and the cat merge

LoRA-soup CAT freezes the trained modules and learns layer-wise weights, so it is a \(\oplus\) of \(k\) frozen line methods \(\alpha_i\mapsto\alpha_iB_iA_i\). Its per-layer image is a span of dimension at most \(k\), much smaller than \(\mathrm{LoRA}_{kr}\). UniPELT and MAM are not \(\oplus\)-words in \(\mathsf{Meth}(\Theta,\theta_0)\). Their adapter and prefix parts live on extensions (Exposé VI), and UniPELT's gates depend on the input.

The obstruction in the observation is the reuse of one pointing, since stacking the same \(A_0\) twice gives a rank-deficient frame. For \(r\ne s\), standard pointings satisfy \(F(r)\oplus F(s)\cong F(r+s)\) exactly when the row space of the rank-\((r+s)\) frame is the direct sum of the other two. Concatenation is strictly associative, so discrete \((\mathbb N,+)\) suffices for a plain strong monoidal functor on unpointed objects. Symmetric strong monoidality needs the permutation groupoid, since swapping and then concatenating permutes the blocks.

“Additive” in Def I.2 (\(\delta\) independent of \(\theta_0\)) is a condition on families \(\theta_0\mapsto M(\theta_0)\). It matters for base change (Exposé V) and plays no role in \(\oplus\).

With scales \(s_r=\alpha/r\) (or \(\alpha/\sqrt r\) for rsLoRA) and any target scale \(s_*\), the rescaled concatenation \(([B_1\ B_2],[c_1A_1;c_2A_2])\) with \(c_i=s_{r_i}/s_*\), where \(r_1=r\) and \(r_2=s\), is an isomorphism over \(\Theta\). It preserves row spaces, so the pointing criterion is unchanged. Hugging Face PEFT's add_weighted_adapter(..., combination_type="cat") takes \(s_*=1\). It multiplies each \(A\) block by \(w_is_{r_i}\), leaves the \(B\) blocks alone, and sets the merged lora_alpha to \(r+s\), so the merged scale is \(1\). In PEFT releases up to v0.21.1 the merged adapter also inherits use_rslora from the first source. For rsLoRA sources its scale is then \((r+s)/\sqrt{r+s}=\sqrt{r+s}\), and the merged update is \(\sqrt{r+s}\) times too large. The main branch now sets use_rslora=False on the merged adapter.

Products

Freeze \(A\) at \(A_0\) and train \(B\), or freeze \(B\) at \(B_0\) and train \(A\). The product of these two one-sided methods is LoRA-XS.

(expand)

(a) A point of the fibre product is a pair \((B,A)\) with \(BA_0=B_0A=:D\). Then \(\mathrm{row}\,D\subseteq\mathrm{row}\,A_0\) and \(\mathrm{col}\,D\subseteq\mathrm{col}\,B_0\), so \(D=B_0RA_0\) for a unique \(R\). Since \(A_0\) has full row rank, \(BA_0=B_0RA_0\) forces \(B=B_0R\), and likewise \(A=RA_0\). So the fibre product is the \(r^2\)-dimensional linear subspace \(\{(B_0R,RA_0)\}\), and \(R\mapsto(B_0R,RA_0)\) is a smooth bijection onto it with smooth inverse \((B,A)\mapsto B_0^+B\). Any pair of maps \((h_1,h_2):T\to Q_{\rm FA}\times Q_{\rm FB}\) that agree over \(\Theta\) factors through it via \(t\mapsto B_0^+h_1(t)\), so this is the product in \(\mathbf{Diff}\). If \(\operatorname{rank}W_0\lt r\), then \(\Sigma_rV_r^\top\) is rank-deficient and the LoRA-XS instance breaks.

(b) \(\mathrm{diag}(\ell)W_0-W_0=\mathrm{diag}(\ell-\mathbf 1)W_0\), and its rank is \(\#\{i:\ell_i\ne1\}\) because the rows of \(W_0\) are independent.∎

[Num: the fibre product of (a) has dimension \(4=r^2\) at \((m,n,r)=(6,8,2)\).]

LoRA-XS is the largest method that factors through both frozen frames. At a split initialisation it is the overlap the split double-counts (Prop II.3), and it is LoRA-XS proper at a PiSSA initialisation, whose frames span the top-\(r\) singular subspaces of \(W_0\). The meet in (b) is only set-level. For \(n\ge2\) no product exists in \(\mathbf{Diff}\): its fibre product is singular at the origin \((b,a,\ell)=(0,0,\mathbf 1)\), where its branches cross, by the rank argument for the cross in Obs I.4(b). (For \(m=n=1\) the fibre product is a graph, and the product exists.) With a fixed support, the meet is (IA)³ masked to \(r\) neurons.

More: the shape of the meet in (b)

The meet is the union of the \(\binom mr\) coordinate \(r\)-planes “rescale at most \(r\) output neurons”. For \(r=1\) the fibre product contains the sheets \(\{b=0\}\) and \(\{a=0\}\), which cross at the origin. The standard base point, with \(b=0\) and \(a_0\ne0\), lies on a smooth sheet, so the singularity sits away from it.

The dictionary so far

The theorems begin when we look closely at one image near one point, which is where Exposé II starts. The full dictionary is the Rosetta stone.

More: the dictionary for this exposé

In PEFT	In the pointed slice	Statement
pretrained checkpoint	base point \(\theta_0\) of a pointed 1-cell \((\Theta,\theta_0,f)\)	Def I.1
a weight-space PEFT method	an object \(\rho:(Q,q_0)\to(\Theta,\theta_0)\) of \(\mathsf{Meth}(\Theta,\theta_0)\)	Def I.2
“\(N\) can do whatever \(M\) can”	a morphism \(h:M\to N\), \(\rho_N\circ h=\rho_M\); at the level of sets, image inclusion	Def I.2, Obs I.4(c)
frozen model, full fine-tuning	initial and terminal objects	Obs I.4(a)
merging into full-precision weights	the canonical morphism \(\rho_M\) to the terminal object	Obs I.4(a)
effective number of parameters	expressive dimension \(d(M)\); the rest is gauge waste	Def I.5
what the first steps can use	first-order image \(S_1\), a functorial invariant	Obs I.6
RoSA, GraLoRA, ReLoRA stages	\(\oplus\)-words, with a Minkowski sum as image	Obs I.7
HF cat merge	the concatenation isomorphism evaluated at trained points	Obs I.7
LoRA-XS	the product \(\mathrm{FA}(A_0)\times\mathrm{FB}(B_0)\)	Prop I.8
adapters, prefixes, ReFT	extensions of the network, outside \(\mathsf{Meth}\)	Def VI.1

The image germ

Exposé I turned a PEFT method into a pointed map \(\rho:(Q,q_0)\to(\Theta,\theta_0)\). Its image \(\mathrm{Im}\,M=\rho(Q)\) is everything it can reach. Fine-tuning travels only a short way from the pretrained weights, so what matters is the image near \(\theta_0\), its germ at \(\theta_0\).

The germ takes few shapes. It is a linear space when \(\rho\) is affine, as for BitFit, LoRA-FA, LoRA-XS, FourierFT and (IA)³. It is a cone when \(\rho\) is multihomogeneous and non-linear and starts where the update vanishes, as LoRA’s \(BA\) does at zero initialisation. It is a piece of a group orbit for exactly orthogonal methods such as exact-Cayley OFT. DoRA adds an output gain to LoRA’s cone (Prop II.8), and HRA meets that cone with a rotation orbit (Thm II.7), each under its stated hypotheses. Adapters and prefixes change the network itself, and their germs live among functions (Exposé VI).

Four invariants of the germ carry most of the story, and isomorphic methods share them (Obs I.6).

• The first-order image \(S_1(M)=d\rho_{q_0}(T_{q_0}Q)\), the directions a first step can take.
• The type of the base point: regular when the first-order deficit \(d(M)-\dim S_1(M)\) vanishes, an apex when it is positive (Def I.5).
• The maximum rank of an update, bounded by a minimum cut (Thm II.10).
• The expressive dimension \(d(M)=\dim\mathrm{Im}\,M\), which falls short of the parameter count \(\lvert M\rvert\) by the gauge waste.

II.A asks where a method starts, II.B what a group-type method can never change (Thm II.5, Cor II.6), and II.C how rank differs from dimension (Prop II.12). Expressivity here means reach, and finding a reachable point is left to Exposé III.

II.A Initialisations: pointings and strata

Every LoRA recipe decides where training starts. LoRA sets \(B_0=0\) and draws \(A_0\) at random. PiSSA takes both factors from the top of the SVD of \(W_0\) and subtracts their product from the frozen weight, LoRA-GA does the same with the first gradient, and CorDA, OLoRA, MiLoRA and EVA choose otherwise. Each recipe is a pointing, a parameter \(q_0\) with the frozen residual that makes \(\rho(q_0)=\theta_0\), and two pointings of one formula can be different objects of \(\mathsf{Meth}(\Theta,\theta_0)\). Figure 2 draws the typical cases.

Translates of one shape

An additive method adds updates from a fixed shape \(C\), for LoRA the cone \(\mathcal M_{\le r}\) of matrices of rank at most \(r\). Pointing it at \(q\) slides the shape so that \(\delta(q)\) lands on \(\theta_0\), and reachable updates become differences of two points of \(C\).

(expand)

(a) At \(q\) the re-pointed map gives \(\theta_0-\delta(q)+\delta(q)=\theta_0\), and its image is \(\theta_0-\delta(q)+C\). (b) Take the union over \(q\) of \(\theta_0+(C-\delta(q))\).

(c) For \(s\ne0\) the shape is \(C=s\,\mathcal M_{\le r}=\mathcal M_{\le r}\). The inclusion \(C-C\subseteq\mathcal M_{\le\min(2r,m,n)}\) is \(\operatorname{rank}(X-Y)\le\operatorname{rank}X+\operatorname{rank}Y\le 2r\). Conversely, \(C\) is a cone and so closed under negation. Split the SVD of a matrix \(Z\) of rank at most \(\min(2r,m,n)\) as \(Z=Z_1+Z_2\) with \(\operatorname{rank}Z_i\le r\), and write \(Z=Z_1-(-Z_2)\). With several boxes \(\delta\) acts box by box, so \(C\) and \(C-C\) are products. The robust form is the same inclusion applied to \(\delta(q)-\delta(q_0)\). For the defects, QLoRA’s \(e\) and QPiSSA’s \(\kappa(W_{\rm res})-W_{\rm res}\) are quantisation errors, generically of full rank, and one-step LoftQ’s \(e-s\,\mathrm{SVD}_r(e)\) has rank at least \(\operatorname{rank}e-r\). For attainment, a split pointing with \(P=\delta(q_0)\) of rank \(r\) has image \((\theta_0-P)+\mathcal M_{\le r}\), so it reaches \(\theta_0+(Y-P)\) for every \(Y\) of rank \(r\), and \(Y-P\) has rank \(2r\) for generic \(Y\) when \(2r\le\min(m,n)\).∎

An initialisation chooses which translate of the shape passes through \(\theta_0\). Split initialisations gain updates \(Y-P\) of rank up to \(2r\) and lose generic updates of every rank \(\ge1\) (Thm II.2(d)). Moving a weight matrix by more than rank \(2r\) needs rebasing, as in ReLoRA (Thm V.3), or another shape.

Three strata

Isomorphisms of methods are pointed diffeomorphisms over \(\Theta\) (Def I.2), such as the gauge \((B,A)\mapsto(Bg^{-1},gA)\), \(g\in GL_r\). They preserve the image and \(S_1\) (Obs I.6) but may change training (Exposé III). Up to isomorphism, a full-rank LoRA initialisation is determined by one subspace or one matrix.

(expand)

• “If” in (a). Two full-row-rank matrices with the same row space satisfy \(A_0'=gA_0\) for a unique \(g\in GL_r\), and \((B,A)\mapsto(Bg^{-1},gA)\) is a pointed isomorphism over \(\Theta\). \(\mathrm T_B\) is symmetric. In \(\mathrm S\), the full-rank factorisations of \(P\) form one \(GL_r\)-orbit, and the residuals agree.
• “Only if” in (a). In \(\mathrm T_A\), \(d\rho_{(0,A_0)}(\dot B,\dot A)=\dot BA_0\), so \(S_1=\{XA_0\}\) is the set of matrices whose row space lies in \(\mathrm{row}\,A_0\). It determines \(\mathrm{row}\,A_0\), and it is an isomorphism invariant by Obs I.6. \(\mathrm T_B\) is symmetric. In \(\mathrm S\), the image \((\theta_0-P)+\mathcal M_{\le r}\) is an isomorphism invariant, and a translate of \(\mathcal M_{\le r}\) determines its vertex. Suppose \(c+\mathcal M_{\le r}=c'+\mathcal M_{\le r}\) with \(D=c-c'\ne0\), so that \(D+\mathcal M_{\le r}=\mathcal M_{\le r}\), and let \(k=\operatorname{rank}D\). If \(k\gt r\), take \(X=0\). Otherwise take \(X\) of rank \(r-k+1\) whose column and row spaces meet those of \(D\) only in \(0\), which is possible since \(r+1\le\min(m,n)\). Then \(\operatorname{rank}(D+X)=r+1\) in both cases, a contradiction. Hence the image determines \(P\), for every pointing.
• Separating the strata. \(S_1(\mathrm T_A)=\{Z:\mathrm{row}\,Z\subseteq V\}\) contains \(e_iv^\top\) for every \(i\) and every \(v\in V\), so it can equal \(S_1(\mathrm T_B)=\{Z:\mathrm{col}\,Z\subseteq U\}\) only if \(U=\mathbb R^m\), which is impossible since \(r\lt m\). When \(m\ne n\) the dimensions \(mr\ne nr\) already differ. \(\mathrm S\) differs from both because \(\dim S_1=r(m+n-r)\) differs from \(mr\) (as \(n\gt r\)) and from \(nr\) (as \(m\gt r\)).
• (b). At a full-rank pair, \(S_1=\{XA_0+B_0Y\}=T_P\mathcal M_r\). In general \(\dim\{XA_0+B_0Y\}=m\operatorname{rk}A_0+n\operatorname{rk}B_0-\operatorname{rk}A_0\operatorname{rk}B_0\) (Prop II.3).
• (c). Classically, \(\mathcal M_r\) is open in \(\mathcal M_{\le r}\) and is a manifold of dimension \(r(m+n-r)\), and \(\mathrm{Sing}\,\mathcal M_{\le r}=\mathcal M_{\le r-1}\). So \(P\), of rank \(r\), is a smooth point, and \(\theta_0\) is a smooth point of the \(\mathrm S\)-image. In the \(\mathrm T\) strata, \(\theta_0\) is the vertex of the non-linear cone \(\theta_0+\mathcal M_{\le r}\), a singular point (Def I.5), and by (b) its deficit is positive, so it is an apex.
• (d). A generic rank-one \(X\) satisfies \(\operatorname{rank}(P+X)=\operatorname{rank}(X-P)=r+1\). So \(\theta_0+X\) lies in the \(\mathrm T\)-image but not in the \(\mathrm S\)-image, and \(\theta_0-P+X\) lies in the \(\mathrm S\)-image but not in the \(\mathrm T\)-image.
• (e). The dimension is Prop II.3. For any pointing, \(S_1=\{XA_0+B_0Y\}\) is the space of matrices that send \((\mathrm{row}\,A_0)^\perp\) into \(\mathrm{col}\,B_0\). Its Frobenius orthogonal complement is \((\mathrm{col}\,B_0)^\perp\otimes(\mathrm{row}\,A_0)^\perp\), whose column and row spans recover \((\mathrm{col}\,B_0)^\perp\) and \((\mathrm{row}\,A_0)^\perp\) when both subspaces are proper. HF’s orthogonal init draws one orthogonal \(Q\in O(r)\) and sets \(A_0=(G_AQ')^\top/10\) and \(B_0=G_BQ''/10\) with Gaussian \(G_A,G_B\), where \(Q'\) and \(Q''\) are the odd- and even-numbered rows of \(Q\) (HF’s q_odd and q_even). Distinct rows of \(Q\) are orthogonal, so \(B_0A_0=0\), with both factors of rank \(r/2\). Its image is \(\theta_0+\mathcal M_{\le r}\), with vertex \(\theta_0\), so it is not in \(\mathrm S\). Its pair \((\mathrm{col}\,B_0,\mathrm{row}\,A_0)\) has dimensions \((r/2,r/2)\), against \((0,r)\) for \(\mathrm T_A\) and \((r,0)\) for \(\mathrm T_B\), so its \(S_1\) differs from theirs. Its deficit is \(\tfrac r2(m+n-\tfrac{3r}2)\gt0\). In low precision the separately rounded factors acquire small extra singular values, so the placement holds only up to a small perturbation.

Numerical check: \(\dim S_1=(14,10,20)\) for \((\mathrm T_A,\mathrm T_B,\mathrm S)\) at \((m,n,r)=(7,5,2)\), and \(11\) for HF’s orthogonal init.∎

Most published schemes are points of the three strata. LoRA-FA and MiCA freeze one factor and are sub-objects, and the last three rows lie outside the strata.

Zero-init LoRA starts at a singular point where the \(A\)-direction is dead at first order, so LoRA and LoRA-FA, trained with the same optimizer from the same \(A_0\), agree to first order (Lemma II.4). Split initialisations start at a smooth point with full first-order reach. That this explains the faster early progress reported for PiSSA and LoRA-GA is a conjecture; LoRA-GA credits gradient alignment, and MiLoRA also starts at a smooth point.

More: the first-order deficit and the data

\(S_1\) counts directions in weight space. The lazy-regime kernel \(J_f\,d\rho\,\Lambda\,d\rho^\top J_f^\top\) also involves the network Jacobian \(J_f\) and the optimizer’s metric \(\Lambda\), and for random-\(A_0\) zero-init LoRA it approximates the full fine-tuning kernel (Malladi et al., 2023). A large first-order deficit need not cost much on the data.

Counting first-order directions

A first step of LoRA from \((B,A)\) adds \(s(\dot BA+B\dot A)\), a matrix with rows in \(\mathrm{row}\,A\) plus one with columns in \(\mathrm{col}\,B\). Counting both families and their overlap gives \(\dim S_1\).

(expand)

The image of \(d\kappa\) is \(\{XA\}+\{BY\}\), since the factor \(s\ne0\) does not change it. Its summands are the matrices with row space in \(\mathrm{row}\,A\), of dimension \(m\operatorname{rk}A\), and the matrices with column space in \(\mathrm{col}\,B\), of dimension \(n\operatorname{rk}B\). A matrix \(D\) lies in both iff \(D\in\mathrm{col}\,B\otimes\mathrm{row}\,A\), of dimension \(\operatorname{rk}A\operatorname{rk}B\), and then \(D=B(B^+DA^+)A\).

The gradient-flow rate is \(\langle G,\dot W\rangle\) with \(\dot W=-s^2(\eta_BGA^\top A+\eta_ABB^\top G)\), because \(\nabla_BL=sGA^\top\) and \(\nabla_AL=sB^\top G\). Adam’s first step is \(\delta B=-\eta_B\,\mathrm{sign}(sGA^\top)\), so \(\langle G,s\,\delta B\,A\rangle=-\lvert s\rvert\eta_B\|GA^\top\|_1\), and likewise for \(A\).

Numerical check: the rank formula holds in 400 random cases, including rank-deficient factors, and the rate matches finite differences.∎

“Better initialisation” mixes three mechanisms, and the proposition separates them.

• Smoothness. PiSSA and LoRA-GA start off the apex, so their first step sees all \(r(m+n-r)\) directions.
• Alignment. At a zero-\(B\) start the rate is \(s^2\eta_B\|GA_0^\top\|_F^2\). Among \(A_0\) with orthonormal rows, the top-\(r\) right singular space of \(G\) maximises it, and that is LoRA-GA’s \(A_0\). Its \(B_0\), the next \(r\) left singular vectors, makes the first step approximate full fine-tuning and does not maximise the rate. EVA’s activation frame is a gradient-free proxy. Under Adam the rate depends on the basis of \(A_0\), so this alignment belongs to gradient flow and SGD.
• Scale. \(\|GA_0^\top\|_F^2\) grows with \(\|A_0\|^2\), and initial scale trades off against learning rate (Thm III.10). HF’s Kaiming default has \(A_0A_0^\top\approx I/3\) and EVA with whiten=False has \(A_0A_0^\top=I\), so fair comparisons normalise \(A_0A_0^\top\).

More: EVA’s frame and the factor scales

EVA’s guarantee assumes inputs independent of the backpropagated errors and bounds the expected per-sample squared gradient norm, a different quantity from the rate above, and HF’s EVA centres the activations. PiSSA’s \(S_r^{1/2}\)-weighted factors and LoRA-GA’s stability scaling also change factor norms (both factors in LoRA-GA’s paper, only \(B\) in HF’s default).

The zero gate

At a zero-\(B\) start, \(BA\) is linear in \(B\) and \(B_0=0\), so moving \(A\) alone changes nothing. The same holds whenever a trainable gate starts closed. Below, \(K=D\rho\,\Lambda\,D\rho^\top\) is the cometric on weights induced by an optimizer’s metric \(\Lambda\) (Exposé III).

(expand)

Since \(\beta(u_0,v)=0\) for all \(v\), differentiating in \(v\) gives \(D_v\beta(u_0,v_0)=0\). The rest is the chain rule. The gradient \(\nabla_v(L\circ\rho)=D_v\rho^\top\nabla L\) vanishes, \(S_1\) is the image of the \(u\)-columns alone, and in \(K=D\rho\,\Lambda\,D\rho^\top\) every term that touches a \(v\)-column is zero, which leaves \(D_u\beta\,\Lambda_{uu}\,D_u\beta^\top\). The function-level variant is the same argument applied to \(F\).

Numerical check: under Adam with \(\eta\in\{10^{-2},10^{-3},10^{-4}\}\), the step-2 move of \(A\) in zero-init LoRA scales as \(\eta\), while the deviation of \(\Delta W\) from the frozen-\(A\) method scales as \(\eta^2\).∎

Instances, assuming no weight decay on \(v\):

• LoRA (\(u=B\)) and VeRA (\(u=b\)).
• AdaLoRA, with \(u\) its diagonal of singular values started at \(0\). \(\theta\) first moves only along the \(r\) directions \(p_iq_i^\top\), so \(\dim S_1=r\) for generic \(P,Q\). \(P\) and \(Q\) move at step one through the orthogonality penalty (HF default orth_reg_weight=0.5), with no first-order effect on \(\theta\).
• LoHa with a zero factor (HF zero-initialises hada_w2_b).
• By the function-level variant, the gate of LLaMA-Adapter and bottleneck adapters whose up-projection starts at zero (Houlsby et al., 2019 start near that point).

HRA’s paired initialisation, though of a different form, shows the same first-order collapse (Thm II.7(b)).

Off the point: quantised bases

QLoRA quantises the frozen weight, so it is LoRA pointed at \(\kappa\theta_0\), with pointing defect \(e=\kappa\theta_0-\theta_0\) (Def V.1, Prop V.2(a)). As \(e\) is generically of full rank, \(\theta_0\) lies outside \(\mathrm{Im}\,\mathrm{LoRA}_r(\kappa\theta_0)\) unless \(\operatorname{rank}e\le r\), which is why Theorem II.1 speaks of exact pointings. When the initialisation also splits, the order of operations decides the defect (Prop V.2(b)). Below, \(W\) is the full-precision weight, \(e=\kappa W-W\) and \(\delta_0=\delta(q_0)\).

The strata of Theorem II.2 classify what a start can express. The dynamics see a finer quotient. For a zero-\(B\) start, SGD sees \(A_0^\top A_0\) and Adam sees \(A_0\) up to signed permutations of its rows (Cor III.13). A dynamical classification of initialisations, including the degenerate pointings, is open.

Klein's Erlangen programme of 1872 defines a geometry by its group of motions and studies what the motions leave unchanged. OFT, GOFT, HRA and ETHER multiply the pretrained matrix by an orthogonal matrix, (IA)³ and SSF's scale multiply it by a diagonal one, and DoRA's magnitude does the same to a LoRA weight. These are methods of action type (Definition I.2), \(\rho(q)=\gamma(q)\cdot W_0\) with \(\gamma\) landing in a group. Whatever that group cannot change, the method cannot change, in training or after any number of merges.

One convention matters. A linear box computes \(y=Wx\) with \(W\in\mathbb R^{m\times n}\), and row \(i\) of \(W\) is output neuron \(i\). The two Gram matrices, over neurons and over inputs, are

\[K_{\rm out}(W)=WW^\top,\qquad K_{\rm in}(W)=W^\top W.\]

A right multiplication \(W\mapsto WR\) acts on the input side and fixes \(K_{\rm out}\); a left multiplication \(W\mapsto PW\) acts on the output side and fixes \(K_{\rm in}\). Papers that store \(W^\top\) swap these words, and we translate every method into this convention.

Complete invariants

A complete invariant of a group action agrees on two matrices exactly when one can be carried to the other. For the five actions in PEFT it is classical. We write \(T_m\) for the invertible diagonal matrices, \(T_m^+=\{\mathrm{diag}(d):d_i\gt0\}\) for its identity component, and \(\odot\) for the entrywise product.

(expand)

(1) If all inner products of rows agree, the map \(\sum_ic_iw_i\mapsto\sum_ic_iw_i'\) is well defined and isometric on the row spans, because \(\|\sum_ic_iw_i\|^2=c^\top WW^\top c\). The two row spans have the same dimension, \(\operatorname{rank}WW^\top\). Extend the map by an isometry of the orthogonal complements to \(R^\top\in O(n)\); then \(W'=WR\). The converse is immediate.

(2) is (1) applied to transposes. (3) follows from the singular value decomposition. (4) and (5) hold entry by entry; for (5) put \(E_{ij}=W'_{ij}/W_{ij}\) on the support and \(1\) off it. ∎

More: where Theorem II.5 comes from

Item (1) follows from the first fundamental theorem of invariant theory for \(O(n)\), or from Witt's extension of isometries. LoHa's gauge in Proposition II.12, the two-sided diagonal torus \(\{\mathrm{diag}(a)\,X\,\mathrm{diag}(b)\}\), differs from the full Hadamard torus in (5).

No PEFT method realises (4) or (5) as a group action. (IA)³ gains and DoRA magnitudes are unconstrained reals, and HiRA's factor \(J+BA\) can vanish entrywise. These methods can create new zeros, so only the one-way statement transfers, that the zeros of \(W_0\) remain zero.

Moving a method from the input side to the output side trades \(K_{\rm out}\) for \(K_{\rm in}\). Letting a scale change sign trades rays for lines, and letting it reach zero leaves only the zeros of \(W_0\).

What exactly orthogonal methods can never do

If every weight a method reaches is \(W_0R\) with \(R\) orthogonal, Theorem II.5(1) and (3) freeze the neuron Gram matrix and the singular values, and merging cannot help, since products of orthogonal matrices are orthogonal. The work lies in saying which implementations are exactly orthogonal, and on which side.

Several methods that look orthogonal fail the hypothesis. Each row below is a separate computation.

(expand)

Use Theorem II.5(1) and (3), and the fact that a product of orthogonal factors is orthogonal, so the image of any number of merge cycles stays in \(W_0\cdot O(n)\). For block-diagonal \(R\), \((WR)_{:,J}=W_{:,J}R_J\), which gives the partial Grams when the partition is fixed.

For BOFT, \(K_{\rm out}(\mathrm{diag}(s)W_0R^\top)=\mathrm{diag}(s)K_{\rm out}(W_0)\mathrm{diag}(s)\), so \(\cos'_{ij}=\mathrm{sign}(s_is_j)\cos_{ij}\) whenever \(s_is_j\ne0\). All cosines therefore survive when the \(s_i\) are nonzero and share one sign, and when no cosine of \(W_0\) vanishes this is also necessary. For GOFT's default code, \(K_{\rm out}(W_0\mathrm{diag}(g)G)=W_0\mathrm{diag}(g)GG^\top\mathrm{diag}(g)W_0^\top=W_0\mathrm{diag}(g)^2W_0^\top\).

For the Cayley–Neumann map, \((I-Q)(I+Q+Q^2+Q^3)=I-Q^4\), so \(R=(I+Q)(I+Q+Q^2+Q^3)=C(Q)(I-Q^4)\) with \(C(Q)=(I+Q)(I-Q)^{-1}\), which is HF's exact map \((I-Q)(I+Q)^{-1}\) evaluated at \(-Q\). \(C(Q)\) is orthogonal and commutes with \(Q^4\), and \((Q^4)^\top=Q^4\) for skew \(Q\). Hence \(R^\top R=(I-Q^4)^2\), and if the eigenvalues of \(Q\) are \(\pm i\theta_k\), the singular values of \(R\) are \(\lvert1-\theta_k^4\rvert\): at most \(1\) when every \(\theta_k\le2^{1/4}\), and \(0\) when some \(\theta_k=1\). With last coefficient \(2\), as in PEFT 0.16–0.17, the same computation gives \(R^\top R=I+4Q^6+4Q^8\). ∎

Numerical check: with \(s=(-0.5,1,\dots,1)\), BOFT changes a cosine by \(1.5\) while \(\lvert\cos\rvert\) is fixed to \(4\times10^{-16}\), and with every \(s_i\lt0\) the cosines are fixed to \(4\times10^{-16}\). HF's default OFT satisfies \(R^\top R=(I-Q^4)^2\) to \(5\times10^{-16}\), with \(\lVert R^\top R-I\rVert=1.5\times10^{-2}\) already for skew entries of size about \(0.07\) at block size \(8\). A conformal block \(\sqrt2\,\mathrm{Rot}(0.3)\) on two of \(n=8\) inputs changes some cosine by \(0.17\).

“Orthogonal fine-tuning preserves knowledge” corresponds to preserving a complete invariant, which exact orthogonality keeps through any number of merges (Theorem V.3, corollary 7), up to rounding. HF's default OFT is only close to orthogonal; with block size \(8\) and fresh skew generators with entries near \(0.05\) in every cycle, its merges shrink \(K_{\rm out}\) by about \(0.2\%\), \(1.3\%\), \(6\%\) and \(22\%\) after 1, 10, 50 and 200 cycles (relative Frobenius change, averaged over draws). The statement concerns reach. It does not show that the invariant causes retention, and the control study of ETHER (Bini et al., 2024) found a non-orthogonal multiplicative variant comparably stable.

HRA is a meet of two images

A Householder reflection \(H_u=I-2uu^\top/\|u\|^2\) is orthogonal and differs from \(I\) by rank one. So \(W_0H_{u_1}\cdots H_{u_r}\) lies in the orthogonal orbit of \(W_0\) and in LoRA's cone \(W_0+\mathcal M_{\le r}\), and for injective \(W_0\) and even \(r\) the Cartan–Dieudonné theorem says that HRA fills the cone's intersection with the rotation orbit. HF PEFT repeats each reflection at initialisation, so to first order each pair can only turn its own vector against the rest of the space, and HRA starts at an apex, as zero-init LoRA does (Theorem II.2).

(expand)

(a) “\(\subseteq\)”. Each \(H_u-I\) has rank one, so \(R=\prod_iH_{u_i}\) fixes \(\bigcap_iu_i^\perp\) and \(\operatorname{rank}(R-I)\le r\). Also \(\det R=(-1)^r=1\). Hence \(W_0R-W_0=W_0(R-I)\) has rank at most \(r\).

“\(\supseteq\)”. Suppose \(W_0R=W_0+X\) with \(\operatorname{rank}X\le r\) and \(R\in SO(n)\). Injectivity gives \(k':=\operatorname{rank}(R-I)\le r\). By the Cartan–Dieudonné theorem in Scherk's sharp form, \(R\) is a product of exactly \(k'\) reflections, and \(\det R=(-1)^{k'}\) forces \(k'\) to be even. Pad with \((r-k')/2\) pairs \(H_uH_u=I\). A single reflection has determinant \(-1\), so \(W_0H_u\), a rank-one update in the \(O(n)\)-orbit, is not in the image.

Dimension. A generic element is determined by its moved space \(\mathrm{im}(R-I)\in\mathrm{Gr}(r,n)\), of dimension \(r(n-r)\), together with a fixed-point-free element of \(SO(r)\) on it, an open set of dimension \(r(r-1)/2\). The sum is \(rn-r(r+1)/2\), and the generic fibre has dimension \(rn-d\). The identity \(H_aH_b=H_{H_ab}H_a\) shows that the Hurwitz move \((u_i,u_{i+1})\mapsto(H_{u_i}u_{i+1},u_i)\) preserves \(\rho\).

(b) Normalise \(\hat w=w/\|w\|\). A direct computation gives

\[\tfrac{d}{d\varepsilon}\big(H_{w+\varepsilon a}H_{w+\varepsilon b}\big)\big|_{\varepsilon=0}=\tfrac{2}{\|w\|}\big(v\hat w^\top-\hat wv^\top\big),\qquad v=(I-\hat w\hat w^\top)(a-b).\]

Every pair equals \(I\) at \(q_0\), so the differential of the product is the sum of the pair differentials, and \(S_1=W_0\cdot\{\sum_iv_i\wedge\hat w_i\}\) with \(v\wedge w:=vw^\top-wv^\top\). This set lies in \(\{\Omega:P_{U^\perp}\Omega P_{U^\perp}=0\}\), because \(y^\top(v\hat w^\top-\hat wv^\top)x=0\) for \(x,y\perp\hat w\). Conversely, that space is spanned by the \(e_a\wedge e_b\) with \(e_a\in U\), and each is a combination of the elements \(\hat w_i\wedge e_b\). Its dimension is \(\dim\mathfrak{so}(n)-\dim\mathfrak{so}(n-k)=kn-k(k+1)/2\), and \(W_0\) is injective. Finally \(d-\dim S_1=k(2n-3k-1)/2\gt0\) for \(n\ge2k+2\).

(c) The Cayley map \(C(\Omega)=(I+\Omega)(I-\Omega)^{-1}\) is a diffeomorphism from a neighbourhood of \(0\in\mathfrak{so}(n)\) onto a neighbourhood of \(I\in SO(n)\), and \(C(\Omega)-I=2\Omega(I-\Omega)^{-1}\), so \(\operatorname{rank}(C(\Omega)-I)=\operatorname{rank}\Omega\). Composing with the linear embedding \(R\mapsto W_0R\) identifies the germs. \(\Sigma_r\) is a closed cone, and it is not linear, because a rank-\(r\) and a rank-\(2\) skew matrix can sum to rank \(r+2\le n\). A closed cone that is \(C^1\) at its vertex is linear, so \(0\) is singular. The apex statement uses (b); a singular image alone would not give a positive deficit.

(d) Over a generic point of the meet, the LoRA fibre is a \(GL_r\)-orbit and the \(SO(n)\) fibre is a point, since \(W_0\) is injective; this gives \(d+r^2\). Over \(W_0\) itself the fibre product contains \(\{(B,0,I)\}\), of dimension \(mr\), so it is not pure-dimensional. If \(C(\Omega)x=-x\), then \(y=(I-\Omega)^{-1}x\) satisfies \((I+\Omega)y=-(I-\Omega)y\), so \(y=0\). For density, perturb the angles \(\pi\) inside the same moved planes. For the Cayley–Neumann factor \(R\), a point \(W_0R\) of \(W_0\cdot O(n)\) needs \(R\) orthogonal, since \(W_0\) is injective, that is \((I-Q^4)^2=I\), which for small \(Q\) forces \(Q=0\). Conversely a skew \(Q\) of rank \(2\) inside one block gives \(\operatorname{rank}(R-I)=2\) with \(R\) not orthogonal, a point of LoRA \(\cap\) OFT outside \(\mathrm{Im}\,\mathrm{HRA}_r\). ∎

Numerical check: at \(n=9\) the paired-init Jacobian has rank \(8\) against generic rank \(15\) for \(r=2\), and \(15\) against \(26\) for \(r=4\).

More: symmetries, Gram–Schmidt and odd rank

The evident continuous symmetries of HRA's gauge are the \(r\) scalings \(u_i\mapsto c_iu_i\); the braid group also acts, by Hurwitz moves. HRA with Gram–Schmidt (apply_GS=True) has the smaller image \(W_0(I-2P_U)\), of dimension \(r(n-r)\), which never contains \(W_0\). For odd \(r\), HF falls back to a random initialisation, and \(\det R=(-1)^r\) shows that no odd-\(r\) HRA can start at \(W_0\).

HRA's “bridge between low-rank and orthogonal adaptation” is, at the level of images, the meet of LoRA with the rotation orbit. It strictly contains LoRA \(\cap\) OFT for exactly orthogonal OFT and is incomparable with it for HF's default OFT. Compare Proposition I.8(a), where LoRA-XS (Bałazy et al., 2024) is a genuine categorical product \(\mathrm{FA}\times\mathrm{FB}\). That HRA's default initialisation is an apex is new here. Its first-order collapse is the multiplicative twin of LoRA's dead \(A\)-direction (Lemma II.4), and whether it slows early training is open (Exposé XI). Rebasing commutes with the meet, so after \(K\) cycles re-HRA reaches \((W_0+\mathcal M_{\le Kr})\cap W_0\cdot SO(n)\) (Theorem V.3, corollary 5).

In Figure 3, choose HRA and press “Reset to \(q_0\)”. The first-order strip reads \(\dim S_1=3\) against \(d=5\), the values of (a) and (b) at \(n=4\), \(r=2\). Move \(u_2\) and the Gram, spectrum and rank lamps stay lit together; “\(u=(e_1,e_2)\)” lands on \(W_0H_{e_1}H_{e_2}\), which exact Cayley OFT never reaches.

LOFT fixes one leaf of the meet

The proof of Theorem II.7 already shows how the meet is built. It is the union of the orbits \(W_0\cdot SO(V)\) over the \(r\)-dimensional input subspaces \(V\), one subgroup orbit for each point of the Grassmannian \(\mathrm{Gr}(r,n)\). Call each orbit a leaf. Leaves overlap along lower-rank strata, so this is a union and not a partition, but a generic point lies on exactly one, fixed by its moved space \(\mathrm{im}(R-I)\). HRA's reflection vectors move the leaf as it trains. LOFT (Zhao et al., 2026) freezes the leaf and makes the choice of leaf the design variable, which the paper calls the adaptation support.

(expand)

(a) A rotation \(T\in O(r)\) gives \(W_0(I+P_r^\top(T-I)P_r)=W_0+W_0P_r^\top(T-I)P_r\), so “\(\supseteq\)” holds. Conversely let \(W=W_0+BXP_r\) with \(B=W_0P_r^\top\), and suppose \(W=W_0S\) with \(S\in O(n)\). Then \(WW^\top=W_0W_0^\top\), and expanding with \(P_rP_r^\top=I_r\) gives \(B\,(X+X^\top+XX^\top)\,B^\top=0\). Since \(B\) has full column rank, \(X+X^\top+XX^\top=0\), that is \((I+X)(I+X)^\top=I\), so \(T=I+X\in O(r)\). For \(P_r=V_r^\top\) we have \(B=U_r\Sigma_r\), and \(\{U_r\Sigma_rXV_r^\top\}=\{U_rR\,\Sigma_rV_r^\top\}\) through \(R=\Sigma_rX\Sigma_r^{-1}\), which is LoRA-XS's image. For injective \(W_0\), \(W_0S\) determines \(S\), so meeting with exact full-block Cayley OFT keeps exactly the \(S(T)\) with \(T\in SO(r)\) and no eigenvalue \(-1\), which is the Cayley image.

(b) Every \(S\in SO(V)\) has \(\operatorname{rank}(S-I)\le r\), which gives “\(\supseteq\)”. Conversely, if \(W_0S-W_0\) has rank at most \(r\) and \(W_0\) is injective, then \(\operatorname{rank}(S-I)\le r\). The fixed space \(F=\ker(S-I)\) has dimension at least \(n-r\), and \(F^\perp\) is invariant. Any \(r\)-dimensional \(V\supseteq F^\perp\) works: \(S\) is the identity on \(V^\perp\subseteq F\) and has determinant \(1\) on \(V\).

(c) \(D\rho_0(E)=W_0P_r^\top Q'(0)(E)P_r\), with \(Q'(0)\) invertible. If this vanishes, then \(W_0P_r^\top Q'(0)(E)=0\) because \(P_r\) has full row rank, and \(E=0\) because \(W_0P_r^\top\) has full column rank. So \(\dim S_1=\lvert M\rvert\ge d(M)\ge\dim S_1\). A smooth pointed arrow would give \(S_1(\mathrm{LOFT})\subseteq S_1(\mathrm{HRA})\) (Observation I.6), and by Theorem II.7(b) and injectivity this asks every \(\Omega\in\mathfrak{so}(V)\) to satisfy \(P_{U^\perp}\Omega P_{U^\perp}=0\). Since \(\dim(V\cap U^\perp)\ge r-r/2\ge2\), any two orthonormal \(a,b\in V\cap U^\perp\) give \(a\wedge b\) violating it.

(d) With \(Q'(0)=\mathrm{id}\), \(\tfrac{d}{dt}L\big(\rho(tE)\big)\big|_{0}=\langle G,W_0P_r^\top EP_r\rangle=\langle P_rW_0^\top GP_r^\top,E\rangle=\langle P_r\operatorname{skew}(W_0^\top G)P_r^\top,E\rangle\), since \(E\) is skew. The bound is a Ky Fan inequality for the skew matrix \(\operatorname{skew}(W_0^\top G)\) compressed to \(V\). ∎

Numerical check (\(n=8\), \(m=10\), \(r=2,4,6\)): the leaf identity, the union, the arrows into exact Cayley OFT and LoRA-XS, and \(\dim S_1=r(r-1)/2\) all hold to about \(10^{-15}\).

Orthogonal methods differ in which leaves they use and how they combine them. Block OFT, GOFT and BOFT multiply rotations of fixed coordinate subspaces; HRA moves its leaf while it trains; PSOFT takes the principal leaf and adds two tori; LOFT takes one leaf, chosen before training from \(W_0\) or from the gradient at \(\theta_0\). By part (d), SkewGrad is the orthogonal counterpart of the alignment mechanism of Proposition II.3, and GradSVD transplants the gradient-based frames of LoRA-GA. With the leaf frozen, merge-and-restart gains nothing. With finitely many supports revisited across restarts, the reach is the orbit of the connected group whose Lie algebra the \(\mathfrak{so}(V_k)\) generate (Theorem V.3(b)); numerically, generic supports with \(r\ge2\) generate all of \(SO(n)\) once \(\lceil n/r\rceil\) of them are used. The principal support does not move when it is re-selected after a merge, and data-chosen supports have to be checked case by case.

DoRA is a torus saturation of LoRA

DoRA normalises each row of LoRA's matrix \(V=W_0+BA\) and multiplies row \(i\) by a learned magnitude \(\mu_i\). That is one row scaling, by \(\mu_i/\|v_i\|\), so every DoRA weight is a diagonal matrix times a LoRA weight, and conversely up to a repair for vanishing rows. We write \(\mathrm{Diag}_m\) for the monoid of diagonal matrices, \(\mathrm{Mon}_m=T_m\rtimes S_m\) for the monomial matrices, and \(N\) for row normalisation.

(expand)

\(\mathrm{diag}(\mu)N(V)=\mathrm{diag}(\mu_i/\|v_i\|)\,V\in\mathrm{Diag}_m\cdot V\). Conversely, take \(\mathrm{diag}(\ell)V\) with \(V=W_0+X\), \(\operatorname{rank}X\le r\), and let \(Z\) be the set of zero rows of \(V\). Put \(V'=W_0+(I-\varepsilon\sum_{i\in Z}E_{ii})X\). Its rows outside \(Z\) are those of \(V\), its rows in \(Z\) are \(\varepsilon w_{0,i}\ne0\), and its update still has rank \(\le r\). With \(\ell'_i=0\) on \(Z\) and \(\ell'_i=\ell_i\) elsewhere, \(\mathrm{diag}(\ell')V'=\mathrm{diag}(\ell)V\) and \(V'\in\mathcal U\). For the \(T_m\) statement, \(DV\in\mathcal U\) with \(D\) invertible forces \(V\in\mathcal U\).

Dimension. \(\rho_{\rm DoRA}=\rho_{\rm LoRA+scale}\circ\varphi\) with \(\varphi(\mu,B,A)=(\mu_i/\|v_i\|,B,A)\), a diffeomorphism of DoRA's domain, so ranks may be computed for \((D,X)\mapsto D(W_0+X)\). For invertible \(D\) the differential at \((D,X)\) is \(D\) times the one at \((I,X)\) after \(\delta\mapsto D^{-1}\delta\). Its image at \((I,X)\) is \(\{\mathrm{diag}(\delta)V\}+T_X\mathcal M_r\); the first summand has dimension \(m\) since \(V\in\mathcal U\), and \(\mathrm{diag}(\delta)V\in T_X\mathcal M_r\) iff \(P_{U^\perp}\mathrm{diag}(\delta)VP_{R^\perp}=0\), where \(VP_{R^\perp}=W_0P_{R^\perp}\). A zero \(\mu_i\) kills row \(i\) of the LoRA directions, leaving only \(\delta_iv_i\) there, where an invertible magnitude also gives every \(e_iy^\top\) with \(y\in R\). Since \(v_i-w_{0,i}\in R\), the rank falls below the formula by at least \(\dim\mathrm{span}(w_{0,i},R)-1\) for each such \(i\), which is positive when \(r\ge2\) or \(w_{0,i}\notin R\). For the generic value, a proof sketch: with bases \(E\) of \(U^\perp\) and \(F\) of \(R^\perp\) the kernel map is \(\delta\mapsto\sum_i\delta_i(E^\top e_i)\otimes(F^\top W_0^\top e_i)\), and \(m\) generic points of the Segre cone in \(\mathbb R^{m-r}\otimes\mathbb R^{n-r}\) span \(\min(m,(m-r)(n-r))\) dimensions.

Covariance. A monomial \(g\) normalises \(\mathrm{Diag}_m\), and \(g\mathcal M_{\le r}h^{-1}=\mathcal M_{\le r}\), so \(g\,\mathrm{Diag}_m(W_0+\mathcal M_{\le r})h^{-1}=\mathrm{Diag}_m(gW_0h^{-1}+\mathcal M_{\le r})\). ∎

Numerical check: Jacobian rank \(9=mn\) at \((m,n,r)=(3,3,2)\), where \(r(m+n-r)+m\) gives \(11\); \(36\) at \((8,8,2)\), matching; \(24\) for a rank-one \(W_0\) at \((6,6,2)\). With one \(\mu_i=0\) at \((8,8,2)\) the rank drops to \(34\).

More: which weight, which norm

Images refer to the merged weight; in training mode with LoRA dropout, HF's DoRA forward is not of the form \(Wx\). The DoRA paper's notation normalises “column-wise”, while both its official code and HF PEFT normalise each output row.

At the level of images, DoRA's “magnitude–direction decoupling” is LoRA plus \(m\) free output gains; everything else happens in training. Under gradient flow the exact gradient of the normalised parametrisation is a change of metric, while the detached backward of the DoRA paper (Sec. 4.3) and HF PEFT (weight_norm.detach()) is a pseudo-gradient that no metric describes. Under Adam the coordinates matter too, as \(\mu_i\) starts at \(\|w_{0,i}\|\) and \(\ell_i\) at \(1\). Prediction X.2 (Conjecture D) uses the equal images to set up a controlled comparison.

The covariance group \(\mathrm{Mon}_m\times GL_n\) decides where Observation II.15 applies. One orthogonal rotation of the residual stream of a gain-fused pre-norm model, as in SliceGPT, acts on the input side of \(q\), \(k\), \(v\), gate and up, where DoRA is \(GL_n\)-covariant, so DoRA on those boxes reaches the same functions after the rotation as before; on \(o\) and down the sufficient condition fails. Training outcomes are a separate question.

The torus ladder

Write a weight as \(W_0\odot(J+Z)\), with \(J\) the all-ones matrix. Output scalings use \(Z=(\ell-\mathbf 1)\mathbf 1^\top\), of rank one; two-sided scalings use \(ab^\top-J\), of rank at most two; HiRA allows any \(Z=BA\) of rank at most \(r\). These choices form a ladder of images, and no rung can fill a zero of \(W_0\).

(expand)

(a) \(W_0\odot(J+BA)=W_0+W_0\odot BA\). \(D_{W_0}\) is diagonal in the standard basis of \(\Theta\), and invertible iff no entry of \(W_0\) vanishes; pushforward along a linear isomorphism preserves dimension and gauge. For the rank, \(W_0\odot(BA)=\sum_k\mathrm{diag}(b_k)W_0\mathrm{diag}(a_k)\), summed over the \(r\) columns \(b_k\) of \(B\) and rows \(a_k\) of \(A\), which has rank \(\le r\operatorname{rank}W_0\).

(b) Direct computation, using \(ab^\top-J=[a,-\mathbf 1][b,\mathbf 1]^\top\in\mathcal M_{\le2}\).

(c) The tangent map at \((\mathbf 1,\mathbf 1)\) is \((\alpha,\beta)\mapsto\big((\alpha_i+\beta_j)(W_0)_{ij}\big)\). Its kernel requires \(\alpha_i+\beta_j=0\) along every support edge, so it is one-dimensional on each connected component. Torus orbits are homogeneous, so the dimension is the same at every point of the orbit.

(d) A \(2\times n\) real block \(X\) is a complex row in \(\mathbb C^n\), and \(\alpha\mathrm{Rot}(\theta)\) acts on it as multiplication by \(\alpha e^{i\theta}\). So the image for one pair is \(\mathbb C\cdot X\), of real dimension \(2\) iff \(X\ne0\), and \((\alpha\mathrm{Rot}X)^\top(\alpha\mathrm{Rot}X)=\alpha^2X^\top X\). In \(\mathrm{RoAd}_2\) each row of a block is an arbitrary vector of \(\mathbb R^2\) written in polar form.

(e) Hadamard multiplication and row or column scaling preserve the zeros of \(W_0\).

(IA)³ and SSF. A monomial \(g\) normalises the diagonal monoid, so \(\mathrm{Mon}_m\times GL_n\) lies in the covariance group of output scalings. Conversely, take \(W_0=e_jc^\top\). Then \(\mathrm{Im}\,M(gW_0h^{-1})=\mathrm{span}\{e_i:i\in\mathrm{supp}(ge_j)\}\,c^\top h^{-1}\), while \(g\,\mathrm{Im}\,M(W_0)\,h^{-1}=\mathbb R\,(ge_j)c^\top h^{-1}\). Equality forces \(\lvert\mathrm{supp}(ge_j)\rvert=1\) for every \(j\), so \(g\) is monomial. This works for all \(m,n\). Input-side scalings are the transpose.

HiRA. Monomial pairs commute with Hadamard products up to relabelling, so \(\mathrm{Mon}_m\times\mathrm{Mon}_n\) lies in the covariance group. Conversely, take \(W_0=E_{ij}\). Then \(\mathrm{Im}=\mathbb R\,E_{ij}\), so covariance under \((g,h)\) forces \(\mathrm{Im}\,\mathrm{HiRA}_r(uv^\top)=\mathbb R\,uv^\top\) with \(u=ge_i\) and \(v=h^{-\top}e_j\). But that image contains \(uv^\top+(u\odot p)(v\odot q)^\top\) for all \(p,q\), which leaves the line \(\mathbb R\,uv^\top\) unless \(\lvert\mathrm{supp}\,u\rvert=\lvert\mathrm{supp}\,v\rvert=1\). So every \(ge_i\) and every \(h^{-\top}e_j\) has a single nonzero entry, and \(g\) and \(h\) are monomial. ∎

Numerical check: \(\mathrm{RoAd}_1\) orbit dimension \(8\) and \(\mathrm{RoAd}_2\) dimension \(16\) at \(m=8\).

More: pointings, LayerNorm folds and HF PEFT details

As a morphism, \(h(\ell)=(\ell-\mathbf 1,\mathbf 1^\top)\) lands in \(\mathrm{HiRA}_1\) pointed at \((0,\mathbf 1^\top)\), not at HiRA's random-\(A\) default. A LayerNorm gain folds exactly for the input norms of pre-norm blocks and a final norm before an untied head, but not for post-sublayer norms (Gemma-2 and 3) or QK-norms; LayerNorm biases, like SSF's shift, leave \(W\) alone. HF HiRA on embeddings starts at \(A_0=0\). HF RoAd pairs output \(j\) with \(j+g/2\) in groups of size \(g\) and also rotates the bias; nothing above depends on the pairing.

(IA)³, SSF's scales, folded LayerNorm gains and HiRA are rungs \(W_0\odot(J+Z)\). DoRA and RoAd can fill zeros of \(W_0\); they become rungs only when \(W_0\) has no zero entries, where every image germ is a rung and the ladder by itself constrains nothing. HiRA's “high rank” is a property of a frame. Its transport \(D_{W_0}\) depends on \(\theta_0\) and does not preserve rank. Two re-based cycles reach \(W_0\odot(J+Z_1)\odot(J+Z_2)\), whose relative update \(Z_1+Z_2+Z_1\odot Z_2\) has rank up to \(\min(2r+r^2,m,n)\), against \(2r\) for LoRA (Proposition II.13).

With HF defaults, (IA)³ scales the output side of the attention projections it targets (\(k\) and \(v\), or \(q\) and \(v\)) and the input side of down, so the residual rotation above, acting on the other side, leaves its reachable functions unchanged. HiRA's covariance group stops at \(\mathrm{Mon}_m\times\mathrm{Mon}_n\) and fails the sufficient condition of Observation II.15. That alone does not prove its reachable functions change; Conjecture A says they do, and Prediction X.4 is the test.

The groups of this exposé see orbits. LoRA's cone is governed instead by the rank of the update, read off a cut in a tensor network, while the size of an image is a dimension, which behaves like a volume. Rank and volume can move in opposite directions, the subject of Exposé II.C.

PEFT papers usually report a method’s expressiveness as a rank and its cost as a parameter count. A third number, the dimension of the set of weights the method can reach, usually goes unreported.

Draw a method as a network of boxes and wires. The rank of its update is limited by the narrowest place where the network can be cut in two, so rank is a cut invariant. The dimension counts the independent directions in which the method can move the weights; we call it a volume invariant, loosely, since no measure is computed. In Figure 4, several methods raise one while lowering the other.

Rank is a cut

In LoRA every signal crosses the inner wire of dimension \(r\) between \(A\) and \(B\), so \(\Delta W=BA\) has rank at most \(r\). The argument works for any method drawn as a tensor network, a string diagram in \((\mathbf{FinVect},\otimes)\). Split the boxes into an input side and an output side, and the network factors through the wires that cross.

(expand)

Write \(V_{\mathcal I},V_{\mathcal O}\) for the input legs attached to \(\mathcal I\) and \(\mathcal O\), \(U_{\mathcal I},U_{\mathcal O}\) likewise for the outputs, and \(C\) for the tensor product of the cut wires. Contracting each side gives \(T_{\mathcal I}:V_{\mathcal I}\to U_{\mathcal I}\otimes C\) and \(T_{\mathcal O}:C\otimes V_{\mathcal O}\to U_{\mathcal O}\). Then

\[[\![\Gamma]\!]=(\mathrm{id}_{U_{\mathcal I}}\otimes T_{\mathcal O})\circ(T_{\mathcal I}\otimes\mathrm{id}_{V_{\mathcal O}})\]

factors through \(U_{\mathcal I}\otimes C\otimes V_{\mathcal O}\), whose dimension is \(w\). Symmetric monoidal structure suffices; no cups or caps are needed. Subadditivity is \(\operatorname{rank}(X+Y)\le\operatorname{rank}X+\operatorname{rank}Y\).

Numerical check: no violation in 283 random networks.

∎

Without the convention on bare strands, \(I_d\otimes BA\) would violate the bound. The bound also depends on the drawing. A naive expansion of HRA’s product of reflections has \(2^r-1\) terms, while a telescoping sum gives the sharp bound \(r\).

More: the bound for common methods

method	presentation used	bound on \(\operatorname{rank}\Delta W\)
LoRA family, \(BA\)	one bond of dimension \(r\)	\(r\)
split initialisations (PiSSA type), relative to \(W_0\)	\(\Delta W=BA-B_0A_0\), two bonds	\(2r\) (Thm II.1)
HRA	telescoping sum \(R-I=\sum_k(H_k-I)H_{k+1}\cdots H_r\)	\(r\)
Givens circuit with \(g\) gates	the same telescoping sum	\(2g\)
ReLoRA after \(K\) phases	formal sum of \(K\) LoRAs	\(Kr\)
GraLoRA	sum of \(k^2\) block LoRAs	\(kr\)
LoHa	Hadamard product through copy spiders	\(r_1r_2\)
HiRA	\(W_0\) factored through a bond of dimension \(\operatorname{rk}W_0\)	\(r\operatorname{rk}W_0\)
LoKr	\(C\otimes BA\), inner rank \(r'\)	\(\operatorname{rk}C\cdot r'\)
FourierFT, \(n_c\) coefficients	sum of real cosine patterns, each of rank \(\le2\)	\(\min(2n_c,m,n)\)
MoRA (reshape), C3A, MPO adapters	no narrow cut	up to full rank

The “high-rank PEFT” literature consists of networks whose min-cut avoids an \(r\)-dimensional bottleneck. A large min-cut is necessary for high rank and does not guarantee it, and attaining it can cost volume.

Dimension is a volume

Moving \(q\in Q\) along a fibre of \(\rho\) changes nothing. For LoRA, \((Bg^{-1},gA)\) gives the same weight as \((B,A)\) for every \(g\in GL_r\), so \(r^2\) of the \(r(m+n)\) parameters are spent on a symmetry and the image has dimension \(r(m+n-r)\). This is the expressive dimension \(d(M)\) of Definition I.5; \(\lvert M\rvert-d(M)\) is the gauge waste.

More: what “gauge” means here

“Gauge” means the symmetry group of a parametrisation. There is no connection, and away from the full-rank stratum the fibres jump in dimension.

(expand)

(a) The generic fibre contains the generic gauge orbit, and \(d=\lvert M\rvert-\dim(\text{generic fibre})\).

LoHa. The torus acts by

\[(B_1,A_1,B_2,A_2)\mapsto(D_1B_1,\,A_1D_2,\,D_1^{-1}B_2,\,A_2D_2^{-1}),\]

which preserves the product because \((D_1XD_2)\odot(D_1^{-1}YD_2^{-1})=X\odot Y\). The joint action of \(GL_{r_1}\times GL_{r_2}\times T_m\times T_n\) has generic stabiliser \(\{(cI,c^{-1}I,cI,c^{-1}I)\}\), so the orbit has dimension \(r_1^2+r_2^2+m+n-1\). Sketch for \(r_1\lt\min(m,n)\): an infinitesimal stabiliser \((\xi_1,\xi_2,\delta_1,\delta_2)\) satisfies \(\delta_1B_1=B_1\xi_1\), so the column space of \(B_1\) is invariant under the diagonal \(\delta_1\). A generic subspace of dimension below \(m\) is invariant only under scalar diagonals, so \(\delta_1=\xi_1=cI\). The equation for \(A_1\) then forces \(\delta_2=-cI\), and those for \(B_2,A_2\) give \(\xi_2=-cI\). The remaining corner shapes were checked numerically up to \(7\times7\). Trivially \(d\le mn\).

GraLoRA. The image is a product of \(k^2\) cones \(\mathcal M_{\le r/k}(\tfrac mk\times\tfrac nk)\), of total dimension \(k^2\cdot\tfrac rk\cdot\tfrac{m+n-r}{k}=r(m+n-r)\). For the rank, write block \((i,j)\) as \(U_{ij}V_{ij}^\top\) with \(U_{ij}\in\mathbb R^{m/k\times r/k}\) and \(V_{ij}\in\mathbb R^{n/k\times r/k}\). Then \(\Delta W=\mathrm{blockdiag}(L_1,\dots,L_k)\,S\), with \(L_i=[U_{i1}\cdots U_{ik}]\in\mathbb R^{m/k\times r}\) and \(S\in\mathbb R^{kr\times n}\) stacking the \(V_{ij}^\top\), each supported on column block \(j\). For generic blocks and \(kr\le\min(m,n)\), each \(L_i\) has full column rank \(r\) and \(S\) has full row rank \(kr\), so \(\operatorname{rank}\Delta W=kr\). In general the rank is at most \(\min(kr,m,n)\). Placing rank-\(r/k\) blocks only on the diagonal would give rank \(r\), not \(kr\).

(c). Since \(d(\mathrm{LoHa}_{r,r})\le F\) and \(d(\mathrm{LoRA}_{2r})=2r(m+n-2r)\) for \(2r\le\min(m,n)\),

\[d(\mathrm{LoRA}_{2r})-d(\mathrm{LoHa}_{r,r})\ge2r(m+n-2r)-F=(m+n-1)-2r^2.\]

LoHa\(_{r,r}\) maps into LoRA\(_{r^2}\) by the face-splitting product \(h=(B_1\bullet B_2,(A_1^\top\bullet A_2^\top)^\top)\), one of the arrow rules of the atlas. For \(r\le2\) this gives \(\mathrm{Im}\,\mathrm{LoHa}_{r,r}\subseteq\mathrm{Im}\,\mathrm{LoRA}_{r^2}\subseteq\mathrm{Im}\,\mathrm{LoRA}_{2r}\), and a positive gap makes the inclusion strict. Without \(m+n\ge2r^2+2\) the argument fails: at \((m,n,r)=(4,4,2)\) both images are 16-dimensional, and a numerical solver found exact LoHa\(_{2,2}\) decompositions of every target tried.

Other rows. The KronA image is a Segre cone, and \(\operatorname{rank}(A\otimes B)=\operatorname{rk}A\operatorname{rk}B\). HRA is Theorem II.7 for \(r\le n-2\); the cases \(r\in\{n-1,n\}\) were checked numerically. DoRA is Proposition II.8. The TT and Tucker counts are the cited results, which assume minimal ranks. The remaining rows are standard. For FourierFT, \(\mathrm{Re}\circ\mathrm{ifft2}\) sends the frequencies \(p\) and \(-p\) to the same real pattern.

Numerical check: LoHa dimensions \(38,59,93,101\) at \((m,n,r)=(8,7,2),(12,10,2),(12,10,3),(20,16,2)\), and \(16=mn\) at \((4,4,2)\), where the uncapped formula gives \(17\). GraLoRA dimensions \(48,64,108\) at \((m,n,r,k)=(8,8,4,2),(12,8,4,2),(12,12,6,3)\); generic rank \(8=kr\) at \((16,16,4,2)\) and \(12=\min(m,n)\) at \((12,12,6,3)\).

∎

When \(m+n\le2r^2+1\), still with \(2r\le\min(m,n)\) but without needing \(r^2\le\min(m,n)\), the count gives no gap, and wherever \(F\) is attained, as the numerical checks find, LoHa\(_{r,r}\) matches or exceeds LoRA\(_{2r}\) in dimension as well as rank, for example at \(r=64\) on a \(4096\times4096\) layer and at LyCORIS-style \(r=32\) on Stable Diffusion layers 320 to 640 wide, where \(r^2\gt\min(m,n)\).

When \(2r^2\) is small next to \(m+n\), as on LLM projections at the usual ranks, LoHa reaches high rank on a thin, curved subvariety, because its Hadamard product spends a whole torus on gauge. GraLoRA keeps LoRA’s dimension and caps every block’s rank.

Kronecker adapters cut the other way

If \(m=m_1m_2\) and \(n=n_1n_2\), an \(m\times n\) matrix is a tensor with output legs \(U_1,U_2\) and input legs \(V_1,V_2\). LoRA’s rank is the rank across \(\{U_1U_2\}\mid\{V_1V_2\}\); a sum of \(k\) Kronecker products has rank at most \(k\) across \(\{U_1V_1\}\mid\{U_2V_2\}\). The cuts disagree sharply, since \(I\otimes I\) is one Kronecker term of full rank.

(expand)

(a) and (b) follow from the Van Loan–Pitsianis rearrangement \(\mathcal R(C\otimes D)=\mathrm{vec}\,C\,\mathrm{vec}\,D^\top\), which is wire-bending; \(\mathcal R\Delta\) is an \((m_1n_1)\times(m_2n_2)\) matrix, and \(\operatorname{rk}_\otimes\Delta=\operatorname{rank}\mathcal R\Delta\).

(c) Write the Schmidt decompositions \(u=\sum\sigma_ix_i\otimes y_i\) and \(v=\sum\tau_jz_j\otimes w_j\). Then

\[uv^\top=\sum_{i,j}\sigma_i\tau_j\,(x_iz_j^\top)\otimes(y_iw_j^\top)\]

is an operator-Schmidt decomposition with Frobenius-orthonormal families and \(\rho_u\rho_v\) nonzero coefficients. Generic \(u,v\) have full Schmidt ranks \(\min(m_1,m_2)\) and \(\min(n_1,n_2)\). If \(m_1\le m_2\) and \(n_1\le n_2\), then \(\kappa_1=m_1n_1\le m_2n_2\), and symmetrically in the other aligned case.

(d) follows from (b) and (c). (e) A generic \(W_0+uv^\top\) lies in \(\mathrm{Im}\,\mathrm{LoRA}_r\) and, by (c), outside \(W_0+\mathcal K_{\le k}\) when \(k\lt\kappa_1\). For LoKr, (b) gives \(\operatorname{rank}(C\otimes BA)=\operatorname{rk}C\cdot\operatorname{rk}(BA)\le\min(m_1,n_1)\,r'\), with equality for full-rank \(C\) and \(BA\) of rank \(r'\). So LoKr reaches rank above \(r\) exactly when \(r\lt\min(m_1,n_1)\,r'\); and LoKr lies in \(W_0+\mathcal K_{\le1}\), so the first clause applies when \(\kappa_1\gt1\).

Numerical check: Schmidt ranks \((1,3)\to3\), \((2,3)\to6\), \((3,3)\to9\), each with ordinary rank \(1\); for the \(2\times4\) split of \(\mathbb R^8\), \(\operatorname{rk}_\otimes(uv^\top)=4\), the largest Kronecker rank there; LoKr with \(s=4\), \(r'=1\) never exceeds rank \(4\).

∎

Unconstrained Kronecker sums are LoRA across a different cut, so calling them higher-rank LoRAs mixes two cuts; with fewer than \(\kappa_1\) terms they miss a generic rank-one update. LoKr with inner rank \(r'\) lies inside LoRA\(_r\) only once \(r\ge\min(m_1,n_1)\,r'\), which is 192 on 768-wide layers and 512 on 4096-wide layers at HF’s default \(r'=8\). Below that, the images are incomparable.

More: LoKr in HF PEFT, Compacter and MPO

HF PEFT factors LoKr’s second matrix as \(BA\) only when its r is below half the larger of \(m_2,n_2\). Otherwise the second factor is a full matrix, LoKr is an unconstrained Kronecker product, and its maximum rank is \(\nu\). On a \(768\times768\) layer split \(24\otimes32\), the maximum rank jumps from 360 at r=15 to 768 at r=16. The decompose_both option also caps \(C\) at rank r.

Compacter’s PHM weights are Kronecker-structured weights inside a nonlinear bottleneck adapter, and MPO adapters are a multi-core reparametrisation that is lossless at full bond dimension. The image comparison does not transfer to either without more work.

Pred Under a loss whose only minimiser is the target weight, LoRA\(_1\) fits \(W_0+uv^\top\) (generic \(u,v\)) exactly and no Kronecker sum with fewer than \(\kappa_1\) terms does. HF PEFT’s factorisation is aligned, so \(\kappa_1\) is maximal there (576 for \(768=24\times32\)). At equal budget neither family should dominate across tasks.

In Figure 5, at rest, a generic \(uv^\top\) on the split \(4\otimes4\) has rank 1 and Kronecker rank 16. LoRA\(_1\) fits it with 32 parameters and a Kronecker sum with 16 terms and 512. For \(I_{16}\) the costs trade places.

Transport of structure

An invertible linear change of frame fixing \(\theta_0\) carries dimension, gauge, first-order image and degree along, since they are defined from \(\rho\) and its differential. It carries rank along only when it respects rows and columns, which the Kronecker rearrangement, Fourier and wavelet frames and the Hadamard rescaling by \(W_0\) do not.

(expand)

\(D(\tau_T\circ\rho)=T\,D\rho\). This gives the statements about \(S_1\), the ranks of the differential, \(d\), \(g\) and the deficit, and \(K^{T_*M}=TD\rho\,\Lambda\,D\rho^\top T^\top\). Since \(T_*\) does not touch \(Q\), \(L\circ\rho_{T_*M}=(L\circ\tau_T)\circ\rho_M\) exactly, so the two objectives on \(Q\) coincide, and the \(\theta\)-trajectories are conjugate by \(\tau_T\). A rule that projects \(\nabla_\theta L\) by a Frobenius least-squares step moves by \(-(D\rho^\top T^\top TD\rho)^+D\rho^\top T^\top\nabla L\) on \(T_*M\), which matches its step on \(M\) for \(L\circ\tau_T\) when \(T^\top T=I\). Orthogonal \(T\) also keep the Frobenius spectrum of \(K\) and isometry-invariant losses such as \(\|\theta-\theta^\star\|^2\). Coordinate permutations of \(Q\) commute with every coordinatewise update rule.

Rebasing: for an additive family with shape \(C\), the transported family has shape \(TC\) at every base, and \(R_K\) is a \(K\)-fold Minkowski sum, so \(R_K(T_*M)=\theta_0+T(C+\dots+C)=\tau_T(R_K(M))\) by induction. For OFT with \(T=2I\), \(n=2\) and a unit base vector, two transported cycles reach norm \(9\), while the transported closure has norms in \([1,3]\).

The circulant identity is the DFT diagonalisation, and \(\mathrm{Re}\circ\mathrm{ifft2}\) sends conjugate frequencies to the same real pattern. For HiRA, two cycles reach \(W_0\odot(J+Z_1)\odot(J+Z_2)\), and \((J+Z_1)\odot(J+Z_2)=J+Z_1+Z_2+Z_1\odot Z_2\).

Numerical check: \(\|K^{T_*M}-TK^MT^\top\|=2\times10^{-14}\) for a Gaussian \(T\); Adam on \(T_*M\) with \(L\) and on \(M\) with \(L\circ\tau_T\) agree to \(8\times10^{-11}\); two HiRA cycles give relative-update rank \(3,8,15\) at \(r=1,2,3\) (\(m=n=30\)).

∎

More: the instances in detail

• The mask readings assume a start at \(\theta_0\). HF’s defaults for FourierFT (a random spectrum unless init_weights=True) and C3A (xavier_uniform) start away from it.
• FourierFT: HF’s \(\Delta W=s\cdot\mathrm{Re}(\mathrm{ifft2}(S))\), with a fixed scaling \(s\), is sparse in the real cosine half of the real Fourier frame. Each sampled conjugate pair \((u,v),(-u,-v)\) adds one dimension of gauge. The image lies in the point-symmetric subspace \(\Delta W_{j,l}=\Delta W_{-j,-l}\), and the frame is orthogonal only up to scale (self-conjugate frequencies have twice the squared norm).
• C3A: block-circulant. Over \(\mathbb R\) the frame is given by the powers \(P^k\) of the cyclic shift (Gram matrix \(bI\)). Over \(\mathbb C\), \(\mathrm{circ}(w)=F^{-1}\mathrm{diag}(Fw)F\) for first column \(w\), and the real form has \(2\times2\) rotation–scaling blocks. HF adds a factor \(1/\mathrm{in\_features}\).
• SHiRA, FISH and Trainable Tokens are masks in the standard frame.
• MiSS/Bone in its default (balance) mode is \(\Delta W=D(\mathbf 1_g^\top\otimes I_r)\), LoRA-FA with a frozen summing projection. Its BAT mode, blockwise \(\Delta W=\mathrm{blocks}(W_0)\,D+D\), is linear in \(D\) and depends on \(\theta_0\).
• LoKr: in HF PEFT the rank constraint on the second factor is active only below the threshold given after Theorem II.11, and decompose_both also constrains the first factor.

Rank is invariant under \(GL_m\times GL_n\) and changes under \(GL(\Theta)\), so “high rank” claims (MoRA, HiRA, LoKr, C3A, SHiRA, WaveFT) depend on a frame that the network function does not respect. Transport keeps every isomorphism invariant (\(d\), \(g\), gauge group, deficit, apex or smooth type, degree) and conjugates the dynamics. In Figure 4 the HiRA dot sits at LoRA’s height and at full rank (Prop II.9(a)).

Pred At matched learning-rate sweeps, the gap between a transported method and its source (Kronecker sums against LoRA, Fourier or wavelet methods against masks) tracks how well each image aligns with a full fine-tuning \(\Delta W\), more than it tracks optimizer effects.

The plug, and a reporting rule

Cut \(W_0\) out of a linear layer and a hole \(V\to U\) remains. A weight-space method fills it with a small network of frozen and trainable boxes, a plug. Every invariant above is read off the plug: the budget (Def I.2), the min-cut, the Jacobian rank (Def I.5), the gauges on its wires, and the merge type, set by whether an activation sits on the path (Thm VI.3).

Here is one \(4096\times4096\) box and one budget, spent three ways (Proposition II.12).

The methods differ by at least 8,063 dimensions and by a factor of four in rank. The LoHa row uses the upper bound \(F\), whose equality is checked numerically at small sizes and open in general.

Pred Benchmarks should report \(d(M)\) beside \(\lvert M\rvert\), with the maximum rank and the cut it refers to. At budgets of a few thousand parameters, rankings at equal effective dimension can change.

At rest, Figure 6 shows LoHa\(_2\) on an \(8\times8\) box, with 64 entries, rank at most 4, Jacobian rank 41 and a gauge orbit of dimension \(23=4+4+15\) that fills the fibre. LoRA\(_4\) costs the same and reaches 7 more dimensions, with LoHa’s image strictly inside it.

Covariance and descent

A method is usually one recipe applied to every pretrained weight. Rename a layer’s input basis by \(h\) and its output basis by \(g\), so that \(W_0\) becomes \(gW_0h^{-1}\); the method is covariant under \((g,h)\) if its image moves the same way.

More: in what sense this is Klein’s

Klein defined a geometry by its group of motions and studied what the group leaves unchanged. For group-type methods this reading is a theorem (Exposé II.B). For cones such as LoRA’s image it is a naming device, and the content is the shape of the image and its covariance group.

LoRA’s image \(W_0+\mathcal M_{\le r}\) is covariant under all of \(GL_m\times GL_n\), since rank ignores bases, while a coordinate mask has a much smaller group. This matters because networks have symmetries of their own, such as permutations of hidden neurons, rotations of the residual stream and the query–key gauge of attention, which change the weights and keep the function.

(expand)

Apply per-box covariance factor by factor; the frozen coordinates are single points, so \(\mathrm{Im}\,M(g\theta_0)=g\,\mathrm{Im}\,M(\theta_0)\). Then

\[\{\Phi_f(\theta):\theta\in\mathrm{Im}\,M(g\theta_0)\}=\{\Phi_f(g\theta):\theta\in\mathrm{Im}\,M(\theta_0)\}=\{\Phi_f(\theta):\theta\in\mathrm{Im}\,M(\theta_0)\}.\]

With independent per-box frames, the joint law of the image is the product of the per-box laws, and each factor moves covariantly in law. Shared frames are excluded because covariance in law box by box does not control the joint law. SliceGPT’s per-block rotations need extra residual matrices, which change the architecture.

Attention gauges. With no positional encoding or an absolute one, any \(g\in GL_{d_k}\) preserves the scores. Under RoPE, \(g\) must commute with every RoPE rotation, which leaves the per-frequency rotation–scalings \((\mathbb C^\times)^{d_k/2}\), shared across a GQA group, or \((\mathbb C^\times)^{r/2}\times GL_{d_k-r}\) when RoPE covers \(r\) of the dimensions. Under QK-norm, \(g\) must in addition be orthogonal up to a positive scale (exactly orthogonal when the norm has \(\varepsilon\gt0\)) and compatible with the norm gains.

Numerical check: a tied rank-one update shared by two boxes violates the conclusion (relative distance up to \(0.30\)), while independent per-box LoRA satisfies it. Under RoPE a generic \(g\in GL_8\) changes attention scores by up to \(313\), a RoPE-commuting \(g\) by \(5\times10^{-14}\).

∎

For one-sided methods ((IA)\(^3\), DoRA, OFT, BOFT), descent depends on the side: a residual rotation acts on the input side of \(q\), \(k\), \(v\), gate and up and on the output side of \(o\) and down. LoHa with \(r\ge2\), HiRA and magnitude-chosen masks pass hidden-unit permutations and fail rotations. The table compares gain-fused models with and without the rotation, since fusing gains is a diagonal rescaling. Failing a sufficient condition does not prove that reachable sets differ.

Conj A Against a gain-fused, unrotated control, a SliceGPT-style rotation measurably changes the fine-tuning results of block OFT/BOFT on residual-input projections, LoHa, HiRA and coordinate masks, because LLM residual streams have a privileged basis. Adam is only \(B_r\)-equivariant (Thm III.12) and HF’s Kaiming-uniform \(A\) is not rotation-invariant in law, so LoRA as trained need not descend even though its image does.

SVD-based initialisations raise the same question. PiSSA, MiLoRA and CorDA split \(W_0\) into a trainable part and a frozen residual by a truncated SVD of \(W_0\), or, for CorDA, of \(W_0\) weighted by the input second moment. Truncated SVD commutes only with rotations and uniform scalings, so these splits depend on the input coordinates, and whitening removes that dependence.

(expand)

Definition II.14 concerns images. For \(r\lt p\) the translation stabiliser of \(\mathcal M_{\le r}\) is \(\{0\}\), so covariance of the image \(W_0-X+\mathcal M_{\le r}\) is equivalent to covariance of the split \(X\).

(a) With \(L'=hL\), \([W_0h^{-1}hL]_r(hL)^{-1}=[W_0L]_rL^{-1}h^{-1}\), and the result does not depend on the square root chosen, since \([W_0LO]_rO^\top=[W_0L]_r\) for \(O\in O(n)\). Also \([W_0L]_r=\Pi_rW_0L\), because the top left singular vectors of \(W_0L\) are the top eigenvectors of \(W_0LL^\top W_0^\top\). On the output side, for \(g\in CO(m)\) the eigenprojector of \(gW_0CW_0^\top g^\top\) is \(g\Pi_rg^{-1}\).

(b) Lemma. For \(1\le r\lt p\), \([Mh]_r=[M]_rh\) for all \(M\) with a gap at \(r\) iff \(h\in CO(n)\). “If”: \(M\,cQ\) has the singular vectors of \(M\) rotated by \(Q\) and the singular values scaled by \(c\). “Only if”: take \(M=\sum_{i\le r+1}\sigma_iu_iv_i^\top\) with distinct \(\sigma\)’s. The top-\(r\) left singular space of \(Mh\) is \(\mathrm{span}\{u_1,\dots,u_r\}\) only if the coupling \(\sigma_r\sigma_{r+1}v_r^\top hh^\top v_{r+1}\) vanishes; requiring this for every orthonormal pair forces \(hh^\top\in\mathbb R_{\gt0}I\).

CorDA IPM’s covariance requires \([W_0Ch^\top]_rh^{-\top}C^{-1}h^{-1}=[W_0C]_rC^{-1}h^{-1}\), which is the lemma for \(h^\top\) applied to \(M=W_0C\), an arbitrary matrix. PiSSA is the lemma for \(h^{-1}\). For the bottom-\(r\) splits (KPM, MiLoRA), \(\langle M\rangle_r=M-[M]_{p-r}\), so the lemma applies at index \(p-r\).

Numerical check: under \(h=3Q\) with \(Q\) orthogonal, CorDA and PiSSA move covariantly to \(2\times10^{-15}\). Under a generic \(h\) only the whitened split does (\(2\times10^{-15}\), against \(0.63\) and \(1.1\)), and an absolute damping \(0.1\) breaks it (\(0.14\)).

∎

CorDA damps \(C\) by a multiple of \((\operatorname{tr}C)\,I\), which keeps the \(CO(n)\) covariance; an absolute damping \(C+\lambda I\) would cut it to \(O(n)\). The released CorDA code also divides each calibration sample by the absolute value of its largest entry, which keeps only positive scalings and coordinate permutations and so breaks even rotation covariance; (b) covers the published formulas only.

A data-aware split ignores the choice of input coordinates if it has the form \(X=F(W_0CW_0^\top)\,W_0\), as the whitened (SVD-LLM-style) split does; Astra’s paper form, a tail eigenprojector of the centred output covariance, has the same form with \(C\) replaced by the centred input covariance, which transforms in the same way. Diagonal rescalings such as norm-gain folding or SmoothQuant change the CorDA and PiSSA splits. Covariance on both sides would need an output-side statistic independent of \(W_0\) and \(C\), such as a gradient second moment, and training dynamics stay non-covariant.

More: EVA

EVA (Paischer et al., 2024) does not split \(W_0\). It starts at \(B_0=0\) and points \(\mathrm{row}\,A_0\) at the top-\(r\) centred principal subspace of the inputs, which is \(CO(n)\)-covariant. At a fixed rank per layer its image \(W_0+\mathcal M_{\le r}\) is \(GL_m\times GL_n\)-covariant under Definition II.14. With HF’s default rho=2, ranks are redistributed by explained-variance ratios, which are invariant only under \(CO(n)\), so the image is \(GL_m\times CO(n)\)-covariant.

Conj C Whitened CorDA or PiSSA is more robust to function-preserving input reparametrisations, and no worse otherwise.

Rank is a cut, dimension a volume and covariance a group. All three can be computed before training. The next exposé turns from what a method can reach to how it moves there.

One experiment first

Training happens upstairs, in the parameter space \(Q\), where each fibre of \(\rho\) collects the parameters that give one weight. The optimizer’s metric on \(Q\), pushed down to weight space, is what the model feels. Every statement below names its optimizer.

Figure 7 shows the cleanest consequence. Train with Adam at \(\varepsilon=0\) from \(B_0=0\), with no weight decay or clipping, one schedule for both groups, and the same data order and dropout masks. Then LoRA+ with ratio \(\lambda\) (\(\eta_B=\lambda\eta_A\)) and plain LoRA with \(\alpha\) multiplied by \(\lambda\), at the same \(\eta_A\) and \(A_0\), produce the same weights at every step (Thm III.10, Cor III.11).

The metric a method carries

A learning rate is a metric in disguise. Gradient descent with rates \(\Lambda\) moves \(q\) by \(-\Lambda\nabla_q\), and pushed down by \(D\rho\) this is the weight gradient multiplied by a positive semidefinite operator.

Cometrics compose the way derivatives do.

(expand)

\(D(\rho\sigma)=D\rho\,D\sigma\), so \(K^{\rho\sigma}=D\rho\,D\sigma\,\Lambda'D\sigma^\top D\rho^\top\). For LoRA, use \(\nabla_B=sGA^\top\), \(\nabla_A=sB^\top G\) and \(\dot W=s(\dot BA+B\dot A)\). Discrete GD adds the term \(s\,\delta B\,\delta A\), of order \(\eta^2\).

Num Finite-difference Jacobians agree with the formula to \(6\times10^{-9}\).

∎

For LoRA, \(\eta_B\,GA^\top A\) lets the update reach all outputs while filtering its inputs through \(\mathrm{row}\,A\), and \(\eta_A\,BB^\top G\) does the reverse. At \(B_0=0\) only the first survives, so the first-step cometric \(s^2\eta_B\,GA_0^\top A_0\) is fixed by \(A_0\) (Cor III.13).

More: what prefix tuning’s MLP changes

Take \(X\) the prefix space (Exposé VI). HF PrefixTuning with prefix_projection=True (Li and Liang’s MLP \(\sigma\)) and False (P-tuning v2, minus its head) have the same prefix image when encoder_hidden_size \(\ge\) num_virtual_tokens \(-1\). Their cometrics are \(D\sigma\Lambda'D\sigma^\top\) and \(\Lambda\), so the MLP makes a flat metric point-dependent; that alone does not explain the instability Li and Liang saw under AdamW.

When a change of chart does not matter

A gauge transformation changes the parameters and keeps the weights. It leaves training unchanged exactly when the cometrics agree, which for LoRA’s gauge \(GL_r\) singles out \(O(r)\).

(expand)

(a) Put \(\ell=L\circ\rho'\), so that \(L\circ\rho=\ell\circ\varphi\). The \(\rho\)-flow is \(\dot q=-\Lambda D\varphi^\top\nabla\ell\), so \(\tfrac{d}{dt}\varphi(q)=-D\varphi\Lambda D\varphi^\top\nabla\ell\). Apply \(D\rho'\) and use \(\nabla\ell=D\rho'^\top\nabla_\theta L\). The \(\rho'\)-flow gives \(-D\rho'\Lambda'D\rho'^\top\nabla_\theta L\). Since \(\nabla_\theta L\) is arbitrary, the criterion follows. For the examples, a linear \(\rho\) has constant \(D\rho\), and \(D\rho\,P=D\rho\) for a linear gauge \(P\); for instance, for \(\rho(x,y)=\theta_0+x+y\) the shear \((x,y)\mapsto(2x,y-x)\) preserves \(\rho\) and \(K\) without being orthogonal. For a pointwise gauge, \(D\varphi_q=D\varphi_{g(q)}+V\) with \(V\) valued in fibre directions, which \(D\rho'\) kills, so the criterion reduces to that of \(\varphi_{g(q)}\), which (b) covers.

(b) The two cometrics agree for all \(G\) iff \[\eta_B\,GA^\top(g^\top g-I)A=\eta_A\,B(I-g^{-1}g^{-\top})B^\top G\quad\text{for every }G.\] An identity \(GX=YG\) for all \(G\) forces \(X=cI_n\) and \(Y=cI_m\) (test it on the matrix units \(E_{ij}\)). Here \(\operatorname{rank}X\le r\lt n\), so \(c=0\), and full row rank of \(A\) gives \(g^\top g=I\). Only the full row rank of \(A\) is used.

Num At a random point a non-orthogonal \(g\) gives a cometric mismatch of about \(18\), an orthogonal one about \(10^{-15}\).

∎

So gradient descent on LoRA is a family of dynamics indexed by \(O(r)\backslash GL_r\cong\mathrm{Sym}^+_r\), via \(g\mapsto g^\top g\). Under GD, LoRA+’s ratio \(\lambda\) is the gauge \(g=\lambda^{1/4}I\) plus an overall change of rate, while rsLoRA is a move on the hyperparameter torus (Thm III.10).

Some published optimizers use instead a cometric that depends only on the weight, at least in an idealised form.

Practice. As published and implemented, these methods use damped inverses, AdamW or a Sylvester-chosen free matrix, which keep only the symmetries of Thm III.12. At \(B_0=0\), off the full-rank locus, (a) and (b) need damping; Zhang and Pilanci report damped scaled GD and scaled AdamW, and HF’s create_riemannian_optimizer and LoRA-Pro’s code damp too. LoRA-Pro’s experiments use \(\dot W=-\Pi_T(\mathrm{Adam}(\Pi_TG))\), which is no gradient flow, and revised LoRA-FA and HF’s create_lorafa_optimizer apply AdamW after the correction, so the basis and scale of \(A_0\) matter too.

(expand)

(a) Substitute \(\nabla_B=sGA^\top\) and \(\nabla_A=sB^\top G\) into \(\delta W=s(\delta BA+B\delta A)+O(\eta^2)\), using \(A^\top(AA^\top)^{-1}A=P_V\) and \(B(B^\top B)^{-1}B^\top=P_U\). For equivariance: under \((Bg^{-1},gA)\) we have \(\nabla_{B'}=\nabla_Bg^\top\) and \((A'A'^\top)^{-1}=g^{-\top}(AA^\top)^{-1}g^{-1}\), so \(\delta B'=\delta B\,g^{-1}\); the same holds for \(A\).

(b) The tangent space is \[T_{W-W_0}\mathcal M_r=\{XA+BY\}=\{D:P_{U^\perp}DP_{V^\perp}=0\},\] and the orthogonal projection onto it is \[G-P_{U^\perp}GP_{V^\perp}=P_UG+GP_V-P_UGP_V=\tilde g.\] For any free matrix \(X\) in Alg. 1, \(s(h_BA+Bh_A)=P_UG+P_{U^\perp}GP_V\), so \(\delta W=-\eta\,\Pi_TG+O(\eta^2)\).

(c) \[\Delta W=s\,\Delta B\,A_0=-\eta\,GA_0^\top(A_0A_0^\top)^{-1}A_0=-\eta\,GP_V,\] exactly, since \(A_0\) is frozen.

(d) For (a) and (b), read off the formulas. For (c), \(XA_0\) has rank \(r\) iff \(X\) does, an open dense condition, and then \(\mathrm{row}(XA_0)=V\), so \(GP_V=GP_{\mathrm{row}(W-W_0)}\) lies in \(\{XA+BY\}\).

Num Gauge defects are about \(10^{-15}\) for (a), against \(52\) for plain GD. LoRA-FA with AdamW from \(A_0\) and from \(gA_0\) differs by \(O(1)\) after a few steps, against \(10^{-15}\) with SGD.

∎

Under (S)GD each method is summarised by its weight-space cometric. LoRA, scaled GD and LoRA-Pro share the image \(W_0+\mathcal M_{\le r}\) and differ only in cometric; LoRA-FA is the sub-object \(W_0+\{XA_0\}\); GaLore (with a gradient-SVD or a random projector) and MeZO are full-parameter methods, listed for comparison. Flora proper compresses only its momentum, takes full-space Adafactor steps and has no cometric of this form (Exposé IV).

Isometries and charges

Split LoRA’s gauge in two. With per-block scalar rates, the antisymmetric generators rotate the rank channels and preserve the rate metric, so the optimizer cannot feel them. Along the symmetric ones, gradient flow conserves a quadratic charge; for LoRA the charges form one matrix \(\Phi\), whose vanishing is the balancedness of deep linear networks.

(expand)

Invariance of \(\rho\) gives \(D\rho_q(\xi q)=0\). Hence \[\tfrac{d}{dt}J_\xi=\langle\Lambda^{-1}\xi q,\dot q\rangle=-\langle\xi q,D\rho_q^\top\gamma\rangle=-\langle D\rho_q\xi q,\gamma\rangle=0.\]

For (ii), \(\Lambda^{-1}\xi\) antisymmetric gives \(\tfrac{d}{d\tau}\big(e^{\tau\xi^\top}\Lambda^{-1}e^{\tau\xi}\big)=0\), equivalently \(k\Lambda k^\top=\Lambda\). From \(\rho\circ k=\rho\) we get \(D\rho_q=D\rho_{kq}k\), so \[K_q=D\rho_{kq}k\Lambda k^\top D\rho_{kq}^\top=K_{kq},\qquad \tfrac{d}{dt}(kq)=-k\Lambda k^\top D\rho_{kq}^\top\gamma=-\Lambda D\rho_{kq}^\top\gamma.\] When \(\gamma\) factors through \((t,\rho)\), \(kq(t)\) solves the same ODE as the trajectory from \(kq_0\); under the stated regularity the solution is unique.

(iii) Write \[\xi=\tfrac12(\xi+\Lambda\xi^\top\Lambda^{-1})+\tfrac12(\xi-\Lambda\xi^\top\Lambda^{-1}).\] For per-channel rates \(\Lambda(B,A)=(BD_B,D_AA)\), \(\Lambda^{-1}\xi_X(B,A)=(BXD_B^{-1},-D_A^{-1}XA)\) is symmetric (antisymmetric) iff \(XD_B^{-1}\) and \(D_A^{-1}X\) both are. Entrywise, \(X_{ji}=\pm X_{ij}\eta_{B,i}/\eta_{B,j}\) and \(X_{ji}=\pm X_{ij}\eta_{A,j}/\eta_{A,i}\), which admit \(X_{ij}\ne0\) iff \(\eta_{B,i}\eta_{A,i}=\eta_{B,j}\eta_{A,j}\). So an off-diagonal pair gives a charge and an isometry exactly when its products agree, and closure holds iff all products agree.

Num With a non-gradient, \(q\)-dependent \(\gamma\) on LoRA, \(\Phi\) drifts by \(1.6\times10^{-12}\). With Euclidean clipping and a non-Euclidean isometry \(k\), the \(\theta\)-trajectories from \(q_0\) and \(kq_0\) differ by \(0.06\).

∎

Instances, under gradient flow or SGD without momentum, with per-block rates. AdamW and heavy-ball momentum fall outside the theorem, since their buffers hold \(D\rho\) at past points.

• LoRA. \(\xi_X(B,A)=(BX,-XA)\) gives \(\Lambda^{-1}\xi_X\) symmetric iff \(X=X^\top\), so the \(r(r+1)/2\) charges are the entries of \(\Phi=B^\top B/\eta_B-AA^\top/\eta_A\). The \(\mathfrak{so}_r\) half of \(\mathfrak{gl}_r\) acts by Euclidean isometries, so (ii) survives global-norm clipping.
• DyLoRA: only \(\mathrm{diag}\,\Phi\), as the gauge reduces to the torus. VeRA: \(\|b\|^2/\eta_b-\|d\|^2/\eta_d\), from \((b,d)\mapsto(cb,d/c)\); NOLA is analogous.
• AdaLoRA (\(P\Lambda Q\)), between masking events, at equal rates and without its orthogonality regulariser (HF default orth_reg_weight=0.5, which breaks them): charges \(\|p_k\|^2-\lambda_k^2\) and \(\|q_k\|^2-\lambda_k^2\).
• HRA: a conserved \(\|u_i\|^2\). ReLU bottleneck adapters: one charge per hidden unit. TT adapters: one conserved matrix per bond.
• DoRA’s detached-norm backward and global-norm clipping keep the form \(\Lambda D\rho_{\rm LoRA}^\top\gamma\), so \(\Phi\) is conserved.

Conservation needs only the shape \(\dot q=-\Lambda D\rho^\top\gamma\), so it holds for any loss and data. With \(\eta\) the common scale of the rates, a GD step changes \(\eta\Phi\) by \(O(\eta^2)\) and an Adam step by \(O(\eta)\), as Panel A of Figure 8 shows.

A conserved charge is fixed at time zero, so it labels initialisations. Weight decay moves it on purpose, and its two conventions move it toward different targets.

(expand)

(a) Substitute the initialisations. Each Kaiming entry has variance \(1/(3n)\) and there are \(n\) entries per row, so the diagonal has mean \(1/3\) with fluctuations \(O(n^{-1/2})\), and the off-diagonal entries have standard deviation \(1/(3\sqrt n)\). HF’s pissa_init divides \(S_r\) by the scale before splitting. Under \(q\mapsto(BO,O^\top A)\) with \(O\in O(r)\), \(\Phi\mapsto O^\top\Phi O\).

(b) Differentiate: \[\tfrac{d}{dt}\tfrac{B^\top B}{\eta_B}=-(AG^\top B+B^\top GA^\top)-2c_B\tfrac{B^\top B}{\eta_B},\] with \(c_B=\lambda\) in the per-time convention and \(c_B=\eta_B\lambda\) in the coupled one. Do the same for \(A\) and subtract.

Num With coupled decay at \(\eta_B/\eta_A=4\), the closed form (taken with \(\eta=\eta_A\)) has relative error of order \(10\); in the valid cases the error is \(10^{-8}\) to \(10^{-4}\), the integrator’s. An Adam step moves \(\eta\Phi\) at first order in the learning rate, an SGD step at second order.

∎

Zero-product starts are maximally unbalanced (Prop III.8). HF’s PiSSA starts with \(\mu_0=B_0^\top B_0-A_0A_0^\top=0\) at every rate, but with \(\Phi_0=0\) only at equal rates.

Both decay conventions gauge-fix at equilibrium, each being a \(\Lambda\)-gradient flow of \(L\) plus a Kempf–Ness potential on the fibres. Per-time decay adds \(\tfrac\lambda2(\|B\|^2/\eta_B+\|A\|^2/\eta_A)\) and drives \(\Phi\to0\); coupled decay adds \(\tfrac\lambda2(\|B\|^2+\|A\|^2)\), with equilibria on the slice \(\mu=0\) of Prop III.7(a). The slices coincide iff \(\eta_A=\eta_B\) [Num: at \(\eta_B/\eta_A=4\) each run reaches its own slice to \(10^{-12}\) and stays \(O(1)\) from the other].

The balanced slice

In Kempf–Ness theory, the least-norm vectors on a closed orbit of a reductive group form one orbit of its compact part, and they are where the moment map vanishes. For LoRA the moment map is \(\mu=B^\top B-AA^\top\), and the least-norm representatives are the balanced ones.

(expand)

(a) For \(\xi\in\mathfrak{gl}_r\) with symmetric part \(\xi_s\), \[\tfrac{d}{dt}\big|_0\big(\|Be^{-t\xi}\|^2+\|e^{t\xi}A\|^2\big)=-2\operatorname{tr}(\xi_s\mu).\] This vanishes for all \(\xi\) iff \(\mu=0\). The function is \(O(r)\)-invariant, so by the polar decomposition \(GL_r=O(r)\exp(\mathrm{Sym}_r)\) it suffices to study \(h(t)=f(e^{t\xi}\cdot x)\) for symmetric \(\xi=P\,\mathrm{diag}(d)P^\top\). Then \[h''(t)=4\sum_id_i^2\big(\|(BP)_{:,i}\|^2e^{-2td_i}+\|(P^\top A)_{i,:}\|^2e^{2td_i}\big)\ge0,\] so \(h\) is convex and critical points are minima. If \(x\) and \(e^{\xi}\cdot x\) are both balanced, then \(h'(0)=h'(1)=0\), so \(h''\equiv0\) on \([0,1]\). Since \(B\) has full column rank, every \((BP)_{:,i}\ne0\), which forces all \(d_i=0\). Hence the balanced points form one \(O(r)\)-orbit. For existence, write \(BA=U\Sigma V^\top\); the representative \(B=U\Sigma^{1/2}O\), \(A=O^\top\Sigma^{1/2}V^\top\) is balanced, with value \(2\operatorname{tr}\Sigma\).

(b) Conservation of \(\Phi\) (Thm III.5) with \(B_0=0\) gives \[B^\top B/\eta_B-AA^\top/\eta_A=-A_0A_0^\top/\eta_A.\] For per-time decay, use Cor III.6(b).

Num On 5 random orbits at \((m,n,r)=(7,5,3)\) the minimum equals \(2\|BA\|_*\) to \(10^{-14}\).

∎

An explicit penalty \(\lambda(\|B\|^2+\|A\|^2)\) equals, at its minimisers, a \((2\lambda/s)\)-nuclear-norm penalty on \(\Delta W=sBA\) of rank \(\le r\), and SGD-coupled decay with coefficient \(\lambda\) a \((\lambda/s)\)-penalty; stationary points of either lie on \(\mu=0\). Per-time decay at unequal rates targets \(\Phi=0\), and decoupled AdamW decay is not covered.

The default start is as unbalanced as possible, and no start meets all three natural wishes.

(expand)

Assume (a) and (b). Then \(0=B_0^\top B_0A_0=cA_0A_0^\top A_0\), so \((A_0^\top A_0)^2=0\). Since \(A_0^\top A_0\) is PSD, it vanishes, so \(A_0=0\) and then \(B_0=0\). At \((0,0)\), \(D\rho(\dot B,\dot A)=s(\dot BA+B\dot A)=0\), and also \(\nabla_B=\nabla_A=0\); so every gradient-based update rule, including Adam, momentum and weight decay, is stationary there. For the identity, \[\operatorname{tr}(B_0^\top B_0A_0A_0^\top)=\operatorname{tr}(B_0^\top(B_0A_0)A_0^\top)=0.\]

∎

Type I starts keep \(\rho\) and pick \(q_0\) in the apex fibre \(\{BA=0\}\): LoRA, EVA, Init[B], all maximally unbalanced. Type II starts change \(\rho\) by an exact split \(\theta_0=\theta_{\rm res}+\delta(q_0)\): PiSSA, MiLoRA, OLoRA, LoRA-GA, CorDA. LoftQ and QPiSSA split only approximately, over a quantised residual, so they are not identity-at-init (Prop V.2). By the trilemma, a balanced, non-stationary, identity-at-init start must be Type II, though Type II starts such as OLoRA and CorDA can be unbalanced; by Thm II.2 they are different objects, incomparable with LoRA.

Two regimes and a crossover

When the charge is a multiple of the identity, the LoRA flow has a closed form. Below a scale \(\sigma^*\), one factor carries the motion, as in LoRA-FA; above it, the factors share the work, as in balanced deep linear networks.

(expand)

Put \(\beta=B/\sqrt{\eta_B}\) and \(\alpha=A/\sqrt{\eta_A}\). Then \(\beta^\top\beta-\alpha\alpha^\top=-\kappa I\) and \(\beta\alpha=Z/\mu\). Let \(T=\beta\beta^\top\) and \(S=\alpha^\top\alpha\).

• \[ZZ^\top/\mu^2=\beta(\beta^\top\beta+\kappa I)\beta^\top=T^2+\kappa T,\] and likewise \(Z^\top Z/\mu^2=S^2-\kappa S\).
• On the rank-\(r\) locus, \(\alpha\alpha^\top\succ0\) and \(\beta^\top\beta\succ0\). The nonzero eigenvalues of \(T\) equal those of \(\beta^\top\beta=\alpha\alpha^\top-\kappa I\), so they exceed \(\max(0,-\kappa)\); those of \(S\) exceed \(\max(0,\kappa)\).
• On these ranges, \(t\mapsto t^2+\kappa t\) and \(s\mapsto s^2-\kappa s\) are strictly increasing and send \(0\) to \(0\), with positive roots \(f_\kappa\) and \(g_\kappa\). The compact SVD matters here, because \(f_\kappa(0)=\lvert\kappa\rvert\) for \(\kappa\lt0\) and \(g_\kappa(0)=\kappa\) for \(\kappa\gt0\), so the full matrix function would be wrong on \(\ker Z\). Hence \(T=Uf_\kappa(\Sigma^2/\mu^2)U^\top\) and \(S=Vg_\kappa(\Sigma^2/\mu^2)V^\top\).
• Finally, \[\dot W=-s^2(\eta_BGA^\top A+\eta_ABB^\top G)=-\mu^2(GS+TG).\]

Off the locus: for \(\kappa\ge0\), \(T=f_\kappa(ZZ^\top/\mu^2)\) still holds as a full matrix function, but \(S\) gains the term \(\kappa(P_{\mathrm{row}A}-P_V)\), since \(S=\alpha^\top\alpha\) also has the eigenvalue \(\kappa\) on \(\mathrm{row}A\ominus\mathrm{row}Z\); symmetrically, for \(\kappa\lt0\), \(T\) gains \(\lvert\kappa\rvert(P_{\mathrm{col}B}-P_U)\). At \(Z=0\) this leaves \(S=\kappa P_{\mathrm{row}A_0}\) and \(T=0\).

Num The formula matches direct integration to \(10^{-12}\), and the pointwise identity holds to \(10^{-15}\) for \(\kappa\in\{-2,-0.3,0,0.3,2,10\}\) and all shape cases.

∎

Default LoRA has \(B_0=0\) and \(A_0A_0^\top=cI\), so \(\kappa=c/\eta_A\) and \(\sigma^*=\tfrac{sc}2\sqrt{\eta_B/\eta_A}\), with \(c\approx1/3\) for HF’s default. LoRA+ multiplies \(\sigma^*\) by \(\sqrt\lambda\) at fixed \(\eta_A\). PiSSA at equal rates has \(\kappa=0\) and stays balanced under gradient flow, though not under Adam.

The theorem is exact for orthonormal frames (EVA, LoRA-GA). For Kaiming starts the eigenvalues of \(A_0A_0^\top\) lie near \(c(1\pm\sqrt{r/n})^2\) (Marchenko–Pastur), so the crossover is a band of relative width \(O(\sqrt{r/n})\); at \(r=256\), \(n=4096\), those of \(3A_0A_0^\top\) span \(0.57\) to \(1.57\). Prediction Under SGD, singular values of \(\Delta W\) switch from one-sided to two-sided growth as they cross \(\sigma^*\).

The hyperparameter torus

Rescale block \(i\) by \(c_i\) and divide \(s\) by the matching power of the \(c_i\). The method is the same object of \(\mathsf{Meth}\), and training is the same if every step rescales alike. Adam ignores the size of a block’s gradient, so its rate must scale like \(c_i\); an SGD step scales with the gradient, which shrinks by \(c_i\), so its rate must scale like \(c_i^2\).

(expand)

The rescaling \(q_i'=c_iq_i\), together with the compensating \(s\), is an isomorphism in \(\mathsf{Meth}\). At corresponding points, \(\nabla_{q_i'}=c_i^{-1}\nabla_{q_i}\). With \(\varepsilon_i'=\varepsilon_i/c_i\), Adam’s normalised step is invariant to the positive rescaling \(c_i^{-1}\) of a block’s whole gradient history, so \(\Delta q_i'=c_i\Delta q_i\) requires \(\eta_i'(t)=c_i\eta_i(t)\), and the decoupled decay term \(\eta_i'\lambda_i'q_i'=c_i\eta_i\lambda_iq_i\) requires \(\lambda_i'=\lambda_i/c_i\).

For SGD and momentum the buffers are linear in the gradients, so the gradient factor \(c_i^{-1}\) combines with the required \(c_i\) to give \(c_i^2\), and coupled decay enters the gradient as \(\lambda_i'q_i'\), giving \(\lambda_i/c_i^2\). Clipping decisions agree when the thresholds scale like the gradients. The time-dependence of \(\eta_i\) plays no role. Conclude by induction over steps.

Num Trajectory differences are exactly \(0.0\) for power-of-two \(c\) and about \(10^{-15}\) otherwise, including a warm-up schedule and rescaled \(\varepsilon\) and decay. With \(\varepsilon\) or decay held fixed, or with global-norm clipping, they are \(10^{-3}\) to \(O(1)\).

∎

Only the torus orbit of the hyperparameters reaches the weights. LoRA is the case \(k=2\), \(d=(1,1)\); \(\delta\) may be any multilinear update, such as VeRA’s in its two scaling vectors or LoHa’s in its four factors.

(expand)

(a) These are the invariants of the 2-torus action. (b) Take \(c_B=1/\lambda\), \(c_A=1\) under Adam, or \(c_B=\lambda^{-1/2}\) under SGD; the single-rate form is \(c_A=\lambda^{1/4}\), \(c_B=\lambda^{-1/4}\). Since \(\sigma_B=0\), the start \(B_0=0\) is unchanged. Thm III.10 gives \(\varepsilon_B'=\varepsilon/c_B\) and \(\mathrm{wd}_B'=\mathrm{wd}/c_B\). (c) Take \(c_B=\sqrt r\).

Num The maximum trajectory difference is exactly \(0.0\) at \(\varepsilon=0\) for power-of-two \(\lambda\) and about \(10^{-14}\) otherwise, against \(0.93\) for the unrescaled control; \(5\times10^{-9}\) at a shared \(\varepsilon=10^{-8}\); \(0.6\) with global-norm clipping at \(0.5\).

∎

So LoRA+, rsLoRA and \(\alpha\)-tuning form one family. The slogan depends on the chart. Holding \(\eta_A\) fixed, LoRA+ is \(\alpha\) multiplied by \(\lambda\); holding \(\eta_B\) fixed, it is LoRA with \(\eta_A\) divided by \(\lambda\).

Prediction Any gain of LoRA+ over LoRA with \(\alpha\) re-tuned by \(\lambda\) must come from what breaks the torus: global-norm clipping (HF Trainer default max_grad_norm=1.0), \(\varepsilon\) or weight decay shared across groups (as in HF PEFT’s LoRA+ optimizer), or group-specific schedules. That optimizer treats embeddings and 1-D parameters separately, so this covers linear modules only.

What the optimizer can see

An optimizer is equivariant under a gauge element \(g\) if starting from \((Bg^{-1},gA)\) gives the transported trajectory. SGD’s Euclidean metric respects rotations of the rank channels; Adam, acting coordinatewise, respects only the signed permutations.

Published variants. Damped scaled GD (\(\delta\gt0\)) is only \(O(r)\)-equivariant, and wrapped in AdamW (Zhang and Pilanci’s scaled AdamW, HF’s create_riemannian_optimizer) it is \(B_r\)-equivariant. LoRA-Pro’s parameter update with the Sylvester-optimal \(X\) is only \(O(r)\)-equivariant, and its discrete \(W\)-step depends on the representative at \(O(\eta^2)\). LoRA-RITE’s Definition 1 asks only for \(W\)-level invariance, yet on the full-rank locus its update is \(GL_r\)-equivariant in the parameters. Followed line by line through its Algorithm 1, a gauge change alters its internal quantities (unmagnified gradients, transported moments) only by orthogonal matrices, and the final multiplication by \((R_B^\top)^+\) supplies the gauge element; the resulting step is its Eq. 15. That puts this Adam-type optimizer in class (c).

(expand)

Under \((Bg^{-1},gA)\) we have \(\nabla_{B'}=\nabla_Bg^\top\) and \(\nabla_{A'}=g^{-\top}\nabla_A\). Equivariance means \(\delta B'=\delta B\,g^{-1}\) and \(\delta A'=g\,\delta A\), for arbitrary gradients.

(a) SGD needs \(g^\top=g^{-1}\); momentum buffers are linear in the gradients.

(b) “\(\supseteq\)”: coordinatewise rules commute with signed permutations, since the moment \(m\) flips sign and \(v\) does not change. “\(\subseteq\)”: the first Adam step from zero state is \(\delta B=-\eta\,\phi(\nabla_B)\) with \(\phi(x)=x/(\lvert x\rvert+\varepsilon)\) coordinatewise. Write \(h=g^{-1}\).

• Case \(\varepsilon=0\). Equivariance reads \(\mathrm{sign}(xg^\top)=\mathrm{sign}(x)h\) for every row vector \(x\). Perturbing \(x\) realises every sign pattern \(s\in\{\pm1\}^r\) with \(xg^\top\) zero-free, so \(sh\in\{\pm1\}^r\) for all \(s\). Flipping \(s_i\) changes \((sh)_j\) by \(2s_ih_{ij}\in\{0,\pm2\}\), so \(h_{ij}\in\{0,\pm1\}\). A column with \(t\ne1\) nonzero entries yields a sum of absolute value \(0\) or \(\ge2\) for suitable \(s\), so each column of \(h\) has exactly one entry \(\pm1\). Hence \(h\), and therefore \(g\), lies in \(B_r\).
• Case \(\varepsilon\gt0\). For small \(x\), \(\phi(x)\approx x/\varepsilon\), which forces \(g\in O(r)\). For large \(x\), \(\phi\to\mathrm{sign}\), which forces \(g\in B_r\).

For the joint torus, take \(g=\mathrm{diag}(d)\). Gradient column \(j\) of \(B\) is scaled by \(d_j\), while equivariance requires the step to be scaled by \(1/d_j\). At \(\varepsilon=0\), Adam’s step is invariant to positive per-coordinate scaling of the gradient, so the rate of column \(j\) must be multiplied by \(1/d_j\), and the rate of row \(j\) of \(A\) by \(d_j\). With \(\varepsilon\gt0\), \[d_jm/(d_j\sqrt v+\varepsilon')=m/(\sqrt v+\varepsilon'/d_j),\] which requires \(\varepsilon'=d_j\varepsilon\); the decoupled decay term \(\eta'\lambda'B'_{:,j}=(\eta/d_j)\lambda'B_{:,j}/d_j\) requires \(\lambda'=d_j\lambda\). Rows of \(A\) are symmetric. Global-norm clipping compares the total gradient norm with a threshold, and a non-orthogonal \(g\) changes that norm.

(c) This is Prop III.4(a); heavy-ball buffers transform like the gradients they accumulate. Damping fails for non-orthogonal \(g\) because \[g^{-1}(gAA^\top g^\top+\delta I)g^{-\top}=AA^\top+\delta g^{-1}g^{-\top}.\]

Num Defects are \(10^{-15}\) for signed permutations, \(11.3\) for a positive diagonal, \(6.6\) for a rotation, and \(5\times10^{-16}\) for the diagonal with jointly rescaled per-channel rates at \(\varepsilon=0\). With \(\varepsilon=10^{-8}\) left unrescaled the joint torus fails by about \(2\%\). LoRA-Pro with the Sylvester \(X\) has a parameter-level defect of \(1.2\) under a non-orthogonal \(g\), against \(2\times10^{-15}\) with \(X=0\) or with an orthogonal \(g\). LoRA-RITE’s Algorithm 1 from \((B,A)\) and from \((Bg^{-1},gA)\) with a generic \(g\) gives parameters that agree to \(10^{-13}\) after 30 steps.

∎

The hierarchy \(B_r\subset O(r)\subset GL_r\) is strict for \(r\ge2\). A \(GL_r\)-equivariant rule, or any rule mapping gauge orbits to gauge orbits, defines an optimizer on the model itself; \(O(r)\)- and \(B_r\)-equivariant rules in general do not. The default, Adam, sees a finite shadow of the gauge.

What a zero-B start chooses

Default LoRA starts at \(B_0=0\) with a random full-rank \(A_0\). The image remembers only \(\mathrm{row}\,A_0\) (Thm II.2). Training remembers what the optimizer’s gauge group cannot move.

(expand)

Only the pointwise “\(\supseteq\)” half of Thm III.12 is used. For SGD and Adam it holds off the full-rank locus, and the initial Adam state \((m,v)=0\) is fixed by \(B_r\). With \(B_0=0\) the term \(B_0g^{-1}\) stays \(0\), so equivariant updates give identical \(W\)-trajectories. Orthogonal \(g\), hence \(O(r)\) and \(B_r\), preserve gradient norms and so commute with global-norm clipping; a general \(g\in GL_r\) does not. Left \(O(r)\)-orbits of full-rank \(A_0\) are classified by \(A_0^\top A_0\) (Thm II.5(2)), and \(GL_r\)-orbits by the row space (proof of Thm II.2). For the example rule, \(\nabla_{A'}=g^{-\top}\nabla_A\) and \(A'A'^\top=gAA^\top g^\top\) give \(\delta A'=g\,\delta A\), and \(\delta B'=\delta B\,g^{-1}\) as in Prop III.4(a).

The converse compares first steps. Under SGD the first step from \((0,A_0)\) is exactly \(\delta W_1=-s^2\eta_BGA_0^\top A_0\), so equal trajectories for every linear loss force \(A_0^\top A_0=A_0'^\top A_0'\). Under Adam at \(\varepsilon=0\) it is \(\delta W_1=-s\eta_B\,\mathrm{sign}(GA_0^\top)A_0\), and row by row \(F(x)=\mathrm{sign}(xA_0^\top)A_0\) is constant off the hyperplanes \(a_j^\perp\) and jumps by \(\pm2a_j\) across each. So \(F\) determines the hyperplanes, hence each row up to a factor, and the size of its jump fixes that factor up to sign: \(A_0'\) is a signed row permutation of \(A_0\).

Num SGD from \(A_0\) and \(QA_0\) agree to \(10^{-15}\), and from \(A_0\) and \(DA_0\) differ by \(0.81\). Adam from \(A_0\) and \(PA_0\) agree to \(10^{-16}\), and from \(A_0\) and \(QA_0\) differ by \(0.98\). The example rule from \(A_0\) and \(gA_0\) agrees to \(3\times10^{-15}\). LoRA-RITE’s Algorithm 1, run as written with its pseudo-inverses from \(B_0=0\), gives the same weights from \(A_0\) and from \(gA_0\) for a generic \(g\) to \(10^{-13}\), and weights that differ by \(O(1)\) for a different row space.

∎

The strata of Thm II.2 are the expressive quotient; each optimizer sees a finer one. Undamped scaled GD is undefined at \(B=0\), so scaled-GD optimizers enter in damped (\(O(r)\)) or AdamW-wrapped (\(B_r\)) form. LoRA-RITE is defined there through pseudo-inverses, but its \(GL_r\)-equivariance is proved only on the full-rank locus.

Two problems stay open. One is the apex fibre modulo \(B_r\) for Adam, stratified by \(\Phi_0\), degenerate pointings included. The other is an Adam analogue of Thm III.9, a conserved or monotone quantity for sign-like updates.

Exposé IV reads the same cometric from the backward side of the lens, where methods that compress gradients have cometrics of their own.

Two sides of one port

A layer does two jobs in training. Forward, it turns parameters and inputs into outputs; backward, it turns a cotangent at the output into cotangents at the input and the parameters. Backpropagation keeps the pair together, and a category theorist calls it a lens.

Every method in Exposés I to III changes the forward job, replacing the parameter port \(\Theta\) by a smaller space \(Q\) and a map \(\rho:Q\to\Theta\). A second family edits only the backward job. GaLore compresses the weight gradient \(G\) to \(P^\top G\), with \(P\) from an SVD of \(G\), runs Adam on that small matrix and maps the result back with \(P\). Every weight trains; only the optimizer state is small. LISA and MeZO follow the same pattern.

When \(\rho\) is linear the two families meet. The pullback of the forward method is then the compression of a backward method, and an optimizer that reads only its gradients cannot tell the two apart. Figure 10 sorts the backward family and draws what backpropagation must store.

Backpropagation as a lens

In the language of Def I.1, a layer is a parametrised map \(f:P\times X\to Y\). Its reverse derivative \(\mathsf R[f](x,y)=J_f(x)^\top y\) takes an output cotangent to an input cotangent, parameters included, and backpropagation is the 2-functor

\[\mathbf{Para}(\mathsf R):\mathbf{Para}(\mathcal C)\longrightarrow\mathbf{Para}(\mathbf{Lens}(\mathcal C)),\qquad f\longmapsto(f,\mathsf R[f]).\]

A reparametrisation \(\rho\) is a 2-cell, so the functor turns it into a lens \((\rho,\mathsf R[\rho])\). Its tangent side is the image, what the method can express (Exposé II). Its cotangent side \(\mathsf R[\rho](q,\gamma)=D\rho_q^\top\gamma\) pulls the weight gradient back to \(Q\) and, with the rates \(\Lambda\), induces the cometric \(K=D\rho\,\Lambda\,D\rho^\top\) (Def III.1). A lens need not come from a map, though, and a backward half written down directly gives the second family.

A backward method has its own cometric \(K_t=a_tc_t\) (Def III.1), for instance \(K(G)=PP^\top G\) for GaLore under SGD. The update must lie in the image of the anchor, so Flora as published, APOLLO and Fira, whose updates leave it, are not backward methods here.

More: why Flora falls outside

Flora compresses its momentum or gradient accumulator with a random projection and hands the decompressed tensor to Adafactor, which acts in the full space, so its update need not lie in the image of the anchor. On resampling it transports its momentum to the new frame. It fits Def IV.1 only in an idealised form with an SGD or heavy-ball base. An earlier version of the table above listed Flora where GaLore with a random projector now stands.

The degree-one collapse

If \(\rho(C)=W_k+\mathcal LC\), training \(C\) and the backward method with \(c=\mathcal L^*\), \(a=\mathcal L\) feed the optimizer the same gradients \(\mathcal L^*G\) and so make the same weight updates. A curved \(\rho\), such as LoRA’s \(BA\), separates them after one step, and so does an optimizer that reads the parameter tensor, \(C\) in one procedure and \(W\) in the other.

(expand)

By induction, \(W_t=W_k+\mathcal LC_t\) along the first procedure. Both procedures therefore feed opt the same sequence \(\mathcal L^*\nabla L(W_t)\) from the same state, and they receive the same outputs \(U_t\). One sets \(C\leftarrow C-\eta U_t\), which moves the weight by \(-\eta\mathcal LU_t\); the other sets \(W\leftarrow W-\eta\mathcal LU_t\) directly. Injectivity of \(\mathcal L\) is never used.

The excluded ingredients see different tensors in the two procedures. Decoupled decay acts on \(C\) in one, pulling \(W\) toward \(W_k\), and on \(W\) in the other, pulling it toward \(0\); coupled decay adds \(\lambda C\) to the gradient in one and \(\lambda W\) in the other. Parameter-scaled steps read \(\mathrm{RMS}(C)\) in one and \(\mathrm{RMS}(W)\) in the other. Full-gradient clipping uses \(\|G\|\) in one and \(\|\mathcal L^*G\|\) in the other.

For GaLore, apply this on each period with \(\mathcal LC=P_kC\) (or \(\mathcal LC=CP_k^\top\) when projecting on the right). GaLore’s update is \(W\leftarrow W-\eta\,\alpha\,P_k\,\mathrm{opt}(P_k^\top G)\), so \(\alpha\) multiplies the learning rate. Since the trajectories agree up to the start of a period, so do the gradients there, and both procedures read off the same \(P_k\). The state handed from one period to the next is the same object in both procedures, so the induction runs across periods. That state holds coordinates relative to \(P_{k-1}\)’s columns, so replacing \(P_k\) by another basis of its span changes what the reused moments mean.

Num Against the official galore_torch optimizers (least squares, \(W\in\mathbb R^{6\times10}\), \(r=2\), \(T=7\), four periods, total movement \(0.046\)), the two procedures agree to \(8\times10^{-17}\) with the state carried over. Resetting the LoRA side’s state separates them by \(0.029\), weight decay \(0.1\) by \(0.020\), HF-default Adafactor by \(0.030\), and flipping the sign of one column of \(P_k\) by \(0.12\). Figure 10 reruns a check of this kind live.

∎

Beyond one-sided LoRA, GaLore adds a frame read off the gradient’s SVD and the carry-over of optimizer state. The carry-over reads moments written in the coordinates of \(P_{k-1}\) as coordinates for \(P_k\), with no change of basis. LDAdam and Flora transport the state instead.

More: scale, implementation, review

Moving GaLore’s \(\alpha\) inside \(\rho\) as a LoRA scale multiplies the compressed gradient by \(\alpha\). Under Adam this gives GaLore’s trajectory only at \(\varepsilon=0\), or with \(\varepsilon\) multiplied by \(\alpha\) as well. This is the exchange of scale for learning rate and \(\varepsilon\) on the hyperparameter torus of Thm III.10.

Both sides must run the same implementation. galore_torch’s AdamW adds \(\varepsilon\) to \(\sqrt v\) before the bias correction, as in transformers, and swapping in torch.optim.Adam on the LoRA side moves the trajectory by about \(4\times10^{-5}\) in the galore_torch check quoted in the proof.

The statement was repaired after review. The optimizer class is restricted to optimizers that do not read the parameter tensor, which excludes HF’s default parameter-scaled Adafactor. With fresh state the GaLore statement failed beyond one period, because GaLore carries its Adam state across projector changes, so the state is now carried on both sides. Decoupled decay and full-gradient clipping are listed as exclusions, injectivity was dropped, and the carry-over is no longer called a transport.

The cotangent family sorts by where the projector comes from and how often it changes.

Prop IV.2 covers the first three rows under its optimizer hypotheses, except EVA and LoRA-GA, which initialise two-sided LoRA and sit in the frozen column only for their first step. LoRA’s map \((B,A)\mapsto W_0+sBA\) has degree two, so its cometric moves with the state (Obs III.2) and the collapse does not apply. The other frozen entries gain nothing from merge-and-restart (Thm V.3, corollary 2); their re-based neighbours move the frame and can reach all of weight space.

When a backward method is a forward one

A backward method chooses at each weight a space of directions \(D_\theta=\mathrm{im}\,a(\theta)\), and it reads as a forward method only if these fit into \(k\)-dimensional surfaces, the images of immersive parametrisations. Frobenius’ theorem tests this. The bracket \([X,Y]\), the gap by which a small loop along \(X\), \(Y\), \(-X\), \(-Y\) fails to close, must stay in \(D\). Discrete steps and the optimizer’s metric ask for more.

(expand)

The first equivalence is Frobenius’ theorem. The tangent spaces of the image of an injective immersion are involutive along that image, which gives the remark about a single \(\rho\), and the bracket in the example is a direct computation. For the discrete statement, \(\theta-\eta\,a(\theta)s\) stays on the leaf for all \(\eta\) and \(s\) iff the leaf contains \((\theta+D_\theta)\cap U\).

For the frame condition, gradient flow of the forward method moves the weight by \(-D\rho\,\Lambda\,D\rho^\top\nabla_\theta L\) (Def III.1), and agreement for every loss forces \(a_tc_t=D\rho\,\Lambda\,D\rho^\top\). This operator is symmetric PSD with range in \(\mathrm{im}\,a\), which gives \(c=Ma^\top\) when \(a\) is injective. When \(\rho\) is an immersion with \(\dim Q=k\), it is a local diffeomorphism onto the leaf, and \(D\rho\) carries the flat metric \(\Lambda^{-1}\) on \(Q\) to the metric \(K^{-1}\) on the leaf, which is therefore flat. A forward method whose \(\rho\) has gauge fibres induces the leaf metric by a submersion instead, and that metric can be curved; balanced full-rank LoRA and \(B\mapsto BB^\top\) are examples. For discrete steps, the forward step lands at \(\rho(q-\eta v)\) with \(v=\Lambda D\rho_q^\top g\) and the backward step at \(\rho(q)-\eta\,D\rho_q v\); equality for all \(\eta\) is affineness along that line.

For the GaLore field, take \(L=\tfrac12\|W\|_F^2\), so \(\nabla L=W\), at \(W=\mathrm{diag}(\sigma_1,\dots,\sigma_{\min(m,n)})\) with \(\sigma_1\gt\dots\gt\sigma_{\min(m,n)}\gt0\), where \(U_r\) is spanned by the first \(r\) basis vectors. The fields \(X_E(W)=\Pi(W)E\), with \(\Pi\) the projector onto \(U_r\), span \(D_W\). First-order perturbation of \(\Pi\) gives, for \(E=e_1e_{r+1}^\top\) and \(F=e_1e_1^\top\), a bracket \([X_E,X_F]\) whose row \(r+1\) is \(\sigma_{r+1}/(\sigma_1^2-\sigma_{r+1}^2)\,e_1^\top\ne0\), so it leaves \(D\). At \(m=n=2\), \(r=1\) and \(W=\mathrm{diag}(2,1)\) this reads \([X_1,X_2]=-\tfrac13e_2e_1^\top\) for \(X_i=u(W)e_i^\top\). With a single sample, \(\nabla L=(Wx-y)x^\top\), and moving \(W\) along \(D\) keeps \(Wx-y\) in \(U_1\), so \(U_1\) is constant along \(D\) and the leaves are affine.

Num For least squares and cross-entropy with several samples, the relative normal component of the bracket is \(0.08\) to \(0.95\) at \((m,n,r)=(4,3,1)\), \((5,4,1)\) and \((4,3,2)\); Figure 10 recomputes it live at \(W\in\mathbb R^{3\times4}\). The diagonal formula above was checked by central differences for several \((m,n,r)\).

∎

More: what review changed

Repaired after two reviews. The literal “only if” was false, since one integral manifold implies involutivity only along itself; the example in the statement shows it. The dynamic hypotheses are now separate. Flatness of the leaf metric is required only of immersive \(\rho\) with \(\dim Q=k\), since gauge-fibred methods escape it: balanced full-rank LoRA induces a curved leaf metric. An earlier claim that full-batch rank-one GaLore is integrable is withdrawn; the diagonal example in the proof refutes it at every rank.

More: in what sense “dual”

Analogy (the lens duality). “Tangent side” and “cotangent side” are exact words here. A forward method acts on tangent vectors through \(D\rho\), and a backward method acts on cotangent vectors through \(c\). Calling GaLore and one-sided LoRA dual is exact only at degree one, and only for optimizers that do not read the parameter tensor. Beyond that range the word organises the zoo and proves nothing.

What the backward pass must remember

Fine-tuning memory is weights, optimizer state, and activations stored for the backward pass. Shrinking \(Q\) shrinks the second. As for the third, a linear box’s reverse derivative \(W^\top y\) ignores its input, and cotangents are needed only downstream of something trainable, so only a cone through the network costs activation memory.

(sketch; expand)

Cotangents are needed only on the downstream cone of the trainable boxes, so everything else can be pruned. A frozen linear box has \(\mathsf R[f](x,y)=W^\top y\), which does not depend on \(x\), whereas a nonlinear box’s reverse derivative depends on its input through some residual. A trainable linear box \(y=Wx\) has parameter-VJP \(\bar y\,x^\top\), and a multiplicative box \(y=l\odot x\) has \(\bar y\odot x\) summed over tokens, so both read \(x\). A bias’s parameter cotangent is its incoming cotangent. A dropout box’s VJP applies the same rescaled mask to \(\bar y\).

The \(i\)-th coordinate of \(P^\top\nabla L\) is the directional derivative \(DL(\theta)[p_i]\), which forward mode computes alongside a forward pass, keeping only the current layer’s values and tangents. The memory and time costs of forward mode are the standard ones of automatic differentiation and are not re-derived here.

∎

The bound reads off the activation memory of the zoo.

• LST reads the backbone through trainable down-projection ladders, so the backbone’s backward pass never runs.
• LoRA-FA stores \(A_0x\) (\(r\) numbers per token) instead of \(x\) (\(n\) numbers), plus the dropout mask unless lora_dropout=0, HF’s default.
• Top-layer adaptation truncates the cone, as long as no checkpointing sends cotangents further down (see the caveat).
• LISA trains the embedding and LM head at every step, so its cone is the whole network. Its unsampled layers’ frozen linear boxes drop their input residuals, so it keeps fewer activations than full fine-tuning or LoRA, though most of its saving is in gradients and optimizer state.
• Prompt tuning has a \(Q\) as small as VeRA’s, FourierFT’s or (IA)³’s but the maximal cone, the whole network, and its prompt tokens enlarge every stored activation.
• MeZO is the case \(k=1\), with a central difference in place of the JVP and no reverse pass.

More: fine print on the readings

LoRA-FA’s saving is real only where no neighbouring nonlinear box already keeps \(x\). MeZO’s estimate is unbiased for the gradient of the Gaussian-smoothed loss, and for a loss with Lipschitz Hessian its bias against \(\nabla L\) is \(O(\varepsilon^2)\). Prompt tuning shares the maximal cone with every method that trains something in the first layer. A LISA variant that froze the embedding would truncate the cone at its lowest sampled layer. GaLore, its random-projector variant and LISA all take the reverse route. Exact unbiasedness, \(\mathbb E[(\nabla L\cdot z)z]=\nabla L\), holds for the forward gradient computed with an exact JVP rather than MeZO’s finite difference.

Repaired after review. The observation is now an upper bound with an explicit notion of residual. Additive trainable boxes need nothing, dropout needs its mask, the JVP route is competitive in time only for very small \(k\), and MeZO’s central difference is biased at order \(\varepsilon^2\). LISA, which trains its embedding, keeps the whole cone while still dropping the residuals of its frozen linear boxes.

So every cotangent method meeting the hypotheses of Prop IV.2 is, period by period, a linear forward method on a moving base. What it reaches, and what a quantiser does to the base, are the subject of Exposé V.

The point moves

Every method so far has been an arrow into one fixed point \((\Theta,\theta_0)\). In practice the point moves. Quantisation moves it once, ReLoRA at every merge, and task arithmetic, LoraHub, TIES and DARE build a new update from several adapters. A PEFT method is a formula that makes sense over any base point, so it is a family \(\theta_0\mapsto M(\theta_0)\), and this exposé relates its members. Four kinds of move occur.

Only part (a) is base change in the algebraic geometer’s sense, and it costs nothing; whether \(\phi_!\) turns LoRA into LoRA is the covariance question of Observation II.15. Parts (b) and (c) borrow the word, and the work lies there and in the combination rules. Figure 11 lets you run the restart computations of Theorem V.3 by hand.

More: what “base change” means here

In algebraic geometry a family over a base \(S\) pulls back along any map \(S'\to S\), and the statements worth having are the ones that survive the change. Part (a) of Definition V.1 is the standard companion of pullback on slice categories, composition with a map of bases. A quantiser changes the network’s function, so there is no commuting square to transport along, and we compute the defect it leaves instead. A rebasing scheme is an iteration, and we compute its reachable set. “Gauge” keeps the sense of Exposé III, the symmetry group of a parametrisation; no connection is involved.

Quantisation and splitting

For an additive method (Def I.2) whose starting point does not depend on the base, such as zero-init LoRA, quantising the base only translates the image, so QLoRA is LoRA with its vertex at \(\kappa\theta_0\). A quantisation error is spread over all singular directions, so \(e\) is generically of full rank and the image misses \(\theta_0\). Methods whose starting point is computed from the base, such as PiSSA, CorDA, OLoRA, LoftQ and DoRA, meet two operations that do not commute, and the order fixes the defect (tabulated in Exposé II.A).

(expand)

(a) is the definition of additivity. With \(\rho=\theta_0+\delta\) and \(\delta\) independent of \(\theta_0\), moving the base by \(v\) translates every value of \(\rho\) by \(v\). For the last clause, \(\kappa\theta_0+\delta(q)=\theta_0\) needs \(\delta(q)=-e\), and LoRA’s \(\delta(q)\) has rank at most \(r\).

(b) Compute \(\rho(q_0)-W\) in each pipeline. Splitting first gives \(\kappa(W_{\rm res})+\delta_0-W\) with \(W_{\rm res}=W-\delta_0\) (for LoftQ, \(W_{\rm res}=W-\delta(q_{t-1})\)). Fitting the error gives \(\kappa W-s\,\mathrm{SVD}_r(e)-W=(e-\mathrm{SVD}_r(e))+(1-s)\,\mathrm{SVD}_r(e)\), and the two terms are orthogonal in the Frobenius inner product. Re-quantising gives \(\kappa(r')+\delta_0-W=(\kappa(r')-r')+(\kappa W-W)\). For DoRA at \(B=0\), \(\rho(q_0)=\mathrm{diag}\big(\mu_0/\|(\kappa\theta_0)_i\|\big)\,\kappa\theta_0\), where \(\mu_0\) is the vector of row norms of whichever base initialised the magnitude.

(c) holds by construction, with deployed weight \(Q_T+s\,L_TR_T\); the failure of monotonicity is observed numerically. (d) In general \(\kappa(\theta_0+\Delta)\ne\kappa\theta_0+\Delta\). On a quantisation group with codes \(c\), scale \(a\) and zero-point \(\beta\), a group-constant \(\Delta\) gives \(ac+\beta+\Delta=ac+(\beta+\Delta)\), so it is absorbed into the zero-point.

Num On one \(256\times256\) NF4 example with \(r=8\), \(s=1\) and \(\|e\|=5.65\), QPiSSA’s defect is \(1.45\), one-step LoftQ’s is \(5.22\), and the LoftQ objective rises in \(27\) of \(60\) half-steps. QPiSSA’s large advantage there relies on a weight with a dominant low-rank part; on a Gaussian weight all three defects are close to \(\|e\|\).

∎

More: LoftQ as deployed

The official LoftQ checkpoints were made with \(r=64\), \(\alpha=16\) and five iterations, so \(s=1/4\). In a referee’s simulation of that configuration on a Gaussian \(512\times512\) NF4 weight, the attained residual was \(0.67\|e\|\) and the deployed defect \(0.96\|e\|\). For \(T\ge2\), HF’s internal residual also omits \(s\), so the objective in (c) is not the defect HF starts from. With a callback, replace_lora_weights_loftq can roll a module back to the zero initialisation, and that module’s defect is then \(e\).

Over a quantised base, exactly orthogonal OFT keeps the neuron geometry of \(\kappa\theta_0\) (Cor II.6) and HiRA keeps the zeros NF4 created (Prop II.9(e)). QA-LoRA makes merging commute with quantisation by living inside the quantiser’s own degrees of freedom.

Restarting: what merge-and-restart can reach

Merge the adapter, restart at the merged point, repeat. Each cycle adds one more copy of an additive shape or applies one more group element, so the long-run reach is whatever one cycle generates. A rank-\(r\) cone generates every matrix, and a subspace, subgroup or submonoid generates only itself. Block-orthogonal factors generate the rotations of each connected piece of their block hypergraph, because the bracket of infinitesimal rotations in the planes \((a,c)\) and \((c,b)\) is the infinitesimal rotation in the plane \((a,b)\).

(expand)

(a) follows by induction on \(K\). If \(C=\mathbb RC\), the monoid generated by \(C\) contains \(-C\) and is closed under scaling, so it is the span.

(b) Each cycle applies one element of \(S\), so \(R_K=S^K\theta_0\), and since \(1\in S\) the sets increase with \(K\). Any neighbourhood of \(1\) contains a symmetric one, and a symmetric neighbourhood of \(1\) in a connected group generates that group as a monoid. The subgroup generated by connected Lie subgroups is the connected Lie subgroup with the generated Lie algebra (Yamabe).

(c) Write \(L_{ab}=E_{ab}-E_{ba}\). Then \([L_{ac},L_{cb}]=L_{ab}\) for distinct \(a,b,c\). Overlapping hyperedges therefore generate \(\mathfrak{so}\) of their union, and \(\bigoplus_C\mathfrak{so}(C)\) is a subalgebra that contains every generator.

The corollaries.

• (1) Rank is subadditive, and an SVD splits into blocks of rank at most \(r\).
• (2) The exact Cayley image is dense and open in \(SO(b)\), and every element of \(SO(b)\) is a product of two elements of it.
• (3) The entries of \(\Lambda_bB\Lambda_d\) are \(b_iB_{ij}d_j\). Their span over \((b,d)\) is all of \(\mathbb R^{m\times r}\) when every \(B_{ij}\ne0\) (take \(b=e_i\), \(d=e_j\)).
• (4) Apply (c) to the butterfly blocks. For the Cayley–Neumann factors, \((I-Q)(I+Q+Q^2+Q^3)=I-Q^4\) gives \(R=C(Q)(I-Q^4)\), where \(C(Q)\) is orthogonal and commutes with the symmetric matrix \(Q^4\). Hence the singular values of \(R\) are \(\lvert1-\mu_k^4\rvert\), at most \(1\) when every \(\mu_k\le2^{1/4}\), and \(K_{\rm out}(WR^\top)=W(I-Q^4)^2W^\top\preceq WW^\top\). Also \(\det R=\prod_k(1-\mu_k^4)^2\ge0\), and \(R\) is singular when some \(\mu_k=1\). The invertible factors generate \(\prod GL^+(b)\) as a group (\(\prod CO^+(2)\) when \(b=2\)): \(R(Q)R(-Q)=(I-Q^4)^2\) gives every positive scaling that is constant on the rotation planes of some skew \(Q\), \(R(Q)\lvert I-Q^4\rvert^{-1}=C(Q)\) for small \(Q\) gives \(SO(b)\), and \(GL^+(b)=SO(b)\cdot D^+\cdot SO(b)\); for \(b=2\) the scalings are scalar. This identifies the group only. Re-merging reaches the monoid, which can be smaller, as the contraction bound shows.
• (5) “\(\subseteq\)” holds because \(R_1R_2-I=R_1(R_2-I)+(R_1-I)\), so ranks add. For “\(\supseteq\)”, an \(R\in SO(n)\) with \(k=\operatorname{rank}(R-I)\le Kr\) is a product of exactly \(k\) reflections, with \(k\) even (Cartan–Dieudonné in Scherk’s form). Split them into \(K\) consecutive groups of even size at most \(r\), pad each group with pairs \(H_uH_u=I\), and observe that each group is an \(\mathrm{HRA}_r\) element. Injectivity of \(W_0\) gives \(\operatorname{rank}W_0(R-I)=\operatorname{rank}(R-I)\).
• (6) A fixed linear shape \(L\) is its own additive monoid, by corollary 2.
• The remaining corollaries are immediate.

Num For the butterfly hypergraph at \(n=64\), \(b=4\), the components have sizes \(4,8,16,32,64\) for \(m'=1,\dots,5\), so it is connected only at \(m'=5\); the referees found the same with HF’s own permutation code. With skew entries of size about \(0.05\), \(K_{\rm out}\) moves by \(0.2\%\), \(1.3\%\), \(6\%\) and \(22\%\) after \(1\), \(10\), \(50\) and \(200\) default-OFT merges, and every one of these moves is a Loewner-order shrinkage.

∎

ReLoRA is Observation I.7 unrolled in time, since a sum of \(K\) rank-\(r\) LoRAs is one LoRA of rank \(Kr\). A fixed additive shape gains from restarts only when it is not closed under addition, as LoRA’s cone and VeRA’s shape are not.

Prediction Re-merged exact-Cayley block OFT and COFT on a fixed partition never leave \(W_0\cdot\prod SO(b)\); going further needs partitions that change between cycles or full-depth butterflies. These are statements about reach, and whether extra reach improves a trained model is the experiment.

Combining: rules that must ignore the gauge

A LoRA module ships as factors \((B,A)\), the model sees only \(BA\), and \((Bg^{-1},gA)\) gives the same product for every \(g\in GL_r\). A combination rule should return the same model for every choice of factors, or two people with the same adapters can merge them into different models. Task arithmetic adds the products and passes. LoraHub adds the \(B\)’s, adds the \(A\)’s and multiplies, and fails, because its cross terms \(B_iA_l\) see a rescaling \((cB,A/c)\) of one module. Tab B of Figure 9 shows the LoraHub merge moving under that rescaling while task arithmetic stays fixed.

(expand)

(a) On the full-rank locus, the fibres of \((B_i,A_i)\mapsto(B_iA_i)\) are \(\prod GL_r\)-orbits (Thm II.2). Off it, invariance alone is not enough. The projector onto \(\mathrm{row}\,A\) is \(GL_r\)-invariant on \(\{\operatorname{rank}A=r\}\), yet it differs at \((0,A)\) and \((0,A')\), which have the same product. For a continuous \(\Phi\), invariance makes \(\Phi\) constant on orbit closures, and the unique closed orbit of a fibre lies in the closure of every orbit of that fibre. The LoraHub formula follows by expanding the product: the diagonal term \(w_2^2B_2A_2\) is unchanged, and each cross term \(w_lw_2B_2A_l\) gains a factor \(c\) while \(w_lw_2B_lA_2\) gains \(c^{-1}\). For Poly and MHR, a common \(g\) cancels inside every product of sums. Whether it is the only continuous symmetry depends on the routing: with at least as many skills as tasks, or with the routing counted as a parameter, there are further ones, such as relabelling the skills.

(b) Flipping column \(j\) of \(B_i\) together with row \(j\) of \(A_i\) preserves the SVD form and changes \(B_iA_l\) for \(l\ne i\). A product of two weighted sums is quadratic in \(w\), unless one factor is shared and left unweighted.

(c) Masks and the rescaling by \(1/(1-p)\) commute with diagonal maps, and an i.i.d. mask is permutation-invariant in law. Trimming by magnitude, electing signs by total mass and taking disjoint means commute with signed coordinate permutations and with \(c\cdot I\) for \(c\ne0\). A rotation changes the magnitudes of individual coordinates. For the factor clause, the all-ones mask has positive probability, and on that event factor-wise DARE is the product of sums \(\big(\sum_iw_iB_i\big)\big(\sum_jw_jA_j\big)/(1-p)^2\), which a gauge \((cB_2,A_2/c)\) on one adapter changes by the formula of (a). A common diagonal gauge commutes with the masks and cancels in every product.

Num DARE with a random non-unit diagonal \(D\) is equivariant to \(2\times10^{-15}\) per mask. TIES is homogeneous to \(9\times10^{-16}\) and fails by \(5.6\) under a generic positive diagonal. The LoraHub change formula matches with three modules to \(2\times10^{-15}\). On the factors of two adapters, a diagonal gauge on one of them moves factor-wise DARE by an amount of order one, while the same gauge shared by both leaves it fixed to \(10^{-15}\).

∎

More: TIES variants and HF options

With per-tensor trimming, TIES is equivariant instead under \(\prod_b\big(\mathbb R^\times\cdot B_{N_b}\big)\), a signed permutation and a scalar per block, together with permutations of equal-size blocks. DARE followed by full TIES is \(\mathbb R^\times\cdot B_N\)-equivariant in distribution. HF’s dare_ties has no magnitude trim, so with total-mass election it is exactly \(T_N\)-equivariant per mask, like DARE. With HF’s majority_sign_method="frequency", sign counts are integers, ties are common and are broken towards \(+1\), so sign flips and negative homotheties fail for ties and dare_ties alike. HF’s linear rule weights \(A_i\) by \(\mathrm{sign}(w_i)\sqrt{\lvert w_is_i\rvert}\) and \(B_i\) by \(\sqrt{\lvert w_is_i\rvert}\), so even on adapters that share a frozen \(A\) it is not exact up to a global factor; the relative weights become \(\sqrt{\lvert w_is_i\rvert}\).

Post-hoc factor merging of independently trained modules, as in LoraHub and HF’s linear, depends on how each module happened to be factorised, which nobody controls after training. Federated LoRA fails through bilinearity instead. Combine LoRAs on products, or only after a joint re-factorisation that produces a shared factor.

Prediction LoraHub behaves like update arithmetic, up to the factor \(\sum_jw_j\), when its modules share the \(A\)-seed and \(A\) stays close to its initialisation. That regime is empirical and favoured by a large ratio \(\eta_B/\eta_A\), as in LoRA+. LoraHub degrades when the modules come from independent seeds.

What base change leaves for the network

Every move here stayed inside weight space, where merging into the base is the canonical arrow to the terminal object of Observation I.4 and always exists. Adapters, prefixes and ReFT change the network itself, and for them merging is a lifting problem whose obstructions the activation function decides. That is Exposé VI.

The weight that absorbs the edit

(IA)³ rescales the hidden units of a feed-forward block by a learned vector \(\ell\), which one would like to fold into a frozen weight after training. The weight that reads the hidden units, \(W_{\rm down}\), absorbs \(\ell\) under every activation. The weight that produces them, \(W_{\rm up}\), absorbs it under SwiGLU for every \(\ell\), under ReLU when no entry is negative, and under GELU only for entries \(0\) or \(1\).

The activation belongs to the network, which a weight-space method never sees (Remark I.3). Adapters, prefixes, prompts and ReFT add boxes the pretrained network lacked, and whether they can be removed depends on their surroundings. Figure 12 runs the rules of this exposé on one block.

Extensions, and merging as a lift

An adapter \(y\mapsto y+U\sigma(Dy)\) adds weights \((U,D)\), and at \(U=0\) the network computes the old function. A prefix adds key–value pairs to every softmax, and ReFT edits hidden states at chosen positions. Each time \(\Theta\) grows to \(\Theta\times\Theta'\) and \(f\) to a larger \(\tilde f\). A prefix always takes softmax mass, and on a head with \(W_oW_v\ne0\) no prefix leaves the output unchanged, so prefixes have no neutral setting there (Thm VI.5(f)).

A merge is an arrow \(\mu\) that makes the square of realisations commute.

\[\begin{array}{ccc} Q & \hphantom{.}\xrightarrow{\qquad\mu\qquad}\hphantom{.} & \Theta\\[2pt] \llap{\scriptstyle\rho\ }\Big\downarrow & & \Big\downarrow\rlap{\scriptstyle\ \Phi_f}\\[2pt] \Theta\times\Theta' & \hphantom{.}\xrightarrow[\qquad\Phi_{\tilde f}\qquad]{}\hphantom{.} & C^\infty(X,Y) \end{array}\]

A weight-space method merges by \(\mu=\rho\), the canonical arrow to full fine-tuning (Obs I.4(a)), so the real questions live on extensions, in the slice of \(\mathbf{Para}\) over the pretrained model (Remark I.3). Box-local mergeability can be read off formulas, and the atlas of Exposé VIII records it as a merge type:

• M1 the edit merges into its own box;
• M1→ the rules move it into a neighbouring box, under side conditions;
• Mq an exact merge exists but leaves the quantised type;
• M∞ no merge: the rules are blocked box-locally, or the edit lies outside them.

“Zero inference latency” runs two questions together, whether the original architecture can compute the trained function at all and whether a local edit finds the weight. A zero up-projection, or the zero gate of LLaMA-Adapter, makes training start at the pretrained function, and by the function-level form of Lemma II.4 only \(U\), or the gate, moves at first.

The activation fixes the privileged basis

An edit moves through a network by sliding along wires, and a linear map slides through a box when it commutes with the box. Activations are the bottleneck. Permutations commute with every elementwise \(\sigma\), and for ReLU so do positive rescalings, since \(\mathrm{ReLU}(ct)=c\,\mathrm{ReLU}(t)\) for \(c\gt0\). The lemma says how little else does.

(expand)

(a) Fix row \(i\) of \(L\); its row vector \(\ell_i\) is nonzero because \(L\) is invertible. Coordinate \(i\) of \(L\,\sigma^{\times d}(x)\) is \(\sum_jL_{ij}\sigma(x_j)\), whose mixed partials \(\partial_j\partial_k\) vanish for \(j\ne k\). Coordinate \(i\) of \(\sigma^{\times d}(Lx)\) is \(\sigma(\ell_i\cdot x)\), with mixed partial \(L_{ij}L_{ik}\sigma''(\ell_i\cdot x)\). Since \(\sigma\) is not affine there is \(t_0\) with \(\sigma''(t_0)\ne0\); choose \(x\) with \(\ell_i\cdot x=t_0\). Then \(L_{ij}L_{ik}=0\) for \(j\ne k\), so each row has a single nonzero entry.

(b) ReLU is not \(C^2\), so argue by kinks instead. The left side of row \(i\) is non-differentiable exactly on \(\bigcup_{j:L_{ij}\ne0}\{x_j=0\}\), and the right side exactly on the hyperplane \(\{\ell_i\cdot x=0\}\). The two sets are equal, and a hyperplane equal to a finite union of coordinate hyperplanes is one of them. So row \(i\) has a single entry \(L_{ij}\), and \(L_{ij}\sigma(t)=\sigma(L_{ij}t)\) at \(t=1\) forces \(L_{ij}\gt0\). Leaky ReLU is identical.

(c) \(P\) commutes with \(\sigma^{\times d}\), so \(D\sigma=\sigma D\), that is \(c\,\sigma(t)=\sigma(ct)\) for each diagonal entry \(c\). Compare Taylor coefficients at \(0\). For GELU and SiLU \(a_2\ne0\), so \(c=c^2\) and \(c=1\). For tanh \(a_3\ne0\), so \(c=c^3\) and \(c=\pm1\). ∎

So rules R2 and R3 below exhaust the invertible linear maps that slide through \(\sigma\), permutations for GELU and SiLU and positive monomials for ReLU. In gated blocks such as SwiGLU every diagonal slides through the linear branch (rule R4), so their hidden-unit group is the full monomial group \(\mathrm{Mon}_{d_{\rm ff}}\).

More: symmetry groups by architecture

Ungated blocks (BERT, GPT-2, ViT) keep \(S_d\) under GELU and SiLU, \(B_d\) under tanh and the positive monomials under ReLU; gated blocks (LLaMA, Mistral, Qwen, Gemma) keep \(\mathrm{Mon}_{d_{\rm ff}}\). LayerNorm’s privileged basis comes from its diagonal gain; a gain-free RMSNorm is \(O(d)\)-equivariant, which SliceGPT and QuaRot (Ashkboos et al., 2024) use to rotate the residual stream (Obs II.15). Prediction X.3 turns the lemma into a permutation test.

Seven local rules

Draw a transformer block as a string diagram, with a copy node where a wire feeds several boxes and a product node in a gated block. An edit is a small box on a wire. To merge, slide it by equalities of maps until a trainable box absorbs it; where it gets stuck names the failing side condition. A positional rotation, a shared weight, a missing bias slot or a second reader of a wire can each block a slide, and all four occur in production models.

(expand)

R1 is closure of linear (affine) maps under composition and sums. R2, R4, R6 and R7 are identities; R7 is the naturality of the cartesian diagonal. For R3, \(\ell\sigma(t)=\sigma(\ell t)\) for all \(\ell\ge0\) and all \(t\) is positive homogeneity of degree one; such a \(\sigma\) is linear on each half-line, and linear if it is \(C^1\). For R5, \(\langle R_mMq,R_nk\rangle=\langle q,M^\top R_{n-m}k\rangle\) for every \(M\), since \(R_m^\top R_n=R_{n-m}\), while \(\langle R_mq,R_nM^\top k\rangle=\langle q,R_{n-m}M^\top k\rangle\). The two agree for all \(q,k\) iff \(M^\top\) commutes with \(R_{n-m}\), and a diagonal matrix commutes with a planar rotation through an angle that is not a multiple of \(\pi\) iff its two entries are equal.

(a) The closure and reader conditions put \(W_cE_q\) in each reader’s class, R7 distributes \(E_q\) to every reader, and position-uniformity lets one weight absorb the edit at every position. The dual case is R1 applied to \(E_q\circ W\). (b) is (a) applied at the box reached. For the converse under RoPE, suppose a query scaling \(D=\mathrm{diag}(\ell)\) merges into some \(W_K'\). Equal logits give \(W_Q^\top R_\Delta W_K'=W_Q^\top DR_\Delta W_K\) for every offset \(\Delta\) that occurs. If \(W_Q\) and \(W_K\) have full row rank, then \(W_K'=R_{-\Delta}DR_\Delta W_K\) for every such \(\Delta\). Causal attention includes \(\Delta=0\), so \(D\) commutes with every occurring \(R_\Delta\), and \(\ell\) is constant on each rotary pair whose angle at some occurring \(\Delta\) is not a multiple of \(\pi\). (c) is the definition of box-local mergeability. (d) is Prop V.2(d).

For the (IA)³ corollary under GELU, suppose \(\ell_i\,\mathrm{GELU}(w_i\cdot x)=\mathrm{GELU}(w_i'\cdot x)\) for all \(x\), with \(w_i\ne0\). On the hyperplane \(w_i\cdot x=0\) the right side vanishes, and GELU vanishes only at \(0\), so \(w_i'=cw_i\). Then \(\ell_i\,\mathrm{GELU}(t)=\mathrm{GELU}(ct)\) for all \(t\), and the Taylor coefficients at \(0\), \(\mathrm{GELU}(t)=t/2+t^2/\sqrt{2\pi}+\dots\), give \(c=\ell_i\) and \(\ell_i=\ell_i^2\). Under ReLU an entry \(\ell_i\lt0\) is impossible, since the left side is then \(\le0\) and somewhere negative, while the right side is \(\ge0\). ∎

Num Under RoPE, a query scaling slid onto \(W_K\) leaves a \(7.6\%\) least-squares residual unless \(\ell\) is pair-constant. Folding post-LN gains into \(W_1\) changes the output. Legal rewrites agree to \(10^{-15}\).

Under SwiGLU, R4 moves \(\ell\) onto the linear branch and R1 fuses it into \(W_{\rm up}\), for \(\ell\) of either sign:

\[\begin{aligned}\mathrm{diag}(\ell)\big(\mathrm{silu}(W_gx)\odot W_ux\big)&=\mathrm{silu}(W_gx)\odot\big(\mathrm{diag}(\ell)\,W_ux\big)\\&=\mathrm{silu}(W_gx)\odot\big((\mathrm{diag}(\ell)\,W_u)\,x\big).\end{aligned}\]

The rules certify a merge and name the absorbing weight; a stuck rewrite certifies only a box-local obstruction. RoPE pins a query scaling to \(W_Q\) unless it is pair-constant. Bias-free layers leave a shift no slot, and a new bias, though nearly free at inference, changes the architecture. Post-LN feeds the normalised wire to the residual too, so R7 cannot carry its gains forward.

More: residual-stream edits that do merge

Some edits outside (a) merge anyway. A diagonal edit after a post-LN folds into that norm’s gain and shift; a shift after a pre-LN block with biases, into its FFN output bias; an orthogonal edit in an untied pre-RMSNorm model, globally, SliceGPT-style. No merge is known for LoReFT’s edit at interior positions.

Linear insertions, and ReFT

A linear inserted module fuses at once by R1, and the question is which weight-space method it becomes. ReFT, read one hidden vector at a time, becomes LoRA on a box the network lacks, the identity, written in homogeneous coordinates \((h,1)\) as \([I_d\mid0]\) so that its last column holds the bias.

(expand)

(a) holds by definition.

(b) \(UDW_0\) ranges over the rank-\(\le r\) matrices whose rows lie in \(\mathrm{row}\,W_0\), which is all of \(\mathcal M_{\le r}\) iff \(\mathrm{row}\,W_0=\mathbb R^n\). If \(W_0\) is invertible, \((U,D)\mapsto(U,DW_0)\) is a linear bijection over \(\Theta\). Conversely, an isomorphism preserves \(\mathrm{Im}\) (Obs I.4(c)), forcing \(\operatorname{rank}W_0=n\), and is a diffeomorphism, forcing \(2mr=r(m+n)\), that is \(m=n\). Pointed case, “if”: \(U_0D_0W_0=0\) and \(W_0\) invertible give \(U_0D_0=0\), and \(\operatorname{rank}(D_0W_0)=\operatorname{rank}A_0=r\) gives \(D_0\) full row rank, so \(U_0=0\). Choose \(g\in GL_r\) with \(A_0=gD_0W_0\) and compose with the gauge \((B,A)\mapsto(Bg^{-1},gA)\). “Only if”: the unpointed part gives \(W_0\) invertible. \(S_1\) is invariant under pointed isomorphism (Obs I.6), and here \(S_1=\{XD_0W_0+U_0Y\}\) for the serial adapter and \(\{XA_0\}=\{Z:\mathrm{row}\,Z\subseteq\mathrm{row}\,A_0\}\) for LoRA. If \((U_0)_{ij}\ne0\), take \(Y=e_jv^\top\) with \(v\notin\mathrm{row}\,A_0\), possible since \(r\lt n\); row \(i\) of \(U_0Y\) is then a nonzero multiple of \(v\), a contradiction. So \(U_0=0\), and \(\mathrm{row}(D_0W_0)=\mathrm{row}\,A_0\).

(c) Expand \(\Phi\). Every \(d\times(d+1)\) matrix of rank \(\le r\) factors as \(R^\top V\) with \(R^\top\) having orthonormal columns (QR), so the image is as stated. \(R^\top\) is injective, so the edit vanishes iff \(W=R\) and \(b=0\). The \(O(r)\)-action preserves \(\Phi\), since \(R^\top O^\top(OW-OR)=R^\top(W-R)\) and \(R^\top O^\top Ob=R^\top b\). The parameter space has dimension \(rd-r(r+1)/2+rd+r\), short of LoRA\(_r\)’s \(2rd+r\) by \(r(r+1)/2\), so no isomorphism exists. The map \((R,W,b)\mapsto(R^\top,[W-R\mid b])\) is the simulation, and it sends the neutral point to \((R_0^\top,0)\). There \(S_1=\{R_0^\top V\}\) has dimension \(r(d+1)\), more than the \(rd\) of LoRA\(_r\) pointed at \((0,A_0)\), so that pointing does not simulate LoReFT (Obs I.6). For the defect, \(R\,\Phi(h)=Wh+b\) because \(RR^\top=I_r\), so the \(R\)-component of the output is the affine map \(h\mapsto Wh+b\). ∎

Num The Jacobian rank is \(22=r(2d+1-r)\) at \(d=6\), \(r=2\), against \(26\) for LoRA\(_r\oplus\)bias. The serial adapter’s image has dimension \(7\) at \((m,n,r)=(5,3,1)\).

Per hidden vector LoReFT, like DiReFT, adds no new geometric type; Wu et al. already read DiReFT as LoRA on hidden representations at certain positions, and name position as what separates ReFT from PEFT. Position also blocks a merge. LoReFT acts at selected prompt positions, outside Theorem VI.3(a), and no merge into the base weights is known.

More: biases, initialisations and whole-sublayer adapters

With a bias \(b_0\), the serial adapter of (b) also moves it by \(UDb_0\). Houlsby et al. initialise both projections near zero, so their adapters start with a small pointing defect. He et al.’s parallel adapter sits beside a whole sublayer, and its linear version edits the skip path.

Attention is a homomorphism followed by a perspective map

Fix a query \(q\). Attention over a context \(K\) of key–value pairs computes

\[\mathrm{Attn}_q(K)=\frac{\sum_{(k,v)\in K}e^{\langle q,k\rangle/\sqrt d}\,v}{\sum_{(k,v)\in K}e^{\langle q,k\rangle/\sqrt d}}.\]

The pair (total weight, weighted sum of values) is additive in the context, and only the division \((Z,S)\mapsto S/Z\), the perspective map of convex analysis, is nonlinear. With a prefix \(P\) before the content \(C\), the output averages what \(P\) alone and \(C\) alone would give, weighted by their shares of the mass, so the prefix is a parallel branch gated by its share.

(expand)

(a) Sums over a disjoint union split, and \(Z=0\) only for the empty multiset.

(b) \(\pi(Z_P+Z_C,S_P+S_C)=\dfrac{S_P+S_C}{Z_P+Z_C}=\lambda\dfrac{S_P}{Z_P}+(1-\lambda)\dfrac{S_C}{Z_C}\).

(c) Each content weight is \(e^{\langle q,k\rangle/\sqrt d}/(Z_P+Z_C)=(1-\lambda)\,e^{\langle q,k\rangle/\sqrt d}/Z_C\), and \(\mathrm{Attn}_q(P)\) is a convex combination of \(V_P\).

(d) This is the definition.

(e) Induction over layers. Under causal masking, prompt positions attend only to prompt positions, and with the stated offset their positional encodings agree in both methods. Without the offset, RoPE sees the content-to-prompt displacement shifted by the prompt length, which the counter-rotation restores.

(f) A softmax over one token equals \(1\), so the original head computes \(W_o(W_vx+b_v)+b_o\). The prefixed output is real-analytic in \((x,\text{prefix})\), and one explicit witness of a nonzero second derivative along a ray makes the prefixes with affine output a nowhere-dense analytic subset. If \(W_o=0\) the head is constant, and if \(W_q=0\) with \(W_k^\top b_q=0\) the gate is constant and the head stays affine, which is why genericity of the frozen weights is needed. For unpointedness, the prefixed output on one token is \(W_o\big[(1-\lambda(x))(W_vx+b_v)+\lambda(x)\,\mathrm{Attn}_{q(x)}(P)\big]+b_o\) with \(\lambda(x)\gt0\). It equals the frozen output iff \(W_o\mathrm{Attn}_{q(x)}(P)=W_o(W_vx+b_v)\) for all \(x\). The left side lies in the compact set \(W_o\,\mathrm{conv}\,V_P\), and the right side is unbounded when \(W_oW_v\ne0\). ∎

Num The identities (a)–(d) and the non-affinity in (f) are checked numerically; Figure 13 recomputes them in your browser.

At the lowest prefixed layer content weights keep their pretrained ratios. Higher layers change only through their modified inputs, to first order additively at each position (Petrov et al., §6), so a prefix cannot impose a new query–key interaction directly; no network-level impossibility follows. Petrov et al. and Li and Liang state that prompt tuning is strictly weaker than prefix tuning; a one-layer example with one virtual token shows it, and the general case is open.

Where each method lands

The verdicts are box-local, and the attention results are per layer. Two questions remain open. Is prefix tuning non-mergeable for the whole network? Is prompt tuning strictly weaker than P-tuning v2 in general causal decoders, and how do the two relate in encoders?

A method is a pointed map \(\rho:(Q,q_0)\to(\Theta,\theta_0)\). Its image germ says what it can express, its fibres and the optimizer’s metric how it moves, base change what a moved \(\theta_0\) does, and extensions cover adapters and prompts. Exposé VII applies this toolkit to fourteen methods; Exposé VIII turns the answers into coordinates and arrows, mapped in Figure 14.

VII. Fourteen methods through the three invariants

Each card gives \((Q,q_0,\rho)\) for \(y=Wx\), \(W\in\mathbb R^{m\times n}\), then the image germ (Exposé II), fibre and metric (III), base change (V) and merge (VI). The top rule’s colour is the modification kind, and the last line links each claim to its result.

VIII.A Seven axes, each read off the formula

(expand)

Each axis has a procedure that uses only \(\rho\), \(q_0\) and the box.

• Kind. Check whether the trained model is \(f(\rho(q),-)\) with the original architecture (R), whether new boxes are inserted and some parameter value makes them the identity (E) or none does (E°), whether only the gradient path or the schedule changes (B), whether the inputs are trained checkpoints (P), and whether the adapter depends on a task, an input or a tenant (F\(_T\)). See Def I.2, Def IV.1 and Def VI.1.
• Shape. Read the degree of \(\rho-\theta_0\) in \(q\). An affine map gives Lin and a multihomogeneous non-linear one gives Cone. A group acting on \(\theta_0\) gives Orb, a torus acting on another method’s image gives Sat (Prop II.8), and an intersection gives Meet (Thm II.7).
• Base point. Evaluate \(\rho(q_0)\) and compute \(S_1=d\rho_{q_0}(T_{q_0}Q)\), then compare \(\dim S_1\) with the generic rank \(d(M)\) (Def I.5). For LoRA this is Thm II.2(b).
• Gauge. Insert \(gg^{-1}\) on each internal wire between trainable boxes and slide diagonals through copy spiders, then confirm \(\lvert M\rvert-d(M)\) by a Jacobian rank at a random point (Prop II.12(a)).
• Covariance. Test the identity of Def II.14 on generators of \(GL_m\times GL_n\), in distribution when frames are random or come from data.
• Merge. Run the rewrite rules R1–R7 of Thm VI.3 with their side conditions, adding the type condition of Prop V.2(d) for a quantised base. The rules are sufficient conditions. They certify M1 and M1→, and M∞ records a box-local obstruction. Whether a whole network admits some other merge is left open.
• Rebasing. Apply Thm V.3(a) to additive shapes and Thm V.3(b) and (c) to actions.

∎

Methods with different shapes or base-point types are never isomorphic, however alike their formulas (Obs I.4(c), Obs I.6).

VIII.B The periodic table

Figure 14 splits reparametrisations by how they touch \(W_0\), additively, by a group action, by a Hadamard product or by selecting coordinates. Some of its cells are empty by theorem. An affine \(\rho\) has the same differential everywhere, so no linear method is an apex, and in the catalogue none is. Corollary 2 of Thm V.3 ties idempotent rebasing to shapes that are fixed subspaces, subgroups or submonoids, and of the catalogue’s 92 idempotent methods, 84 are linear shapes and 7 are orbits. The other, WASI, trains on the rank-\(K\) variety \(\mathcal M_{\le K}\) itself, a set that does not move with the base.

VIII.C When one method simulates another

“Generalises” and “is a special case of” get one reading here. \(N\) simulates \(M\) when a pointed smooth map \(h:Q_M\to Q_N\) satisfies \(\rho_N\circ h=\rho_M\) (Def I.2), so \(N\) reaches every weight \(M\) reaches. Five rules produce every arrow in the atlas; three tests rule arrows out.

(expand)

The rules produce arrows. In each case \(h\) is smooth and pointed, and \(\rho_N\circ h=\rho_M\) is checked by multiplying out. A1 is Obs I.4(a), since \(\rho_M\circ\mathrm{const}_{q_0}=\theta_0\) and \(\mathrm{id}\circ\rho_M=\rho_M\). A2 holds because freezing \(q''\) at \(q_0''\) leaves the rest of the formula unchanged. For A3, take VeRA, where \(W_0+\Lambda_bB\Lambda_dA=W_0+(\Lambda_bB\Lambda_d)A\) and \(h\) sends \(b_0=0\) to LoRA-FA’s base point \(0\). A4 is the universal property of the fibre product (Obs I.4(b), Prop I.8(a)); for a meet such as HRA only the inclusion of images holds (Thm II.7). A5 is the identity below, from Obs I.7.

\[\rho_{M\oplus N}(q,q_0^N)=\theta_0+\delta_M(q)+\delta_N(q_0^N)=\rho_M(q).\]

The tests rule arrows out. An arrow gives \(\mathrm{Im}\,M=\rho_N(h(Q_M))\subseteq\mathrm{Im}\,N\), which is O1. Images are semialgebraic, or definable when softmax enters, so dimension is monotone under inclusion; and every update in \(\mathrm{Im}\,M-\theta_0\) also lies in \(\mathrm{Im}\,N-\theta_0\), so its rank is bounded by \(\rho_{\max}(N)\). Together these give O2. The chain rule at \(q_0\) gives \(S_1(M)\subseteq S_1(N)\) (Obs I.6), the first half of O3. For the strata, the images of \(\mathrm T\)-type and \(\mathrm S\)-type objects are incomparable (Thm II.2(d)). Inside the \(\mathrm T\) strata, \(S_1(\mathrm T_A)=\{XA_0\}\) contains matrices whose column space leaves \(\mathrm{col}\,B_0'\), so it does not lie in \(S_1(\mathrm T_B)=\{B_0'Y\}\), and symmetrically.

Transport. \(T_*\) is an automorphism of the slice with inverse \((T^{-1})_*\) (Prop II.13). It identifies \(M\) and \(T_*M\) as one object in two frames and does not by itself give an arrow between them.

∎

More: which published relations survive

Some become exact. VeRA is LoRA-FA\((A)\) with a constrained factor, and rsLoRA and a linear parallel adapter are LoRA. Others are weaker than they sound. HRA’s arrow to LoRA\(_r\) lands on a degenerate pointing (\(B_0A_0=0\)) outside the strata of Thm II.2, and LoRA is a special case of AdaLoRA only unpointed, their first-order images having dimensions \(mr\) and \(r\) (Lemma II.4).

The literature in time

Figure 16 dates each method by its first arXiv submission. Landmarks come in rough order, with bottleneck adapters in 2019, prefixes and LoRA’s cone in 2021, the torus of (IA)\(^3\) in 2022, base change with QLoRA in 2023, and GaLore’s moving frame in 2024, the year of 113 of the 382 methods.

What a budget buys

Proposition II.12 separates the budget \(\lvert M\rvert\), what a method trains, from the expressive dimension \(d(M)\), the directions it can move the weights in. The gap is gauge waste (Def I.5), per matrix \(r^2\) for LoRA, one for VeRA (numerically), zero for injective linear methods, at least \(2r^2+m+n-1\) for LoHa\(_{r,r}\) (Prop II.12). At 131,072 parameters on a \(4096\times4096\) matrix, LoRA\(_{16}\) reaches 130,816 directions at rank up to 16, LoHa\(_{8,8}\) at most 122,753 at rank up to 64. Rankings at matched budget and at matched \(d(M)\) can differ (Prediction X.10).

The catalogue

Everything above rests on a catalogue of 382 methods and 138 papers, checked against arXiv and, for library claims, Hugging Face PEFT as of 30 September 2026; 82 methods ship in that library. Each method entry gives formula, budget, locus, kind, merge verdict, initialisation, relations, sources, coordinates and a categorical reading, most with arrows and obstructions. Every entry has BibTeX and an address such as #m-lora.

The atlas is meant to grow. A new method enters by answering the seven questions and exhibiting its arrows, each with its map \(h\); a disputed comparison is settled by an explicit \(h\) or one of the three tests.

A fine-tuning paper says that LoRA+ gives \(B\) a larger learning rate than \(A\). The relative theory says that, under the hypotheses of Theorem III.10, LoRA+ lies on the torus orbit of plain LoRA with a larger \(\alpha\). Passing between such sentences lets each side borrow the other’s theorems, and Figure 19 is a dictionary for the passage.

How to read an entry

An entry claims that a phrase names a definite object, on which claims about the phrase can be checked. There are three strengths.

The box fixes each borrowed word’s sense; Exposé XI says more (Analogy vs theorem).

Most of the right-hand column concerns images, which measure expressivity rather than learnability; entries with dynamical claims state their optimizer.

Six entries, one from each exposé

I. “Method \(N\) can do whatever \(M\) can”

The weak reading asks that \(N\) reach every weight \(M\) reaches; the useful one asks that \(N\) also follow \(M\)’s parameter paths, through a map \(h\) with \(\rho_N\circ h=\rho_M\). Methods at \(\theta_0\) with such arrows form the pointed slice \(\mathsf{Meth}(\Theta,\theta_0)\) (Def I.2). The frozen model is initial, full fine-tuning is terminal, and the arrow from \(M\) to full fine-tuning is \(\rho_M\) itself (Obs I.4(a)). Image inclusion gives only a map of sets over \(\Theta\), which need not be smooth (Obs I.4(c)).

(expand)

The only pointed map from the one-point space into \(Q\) is \(*\mapsto q_0\), and it lies over \(\Theta\) because \(\rho(q_0)=\theta_0\). So the frozen model is initial. An arrow \(h\) from \(M\) into \((\Theta,\theta_0,\mathrm{id})\) must satisfy \(\mathrm{id}\circ h=\rho_M\), so it exists and is unique. For the gap between sets and smooth maps, take \(\rho_M(t)=t\) and \(\rho_N(u)=u^3\) on \(\mathbb R\), both pointed at \(0\). The images agree, and the only map with \(\rho_N\circ h=\rho_M\) is the cube root, which is not differentiable at \(0\).

∎

Every weight-space method pointed at \(\theta_0\) lies between the frozen model and full fine-tuning, and in exact arithmetic on a full-precision base its merge is that canonical arrow. Merging questions remain for extensions such as adapters, prefixes and ReFT (reading VI), for quantised bases (Prop V.2) and for combinations of adapters (Prop V.4).

More: a first-order test, and defaults that start elsewhere

The first-order image \(S_1\) detects part of this gap. In the cube-root example \(S_1(M)=\mathbb R\) is not inside \(S_1(N)=0\), so no smooth simulation exists (Obs I.6). Several initialisations start away from \(\theta_0\) (HF’s default FourierFT spectrum and VB-LoRA bank, init_lora_weights=False, LoRA-XS’s small random \(R\), quantised bases) and carry a pointing defect (Exposé V).

II. “The effective number of parameters”

A parameter count measures \(Q\), while expressivity lives in the image. The difference is spent on the fibres of \(\rho\), along which the gauge group moves. LoRA spends \(r^2\) parameters there, since \((Bg^{-1},gA)\) and \((B,A)\) give the same weights for every \(g\in GL_r\).

With \(\lvert M\rvert=\dim Q\) and \(d(M)=\dim\operatorname{Im}M\) (Def I.5), \(d(M)\le\lvert M\rvert-\dim(\text{generic gauge orbit})\), with equality when the gauge is locally transitive on generic fibres (Prop II.12(a)). For LoRA\(_r\) on an \(m\times n\) box with \(r\le\min(m,n)\), \(d=r(m+n-r)\). Compare LoHa\(_{r,r}\) with LoRA\(_{2r}\) at their shared budget \(2r(m+n)\), with \(2r\le\min(m,n)\). If \(2r^2\lt m+n-1\), LoHa reaches at least \((m+n-1)-2r^2\) fewer dimensions; if moreover \(r^2\le\min(m,n)\), it trades them for rank up to \(r^2\) when \(r\ge3\), and lies strictly inside LoRA\(_{2r}\) when \(r\le2\) (Prop II.12(c)). If \(2r^2\ge m+n-1\), as for \(r=32\) on layers 320 to 640 wide, LoHa matches or beats LoRA\(_{2r}\) in rank, and in dimension by its numerically checked dimension formula. Rank, in turn, is bounded by a minimum cut of the method’s tensor network (Thm II.10).

(expand)

The generic fibre contains the generic gauge orbit, and \(d=\lvert M\rvert-\dim(\text{generic fibre})\). For LoRA, the full-rank factorisations of a rank-\(r\) matrix form a single \(GL_r\)-orbit on which \(GL_r\) acts freely, so the generic fibre has dimension \(r^2\). For LoHa\(_{r_1,r_2}\), the product \(B_1A_1\odot B_2A_2\) is preserved by \(GL_{r_1}\times GL_{r_2}\) and by the diagonal torus \((B_1,A_1,B_2,A_2)\mapsto(D_1B_1,A_1D_2,D_1^{-1}B_2,A_2D_2^{-1})\), since \((D_1XD_2)\odot(D_1^{-1}YD_2^{-1})=X\odot Y\). At generic parameters the joint stabiliser is one-dimensional. Indeed \(D_1B_1g_1^{-1}=B_1\) makes every row of \(B_1\) a left eigenvector of \(g_1\), and \(m\ge r_1+1\) rows in general position force \(g_1=cI\) and \(D_1=cI\); the other three equations then give \(D_2=g_2=c^{-1}I\). So the orbits have dimension \(r_1^2+r_2^2+m+n-1\). At \(r_1=r_2=r\) this bounds \(d(\mathrm{LoHa}_{r,r})\) by \(2r(m+n)-2r^2-(m+n-1)\), and subtracting it from \(d(\mathrm{LoRA}_{2r})=2r(m+n)-4r^2\) gives the gap. The rank bound \(r^2\) comes from \(\operatorname{rank}(X\odot Y)\le\operatorname{rank}X\cdot\operatorname{rank}Y\), and it is attained when \(r^2\le\min(m,n)\). For \(r\le2\), LoHa\(_{r,r}\) maps into LoRA\(_{r^2}\), whose image lies in that of LoRA\(_{2r}\); when the gap is positive, the containment is strict.

∎

Report \(d(M)\) beside \(\lvert M\rvert\) in every comparison (Prediction X.10).

III. “The LoRA+ ratio, \(\alpha\), the init scale, rsLoRA”

Multiplying \(B\) by \(c\gt0\) and dividing the scale \(s\) by \(c\) leaves the weights unchanged, and Adam with \(\varepsilon=0\) ignores a positive rescaling of a block’s gradient history. Rescaling the block’s learning rate too then reproduces the run step for step, and the per-block factors form a torus acting on the hyperparameters.

Theorem III.10 makes this exact for any multihomogeneous method, assuming no parameter-dependent regulariser, no global clipping, the same data order and dropout masks, and exact arithmetic. For LoRA (Corollary III.11(b)), under Adam with \(\varepsilon=0\), no weight decay or clipping and one schedule for both groups, LoRA+ with ratio \(\lambda\) at scale \(s\) produces exactly the trajectory of plain LoRA at scale \(\lambda s\), with the same \(\eta_A\) and \(A_0\). For \(\varepsilon\gt0\) this needs \(\varepsilon_B'=\lambda\varepsilon\), and with AdamW a decay \(\lambda\cdot\mathrm{wd}\) on \(B\). Under SGD the scale is \(\sqrt\lambda\,s\).

(expand)

Take \(c_B=1/\lambda\) and \(c_A=1\), so that \(B'=c_BB\) and \(s'=s/c_B=\lambda s\). LoRA+ has \(\eta_B=\lambda\eta_A\), and the rescaled run is asked to use \(\eta_B'=c_B\eta_B=\eta_A\), which is plain LoRA. The start \(B_0=0\) stays \(0\). At corresponding points \(\nabla_{B'}=c_B^{-1}\nabla_B\). With \(\varepsilon=0\), Adam’s step does not change when a block’s whole gradient history is multiplied by a positive constant, so the step in \(B'\) is \(\eta_B'/\eta_B\) times the step in \(B\), and \(\Delta B'=c_B\Delta B\) holds exactly when \(\eta_B'=c_B\eta_B\). The weights agree at step \(0\), and induction over steps carries the agreement along the whole run. With \(\varepsilon\gt0\) the same computation asks for \(\varepsilon_B'=\varepsilon/c_B\), and decoupled decay for \(\mathrm{wd}_B'=\mathrm{wd}/c_B\). Under SGD the gradient factor \(c_B^{-1}\) and the required \(c_B\) combine into \(\eta_B'=c_B^2\eta_B\), which gives \(c_B=\lambda^{-1/2}\) and scale \(\sqrt\lambda\,s\).

∎

Only the torus orbit of \((s,\eta_A,\eta_B,\sigma_A)\) matters. Under Adam it is fixed by \(s\eta_A\eta_B\) and \(\sigma_A/\eta_A\) (Cor III.11(a)), so at a fixed rank LoRA+, rsLoRA and \(\alpha\)-tuning are one family. A gain of LoRA+ over LoRA with \(\alpha\) multiplied by \(\lambda\) needs something that breaks the torus, such as global-norm clipping (a Hugging Face Trainer default), an \(\varepsilon\) or decay shared across groups, or different schedules (Prediction X.1). Figure 7 runs both trainings.

More: the same fibres under gradient flow

Entry 22 reads the same fibres under gradient flow. With block-scalar rates the gauge yields the conserved charge \(\Phi=B^\top B/\eta_B-AA^\top/\eta_A\) (Thm III.5), after Du, Hu and Lee, 2018, Arora et al., 2018, Zhao et al., 2023 and Marcotte, Gribonval and Peyré, 2023. Balanced points \(B^\top B=AA^\top\) form the Kempf–Ness slice (Prop III.7; Kempf and Ness, 1979; Richardson and Slodowy, 1990).

IV. “GaLore versus LoRA”

Backpropagation makes each map a lens, running forward from parameters to outputs and backward from output gradients to parameter gradients. LoRA restricts the forward side. GaLore compresses the gradient before the optimizer and expands its output after.

Let \(\rho_k(C)=W_k+\mathcal LC\) with \(\mathcal L\) linear and \(C_0=0\). Let the optimizer’s state and output depend only on the gradients fed to it, with one implementation on both sides (SGD, heavy-ball or Nesterov momentum, Adam without weight decay, Adafactor without parameter scaling). Then training \(C\) and merging at the end of a period gives the same \(W\)-trajectory as the backward method with compression \(\mathcal L^*\) and anchor \(\mathcal L\), started at \(W_k\) from the same optimizer state (Prop IV.2). Decoupled weight decay, LAMB and LARS trust ratios, parameter-scaled Adafactor and full-gradient clipping break the identity.

(expand)

By induction on the step, the weights of the second procedure equal \(W_k+\mathcal LC_t\), where \(C_t\) is the state of the first. Both procedures feed the optimizer the same sequence \(\mathcal L^*\nabla L(W_t)\) from the same state, so they receive the same outputs \(U_t\). One sets \(W\leftarrow W-\eta\,\mathcal LU_t\), and the other sets \(C\leftarrow C-\eta\,U_t\), which moves \(W_k+\mathcal LC\) by the same amount. The excluded ingredients see different tensors in the two procedures. Decoupled decay acts on \(C\) in one and on \(W\) in the other, parameter-scaled steps read the size of \(C\) in one and of \(W\) in the other, and full-gradient clipping uses \(\lVert G\rVert\) in one and \(\lVert\mathcal L^*G\rVert\) in the other.

∎

Under these hypotheses, GaLore with a projector \(P_k\) fixed on each period is one-sided LoRA \(W_k+P_kC\), merged and restarted every period with \(C\)’s optimizer state carried over unchanged. It adds two things to one-sided LoRA, a frame from the gradient’s SVD and that carry-over, which reads moments written for \(P_{k-1}\) as if written for \(P_k\). LDAdam transports them instead.

More: the rest of the cotangent family

Under the same hypotheses the identity (Prop IV.2, Prop IV.3) covers LoRA-FA and GaLore with random projectors, MeZO up to its finite-difference estimate, GaLore with gradient SVDs, and BitFit and LISA (without weight decay) with coordinate blocks. Flora as published, APOLLO and Fira update outside their projected subspace and are not covered.

V. “QLoRA and LoftQ”

A quantiser \(\kappa\) moves the pretrained point to \(\kappa\theta_0\) and changes the function. Additive methods commute with translation, \(M(\theta+v)=\tau_v\circ M(\theta)\), so QLoRA is LoRA attached at \(\kappa\theta_0\), with pointing defect \(e=\kappa\theta_0-\theta_0\) (Def V.1), and \(\theta_0\) lies outside its image unless \(\operatorname{rank}e\le r\) (Prop V.2(a)). Splitting initialisations come in three orders with three different defects (Prop V.2(b)). Write \(W\) for the full-precision weight, \(\delta_0\) for the initial adapter, \(W_{\rm res}=W-\delta_0\) and \(s=\alpha/r\).

(expand)

Translation is the definition of additivity. The image of LoRA\(_r\) pointed at \(\kappa\theta_0\) is \(\kappa\theta_0+\mathcal M_{\le r}\), which contains \(\theta_0\) iff \(-e\in\mathcal M_{\le r}\), that is, iff \(\operatorname{rank}e\le r\). For the table, compute \(\rho(q_0)-W\) in each pipeline. Splitting first gives \(\kappa(W_{\rm res})+\delta_0-W=\kappa(W_{\rm res})-W_{\rm res}\). Fitting the error stores the factors of \(-\mathrm{SVD}_r(e)\), which the forward pass multiplies by \(s\), so \(\delta_0=-s\,\mathrm{SVD}_r(e)\) and the defect is \(\kappa W-s\,\mathrm{SVD}_r(e)-W=e-s\,\mathrm{SVD}_r(e)\). Re-quantising the split residual gives \(\kappa(r')+\delta_0-W=e+\kappa(r')-r'\).

∎

Measured from its own start, QLoRA obeys the rank bound of any LoRA (Thm II.1(c)); measured from \(\theta_0\), its update also contains the generically full-rank defect. Since \(e-\mathrm{SVD}_r(e)\) is orthogonal to \(\mathrm{SVD}_r(e)\), one-step LoftQ in HF beats QLoRA in Frobenius norm only for \(0\lt s\lt2\). Further LoftQ iterations need not decrease their objective, and at \(s=1\) the output is an approximate section, pointed up to its attained residual (Prop V.2(c)).

VI. “Merging an adapter into the weights”

Reading I settled merging for weight-space methods on a full-precision base. Adapters, prefixes and (IA)\(^3\) scalings insert new boxes and live on an extension of \(f\), so merging one is a lifting problem along the realisation \(\theta\mapsto f(\theta,-)\).

A method \(\rho\) on an extension \(\tilde f\) is mergeable if some \(\mu:Q\to\Theta\) has \(\Phi_{\tilde f}\circ\rho=\Phi_f\circ\mu\) (Def VI.1). Theorem VI.3 gives sufficient conditions box by box, through seven rewrite rules with side conditions such as position-uniform edits, unshared absorbing weights and bias slots for shifts. For instance, HF applies the feed-forward vector \(\ell\) of (IA)\(^3\) to the input of \(W_{\rm down}\), where it always fuses. It fuses into \(W_{\rm up}\) instead under SwiGLU or GeGLU for every \(\ell\), and under ReLU when \(\ell\ge0\). Under GELU, an entry \(\ell_i\) on a nonzero row of \(W_{\rm up}\) fuses there only if it is \(0\) or \(1\).

(expand)

The scaling of the input of \(W_{\rm down}\) is absorbed as \(W_{\rm down}\mathrm{diag}(\ell)\) by fusion (R1), whatever the activation. In a gated block the hidden activation is \(\mathrm{silu}(W_gx)\odot W_ux\), and the spider slide (R4), \(\mathrm{diag}(\ell)(a\odot b)=a\odot(\mathrm{diag}(\ell)b)\), moves the scaling onto the linear branch, so \(W_u\mapsto\mathrm{diag}(\ell)W_u\) for every \(\ell\). Under ReLU the positive-diagonal slide (R3), \(\mathrm{diag}(\ell)\sigma^{\times d}=\sigma^{\times d}\mathrm{diag}(\ell)\) for \(\ell\ge0\), does the same; this slide holds exactly for positively homogeneous \(\sigma\). GELU is not positively homogeneous. Absorbing \(\ell_i\ne0\) into a nonzero row \(w\) of \(W_{\rm up}\) would need a row \(w'\) with \(\ell_i\,\mathrm{GELU}(w\cdot x)=\mathrm{GELU}(w'\cdot x)\) for all \(x\). The left side is constant along directions orthogonal to \(w\), and the right side is not unless \(w'\) is parallel to \(w\), so \(w'=cw\), and then \(\ell_i\,\mathrm{GELU}(t)=\mathrm{GELU}(ct)\) for all \(t\). Since \(\mathrm{GELU}(t)=t/2+t^2/\sqrt{2\pi}+O(t^4)\), the coefficients of \(t\) and \(t^2\) give \(c=\ell_i\) and \(\ell_i=\ell_i^2\). With biases the same argument gives \(\ell_i\,\mathrm{GELU}(u)=\mathrm{GELU}(cu+\beta)\), and \(u=0\) forces \(\beta=0\), because GELU vanishes only at \(0\). An entry \(\ell_i=0\) is absorbed by zeroing its row, since \(\mathrm{GELU}(0)=0\).

∎

“Zero inference latency” is certified by local rewriting, a sufficient test that also names the absorbing weight; the \(W_{\rm up}\) route is idle in practice, as \(W_{\rm down}\) always works. Nonlinear adapters, input-routed mixtures and prefixes do not merge box-locally. At its layer a prefix is a gated parallel adapter (Thm VI.5(b)), not box-locally mergeable for generic frozen weights and prefixes (Thm VI.5(f)); whole-network merging of prefixes is open.

Reading from the right

Some mathematical constructions are methods already in use.

• The product \(\mathrm{FA}(A_0)\times\mathrm{FB}(B_0)\) of the two one-sided LoRAs is the sandwich \(R\mapsto W_0+B_0RA_0\). In the SVD frames of \(W_0\), with \(\sigma_r\gt0\), it is LoRA-XS pointed at \(R=0\) (Prop I.8(a); Bałazy et al., 2024).
• For injective \(W_0\) and even \(r\) with \(2\le r\le n-2\), LoRA’s image meets the rotation orbit \(W_0\cdot SO(n)\) exactly in the image of HRA without Gram–Schmidt, the HF default (Thm II.7; Yuan et al., 2024). This is a meet of sets, and HRA is not the categorical product.
• Transport along a fixed \(T\in GL(\Theta)\) is an automorphism of the slice. It carries coordinate masks to FourierFT (with no conjugate frequency pair sampled), WaveFT (when its padded inverse wavelet transform is invertible) and C3A, each started at zero, and to LoRA-XS and SVFT in the SVD frame of \(W_0\), which depends on \(\theta_0\). The inverse of the rearrangement of Van Loan and Pitsianis, 1993 carries LoRA\(_k\) to unconstrained sums of \(k\) Kronecker products (Prop II.13).
• The monoidal sum \(\oplus\) adds updates, so its image is a Minkowski sum. RoSA, GraLoRA and the stages of ReLoRA, unrolled in time, are words in \(\oplus\) (Obs I.7).

Products and meets can be tried on other pairs too. Meets of images always exist as sets of weights, while products can fail to exist (Obs I.4(b)). Which meets admit a polynomial chart, as LoRA-XS and HRA’s image do, is an open problem.

Why the theory should risk something

A framework that only re-describes known methods cannot be wrong, and so it earns little trust. The claims below say what a training run will do. Each inherits the optimizer scope of the result it rests on; a claim about images needs none, because no run can leave the image.

Read the kind column as a warning label. Theorem means the sharp half cannot fail inside its hypotheses. Theorem + data pairs a proved mechanism with an empirical claim about its size. Conjecture means we expect it and have not proved it. The toys check mechanisms in float64 on small matrices; the experiments on real models are still to be run.

Predictions about optimizers

LoRA has four knobs, the scale \(s=\alpha/r\), the rates \(\eta_A,\eta_B\) and the initial size \(\sigma_A\) of \(A\). A two-dimensional torus of rescalings moves all four without changing any weight. LoRA+ turns one knob, the ratio \(\lambda=\eta_B/\eta_A\), and under Adam that turn lands on the orbit of plain LoRA with a larger \(\alpha\).

More: why the test avoids \(\varepsilon=0\)

Literal \(\varepsilon=0\) needs the convention \(0/0=0\). At \(B_0=0\) the gradient of \(A\) vanishes, and a stock Adam divides zero by zero there.

Under gradient flow LoRA conserves the charge \(\Phi=B^\top B/\eta_B-AA^\top/\eta_A\), and when \(\Phi\) is isotropic the dynamics have a closed form. Small singular values of the update grow through \(B\) alone, as in LoRA-FA; large ones grow through both factors in balance.

LoRA+ moves \(\sigma^*\) up by \(\sqrt\lambda\), so more of the update spends longer in the LoRA-FA regime. PiSSA at equal rates has \(\kappa=0\), and under gradient flow it is balanced from the first step.

Part of LoRA’s gauge is a symmetry of \(\rho\), and gradient flow conserves its charge whatever signal drives it. A discrete step conserves it up to second order in the step size; nothing forces Adam’s sign-like step to.

(expand)

Write \(g_B=\nabla_B=sGA^\top\) and \(g_A=\nabla_A=sB^\top G\) for one weight-space gradient \(G\). Expanding the step,

\[\tfrac1{\eta_B}B^{+\top}B^+=\tfrac1{\eta_B}B^\top B-\eta\big(g_B^\top B+B^\top g_B\big)+\eta^2\eta_B\,g_B^\top g_B,\] \[\tfrac1{\eta_A}A^+A^{+\top}=\tfrac1{\eta_A}AA^\top-\eta\big(g_AA^\top+Ag_A^\top\big)+\eta^2\eta_A\,g_Ag_A^\top.\]

The first-order terms agree, since \(g_B^\top B=sAG^\top B=Ag_A^\top\) and \(B^\top g_B=sB^\top GA^\top=g_AA^\top\). They cancel in the difference, which leaves \(\Delta\Phi=\eta^2(\eta_Bg_B^\top g_B-\eta_Ag_Ag_A^\top)\). The argument uses only that both gradients come from one \(G\), so it covers minibatches, LoRA dropout on the input of \(A\), and global clipping when \(g_A,g_B\) denote the gradients actually applied.

∎

Predictions about images

DoRA splits each output neuron into a length and a direction. At the level of images that adds exactly one free gain per output neuron to LoRA, so anything else that separates the two happens in training, where Adam notices the coordinates a parameter is written in.

A rank-one update writes one new feature into many neurons, and LoRA does this with a single rank. A Kronecker sum measures rank across a different cut of the same four-leg tensor, where, for the splits HF PEFT uses, a generic rank-one update is as complex as any matrix can be.

(expand)

The Van Loan–Pitsianis rearrangement \(\mathcal R\) permutes the entries of a matrix. It is therefore a linear isometry for the Frobenius norm, and by Thm II.11(a) it carries the set of \(k\)-term Kronecker sums onto the matrices of rank at most \(k\). The distance from \(uv^\top\) to that set equals the distance from \(\mathcal R(uv^\top)\) to the rank-\(\le k\) matrices, which by Eckart–Young is the root of the sum of the squared singular values beyond the \(k\)-th. The proof of Thm II.11(c) shows that the singular values of \(\mathcal R(uv^\top)\) are the products \(\sigma_i\tau_j\).

∎

Parameter count measures what a method pays, expressive dimension what it can reach. The difference is gauge, parameters that move without moving the weights. It ranges from one parameter for VeRA through \(r^2\) for LoRA\(_r\) to at least \(2r^2+m+n-1\) for LoHa\(_{r,r}\), which has the budget of LoRA\(_{2r}\).

Predictions about symmetry

The hidden units of a feed-forward block carry no order. Permute them with the matching rows and columns, and the network computes the same function. A method that never looks at unit order should not notice; a method built on a frame, a reshape or a block partition of the hidden coordinates should.

If the prediction holds, seed variance understates the variance of methods tied to unit order across equally good checkpoints.

SliceGPT rotates the residual stream of a model without changing what it computes. A method whose image rotates with the weights reaches the same functions afterwards; a method built on the stream’s coordinates may lose or gain reach.

PiSSA and CorDA choose their split from a truncated SVD, which follows the coordinates under rotations and uniform scalings and changes when they are stretched unevenly. Whitening the input first removes that dependence from the split.

Predictions about base change and merging

Merging and restarting lets a method compose its own moves. For an additive cone such as ReLoRA’s, composition keeps buying rank; for block OFT with a fixed partition, a product of block rotations is again a block rotation.

LoraHub mixes adapters factor by factor. A factor belongs to no adapter in particular, since \((B,A)\) and \((cB,A/c)\) give the same update, so the mix depends on choices the adapters do not record unless the modules share a factor.

A safe rule combines LoRAs on the products \(B_iA_i\), or on factors only when the modules share one.

One more prediction, and how to read a failure

One expectation is not on the list. By Prop II.13, unconstrained Kronecker sums are LoRA after a rearrangement of entries, and FourierFT (with no sampled conjugate pair), WaveFT (when HF’s pad–IDWT–crop map is invertible) and C3A are coordinate masks in a fixed frame; an optimizer that sees the loss only through the trainable parameters runs each exactly as its source on a transported loss. At matched learning-rate sweeps we therefore expect the gap between a transported method and its source to follow how well each image aligns with the task’s update, measured by projecting a full fine-tuning \(\Delta W\) onto each image.

A failure inside the hypotheses of X.1, X.5, X.6, or of a proved half, would mean an error in a proof, and we would mark that result refuted; a failure outside them shows which hypothesis matters. A failure of X.2, X.4 or X.11 would refute Conjecture D, A or C.

What the borrowed words mean

Words borrowed from geometry and physics compress a great deal, and each also carries connotations no theorem here supports.

Three scopes travel with every claim. Expressivity is a different property from learnability. Only Exposé III is dynamical, and each of its results names its optimizer. Mergeability is proved box by box under explicit side conditions, and whole-network statements are open. Most of the mathematics is real algebraic geometry, invariant theory, Lie generation and gradient-flow ODEs; the category theory does its own work in seven places: the interval [frozen, full FT] and simulation; invariants finer than the image; a product that yields LoRA-XS, and meets that organise others; base change along maps that change the function; the lens; naturality of gradient descent and of combination rules; and rank as a min-cut.

Some statements rest on less than a full proof. The equalities in the dimensions of LoHa (Prop II.12) and VeRA are checked numerically; the bounds are proved. For VeRA with \(r\le n\) and generic frozen factors the equality also follows from the toric-cone description of C.14, and the generic equality for DoRA (Prop II.8) is proved. Prop V.4(a) in its continuous case and the converse in Cor III.13 have complete proofs, and Prop IV.3 is argued around Frobenius’ theorem. theory/framework_checks.py re-runs the main numbers in about a minute (see Reproducing the numbers).

Open problems

Each grew out of a proof that stopped short, and each is stated so that a solution would be recognisable.

1. LoHa’s dimension. Prop II.12 proves \(d(\mathrm{LoHa}_{r_1,r_2})\le\min\big(mn,\,r_1(m+n-r_1)+r_2(m+n-r_2)-(m+n-1)\big)\), with equality checked only numerically. Prove it in general, and decide whether the Hadamard variety is closed.
2. Initialisations, dynamically. Thm II.2 classifies three strata by expressivity, and Cor III.13 shows that Adam sees \(A_0\) only up to signed row permutations. Classify the apex fibre modulo \(B_r\), stratified by the charge \(\Phi_0\) (Cor III.6), degenerate pointings included.
3. HRA’s apex in practice. HRA’s paired initialisation, HF PEFT’s default, starts at an apex (Thm II.7(b), for injective \(W_0\) and even \(r\le n-2\)), with \(\dim S_1=kn-\tfrac{k(k+1)}2\), \(k=r/2\), against \(d=rn-\tfrac{r(r+1)}2\). Does it slow early training against a regular start at matched budget?
4. Charts for meets. LoRA-XS (Prop I.8(a)) and HRA’s image (Thm II.7(a)) are meets with a polynomial chart; LoRA meet output scaling (Prop I.8(b)) is a union of coordinate planes. When does a meet of two named methods admit one?
5. The frontier. Incomparable pairs are common: split against zero-init LoRA (Thm II.2(d)), Kronecker sums or GraLoRA against LoRA (Thm II.11, Prop II.12(c)). Find the size-\(k\) frontier of \(\mathsf{Meth}\), whose maximal elements are pairwise incomparable.
6. An Adam analogue of the crossover. Thm III.5 and Thm III.9 hold for gradient flow. Find a conserved or monotone quantity for sign-like updates, whose symmetry is only \(B_r\) (Thm III.12).
7. Prefixes and prompts. For a vanilla softmax head with \(W_o\ne0\) and \(W_q\ne0\), a generic prefix is not box-locally mergeable (Thm VI.5(f)); settle the whole network. In causal decoders every soft prompt is a deep prefix (Thm VI.5(e)); Petrov, Torr and Bibi and Li and Liang assert that the inclusion is strict, as it is in a one-layer, one-token example. Prove the generic case, and settle encoders.

Conjectures A, C and D are open as well; they appear above as Predictions X.4, X.11 and X.2.

Debts

Much of what this site proves was known in another language or setting. A re-derivation earns its place only if the new language adds something, so the box states what we add next to each debt.

The referee record

Every numbered result went to two independent model referees, who checked it against its proof, the literature and the Hugging Face PEFT source. Of the 41 results, 14 stood as first written, with wording fixes, and 27 were repaired by an added hypothesis, a narrowed claim, a replaced proof or a withdrawn reading. A second review attacked the repaired statements, found a false or overreaching clause in 19 of them, and those were narrowed again. Only the final statements appear on this site; the notes are in theory/propositions.json and theory/round2_verdicts.json.

More: what changed, result by result

Result	Status	What the referees changed
Obs I.4 universal properties	repaired	Products need a fibre product in \(\mathbf{Diff}\) and can fail to exist (\(\mathrm{LoRA}_1\times\text{Frozen}\) on \(1\times1\) weights).
Obs I.6 \(S_1\) is functorial	verified	Monotonicity stated only along pointed morphisms.
Obs I.7 additive monoidal structure	repaired twice	\(r\mapsto\mathrm{LoRA}_r\) is strong monoidal only unpointed or at degenerate pointings (\(\dim S_1\) is \(mr\) against \(2mr\)); HF’s cat merge described exactly.
Prop I.8 a product and a meet	verified	LoRA-XS needs \(\operatorname{rank}W_0\ge r\); the set-level meet in (b) is no new method.
Thm II.1 pointings of an additive method	verified	Stated for exact pointings, with a robust form; credited to Meng et al.
Thm II.2 three strata of LoRA initialisations	repaired	Full-rank pointings only, and only as a set; HF’s orthogonal init and QLoRA/LoftQ defects lie outside.
Prop II.3 first-order rank formula	repaired	Scale and gradient flow made explicit; Adam’s first step has an \(\ell_1\) rate; LoRA-GA’s \(A_0\), not its \(B_0\), maximises the rate.
Lemma II.4 zero gate	verified	Hypotheses stated; a function-level variant for LLaMA-Adapter; AdaLoRA instance corrected.
Thm II.5 complete invariants	verified	Methods that can create zeros ((IA)\(^3\), HiRA) keep only “zeros of \(W_0\) stay zero”.
Cor II.6 what orthogonal methods never do	repaired twice	Exactly orthogonal input-side maps only; BOFT keeps cosines up to sign; HF’s Cayley–Neumann OFT and GOFT’s scaler excluded.
Thm II.7 HRA as a meet; its apex	repaired twice	An image-level meet with \(W_0\cdot SO(n)\), no product; strictly above LoRA \(\cap\) OFT only for exactly orthogonal OFT.
Prop II.8 DoRA’s image	repaired twice	\(\mathrm{Diag}_m\cdot(W_0+\mathcal M_{\le r})\) for \(W_0\) without zero rows; rank formula needs nonzero magnitudes; Conj D made falsifiable.
Prop II.9 the torus ladder	repaired	RoAd\(_1\) is (IA)\(^3\) over \(\mathbb C\); only zeros of \(W_0\) persist; (IA)\(^3\)’s FFN vector acts on the input of \(W_{\rm down}\).
Thm II.10 cut-rank bound	verified	A large min-cut is necessary for high rank but not sufficient.
Thm II.11 Kronecker incomparability	repaired twice	\(\kappa_1\) is the maximal Kronecker rank for aligned splits; LoKr with \(r'\le\min(m_2,n_2)\) lies in LoRA\(_r\) iff \(r\ge\min(m_1,n_1)r'\).
Prop II.12 gauge census	repaired twice	GraLoRA’s maximum rank is \(\min(kr,m,n)\); LoHa loses dimensions to LoRA\(_{2r}\) only when \(2r^2\lt m+n-1\).
Prop II.13 transport of structure	repaired twice	Any \(T\in GL(\Theta)\); commutes with rebasing for additive families and fixed \(T\); FourierFT needs no conjugate pair.
Obs II.15 descent	repaired twice	Reachable sets only, for box-wise methods without shared frames; residual rotations need a gain-fused pre-norm model.
Prop II.16 covariant splits	repaired twice	MiLoRA and CorDA-KPM are bottom-\(r\) splits; trace-relative damping keeps \(CO(n)\); shipped CorDA code is not covered.
Obs III.2 chain rule for cometrics	verified	Scope stated: gradient flow, no LoRA dropout, constant rates.
Thm III.3 naturality of gradient descent	verified	The title “natural only along isometries” was false in general and was replaced.
Prop III.4 cometrics that descend	repaired	Undamped (S)GD only; the published methods use AdamW or damping.
Thm III.5 Noether charges	repaired twice	Isometries need the signal to factor through \((t,\rho(q))\); relabelled Known; per-channel-rate remark needs distinct rate products.
Cor III.6 charge at initialisation; decay	repaired	The decay commentary was reversed: coupled decay tends to \(\mu=0\), per-time decay to \(\Phi=0\).
Prop III.7 Kempf–Ness slice	verified	Uniqueness up to \(O(r)\) proved; the Loewner floor renamed.
Prop III.8 balance–escape trilemma	verified	Balance generalised to \(B_0^\top B_0=cA_0A_0^\top\).
Thm III.9 isotropic-charge dynamics	verified	Pointwise on the rank-\(r\) locus; the Kaiming crossover is a band.
Thm III.10 the hyperparameter torus	verified	Hypotheses explicit; extended to schedules, per-block \(\varepsilon\), decay and clipping.
Cor III.11 LoRA+ and rsLoRA	verified	A shared warm-up keeps the torus; linear modules only.
Thm III.12 optimizer gauge hierarchy	repaired twice	Adam respects \(B_r\), positive diagonals only with per-channel rates, \(\varepsilon\) and decay; damping cuts \(GL_r\) to \(O(r)\).
Cor III.13 what zero-\(B\) chooses	repaired	The apex fibre is \(\{BA=0\}\); the \(GL_r\) case needs equivariance along the whole trajectory.
Prop IV.2 degree-one collapse	repaired	Decoupled decay, default Adafactor and full clipping break it; GaLore carries optimizer state.
Prop IV.3 integrability	repaired twice	The “only if” was false; the top-\(r\) gradient field is non-involutive for every \(r\ge1\).
Obs IV.4 the backprop cone	repaired twice	An upper bound; forward mode pays off only for \(k=O(1)\).
Prop V.2 quantisation and splitting	repaired twice	Three orders, three defects; HF’s LoftQ init ignores the LoRA scale; QA-LoRA merges on-grid.
Thm V.3 rebasing closure	repaired twice	BOFT reaches \(SO(n)\) only at full depth; HF’s Cayley–Neumann merges contract.
Prop V.4 combination rules	repaired twice	The “iff” needs full rank or continuity; factor-wise DARE is gauge-dependent too.
Lemma VI.2 activations fix the basis	verified	Gated FFNs have the full monomial group.
Thm VI.3 the merge calculus	repaired twice	Sufficient conditions only; RoPE needs scales constant on rotary pairs.
Prop VI.4 linear insertions, ReFT	repaired twice	LoReFT has dimension \(r(2d+1-r)\) and a pyreft pointing defect; the serial-adapter criterion needs \(\operatorname{rank}A_0=r\).
Thm VI.5 attention as a homomorphism	repaired twice	Rigidity is per layer; the prompt-to-prefix map credited to Petrov, Torr and Bibi.

A repair is a record of an error that was caught. We keep the record public so that a reader can weigh each result by its history, and so that the next error, when someone finds it, has an obvious place to go.

How to use this appendix

This appendix is for readers who know linear algebra, multivariable calculus, basic probability and the PEFT literature, and who have not studied category theory, Lie groups or real algebraic geometry. It skips what ML readers already use, such as the SVD, rank, orthogonal matrices, eigenvalues, Jacobians, gradients, Adam, LoRA and the transformer block. Each of the 93 entries gives a short definition or statement with its hypotheses, a small example from fine-tuning, the places on the site that use it, and usually a textbook for further reading. Definitions carry an ochre rule, standard facts quoted from the literature a dotted teal one, and the one remark a grey one.

Part A covers categories, slices and base change, Part B monoidal categories, string diagrams and lenses, Part C manifolds, cones and real algebraic geometry, and Part D groups, symmetry and conservation laws. The parts can be read in any order, since each entry links to the entries it depends on at their first use. In the exposés, the first use of many key terms is underlined with dots and links to its entry here, and each entry ends with links back to the results that rely on it.

A few symbols from the exposés recur in the examples. A method \(M=(Q,q_0,\rho)\) is a smooth map \(\rho:Q\to\Theta\) from trainable parameters to weights with \(\rho(q_0)=\theta_0\) (A.8), and \(\lvert M\rvert=\dim Q\) is its parameter budget. The expressive dimension \(d(M)\), the first-order image \(S_1\) and the first-order deficit are defined in C.3; the base point is regular for \(M\) when the deficit is \(0\) and an apex when it is positive. LoRA is \(\rho(B,A)=W_0+sBA\) with \(B\in\mathbb R^{m\times r}\) and \(A\in\mathbb R^{r\times n}\), and its full-rank pointings fall into the strata \(\mathrm T_A\), \(\mathrm T_B\) and \(\mathrm S\) of C.19. Block-scalar rates give each factor one learning rate, \(\eta_B\) for \(B\) and \(\eta_A\) for \(A\) (D.23).

The index below lists every entry with its area and the first places it is used. The box scrolls, and its header stays in view.

A. Categories without tears

20 entries

Part A collects the category theory the exposés use, from the definition of a category up to base change, and illustrates every notion in one running example, the pointed slice \(\mathsf{Meth}(\Theta,\theta_0)\) of weight-space methods. Entries A.1 to A.8 set up categories, functors and slices, A.9 to A.16 cover universal properties, limits and orders, and A.17 to A.19 cover transport, lifting and base change, with A.20 listing the constructions the site names and leaves unused.

Categories and slices

Definition A.1 (category). A category \(\mathcal C\) has objects and, for any two objects \(X,Y\), a set \(\mathcal C(X,Y)\) of morphisms, or arrows, \(f:X\to Y\). Arrows \(f:X\to Y\) and \(g:Y\to Z\) have a composite \(g\circ f:X\to Z\). Composition is associative, and every object has an identity arrow \(\mathrm{id}_X\) with \(f\circ\mathrm{id}_X=f=\mathrm{id}_Y\circ f\).

The site uses \(\mathbf{Set}\) (sets and functions), \(\mathbf{Diff}\) (smooth manifolds and smooth maps), \(\mathbf{FinVect}\) (finite-dimensional real vector spaces and linear maps, where tensor networks are evaluated) and the pointed categories \(\mathbf{Set}_*\) and \(\mathbf{Diff}_*\) of A.7. In \(\mathsf{Meth}(\Theta,\theta_0)\) the objects are methods \(M=(Q,q_0,\rho)\), and an arrow \(h:M\to N\) is a pointed smooth map \(h:Q_M\to Q_N\) with \(\rho_N\circ h=\rho_M\), read “\(N\) simulates \(M\)”. Simulations compose, since \(\rho_P\circ k\circ h=\rho_N\circ h=\rho_M\) for \(h:M\to N\) and \(k:N\to P\). For instance, \(h(B)=(B,A_0)\) is an arrow from LoRA-FA\((A_0)\) to LoRA pointed at \((0,A_0)\). The simulation arrows of the atlas are arrows of this category, each recorded with its map \(h\).

Used in: Notation, Def I.2, Exposé 0–I, VIII.C, Figure 15. Reading: Leinster, Basic Category Theory (2014), Ch. 1.

Definition A.2 (commutative diagram). A diagram is a family of objects joined by arrows, drawn as a directed graph. It commutes if any two directed paths with the same start and the same end compose to the same arrow. A triangle commutes when one side equals the composite of the other two, and a square commutes when its two routes between opposite corners agree.

A simulation is a commuting triangle over \(\Theta\), whose two paths from \(Q_M\) to \(\Theta\) are

\[Q_M\xrightarrow{\;h\;}Q_N\xrightarrow{\;\rho_N\;}\Theta\qquad\text{and}\qquad Q_M\xrightarrow{\;\rho_M\;}\Theta.\]

For \(M=\mathrm{FA}(A_0)\), \(N\) LoRA pointed at \((0,A_0)\) and \(h(B)=(B,A_0)\), both paths send \(B\) to the weight \(W_0+BA_0\) (scale absorbed into \(B\)), so the triangle commutes. Figure 1 draws, for each method, the triangle of Figure A.1, which commutes only up to a pointing defect when the method starts away from \(\theta_0\) (A.19). Commuting squares, used for merging on extensions, return in A.18.

Used in: Figure 1, Def I.2, Def VI.1, Def V.1, Exposé V. Reading: Leinster (2014), §1.1.

Definition A.3 (isomorphism). An arrow \(f:X\to Y\) is an isomorphism, written \(X\cong Y\), if some \(g:Y\to X\) satisfies \(g\circ f=\mathrm{id}_X\) and \(f\circ g=\mathrm{id}_Y\). The inverse \(g\) is then unique. In \(\mathbf{Diff}\) the isomorphisms are the diffeomorphisms, so a smooth bijection need not be one, as \(t\mapsto t^3\) shows. In \(\mathsf{Meth}\) an isomorphism \(M\cong N\) is a pointed diffeomorphism \(h:Q_M\to Q_N\) with \(\rho_N\circ h=\rho_M\). Its inverse lies over \(\Theta\) automatically, since \(\rho_M\circ h^{-1}=\rho_N\).

The LoRA gauge (D.21) \((B,A)\mapsto(Bg^{-1},gA)\), \(g\in GL_r\), is an isomorphism from LoRA pointed at \((B_0,A_0)\) to LoRA pointed at \((B_0g^{-1},gA_0)\). Other examples are rsLoRA \(\cong\) LoRA by \((B,A)\mapsto(\sqrt r\,B,A)\), and \(\mathrm{LoRA}_r\oplus\mathrm{LoRA}_s\cong\mathrm{LoRA}_{r+s}\) by concatenating factors (B.14). Isomorphic methods have the same image, the same \(S_1\) and the same first-order deficit (C.3, Obs I.6). They can still train differently, because the optimizer acts on the coordinates of \(Q\). For LoRA’s gauge with block-scalar rates (one rate per factor), gradient flow is unchanged exactly when \(g\in O(r)\), at points where \(A\) has full row rank and \(r\lt n\) (Thm III.3; D.23).

Used in: Obs I.6, Obs I.7, Three strata, Thm II.2, Prop VI.4, VIII.C.

Definition A.4 (functor; functorial invariant). A functor \(F:\mathcal C\to\mathcal D\) sends each object \(X\) to an object \(FX\) and each arrow \(f:X\to Y\) to an arrow \(Ff:FX\to FY\), with \(F(g\circ f)=Fg\circ Ff\) and \(F(\mathrm{id}_X)=\mathrm{id}_{FX}\). A functor sends isomorphisms to isomorphisms, so \(FX\) is an isomorphism invariant of \(X\). A forgetful functor discards structure, as \(\mathbf{Diff}_*\to\mathbf{Diff}\) discards the base point.

The image is a functor \(\mathrm{Im}\) from \(\mathsf{Meth}(\Theta,\theta_0)\) to the subsets of \(\Theta\) containing \(\theta_0\), ordered by inclusion (a poset, A.16), because an arrow \(h:M\to N\) gives \(\mathrm{Im}\,M=\rho_N(h(Q_M))\subseteq\mathrm{Im}\,N\). In the same way the chain rule at \(q_0\) makes \(S_1\) a functor to the linear subspaces of \(\Theta\), with \(S_1(M)\subseteq S_1(N)\) along every pointed arrow. Read backwards, functoriality gives the atlas test O1 and the first half of O3. If \(\mathrm{Im}\,M\not\subseteq\mathrm{Im}\,N\) or \(S_1(M)\not\subseteq S_1(N)\), then \(N\) does not simulate \(M\). Base change \(\phi_!\) (A.19) and transport \(T_*\) (A.17) are functors between slices.

Used in: Obs I.4(c), Obs I.6, VIII.C, Prop II.13, Def V.1, Backpropagation as a lens. Reading: Leinster (2014), §1.2.

Definition A.5 (natural transformation; naturality). For functors \(F,G:\mathcal C\to\mathcal D\), a natural transformation \(\alpha:F\Rightarrow G\) is a family of arrows \(\alpha_X:FX\to GX\) with \(Gf\circ\alpha_X=\alpha_Y\circ Ff\) for every \(f:X\to Y\). This naturality square says that applying \(\alpha\) and then moving along \(f\) agrees with moving first and applying \(\alpha\) after. A group \(K\) can be viewed as a category with one object whose arrows are the elements of \(K\). A functor from it to \(\mathbf{Set}\) is then a set with a \(K\)-action, and a natural transformation between two such functors is exactly a \(K\)-equivariant map.

The copy map \(\Delta_X(x)=(x,x)\) is natural, \(\Delta_Y\circ f=(f\times f)\circ\Delta_X\), which is rule R7 of the merge calculus (B.4). Elsewhere on the site, natural for a group of changes of chart means equivariant in this sense (D.18). Task arithmetic is natural for all of \(GL(\Theta)\), and DARE for invertible diagonal matrices at a fixed mask (Prop V.4(c)). Gradient flow on LoRA commutes with the gauge \(g\) exactly when the induced cometrics (D.23) agree, which for block-scalar rates singles out \(O(r)\) (Thm III.3).

Used in: Where the category theory works, Thm III.3, When a change of chart does not matter, Prop V.4, Seven local rules, Thm VI.3. Reading: Leinster (2014), §1.3.

Definition A.6 (slice category). For an object \(X\) of \(\mathcal C\), the slice \(\mathcal C/X\) has as objects the arrows \(a:A\to X\), called objects over \(X\), and as morphisms \((A,a)\to(B,b)\) the arrows \(h:A\to B\) of \(\mathcal C\) with \(b\circ h=a\). Composition and identities are those of \(\mathcal C\). The object \(X\) is the base, and \(a\) is the structure map.

\(\mathbf{Diff}/\Theta\) consists of smooth maps \(\rho:Q\to\Theta\) into weight space without base points, and its morphisms are the weak morphisms of Def I.2, which need not send \(q_0\) to \(q_0'\). A slice always has a terminal object, the identity \((X,\mathrm{id}_X)\), which here is full fine-tuning. Working in a slice is the relative point of view of A.19, in which an object is studied together with its map to a base.

Used in: Notation, Def I.2, Where the category theory works, Rosetta entries 3 and 4, Seven borrowed words. Reading: Mac Lane, Categories for the Working Mathematician (2nd ed., 1998), §II.6.

Definition A.7 (coslice; pointed objects; opposite category). The coslice \(X/\mathcal C\) has as objects the arrows \(x:X\to A\) out of \(X\), and as morphisms \((A,x)\to(B,x')\) the arrows \(h:A\to B\) with \(h\circ x=x'\). For the one-point space \(X=\mathrm{pt}\), an arrow \(\mathrm{pt}\to Q\) is a point \(q_0\), so \(\mathbf{Diff}_*:=\mathrm{pt}/\mathbf{Diff}\) is the category of pointed manifolds \((Q,q_0)\) and pointed maps, those with \(h(q_0)=q_0'\). Likewise \(\mathbf{Set}_*:=\mathrm{pt}/\mathbf{Set}\). The opposite category \(\mathcal C^{\rm op}\) has the same objects and every arrow reversed, and \((\mathcal C/X)^{\rm op}\cong X/\mathcal C^{\rm op}\).

A method sends its initialisation to the checkpoint, \(\rho(q_0)=\theta_0\), so it is an arrow of \(\mathbf{Diff}_*\), and a weak morphism forgets the base points and lives in \(\mathbf{Diff}\). Opposites enter in Remark I.3. A simulation \(h:M\to N\) runs from \(Q_M\) to \(Q_N\). The network with \(Q_M\)-parameters is obtained from the one with \(Q_N\)-parameters by precomposing with \(h\), \(f\circ(\rho_M\times\mathrm{id}_X)=f\circ(\rho_N\times\mathrm{id}_X)\circ(h\times\mathrm{id}_X)\), with \(X\) the input space. Read as a change of parameters for networks, \(h\) therefore points the other way, from the \(Q_N\)-network to the \(Q_M\)-network, and Remark I.3 records this reversal with an opposite category (B.16).

Used in: Def I.2, Remark I.3, Obs I.4, What Para adds.

Definition A.8 (the pointed slice \(\mathsf{Meth}(\Theta,\theta_0)\)). Combining A.6 and A.7, \(\mathsf{Meth}(\Theta,\theta_0):=\mathbf{Diff}_*/(\Theta,\theta_0)\). An object is a pointed smooth map \(\rho:(Q,q_0)\to(\Theta,\theta_0)\), and a morphism \(h:M\to N\) is a pointed smooth map with \(\rho_N\circ h=\rho_M\). Equivalently, an object is a factorisation \(\mathrm{pt}\xrightarrow{\,q_0\,}Q\xrightarrow{\,\rho\,}\Theta\) of the point \(\theta_0:\mathrm{pt}\to\Theta\), and a morphism is a map between the middle terms that is compatible with both ends.

On one box, LoRA is \(\big(\mathbb R^{m\times r}\times\mathbb R^{r\times n},\,(0,A_0),\,(B,A)\mapsto W_0+sBA\big)\), and OFT is \(Q\mapsto W_0\,C(Q)\) on skew matrices (block-diagonal in OFT), pointed at \(0\), with \(C\) the Cayley map. Full fine-tuning is \((\Theta,\theta_0,\mathrm{id})\) and the frozen model is \((\mathrm{pt},*,\theta_0)\). The network \(f\) appears nowhere in the definition, so the category depends only on \((\Theta,\theta_0)\) (Remark I.3). The factorisation picture already shows why the frozen model and full fine-tuning sit at the two ends (A.9).

Used in: Notation, Def I.2, Pretrained models and methods, Remark I.3, Rosetta entries 3 and 4.

Universal properties

Definition A.9 (initial and terminal objects; universal property). An object \(I\) is initial if every object \(X\) receives exactly one arrow \(I\to X\), and \(T\) is terminal if every \(X\) has exactly one arrow \(X\to T\). Each is unique up to a unique isomorphism when it exists. Characterising an object by the arrows into or out of it in this way is called a universal property. In \(\mathbf{Set}\) the empty set is initial and a one-point set is terminal, and in \(\mathbf{Set}_*\) and \(\mathbf{Diff}_*\) the one-point space is both.

In \(\mathsf{Meth}(\Theta,\theta_0)\) the frozen model \((\mathrm{pt},*,\theta_0)\) is initial. The only pointed map \(\mathrm{pt}\to Q\) is \(*\mapsto q_0\), and it lies over \(\Theta\) because \(\rho(q_0)=\theta_0\). Full fine-tuning \((\Theta,\theta_0,\mathrm{id})\) is terminal, since an arrow \(h\) from \(M\) must satisfy \(\mathrm{id}\circ h=\rho_M\). So the unique arrow \(M\to\) Full FT is \(\rho_M\) itself, the merge, and every weight-space method that starts at \(\theta_0\) sits in the commuting triangle of Figure A.1.

Used in: Obs I.4(a), Frozen and full fine-tuning, Figure 1, Rosetta entries 3 and 4, VIII.C (rule A1), What base change leaves for the network.

Definition A.10 (product). A product of \(X\) and \(Y\) is an object \(X\times Y\) with projections \(p_X:X\times Y\to X\) and \(p_Y:X\times Y\to Y\) such that for every object \(T\) and arrows \(u:T\to X\), \(v:T\to Y\) there is exactly one arrow \(\langle u,v\rangle:T\to X\times Y\) with \(p_X\circ\langle u,v\rangle=u\) and \(p_Y\circ\langle u,v\rangle=v\). It is unique up to unique isomorphism, and it need not exist. In a slice \(\mathcal C/Z\), the product of \((X,a)\) and \((Y,b)\) is the pullback of \(a\) and \(b\) in \(\mathcal C\) (A.11).

In \(\mathsf{Meth}\) a product \(M\times N\) has arrows to \(M\) and to \(N\), so both factors simulate it, and every method \(T\) with arrows \(u:T\to M\) and \(v:T\to N\) maps to it by exactly one arrow compatible with \(u\) and \(v\). When it exists, its image is \(\mathrm{Im}\,M\cap\mathrm{Im}\,N\). For the one-sided LoRAs with frozen frames \(B_0\) of full column rank and \(A_0\) of full row rank,

\[\mathrm{FA}(A_0)\times\mathrm{FB}(B_0)\;\cong\;\big(\mathbb R^{r\times r},\,0,\,R\mapsto W_0+B_0RA_0\big),\]

with projections \(R\mapsto B_0R\) and \(R\mapsto RA_0\). In the SVD frames of \(W_0\), when \(\operatorname{rank}W_0\ge r\), this is LoRA-XS (Prop I.8(a)). The sum \(\oplus\) of Obs I.7 is a different operation, whose image is a Minkowski sum (B.14).

Used in: Obs I.4(b), Prop I.8, Products, Prop II.3, VIII.C (rule A4), Reading from the right. Reading: Leinster (2014), §5.1.

Definition A.11 (pullback; fibre product). Given \(f:X\to Z\) and \(g:Y\to Z\), a pullback is an object \(P\) with arrows \(p:P\to X\) and \(q:P\to Y\) such that \(f\circ p=g\circ q\), and such that whenever \(u:T\to X\) and \(v:T\to Y\) satisfy \(f\circ u=g\circ v\), exactly one \(t:T\to P\) has \(p\circ t=u\) and \(q\circ t=v\) (Figure A.2). In \(\mathbf{Set}\) the pullback is the fibre product \(X\times_ZY=\{(x,y):f(x)=g(y)\}\). In \(\mathbf{Diff}\), if this set is an embedded submanifold of \(X\times Y\), it is the pullback, and transversality (C.7) is the standard sufficient condition for that. A pullback in \(\mathbf{Diff}\) can fail to exist.

Products in \(\mathsf{Meth}\) are fibre products \(Q_M\times_\Theta Q_N=\{(q,q'):\rho_M(q)=\rho_N(q')\}\). For \(\mathrm{FA}(A_0)\) and \(\mathrm{FB}(B_0)\), with the ranks of A.10, this set is the linear space \(\{(B_0R,RA_0)\}\), so the product exists. For \(\mathrm{LoRA}_1\) and the frozen model on \(1\times1\) weights the set is the cross \(\{ba=0\}\subset\mathbb R^2\), no manifold structure on it has the universal property, and the product does not exist (A.13). Pulling back along the point \(\theta_0:\mathrm{pt}\to\Theta\) gives the fibre \(\rho^{-1}(\theta_0)\), the underlying set of \(M\times\text{Frozen}\) whenever that product exists.

More: HRA is a meet of images, not a product

Take \(W_0\) injective and \(r\) even with \(2\le r\le n-2\). The fibre product of \(\mathrm{LoRA}_r\) with the rotation orbit \(W_0\cdot SO(n)\) has local dimension \(d(\mathrm{HRA}_r)+r^2=rn+r(r-1)/2\) over generic points of the meet (C.19). This exceeds \(rn=\lvert\mathrm{HRA}_r\rvert\) because \(r\ge2\), so HRA is not the product of the two methods, and Thm II.7 keeps only the meet of their images (A.16).

Used in: Obs I.4(b), Prop I.8, Prop II.3, Thm II.7(d), Cor III.13.

Definition A.12 (limit; creation of limits). A cone over a diagram \(D\) in \(\mathcal C\) is an object \(L\) with an arrow to every object of \(D\), such that each triangle formed with an arrow of \(D\) commutes. A limit of \(D\) is a cone through which every other cone factors by exactly one arrow. This categorical cone is unrelated to the geometric cones of C.10. Terminal objects, products and pullbacks are the limits of the empty diagram, of two objects, and of two arrows with a common target. A functor \(U:\mathcal C\to\mathcal D\) creates limits if, for every diagram \(D\) in \(\mathcal C\) and every limit cone of \(U\circ D\), exactly one cone over \(D\) is sent by \(U\) to that limit cone, and this cone is a limit of \(D\).

The forgetful functor \(\mathbf{Diff}_*\to\mathbf{Diff}\) creates limits (A.14). Concretely, when the underlying manifolds have a pullback, the pullback of pointed manifolds is that pullback, pointed at the pair of base points. This is how Obs I.4(b) computes products of methods as fibre products in \(\mathbf{Diff}\).

Used in: Obs I.4(b). Reading: Mac Lane (1998), Ch. V; Riehl, Category Theory in Context (2016), Ch. 3.

Definition A.13 (adjunction). A functor \(F:\mathcal C\to\mathcal D\) is left adjoint to \(G:\mathcal D\to\mathcal C\), and \(G\) is right adjoint to \(F\), written \(F\dashv G\), if there are bijections \(\mathcal D(FX,Y)\cong\mathcal C(X,GY)\) natural in \(X\) and \(Y\). Many forgetful functors are right adjoints, whose left adjoints add the forgotten structure freely.

The forgetful functor \(\mathbf{Diff}_*\to\mathbf{Diff}\) is right adjoint to \(X\mapsto(X\sqcup\mathrm{pt},\mathrm{pt})\), which adds a disjoint base point, since a pointed map out of \(X\sqcup\mathrm{pt}\) is the same as an arbitrary map out of \(X\) (a manifold may have components of different dimensions). Right adjoints preserve limits (A.14), so a product of methods, if it exists, is also a pullback in \(\mathbf{Diff}\) and can be tested with unpointed maps. Obs I.4(b) uses this to rule out \(\mathrm{LoRA}_1\times\text{Frozen}\) on \(1\times1\) weights. The axis curves \(t\mapsto(t,0)\) and \(t\mapsto(0,t)\) in the cross would lift smoothly through the point over the origin, so the differential of the projection from the product to the parameter plane \(\mathbb R^2\) of \(\mathrm{LoRA}_1\) would have rank 2 there. That projection would then be a submersion near the point (C.4), and its image would contain an open set, which the cross does not.

Used in: Obs I.4(b). Reading: Leinster (2014), Ch. 2 and Ch. 6.

Fact A.14 (two standard facts about limits). Let \(\mathcal C\) be a category.

(a) For every object \(X\) of \(\mathcal C\), the forgetful functor \(X/\mathcal C\to\mathcal C\) creates limits. In particular \(\mathbf{Diff}_*\to\mathbf{Diff}\) does.

(b) A right adjoint preserves every limit that exists in its domain.

More: the idea behind (a) and (b)

(a) The structure arrows \(X\to D_j\) of a diagram in \(X/\mathcal C\) form a cone in \(\mathcal C\), which factors through the limit by exactly one arrow, and that arrow is the structure map of the limit. (b) \(\mathcal C(X,G\lim D)\cong\mathcal D(FX,\lim D)\cong\lim\mathcal D(FX,D_j)\cong\lim\mathcal C(X,GD_j)\), naturally in \(X\).

Obs I.4(b) combines them. Products in \(\mathsf{Meth}\) are pullbacks in \(\mathbf{Diff}_*\), which by (a) exist whenever the underlying pullback exists in \(\mathbf{Diff}\), and by (b) and A.13 only then. The transversality hypothesis of C.7 fails at the base point whenever \(\lvert M\rvert+\lvert N\rvert\lt\dim\Theta\), with \(\lvert M\rvert=\dim Q_M\) (C.3), so the products that exist, such as LoRA-XS, come from other structure (A.11).

Used in: Obs I.4(b), Frozen and full fine-tuning. Reading: Mac Lane (1998), Ch. V; Riehl (2016), Ch. 3 and 4.

Definition A.15 (monomorphism; sub-object). An arrow \(m:A\to B\) is a monomorphism if \(m\circ u=m\circ v\) implies \(u=v\) for all arrows \(u,v:T\to A\). A sub-object of \(B\) is a monomorphism into \(B\), two of them being identified when they differ by an isomorphism over \(B\). Testing with maps out of one- and two-point manifolds shows that the monomorphisms of \(\mathbf{Diff}\), and those of \(\mathsf{Meth}\), are exactly the arrows whose underlying map is injective. Such a map need not be an embedding, as \(t\mapsto t^3\) shows.

Freezing coordinates at their base values gives an injective affine map, hence a sub-object, written \(\hookrightarrow\) in the atlas (rule A2). Examples are \(\mathrm{FA}(A_0)\hookrightarrow\mathrm{LoRA}\) pointed at \((0,A_0)\), by \(B\mapsto(B,A_0)\), and MiCA \(\hookrightarrow\) LoRA pointed at \((B_0,0)\), by \(A\mapsto(B_0,A)\). These maps are embeddings, a stronger property than injectivity. The site writes sub-object and never monomorphism. In Cor III.13, LoRA-FA is a sub-object of LoRA and so a different object, while random-\(A\) LoRA and EVA are the same map \(\rho\) with two different base points in its fibre over \(\theta_0\).

Used in: Def I.2, VIII.C (rule A2), Cor III.13.

Definition A.16 (posets; meets; the expressivity order). A preorder is a category with at most one arrow between any two objects; write \(x\le y\) when there is one. It is a poset if \(x\le y\le x\) forces \(x=y\). In a poset the product of \(x\) and \(y\) is their meet \(x\wedge y\), the greatest lower bound, and for subsets ordered by inclusion the meet is the intersection. Two elements are incomparable if neither \(x\le y\) nor \(y\le x\), and \(x\) is maximal if \(x\le y\) implies \(x=y\). Every category has a poset reflection, in which \(X\le Y\) when some arrow \(X\to Y\) exists and \(X,Y\) are identified when arrows run both ways. A Hasse diagram draws a finite poset with \(y\) above \(x\) when \(x\lt y\), joining them by an edge when nothing lies strictly between.

The poset reflection of \(\mathsf{Meth}\), which we call the expressivity order, orders methods by simulation. Figure 15 draws it for the core methods as a layered Hasse diagram, with mutually simulating families on one rung. A meet of images always exists and need not be the image of a product. For injective \(W_0\) and even \(r\le n-2\), the image of HRA without Gram–Schmidt is the meet of LoRA’s cone with the rotation orbit \(W_0\cdot SO(n)\) (D.11), and HRA is not their product (Thm II.7). Split and zero-init LoRA have incomparable images (Thm II.2(d)), so neither simulates the other. The frontier problem asks for the maximal methods of budget at most \(k\), which are automatically pairwise incomparable.

Used in: Obs I.4(c), Prop I.8(b), Thm II.2, Thm II.7, HRA is a meet of two images, Figure 15, Open problems. Reading: Davey and Priestley, Introduction to Lattices and Order (2nd ed., 2002), Ch. 1 and 2.

Base change and lifting

Definition A.17 (automorphism of a category; transport of structure). An automorphism of a category \(\mathcal C\) is a functor \(F:\mathcal C\to\mathcal C\) with an inverse functor, so that both composites are identity functors. It carries initial objects, products, isomorphisms and every other notion defined from arrows alone to notions of the same kind. Transport of structure builds such a functor from an invertible map of the underlying data.

For \(T\in GL(\Theta)\) put \(\tau_T(\theta)=\theta_0+T(\theta-\theta_0)\) and \(T_*M:=(Q,q_0,\tau_T\circ\rho)\). An arrow \(h:M\to N\) is also an arrow \(T_*M\to T_*N\), since \(\tau_T\circ\rho_N\circ h=\tau_T\circ\rho_M\), so \(T_*\) is a functor, with inverse \((T^{-1})_*\). It keeps \(d(M)\) and the first-order deficit (C.3) and the gauge group (D.21), and moves \(S_1\) to \(TS_1\). It does not keep rank, which depends on the matrix shape of \(\Theta\) that \(T\) need not respect. Sums of \(k\) Kronecker products are \(\mathcal R^{-1}_*\mathrm{LoRA}_k\), LoRA on the matrix rearranged by Van Loan and Pitsianis and carried back (B.11), and HiRA is \((D_{W_0})_*\mathrm{LoRA}\) with \(D_{W_0}(X)=W_0\odot X\), when \(W_0\) has no zero entries. The atlas records \(M\) and \(T_*M\) as one object in two frames, with no arrow between them.

Used in: Prop II.13, Exposé II.C, VIII.C (rule T), Reading from the right.

Definition A.18 (lift; lifting problem; obstruction). Given arrows \(p:E\to B\) and \(f:A\to B\), a lift of \(f\) along \(p\) is an arrow \(g:A\to E\) with \(p\circ g=f\). A lifting problem asks whether a lift exists, and an obstruction is an invariant whose value shows that none does. A morphism \(h:M\to N\) of \(\mathsf{Meth}\) is exactly a pointed smooth lift of \(\rho_M\) along \(\rho_N\).

In \(\mathbf{Set}\) a lift exists once \(f(A)\subseteq p(E)\), by choosing preimages, so smoothness is the real constraint. For \(\rho_M(t)=t\) and \(\rho_N(u)=u^3\) on \(\mathbb R\) the images agree, and the only lift is the cube root, which is not differentiable at \(0\). A pointed smooth lift forces \(S_1(M)\subseteq S_1(N)\) by the chain rule, so \(S_1\) is a first-order obstruction, and here \(S_1(M)=\mathbb R\not\subseteq S_1(N)=0\). For \(t^3\) against \(u^5\) both \(S_1\) vanish, yet the only lift \(t^{3/5}\) is not \(C^1\), so that obstruction is invisible to \(S_1\). Merging a method on an extension is again a lifting problem, to find \(\mu\) with \(\Phi_f\circ\mu=\Phi_{\tilde f}\circ\rho\), a lift along the realisation \(\Phi_f\) (B.16, Def VI.1).

Used in: Obs I.4(c), Frozen and full fine-tuning, Obs I.6, Def VI.1, Extensions, and merging as a lift, Rosetta entry 5.

Definition A.19 (base change; pushforward \(\phi_!\); the relative point of view). A map of bases \(\phi:S'\to S\) induces the pushforward \(\phi_!:\mathcal C/S'\to\mathcal C/S\), \((X,x)\mapsto(X,\phi\circ x)\), by postcomposition. When every object over \(S\) has a pullback along \(\phi\), it also induces the pullback \(\phi^*:\mathcal C/S\to\mathcal C/S'\), \((Y,y)\mapsto(Y\times_SS',\text{projection})\), and then \(\phi_!\dashv\phi^*\). In algebraic geometry, base change usually means \(\phi^*\).

Def V.1(a) uses \(\phi_!\) on pointed slices. A function-preserving pointed reparametrisation \(\phi:(\Theta',\theta_0')\to(\Theta,\theta_0)\), with \(f\circ(\phi\times X)=f'\), gives \(\phi_!:\mathsf{Meth}(\Theta',\theta_0')\to\mathsf{Meth}(\Theta,\theta_0)\), \(\rho\mapsto\phi\circ\rho\), for instance for a permutation of hidden units. Whether \(\phi_!\) carries LoRA to LoRA is the covariance question of Obs II.15. A quantiser changes the function and induces no such functor, so QLoRA and ReLoRA only borrow the name, and Exposé V computes pointing defects and reachable sets for them.

Grothendieck’s relative point of view replaces a space \(X\) by a morphism \(X\to S\), an object over a base. Properties then belong to the morphism, and the useful ones are those that survive pulling back along every change of base \(S'\to S\). The fibre over a point \(s\) is itself the pullback along \(s:\mathrm{pt}\to S\). The site takes this stance in a modest form. A method is a morphism over the base point \(\theta_0\), and quantisation, restarts and merging are read as moves of that base point.

Used in: Exposé 0–I, Where the category theory works, Def V.1, Exposé V, Rosetta entry 28, Seven borrowed words. Reading: Vakil, The Rising Sea: Foundations of Algebraic Geometry, on fibred products and base change.

Remark A.20 (constructions the site names and leaves unused). Coends, monads, sheaves, stacks, schemes, Grothendieck’s descent theory (descent data, effective descent morphisms), the Grothendieck construction \(\int\mathsf{Meth}\to\mathbf{PNet}\) and coKleisli 2-cells appear on the site only in lists of constructions it leaves out (Where the category theory works, What the borrowed words mean). No result depends on them. The elementary descent of D.22, factoring a map that is constant on orbits through the orbit space, is a different and much simpler notion, and several results use it.

For orientation only, the Grothendieck construction would assemble the slices \(\mathsf{Meth}(\Theta,\theta_0)\) of all pretrained models into one category of pairs (network, method), with a projection to the category \(\mathbf{PNet}\) of pointed networks and function-preserving reparametrisations. A monad is an endofunctor \(T\) with natural transformations \(\mathrm{id}\Rightarrow T\) and \(TT\Rightarrow T\) satisfying unit and associativity laws, and a sheaf assigns data to open sets so that compatible local data glue to unique global data.

Used in: Where the category theory works, Seven borrowed words, What the borrowed words mean.

B. Monoidal categories, string diagrams and lenses

17 entries

Part A compared methods through the arrows between them. Part B covers the algebra behind drawing a method as boxes and wires, reading a rank bound off a cut of the drawing, and treating backpropagation as a pair of maps. Entries B.1 to B.6 set up monoidal categories and their diagrams, B.7 to B.13 turn them into rank and dimension counts for matrices built from small factors, and B.14 to B.17 cover sums of methods, parametrised maps and lenses.

Monoidal categories

Definition B.1 (monoidal and symmetric monoidal categories). A monoidal category is a category \(\mathcal C\) with a functor \(\otimes:\mathcal C\times\mathcal C\to\mathcal C\), a unit object \(I\), and natural isomorphisms \((A\otimes B)\otimes C\cong A\otimes(B\otimes C)\) and \(I\otimes A\cong A\cong A\otimes I\), the coherence isomorphisms, which satisfy the pentagon and triangle equations. Mac Lane’s coherence theorem then makes every diagram built from them commute, so brackets can be dropped. The category is symmetric if it also has natural isomorphisms \(\sigma_{A,B}:A\otimes B\to B\otimes A\) with \(\sigma_{B,A}\sigma_{A,B}=\mathrm{id}\) that are compatible with the associator (the hexagon equation). Examples are \((\mathbf{FinVect},\otimes,\mathbb R)\), \((\mathbf{FinVect},\oplus,0)\) and \((\mathbf{Diff},\times,\mathrm{pt})\).

On methods, \(M\oplus N\) runs the two parameter spaces side by side and adds their updates (B.14). This makes \(\mathsf{Meth}(\Theta,\theta_0)\) symmetric monoidal with the frozen model as unit. Its coherence isomorphisms are those of \(\times\) on pointed manifolds, and they lie over \(\Theta\) because addition is associative and commutative.

Used in: Obs I.7, Sums, Where the category theory works, Thm II.10, Thm VI.3. Reading: Mac Lane (1998), Ch. VII and XI.

Definition B.2 (direct sum and tensor product in FinVect). \(\mathbf{FinVect}\) has finite-dimensional real vector spaces as objects and linear maps as morphisms, and two monoidal products. The direct sum \(V\oplus W\) is a biproduct, both the categorical product and the coproduct, with \(\dim(V\oplus W)=\dim V+\dim W\); on matrices \(f\oplus g\) is block-diagonal. The tensor product \(V\otimes W\) has \(\dim(V\otimes W)=\dim V\cdot\dim W\) and is not a categorical product; on matrices \(f\otimes g\) is the Kronecker product (B.10). Each bilinear map \(V\times W\to U\) factors through a unique linear map \(V\otimes W\to U\).

Two boxes drawn side by side therefore mean a block-diagonal map in the activation diagrams of Exposé VI, whose monoidal product is \(\times=\oplus\), and a Kronecker product in a tensor network. Dimensions multiply along \(\otimes\), which is why the weight of a cut in Thm II.10 is a product of wire dimensions. LoRA’s residual branch \(W_0+BA\) uses \(\oplus\), since it sends one input to two branches and adds their outputs (Figure B.1).

Used in: Thm II.10, Exposé II.C, Thm VI.3, Obs I.7.

Definition B.3 (string diagrams). A string diagram draws a morphism \(f:A\to B\) of a monoidal category as a box with an input wire \(A\) and an output wire \(B\). Composition \(g\circ f\) joins boxes in series, \(f\otimes g\) places them side by side, an identity is a bare wire, the unit \(I\) is drawn as no wire, and in a symmetric category \(\sigma\) is a crossing. Joyal and Street proved that the pictures are sound and complete. In a monoidal category, diagrams related by a planar deformation denote the same morphism; in a symmetric one, diagrams with the same boxes joined in the same way do. Laws such as \((g\otimes g')\circ(f\otimes f')=(g\circ f)\otimes(g'\circ f')\) can then be read off the picture.

The plug of Exposé II.C is a string diagram. LoRA is a box \(A:\mathbb R^n\to\mathbb R^r\) followed by \(B:\mathbb R^r\to\mathbb R^m\), drawn beside \(W_0\) (Figure B.1, read left to right). The merge calculus moves an edit along wires by equalities of maps, so a successful slide proves that the merged network computes the same function.

Used in: Exposé II.C, Thm II.10, Figure 6, Seven local rules, Analogy in Exposé VI, Thm VI.3. Reading: Selinger, “A survey of graphical languages for monoidal categories” (2011).

Definition B.4 (cartesian categories; copy and discard). A monoidal category is cartesian when \(\otimes\) is the categorical product \(\times\) and \(I\) is a terminal object \(1\). Every object then has a copy map \(\Delta_X:X\to X\times X\), \(x\mapsto(x,x)\), and a discard map \({!}_X:X\to1\), and copy is natural, \((f\times f)\circ\Delta_X=\Delta_Y\circ f\) for every \(f:X\to Y\). Conversely, by a theorem of Fox (1976), a symmetric monoidal category is cartesian exactly when every object carries a commutative comonoid, a copy and a discard map obeying coassociativity, counit and commutativity laws (B.6), compatible with \(\otimes\) and natural in all morphisms. \(\mathbf{Set}\), \(\mathbf{Diff}\) and \((\mathbf{FinVect},\oplus)\) are cartesian. \((\mathbf{FinVect},\otimes)\) is not, since \(x\mapsto x\otimes x\) is not linear.

Naturality of copy is rule R7 of the merge calculus. A linear edit \(E\) on a wire read by \(W_Q\), \(W_K\) and \(W_V\) equals \(E\) applied on each of the three branches, so it is absorbed as \(W_QE\), \(W_KE\), \(W_VE\). The site builds \(\mathbf{Para}(\mathcal C)\) (B.16) over a cartesian \(\mathcal C\). The linear copy of B.6 copies basis vectors only and is a different map.

Used in: Def I.1, Seven local rules, Analogy in Exposé VI, Thm VI.3.

Definition B.5 (compact closed categories; cups, caps and wire-bending). A symmetric monoidal category is compact closed if every object \(A\) has a dual \(A^*\) with a cup \(\eta_A:I\to A^*\otimes A\) and a cap \(\varepsilon_A:A\otimes A^*\to I\) satisfying the snake equations \((\varepsilon_A\otimes\mathrm{id}_A)(\mathrm{id}_A\otimes\eta_A)=\mathrm{id}_A\) and \((\mathrm{id}_{A^*}\otimes\varepsilon_A)(\eta_A\otimes\mathrm{id}_{A^*})=\mathrm{id}_{A^*}\), with the unitors \(I\otimes A\cong A\cong A\otimes I\) of B.1 suppressed. Drawn as bent wires, cups and caps turn an input wire into an output wire and back. In \(\mathbf{FinVect}\) the cap is evaluation, the cup is \(1\mapsto\sum_ie_i^*\otimes e_i\), and bending gives \(\mathrm{Hom}(V,U)\cong U\otimes V^*\). With a fixed basis, a box slid around a cup or cap comes out as its transpose.

Bending the input wire of an \(m\times n\) matrix gives its vectorisation in \(\mathbb R^m\otimes\mathbb R^n\), and the Van Loan–Pitsianis rearrangement (B.11) is a composite of bends and crossings. Rule R5 of the merge calculus, \(\langle Mq,k\rangle=\langle q,M^\top k\rangle\), slides \(M\) around the cap that computes an attention score. The cut-rank bound uses only \(\otimes\) and crossings.

Used in: Thm II.11, Thm VI.3, Exposé II.C. Reading: Coecke and Kissinger, Picturing Quantum Processes (2017).

Definition B.6 (Frobenius algebras and spiders). A Frobenius algebra on an object \(V\) is a monoid \(\mu:V\otimes V\to V\), \(\eta:I\to V\) and a comonoid \(\delta:V\to V\otimes V\), \(\varepsilon:V\to I\) obeying the Frobenius law \((\mu\otimes\mathrm{id})(\mathrm{id}\otimes\delta)=\delta\mu=(\mathrm{id}\otimes\mu)(\delta\otimes\mathrm{id})\). A basis \(e_1,\dots,e_n\) of \(\mathbb R^n\) gives one, with copy \(\delta(e_i)=e_i\otimes e_i\), merge (the multiplication; unrelated to merging an adapter into the weights) \(\mu(e_i\otimes e_j)=e_i\) if \(i=j\) and \(0\) otherwise, \(\eta=\sum_ie_i\) and \(\varepsilon(e_i)=1\). It is commutative and special (\(\mu\delta=\mathrm{id}\)). For such algebras the spider theorem says that every connected diagram of \(\mu,\eta,\delta,\varepsilon\) with \(k\ge1\) inputs and \(l\) outputs equals one spider, which sends \(e_i^{\otimes k}\) to \(e_i^{\otimes l}\) and every other basis tensor to \(0\).

Merge multiplies coordinates, \(\mu(x\otimes y)=x\odot y\). So for matrices \(X,Y:\mathbb R^n\to\mathbb R^m\) the Hadamard product is the spider composite \(X\odot Y=\mu_m\circ(X\otimes Y)\circ\delta_n\). LoHa is a copy spider on the input feeding \(A_1\) and \(A_2\), then \(B_1\) and \(B_2\), then a merge spider on the output, and in Thm II.10 a spider split by the cut counts its dimension once. The copy and \(\odot\) nodes of Exposé VI live in a cartesian category (B.4), where copy is \(x\mapsto(x,x)\); together they fail the Frobenius law, so there “spider” is only a name.

Used in: Exposé II.C, Thm II.10, Prop II.12, Analogy in Exposé VI. Reading: Coecke, Pavlovic and Vicary (2008).

Tensor networks and rank

Definition B.7 (tensor networks, contraction, bond dimension). A tensor network \(\Gamma\) is a string diagram in \((\mathbf{FinVect},\otimes)\), possibly with spiders. Each box is a tensor and each wire a finite-dimensional space. A wire joining two boxes means contraction, a sum over the index the wire carries. The open wires (legs) are the inputs \(V\) and outputs \(U\), and contracting every internal wire gives the value \([\![\Gamma]\!]:V\to U\). The dimension of an internal wire is its bond dimension. One value has many presentations, and quantities read off a network, such as its cuts, depend on the presentation.

LoRA’s update is the two-box network \(B\circ A\) with one bond of dimension \(r\), and \(\Delta W=BA\) contracts it, \((BA)_{ij}=\sum_{k\le r}B_{ik}A_{kj}\). HiRA factors \(W_0\) through a bond of dimension \(\operatorname{rk}W_0\), while MoRA, C3A and MPO adapters have no narrow cut between inputs and outputs (the table after Thm II.10).

Used in: Thm II.10, Exposé II.C, Figure 6, Where the category theory works, Known results. Reading: Orús, “A practical introduction to tensor networks” (2014).

Fact B.8 (the min-cut bound on rank). Let \(\Gamma\) be a tensor network with input legs \(V\) and output legs \(U\), with every wire attached to a box (put an identity box on any bare strand). Split the boxes into an input side \(\mathcal I\) and an output side \(\mathcal O\). The weight \(w\) of the cut is the product of the dimensions of the wires joining the two sides and of the legs attached on the wrong side (input legs at \(\mathcal O\), output legs at \(\mathcal I\)); a spider with legs on both sides contributes its dimension once. Then \(\operatorname{rank}[\![\Gamma]\!]\le\min_{\text{cuts}}w\). For sums of networks, \(\operatorname{rank}(X+Y)\le\operatorname{rank}X+\operatorname{rank}Y\).

More: the idea of the proof

Contract each side separately. The value becomes a map into the tensor product of the crossing wires and wrong-side legs followed by a map out of it, and that space has dimension \(w\).

For LoRA the cut between \(A\) and \(B\) crosses one wire, so \(\operatorname{rank}BA\le r\) (Figure B.1). For LoHa the cut with \(A_1,A_2\) on the input side crosses two bonds and gives \(r_1r_2\). The bound depends on the presentation. HRA’s product of \(r\) reflections expands into \(2^r-1\) terms of rank at most one, while the telescoping sum \(R-I=\sum_k(H_k-I)H_{k+1}\cdots H_r\) gives \(r\).

Used in: Thm II.10, Exposé II.C, Figure 6, Prop II.12, Known results. Reading: Kolda and Bader, “Tensor decompositions and applications” (2009).

Definition B.9 (tensor trains, Tucker and MPO; gauge on bonds). A tensor train (TT) writes a tensor with legs of dimensions \(n_1,\dots,n_d\) as a chain of cores \(G_k\in\mathbb R^{r_{k-1}\times n_k\times r_k}\), \(r_0=r_d=1\), joined by bonds of dimensions \(r_1,\dots,r_{d-1}\). An MPO is the same chain with one input and one output leg on each core. Tucker joins a core \(C\in\mathbb R^{r_1\times\cdots\times r_d}\) to factor matrices \(U_k\in\mathbb R^{n_k\times r_k}\). Inserting \(g^{-1}g=\mathrm{id}\), \(g\in GL_{r_e}\), on a bond \(e\) and absorbing each factor into its neighbouring box leaves the value unchanged, so the gauge (D.21) contains \(\prod_eGL_{r_e}\). When every \(r_e\) is the rank of the matching unfolding (minimal ranks), the tensors of these ranks form a manifold of dimension \(\sum(\text{core sizes})-\sum_er_e^2\).

LoRA is the two-core case, \(B\) and \(A\) joined by one bond, with gauge \((B,A)\mapsto(Bg^{-1},gA)\) and, for \(r\le\min(m,n)\), image dimension \(r(m+n)-r^2\). TT and Tucker adapters have \(d(M)=\lvert M\rvert-\sum_er_e^2\) at minimal ranks (Prop II.12), and under gradient flow each bond carries a conserved matrix (Isometries and charges).

Used in: Prop II.12, Exposé II.C, Isometries and charges. Reading: Oseledets (2011); Holtz, Rohwedder and Schneider (2012); Koch and Lubich (2010).

Definition B.10 (Kronecker product, vec, tensor product of subspaces). For \(C\in\mathbb R^{m_1\times n_1}\) and \(D\in\mathbb R^{m_2\times n_2}\), the Kronecker product \(C\otimes D\in\mathbb R^{m_1m_2\times n_1n_2}\) is the block matrix \([c_{ij}D]\), the matrix of \(C\otimes D\) in the product basis. It obeys the mixed-product rule \((C\otimes D)(E\otimes F)=CE\otimes DF\), and \(\operatorname{rank}(C\otimes D)=\operatorname{rank}C\cdot\operatorname{rank}D\). The map \(u\otimes v\mapsto uv^\top\) identifies \(\mathbb R^m\otimes\mathbb R^n\) with \(\mathbb R^{m\times n}\). For subspaces \(U\subseteq\mathbb R^m\) and \(V\subseteq\mathbb R^n\), \(U\otimes V\) is then the space of matrices with column space in \(U\) and row space in \(V\), of dimension \(\dim U\cdot\dim V\).

LoKr’s update is \(C\otimes BA\) with \(C\) ranging over all \(m_1\times n_1\) matrices, \(B\in\mathbb R^{m_2\times r'}\) and \(A\in\mathbb R^{r'\times n_2}\). For inner rank \(r'\le\min(m_2,n_2)\) its rank \(\operatorname{rk}C\cdot\operatorname{rk}(BA)\) is at most \(\min(m_1,n_1)\,r'\), and this is attained. So \(\mathrm{Im}\,\mathrm{LoKr}\) lies inside the image \(W_0+\mathcal M_{\le r}\) of zero-init LoRA\(_r\) exactly when \(r\ge\min(m_1,n_1)\,r'\) (Thm II.11(e)). At a LoRA point \((B,A)\), the directions reached by moving one factor alone overlap in \(\{XA\}\cap\{BY\}=\mathrm{col}\,B\otimes\mathrm{row}\,A\), of dimension \(\operatorname{rk}A\cdot\operatorname{rk}B\) (Prop II.3).

Used in: Thm II.11, Exposé II.C, Prop II.3, Thm II.2, LoHa / LoKr in the atlas. Reading: Horn and Johnson, Topics in Matrix Analysis (1991), Ch. 4.

Fact B.11 (Van Loan–Pitsianis rearrangement; Kronecker rank). Let \(m=m_1m_2\) and \(n=n_1n_2\), and cut \(\Delta\in\mathbb R^{m\times n}\) into an \(m_1\times n_1\) grid of \(m_2\times n_2\) blocks \(\Delta_{ij}\). The rearrangement \(\mathcal R(\Delta)\in\mathbb R^{m_1n_1\times m_2n_2}\) has the rows \(\mathrm{vec}(\Delta_{ij})^\top\), ordered with \(i\) running fastest. Then \(\mathcal R(C\otimes D)=\mathrm{vec}\,C\,\mathrm{vec}\,D^\top\), and \(\mathcal R\) is a linear bijection that permutes entries, hence a Frobenius isometry. So the Kronecker rank \(\operatorname{rk}_\otimes\Delta\), the least \(k\) with \(\Delta=\sum_{i\le k}C_i\otimes D_i\), equals \(\operatorname{rank}\mathcal R(\Delta)\), and the nearest sum of \(k\) Kronecker products in Frobenius norm comes from the rank-\(k\) truncated SVD of \(\mathcal R(\Delta)\) (Van Loan and Pitsianis, 1993).

More: the idea of the proof

Block \((i,j)\) of \(C\otimes D\) is \(c_{ij}D\), so its row in \(\mathcal R\) is \(c_{ij}\,\mathrm{vec}(D)^\top\). Linearity extends this to sums, and Eckart–Young applies to \(\mathcal R(\Delta)\) because \(\mathcal R\) preserves the Frobenius norm.

In string-diagram terms \(\mathcal R\) bends and crosses wires, regrouping the four legs of \(\Delta\) from \(U_1U_2\mid V_1V_2\) to \(U_1V_1\mid U_2V_2\). A sum of \(k\) Kronecker products is therefore LoRA\(_k\) transported by \(\mathcal R^{-1}\) (Prop II.13), with rank taken across another cut. For \(N=s^2\), \(I_N=I_s\otimes I_s\) has Kronecker rank \(1\) and rank \(N\). Figure 5 animates \(\mathcal R\). On this site a Kronecker sum means a sum of Kronecker products; matrix analysis uses the same name for \(C\otimes I+I\otimes D\).

Used in: Thm II.11, Prop II.13, Figure 5, Prediction X.5, Known results, Key references. Reading: Van Loan, “The ubiquitous Kronecker product” (2000).

Definition B.12 (Schmidt and operator-Schmidt decompositions). Every \(u\in\mathbb R^{m_1}\otimes\mathbb R^{m_2}\) has a Schmidt decomposition \(u=\sum_{i\le\rho}\sigma_ix_i\otimes y_i\) with \(\sigma_i\gt0\) and orthonormal families \((x_i)\), \((y_i)\). It is the SVD of the \(m_1\times m_2\) reshape of \(u\) (\(x\otimes y\mapsto xy^\top\)), and the Schmidt rank \(\rho_u\) is the rank of that reshape, at most \(\min(m_1,m_2)\). The SVD of \(\mathcal R(\Delta)\) likewise gives the operator-Schmidt decomposition \(\Delta=\sum_{k\le K}s_kC_k\otimes D_k\), with \(s_k\gt0\), Frobenius-orthonormal families \((C_k)\) and \((D_k)\), and \(K=\operatorname{rk}_\otimes\Delta\).

If \(u\) and \(v\) have Schmidt ranks \(\rho_u\) and \(\rho_v\), the mixed-product rule gives \(uv^\top=\sum_{i,j}\sigma_i\tau_j\,(x_iz_j^\top)\otimes(y_iw_j^\top)\), an operator-Schmidt decomposition with \(\rho_u\rho_v\) terms, so \(\operatorname{rk}_\otimes(uv^\top)=\rho_u\rho_v\). A generic rank-one update, which LoRA\(_1\) reaches, thus needs \(\kappa_1=\min(m_1,m_2)\min(n_1,n_2)\) Kronecker terms (Thm II.11(c)).

Used in: Thm II.11, Figure 5, Prediction X.5, Known results. Reading: Nielsen and Chuang, Quantum Computation and Quantum Information (2000), §2.5.

Fact B.13 (rank of a Hadamard product; face-splitting product). For \(X,Y\in\mathbb R^{m\times n}\), \(\operatorname{rank}(X\odot Y)\le\operatorname{rank}X\cdot\operatorname{rank}Y\). Explicitly, if \(X=B_1A_1\) and \(Y=B_2A_2\) with inner dimensions \(r_1\) and \(r_2\), then \[X\odot Y=(B_1\bullet B_2)\,(A_1^\top\bullet A_2^\top)^\top,\] where the face-splitting product \(B_1\bullet B_2\in\mathbb R^{m\times r_1r_2}\) has as row \(i\) the Kronecker product of row \(i\) of \(B_1\) and row \(i\) of \(B_2\). In particular \(W_0\odot(BA)=\sum_{k\le r}\mathrm{diag}(b_k)\,W_0\,\mathrm{diag}(a_k)\), summed over the columns \(b_k\) of \(B\) and the rows \(a_k\) of \(A\), has rank at most \(r\operatorname{rank}W_0\) (Horn and Johnson, 1991, Ch. 5).

More: the idea of the proof

\(X\odot Y\) is the submatrix of \(X\otimes Y\) on the rows \((i,i)\) and the columns \((j,j)\). In pictures it is the spider composite \(\mu\circ(X\otimes Y)\circ\delta\) of B.6, which factors through the two inner wires.

LoHa\(_{r_1,r_2}\) therefore has rank at most \(r_1r_2\), and \(h=(B_1\bullet B_2,(A_1^\top\bullet A_2^\top)^\top)\) is the atlas arrow LoHa\(_{r_1,r_2}\to\) LoRA\(_{r_1r_2}\). HiRA’s update \(W_0\odot BA\) has rank at most \(r\operatorname{rank}W_0\), which allows full rank when \(W_0\) has full rank (Prop II.9(a)).

Used in: Prop II.12, Thm II.10, Prop II.9, When one method simulates another, Rosetta entries 6 and 11.

Sums, Para and lenses

Definition B.14 (sums of methods; Minkowski sums, differences and translates). For subsets \(S,T\) of a vector space, the Minkowski sum is \(S+T=\{s+t:s\in S,\ t\in T\}\), the Minkowski difference is \(S-T=\{s-t:s\in S,\ t\in T\}\), a translate of \(S\) is \(v+S=\{v+s:s\in S\}\), and \(C^{+K}\) is the sum of \(K\) copies of \(C\). A sum is the image of \(S\times T\) under addition, so for semialgebraic sets \(\dim(S+T)\le\dim S+\dim T\) (C.18), and likewise for differences. If \(0\in C\), the additive monoid (D.10) generated by \(C\) is \(\mathrm{Mon}(C)=\bigcup_KC^{+K}\), and if moreover \(\mathbb RC=C\) it is \(\mathrm{span}\,C\). Writing \(\rho_M=\theta_0+\delta_M\), the sum of methods \(M\oplus N=(Q_M\times Q_N,\,(q_0,q_0'),\,\theta_0+\delta_M+\delta_N)\) has image \(\theta_0+(\mathrm{Im}\,\delta_M+\mathrm{Im}\,\delta_N)\), and \(\oplus\) is a symmetric monoidal structure with the frozen model as unit (B.1).

Concatenation \(([B_1\ B_2],[A_1;A_2])\) is an isomorphism \(\mathrm{LoRA}_r\oplus\mathrm{LoRA}_s\cong\mathrm{LoRA}_{r+s}\) of unpointed objects, with the \(A\)-blocks rescaled to the target scale when the scales differ, as under \(\alpha/r\) scaling (Obs I.7). It is also a pointed isomorphism onto \(\mathrm{LoRA}_{r+s}\) pointed at the concatenated initialisation. RoSA and GraLoRA are \(\oplus\)-words. The categorical product of Obs I.4, when it exists, has image \(\mathrm{Im}\,M\cap\mathrm{Im}\,N\) (A.10). Merging and restarting \(K\) times with a fixed shape \(C\) reaches \(\theta_0+C^{+K}\). Since \(\mathcal M_{\le r}^{+K}=\mathcal M_{\le\min(Kr,m,n)}\), ReLoRA reaches every matrix after \(\lceil\min(m,n)/r\rceil\) cycles (Thm V.3).

Re-pointing an additive method \(\rho=\theta_b+\delta\) with shape \(C=\mathrm{Im}\,\delta\) at a parameter \(q\), so that \(\theta_b=\theta_0-\delta(q)\), gives the translate \(\theta_0+(C-\delta(q))\) as image, and all exact pointings together reach \(\theta_0+(C-C)\). For LoRA, \(\mathcal M_{\le r}\) is closed under negation and \(\mathcal M_{\le r}+\mathcal M_{\le r}=\mathcal M_{\le\min(2r,m,n)}\) by splitting an SVD (C.12), so \(C-C=\mathcal M_{\le\min(2r,m,n)}\), which is why no exact pointing reaches an update of rank above \(2r\) (Thm II.1).

Used in: Obs I.7, Sums, Thm II.1, Prop II.13, Thm V.3, Restarting, Reading from the right.

Definition B.15 (strong monoidal functors; \((\mathbb N,+)\) and the permutation groupoid). A strong monoidal functor \(F:(\mathcal C,\otimes,I)\to(\mathcal D,\otimes,J)\) is a functor with isomorphisms \(\phi_{A,B}:FA\otimes FB\to F(A\otimes B)\), natural in \(A\) and \(B\), and \(\phi_0:J\to FI\), compatible with associators and unitors. It is symmetric if also \(F(\sigma_{A,B})\circ\phi_{A,B}=\phi_{B,A}\circ\sigma_{FA,FB}\). The discrete monoidal category \((\mathbb N,+)\) has the natural numbers as objects, only identity morphisms, \(r\otimes s=r+s\) and identity symmetry. The permutation groupoid \(\mathbb P\) has the same objects, the symmetric group \(S_r\) as the automorphisms of \(r\) and no other morphisms, with \(\otimes\) given by \(+\) and block sums of permutations, and the block transposition in \(S_{r+s}\) as symmetry.

Concatenation (B.14) makes \(r\mapsto\mathrm{LoRA}_r\) strong monoidal from \((\mathbb N,+)\) into unpointed methods. It is not symmetric there, because \(\phi_{s,r}\circ\sigma\) lists the blocks in the other order. On \(\mathbb P\) the block transposition acts on \(\mathrm{LoRA}_{r+s}\) by permuting inner channels, \((B,A)\mapsto(BP^{-1},PA)\), and the symmetry square commutes. Pointed objects behave differently. If \(F(r)\) and \(F(2r)\) are standard \(\mathrm T_A\) pointings (\(B_0=0\) and \(A_0\) of full row rank, C.19), which needs \(2r\lt\min(m,n)\), then \(F(r)\oplus F(r)\not\cong F(2r)\) as pointed objects, since \(\dim S_1\) is \(mr\) against \(2mr\) (Obs I.7).

Used in: Obs I.7, The referee record. Reading: Mac Lane (1998), Ch. XI.

Definition B.16 (bicategories and the Para construction). A bicategory has objects, 1-cells between objects and 2-cells between parallel 1-cells. 2-cells compose vertically and horizontally, and composition of 1-cells is associative and unital up to coherent invertible 2-cells. For a symmetric monoidal category \(\mathcal C\), here always cartesian, the bicategory \(\mathbf{Para}(\mathcal C)\) has the objects of \(\mathcal C\). A 1-cell \(X\to Y\) is a pair \((P,\,f:P\times X\to Y)\), and \((P',g):Y\to Z\) composes with it to \((P'\times P,\ (p',p,x)\mapsto g(p',f(p,x)))\). A 2-cell \((P,f)\Rightarrow(Q,g)\) is a reparametrisation \(r:Q\to P\) with \(g=f\circ(r\times\mathrm{id}_X)\). A 2-functor (pseudofunctor) between bicategories sends objects, 1-cells and 2-cells to objects, 1-cells and 2-cells, and preserves both compositions up to coherent invertible 2-cells.

A linear layer is the 1-cell \(\big(\mathbb R^{m\times n},\,(W,x)\mapsto Wx\big):\mathbb R^n\to\mathbb R^m\), and stacking a second layer \(\mathbb R^m\to\mathbb R^k\) composes 1-cells, with parameter space \(\mathbb R^{k\times m}\times\mathbb R^{m\times n}\). A pretrained model is a pointed 1-cell \(F=(\Theta,\theta_0,f)\) of \(\mathbf{Para}(\mathbf{Diff})\), and its realisation \(\Phi_f:\Theta\to C^\infty(X,Y)\), \(\theta\mapsto f(\theta,-)\), records the function at each weight. LoRA is a 2-cell \(F\Rightarrow(Q,f\circ(\rho\times\mathrm{id}_X))\), the network with \((B,A)\) in its parameter slot, and a simulation is a 2-cell between two such cells (Remark I.3). An adapter extension is a 2-cell into \(F\), and merging it is a lift through \(\Phi_f\) (Def VI.1). Composition is associative only up to \((P''\times P')\times P\cong P''\times(P'\times P)\), hence a bicategory; over a strict monoidal category it is a 2-category.

Used in: Def I.1, What Para adds, Remark I.3, Def VI.1, Backpropagation as a lens, Key references. Reading: Leinster, “Basic bicategories” (1998).

Definition B.17 (lenses, reverse derivatives and \(\mathbf{Para}(\mathsf R)\)). In a cartesian category, a lens \((A,A')\to(B,B')\) is a pair of maps, a forward part \(f:A\to B\) and a backward part \(f^\sharp:A\times B'\to A'\). Lenses compose by \((g,g^\sharp)\circ(f,f^\sharp)=\big(g\circ f,\ (a,c')\mapsto f^\sharp(a,g^\sharp(f(a),c'))\big)\) and form the category \(\mathbf{Lens}(\mathcal C)\). For smooth maps between open subsets of Euclidean spaces, the reverse derivative is \(\mathsf R[f](x,y)=J_f(x)^\top y\). The chain rule \(J_{g\circ f}(x)^\top=J_f(x)^\top J_g(f(x))^\top\) says that \(f\mapsto(f,\mathsf R[f])\) is a functor into \(\mathbf{Lens}\). Applied to parametrised maps it gives the 2-functor \(\mathbf{Para}(\mathsf R):\mathbf{Para}(\mathcal C)\to\mathbf{Para}(\mathbf{Lens}(\mathcal C))\) of Cruttwell et al., which is backpropagation.

LoRA’s forward part is \(\rho(B,A)=W_0+sBA\), and its backward part pulls the weight gradient \(G\) back to \(\mathsf R[\rho]((B,A),G)=(sGA^\top,\,sB^\top G)\). GaLore keeps the forward part \(\mathrm{id}_\Theta\) and uses the compression \(G\mapsto P^\top G\) with anchor \(P\), a lens whose backward part is not the reverse derivative of its forward part (Def IV.1). For linear \(\rho(C)=W_k+\mathcal LC\), \(\mathsf R[\rho](C,G)=\mathcal L^*G\) does not depend on \(C\), which is why the two readings agree at degree one (Prop IV.2).

Used in: Exposé IV, Backpropagation as a lens, Def IV.1, Prop IV.2, Rosetta entry 25, Where the category theory works, Seven borrowed words. Reading: Cruttwell, Gavranović, Ghani, Wilson and Zanasi, “Categorical foundations of gradient-based learning” (arXiv 2021; ESOP 2022).

C. Manifolds, germs and cones

22 entries

Parts A and B used only the arrows of a method and its wiring. Part C turns to its geometry. A PEFT method on this site is a smooth map \(\rho:(Q,q_0)\to(\Theta,\theta_0)\), and its expressivity is read off the set \(\mathrm{Im}\,M=\rho(Q)\) near \(\theta_0\). This part collects the geometry behind that reading, from manifolds, differentials and rank for the map to germs, cones, singular points and semialgebraic dimension for the image, which is often not a manifold.

Manifolds and smooth maps

Definition C.1 (smooth manifolds and smooth maps). A smooth manifold of dimension \(k\) is a space covered by charts, bijections from open pieces of it onto open subsets of \(\mathbb R^k\), whose changes of chart are smooth. Every manifold on this site is an open subset of a vector space or an embedded submanifold of one (C.4), or is diffeomorphic to one, as quotients (D.22) and the Grassmannian (C.15) are. A diffeomorphism is a smooth bijection with smooth inverse, and \(\mathrm{Diff}(Q)\) is the group of diffeomorphisms of \(Q\) under composition. A local diffeomorphism maps an open neighbourhood of each point diffeomorphically onto an open set; by the inverse function theorem, \(\rho\) does this near \(q\) whenever \(d\rho_q\) (C.2) is invertible.

LoRA’s parameter space \(\mathbb R^{m\times r}\times\mathbb R^{r\times n}\) is a vector space, and HRA’s \(\{(u_1,\dots,u_r):u_i\ne0\}\) is open in \((\mathbb R^n)^r\). The rotation group \(SO(n)\) is a compact manifold of dimension \(n(n-1)/2\), and the Cayley map (D.7) is a diffeomorphism from \(\mathfrak{so}(n)\) onto an open dense subset of it. A method is a pointed smooth map, and its gauge group \(\{\varphi\in\mathrm{Diff}(Q):\rho\circ\varphi=\rho\}\) (D.21) contains LoRA’s \((B,A)\mapsto(Bg^{-1},gA)\) for every \(g\in GL_r\).

Used in: Def I.2, Def I.5, Thm II.7, Prop IV.3. Reading: Lee, Introduction to Smooth Manifolds (2nd ed., 2013), Ch. 1 and 2.

Definition C.2 (tangent spaces, differentials and covectors). For a manifold \(Q\subseteq\mathbb R^N\), the tangent space \(T_qQ\) is the set of velocities \(c'(0)\) of smooth curves \(c\) in \(Q\) with \(c(0)=q\). It is a linear space of dimension \(\dim Q\), all of \(\mathbb R^N\) when \(Q\) is open. The differential of a smooth \(\rho\) at \(q\) is the linear map \(d\rho_q:T_qQ\to T_{\rho(q)}\Theta\), \(c'(0)\mapsto(\rho\circ c)'(0)\). In coordinates it is the Jacobian, and the chain rule reads \(d(\rho\circ h)_q=d\rho_{h(q)}\circ dh_q\). The cotangent space \(T^*_\theta\Theta\) is the dual of \(T_\theta\Theta\). The differential \(dL_\theta\) of a loss is a covector, and it becomes a gradient vector \(\nabla L(\theta)\) only once an inner product is chosen.

For LoRA, \(d\rho_{(B,A)}(\dot B,\dot A)=s(\dot BA+B\dot A)\). For a rotation orbit, \(T_I\,SO(n)=\mathfrak{so}(n)\) and \(T_{W_0R}\big(W_0\cdot SO(n)\big)=W_0R\,\mathfrak{so}(n)\). Obs I.6 is the chain rule at \(q_0\). Thm III.5 drives the parameters by any cotangent signal \(\gamma(t)\in T^*_{\rho(q)}\Theta\), pulled back to \(Q\) as \(\gamma\circ d\rho_q\), which is \(D\rho^\top\gamma\) in coordinates.

Used in: Notation, Def I.5, Obs I.6, Thm II.2, Thm II.7, Thm III.5, Def IV.1. Reading: Lee (2013), Ch. 3 and 11.

Definition C.3 (rank, expressive dimension and first-order image). The rank of \(\rho\) at \(q\) is \(\operatorname{rank}d\rho_q\). It is lower semicontinuous, since a nonzero \(k\times k\) minor of the Jacobian stays nonzero nearby, so \(\{q:\operatorname{rank}d\rho_q\ge k\}\) is open. The generic rank is \(\max_q\operatorname{rank}d\rho_q\). If \(\rho\) is real-analytic and \(Q\) is connected, the generic rank is attained on an open dense set whose complement has measure zero (C.22). The site writes \(\lvert M\rvert=\dim Q\) for the parameter budget, \(d(M)\) for the generic rank, \(S_1=d\rho_{q_0}(T_{q_0}Q)\) for the first-order image and \(d(M)-\dim S_1\) for the first-order deficit. For definable \(\rho\) on an open definable \(Q\), as for every method on the site, C.18(d) shows that \(d(M)=\dim\mathrm{Im}\,M\), the expressive dimension. The deficit vanishes exactly when \(q_0\) is a point of maximal rank; the base point \(\theta_0\) is then called regular for \(M\), and an apex when the deficit is positive (Def I.5).

Prop II.3 gives LoRA’s rank at every point, \(m\operatorname{rk}A+n\operatorname{rk}B-\operatorname{rk}A\operatorname{rk}B\). At a random pair it is \(r(m+n-r)=d\); at the zero-init point \((0,A_0)\) it is \(mr\), a deficit of \(r(n-r)\). Since the points of lower rank have measure zero, a float64 Jacobian rank at one random point computes \(d(M)\) with probability one, up to the tolerance that decides numerical rank. The plug workbench and the check script do exactly this.

Used in: Notation, Def I.5, Prop II.3, Prop II.12, Figure 6, reproducing the numbers.

Definition C.4 (immersions, submersions and embedded submanifolds). A smooth \(\rho:Q\to\Theta\) is an immersion if every \(d\rho_q\) is injective (rank \(\dim Q\)), and a submersion if every \(d\rho_q\) is surjective (rank \(\dim\Theta\)). A subset \(X\subseteq\mathbb R^N\) is an embedded submanifold of dimension \(k\) if near each of its points it is the zero set of \(N-k\) smooth functions with linearly independent gradients. A submersion is an open map, so its image contains an open set. An embedding, an injective immersion that is a homeomorphism onto its image, has an embedded submanifold as image; an injective immersion such as the figure eight need not.

The one-sided method \(\mathrm{FA}(A_0):B\mapsto W_0+BA_0\) is an injective immersion with affine image \(W_0+\{BA_0\}\) when \(A_0\) has full row rank. On pairs of full rank, \((B,A)\mapsto BA\) is a submersion onto \(\mathcal M_r\) (C.12). Obs I.4(b) uses openness, in the argument recalled in A.13. Prop IV.3 asks for injective immersions \((\mathbb R^k,0)\to(U,\theta)\) tangent to a distribution.

Used in: Obs I.4, Prop IV.3, integrability. Reading: Lee (2013), Ch. 4 and 5.

Fact C.5 (constant-rank theorem). Let \(\rho:Q\to\Theta\) be smooth with \(\operatorname{rank}d\rho_q=k\) for all \(q\) near \(p\). Then there are charts centred at \(p\) and at \(\rho(p)\) in which \(\rho\) reads \((x_1,\dots,x_{\dim Q})\mapsto(x_1,\dots,x_k,0,\dots,0)\). Hence some neighbourhood \(V\) of \(p\) has \(\rho(V)\) an embedded \(k\)-dimensional submanifold, and \(\rho^{-1}(\rho(p))\cap V\) an embedded submanifold of dimension \(\dim Q-k\). By C.3 the hypothesis holds wherever the rank is maximal.

More: the idea of the proof

Choose coordinates in which the top-left \(k\times k\) block of the Jacobian is invertible. The inverse function theorem straightens the first \(k\) output coordinates. Constant rank then forces the other outputs to depend only on those \(k\), and a second change of coordinates in the target removes them.

The site uses this theorem without naming it. Near a point of maximal rank, \(\rho\) is locally a projection onto a sheet of dimension \(d(M)\), with fibres of dimension \(\lvert M\rvert-d(M)\), the gauge waste. For LoRA at a full-rank pair the sheet is a piece of \(W_0+\mathcal M_r\) and the fibre a piece of a \(GL_r\)-orbit. Prop IV.3 makes the analogous constant-rank assumption for distributions, where Frobenius’ theorem (D.25) takes the place of this one.

Used in: Def I.5, Prop II.12, C.18. Reading: Lee (2013), Ch. 4 (the rank theorem).

Definition C.6 (fibres and fibre dimension). The fibre of \(\rho\) over \(\theta\) is \(\rho^{-1}(\theta)\), the set of parameters that give the weight \(\theta\). If \(\operatorname{rank}d\rho_q=d(M)\), C.5 makes the fibre through \(q\) a submanifold of dimension \(\lvert M\rvert-d(M)\) near \(q\). For real-analytic \(\rho\) on connected \(Q\) this holds for all \(q\) outside a closed set of measure zero, and these are the generic fibres. Fibres through points of lower rank can have a different dimension. The fibres of a fibre bundle all look alike locally, so a map whose fibre dimension jumps is no fibre bundle.

For LoRA on the full-rank stratum, the fibre through \((B,A)\) is the orbit \(\{(Bg^{-1},gA):g\in GL_r\}\), of dimension \(r^2=r(m+n)-r(m+n-r)\) (D.21). For zero-init LoRA the fibre over \(\theta_0\) is \(\{(B,A):BA=0\}\). It contains \(\{B=0\}\), of dimension \(rn\gt r^2\), and is itself a union of strata, such as \(\{(0,A):\operatorname{rank}A=r\}\) (Cor III.13).

Used in: Exposé III, Def I.5, Prop II.12, Cor III.13, Rosetta entries 6 and 11, Seven borrowed words.

Definition C.7 (fibre products, transversality and clean intersection). For \(\rho_M:Q_M\to\Theta\) and \(\rho_N:Q_N\to\Theta\), the set-level fibre product is \(P=\{(q,q'):\rho_M(q)=\rho_N(q')\}\subseteq Q_M\times Q_N\). The maps are transverse, \(\rho_M\pitchfork\rho_N\), if \(d\rho_M(T_qQ_M)+d\rho_N(T_{q'}Q_N)=T_\theta\Theta\) at every \((q,q')\in P\); then \(P\) is an embedded submanifold of dimension \(\lvert M\rvert+\lvert N\rvert-\dim\Theta\), and it is the pullback of \(\rho_M\) and \(\rho_N\) in \(\mathbf{Diff}\) (A.11). The intersection is clean if \(P\) is an embedded submanifold with \(T_{(q,q')}P=\{(v,w):d\rho_Mv=d\rho_Nw\}\) at every point. Transverse maps meet cleanly, and so do affine maps.

Transversality at \((q_0,q_0')\) needs \(\lvert M\rvert+\lvert N\rvert\ge\dim\Theta\), which fails for any two small methods, so the products the site builds come from clean cases. The affine methods \(\mathrm{FA}(A_0)\) and \(\mathrm{FB}(B_0)\), with \(B_0\) of full column rank and \(A_0\) of full row rank, have the linear fibre product \(\{(B_0R,RA_0)\}\) of dimension \(r^2\), which is LoRA-XS (Prop I.8). On \(\Theta=\mathbb R\), \(\mathrm{LoRA}_1\) and the frozen model have the cross \(\{BA=0\}\) as fibre product. It is no manifold at the origin, and no product exists (A.13).

Used in: Obs I.4, Prop I.8, Thm II.7(d). Reading: Guillemin and Pollack, Differential Topology (1974), Ch. 1; Lee (2013), Ch. 6.

Germs, cones and singular points

Definition C.8 (germs). Two subsets \(X,Y\subseteq\Theta\) have the same germ at \(\theta_0\) if \(X\cap U=Y\cap U\) for some neighbourhood \(U\) of \(\theta_0\); two maps have the same germ at \(q_0\) if they agree near \(q_0\). The germs \((X,\theta_0)\) and \((Y,0)\) are diffeomorphic if a diffeomorphism \(\varphi\) between neighbourhoods \(U\ni\theta_0\) and \(V\ni0\) has \(\varphi(\theta_0)=0\) and \(\varphi(X\cap U)=Y\cap V\). Local properties, such as the dimension at \(\theta_0\), smoothness at \(\theta_0\) and the tangent cone (C.11), depend only on the germ.

The image germ of a method is the germ of \(\mathrm{Im}\,M=\rho(Q)\) at \(\theta_0\). It uses all of \(Q\), so it can be larger than the image of a small neighbourhood of \(q_0\). For zero-init LoRA, \(sBA\) with \(B\) small and \(A\) arbitrary is a small update with any row space, so the germ is that of the whole cone \(\theta_0+\mathcal M_{\le r}\); parameters near \((0,A_0)\) only give updates with row space inside \(\mathrm{row}\,A\) for some \(A\) near \(A_0\). For injective \(W_0\) and even \(r\) with \(2\le r\le n-2\), HRA’s image germ at \(W_0\) is the image of the germ of \(\Sigma_r\) at \(0\) under the embedding \(\Omega\mapsto W_0\,C(\Omega)\) of \(\mathfrak{so}(n)\) into \(\mathbb R^{m\times n}\) (C.13, D.7).

Used in: the image germ, three invariants, Thm II.7, the seven axes.

Definition C.9 (smooth and singular points). A point \(x\) of a set \(X\subseteq\mathbb R^N\) is a smooth point if \(X\cap U\) is a \(C^1\) embedded submanifold for some neighbourhood \(U\) of \(x\), and a singular point otherwise. The singular points form the singular locus \(\mathrm{Sing}\,X\). Algebraic geometry defines \(\mathrm{Sing}\) through the Jacobian of defining polynomials, and for \(\mathcal M_{\le r}\) the two notions agree. Smoothness is a property of a set. The site’s regular base point and apex, defined by a zero or positive first-order deficit (C.3, Def I.5), are properties of a method at \(q_0\).

The cross \(\{BA=0\}\subseteq\mathbb R^2\) is singular at the origin and smooth elsewhere. For \(1\le r\lt\min(m,n)\), \(\mathrm{Sing}\,\mathcal M_{\le r}=\mathcal M_{\le r-1}\). A split initialisation has image \((\theta_0-P)+\mathcal M_{\le r}\) with \(\operatorname{rank}P=r\), so \(\theta_0\) is a smooth point; zero-init LoRA starts at the singular vertex. A figure-eight curve passing through its crossing with nonzero velocity has deficit \(0\) there, though its image is singular, so smooth and regular are different notions.

Used in: Def I.5, Thm II.2, Figure 2, Thm II.7, Erlangen. Reading: Harris, Algebraic Geometry: A First Course (1992), Lecture 14.

Definition C.10 (cones and vertices). A set \(X\) in a vector space is a cone with vertex \(v\) if \(v\in X\) and \(v+t(x-v)\in X\) for all \(x\in X\) and \(t\ge0\), and linear if it is an affine subspace. If \(\rho=\theta_0+\delta\) with \(\delta(tu,w)=t\,\delta(u,w)\) for \(t\ge0\) on a block \(u\) of parameters, and \(Q\) is closed under \(u\mapsto tu\), then \(\mathrm{Im}\,M\) is a cone with vertex \(\theta_0\). A cone that is a \(C^1\) embedded submanifold near its vertex \(v\) equals \(v+T_vX\), so the vertex of a non-linear cone is a singular point.

More: why a smooth cone is linear

Each ray from \(v\) lies in \(X\), so \(X\subseteq v+T_vX\). Near \(v\), \(X\) is a submanifold of full dimension in this affine space, so it contains a neighbourhood of \(v\) there, and scaling spreads it over all of \(v+T_vX\).

\(\mathcal M_{\le r}\) is a closed cone with vertex \(0\), and it is not linear for \(1\le r\lt\min(m,n)\), since a rank-\(r\) and a rank-one matrix can sum to rank \(r+1\). Zero-init LoRA has \(\delta=sBA\), linear in \(B\), so its image \(W_0+\mathcal M_{\le r}\) has singular vertex \(W_0\). Its deficit \(r(n-r)\) makes \(\theta_0\) an apex too; the vertex is a property of the set, the apex one of the method. Multihomogeneous updates such as LoHa’s give cones in the same way (D.19).

Used in: Def I.5, the image germ, Thm II.1, Thm II.2, Figure 2, Thm II.7.

Fact C.11 (first-order image versus image). The tangent cone of a set \(X\) at \(x\) is the set of limits of \(t_k(x_k-x)\) with \(x_k\in X\), \(x_k\to x\) and \(t_k\gt0\); for a closed cone with vertex \(v\) it is \(X-v\). For a method \(M\): (a) \(S_1\) lies in the tangent cone of \(\mathrm{Im}\,M\) at \(\theta_0\); (b) \(\dim S_1\le d(M)\); (c) if \(\mathrm{Im}\,M\) is a \(C^1\) manifold of dimension \(d(M)\) near \(\theta_0\), then \(S_1\subseteq T_{\theta_0}\mathrm{Im}\,M\), with equality exactly when the deficit is \(0\).

For (a), \(d\rho_{q_0}(\xi)\) is the velocity of the curve \(\rho(q_0+t\xi)\) inside the image; for (c), \(\rho\) maps a neighbourhood of \(q_0\) into the manifold. For \(\mathrm T_A\)-pointed LoRA (\(B_0=0\), C.19) the tangent cone at \(\theta_0\) is \(\mathcal M_{\le r}\) itself, and inside it \(S_1=\{XA_0\}\), the matrices with row space in \(\mathrm{row}\,A_0\), is a linear space of dimension \(mr\lt r(m+n-r)\). Figure 2 draws \(S_1\) as one ruling of the cone. At a split pointing case (c) applies, and \(S_1=T_P\mathcal M_r\).

Used in: Def I.5, Obs I.6, Thm II.2, Figure 2. Reading: Harris (1992), Lecture 20.

Matrix varieties

Definition C.12 (the determinantal variety). \(\mathcal M_{\le r}=\{X\in\mathbb R^{m\times n}:\operatorname{rank}X\le r\}\), the determinantal variety, is the zero set of all \((r+1)\times(r+1)\) minors, a closed cone with vertex \(0\). Let \(1\le r\lt\min(m,n)\). The matrices of rank exactly \(r\) form an open dense subset \(\mathcal M_r\) of \(\mathcal M_{\le r}\), an embedded submanifold of dimension \(r(m+n-r)\), and \(\mathrm{Sing}\,\mathcal M_{\le r}=\mathcal M_{\le r-1}\). If \(X\in\mathcal M_r\) has column space \(U\) and row space \(V\), and \(X=BA\) with \(B\in\mathbb R^{m\times r}\) and \(A\in\mathbb R^{r\times n}\), then \[T_X\mathcal M_r=\{\dot BA+B\dot A\}=\{D:P_{U^\perp}DP_{V^\perp}=0\}.\] Also \(\mathcal M_{\le r}-\mathcal M_{\le r}=\mathcal M_{\le\min(2r,m,n)}\).

Zero-init LoRA has image \(W_0+\mathcal M_{\le r}\), of dimension \(130{,}816\) for a \(4096\times4096\) matrix at \(r=16\). On full-rank pairs, \((B,A)\mapsto BA\) is a submersion onto \(\mathcal M_r\) whose fibres are \(GL_r\)-orbits, which identifies \(\mathcal M_r\) with the quotient \((\mathbb R^{m\times r}_*\times\mathbb R^{r\times n}_*)/GL_r\) (D.22). The difference formula is why no pointing of \(\mathrm{LoRA}_r\) reaches an update of rank above \(2r\) (B.14, Thm II.1).

Used in: Notation, translates of one shape, Thm II.1, Thm II.2, Prop II.8, Exposé II.C, Prop III.4. Reading: Harris (1992), Lecture 9.

Definition C.13 (the skew-determinantal cone). Every skew-symmetric matrix has even rank. For even \(r\) with \(2\le r\le n-2\), the skew-determinantal cone \(\Sigma_r=\{\Omega\in\mathfrak{so}(n):\operatorname{rank}\Omega\le r\}\) is a closed cone with vertex \(0\) and dimension \(r(n-r)+r(r-1)/2=rn-r(r+1)/2\), counting a range in \(\mathrm{Gr}(r,n)\) and a nondegenerate skew form on it. Its smooth points are the matrices of rank exactly \(r\), and \(\mathrm{Sing}\,\Sigma_r=\Sigma_{r-2}\). It is not linear, since skew matrices of rank \(r\) and \(2\) can sum to rank \(r+2\le n\), so its vertex is singular.

For injective \(W_0\) and even \(r\), HRA’s image near \(W_0\) consists of the \(W_0R\) with \(R\in SO(n)\) near \(I\) and \(\operatorname{rank}(R-I)\le r\). The Cayley map satisfies \(\operatorname{rank}(C(\Omega)-I)=\operatorname{rank}\Omega\) (D.7), so the image germ at \(W_0\) is the image of the germ of \(\Sigma_r\) at \(0\) under the embedding \(\Omega\mapsto W_0\,C(\Omega)\), \(W_0\) is a singular vertex, and \(d(\mathrm{HRA}_r)=\dim\Sigma_r\). At the paired initialisation \(u_{2i-1}=u_{2i}=w_i\) (\(i\le k=r/2\), with the \(w_i\) independent), put \(U=\mathrm{span}\{w_1,\dots,w_k\}\), so \(\dim U=k\). Then \(S_1=W_0\cdot\mathcal S\) with \[\mathcal S=\{\Omega\in\mathfrak{so}(n):P_{U^\perp}\Omega P_{U^\perp}=0\}=U\wedge\mathbb R^n\] (D.5), a linear space of skew matrices of rank at most \(2k=r\) and of dimension \(kn-k(k+1)/2\lt d(\mathrm{HRA}_r)\). It lies inside \(\Sigma_r\), so \(S_1\) lies inside the tangent cone \(W_0\cdot\Sigma_r\) (Thm II.7(b)).

Used in: Thm II.7, HRA is a meet.

Definition C.14 (Segre and toric cones). The Segre cone in \(\mathbb R^p\otimes\mathbb R^q\cong\mathbb R^{p\times q}\) is the set \(\{a\otimes b\}=\mathcal M_{\le1}\) of rank-one tensors, a closed cone of dimension \(p+q-1\) that spans the whole space and is non-linear when \(p,q\ge2\). It is the simplest toric cone, the closure of the image of a torus \((\mathbb R^\times)^k\) under a monomial map, here \((a,b)\mapsto(a_ib_j)_{ij}\). Since rank-one matrices span \(\mathbb R^{p\times q}\), \(k\) generic rank-one matrices span \(\min(k,pq)\) dimensions.

Under the Van Loan–Pitsianis rearrangement (B.11), KronA’s update \(A\otimes B\) becomes \(\mathrm{vec}(A)\,\mathrm{vec}(B)^\top\), so its image is a linear copy of a Segre cone (Prop II.12). VeRA’s update \(\Lambda_bB\Lambda_dA\) has entries \(\sum_k(b_id_k)B_{ik}A_{kj}\), a fixed linear function of the rank-one matrix \(bd^\top\). This function is injective when \(B\) has no zero entry and \(A\) has full row rank, so the image is a toric cone of dimension \(m+r-1\) with vertex \(\theta_0\), as VeRA’s card records. Prop II.8 uses the spanning fact for DoRA, with \(m\) points of the Segre cone in \(\mathbb R^{m-r}\otimes\mathbb R^{n-r}\).

Used in: Prop II.8, Prop II.12, VeRA. Reading: Harris (1992), Lecture 2; Cox, Little and Schenck, Toric Varieties (2011), Ch. 1.

Definition C.15 (the Grassmannian). The Grassmannian \(\mathrm{Gr}(r,n)\) is the set of \(r\)-dimensional linear subspaces of \(\mathbb R^n\), a compact manifold of dimension \(r(n-r)\). One model is the full-row-rank \(r\times n\) matrices up to \(A\mapsto gA\), \(g\in GL_r\), through \(A\mapsto\mathrm{row}\,A\); another is the orthogonal projectors \(P=P^\top=P^2\) with \(\operatorname{tr}P=r\). Near \(V\), the graphs of linear maps \(V\to V^\perp\) give a chart onto \(\mathrm{Hom}(V,V^\perp)\cong\mathbb R^{r(n-r)}\). \(\mathrm{Gr}(r,n)\) carries a unique \(O(n)\)-invariant probability measure, the uniform or Haar measure, and the row space of a Gaussian \(A\) has this law (D.14).

\(\mathrm T_A\)-pointed LoRA is classified up to isomorphism by \(\mathrm{row}\,A_0\in\mathrm{Gr}(r,n)\) (Thm II.2), and its deficit \(r(n-r)\) equals \(\dim\mathrm{Gr}(r,n)\). First steps keep the row space of the update inside \(\mathrm{row}\,A_0\), and the missing directions are those that tilt it. For injective \(W_0\) and even \(r\) with \(2\le r\le n-2\), HRA’s image is the union of the orbits \(W_0\cdot SO(V)\) over \(V\in\mathrm{Gr}(r,n)\), and its dimension \(r(n-r)+r(r-1)/2\) counts a subspace and a rotation inside it. LoRA-FA, which trains \(B\) with \(A_0\) frozen, sees \(A_0\) only through \(V=\mathrm{row}\,A_0\) once its gradient is corrected as in Prop III.4(c).

Used in: Thm II.2, checks, Thm II.7, LOFT, LOFT example, Prop III.4, Cor III.13. Reading: Lee (2013), Ch. 1; Harris (1992), Lecture 6.

Tame sets and genericity

Fact C.16 (images of smooth maps need not be manifolds). Even a polynomial map can have an image that is no manifold. (a) Cusps. \(t\mapsto(t^2,t^3)\) is injective with image \(\{y^2=x^3\}\), which is not \(C^1\) at \(0\). (b) Crossings and cones. The cross \(\{BA=0\}\subseteq\mathbb R^2\) and the cone \(\mathcal M_{\le r}\) are singular at the origin. (c) Missing limits. \((x,y)\mapsto(x,xy)\) has image \(\{u\ne0\}\cup\{(0,0)\}\) in the \((u,v)\)-plane, neither open nor closed. (d) Dense images. The line of irrational slope is an injective immersion into the torus with dense image.

PEFT images show (b) routinely, as in the LoRA cone and HRA’s skew cone, so the site measures them with tools that need no manifold structure, such as germs (C.8), tangent cones (C.11) and the dimension theory of semialgebraic and definable sets (C.17, C.18). Whether LoHa’s Hadamard image is closed, the question raised by (c), is open.

Used in: Obs I.4, Thm II.2, Thm II.7, open problems.

Definition C.17 (semialgebraic and definable sets). A semialgebraic set is a finite union of sets \(\{x\in\mathbb R^N:p(x)=0,\ q_1(x)\gt0,\dots,q_k(x)\gt0\}\) with polynomials \(p,q_i\), and a map is semialgebraic if its graph is. A set is definable in \(\mathbb R_{\exp}\) if a first-order formula defines it from polynomials with real coefficients, \(\exp\), equality, order, Boolean operations and quantifiers over \(\mathbb R\). Wilkie proved that \(\mathbb R_{\exp}\) is o-minimal, meaning every definable subset of \(\mathbb R\) is a finite union of points and intervals. Semialgebraic sets are the definable sets that need no \(\exp\).

LoRA, DoRA (whose row norms involve square roots), HRA and exact Cayley OFT are built from polynomials, quotients and square roots, so their images are semialgebraic by C.18(a). A softmax brings in \(\exp\), and maps that contain one are definable in \(\mathbb R_{\exp}\).

Used in: where the category theory works, arrow rules. Reading: Bochnak, Coste and Roy, Real Algebraic Geometry (1998), Ch. 2; Wilkie, J. Amer. Math. Soc. 9 (1996).

Fact C.18 (Tarski–Seidenberg and definable dimension). (a) The image of a semialgebraic set under a semialgebraic map is semialgebraic (Tarski–Seidenberg); images of definable sets under definable maps are definable. (b) Every definable set \(X\) is a finite disjoint union of \(C^1\) embedded submanifolds, and \(\dim X\), the largest of their dimensions, does not depend on the choice. (c) \(X\subseteq Y\) implies \(\dim X\le\dim Y\), \(\dim f(X)\le\dim X\) for definable \(f\), and \(\dim(\overline X\setminus X)\lt\dim X\) for nonempty \(X\). (d) For a definable \(C^1\) map \(\rho\) on an open definable \(Q\subseteq\mathbb R^k\), \(\dim\rho(Q)=\max_q\operatorname{rank}d\rho_q\).

More: the idea of (d)

Near a point of maximal rank, C.5 gives a sheet of that dimension inside \(\rho(Q)\). Conversely, a definable section of \(\rho\), \(C^1\) on a top-dimensional piece of \(\rho(Q)\), composes with \(\rho\) to the identity there, so the rank is at least that dimension.

Part (d) is the identity \(d(M)=\dim\mathrm{Im}\,M\) of Def I.5. Part (c) is the monotonicity behind test O2, since an arrow \(M\to N\) gives \(\mathrm{Im}\,M\subseteq\mathrm{Im}\,N\) and hence \(d(M)\le d(N)\); for instance \(\mathrm{LoRA}_r\not\to\) LoRA-FA because \(r(m+n-r)\gt mr\).

Used in: Def I.5, arrow rules, where the category theory works. Reading: van den Dries, Tame Topology and O-minimal Structures (1998), Ch. 3 and 4; Bochnak, Coste and Roy (1998), Ch. 2.

Definition C.19 (stratification and local dimension). A stratification of a set \(X\) is a partition into finitely many embedded submanifolds, the strata, such that the closure of each stratum is a union of strata. The local dimension \(\dim_xX\) is the dimension of the germ of \(X\) at \(x\), the largest dimension of a stratum whose closure contains \(x\). \(X\) is pure-dimensional if \(\dim_xX\) is the same at every point. Definable sets admit stratifications, so these notions make sense for every image on the site.

The cross \(\{ba=0\}\subseteq\mathbb R^2\) is stratified as the origin and the four open half-axes, and it is pure of dimension \(1\). Likewise \(\mathcal M_{\le r}=\mathcal M_0\sqcup\mathcal M_1\sqcup\dots\sqcup\mathcal M_r\) with \(\overline{\mathcal M_k}=\mathcal M_{\le k}\), and it is pure of dimension \(r(m+n-r)\). For \(1\le r\lt\min(m,n)\), Thm II.2 singles out three strata of full-rank LoRA pointings, \(\mathrm T_A\) (\(B_0=0\), \(\operatorname{rank}A_0=r\)), \(\mathrm T_B\) (\(A_0=0\), \(\operatorname{rank}B_0=r\)) and the open dense \(\mathrm S\) (\(\operatorname{rank}B_0A_0=r\)), and the fibre \(\{BA=0\}\) over \(\theta_0\) is itself stratified (Cor III.13). The HRA fibre product of Thm II.7(d) has local dimension \(d(\mathrm{HRA}_r)+r^2=rn+r(r-1)/2\) over generic points of the meet and contains \(\{(B,0,I)\}\), of dimension \(mr\), over \(W_0\), so it is not pure-dimensional when \(mr\gt rn+r(r-1)/2\).

Used in: Thm II.2, three strata, Thm II.7, Cor III.13, open problems, Seven borrowed words. Reading: van den Dries (1998), Ch. 4; Bochnak, Coste and Roy (1998), Ch. 9.

Definition C.20 (moduli sets and moduli spaces). Classifying objects up to isomorphism gives a set of classes. A moduli space also carries a topology or geometry for which continuous or smooth families of objects correspond to continuous or smooth maps into it, ideally with a universal family over it from which every family is pulled back. A disjoint union of spaces \(X\sqcup Y\) has the topology in which \(X\) and \(Y\) are both open and closed.

Thm II.2 classifies the three strata of LoRA pointings, as a set, by \(\mathrm{Gr}(r,n)\sqcup\mathrm{Gr}(r,m)\sqcup\mathcal M_r\). The quotient topology does not split this way, since \((\varepsilon B',A_0)\in\mathrm S\) tends to \((0,A_0)\in\mathrm T_A\) as \(\varepsilon\to0\), so \(\mathrm T_A\) lies in the closure of \(\mathrm S\). No universal family is built, and the site uses “moduli” as an analogy.

Used in: Thm II.2, Rosetta, Seven borrowed words. Reading: Harris (1992), Lecture 4.

Definition C.21 (generic properties). A property of points of \(\mathbb R^N\) holds generically if the set where it fails lies in \(\{f=0\}\) for some nonzero polynomial \(f\). The set \(\{f=0\}\) is closed, nowhere dense and of Lebesgue measure zero, so a generic property holds with probability one at a random point whose law has a density, such as a Gaussian or uniform initialisation. For points of a parametrised set, such as a Segre cone, “generic” refers to generic parameters. To prove genericity it suffices to write failure as the vanishing of polynomials and exhibit one point where one of them is nonzero. A weaker notion, almost every point, asks only that the failure set have Lebesgue measure zero, as a countable union of proper algebraic subsets does.

Thm II.2(d) uses a generic rank-one update, Prop II.8 gives DoRA’s dimension \(\min\big(mn,\,r(m+n-r)+m\big)\) for generic \(W_0\), and Prop V.2 notes that QLoRA’s defect \(e=\kappa W-W\) has full rank for almost every \(W\), in the weaker sense. The quantiser \(\kappa\) is defined cell by cell rather than by one polynomial, so the argument runs cell by cell, and the failure set is a countable union of proper algebraic subsets of quantisation cells, which has measure zero. A pretrained \(W_0\) is not random, and structured weights can lie in the exceptional set. For a rank-one \(W_0=ac^\top\) with all \(a_i\ne0\) and \(r\lt n\), Prop II.8 finds DoRA’s dimension \(r(m+n-r)+m-r\), strictly below the generic value exactly when \(r\lt m\) and \(r\le n-2\).

Used in: Thm II.2, Prop II.8, Thm II.11, Prop II.12, Prop V.2.

Definition C.22 (real-analytic functions and their zero sets). A function is real-analytic on an open set if near every point it equals a convergent power series. Polynomials, \(\exp\), softmax, \(\tanh\), GELU and SiLU are real-analytic, and so are sums, products, compositions and quotients with nonvanishing denominator. On a connected open set, a real-analytic function that vanishes on a nonempty open subset vanishes identically, and the zero set of one that is not identically zero is closed, nowhere dense and of measure zero.

Lemma VI.2 compares Taylor coefficients at \(0\). A nonzero scale \(c\) with \(\sigma(ct)=c\,\sigma(t)\) must satisfy \(ca_k=c^ka_k\) for every coefficient \(a_k\) of \(\sigma\), so \(c=1\) for GELU and SiLU, whose \(a_2\ne0\). Thm VI.5(f) uses the zero-set fact. A prefixed head is analytic in the prefix, and one prefix with a nonzero second derivative makes the prefixes with affine output a nowhere-dense set. The same fact makes the maximal-rank set of an analytic \(\rho\) open and dense (C.3).

Used in: Lemma VI.2, Thm VI.5, Rosetta entry 5. Reading: Krantz and Parks, A Primer of Real Analytic Functions (2nd ed., 2002).

D. Groups, symmetry and conservation

34 entries

Part D turns to symmetry. Most PEFT parametrisations have symmetries, since different trainable parameters can give the same weights and some transformations of weight space carry a method’s image to itself. This part collects the group theory the site uses to describe those symmetries, the invariants they leave untouched, and the quantities they conserve under gradient flow.

Groups and their Lie algebras

Definition D.1 (Lie groups; the matrix groups of the site). A Lie group is a group that is also a smooth manifold, with smooth multiplication and inversion. Every closed subgroup of \(GL_n(\mathbb R)\) is a Lie group (Cartan’s closed-subgroup theorem), and all groups below are of this kind. Its dimension is its dimension as a manifold.

group	elements
\(GL_r\) dimension \(r^2\)	invertible \(r\times r\) matrices
\(O(n)\), \(SO(n)\) dimension \(n(n-1)/2\)	\(R^\top R=I\); for \(SO(n)\) also \(\det R=1\)
\(CO(n)\) dimension \(n(n-1)/2+1\)	\(cR\) with \(c\gt0\), \(R\in O(n)\), so \(CO(n)=\mathbb R_{\gt0}\cdot O(n)\); equivalently \(g^\top g\in\mathbb R_{\gt0}I\)
\(Sp(2r,\mathbb R)\) dimension \(r(2r+1)\)	\(M^\top JM=J\), with \(J=\begin{pmatrix}0&I_r\\-I_r&0\end{pmatrix}\)
\(T_m\) dimension \(m\)	invertible diagonal \(m\times m\) matrices (D.2)

LoRA’s gauge is \(GL_r\), acting on factors by \(g\cdot(B,A)=(Bg^{-1},gA)\). Exactly orthogonal OFT keeps the weight inside the orbit \(W_0\cdot O(n)\), and HRA with even \(r\) keeps it inside \(W_0\cdot SO(n)\). \(CO(n)\) is exactly the input-side covariance group (D.17) of the PiSSA, MiLoRA and paper-form CorDA splits. LoCO’s skew generator satisfies \(UV^\top-VU^\top=ZJZ^\top\) with \(Z=[U\ V]\), so it is unchanged under \(Z\mapsto ZM\) for every \(M\in Sp(2r,\mathbb R)\).

Used in: Notation, Def I.5, Thm II.5, Prop II.16, LOFT, Thm III.5, Thm V.3. Reading: Hall, Lie Groups, Lie Algebras, and Representations (2015), Ch. 1.

Definition D.2 (tori, diagonal monoids and saturation). The diagonal torus \(T_m=\{\mathrm{diag}(d):d_i\ne0\}\cong(\mathbb R^\times)^m\) is a commutative Lie group with \(2^m\) connected components, and \(T_m^+=\{\mathrm{diag}(d):d_i\gt0\}\cong\mathbb R_{\gt0}^m\) is the component of the identity. \(\mathrm{Diag}_m\), the set of all diagonal matrices including singular ones, is a monoid under multiplication (D.10) whose invertible elements form \(T_m\). Identifying \(\mathbb R^2\) with \(\mathbb C\), the scaled rotations \(a\,\mathrm{Rot}(\theta)\) with \(a\gt0\) form \(CO^+(2)\cong\mathbb C^\times\), through \(a\,\mathrm{Rot}(\theta)\mapsto ae^{i\theta}\). If a group or monoid \(G\) acts on a space containing a set \(X\), the saturation \(G\cdot X=\{g\cdot x:g\in G,\ x\in X\}\) is the smallest \(G\)-stable set containing \(X\).

(IA)\(^3\) on keys and values maps \(\ell\mapsto\mathrm{diag}(\ell)W_0\), so its image \(\mathrm{Diag}_m\cdot W_0\) is the closure of the torus orbit \(T_m\cdot W_0\). DoRA’s image is the saturation \(\mathrm{Diag}_m\cdot(W_0+\mathcal M_{\le r})\) of LoRA’s cone. RoAd\(_1\) is the left action of \(CO^+(2)^{m/2}\cong(\mathbb C^\times)^{m/2}\) on pairs of output rows, which makes it (IA)\(^3\) over \(\mathbb C\). LoHa’s gauge contains a two-sided diagonal torus.

Used in: Thm II.5, DoRA, Prop II.8, Prop II.9, Prop II.12, The hyperparameter torus.

Definition D.3 (permutations, monomial matrices, semidirect products, normalisers). \(S_d\) is the group of permutations of \(d\) objects, realised by permutation matrices. A matrix is monomial if each row and each column has exactly one nonzero entry. Every monomial matrix factors uniquely as \(DP\) with \(D\in T_m\) and \(P\) a permutation matrix, and \(T_m\) is normal in the group \(\mathrm{Mon}_m\) of monomial matrices, so \(\mathrm{Mon}_m=T_m\rtimes S_m\). Here a semidirect product \(N\rtimes H\) is a group with a normal subgroup \(N\) and a subgroup \(H\) such that \(N\cap H=\{e\}\) and \(NH\) is the whole group. The signed permutations \(B_r=\{\pm1\}^r\rtimes S_r=\mathrm{Mon}_r\cap O(r)\) are the monomial matrices with entries \(\pm1\), and \(\lvert B_r\rvert=2^rr!\). The normaliser of a subset \(H\subseteq G\) is \(N_G(H)=\{g\in G:gHg^{-1}=H\}\).

A monomial \(g\) satisfies \(g\,\mathrm{Diag}_m\,g^{-1}=\mathrm{Diag}_m\), and in \(GL_m\) the normaliser of \(T_m\) is exactly \(\mathrm{Mon}_m\). This puts \(\mathrm{Mon}_m\times GL_n\) inside the covariance group (D.17) of output scalings such as (IA)\(^3\) and DoRA’s magnitude. Adam acts coordinatewise, so on LoRA’s full-rank locus with channel-uniform hyperparameters (one learning rate, \(\varepsilon\) and decay per factor) it is equivariant, of LoRA’s gauge, only under \(B_r\subset O(r)\subset GL_r\) (Thm III.12). By Lemma VI.2, an invertible linear map that commutes with a non-affine \(C^2\) activation is monomial.

Used in: DoRA, Prop II.8, Prop II.9, Thm III.12, Cor III.13, Prop V.4, Lemma VI.2, The privileged basis.

Definition D.4 (connectedness and the identity component). The identity component \(G^\circ\) of a Lie group \(G\) is the connected component containing \(e\). It is an open normal subgroup. \(GL_n\) has two components, told apart by the sign of the determinant, and \(GL^+(n)=\{\det\gt0\}\) is the identity component; likewise \(SO(n)\subset O(n)\) and \(T_m^+\subset T_m\). A subset \(U\) is symmetric if \(U^{-1}=U\). Every neighbourhood of \(e\) contains a symmetric one, and in a connected group a symmetric neighbourhood \(U\) of \(e\) generates the group as a monoid, so every element is a finite product \(u_1\cdots u_k\) with \(u_i\in U\).

In merge-and-restart of a method of action type (D.12), each cycle applies one element of a set \(S\). If \(S\) contains a neighbourhood of the identity in a connected group \(H\), repeated merging reaches all of \(H\) (Thm V.3(b)). HF’s Cayley–Neumann OFT factors have \(\det R\ge0\), so the invertible ones lie in \(GL^+(b)\); they generate \(\prod GL^+(b)\) as a group, or \(\prod CO^+(2)\) when \(b=2\).

Used in: Complete invariants, Thm V.3, Corollaries of Thm V.3.

Definition D.5 (Lie algebras; the commutator; \(\mathfrak{gl}\), \(\mathfrak{so}\), \(\mathrm{Sym}\), \(\wedge\)). The Lie algebra of a matrix group \(G\subseteq GL_n\) is \(\mathfrak g=\{X:e^{tX}\in G\text{ for all }t\in\mathbb R\}\), the tangent space of \(G\) at \(I\). It is a vector space closed under the commutator \([X,Y]=XY-YX\). The site uses \(\mathfrak{gl}_r\) (all \(r\times r\) matrices, for \(GL_r\)), \(\mathfrak{so}(n)\) (skew matrices \(\Omega^\top=-\Omega\), for \(O(n)\) and \(SO(n)\)), the diagonal matrices (for \(T_m\)), \(\mathbb RI\oplus\mathfrak{so}(n)\) (for \(CO(n)\)) and \(\mathfrak{sp}(2r)=\{X:X^\top J+JX=0\}\). For a subspace \(V\subseteq\mathbb R^n\), \(\mathfrak{so}(V)\) consists of the skew maps of \(V\) extended by \(0\) on \(V^\perp\). The symmetric matrices \(\mathrm{Sym}_r\) do not form a Lie algebra, since \([S,S']\) is skew. Write \(v\wedge w=vw^\top-wv^\top\in\mathfrak{so}(n)\), \(U\wedge V\) for the span of the \(u\wedge v\) with \(u\in U\), \(v\in V\), and \(L_{ab}=e_a\wedge e_b=E_{ab}-E_{ba}\). Then \([L_{ac},L_{cb}]=L_{ab}\) for distinct \(a,b,c\).

For injective \(W_0\), at HRA’s paired initialisation \(u_{2i-1}=u_{2i}=w_i\) (\(i\le k=r/2\), with independent \(w_i\) and \(U=\mathrm{span}\{w_i\}\) of dimension \(k\)), the first-order image is \(S_1=W_0\cdot\{\Omega\in\mathfrak{so}(n):P_{U^\perp}\Omega P_{U^\perp}=0\}=W_0\cdot(U\wedge\mathbb R^n)\), a subspace of \(W_0\cdot\mathfrak{so}(n)\) of dimension \(kn-k(k+1)/2\) (C.13). LOFT trains a skew \(E\in\mathfrak{so}(r)\) inside a fixed \(r\)-dimensional input subspace. The bracket identity lets rotations of overlapping coordinate blocks generate rotations of their union (D.8).

Used in: Thm II.7, LOFT, LOFT fixes one leaf, Thm III.5, Thm V.3. Reading: Hall (2015), Ch. 3.

Definition D.6 (the exponential map; one-parameter subgroups). For \(X\in\mathfrak g\), the curve \(t\mapsto e^{tX}\) of matrix exponentials is a one-parameter subgroup, a smooth homomorphism \((\mathbb R,+)\to G\), and every continuous one-parameter subgroup has this form for a unique \(X\), its generator. The exponential map \(\mathfrak g\to G\) has derivative \(\mathrm{id}\) at \(0\), so it maps a neighbourhood of \(0\) diffeomorphically onto a neighbourhood of \(I\). It is onto for \(SO(n)\) and not onto for \(GL^+(n)\), since \(\mathrm{diag}(-1,-2)\) is \(e^X\) for no real \(X\).

The one-parameter subgroup \(t\mapsto(Be^{-tX},e^{tX}A)\) of LoRA’s gauge moves along a fibre without changing \(BA\). Thm III.5 builds its isometries as \(k=e^{\tau\xi}\) from generators \(\xi\) with \(\Lambda^{-1}\xi\) antisymmetric. LOFT may use the exponential as its map \(\mathfrak{so}(r)\to SO(r)\), and the Kempf–Ness argument (D.30) differentiates the norm along \(e^{t\xi}\).

Used in: Thm III.5, Prop III.7, LOFT, Thm V.3. Reading: Hall (2015), Ch. 2.

Definition D.7 (the Cayley map and its Neumann truncation). For skew \(\Omega\) the eigenvalues are imaginary, so \(I-\Omega\) is invertible, and the Cayley map is \[C(\Omega)=(I+\Omega)(I-\Omega)^{-1}.\] It lands in \(SO(n)\), and \(C(\Omega)-I=2\Omega(I-\Omega)^{-1}\), so \(\operatorname{rank}(C(\Omega)-I)=\operatorname{rank}\Omega\). Its expansion \(C(\Omega)=I+2\Omega+2\Omega^2+\cdots\) shows that its derivative at \(0\) is \(2\,\mathrm{id}\). It is a diffeomorphism from \(\mathfrak{so}(n)\) onto the open dense set of rotations with no eigenvalue \(-1\), with inverse \(R\mapsto(R-I)(R+I)^{-1}\), so it misses, for instance, \(\mathrm{diag}(-1,-1,1,\dots,1)\).

Exact-Cayley OFT and block OFT are orthogonal, so they preserve \(K_{\rm out}=WW^\top\) (D.15) and the spectrum. HF PEFT’s default (since version 0.18) truncates \((I-Q)^{-1}\) to \(I+Q+Q^2+Q^3\), which gives \(R=C(Q)(I-Q^4)\). As \(Q^4\) is symmetric and commutes with \(C(Q)\), \(R^\top R=(I-Q^4)^2\ne I\), and for small \(Q\) each merge shrinks \(K_{\rm out}\). The rank identity is how Thm II.7(c) identifies HRA’s image germ with the cone of skew matrices of rank at most \(r\).

Used in: The thesis, Cor II.6, Thm II.7, Figure 3, Corollaries of Thm V.3.

Definition D.8 (Lie subgroups and subalgebras; the generated Lie algebra). A Lie subgroup of \(G\) is a subgroup with a Lie group structure for which the inclusion is an injective immersion; it need not be closed. Connected Lie subgroups \(H\subseteq G\) correspond one to one to Lie subalgebras \(\mathfrak h\subseteq\mathfrak g\), with \(H\) the subgroup generated by \(e^{\mathfrak h}\). The Lie algebra generated by subalgebras \(\mathfrak h_1,\dots,\mathfrak h_p\), written \(\mathrm{Lie}\langle\mathfrak h_1,\dots,\mathfrak h_p\rangle\), is the smallest subalgebra containing them, the span of their elements and all iterated brackets of them.

Block OFT, BOFT factors and Givens rotations use \(\mathfrak{so}(I_j)\) for coordinate blocks \(I_j\subseteq\{1,\dots,n\}\). Since \([L_{ac},L_{cb}]=L_{ab}\), two overlapping blocks generate \(\mathfrak{so}\) of their union, and \(\mathrm{Lie}\langle\mathfrak{so}(I_j)\rangle_j=\bigoplus_C\mathfrak{so}(C)\), summed over the connected components \(C\) of the hypergraph with vertices \(\{1,\dots,n\}\) and hyperedges \(I_j\). Two vertices share a component when a chain of hyperedges, each meeting the next, joins them. Butterfly factors connect this hypergraph only at full depth.

Used in: Thm V.3, Figure 11, Corollaries of Thm V.3, LOFT fixes one leaf. Reading: Lee (2013), Ch. 19; Hall (2015), Ch. 5.

Fact D.9 (Yamabe’s theorem). Every path-connected subgroup of a Lie group is a Lie subgroup. Consequently, if \(H_1,\dots,H_p\) are connected Lie subgroups of \(G\), the subgroup they generate is the connected Lie subgroup \(H\) with \(\mathrm{Lie}(H)=\mathrm{Lie}\langle\mathfrak h_1,\dots,\mathfrak h_p\rangle\). (H. Yamabe, Osaka Math. J. 2, 1950; a short proof is in M. Goto, Proc. AMS 20, 1969.)

The subgroup generated by the \(H_i\) is path-connected, because products of paths are paths, so the theorem makes it a Lie subgroup and the correspondence of D.8 names it. With D.4, this is why a re-merged action method whose cycles contain a neighbourhood of the identity in each \(H_i\) reaches exactly the connected subgroup with Lie algebra \(\mathrm{Lie}\langle\mathfrak h_1,\dots,\mathfrak h_p\rangle\), and why exact-Cayley block-orthogonal factors reach \(\prod_CSO(C)\).

Used in: Thm V.3, Corollaries of Thm V.3.

Definition D.10 (monoids, units, monoid orbits, homomorphisms). A monoid is a set with an associative product and an identity. Its invertible elements form its group of units. The submonoid generated by a subset \(S\) consists of all finite products \(s_1\cdots s_k\) of elements of \(S\), the empty product being the identity. A monoid acts on a set when \(1\cdot x=x\) and \(s\cdot(t\cdot x)=(st)\cdot x\), and the monoid orbit of \(x\) is \(\{s\cdot x\}\). A homomorphism of monoids preserves products and the identity. Examples are \(\mathrm{Diag}_m\), \(M_2(\mathbb R)\) and \(\mathrm{End}(\Theta)\) under multiplication, with units \(T_m\), \(GL_2(\mathbb R)\) and \(GL(\Theta)\), and finite multisets under disjoint union \(\uplus\). Additive monoids \(\mathrm{Mon}(C)\) are in B.14.

Merge-and-restart reaches the orbit of the monoid that one cycle generates, which is \(\theta_0+\mathrm{Mon}(C)\) for an additive method \(\rho=\theta_0+\delta\) with shape \(C=\mathrm{Im}\,\delta\) (B.14) and \(\langle S\rangle\cdot\theta_0\) for a method of action type (D.12) whose cycle applies elements of \(S\). RoAd\(_2\)’s blocks range over \(M_2(\mathbb R)\). For a fixed query, attention’s pair (total weight, weighted sum of values) is a homomorphism from multisets of key–value pairs to \((\mathbb R_{\ge0}\times\mathbb R^{d_v},+)\) (D.34).

Used in: Thm V.3, Restarting, DoRA, Prop II.9, Thm VI.5, Attention.

Fact D.11 (Cartan–Dieudonné theorem; Scherk’s sharp form). For nonzero \(u\in\mathbb R^n\), the reflection \(H_u=I-2uu^\top/\|u\|^2\) fixes the hyperplane \(u^\perp\) and has \(\det H_u=-1\). Every \(R\in O(n)\) is a product of at most \(n\) reflections (Cartan–Dieudonné). In Scherk’s sharp form, if \(k=\operatorname{rank}(R-I)\), then \(R\) is a product of exactly \(k\) reflections and of no fewer. Hence \(\det R=(-1)^k\), and every \(R\in SO(n)\) is a product of \(\operatorname{rank}(R-I)\) reflections, an even number. (P. Scherk, Proc. AMS 1, 1950; E. Artin, Geometric Algebra, 1957, Ch. III.)

More: idea of the proof

A product of \(j\) reflections fixes an intersection of \(j\) hyperplanes, so \(\operatorname{rank}(R-I)\le j\). Conversely, pick \(x\) with \(Rx\ne x\) and put \(u=Rx-x\). Then \(H_u\) swaps \(Rx\) and \(x\) and fixes every fixed vector of \(R\), so \(H_uR\) fixes one more dimension, and induction on \(k\) finishes.

HRA multiplies \(r\) reflections, so \(\operatorname{rank}(R-I)\le r\), and \(\det R=1\) for even \(r\). Scherk’s form gives the converse. For injective \(W_0\) and even \(r\), HRA reaches all of \((W_0+\mathcal M_{\le r})\cap W_0\cdot SO(n)\), padding with pairs \(H_uH_u=I\), and re-merged HRA reaches \(W_0\cdot SO(n)\) within \(\lceil n/r\rceil\) cycles.

Used in: Thm II.7, HRA is a meet, Corollaries of Thm V.3.

Actions, orbits and invariants

Definition D.12 (group actions). A (left) action of a group \(G\) on a set \(X\) is a map \((g,x)\mapsto g\cdot x\) with \(e\cdot x=x\) and \(g\cdot(h\cdot x)=(gh)\cdot x\). A right action \(x\cdot g\) satisfies \(x\cdot(gh)=(x\cdot g)\cdot h\), and \(g\cdot x:=x\cdot g^{-1}\) turns it into a left one. The action is smooth if \(G\times X\to X\) is smooth, and linear if \(X\) is a vector space and each map \(x\mapsto g\cdot x\) is linear.

On weights, \(W\mapsto WR\) is a right action of \(O(n)\) on the input side and \(W\mapsto PW\) a left action of \(O(m)\) on the output side, and \((g,h)\cdot W=gWh^{-1}\) is a left action of \(GL_m\times GL_n\). On LoRA’s factors, \(g\cdot(B,A)=(Bg^{-1},gA)\) is a linear left action of \(GL_r\). A method is of action type when \(\rho(q)=\gamma(q)\cdot\theta_0\) for a map \(\gamma\) into a group acting on \(\Theta\), as for OFT and HRA. (This \(\gamma\) follows the exposés and is unrelated to the cotangent signal \(\gamma(t)\) of C.2 and D.26.) (IA)\(^3\) has the same form with the monoid \(\mathrm{Diag}_m\) in place of a group, and DoRA’s magnitude applies that monoid to LoRA’s weight.

Used in: Def I.2, Notation, Erlangen, Thm II.5, Thm III.5.

Definition D.13 (orbits, stabilisers, free actions). The orbit of \(x\) is \(G\cdot x=\{g\cdot x:g\in G\}\), its closure is the orbit closure, and its stabiliser is the closed subgroup \(G_x=\{g:g\cdot x=x\}\). For a smooth action of a Lie group, \(g\mapsto g\cdot x\) induces an injective immersion \(G/G_x\to X\) with image \(G\cdot x\), so the orbit is an immersed submanifold of dimension \(\dim G-\dim G_x\). The infinitesimal stabiliser \(\{\xi\in\mathfrak g:\xi\cdot x=0\}\) is the Lie algebra of \(G_x\). The action is free if every stabiliser is trivial, and the generic stabiliser is the stabiliser at a generic point, one in a suitable dense open set. Each map \(x\mapsto g\cdot x\) carries an orbit onto itself, so all its points look alike; orbits are homogeneous.

\(GL_r\) acts freely on pairs with \(\operatorname{rank}A=r\), since \(gA=A\) forces \(g=I\), so LoRA’s generic fibre, a single orbit, has dimension \(r^2\). LoHa’s joint gauge \(GL_{r_1}\times GL_{r_2}\times T_m\times T_n\) has a one-dimensional generic stabiliser, so its orbits have dimension \(r_1^2+r_2^2+m+n-1\). By homogeneity, the two-sided torus orbit \(\{\mathrm{diag}(a)W_0\mathrm{diag}(b)\}\) has dimension \(m+n-c(W_0)\) at each of its points, with \(c(W_0)\) the number of connected components of the support graph, the bipartite graph on the \(m\) rows and \(n\) columns with an edge \(\{i,j\}\) where \((W_0)_{ij}\ne0\). The images of (IA)\(^3\) and RoAd\(_1\) are orbit closures.

Used in: Erlangen, Thm II.5, Prop II.9, The image germ, Prop II.12, Rosetta entries 6 and 11. Reading: Lee (2013), Ch. 7 and 21.

Definition D.14 (Haar measure; invariant measures on homogeneous spaces). A compact Lie group \(K\) carries a unique invariant probability measure, its Haar measure. If \(K\) acts transitively on \(X\), so that \(X\cong K/K_x\) is a homogeneous space, then \(X\) carries a unique \(K\)-invariant probability measure, the image of Haar measure under \(k\mapsto k\cdot x\). The Grassmannian \(\mathrm{Gr}(r,n)\) of \(r\)-dimensional subspaces of \(\mathbb R^n\) (C.15) is a homogeneous space of \(O(n)\).

If \(A_0\in\mathbb R^{r\times n}\) has i.i.d. Gaussian entries, its law is unchanged by \(A_0\mapsto A_0R\) for \(R\in O(n)\), and \(A_0\) has rank \(r\) almost surely. So \(\mathrm{row}\,A_0\) is an \(O(n)\)-invariant random point of \(\mathrm{Gr}(r,n)\), and by uniqueness it is Haar-distributed. Vanilla LoRA with a Gaussian \(A_0\) therefore picks a Haar-random point of \(\mathrm{Gr}(r,n)\), the space that classifies its stratum \(\mathrm T_A\). Kaiming-uniform entries are not rotation invariant, so under HF’s default the point is not exactly Haar-distributed.

Used in: Three strata, Thm II.2. Reading: Mattila, Geometry of Sets and Measures in Euclidean Spaces (1995), Ch. 3.

Definition D.15 (invariants and complete invariants). A map \(f:X\to Y\) is invariant under an action if \(f(g\cdot x)=f(x)\) for all \(g\) and \(x\), so it is constant on orbits. It is a complete invariant if moreover \(f(x)=f(x')\) implies \(x'\in G\cdot x\); it then identifies the orbit space \(X/G\) with the image of \(f\).

action on \(W\in\mathbb R^{m\times n}\)	complete invariant
\(W\mapsto WR\), \(R\in O(n)\) (input side)	\(K_{\rm out}=WW^\top\)
\(W\mapsto PW\), \(P\in O(m)\) (output side)	\(K_{\rm in}=W^\top W\)
\(W\mapsto PWR\), \(P\in O(m)\), \(R\in O(n)\)	singular values with multiplicity
\(W\mapsto DW\), \(D\in T_m^+\), \(W\) without zero rows	the rays \(\mathbb R_{\gt0}w_i\) of the rows
\(W\mapsto E\odot W\), \(E\in(\mathbb R^\times)^{m\times n}\)	the zero pattern

If a method’s image lies in one orbit, every invariant of that action stays fixed through training and merging. This is why exactly orthogonal methods never change \(K_{\rm out}\) or the spectrum (Cor II.6). Isomorphism invariants of methods, such as \(S_1\), \(d(M)\) and the first-order deficit, play the same role one level up, since methods with different values cannot be isomorphic.

Used in: Thm II.5, Complete invariants, Figure 3, Cor II.6, Obs I.6, Seven borrowed words.

Fact D.16 (first fundamental theorem for \(O(n)\); Witt’s extension theorem). (First fundamental theorem.) Every polynomial in the rows \(w_1,\dots,w_m\in\mathbb R^n\) of \(W\) that is invariant under \(W\mapsto WR\), \(R\in O(n)\), is a polynomial in the inner products \(\langle w_i,w_j\rangle\), the entries of \(WW^\top\). For a compact group acting linearly, invariant polynomials separate orbits, so \(WW^\top\) is a complete invariant. (Witt’s extension theorem, Euclidean case.) A linear isometry between two subspaces of \(\mathbb R^n\) extends to an element of \(O(n)\). (H. Weyl, The Classical Groups, 1939, Ch. II; Goodman and Wallach, Symmetry, Representations, and Invariants, 2009, Ch. 5.)

More: Witt gives the orbit separation directly

If \(WW^\top=W'W'^\top\), the map \(\sum_ic_iw_i\mapsto\sum_ic_iw_i'\) is well defined and isometric on the row spaces, because \(\|\sum_ic_iw_i\|^2=c^\top WW^\top c\). Extend it by an isometry between the orthogonal complements to \(R^\top\in O(n)\); then \(W'=WR\).

This is item (1) of Thm II.5, the reason exactly orthogonal input-side methods are pinned to \(K_{\rm out}(W_0)\). Applied to \(W^\top\), it gives \(K_{\rm in}\) for the output side.

Used in: Thm II.5, Known results.

Definition D.17 (Klein’s Erlangen programme). Klein (1872) proposed to define a geometry by a space together with a group acting on it, and to study the properties the group preserves. Euclidean geometry keeps lengths and angles, the invariants of rigid motions. Affine geometry keeps parallelism and ratios along parallel lines but not lengths or angles, because its group is larger. A larger group leaves fewer invariants.

For a group-type method the reading is a theorem, because the image lies in one orbit and so the method can change nothing the group preserves. For a cone such as LoRA’s image the word only names its covariance group, the largest \(\mathcal G_M\subseteq GL_m\times GL_n\) with \(\mathrm{Im}\,M(gW_0h^{-1})=g\,\mathrm{Im}\,M(W_0)\,h^{-1}\) (D.18). For LoRA this is all of \(GL_m\times GL_n\), for (IA)\(^3\) on keys and values it is \(\mathrm{Mon}_m\times GL_n\), and for the PiSSA split it is \(CO(m)\times CO(n)\).

Used in: Erlangen, What “Erlangen” means here, Def II.14, The borrowed words, Seven borrowed words. Reading: Sharpe, Differential Geometry: Cartan’s Generalization of Klein’s Erlangen Program (1997), Ch. 4.

Definition D.18 (equivariance and invariance of maps, update rules and merge rules). A map \(F:X\to Y\) between spaces with \(G\)-actions is equivariant if \(F(g\cdot x)=g\cdot F(x)\), and invariant if \(F(g\cdot x)=F(x)\). An update rule on parameters is equivariant under a subgroup \(H\) of the gauge if, started from \(h\cdot q\) with correspondingly transformed optimizer state, it produces \(h\) applied to the iterates started from \(q\). A rule that uses randomness is equivariant in distribution if \(F(g\cdot x)\) and \(g\cdot F(x)\) have the same law for every \(x\).

On LoRA’s full-rank locus, with channel-uniform hyperparameters, the largest such subgroup of \(GL_r\) is \(O(r)\) for SGD, \(B_r\) for Adam and all of \(GL_r\) for undamped scaled GD (D.23, Thm III.12). An equivariant rule started from gauge-related points traces the same weights, since \(\rho(h\cdot q)=\rho(q)\). Merge rules should be invariant under each adapter’s gauge. Task arithmetic \(\sum_iw_is_iB_iA_i\) is, while LoraHub’s \((\sum_iw_iB_i)(\sum_jw_jA_j)\) is not. On a weight space \(\mathbb R^N\), DARE’s random drop-and-rescale is equivariant under the invertible diagonal matrices \(T_N\) for each fixed mask, and under the monomial matrices \(\mathrm{Mon}_N\) (D.3) in distribution.

Used in: Def II.14, Obs II.15, Thm III.12, Cor III.13, Figure 9, Prop V.4.

Definition D.19 (multihomogeneous maps; multidegree). Split the parameters into blocks \(q=(q_1,\dots,q_k)\). A map \(\delta\) is multihomogeneous of multidegree \((d_1,\dots,d_k)\) if \(\delta(c_1q_1,\dots,c_kq_k)=c_1^{d_1}\cdots c_k^{d_k}\,\delta(q)\) for all real \(c_i\). LoRA’s \(BA\) has multidegree \((1,1)\), and LoHa’s \((B_1A_1)\odot(B_2A_2)\) has multidegree \((1,1,1,1)\).

If some \(d_i\ge1\), the image of \(\delta\) contains \(0\) and is closed under multiplication by every nonnegative scalar, so it is a cone with vertex \(0\) (C.10). For a method \(\rho=\theta_b+s\,\delta\) started where \(\delta(q_0)=0\), so that \(\theta_b=\theta_0\), the image is the cone \(\theta_0+s\,\mathrm{Im}\,\delta\) with vertex \(\theta_0\). For \(c\in\mathbb R^k_{\gt0}\), the change \(q_i\mapsto c_iq_i\) together with \(s\mapsto s\prod_ic_i^{-d_i}\) leaves \(\rho\) unchanged. That rescaling is an isomorphism of methods and the source of the hyperparameter torus (D.20).

Used in: The image germ, Thm III.10, Seven axes.

Definition D.20 (orbit space of a positive torus). Let \(\mathbb R^k_{\gt0}\) act on positive variables \(x\in\mathbb R^N_{\gt0}\) by \(x_j\mapsto x_j\prod_ic_i^{a_{ji}}\), with a real weight matrix \(a=(a_{ji})\). In logarithms this is translation by the column space of \(a\), so the orbits are parallel affine subspaces and the orbit space is a vector space of dimension \(N-\operatorname{rank}a\). The monomials \(\prod_jx_j^{b_j}\) with \(b^\top a=0\) are invariant, and a basis of such vectors \(b\) gives a complete set of invariants.

Under the hypotheses of Thm III.10, taken with Adam at \(\varepsilon=0\), no weight decay or clipping and a shared schedule (otherwise the per-block \(\varepsilon\), decay and clipping thresholds rescale as well), the torus acts on LoRA’s hyperparameters by \(s\mapsto s/(c_Ac_B)\), \(\eta_A\mapsto c_A\eta_A\), \(\eta_B\mapsto c_B\eta_B\), \(\sigma_A\mapsto c_A\sigma_A\), with \(\sigma_A\) the scale of \(A\)’s initialisation, without changing the weight trajectory. The invariants are \(s\eta_A\eta_B\) and \(\sigma_A/\eta_A\), so only two of the four knobs matter. LoRA+ with ratio \(\lambda\) replaces \(\eta_B\) by \(\lambda\eta_B\), which changes the invariants exactly as replacing \(\alpha\) by \(\lambda\alpha\) does. Under SGD the rates scale by \(c_i^2\), and the invariants become \(s^2\eta_A\eta_B\) and \(\sigma_A^2/\eta_A\).

Used in: Thm III.10, Cor III.11, Figure 7, Rosetta entry 23, Prediction X.1.

Gauge, quotients and metrics

Definition D.21 (gauge symmetry of a parametrisation). The gauge group of a method \(\rho:Q\to\Theta\) is the group of diffeomorphisms \(\varphi\) of \(Q\) with \(\rho\circ\varphi=\rho\). If a Lie group \(G\) acts on \(Q\) with \(\rho(g\cdot q)=\rho(q)\), each orbit lies inside a fibre \(\rho^{-1}(\theta)\) (C.6), so \(d(M)\le\lvert M\rvert-\dim(\text{generic orbit})\). The action is locally transitive on a fibre when its orbits are open in the fibre, that is, when orbits and fibre have the same dimension there; on generic fibres this gives \(d(M)=\lvert M\rvert-\dim(\text{generic orbit})\), and the site calls such a \(G\) the generic gauge. Here the word means only this symmetry group; no connection is involved.

LoRA’s \(GL_r\) acts freely and locally transitively on full-rank fibres, so \(d=r(m+n)-r^2=r(m+n-r)\), and \(r^2\) parameters are gauge waste. HRA’s generic fibres have dimension \(r(r+1)/2\), while its \(r\) scalings \(u_i\mapsto c_iu_i\) sweep out only \(r\) of those dimensions, so the scalings do not fill the fibres; Thm II.7(a) describes the rest of the gauge as non-linear. For \(r=2\) the fibre through \((u_1,u_2)\) has dimension \(3\), made up of the two scalings and of rotating \(u_1\) and \(u_2\) by a common angle inside their plane, which keeps the angle between them and hence \(H_{u_1}H_{u_2}\), the rotation of that plane by twice the angle. The plane depends on the point, so this move is not a linear action on parameters. The Hurwitz moves \((u_i,u_{i+1})\mapsto(H_{u_i}u_{i+1},u_i)\) preserve the product of reflections, since \(H_aH_b=H_{H_ab}H_a\), and they satisfy the braid relations, so the braid group on \(r\) strands, unrelated to the signed permutations \(B_r\), acts on HRA’s parameters. These moves are discrete and add no dimension to the fibre.

Used in: Def I.5, Invariants of a method, Notation, Rank and dimension, Prop II.12, Thm II.7, Noether; gauge, Seven borrowed words.

Fact D.22 (quotients and descent; the quotient manifold theorem). For an action of \(G\) on \(X\), the orbit space \(X/G\) is the set of orbits, with the map \(\pi:x\mapsto G\cdot x\). A map \(f:X\to Y\) that is constant on orbits factors uniquely as \(f=\bar f\circ\pi\); this is descent. (Quotient manifold theorem.) If a Lie group acts smoothly, freely and properly on a manifold \(X\), then \(X/G\) is a manifold of dimension \(\dim X-\dim G\), \(\pi\) is a submersion, and \(\bar f\) is smooth whenever \(f\) is. Properness means that \((g,x)\mapsto(g\cdot x,x)\) is a proper map, one under which preimages of compact sets are compact. (Lee, 2013, Thm 21.10.)

\(GL_r\) acts freely and properly on pairs \((B,A)\) of full rank, and \((B,A)\mapsto BA\) identifies the quotient with \(\mathcal M_r\), of dimension \(r(m+n)-r^2\). A rule for combining LoRA factors is a function of the adapters exactly when it is constant on the fibres of \((B_i,A_i)\mapsto(B_iA_i)\), and a cometric on factors that depends only on \(W\) descends to \(\mathcal M_r\).

Used in: Prop V.4, Prop III.4, Obs II.15, When a change of chart does not matter.

Definition D.23 (metrics, cometrics and the pulled-back cometric). A metric on a vector space \(V\) is a positive-definite inner product, equivalently a symmetric positive-definite map \(g:V\to V^*\), that is, \(\langle gv,v\rangle\gt0\) for \(v\ne0\). A cometric is a symmetric positive semidefinite map \(K:V^*\to V\). It turns a covector, such as a gradient \(\nabla_\theta L\), into a velocity \(-K\nabla_\theta L\). A metric gives the cometric \(K=g^{-1}\), while a cometric may be degenerate. Metrics pull back along immersions, and cometrics push forward along any smooth map. For instance, gradient flow \(\dot q=-\Lambda\nabla_q(L\circ\rho)\) moves the weights by \(\dot\theta=-K_q\nabla_\theta L\), with \[K_q=D\rho_q\,\Lambda\,D\rho_q^\top.\]

For LoRA with block-scalar rates, one rate per factor, \(\Lambda=\mathrm{diag}(\eta_BI,\eta_AI)\) on \((B,A)\) and \(K(G)=s^2(\eta_BGA^\top A+\eta_ABB^\top G)\). Two points of one fibre give the same weight velocity for every loss exactly when their cometrics agree. At points where \(A\) has full row rank and \(r\lt n\), this holds for LoRA’s linear gauge and plain gradient descent exactly along \(O(r)\). A rule descends to a cometric on weights when it gives the same weight velocity at all points of each fibre; with a point-dependent preconditioner in place of \(\Lambda\), undamped scaled GD and the SGD variant of LoRA-Pro do, on the full-rank locus. Scaled GD multiplies the \(B\)-gradient by \((AA^\top)^{-1}\) and the \(A\)-gradient by \((B^\top B)^{-1}\), and undamped means that no ridge term \(\delta I\) is added before inverting (Prop III.4).

Used in: Def III.1, Obs III.2, Thm III.3, Prop III.4, The metric a method carries, Three invariants.

Definition D.24 (Riemannian gradient flow; tangent projection; retraction). On a submanifold \(N\subseteq\Theta\) with the metric induced by the Frobenius inner product, the Riemannian gradient of \(L|_N\) at \(\theta\) is the orthogonal projection \(\Pi_{T_\theta N}\nabla L(\theta)\) of the ambient gradient onto the tangent space, and the Riemannian gradient flow is \(\dot\theta=-\Pi_{T_\theta N}\nabla L(\theta)\). A discrete step along the tangent space leaves \(N\), so it is followed by a retraction, a smooth map \(R_\theta\) from a neighbourhood of \(0\) in \(T_\theta N\) to \(N\) with \(R_\theta(0)=\theta\) and \(DR_\theta(0)=\mathrm{id}\).

At \(W=W_0+X\) with \(X\in\mathcal M_r\), \(U=\mathrm{col}\,X\) and \(V=\mathrm{row}\,X\), the tangent space is \(\{D:P_{U^\perp}DP_{V^\perp}=0\}\) (C.12) and the projection is \(P_UG+GP_V-P_UGP_V\). This is LoRA-Pro’s equivalent gradient, so the SGD variant of LoRA-Pro follows the Riemannian gradient flow on \(W_0+\mathcal M_r\). Truncating the SVD of \(X+\xi\) to rank \(r\) is a retraction onto \(\mathcal M_r\), the one Riemannion uses.

Used in: Prop III.4, When a change of chart does not matter, Thm III.12. Reading: Absil, Mahony and Sepulchre, Optimization Algorithms on Matrix Manifolds (2008), Ch. 3 and 4.

Fact D.25 (distributions, Lie brackets and Frobenius’ theorem). A distribution \(D\) of rank \(k\) on an open set \(U\) assigns to each point \(\theta\) a \(k\)-dimensional subspace \(D_\theta\) of directions, smoothly in \(\theta\); the word is unrelated to probability distributions. The Lie bracket of vector fields is \([X,Y]=DY\cdot X-DX\cdot Y\); flowing along \(X\), \(Y\), \(-X\), \(-Y\) for time \(t\) misses the starting point by \(t^2[X,Y]+O(t^3)\). \(D\) is involutive if \([X,Y]\) lies in \(D\) whenever \(X\) and \(Y\) do. (Frobenius.) \(D\) is involutive iff every point lies on a \(k\)-dimensional integral manifold, a submanifold whose tangent space at each of its points \(\theta\) is \(D_\theta\). A maximal connected integral manifold is a leaf. (Lee, 2013, Ch. 19.) The Frobenius algebras of B.6 and the Frobenius norm share only the name.

On \(\mathbb R^3\), \(D=\mathrm{span}\{\partial_x,\ \partial_y+x\,\partial_z\}\) is not involutive, since \([\partial_x,\partial_y+x\,\partial_z]=\partial_z\notin D\), so no surface is tangent to \(D\). A backward method with anchor \(a(\theta)\) chooses \(D_\theta=\mathrm{im}\,a(\theta)\), and it reads as a forward method only along integral manifolds. GaLore with a projector fixed on each period, LISA and MeZO have constant distributions, which are involutive with affine leaves. Full-batch GaLore with \(T=1\), read as \(D_W=\mathrm{Hom}(\mathbb R^n,U_r(\nabla L(W)))\) wherever a singular-value gap exists, already fails to be involutive for \(L=\tfrac12\|W\|_F^2\).

Used in: When a backward method is a forward one, Prop IV.3.

Conservation and balance

Fact D.26 (Noether’s theorem for gradient flows). Let a Lie group act linearly on \(Q=\mathbb R^N\) with \(\rho(k\cdot q)=\rho(q)\), let \(\Lambda\succ0\) be a constant rate matrix, and let \(\gamma(t)\) be any time-dependent covector on \(\Theta\), a gradient or not. Along every solution of \(\dot q=-\Lambda\,D\rho_q^\top\gamma(t)\), and for every generator \(\xi\) of the action with \(\Lambda^{-1}\xi\) symmetric, the charge \[J_\xi(q)=\tfrac12\langle q,\Lambda^{-1}\xi\,q\rangle\] is constant. (Kunin et al., 2021; Zhao et al., 2023; Marcotte, Gribonval and Peyré, 2023.)

More: the one-line proof

Invariance of \(\rho\) gives \(D\rho_q(\xi q)=0\). Hence \(\tfrac{d}{dt}J_\xi=\langle\Lambda^{-1}\xi q,\dot q\rangle=-\langle\xi q,D\rho_q^\top\gamma\rangle=-\langle D\rho_q\,\xi q,\gamma\rangle=0\).

For scalar LoRA \(w=w_0+ba\) with rates \(\eta_B,\eta_A\), the charge \(b^2/\eta_B-a^2/\eta_A\) is conserved, so every trajectory runs along a hyperbola, the Noether hyperbolas of Figure 8. Noether (1918) derived conservation laws from symmetries of a Lagrangian; this version needs no Lagrangian and holds for any loss and data. For LoRA with rates \(\eta_B,\eta_A\), the generators \(\xi_X(B,A)=(BX,-XA)\) pass the test exactly for symmetric \(X\), and the \(r(r+1)/2\) charges are the entries of \(\Phi=B^\top B/\eta_B-AA^\top/\eta_A\). Minibatches, LoRA dropout, global-norm clipping and DoRA’s detached norm keep the required form. Momentum and Adam do not, and under discrete SGD the charge drifts, by an amount relative to the charge of order \(\eta^2\) per step.

Used in: Thm III.5, Isometries and charges, Noether; gauge, Figure 8, Things you can check, Prediction X.8, Seven borrowed words.

Definition D.27 (Cartan decomposition; polar decomposition). The map \(\vartheta(\xi)=-\xi^\top\) is an automorphism of \(\mathfrak{gl}_r\) of order two, a Cartan involution. Its \(+1\) and \(-1\) eigenspaces give the Cartan decomposition \(\mathfrak{gl}_r=\mathfrak{so}(r)\oplus\mathrm{Sym}_r\), \(\xi=\tfrac12(\xi-\xi^\top)+\tfrac12(\xi+\xi^\top)\), with \([\mathfrak{so},\mathfrak{so}]\subseteq\mathfrak{so}\), \([\mathfrak{so},\mathrm{Sym}]\subseteq\mathrm{Sym}\) and \([\mathrm{Sym},\mathrm{Sym}]\subseteq\mathfrak{so}\). Its group version is the polar decomposition, which says that \((k,S)\mapsto k\,e^{S}\) is a diffeomorphism \(O(r)\times\mathrm{Sym}_r\to GL_r\). Hence \(g\mapsto g^\top g\) identifies the cosets \(O(r)g\) with the positive-definite matrices, \(O(r)\backslash GL_r\cong\mathrm{Sym}^+_r\).

With block-scalar rates (D.23), the antisymmetric generators \(X\in\mathfrak{so}(r)\) of LoRA’s gauge act by rotations \((BO^\top,OA)\) that preserve the rate metric, so the optimizer cannot feel them; they are isometries. The symmetric generators \(X\in\mathrm{Sym}_r\) give the conserved charges of D.26. For general rates the split needs \(\mathfrak g\) to be closed under \(\xi\mapsto\Lambda\xi^\top\Lambda^{-1}\). Gradient descent on LoRA is thus a family of dynamics indexed by \(O(r)\backslash GL_r\), and the Kempf–Ness argument needs only curves \(e^{t\xi}\) with \(\xi\) symmetric.

Used in: Thm III.5, Isometries and charges, Three invariants, The thesis, Prop III.7, When a change of chart does not matter. Reading: Knapp, Lie Groups Beyond an Introduction (2002), §VI.2; Horn and Johnson, Matrix Analysis (2013), §7.3.

Definition D.28 (balancedness; the conserved \(B^\top B-AA^\top\)). A factorisation \(W=BA\) is balanced if \(B^\top B=AA^\top\), and rate-weighted balanced if \(\Phi=B^\top B/\eta_B-AA^\top/\eta_A=0\), that is, \(B^\top B=cAA^\top\) with \(c=\eta_B/\eta_A\). For a deep linear network \(W=W_L\cdots W_1\) the same word means \(W_{j+1}^\top W_{j+1}=W_jW_j^\top\) for every \(j\).

Under gradient flow with block-scalar rates, \(\Phi\) is conserved (D.26), so rate-weighted balancedness, once it holds, holds forever; at equal rates the two notions agree. Zero-product starts \(B_0A_0=0\) have \(\langle B_0^\top B_0,A_0A_0^\top\rangle_F=0\), which makes them maximally unbalanced for given norms of the two Gram matrices, since the inner product of two positive semidefinite matrices is never negative. From \(B_0=0\), conservation gives \(B^\top B=\tfrac{\eta_B}{\eta_A}(AA^\top-A_0A_0^\top)\), so \(AA^\top\) never drops below \(A_0A_0^\top\) in the Loewner order. HF’s PiSSA starts with \(B_0^\top B_0=A_0A_0^\top\). When \(\Phi\) is isotropic, singular values of \(\Delta W\) well above the crossover scale \(\sigma^*\) of Thm III.9, set by the charge, the scale and the rates, follow the balanced dynamics of Arora, Cohen and Hazan.

Used in: Isometries and charges, Prop III.7, Prop III.8, Thm III.9, References. Reading: Arora, Cohen and Hazan (ICML 2018); Du, Hu and Lee (NeurIPS 2018).

Definition D.29 (the moment map). Let a group \(G\subseteq GL(V)\) that is closed under transpose act on a real inner-product space \(V\), and split \(\mathfrak g=\mathfrak k\oplus\mathfrak p\) into its antisymmetric and symmetric elements. The (real) moment map \(\mu:V\to\mathfrak p\) is defined by \[\langle\mu(v),\xi\rangle=\tfrac12\tfrac{d}{dt}\Big|_{t=0}\|e^{t\xi}v\|^2=\langle\xi v,v\rangle\qquad(\xi\in\mathfrak p).\] It records how the norm changes along the symmetric directions of the orbit; the antisymmetric directions preserve the norm.

For LoRA’s gauge, \(\tfrac{d}{dt}\big|_{0}\big(\|Be^{-tX}\|^2+\|e^{tX}A\|^2\big)=-2\operatorname{tr}(X\mu)\) for symmetric \(X\), with \(\mu=B^\top B-AA^\top\), so this \(\mu\) is the moment map up to sign. Its zero set is the balanced slice, and at equal rates \(\eta_A=\eta_B=\eta\) it equals \(\eta\Phi\), a multiple of the Noether charge.

Used in: The balanced slice, Prop III.7, Cor III.6, Figure 8.

Fact D.30 (Kempf–Ness theorem, real form). Let \(G\subseteq GL(V)\) be a real algebraic group, cut out by polynomial equations, that is closed under transpose (such a group is reductive), for instance \(GL_r\) acting on LoRA factors, and let \(K=G\cap O(V)\). For \(v\in V\), (i) \(g=e\) is a critical point of \(g\mapsto\|g\cdot v\|^2\) iff \(\mu(v)=0\), and then \(v\) has least norm on \(G\cdot v\); (ii) \(G\cdot v\) is closed iff it contains a point with \(\mu=0\); (iii) on a closed orbit, the points with \(\mu=0\) form a single \(K\)-orbit; (iv) the closure of every orbit contains exactly one closed orbit. (Kempf and Ness, 1979; Birkes, 1971; Richardson and Slodowy, 1990.)

More: idea of the proof

For symmetric \(\xi\), \(\|e^{t\xi}v\|^2=\sum_ie^{2t\lambda_i}\|v_i\|^2\) in an eigenbasis of \(\xi\), which is convex in \(t\). By the polar decomposition every orbit point is \(k\,e^{\xi}v\) with \(k\in K\) norm-preserving, so critical points are minima and two minima differ by an element of \(K\).

For \(m=n=r=1\), \(t\in\mathbb R^\times\) acts by \(t\cdot(b,a)=(b/t,ta)\) and \(\mu=b^2-a^2\). The orbits are the hyperbolas \(ba=c\ne0\), which are closed and meet \(\mu=0\) in the two points \(\pm(b,a)\) with \(\lvert b\rvert=\lvert a\rvert\), one \(O(1)\)-orbit; the two punctured axes, which are not closed and avoid \(\mu=0\); and the origin. For LoRA in general, \(K=O(r)\), and a full-rank orbit is closed, being a whole fibre of \((B,A)\mapsto BA\). The least-norm factorisations of a fixed \(BA\) are the balanced ones, they form one \(O(r)\)-orbit, and there \(\|B\|_F^2+\|A\|_F^2=2\|BA\|_*\). Weight decay adds a norm to the loss (Cor III.6). With learning-rate-coupled decay, which multiplies the decay of each factor by its rate, \(\dot\Phi=-2\lambda\mu\), so the equilibria lie on \(\mu=0\), and \(\mu\) decays exponentially at equal rates. With per-time decay, the same \(\lambda\) for both factors, \(\dot\Phi=-2\lambda\Phi\), so \(\Phi\to0\).

Used in: The balanced slice, Prop III.7, Cor III.6, Combining, Known results, Atlas.

Fact D.31 (the closed orbit in each fibre of \((B,A)\mapsto BA\)). For \(GL_r\) acting on factor pairs, fix \(W\) of rank \(k\le r\). Among the pairs with \(BA=W\), those with \(\operatorname{rank}B=\operatorname{rank}A=k\) form a single orbit. It is closed, and it lies in the closure of every orbit in the fibre \(\{BA=W\}\). Consequently a continuous \(GL_r\)-invariant function is constant on each fibre of \((B,A)\mapsto BA\). For a complex reductive group acting on a vector space, every fibre of the quotient map likewise contains exactly one closed orbit (Mumford, Fogarty and Kirwan, Geometric Invariant Theory, 1994, Ch. 1).

For \(m=n=r=1\) (D.30), the fibre \(\{ba=0\}\) consists of the origin and the two punctured axes, and the origin, its only closed orbit, lies in the closure of both axes. This is the step behind Prop V.4(a). A continuous rule for combining LoRA factors that is invariant under each adapter’s \(GL_r\) is automatically a function of the adapters \(B_iA_i\), also off the full-rank locus. Continuity matters. The projector onto \(\mathrm{row}\,A\), defined where \(\operatorname{rank}A=r\), is invariant, yet it differs at \((0,A)\) and \((0,A')\), which have the same product \(0\).

Used in: Prop V.4, Combining.

Matrix and convexity tools

Fact D.32 (Ky Fan’s maximum principle). Let \(S\in\mathbb R^{n\times n}\) be symmetric with eigenvalues \(\lambda_1\ge\dots\ge\lambda_n\), and let \(1\le k\le n\). Then \[\max\{\operatorname{tr}(PSP^\top):P\in\mathbb R^{k\times n},\ PP^\top=I_k\}=\lambda_1+\dots+\lambda_k,\] attained when the rows of \(P\) span a top-\(k\) eigenspace. For any matrix \(X\) with singular values \(\sigma_1\ge\sigma_2\ge\cdots\), applying this to \(S=X^\top X\) gives \(\max_P\|XP^\top\|_F^2=\sigma_1^2+\dots+\sigma_k^2\), and for square \(X\) also \(\|PXP^\top\|_F^2\le\|XP^\top\|_F^2\le\sigma_1^2+\dots+\sigma_k^2\). (K. Fan, PNAS 35, 1949; Bhatia, Matrix Analysis, 1997, Ch. III.)

Among \(A_0\) with \(A_0A_0^\top=cI\), LoRA’s initial rate \(\|GA_0^\top\|_F^2\) under gradient flow is largest when \(\mathrm{row}\,A_0\) is the top-\(r\) right singular space of \(G\), which is LoRA-GA’s choice for \(A_0\). For LOFT, the squared norm of the pulled-back gradient \(P_r\operatorname{skew}(W_0^\top G)P_r^\top\) is at most \(2\sum_{k\le\lfloor r/2\rfloor}\mu_k^2\), where \(\pm i\mu_k\) are the eigenvalues of the skew matrix \(\operatorname{skew}(W_0^\top G)\), whose singular values are \(\mu_1,\mu_1,\mu_2,\mu_2,\dots\). The floor appears because a skew matrix compressed to an odd-dimensional subspace has a zero eigenvalue, and Cauchy interlacing bounds the rest.

Used in: LOFT, Prop II.3.

Fact D.33 (Sylvester equations). For \(S\in\mathbb R^{p\times p}\), \(T\in\mathbb R^{q\times q}\) and \(C\in\mathbb R^{p\times q}\), the Sylvester equation \(SX+XT=C\) has a unique solution \(X\) for every \(C\) iff \(S\) and \(-T\) have no eigenvalue in common, since the eigenvalues of \(X\mapsto SX+XT\) are the sums \(\lambda_i(S)+\lambda_j(T)\). If \(S\) and \(T\) are symmetric positive definite, this holds and \(X=\int_0^\infty e^{-tS}Ce^{-tT}\,dt\). The Bartels–Stewart algorithm solves the equation through Schur forms of \(S\) and \(T\). (Horn and Johnson, 1991, Ch. 4.)

LoRA-Pro’s equivalent gradient leaves a free \(r\times r\) matrix \(X\). The published method picks it by solving \(B^\top B\,X+X\,AA^\top=C\), with \(C\) built from the gradient. On the full-rank locus both Gram matrices are positive definite, so \(X\) exists and is unique. The first-order weight velocity does not depend on \(X\), while the resulting parameter update is only \(O(r)\)-equivariant.

Used in: Thm III.12, When a change of chart does not matter, Figure 9.

Definition D.34 (the perspective map; convex combinations). The perspective map is \(\pi(Z,S)=S/Z\) on \(\{(Z,S)\in\mathbb R\times\mathbb R^d:Z\gt0\}\). It is constant on rays, \(\pi(tZ,tS)=\pi(Z,S)\) for \(t\gt0\), and it turns sums into convex combinations, \[\pi(Z_1+Z_2,S_1+S_2)=\lambda\,\pi(Z_1,S_1)+(1-\lambda)\,\pi(Z_2,S_2),\qquad\lambda=\frac{Z_1}{Z_1+Z_2}.\]

For a fixed query \(q\), attention is \(\mathrm{Attn}_q(K)=\pi(\Sigma_q(K))\), where \(\Sigma_q(K)=\big(\sum e^{\langle q,k\rangle/\sqrt d},\ \sum e^{\langle q,k\rangle/\sqrt d}\,v\big)\) is a monoid homomorphism on multisets of key–value pairs (D.10). With a prefix \(P\) before the content \(C\), the identity above gives \(\mathrm{Attn}_q(P\uplus C)=(1-\lambda_q)\mathrm{Attn}_q(C)+\lambda_q\mathrm{Attn}_q(P)\), with \(\mathrm{Attn}_q(P)\) in the convex hull \(\mathrm{conv}\,V_P\) of the prefix values, so a prefix acts as a parallel branch gated by its share of the mass.

Used in: Thm VI.5, Attention, Figure 13. Reading: Boyd and Vandenberghe, Convex Optimization (2004), §2.3.3 and §3.2.6.

How to cite

Cite the atlas with the entry in the panel below, and give the number of any result you use. “Thm III.10” names the same statement here, in theory/framework.md and in theory/propositions.json. A result marked Known is due to the authors in its margin. We add only its re-derivation and its reading for fine-tuning, so please cite them for the result itself.

Key references

These thirty-two papers carry most of the weight, and each note says where one enters. The citation panel has the full bibliography, and the catalogue a card for every method.

Categorical foundations

1. Brendan Fong, David I. Spivak and Rémy Tuyéras. Backprop as Functor: a compositional perspective on supervised learning. LICS 2019. Learners as parametrised lenses (Definition I.1, Exposé IV).
2. G. S. H. Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson and Fabio Zanasi. Categorical Foundations of Gradient-Based Learning. ESOP 2022. The bicategory \(\mathbf{Para}\) and its 2-cells (Definition I.1).
3. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick and Graham Neubig. Towards a Unified View of Parameter-Efficient Transfer Learning. ICLR 2022. Prefix tuning as a gated parallel adapter (Thm VI.5(b)).

Symmetry and dynamics

4. Simon S. Du, Wei Hu and Jason D. Lee. Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced. NeurIPS 2018. Balancedness, the principle behind Thm III.5.
5. Sanjeev Arora, Nadav Cohen and Elad Hazan. On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization. ICML 2018. Balanced deep linear dynamics, one regime of Thm III.9.
6. Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu and Nima Dehmamy. Symmetries, Flat Minima, and the Conserved Quantities of Gradient Flow. ICLR 2023. Thm III.5 is their Prop. 5.1, re-derived for PEFT; see also Marcotte, Gribonval and Peyré, 2023.
7. Salma Tarmoun, Guilherme França, Benjamin D. Haeffele and René Vidal. Understanding the Dynamics of Gradient Flow in Overparameterized Linear Models. ICML 2021. The closed form behind Thm III.9.
8. Hancheng Min, Salma Tarmoun, René Vidal and Enrique Mallada. On the Explicit Role of Initialization on the Convergence and Implicit Bias of Overparametrized Linear Networks. ICML 2021. The role of initial imbalance (Thm III.9).
9. George Kempf and Linda Ness. The length of vectors in representation spaces. Algebraic Geometry, Lecture Notes in Mathematics 732, Springer, 1979. Least-norm points of closed orbits; for LoRA, the balanced factorisations (Prop III.7).
10. Bamdev Mishra, Gilles Meyer, Silvère Bonnabel and Rodolphe Sepulchre. Fixed-rank matrix factorizations and Riemannian low-rank optimization. Computational Statistics, 2014. The quotient geometry of fixed-rank factorisations (Prop III.4).
11. Tian Tong, Cong Ma and Yuejie Chi. Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent. JMLR, 2021. Scaled gradient descent (Prop III.4, Thm III.12).
12. Fangzhao Zhang and Mert Pilanci. Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models. ICML 2024. Scaled GD for LoRA, equivariant under \(GL_r\) undamped, \(O(r)\) damped and \(B_r\) inside AdamW (Thm III.12).
13. Soufiane Hayou, Nikhil Ghosh and Bin Yu. LoRA+: Efficient Low Rank Adaptation of Large Models. ICML 2024. Under Adam with \(\varepsilon=0\), no decay or clipping and one schedule, its learning-rate ratio acts exactly as a rescaling of \(\alpha\) (Cor III.11).
14. John Schulman and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 2025. The LoRA/Adam case of Thm III.10 and Cor III.11; the general torus and the SGD law are added here.

The image of a method

15. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. Zero-initialised, it starts at the apex of its cone (Thm II.2).
16. Soufiane Hayou, Nikhil Ghosh and Bin Yu. The Impact of Initialization on LoRA Finetuning Dynamics. NeurIPS 2024. Zero-\(B\) and zero-\(A\) starts, the strata \(\mathrm T_A\) and \(\mathrm T_B\) of Thm II.2.
17. Fanxu Meng, Zhaohui Wang and Muhan Zhang. PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024. A split start, the stratum \(\mathrm S\) of Thm II.2; its conversion to LoRA rests on Thm II.1.
18. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng and Min-Hung Chen. DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024. For \(W_0\) with no zero row, the same image as LoRA followed by a per-row scale (Prop II.8).
19. Zeju Qiu, Weiyang Liu, Haiwen Feng et al. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. NeurIPS 2023. What exactly orthogonal OFT keeps (Cor II.6); HF's default Cayley–Neumann form is not covered.
20. Weiyang Liu, Zeju Qiu, Yao Feng et al. Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. ICLR 2024. What its output scale changes (Cor II.6) and what re-merging reaches (Thm V.3).
21. Shen Yuan, Haotian Liu and Hongteng Xu. Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation. NeurIPS 2024. Under the hypotheses of Thm II.7, its image is the intersection of LoRA's image with the rotation orbit \(W_0\cdot SO(n)\), and HF's paired start is an apex.
22. Nam Hyeon-Woo, Moon Ye-Bin and Tae-Hyun Oh. FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning. ICLR 2022. LoHa, rank against dimension at the budget of LoRA\(_{2r}\) (Prop II.12).
23. Charles F. Van Loan and Nikos Pitsianis. Approximation with Kronecker Products. In Linear Algebra for Large Scale and Real-Time Applications, Springer, 1993. The rearrangement behind Thm II.11.

Base change and the cotangent side

24. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. LoRA pointed at \(\kappa\theta_0\), with defect \(\kappa\theta_0-\theta_0\) (Prop V.2).
25. Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira and Anna Rumshisky. ReLoRA: High-Rank Training Through Low-Rank Updates. ICLR 2024. \(K\) cycles reach exactly the updates of rank \(\le\min(Kr,m,n)\) (Thm V.3).
26. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar and Yuandong Tian. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML 2024. With no weight decay or global clipping, one-sided LoRA re-based each period, with optimizer state carried over (Prop IV.2).
27. Lucas Torroba-Hennigen, Hunter Lang, Han Guo and Yoon Kim. On the Duality between Gradient Transformations and Adapters. arXiv, 2025. The duality at the core of Prop IV.2.

Extensions and merging

28. Neil Houlsby, Andrei Giurgiu, Stanisław Jastrzębski et al. Parameter-Efficient Transfer Learning for NLP. ICML 2019. Bottleneck adapters, the model case of an extension (Exposé VI).
29. Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL 2021. Trainable keys and values at every layer (Thm VI.5).
30. Aleksandar Petrov, Philip H. S. Torr and Adel Bibi. When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations. ICLR 2024. Per-layer rigidity, and the soft prompt as a deep prefix (Thm VI.5(c), (e)).
31. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal and Colin Raffel. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS 2022. (IA)\(^3\), whose feed-forward scale always folds into \(W_{\rm down}\) and, for some activations, into \(W_{\rm up}\) (Thm VI.3).
32. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du and Min Lin. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. COLM 2024. Factor-wise composition, which depends on how each adapter is factored (Prop V.4).

How the catalogue was built

Every entry was written once and then attacked once, by a separate reader looking for what the first got wrong. The literature was split into the thirteen families below, plus four gap-filling batches and two addenda. The 2026 sweep was run in two passes on 2 October 2026, after the authors reported methods missing.

Titles, authors and first-version years were checked against the arXiv metadata, and venues against the official records. Formulas and parameter counts were checked against the papers, and against the official code where an initialisation was in doubt. Claims about Hugging Face PEFT were checked against its source at commit 85be3f3 of 30 September 2026. Every change was logged with its reason, 905 in all.

The families overlap, so methods were merged on a canonical identifier and a table of aliases, since matching by name would fold LoRA+ into LoRA and O-LoRA into OLoRA. The BibTeX in the citation panel comes from the arXiv metadata of 493 identifiers, with full author lists, and from hand-written records for the nine entries without one.

Each method then received the seven coordinates of Exposé VIII, read off its formula. The atlas records 403 arrows between methods, each with an explicit map \(h\) satisfying \(\rho_N\circ h=\rho_M\), and 485 obstructions, each naming the test that rules an arrow out.

How the theorems were checked

Each statement was checked twice, for its proof and for its fit to the methods as published and as coded. The framework was synthesised from four proposals and three judge reports. Two independent model referees then attacked each of its 41 numbered statements. Fourteen stood as first written, and 27 were repaired; only the repaired forms appear here.

A second round re-examined 26 of the repaired statements in 48 reports. It raised a major objection to 18, mostly to a side clause: a missing hypothesis, a remark that went past its proof, or a description of a method that matched only one implementation. For example, HF PEFT's LoftQ initialiser turned out to ignore the LoRA scale \(\alpha/r\) (Prop V.2). The statements shown here were revised to what that round supports.

Statuses and first-round notes are in theory/propositions.json, and the second-round reports in theory/round2_verdicts.json. The Rosetta stone shows a result's status when you hover its number.

Reproducing the numbers

python3 theory/framework_checks.py re-runs the [Num] computations (sections 1 to 11 of the script) and the main numbers of the referee round (section 12). It uses float64 PyTorch autograd, measures the dimension of an image as a Jacobian rank, and runs optimizers step by step. It is seeded and takes under a minute. The few remaining numbers come from the referees' scripts and those in theory/proposals/.

Each line prints a value beside its prediction, and some are controls that should fail. Under Adam with \(\varepsilon=0\), LoRA+ with ratio \(\lambda=8\) and plain LoRA with \(\alpha\) multiplied by \(\lambda\) differ by exactly \(0.0\) over 200 steps, bit for bit since \(\lambda\) is a power of two; without the rescaling they differ by \(0.93\). Adam's first step has a defect near \(10^{-15}\) under signed permutations of the rank coordinates, and \(6.6\) and \(11.3\) under a rotation and a diagonal rescaling, as Thm III.12 predicts.

Status legend

Labels on results and the referee chips are explained in How to read the labels. The catalogue's chips mean the following.

Colophon

The site is one static page that runs from a web host, from disk or in a sandboxed artifact. Its data file is generated from research/ and theory/. MathJax 3.2.2 sets the mathematics, and D3 7.9.0 with a small linear-algebra kernel in plain JavaScript draws the figures. Type is Source Serif 4 for text, headings and the title, IBM Plex Sans for labels and controls, and IBM Plex Mono for numbers and code. Moss marks what holds or merges, seal an obstruction, ochre a condition.

The chapters are called exposés after the numbered talks of Grothendieck's seminars, whose habit we also borrow, of studying the morphism and letting the right generality do the work.

Thanks go to the authors of the 382 methods and 138 papers catalogued here and to the maintainers of Hugging Face PEFT.

In brief

• 1 October 2026, 17:06 AEST, to 4 October 2026, about 01:00 AEST, when the first full version of this section was written: about 56 hours of wall-clock time. At least one agent was running for about 23 of those hours. The rest was a 43-minute wait for an account usage reset, two interruptions, and long pauses between requests.
• The authors directed one orchestrating session of Claude (Anthropic's Claude Opus 5.5) running in Claude Code, in short messages and a few answers to multiple-choice questions. That session wrote and launched 20 multi-agent workflows, which did most of the research, writing and checking, and two small ones that checked and redrafted this section. It did the planning, integration and publishing itself.
• Those workflows started 422 agent runs (420 distinct agent calls, two of which were started twice). 358 completed, 29 failed when the account reached a spend limit, and 35 were stopped or cut off.
• The agents processed about 117 million fresh tokens: prompts written to the cache plus generated text, of which 8.6 million were generated. They also re-read 4.1 billion tokens from the prompt cache. The orchestrating session added about 10 million fresh tokens and 0.2 billion cache reads.
• The outputs are a website (15 sections including this one, 20 interactive figures plus an animated hero, and a checked catalogue of 382 methods and 138 papers) and a paper of about 400 pages with the same numbering.
• On 5 October a later session made a narrated film of the atlas, 16 minutes 44 seconds long, and a two-minute teaser. Its five agents processed about 5.0 million fresh tokens, and the session itself about 1.4 million.

Who did what

The authors think that when the literature on parameter-efficient fine-tuning (PEFT) already holds several hundred methods, the field gains more from the approach Grothendieck pictured as a rising sea than from one more method. This atlas catalogues 382 of them.

In his memoir Récoltes et Semailles, written 1983–1986, Grothendieck compared a hard problem to a nut. One can strike the shell with hammer and chisel, at point after point, until it cracks. Or one can soak the nut in a softening liquid and let time pass, until the pressure of a hand is enough. On a larger scale the second way is the rising sea. It advances so slowly and quietly that nothing seems to change, until it surrounds the problem and the problem is dissolved by a theory that reaches well beyond it (Grothendieck 2022; McLarty 2007). In the authors' analogy, a 383rd method would be one more strike of the chisel, and the sea is a theory in which the existing methods appear as instances of one notion.

The authors have packaged this way of working as a Claude Code skill. It asks for the natural home of a result, for objects defined by their relationships and for the relative point of view, so that hard results dissolve. Its binding rule is that every reframe must pay, by a shorter proof, a dropped assumption or a real unity, or be rejected.

This view shaped the question the authors chose, to re-found PEFT "as Grothendieck would" and to make the result an interactive site. They also made the main decisions of scope and presentation. Their instructions to the session were short messages: pointing out some missing methods in the literature, asking for a sweep of 2026 work, a mathematical appendix, more readable typefaces, a LaTeX paper and this section, supplying the author list and the title, and asking the session to resume after interruptions.

The orchestrating session did the following:

• planned the work and wrote the seed sketch of the theory;
• designed the visual identity, the data pipeline, the build and test tools and the LaTeX template;
• launched and resumed the workflows, read their reports and decided what to run next;
• applied some changes itself, such as the new typography rules, and rebuilt, tested and published the site, and built the paper.

The sub-agents researched the literature, proposed, judged and refereed the theory, classified the methods, wrote the sections, drew and tested the figures, and reviewed each other's work. They also made the remaining decisions, including which 2026 candidates entered the catalogue and what was cut from the site in its final review.

A workflow is a script for Claude Code's Workflow tool. It starts sub-agents in parallel or in sequence, hands each a prompt and files, and keeps a journal of their results. Every sub-agent ran the same model, claude-opus-5-5, whose training data ends in June 2026. All had web search and fetch and a shell on one Apple M5 laptop, with Python 3.9 (NumPy, PyTorch, SciPy, Matplotlib), Node.js, headless Chrome and TeX Live 2026. Below, an independent check means one made by a separate agent with its own context and instructions.

Finding the material

The literature work was split into thirteen families:

• the Hugging Face PEFT library;
• LoRA and its training-dynamics variants;
• initialisation and rank allocation;
• sharing and extreme efficiency;
• structured shapes;
• multiplicative and orthogonal methods;
• adapters;
• prompts and states;
• selective fine-tuning;
• quantisation and memory;
• composition and merging;
• theory and foundations;
• the 2025–2026 frontier.

One agent swept each family. It worked from the arXiv API, Semantic Scholar, OpenReview, the ACL Anthology, the official conference listings and the Hugging Face PEFT source code. Each sweeper was told to read a paper's method section before writing down its formula. As each family finished, an independent fact-checker re-verified every entry:

• title, first author and first-version year against the arXiv API;
• the venue against the proceedings;
• the formula against the method section;
• the parameter count, re-derived independently;
• mergeability and initialisation;
• Hugging Face PEFT support, against the repository at commit 85be3f3 (30 September 2026).

A completeness critic then compared the whole catalogue with the library and with its own knowledge of the field. It listed 34 missing methods, and 32 of them were researched and checked in four more batches. The checkers, including those of the four gap-filling batches, made 613 corrections before the first release. Merging duplicates on a canonical identifier and an alias table gave 320 methods and 124 papers. Matching by name would have folded LoRA+ into LoRA.

After the first build, the authors pointed out methods missing from the catalogue. They were added and refereed, and a dedicated sweep of 2026 work followed, in which six scouts covered:

• arXiv submissions from January to September 2026, one scout per quarter, by keyword queries over titles and abstracts;
• the accepted papers of ICLR 2026 and ICML 2026, with ACL and CVPR 2026;
• the citation graph of ten anchor PEFT papers and the 2026 Hugging Face PEFT releases.

A triage agent then chose which candidates entered the atlas. It admitted new 2026 methods with at least one signal (a 2026 venue, inclusion in Hugging Face PEFT, citations, a widely used codebase, or a structure the seven coordinates would treat differently), at most 36 per pass. The first triage saw only the first 105 of the 265 candidates not already in the atlas, the last of them cut off mid-entry, because the orchestrator had truncated the list it passed in. A second pass triaged the remaining 161 candidates and the scouts' overflow lists. Together the passes added 61 methods and 14 theory papers. They were researched and fact-checked in the same way, giving 382 methods, 138 papers and 905 corrections in total.

The BibTeX of the 502 catalogue entries is built from arXiv metadata for the 493 that have an arXiv identifier, and the other nine are built from the catalogue's own fields. Eight of those nine were corrected by hand for the paper. The paper's 130 other references, mostly mathematics and related work, were written by the agents who cited them, who were told to check every field against the publisher, DOI or arXiv page.

From examples to a theory

The orchestrator wrote a seed sketch of eleven numbered points, about 1,000 words, which the architects could adopt, correct or reject. Its points were:

• methods as reparametrisations in Para and as objects of a pointed slice;
• frozen and full fine-tuning as initial and terminal;
• mergeability as a lifting problem;
• Klein's Erlangen programme for group-type methods;
• the \(GL_r\) gauge of LoRA and its conserved charge;
• string diagrams for structured updates;
• a monoidal sum;
• base change;
• the lens duality between forward and backward methods;
• prompts and prefixes as learned input states;
• naturality, so that a method defined once per layer type extends to every network.

Four architect agents each developed a complete framework from this sketch, with definitions, propositions and proofs, from a different point of view:

• parametric lenses;
• Grothendieck's relative point of view;
• the Erlangen programme with gauge symmetry;
• string diagrams.

Three judge agents scored all four proposals, each from an assigned point of view:

• a category theorist's, for rigour;
• a PEFT researcher's, for faithfulness to the methods;
• a science communicator's, for clarity and visual potential.

All three recommended the relative point of view as the backbone. One synthesis agent then built the framework on it. As the judges advised, it took the lens and cometric from the parametric-lens proposal, the covariance and optimizer gauge from the Erlangen proposal and the diagrams from the string-diagram proposal, and it cut the constructions they judged decorative. The standing rule was that a categorical construction must explain, classify or predict something about PEFT, or be cut. The result was 41 numbered results, a seven-axis classification and a dictionary between the two vocabularies.

Generalisation usually followed one pattern:

1. Take an observation about a single method. Zero-initialised LoRA cannot move in some directions at the first step.
2. Find the invariant it depends on. Here it is the first-order image compared with the dimension of the image germ.
3. State the result for every method with that coordinate value. Any method whose base point is the vertex of a cone has the same deficit.
4. Look for consequences the source papers do not state. Householder reflection adaptation at its default start is also an apex.
5. Test the claim numerically and send it to referees.

The same pattern turned "LoRA+ helps" into an exact statement about Adam's invariance to per-block rescaling. It turned "orthogonal methods preserve structure" into complete invariants of each group action. It turned "this adapter can be merged" into a local rewriting test. The test certifies exact merging under stated side conditions, names the weight that absorbs the update, and depends on the activation, the normalisation and the bias slots.

The seven coordinates are decidable from a method's formula. That made it possible to classify the first 320 methods in ten batches of 32, and the 62 later additions in three more. In each batch a second agent re-derived every coordinate, checked every arrow's factorisation and base point by hand, and spot-checked a few gauge dimensions and first-order deficits numerically. A consolidator then reconciled the first ten batches with each other and with the theory, and the final site review made the prose agree with the data. Every arrow between methods records its explicit map h with \(\rho_N\circ h=\rho_M\) (403 of them). Every recorded non-arrow names the test that rules it out (485).

How category theory unifies the field here

The unifying move is to study each method through its map into weight space. A weight-space method is a pointed map from its small parameter space into weight space that sends its starting point to the pretrained weights. Comparing methods then means comparing maps:

• a method N simulates a method M when M's map factors through N's, that is \(\rho_N\circ h=\rho_M\) for a pointed map h;
• doing nothing and full fine-tuning are the initial and terminal objects, so they bracket every weight-space method;
• three invariants of the map carry most of the content: its image germ, its fibres with the optimizer's metric, and its behaviour under base change.

Extensions such as adapters and prefixes, which change the network itself, fit through a realisation map. Mergeability becomes the question of whether a lift exists.

Exposé XI lists what is analogy and what is theorem. The words Noether, Erlangen, gauge and Grothendieck are used in a precise, limited sense, and the paper says which.

How the results were checked

Verification ran in layers:

• Sources. Every catalogue entry was checked by an agent other than the one that wrote it, against arXiv metadata, the paper and the library source.
• Adversarial model referees, three rounds. Each of the 41 results went to two referee agents with different lenses: mathematical correctness, and faithfulness to the papers and code. They were told to default to "refuted" when in doubt.
  • Round 1 produced 82 verdicts, 45 of them major objections (41 refutations). 27 results were repaired and none was false outright.
  • Round 2 re-examined 26 of the 27 repaired results in 48 reports. Six of its 54 runs were stopped by the spend limit, so Corollary III.13 received no second-round report and four results received one; in the final site review the section editors took the missing referees' place. The reports raised 28 major objections against 18 results, and 19 results were narrowed again.
  • Each round-2 report ended with the strongest statement its referee would sign. These became one public statement per result in propositions.json. The writers and referees of the site and the paper were told to state nothing stronger, and checked for it.
  • A third round, four mathematical referees during the writing of the paper, made 63 further fixes.
  • Those fixes were then carried back into the site, the data and the theory files, 108 edits in all.
• Numerical checks. framework_checks.py, one of the paper's data files, re-runs most claims marked [Num] in about a minute. The paper's Appendix E lists the few that rest only on the referees' own replications. Each interactive figure computes its numbers in the browser. The engineers of the theory-driven figures checked their key numbers against framework_checks.py or a separate NumPy computation, and in the final site review a second agent spot-checked the key numbers of every figure.
• Engineering checks. Every figure was rendered headlessly in light and dark themes at desktop and phone widths, and its builder looked at the images. A whole-site test drives Chrome to check console errors, unmounted figures, overflow, MathJax errors and broken in-page links. The paper is compiled from a clean copy of its sources with pdfLaTeX alone and checked for errors and undefined references.
• This section. Two agents checked every number and date in it against the workflow journals, the agents' transcripts and the project files, and its text was corrected where they disagreed. Its opening was then redrafted by three agents writing independently, merged by a fourth and checked by two more, one for facts and one for style.

These checks do not amount to human peer review. Every proposer, writer and referee was the same model, so their errors can be correlated, and the referees of the paper also edited the files they reviewed. A numerical check is not a proof, and every check runs on small float64 examples. No fine-tuning run on a real model was made, so the eleven predictions of Exposé X are untested. As Exposé XI records, the dimension equalities for LoHa and, in general, for VeRA are checked numerically; one clause of corollary 4 of Theorem V.3 is labelled a proof sketch, although the paper's Appendix C gives an argument; and Conjectures A, C and D are open. The catalogue is a selection and can still miss methods. Its 2026 additions came from keyword queries on arXiv, four conference lists and one citation graph, filtered by a triage agent, and the completeness critic's own knowledge of the field ends with the model's training data. In the catalogue, 51 method entries and 7 paper entries carry a medium-confidence flag. Some methods were missing until the authors noticed them. The authors are responsible for what the site and the paper say.

Building the site and the paper

The site is a static page. Its prose lives in fifteen HTML sections, counting this one, and its data is generated from the verified catalogue and the theory files. Each of its 21 figure scripts was written by an engineer agent (four were finished by a second agent after the spend limit cut off the first), self-tested and later reviewed by another. The page runs from any static host, straight from disk and inside a sandboxed artifact. Its data is a script, so nothing is fetched at run time apart from D3, MathJax and the fonts, which come from public CDNs.

The typography was changed once, at the authors' request. The session set one passage of the site in four typeface schemes, and the authors chose the one it recommended, with Source Serif 4 for text, IBM Plex Sans for labels and IBM Plex Mono for numbers. Seven engineer agents then raised the text inside all 21 figure modules to at least 12 pixels where a reader must read it, and 11 pixels for tick labels. Later the authors found the display face of the title unattractive, and the title was reset in Source Serif 4 as well.

The paper uses a template designed for it: SGA-style exposés with Roman numerals, so that every result has the same number on the site and in the paper. Twenty-one writer agents converted the full 35,000-word theory into LaTeX, with complete proofs in Appendix C of the paper. Six figure engineers redrew the key figures as vector PDFs in the same typefaces. An assembler, four mathematical referees, a typesetting referee and a finalizer took it to a clean build of 393 pages, about 400 with this section's counterpart, Appendix F.

The film

On the evening of 5 October, at the authors' request, a later session of Claude Code made a narrated film of the atlas, 16 minutes 44 seconds long, and then a two-minute teaser for X. The authors chose the narrating voice, Sulafat from Gemini 3.8 Flash TTS, and a quiet visual style from a short list of options.

The session wrote the script and checked each claim against the propositions and the paper, and a separate agent then fact-checked it adversarially. Every number on screen comes from a short computation in video/sim/. The opening, for example, trains LoRA+ and plain LoRA with a scaled alpha side by side, and the largest difference between their weights stays exactly zero. The session wrote a shared Manim library, a brief and one chapter, and four agents built the other seven chapters from the brief. The session then synthesised the narration beat by beat, timed each animation cue to the pauses detected in the audio, reviewed every chapter on contact sheets of its frames, and assembled the film with captions and chapter marks. The film plays on the companion site, and the title page of the paper links to it.

Cost and interruptions

The table counts every started agent run, one transcript each: the 422 runs of the workflows and the film's five agents, a script fact-checker and four chapter builders. Cache re-reads are not counted (4.1 billion tokens for the workflows, 0.16 billion for the film's agents), and neither are the sessions that directed the work: the orchestrating session of the workflows, and the film's session, which added 1.4 million fresh tokens, 0.55 million of them generated. Also not counted are the two later workflows that checked and redrafted this section (nine agents), and the film's narration: about 15 minutes of speech from Gemini 3.8 Flash TTS, plus voice tests and a few retakes, which Google bills separately.

The work was interrupted three times:

• at 20:52 on 1 October an account spend limit made 29 agent runs fail, 12 of them mid-run and 17 within seconds of starting;
• at about 20:40 on 2 October the host machine crashed while the paper was being written;
• at about 23:10 on 2 October the session was cut off again, and it resumed at 02:04 the next morning.

Each time, the orchestrator inspected the workflow journals and launched fresh workflows containing only the unfinished items, telling each agent to read and complete its predecessor's partial file. The six stopped second-round referee runs were not repeated; the final site review covered them instead. A first attempt to resume in place re-ran agents that had already finished. The workflow tool replays only the longest unchanged prefix of agent calls, so that attempt was stopped within two minutes. The other process lessons were:

• pass large inputs to agents through files, never as truncated prompt text;
• refereeing a repaired result again is worth it;
• corrections found while writing the paper must flow back to the site.

Reproducibility

The catalogue and the checks accompany the paper as data files:

• catalogue.csv and catalogue.json, with arrows.csv and obstructions.csv;
• propositions.json, which holds each numbered result with its statement, proof, public statement and the notes of both review rounds;
• framework_checks.py, the numerical checks.

The LaTeX source carries the text and the generated tables, and the site's figure code can be read in the page. The public repository, github.com/xiaoyulics/peft-atlas, holds the rest of the record:

• the raw and verified literature files and the classification batches, in research/;
• the full theory text and the round-2 reports, in theory/;
• the build and test scripts, in site/build, site/test and paper/tools, and the scripts of the paper's figures, in paper/figures/src;
• the film's script, scenes and computations, in video/.

The 20 workflow scripts, which contain every agent prompt, and the agents' transcripts stay with the authors.