Low-Rank Adaptation (LoRA)

Fine-tuning a model sounds simple until the optimizer state arrives. Updating every weight means storing gradients for every weight and, with an optimizer such as Adam, maintaining additional state alongside them. A model that fits in memory for inference may be nowhere close to fitting for full fine-tuning.

LoRA changes the part of that bill associated with trainable weights. It freezes a pretrained matrix and learns a low-rank update beside it. The base model still performs the large matrix multiplications; LoRA does not turn a seven-billion-parameter model into a tiny one. What it does is make the learned update small enough to train, store, copy, and sometimes serve independently.

That distinction is easy to lose. LoRA is often described with spectacular compression ratios or as a way to fine-tune almost any model on consumer hardware. Neither is a property of the method. The saving depends on which matrices are adapted, their shapes, the chosen rank, optimizer state, activation memory, precision, sequence length, and whether the frozen model is quantized or sharded.

This article works through those details, including one initialization quirk that is obvious in the derivatives but easy to miss in code.

The Update LoRA Actually Learns

Consider a pretrained linear layer with weight

Full fine-tuning learns an unrestricted update $\Delta W$ of the same shape. LoRA instead writes the update as the product of two smaller matrices:

where

and $r$ is much smaller than either matrix dimension. The forward pass becomes

The scalar $s$ controls the update scale. The original LoRA parameterization commonly uses

with a user-selected $\alpha$. Some later variants use a different rank-dependent scale.

The important word is constrains. For one adapted matrix, $BA$ has rank at most $r$, so LoRA searches within a restricted family of updates. The original paper found that useful adaptation updates were rank-deficient in its experiments. That is empirical motivation, not a theorem that every task, layer, or model needs only a tiny rank.

Across a deep network, the situation is richer than one low-rank matrix suggests. Several LoRA-modified layers are separated by nonlinearities and attention operations. A low-rank update at each selected matrix does not make the end-to-end change to the model a single low-rank function.

Why LoRA Can Work at All

A pretrained model has already spent most of its training budget learning language, syntax, and a large collection of reusable features. Fine-tuning usually does not ask it to start over. It asks for a correction: follow this response format, use this domain’s terminology, prefer this kind of answer, or solve one task more reliably.

LoRA is a bet that those corrections are structured.

Consider the simplest case, a rank-one update. Let $a^T$ be one row of $A$ and $b$ the corresponding column of $B$. Its contribution to the layer is

Read the expression from right to left. The scalar $a^Tx$ measures how strongly the current activation points along one learned input direction. The vector $b$ then adds a learned output direction in proportion to that measurement. In less mathematical language, one vector decides when the correction should activate, and the other decides what correction to add.

Rank $r$ supplies $r$ of these channels:

The directions do not usually have tidy labels such as “Python” or “politeness.” They are distributed numerical features. Still, the operation is concrete: LoRA learns a small collection of input-output correlations instead of allowing an arbitrary change to every entry of $W_0$.

There is a useful clue in ordinary backpropagation. For a linear layer $y=Wx$, let $g=\partial\mathcal{L}/\partial y$ be the gradient arriving from the rest of the network. For one example,

That matrix is rank one. A minibatch gradient is a sum of such outer products:

Across many steps, a full fine-tuning update can certainly become high rank. But examples from one task often produce correlated activations and correlated error signals. If the same few correlations keep appearing, most of the useful update can concentrate in a small number of matrix directions. LoRA is designed to learn those directions directly.

The singular values of a full update make the idea precise. Any update can be written as

If a few $\sigma_j$ values dominate, a low-rank matrix captures much of the update’s action. The original LoRA paper found this kind of rank deficiency in several adaptation experiments. LoRA does not first perform full fine-tuning and truncate its update; it optimizes the factors from the beginning, so its training path is different. The spectrum only explains why the restriction can be reasonable.

The restriction is also applied many times. A transformer may receive LoRA updates in dozens of attention and feed-forward projections. Each update is low rank at its own layer, but the layers are separated by attention, normalization, residual connections, and nonlinearities. The change to the complete network is therefore much richer than one rank-$r$ matrix.

Research on the intrinsic dimension of fine-tuning offers related evidence: some language-model tasks can be learned in a parameter subspace far smaller than the full model. It is not a proof of LoRA. A random low-dimensional parameter subspace and a low-rank update to selected matrices are different constraints. Both results simply point in the same direction—useful adaptation often needs fewer independent degrees of freedom than the parameter count suggests.

This is why rank 8 can be enough in one experiment and fail badly in another. Changing an answer format or specializing a model in a familiar domain may require only a compact correction. Adding new vocabulary, forcing reliable unlearning, or teaching a capability missing from the base model can demand different target modules, a larger rank, or full fine-tuning. LoRA works when the pretrained model already contains most of what the task needs and the remaining correction has a low-rank structure. That condition must be tested, not assumed.

Count the Parameters Before Quoting a Percentage

A full update to $W_0$ contains

parameters. A LoRA update contains

For a square $4096\times4096$ projection and rank $r=8$:

full parameters, while LoRA adds

That is about $0.39\%$ of that matrix. If only the query and value projections are adapted across 32 identical layers, the adapter contains about 4.2 million parameters. If every attention and MLP projection is adapted, the count is much larger. If embeddings or the language-model head are trainable, it grows again.

This is why a claim such as “LoRA trains 0.1% of the model” is incomplete. It may describe one configuration, but it is not a fixed LoRA ratio. A useful experiment reports at least:

• rank and scaling;
• every targeted module;
• trainable and total parameter counts;
• whether biases, embeddings, or the output head are trained;
• the base model and its exact architecture.

Zero Output, Asymmetric First Gradients

A common initialization draws $A$ from a random distribution and initializes $B$ to zero. Then

so the adapted layer initially produces exactly the base layer’s output. This is a valuable property: inserting LoRA does not perturb the model before training begins.

It also makes the first optimization step asymmetric. Let $G$ be the gradient of the loss with respect to $\Delta W$. Ignoring the scalar $s$ for a moment,

At initialization, $B=0$. Matrix $B$ can receive a non-zero gradient because $A$ is already non-zero, but the gradient of $A$ is initially zero. After $B$ moves away from zero, both factors can learn.

Some implementations reverse the roles or offer alternative initialization schemes. The invariant that matters is that the initial low-rank branch should behave as intended. Do not assume two libraries use the same convention merely because both call the method LoRA.

Initializing both factors to zero is a dead start. Each factor’s gradient contains the other factor, so neither can move. Initializing both randomly avoids that problem but perturbs the base model immediately. The random/zero convention is a compromise: preserve the initial function while leaving one factor able to learn on the first step.

The Factors Are Not a Unique Solution

The model uses the product $BA$, not a particular pair of factors. For any non-zero scalar $c$,

The effective weight update is identical, but the factor norms, gradients, Adam moments, clipping behaviour, and weight-decay penalty can differ. This scale symmetry has several practical consequences:

• a large norm in $A$ is not automatically a large functional update;
• clipping $A$ and $B$ independently can change the optimization path even when $BA$ is well behaved;
• applying the same weight decay to both factors is not equivalent to decaying $\Delta W$ directly;
• optimizer checkpoints matter if training is resumed, because two equivalent products can carry different optimizer state.

The zero initialization adds another asymmetry: $B$ learns before $A$. LoRA+ studies this optimization imbalance and proposes different learning rates for the two factors. Its results are evidence that “one learning rate for every adapter tensor” is not a neutral choice, but the published ratio is still a hyperparameter to validate rather than a universal constant.

When diagnosing training, inspect the effective update as well as its factors. Useful quantities include

per layer, alongside gradient norms, activation statistics, and validation metrics. Factor norms alone can tell a misleading story.

A Small Implementation That Checks Its Own Claims

The following module is deliberately small. Production libraries handle adapter selection, mixed precision, quantization, serialization, and distributed training; this version exposes the underlying operation.

import math

import torch
import torch.nn.functional as F
from torch import nn


class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, rank: int, alpha: float):
        super().__init__()
        if rank <= 0:
            raise ValueError("rank must be positive")

        self.base = base
        self.rank = rank
        self.scaling = alpha / rank

        for parameter in self.base.parameters():
            parameter.requires_grad_(False)

        self.A = nn.Parameter(torch.empty(rank, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))

        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))

    def delta_weight(self) -> torch.Tensor:
        return self.scaling * (self.B @ self.A)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        base_output = self.base(x)
        adapter_output = F.linear(F.linear(x, self.A), self.B)
        return base_output + self.scaling * adapter_output

    @torch.no_grad()
    def merged_weight(self) -> torch.Tensor:
        return self.base.weight + self.delta_weight()


torch.manual_seed(7)

layer = LoRALinear(nn.Linear(16, 12), rank=4, alpha=8)
x = torch.randn(5, 16)
target = torch.randn(5, 12)

# B starts at zero, so inserting the adapter changes nothing yet.
torch.testing.assert_close(layer(x), layer.base(x))

loss = F.mse_loss(layer(x), target)
loss.backward()

# With this initialization, A's first gradient is zero while B can move.
print("A gradient:", layer.A.grad.norm().item())
print("B gradient:", layer.B.grad.norm().item())

optimizer = torch.optim.SGD([layer.A, layer.B], lr=0.1)
optimizer.step()
optimizer.zero_grad()

# Merging BA into the frozen weight preserves the adapted computation.
unmerged = layer(x)
merged = F.linear(x, layer.merged_weight(), layer.base.bias)
torch.testing.assert_close(unmerged, merged, rtol=1e-5, atol=1e-6)

That final assertion is more informative than the phrase “zero-overhead inference.” It establishes mathematical equivalence for this ordinary floating-point layer. Serving introduces additional choices that the equation alone does not settle.

Which Matrices Should Be Adapted?

The original LoRA experiments often targeted query and value projections. That is a historical configuration, not a universal optimum. Modern decoder models contain several plausible targets:

• query, key, value, and attention-output projections;
• gate, up, and down projections in the feed-forward block;
• embeddings or the output head in tasks that need vocabulary changes;
• modality projectors in vision-language systems.

Adapting more projections increases capacity and trainable parameters. It can also improve performance when a narrow attention-only adapter is insufficient. The correct choice depends on the task and budget, so “all linear layers” and “query/value only” should be treated as baselines to compare rather than doctrine.

Module matching is a mundane source of failed runs. A configuration can silently target fewer layers than expected because model implementations use names such as q_proj, query_key_value, or a fused projection. Print the selected modules and trainable parameter count before starting a long job.

Biases require an explicit decision too. The base bias in the example above remains frozen. Training it adds a small number of parameters but changes what should be saved, merged, and compared.

Orientation and fused projections

The equations in this article use a weight shaped

which matches torch.nn.Linear. Some transformer implementations store an equivalent projection in the opposite orientation. Hugging Face models derived from GPT-2’s Conv1D are a familiar example. Adapter libraries often expose an option such as fan_in_fan_out for this reason. Getting it wrong can produce a shape error; a more dangerous custom implementation can transpose the update consistently enough to run while adapting the wrong axes.

Fused projections need equal care. A model may store query, key, and value weights in one tensor. Targeting that module adapts the fused object unless the implementation supports slices or separate parameter views. It is not equivalent to the common “query and value only” configuration. With grouped-query attention, the query projection can also have a different output width from the key and value projections, so parameter counts based on four identical square attention matrices are wrong.

Inspect actual module types, weight shapes, selected parameter names, and trainable counts from the loaded checkpoint. Architecture diagrams and configuration field names are not sufficient.

Token embeddings and tied output heads

Adding tokens changes the problem. A newly resized embedding row remains random or fixed if embeddings are frozen, and a LoRA attached only to attention projections cannot train that row. The output head may be tied to the embedding matrix or stored separately, depending on the model.

If the task adds vocabulary, decide explicitly whether to train and save the affected embedding and language-model-head weights. Many adapter libraries provide a modules_to_save mechanism for full, non-LoRA tensors. Forgetting it can produce a checkpoint that worked in the training process but cannot reconstruct the same model after reload.

Tokenizer state belongs to the adapter release for the same reason. Special tokens, chat templates, padding direction, and token IDs can change behaviour without changing a single LoRA weight.

What LoRA Saves—and What It Does Not

Training memory is not one number. It includes:

1. model weights;
2. gradients;
3. optimizer state;
4. saved activations;
5. temporary kernel workspaces and communication buffers.

LoRA sharply reduces gradients and optimizer state because those are needed only for the adapter parameters. It also avoids storing a full trainable copy of each model for every task.

The frozen base weights still have to live somewhere. The forward pass still uses them, and the backward pass still propagates gradients through the network to reach adapters in earlier layers. Sequence length, microbatch size, hidden width, attention implementation, and activation checkpointing can therefore dominate memory even when the adapter itself is tiny.

For intuition, seven billion BF16 parameters require roughly 14 GB for the raw weights alone. This ignores allocator overhead, buffers, activations, and every other part of the process. A small adapter does not make that base storage disappear.

The practical question is not “How many parameters are trainable?” but “Which part of my peak memory changed?” Measure allocated and reserved device memory during the actual step. A parameter count cannot reveal fragmentation or a temporary attention allocation.

LoRA and QLoRA Solve Different Parts of the Bill

QLoRA stores the frozen base model in a 4-bit representation and backpropagates through its dequantized computations into LoRA adapters. The QLoRA paper introduced NF4 for normally distributed weights, double quantization for quantization constants, and paged optimizers to manage memory spikes.

The distinction is simple:

• LoRA reduces the trainable update and its optimizer state.
• QLoRA also reduces storage for the frozen base weights.

Four-bit storage does not mean the entire training job consumes exactly half a byte per model parameter. Quantization scales and metadata take space, computation uses wider types, and activations remain. Hardware and kernel support also matter.

QLoRA is useful when the base model is the memory bottleneck, but quantization may introduce error and operational constraints. Compare it with non-quantized LoRA under the same data and evaluation instead of assuming the result will be identical.

Three dtypes may be involved at once: the storage format of the frozen weight, the compute dtype used after dequantization, and the dtype of the adapter parameters. A nominally four-bit run can still perform matrix multiplication in BF16 while keeping selected normalization or output tensors in wider precision. Report all three rather than calling the entire job “4-bit training.”

Quantization error also changes the starting function before the adapter learns anything. LoftQ addresses this interaction by choosing the quantized base and low-rank initialization together so that their sum better approximates the original full-precision weight. This can be useful in aggressive quantization regimes, but it is a different initialization procedure from loading an arbitrary 4-bit checkpoint and attaching a zero-output adapter.

Finally, do not merge into a low-bit tensor in place and assume the operation is reversible. A safe release process keeps the original adapter, exact base revision, quantization recipe, and—when possible—a higher-precision merge source. Dequantize, merge, requantize, and evaluate the resulting artifact as a new model release.

Rank and Scaling Are Coupled

Increasing $r$ increases adapter capacity and cost. Under the original $\alpha/r$ scaling, it also changes the scale applied to the update. A rank comparison is therefore hard to interpret if $\alpha$, initialization, learning rate, and target modules change at the same time.

Rank-stabilized LoRA, usually called rsLoRA, proposes

instead of $\alpha/r$. Its motivation is to keep the contribution and gradients from shrinking too aggressively as rank grows. It is a useful option for rank sweeps, not proof that higher rank will help a given task.

There is no dependable rule that rank 8 is enough, rank 16 is safer, or larger models always need larger ranks. Language adaptation, a new output format, code generation, and a major domain shift may place very different demands on the update. Tune rank as part of an explicit capacity budget.

Layer-wise rank does not have to be uniform. AdaLoRA, for example, allocates a parameter budget adaptively across matrices. That adds scheduling and implementation complexity, so it should earn its place through a comparison with a well-tuned fixed-rank baseline.

Configured rank is only an upper bound

The product satisfies

Training can use fewer than $r$ meaningful directions. To see whether a rank budget is active, compute the singular values $\sigma_1\geq\cdots\geq\sigma_r$ of the merged update and inspect the cumulative energy

If a few singular values carry nearly all the energy, a higher configured rank may be unused. If energy remains spread across all $r$ directions and quality improves as rank grows, the constraint may still be binding. This diagnostic is not a replacement for held-out evaluation, but it is more informative than treating the configuration value as the update’s achieved rank.

For a small model where full fine-tuning is affordable, comparing the singular spectrum of the full update $W_{FT}-W_0$ with LoRA updates can reveal whether the chosen target matrices and ranks capture the same dominant directions. It does not prove that truncating the full update will reproduce training—the optimization paths differ—but it gives the rank discussion evidence.

Merging Changes the Serving Trade-off

For an ordinary floating-point weight, the adapter can be folded into the base:

After merging, the layer needs no separate adapter matrix multiplications. This is the basis of LoRA’s no-additional-layer inference claim.

There are several qualifications:

• Switching tasks dynamically is easier when adapters remain separate.
• Serving many adapters in one batch may require specialized batching and kernels.
• Merging into a quantized base can require dequantization and requantization, which may introduce additional error.
• Once weights are merged, the original base and adapter must be retained separately if the merge needs to be reversed exactly.
• Floating-point operation order means merged and unmerged results can differ slightly even when they represent the same equation.

A production evaluation should report whether the adapter is merged, how many adapters share the server, and the latency distribution—not just model quality.

LoRA dropout is another easy source of a false merge test. During training, dropout may be applied only on the adapter branch. Compare merged and unmerged outputs in evaluation mode, where dropout is disabled. Agreement in training mode is neither expected nor stable.

Multiple adapters on the same matrix can be added algebraically:

The combined update can have rank as high as the sum of the adapter ranks. The arithmetic is valid; the behaviour is not guaranteed to compose. Two adapters trained independently may rely on conflicting representation directions or chat templates. Evaluate the mixture on every constituent task and on interactions between them.

Repeatedly merging into already rounded weights can accumulate numerical error and makes provenance difficult to recover. Build each release from the immutable base plus named adapter artifacts, then save the result once.

Distributed Training Still Moves More Than Adapter Gradients

In data parallel training, synchronizing adapter gradients can be dramatically cheaper than synchronizing gradients for every base weight. That does not make total communication proportional only to $r(d_{\text{in}}+d_{\text{out}})$.

The base model may still be sharded with FSDP or ZeRO. Tensor parallelism exchanges activations within layers, pipeline parallelism sends activations between stages, and sequence or context parallelism introduces its own collectives. LoRA reduces one category of communication; it does not remove the distributed system underneath the model.

Small trainable tensors can also interact poorly with aggressive sharding or produce many tiny collectives. Inspect the framework’s wrapping policy, communication trace, and step-time breakdown rather than inferring throughput from the adapter parameter count.

Variants Worth Distinguishing

The LoRA family has accumulated enough names to become confusing. Four ideas cover much of the practical design space:

DoRA was motivated by measured differences between full fine-tuning and LoRA updates. Its published results are encouraging across several language and vision-language tasks, but they remain results under named models and datasets. “Variant X beats LoRA” is never complete without the experimental setting and baseline tuning.

PiSSA is sometimes described as an initialization swap, but that shorthand hides an important detail. It places principal components of $W_0$ in the trainable factors and the remainder in a frozen residual so that the combined initial layer still represents the pretrained weight. Saving only the trained factors without the corresponding residual or conversion metadata is not enough to reconstruct the intended model.

An Adapter Checkpoint Is Not Self-Describing

An adapter file is meaningful only relative to the model it modifies. A professional release should pin:

• the exact base-model revision and license;
• tokenizer files, added tokens, and chat template;
• target module names and their expected shapes;
• rank, alpha, scaling convention, dropout, bias policy, and initialization;
• any fully trained modules_to_save;
• training and adapter-library versions;
• base, compute, and adapter dtypes;
• quantization configuration;
• whether the artifact is separate, merged, or converted from another parameterization.

Load the saved artifact into a fresh process and compare it with the training process on fixed inputs. This catches missing embedding rows, a wrong base revision, module-name drift, and save/load dtype changes. Testing only the in-memory model before serialization does not test the checkpoint.

For a merged release, retain a manifest that maps it back to the immutable base and adapter digests. The merged tensor alone cannot say which adapter scale, tokenizer, or quantization path produced it.

A Credible LoRA Experiment

A useful fine-tuning report should make it possible to separate the method from the surrounding choices. At minimum, keep the following visible:

• exact base checkpoint, tokenizer, chat template, and library versions;
• dataset construction, contamination checks, split, and token counts;
• target modules, rank, scaling, dropout, precision, and initialization;
• optimizer, learning-rate schedule, batch construction, and total training tokens;
• trainable parameters and peak device memory;
• training throughput and wall-clock time;
• held-out loss plus a task-relevant evaluation;
• results from more than one seed when the run is small enough for variance to matter;
• merged and unmerged inference latency if serving efficiency is part of the claim.

Compare against something meaningful. Depending on available compute, that might be full fine-tuning, QLoRA, several ranks, attention-only versus all-linear targets, or even a frozen-model baseline. Giving every method a wildly different token budget or tuning effort answers a different question.

Validation metrics do not have to improve monotonically. They fluctuate, and a training loss can continue falling while generalization worsens. Likewise, there is no reason the norms of $\nabla_A$ and $\nabla_B$ should converge to a ratio of one. Monitor them for explosions, vanishing updates, and layer imbalance—not against an invented universal target.

LoRA Is Not an Unlearning Mechanism

Freezing the base preserves its weights; it does not erase what they encode. An adapter can suppress an unwanted output on an evaluation set while the underlying representation remains available under another prompt or after the adapter is removed.

This matters for backdoor removal, privacy deletion, and safety remediation. Recent work on removing backdoors with LoRA found that standard low-rank updates could lack the spectral strength and alignment needed to cancel trigger-sensitive directions in the frozen model. That is one study of a specific failure setting, not a verdict on every adapter, but it exposes the wrong default assumption: good clean-task accuracy does not prove that an unwanted behaviour has been removed.

An unlearning claim needs adversarial evaluation against the behaviour being removed, comparison with stronger baselines, and a threat model covering adapter removal. When deletion must apply to the model itself, full or otherwise targeted modification of the base may be necessary.

When LoRA Is the Wrong Constraint

LoRA is especially attractive when many adaptations share one base model, when storage and optimizer state are limiting, or when experiments need cheap, swappable checkpoints.

It may be the wrong choice when:

• the task requires a large representational shift that a chosen low rank cannot express;
• full fine-tuning is affordable and consistently produces a worthwhile gain;
• the serving system cannot batch or switch adapters efficiently;
• the base model itself does not fit and quantization or sharding is unavailable;
• vocabulary or architectural changes dominate the adaptation;
• maintaining many interacting adapters becomes more complex than maintaining a smaller number of complete models.

Adapters are not automatically composable. Adding or interpolating two low-rank updates is mathematically easy, but the behaviours learned by those updates can interfere. Composition needs evaluation on the combined task, including regressions that neither adapter shows alone.

Conclusion

LoRA is a constrained update, not a miniature model and not a hardware exemption. Its appeal comes from a useful engineering bargain: keep the expensive pretrained weights fixed and make each learned adaptation comparatively small.

The bargain is strongest when the constraint matches the task. That has to be measured. Count the actual targeted parameters, account for the whole memory footprint, inspect the initialization, distinguish LoRA from QLoRA, and test serving in the form that will really be deployed. Once those details are explicit, LoRA stops looking like magic and starts looking like what it is: a simple factorization with unusually practical consequences.

References

• Aghajanyan, Zettlemoyer, and Gupta, Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2020).
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021).
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023).
• Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023).
• Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (2023).
• Li et al., LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models (2023).
• Hayou, Ghosh, and Yu, LoRA+: Efficient Low Rank Adaptation of Large Models (2024).
• Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation (2024).
• Meng, Wang, and Zhang, PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024).
• Luong and Chen, Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models (2026).