PyTorch nn.Linear: Shapes, Weights and the Multiply Error

nn.Linear is PyTorch’s fully connected layer. It takes a number of input features and produces a number of output features:

layer = nn.Linear(in_features=4, out_features=3)

y = layer(x)        # y = x @ W.T + b

Two details save a lot of debugging. The weight is stored as (out_features, in_features), and the layer only transforms the last dimension of your input.

Shapes, weights and errors below are real output from Python 3.12.5, PyTorch 2.14.0+cpu.

Using nn.Linear in PyTorch

Create the layer with the two sizes, then call it like a function:

import torch
import torch.nn as nn

torch.manual_seed(0)

layer = nn.Linear(in_features=4, out_features=3)

x = torch.randn(2, 4)          # a batch of 2 samples, 4 features each
y = layer(x)

print("input :", tuple(x.shape))
print("output:", tuple(y.shape), "<- 4 features became 3")
print()
print(y)

Output:

input : (2, 4)
output: (2, 3) <- 4 features became 3

tensor([[-0.5676,  0.0106, -0.2052],
        [-1.2508,  0.6045, -0.4310]], grad_fn=<AddmmBackward0>)

The input’s last dimension must equal in_features. The output has the same shape with that last number swapped for out_features.

Calling layer(x) runs forward. Never call layer.forward(x) directly, because that skips the hooks PyTorch relies on.

Why the nn.Linear weight shape looks backwards

This trips up everyone reading the weights for the first time:

import torch
import torch.nn as nn

torch.manual_seed(0)
layer = nn.Linear(4, 3)

print("Linear(4, 3)")
print("  weight:", tuple(layer.weight.shape), "  <- (out_features, in_features)")
print("  bias  :", tuple(layer.bias.shape))
print()
print("that is the TRANSPOSE of what most people expect, because the")
print("forward pass is  y = x @ W.T + b")
print()

x = torch.randn(2, 4)
manual = x @ layer.weight.T + layer.bias
print("layer(x) equals x @ W.T + b ?", torch.allclose(layer(x), manual))

print()
print("parameter count:", sum(p.numel() for p in layer.parameters()), "= 4*3 weights + 3 biases")

Output:

Linear(4, 3)
  weight: (3, 4)   <- (out_features, in_features)
  bias  : (3,)

that is the TRANSPOSE of what most people expect, because the
forward pass is  y = x @ W.T + b

layer(x) equals x @ W.T + b ? True

parameter count: 15 = 4*3 weights + 3 biases

The layer computes x @ W.T + b, so storing W transposed means the transpose happens once at definition rather than on every forward pass.

The allclose check in that output confirms it. Writing the maths by hand gives the same numbers as calling the layer.

Attribute	Shape for Linear(4, 3)
layer.weight	(3, 4)
layer.bias	(3,)
Parameters	4 × 3 + 3 = 15

What input shapes does nn.Linear accept?

Any shape at all, as long as the last dimension matches:

import torch
import torch.nn as nn

layer = nn.Linear(4, 3)

# nn.Linear transforms the LAST dimension and leaves the rest alone
for shape in [(4,), (2, 4), (5, 7, 4), (8, 16, 32, 4)]:
    x = torch.randn(*shape)
    print(f"{str(shape):<18} -> {tuple(layer(x).shape)}")

print()
print("only the final 4 has to match in_features")
print("everything in front of it is treated as batch dimensions")

Output:

(4,)               -> (3,)
(2, 4)             -> (2, 3)
(5, 7, 4)          -> (5, 7, 3)
(8, 16, 32, 4)     -> (8, 16, 32, 3)

only the final 4 has to match in_features
everything in front of it is treated as batch dimensions

This is why nn.Linear works unchanged on sequence data shaped (batch, sequence, features). Each position is transformed independently.

It also means you rarely need to reshape before a linear layer, only flatten when coming from a convolution. See PyTorch flatten for that step.

Fixing the nn.Linear mat1 and mat2 shape error

This is the most common nn.Linear error, and the numbers in it tell you exactly what to change:

import torch
import torch.nn as nn

layer = nn.Linear(4, 3)          # expects 4 features

wrong = torch.randn(2, 5)        # but this has 5

try:
    layer(wrong)
except RuntimeError as err:
    print("RuntimeError:", err)

print()
print("read the numbers: mat1 is your input (2x5), mat2 is the weight (4x3)")
print("the 5 and the 4 are what disagree, so in_features should be 5")

Output:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x5 and 4x3)

read the numbers: mat1 is your input (2x5), mat2 is the weight (4x3)
the 5 and the 4 are what disagree, so in_features should be 5

• mat1 is your input, printed as batch × features.
• mat2 is the weight, printed as in_features × out_features.
• The two inner numbers must match. Here 5 and 4 do not.
• Fix it by setting in_features=5, or by reshaping the input.

When the input comes from a convolution, print x.shape just before the linear layer rather than calculating it by hand. The flattened size is easy to get wrong.

How nn.Linear initialises its weights

PyTorch does not start from zeros. Weights are drawn from a uniform distribution whose width depends on the input size:

import math
import torch
import torch.nn as nn

torch.manual_seed(0)

for in_features in (4, 100, 1000):
    layer = nn.Linear(in_features, 5)
    bound = 1 / math.sqrt(in_features)
    print(f"in_features={in_features:<5} bound=+/-{bound:.4f}  "
          f"actual min={layer.weight.min():.4f} max={layer.weight.max():.4f}")

print()
print("weights start uniform on +/- 1/sqrt(in_features)")
print("wider layers start with smaller weights, which keeps activations stable")

Output:

in_features=4     bound=+/-0.5000  actual min=-0.4777 max=0.3964
in_features=100   bound=+/-0.1000  actual min=-0.0998 max=0.1000
in_features=1000  bound=+/-0.0316  actual min=-0.0316 max=0.0316

weights start uniform on +/- 1/sqrt(in_features)
wider layers start with smaller weights, which keeps activations stable

The bound is 1 / sqrt(in_features). A layer with 1000 inputs starts with much smaller weights than one with 4.

That scaling keeps the size of the activations roughly constant as data moves through the network, which is what stops deep models exploding or vanishing on the first pass.

Initialising a linear layer to all zeros would make every output identical, and every gradient identical with it, so the layer could never learn.

Turning off the nn.Linear bias with bias=False

The bias is on by default. Switching it off replaces the tensor with None entirely:

import torch
import torch.nn as nn

with_bias = nn.Linear(4, 3)
without = nn.Linear(4, 3, bias=False)

print("bias=True  ->", tuple(with_bias.bias.shape))
print("bias=False ->", without.bias, "  <- None, not a tensor of zeros")

print()
print("parameters with bias   :", sum(p.numel() for p in with_bias.parameters()))
print("parameters without bias:", sum(p.numel() for p in without.parameters()))

print()
print("drop the bias when the next layer is a BatchNorm, which adds its own shift")

Output:

bias=True  -> (3,)
bias=False -> None   <- None, not a tensor of zeros

parameters with bias   : 15
parameters without bias: 12

drop the bias when the next layer is a BatchNorm, which adds its own shift

That matters if you loop over parameters: layer.bias is None, not a tensor of zeros, so a naive .shape call raises. The same care applies when you summarise a model.

The usual reason to drop it is a following BatchNorm layer, which subtracts a mean and adds its own shift, making the linear bias redundant.

Stacking nn.Linear layers into a network

A classifier is a few linear layers with a non-linear function between them:

import torch
import torch.nn as nn

torch.manual_seed(0)

model = nn.Sequential(
    nn.Linear(784, 128),     # 28x28 image flattened
    nn.ReLU(),
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Linear(64, 10),       # 10 classes
)

x = torch.randn(32, 784)     # a batch of 32
print("input :", tuple(x.shape))
print("output:", tuple(model(x).shape))

print()
total = sum(p.numel() for p in model.parameters())
print(f"total parameters: {total:,}")
for name, module in model.named_children():
    if isinstance(module, nn.Linear):
        n = sum(p.numel() for p in module.parameters())
        print(f"  layer {name}: {module.in_features:>4} -> {module.out_features:<4} {n:>8,} params")

print()
print("without the ReLU between them, three Linear layers collapse into one")

Output:

input : (32, 784)
output: (32, 10)

total parameters: 109,386
  layer 0:  784 -> 128   100,480 params
  layer 2:  128 -> 64      8,256 params
  layer 4:   64 -> 10        650 params

without the ReLU between them, three Linear layers collapse into one

Look at where the parameters concentrate. The first layer holds roughly 100,000 of them because 784 inputs is a lot to connect.

The ReLU between the layers is not decoration. Two linear layers with nothing between them are mathematically equivalent to a single linear layer, so the extra depth would buy you nothing.

For the loss function that usually follows, see cross entropy loss or MSELoss.

nn.LazyLinear when you don’t know the input size

If the input size depends on earlier layers, LazyLinear works it out on the first forward pass:

import torch
import torch.nn as nn

# LazyLinear works out in_features the first time you pass data through
layer = nn.LazyLinear(out_features=3)
print("before any input:", type(layer.weight).__name__)

layer(torch.randn(2, 6))
print("after a (2, 6) input:", tuple(layer.weight.shape), "<- it inferred 6")

print()
print("useful when the input size depends on earlier layers you have not worked out yet")

Output:

before any input: UninitializedParameter
after a (2, 6) input: (3, 6) <- it inferred 6

useful when the input size depends on earlier layers you have not worked out yet

Before any data arrives the weight is an UninitializedParameter with no shape. Afterwards it is an ordinary linear layer.

It saves hand-calculating the flattened size after a stack of convolutions. Run one batch through, then read the shape back.

Common nn.Linear mistakes

Symptom	Cause	Fix
mat1 and mat2 shapes cannot be multiplied	Input features do not match	Match in_features to the input
Weight shape looks transposed	Stored as (out, in)	Expected, the forward uses W.T
AttributeError on bias.shape	bias=False makes it None	Check for None first
Model will not learn	No activation between layers	Add ReLU between them
Shape error after a conv layer	Not flattened	Add nn.Flatten()

More PyTorch layer guides:

• PyTorch flatten
• PyTorch nn.Conv2d
• PyTorch Conv1d
• PyTorch MSELoss
• Cross entropy loss in PyTorch
• PyTorch model summary

Frequently asked questions