python 3.x - Pytorch BatchNorm3d / InstanceNorm3d not working when data size (1,C,1,1,1)

Question

I'm training a neural network in PyTorch which at some point has a BatchNorm3d(C).

Normally, I'm training it with a batch size of 1, and the input of this specific level will then be of shape (1, C, 1, 1, 1).

Unfortunately, the BatchNorm then fails with the error message:

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 32, 1, 1, 1])

The same happens, when I use the InstanceNorm3d. It works perfectly fine when I use a batch size greater than two (i.e. the input will then be of shape (2, C, 1, 1, 1).

Does anyone know a solution to this problem? What am I missing?

The problem can be reproduced with the following snippet:

import torch
x_working = torch.ones([2,32,1,1,1])
x_not_working = torch.ones([1,32,1,1,1])
norm = torch.nn.InstanceNorm3d(32)
out_working = norm(x_working)
out_not_working=norm(x_not_working)

Answer 1

All these normalization layers keep running estimates of the mean and variance of your data over the batch dimension (see doc). Since the variance is computed with the unbiased estimator (notice the n-1 in the denominator), the computation cannot work with less than 2 data points. Therfore, you need a batch size of at least 2 to use these layers.

Note that the variance of 1 data point - if pytorch agreed to compute it - would always be 0, so not really interesting a result. Actually, Batchnorm is known to require significantly larger batch sizes (you often finds 32 or 64 in the scientific literature) to work properly. This strong requirement lead to the development of layerNorm and groupNorm (of which instanceNorm is a particular case I believe).