PyTorch Batch Normalization

In this Python tutorial, we will learn about PyTorch batch normalization in python and we will also cover different examples related to Batch Normalization using PyTorch. And, we will cover these topics.

PyTorch batch normalization
PyTorch batch normalization example
PyTorch batch normalization implementation
PyTorch batch normalization 1d
PyTorch batch normalization 2d
PyTorch batch normalization 3d
PyTorch batch normalization lstm
PyTorch batch normalization conv2d
PyTorch batch normalization running mean
PyTorch batch normalization eval mode

Table of Contents

PyTorch batch normalization

In this section, we will learn about how exactly the bach normalization works in python. And for the implementation, we are going to use the PyTorch Python package.

Batch Normalization is defined as the process of training the neural network which normalizes the input to the layer for each of the small batches.
This process stables the learning process and also reduces the number of epochs require to train the model.

Code:

In the following code, we will import some libraries from which we can train the deep neural network.

nn.BatchNorm1d is used to normalize the data to 0 mean and the unit variance.
inputval = torch.randn(22, 102) is used to generate the random numbers.
outputval = n(input) is used to get the output value.
print(outputval) is used to print the output on the screen.

import torch
import torch.nn as nn
n = nn.BatchNorm1d(102)
# Without Learnable Parameters
n = nn.BatchNorm1d(102, affine=False)
inputval = torch.randn(22, 102)
outputval = n(input)
print(outputval)

Output:

After running the above code, we get the following output in which we can see that the batch normalization is done on the screen.

Also, check: PyTorch Save Model

PyTorch batch normalization example

In this section, we will learn about how the PyTorch batch normalization example runs in python.

Pytorch batch normalization is a process of training the neural network. During training the network this layer keep guessing its computed mean and variance.

Code:

In the following code, we will import some libraries from which we can train the neural network and also evaluate its computed mean and variance.

nn.Flatten() is used to define a continuous range of dim into a tensor.
nn.Linear() is used to create a feed-forward network.
nn.BatchNorm1d() is used to normalize the data to 0 mean and the unit variance.
torch.manual_seed(44) is used to set the fixed random number seed.
mlp = Multilayerpercepron() is used to initialize the multilayer perceptron.
currentloss = 0.0 is used to set the current loss value.
optimizer.zero_grad() is used to zero the gradients.

import torch
import os
from torch import nn
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader
from torchvision import transforms

class Multilayerpercepron(nn.Module):
 
  def __init__(self):
    super().__init__()
    self.layers = nn.Sequential(
      nn.Flatten(),
      nn.Linear(32 * 32 * 3, 64),
      nn.BatchNorm1d(64),
      nn.ReLU(),
      nn.Linear(64, 32),
      nn.BatchNorm1d(32),
      nn.ReLU(),
      nn.Linear(32, 10)
    )


  def forwardfun(self, y):
    '''Forward pass'''
    return self.layers(y)
  
  
if __name__ == '__main__':
  
  torch.manual_seed(44)
  
  # Prepare CIFAR-10 dataset
  dataset = CIFAR10(os.getcwd(), download=True, transform=transforms.ToTensor())
  trainloader = torch.utils.data.DataLoader(dataset, batch_size=10, shuffle=True, num_workers=1)
 
  mlp = Multilayerpercepron()
  
  # Define the loss function and optimizer
  lossfunction = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(mlp.parameters(), lr=1e-4)
  
  # Run the training loop
  for epoch in range(0, 7): 
    
    # Print epoch
    print(f'Starting epoch {epoch+1}')
    
    currentloss = 0.0
    
    # Iterate over the DataLoader for training data
    for i, data in enumerate(trainloader, 0):
      
      # Get inputs
      inputs, targets = data
      optimizer.zero_grad()
      # Perform forward pass
      outputs = mlp(inputs)
      
      # Compute loss
      loss = lossfunction(outputs, targets)
      
      # Perform backward pass
      loss.backward()
      
      # Perform optimization
      optimizer.step()
      
      # Print statistics
      currentloss += loss.item()
      if i % 502 == 499:
          print('Loss after mini-batch %5d: %.3f' %
                (i + 1, currentloss / 502))
          currentloss = 0.0

  print('Training process has been finished.')

Output:

In the following output, we can see the train neural network that normalizes input to the layer for each of the small batches and also reduces the number of epochs.

Also, read: Tensorflow in Python

PyTorch batch normalization implementation

In this section, we will learn about how to implement PyTorch batch normalization in Python.

PyTorch batch normalization implementation is used to train the deep neural network which normalizes the input to the layer for each of the small batches.

Code:

In the following code, we will import some libraries from which we can implement batch normalization.

train_dataset=datasets.MNIST() is used as the training dataset.
test_dataset=datasets.MNIST() is used as the test dataset.
nn.BatchNorm2d() is used as the number of dimensions that output from the last layer and come into the batch norm layer.
nn.Dropout() is used as a dropout unit in a neural network.
torch.flatten() is used as flatter input by reshaping it into a one-dimension tensor.
test_loss+=f.nll_loss(output,label,reduction=’sum’).item() is used to calculate the test loss.
pred=output.argmax(dim=1,keepdim=True) is used to predict the output.
accuracy+=pred.eq(label.view_as(pred)).sum().item() is used to predict the accuracy.
print(‘\nTest Set: Average loss: {:.6f}, Accuracy: {}’.format( test_loss, accuracy)) is used to to print the test set.

import torch
import torch.nn as nn
import torch.nn.functional as fun
import torch.optim as opt

from torchvision import datasets,transforms
from torch.optim.lr_scheduler import StepLR
torch.manual_seed(52)
batch_size=34
epochs=12
lr=0.01
is_cuda=torch.cuda.is_available()
device=torch.device("cuda" if is_cuda else "cpu")
print(device)
transform=transforms.Compose([
                              transforms.ToTensor(),
                              transforms.Normalize((0.1309,),(0.3082,))])

train_dataset=datasets.MNIST('../data',train=True,download=True,transform=transform)
test_dataset=datasets.MNIST('../data',train=False,transform=transform)
train_loader=torch.utils.data.DataLoader(train_dataset,batch_size)
test_loader=torch.utils.data.DataLoader(test_dataset,batch_size)
class model(nn.Module):
  def __init__(self):
    super(model,self).__init__()
    self.conv=nn.Conv2d(1,32,3,1)
    self.conv_bn=nn.BatchNorm2d(32)
    
    self.conv1=nn.Conv2d(32,64,3,1)
    self.conv1_bn=nn.BatchNorm2d(64)
    
    self.dropout=nn.Dropout(0.25)
    
    self.fc=nn.Linear(9216,128)
    self.fc_bn=nn.BatchNorm1d(128)
    
    self.fc1=nn.Linear(128,10)
  def forwardfun(self,y):
    y=self.conv(y)
    y=fun.relu(self.conv_bn(y))
    
    y=self.conv1(y)
    y=fun.relu(self.conv1_bn(y))
    
    y=fun.max_pool2d(y,2)
    y=self.dropout(y)
    
    y=torch.flatten(y,1)
    
    y=self.fc(y)
    y=fun.relu(self.fc_bn(y))
    
    y=self.fc1(y)
    output=fun.log_softmax(y,dim=1)
    return output
model=model().to(device)
print(model)
optimizer=opt.Adadelta(model.parameters(),lr=lr)
def train(epoch):
  model.train()
  for batch_id,(image,label) in enumerate(train_loader):
    image,label=image.to(device),label.to(device)
    optimizer.zero_grad()
    output=model(image)
    loss=fun.nll_loss(output,label)
    loss.backward()
    optimizer.step()

    if batch_id % 1000==0:
      print('Train Epoch: {} \tLoss: {:.8f}'.format(epoch,loss.item()))

def test():
  model.eval()
  test_loss=0
  accuracy=0
  with torch.no_grad():
    for image,label in test_loader:
      image,label=image.to(device),label.to(device)
      output=model(image)

      test_loss+=fun.nll_loss(output,label,reduction='sum').item()

      pred=output.argmax(dim=1,keepdim=True)

      accuracy+=pred.eq(label.view_as(pred)).sum().item()

    test_loss/=len(test_loader.dataset)

    print('\nTest Set: Average loss: {:.6f}, Accuracy: {}'.format(
test_loss, accuracy))
#after batch normalization
scheduler=StepLR(optimizer,step_size=1,gamma=0.7)
for epoch in range(1,epochs+1):
  train(epoch)
  test()
  scheduler.step()

Output:

After running the above code, we get the following output in which we can see that the train epoch loss is printed on the screen.

PyTorch batch normalization implementation

Read: Pandas in Python

PyTorch batch normalization 1d

In this section, we will learn about the PyTorch batch normalization 1d in python.

PyTorch batch normalization 1d is a technique used to build a neural network faster and more stable.

Syntax:

The following syntax is of batch normalization 1d.

torch.nn.BatchNorm1d(num_features,eps=1e-05,momentum=0.1,affine=True,track_running_status=True,device=None,dtype=None)

Parameters used in batch normalization1d :

num_features is defined as C the expected input of size (N, C, L).
eps is used as a demonstrator to add a value for numerical stability.
momentum is used as a value running_mean and running_var computation.
affine is defined as a boolean value if the value is set to true this module has learnable affine parameters.
track_running_status is defined as a boolean value if the value is set to true this module tracks the running mean and variance. If the value is set to false this module does not track the running mean and the variance.

Example:

In the following example, we will import some libraries from which we are creating the batch normalization 1d.

a = nn.BatchNorm1d(120) is a learnable parameter.
a = nn.BatchNorm1d(120, affine=False) is used as without learnable parameter.
inputs = torch.randn(40, 120) is used to generate the random inputs.
print(outputs) is used to print the output values.

import torch
from torch import nn

a = nn.BatchNorm1d(120)

a = nn.BatchNorm1d(120, affine=False)
inputs = torch.randn(40, 120)
outputs = a(inputs)
print(outputs)

Output:

In the following output, we can see the batch normalization 1d value is printed on the screen.

Read: PyTorch Tensor to Numpy

PyTorch batch normalization 2d

In this section, we will learn about the PyTorch batch normalization 2d in python.

PyTorch batch normalization 2d is a technique to construct the deep neural network and the batch norm2d is applied to batch normalization above 4D input.

Syntax:

The following syntax is of batch normalization 2d.

torch.nn.BatchNorm2d(num_features,eps=1e-05,momentum=0.1,affine=True,track_running_statats=True,device=None,dtype=None)

Parameters used in natch normalization 2d:

num_features is defined as C the expected input of size (N, C, H,W).
eps is used as a demonstrator to add a value for numerical stability.
momentum is used as a value running_mean and running_var computation.
affine is defined as a boolean value if the value is set to true this module has learnable affine parameters.
track_running_status is defined as a boolean value if the value is set to true this module tracks the running mean and variance. If the value is set to false this module does not track the running mean and the variance.

Example:

In the following example, we will import some libraries from which we are creating the batch normalization 2d.

a = nn.BatchNorm2d(120) is used with learnable parameters.
a = nn.BatchNorm2d(120, affine=False) is used without learnable parameters.
inputs = torch.randn(20, 120, 55, 65) is used to generates the random numbers.
outputs = a(inputs) is used to get the output.
print(outputs) is used to print the output.

import torch
from torch import nn

a = nn.BatchNorm2d(120)

a = nn.BatchNorm2d(120, affine=False)
inputs = torch.randn(20, 120, 55, 65)
outputs = a(inputs)
print(outputs)

Output:

After running the above code, we get the following output in which we can see that the PyTorch batch normalization 2d data is printed on the screen.

Read: What is Scikit Learn in Python

PyTorch batch normalization 3d

In this section, we will learn about the PyTorch batch normalization 3d in python.

PyTorch bach normalization 3d is defined as a process to create deep neural networks and the bachnorm3d is applied to batch normalization above 5D inputs.

Syntax:

The following syntax is of batch normalization 3d.

torch.nn.BatchNorm3d(num_features,eps=1e-05,momentum=0.1,affine=True,track_running_status=True,device=None,dtype=None)

Parameters used in batch normalization 3d:

num_features is defined as C the expected input of size (N, C, D, H, W).
eps is used as a demonstrator to add a value for numerical stability.
momentum is used as a value running_mean and running_var computation.
affine is defined as a boolean value if the value is set to true this module has learnable affine parameters.
track_running_status is defined as a boolean value if the value is set to true this module tracks the running mean and variance. If the value is set to false this module does not track the running mean and the variance.

Example:

In the following example, we will import some libraries from which we can create a batch normalization 3d.

a = nn.BatchNorm3d(130) is used with learnable parameters.
a = nn.BatchNorm3d(130, affine=False) is used without learnable parameters.
inputs = torch.randn(50, 130, 65, 75, 40) is used to generate random number of inputs.
print(outputs) is used to print the outpputs.

import torch
from torch import nn

a = nn.BatchNorm3d(130)
a = nn.BatchNorm3d(130, affine=False)
inputs = torch.randn(50, 130, 65, 75, 40)
outputs = a(inputs)
print(outputs)

Outputs:

After running the above code, we get the following output. In the output, we can see that the BatchNorm3d applied to batch normalization 3d above 5D inputs and the output is printed on the screen.

Read PyTorch Binary Cross Entropy

PyTorch batch normalization lstm

In this section, we will learn PyTorch batch normalization lstm in python.

LSTM stands for long short-term memory. The LSTM is a class of Recurrent neural networks and the recurrent neural network is a class of artificial neural networks.
In PyTorch, batch normalization lstm is defined as the process create to automatically normalized the inputs to a layer in a deep neural network.

Code:

In the following code, we will import some libraries from which we can create the deep neural network and automatically normalized input to the layer.

nn.Flatten() is used as flatten input by reshaping it into a one-dimensional tensor.
nn.Linear() is used to create a feed-forward network.
torch.manual_seed(44) is used to set a fixed random number of seeds.
dataset = CIFAR10() is used to prepare a CIFAR-10 dataset.
mlp = mlp() is used to initialize the multilayer perceptron.
lossfunction = nn.CrossEntropyLoss() is used to define the loss function.
print(f’Starting epoch {epoch+1}’) is used to print epochs.
currentloss = 0.0 is used to set the current loss value.
loss = lossfunction(output, target) is used to compute the loss.
loss.backward() is used to perform the backward pass.
optimizers.step() is used to perform optimization.

import torch
import os
from torch import nn
from torchvision.datasets import CIFAR10
from torch.utils.data import DataLoader
from torchvision import transforms

class mlp(nn.Module):
 
  def __init__(self):
    super().__init__()
    self.layers = nn.Sequential(
      nn.Flatten(),
      nn.Linear(32 * 32 * 3, 64),
      nn.BatchNorm1d(64),
      nn.ReLU(),
      nn.Linear(64, 32),
      nn.BatchNorm1d(32),
      nn.ReLU(),
      nn.Linear(32, 10)
    )


  def forwardfun(self, y):
    '''Forward pass'''
    return self.layers(y)
  
  
if __name__ == '__main__':
  

  torch.manual_seed(44)
  

  dataset = CIFAR10(os.getcwd(), download=True, transform=transforms.ToTensor())
  trainloader = torch.utils.data.DataLoader(dataset, batch_size=10, shuffle=True, num_workers=1)
  

  mlp = mlp()
  

  lossfunction = nn.CrossEntropyLoss()
  optimizers = torch.optim.Adam(mlp.parameters(), lr=1e-4)
  
  # Run the training loop
  for epoch in range(0, 10): 
    

    print(f'Starting epoch {epoch+1}')
    
    
    currentloss = 0.0
    
    # Iterate over the DataLoader for training data
    for i, data in enumerate(trainloader, 0):
      

      input, target = data
      

      optimizers.zero_grad()
      

      output = mlp(input)
      

      loss = lossfunction(output, target)
      

      loss.backward()
    
      optimizers.step()
      
      # Print statistics
      currentloss += loss.item()
      if i % 550 == 499:
          print(' Loss After Mini-batch  %5d: %.3f' %
                (i + 1, currentloss / 550))
          currentloss = 0.0

Output:

After running the above code, we get the following output in which we can see that the PyTorch batch normalization lstm is printed on the screen.

Read PyTorch Logistic Regression

PyTorch batch normalization conv2d

In this section, we will learn about how PyTorch batch normalization conv2d works in python.

Before moving forward should have a piece of knowledge about conv2d. The conv2d is a 2d convolution layer that creates the convolution kernel.
It also binds with the layer inputs and helps to produce tensor outputs.
Batch normalization is a technique to make neural networks faster and more stable.

Code:

In the following code, we will import some libraries from which we can create a neural network with the help of conv2d.

transform=transforms.Compose() is used to transform the data.
train_dataset=datasets.MNIST() is used to create the train dataset.
test_dataset=datasets.MNIST() is used to create the test dataset.
train_loader=torch.utils.data.DataLoader() is used to load the train data.
test_loader=torch.utils.data.DataLoader() is used to load the test data.
nn.BatchNorm2d() is used as the number of dimensions that output from the last layer and come into the batch norm layer.
nn.Dropout() is used as a dropout unit in a neural network.
torch.flatten() is used as flatter input by reshaping it into a one-dimension tensor.

import torch
import torch.nn as nn
import torch.nn.functional as fun
import torch.optim as opt

from torchvision import datasets,transforms
from torch.optim.lr_scheduler import StepLR
torch.manual_seed(52)
batch_size=34
epochs=12
lr=0.01
is_cuda=torch.cuda.is_available()
device=torch.device("cuda" if is_cuda else "cpu")
print(device)
transform=transforms.Compose([
                              transforms.ToTensor(),
                              transforms.Normalize((0.1309,),(0.3082,))])

train_dataset=datasets.MNIST('../data',train=True,download=True,transform=transform)
test_dataset=datasets.MNIST('../data',train=False,transform=transform)
train_loader=torch.utils.data.DataLoader(train_dataset,batch_size)
test_loader=torch.utils.data.DataLoader(test_dataset,batch_size)
class model(nn.Module):
  def __init__(self):
    super(model,self).__init__()
    self.conv=nn.Conv2d(1,32,3,1)
    self.conv_bn=nn.BatchNorm2d(32)
    
    self.conv1=nn.Conv2d(32,64,3,1)
    self.conv1_bn=nn.BatchNorm2d(64)
    
    self.dropout=nn.Dropout(0.25)
    
    self.fc=nn.Linear(9216,128)
    self.fc_bn=nn.BatchNorm1d(128)
    
    self.fc1=nn.Linear(128,10)
  def forwardfun(self,y):
    u=self.conv1(y)
    y=fun.relu(self.conv1_bn(y))
    
    y=self.conv2(y)
    y=fun.relu(self.conv2_bn(y))
    
    y=fun.max_pool2d(y,2)
    y=self.dropout1(y)
    
    y=torch.flatten(y,1)
    
    y=self.fc(y)
    y=fun.relu(self.fc_bn(y))
    
    y=self.fc1(y)
    output=fun.log_softmax(y,dim=1)
    return output
model=model().to(device)
print(model)

Output:

After running the above code, we get the following output in which we can see that the convolution layer binds with inputs and helps to produce tensor output.

PyTorch batch normalization running mean

In this section, we will learn about how to calculate the PyTorch batch normalization running mean in Python.

PyTorch batch normalization running mean is defined as a process of training the neural network.
During training the layer to keep running and in the running process they estimated the computed mean.

Code:

In the following code, we will import some libraries from which we can calculate the running mean.

exponential_average_factor = 1.0 / float(self.num_batches_tracked) is use cumulation moving average.
exponential_average_factor = self.momentum is used exponential moving average.
var = input.var([0, 2, 3], unbiased=False) is use baised var in train.
self.running_var = exponential_average_factor * var * m / (m – 1)\+ (1 -exponential_average_factor) * self.running_var is used to updating the running_var with unbiased var.
print(“printing bn1 running mean from NET during forward”) is used to print the output.

import torch
import torch.nn as nn
import torch.nn.functional as fun
import torch.optim as opt
from torch.distributions import uniform
import torch.backends.cudnn as cudnn
import torchvision
import torchvision.transforms as transforms
from torch.nn.parameter import Parameter

class BatchNorm2d(nn.BatchNorm2d):
    def __init__(self, num_features, eps=1e-5, momentum=0.1,
                 affine=True, track_running_stats=True):
        super(BatchNorm2d, self).__init__(
            num_features, eps, momentum, affine, track_running_stats)

    def forward(self, input):
        self._check_input_dim(input)

        exponential_average_factor = 0.0

        if self.training and self.track_running_stats:
            if self.num_batches_tracked is not None:
                self.num_batches_tracked += 1
                if self.momentum is None: 
                    exponential_average_factor = 1.0 / float(self.num_batches_tracked)
                else:  
                    exponential_average_factor = self.momentum

        # calculate running estimates
        if self.training:
            mean = input.mean([0, 2, 3])

            var = input.var([0, 2, 3], unbiased=False)
            m = input.numel() / input.size(1)
            with torch.no_grad():
                self.running_mean = exponential_average_factor * mean\
                    + (1 - exponential_average_factor) * self.running_mean
           
                self.running_var = exponential_average_factor * var * m / (m - 1)\
                    + (1 - exponential_average_factor) * self.running_var
        else:
            mean = self.running_mean
            var = self.running_var

        input = (input - mean[None, :, None, None]) / (torch.sqrt(var[None, :, None, None] + self.eps))
        if self.affine:
            input = input * self.weight[None, :, None, None] + self.bias[None, :, None, None]

        return input


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.bn = BatchNorm2d(64)
        print(" printing bn mean when init")
        print(self.bn.running_mean)
        print(" printing bn when init")
        print(self.bn.running_mean)
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.classifier = nn.Linear(64, 10)

    def forward(self, y):
        y = self.conv(y)
        y = self.bn(y)
        y = fun.relu(y)
        y = self.pool(y)
        y = self.avgpool(y)
        y = y.view(y.size(0), -1)
        y = self.classifier(y)
      
        print("printing bn running mean from NET during forward")
        print(net.module.bn.running_mean)
        print("printing bn running mean from SELF. during forward")
        print(self.bn.running_mean)
        print("printing bn running var from NET during forward")
        print(net.module.bn.running_var)
        print("printing bn running mean from SELF. during forward")
        print(self.bn.running_var)
        return y

# Data
print('Preparing data..')
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])


transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])


train_set = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=64, shuffle=True, num_workers=2)
test_set = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)
test_loader = torch.utils.data.DataLoader(test_set, batch_size=64, shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# Model
print('Building model..')
net = Net()
net = torch.nn.DataParallel(net).cuda()
print('Number of GPU {}'.format(torch.cuda.device_count()))

criterion = nn.CrossEntropyLoss()
optimizer = opt.SGD(net.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)

# Training
def train(epoch):
    print('\nEpoch: %d' % epoch)
    net.train()
    train_loss = 0
    correct = 0
    total = 0

    for batch_idx, (inputs, targets) in enumerate(train_loader):
        inputs, targets = inputs.cuda(), targets.cuda()
        outputs = net(inputs)
        loss = criterion(outputs, targets)
       
        print("printing bn running mean FROM net after forward")
        print(net.module.bn.running_mean)
        print("printing bn running var FROM net after forward")
        print(net.module.bn.running_var)
        
        break


for epoch in range(0, 1):
    train(epoch)

Output:

In the following output, we can see that the running mean of training data is calculated on the screen.

PyTorch batch normalization running mean

PyTorch batch normalization eval mode

In this section, we will learn about PyTorch batch normalization eval mode in python.

PyTorch batch normalization is defined as a process of training the neural network. And using the eval mode is just a kind of switch which works on certain layers of the training and the evaluating time.

Code:

In the following code, we will import some libraries from which we can train the network and eval that network.

torch.manual_seed(34) is used to set a fixed random seed.
X = torch.rand(200,1,200) is used to generate the random numbers.
print(batchnormalization .running_var) is used to print the batch normalization running var.
print(f’momentum = {momentum} yields {batchnormalization (X).mean()} for eval mode’) is used to print the eval mode.

import torch
import torch.nn as nn
for momentum in [None, 1]:
     torch.manual_seed(34)
     batchnormalization = nn.BatchNorm1d(1, momentum=momentum)
     X = torch.rand(200,1,200)

     print(batchnormalization .running_var)
     batchnormalization .train()
     print(f'momentum = {momentum} yields {batchnormalization (X).mean()} for train mode')
     print(batchnormalization .running_var)

     batchnormalization .running_var.data.mul_(1 - 1 / (200*200))
     batchnormalization .eval()
     print(f'momentum = {momentum} yields {batchnormalization (X).mean()} for eval mode')

Output:

After running the above code, we get the following output in which we can see the PyTorch batch normalization eval mode data is printed on the screen.

So, in this tutorial, we discussed PyTorch Batch Normalization and we have covered different examples related to its implementation. Here is the list of examples that we have covered.

PyTorch batch normalization
PyTorch batch normalization example
PyTorch batch normalization implementation
PyTorch batch normalization 1d
PyTorch batch normalization 2d
PyTorch batch normalization 3d
PyTorch batch normalization lstm
PyTorch batch normalization conv2d
PyTorch batch normalization running mean
PyTorch batch normalization eval mode

Bijay Kumar

I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.

enjoysharepoint.com/