Tensorflow custom loss function

In this Python tutorial, we will learn how to use the custom loss function in Python TensorFlow. Also, we will cover the following topics.

• Tensorflow custom loss function
• Tensorflow custom loss function Keras
• Tensorflow custom loss function multiple outputs
• Tensorflow custom loss function gradient
• Tensorflow custom loss function numpy
• Tensorflow load model with a custom loss function

Tensorflow custom loss function

• Here we are going to use the custom loss function in Python TensorFlow.
• Loss functions are the main parts of a machine learning model. If you have been working in data science then, you must have heard it.
• Loss functions, also called cost functions are the special types of functions, which help us minimize the error, and get the possibility to the expected output.
• The Mean Squared Error, the loss is the default loss to use for regression problems. The mean squared error loss function can be used in Keras by declaring ‘mse’ or ‘mean_squared_error’ as the loss function when compiling the model.

Syntax:

Here is the Syntax of tf.Keras.Sequential() function in TensorFlow Keras.

tf.keras.Sequential
                   (
                    layers=None,
                    name=None
                   )

• It consists of a few parameters
  • layers: This parameter indicates the list of layers to insert the model.
  • name: It is an optional parameter and it specifies the name of the operation.

Example:

import keras
import numpy as np

new_true_val = np.array([[28.0,16.0]]) 
new_pred_val = np.array([[13.0, 28.0]])

new_tens = keras.losses.MSE(new_true_val, new_pred_val)

print(f'Mean squared value {new_tens.numpy()}')
model = keras.Sequential([
                     keras.layers.Dense(20, input_shape=(1,), activation='relu'),
                     keras.layers.Dense(1)

])
model.compile(loss='mse', optimizer='adam')
model.fit(np.array([[15.0],[25.0], [55.0],[65.0],[75.0],[85.0],[95.0], [85.0]]), np.array([3, 18, 12,42,36, 72,18, 16]), epochs=10)

In the following given code first, we have imported the Keras and NumPy library. After that, we used the Keras.losses.MSE() function and assign the true and predicted value.

Next, we created a model by using the Keras.Sequential() function and within this function, we have set the input shape and activation value as an argument. After creating the model we have compiled and fit the model.

Here is the Screenshot of the following given code

Read: TensorFlow next_batch + Examples

Tensorflow custom loss function Keras

• In this section, we will discuss how to use the custom loss function in Tensorflow Keras.
• The main purpose of loss functions is to generate the quantity that a model should seek to minimize during training time. A loss function is one of the two parameters required for executing a Keras model.
• Loss functions are declaring by a loss class (e.g. keras.losses.SparseCategoricalCrossentropy). All losses are also given as function handles (e.g. keras.losses.sparse_categorical_crossentropy).
• In Keras, loss functions are passed during the compile stage. In this example, we’re defining the loss function by creating an instance of the loss class. Using the class is simple because you can pass some additional parameters.

Example:

Let’s take an example and check how to use the custom loss function in TensorFlow Keras.

Source Code:

import tensorflow as tf

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tensorflow.keras.utils import plot_model
from tensorflow.keras import backend as K

new_arr1= np.array([-2.0, 1.0, 1.0, 1.0, 6.0, 3.0], dtype=np.float64)

new_arr2 = np.array([-6.0, -3.0, 2.0, 1.0, 0.0, 4.0], dtype=np.float64)
plt.scatter(new_arr1, new_arr2);

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=[1])
])

model.compile(optimizer='sgd', loss='mean_squared_error')
model.fit(new_arr1, new_arr2, epochs=500, verbose=0)

new_mean_squ_val = model.predict([20])
new_mean_squ_val

plt.scatter(new_arr1, new_arr2)
plt.scatter(20, new_mean_squ_val, c='r');

def my_huber_loss(new_true_val, new_pred_val):
    new_thres_val = 1.
    error = new_true_val - new_pred_val
    smal_err = tf.abs(error) <= new_thres_val
    new_err_loss = tf.square(error) / 2
    max_err_loss = new_thres_val * (tf.abs(error) - new_thres_val / 2)
    return tf.where(smal_err, new_err_loss, max_err_loss)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(units=1, input_shape=[1,])
])

model.compile(optimizer='sgd', loss=my_huber_loss)
model.fit(new_arr1, new_arr2, epochs=500, verbose=0)

new_predict_val = model.predict([20.0])
new_predict_val

plt.scatter(new_arr1, new_arr2);
plt.scatter(20.0, new_mean_squ_val, label='mse');
plt.scatter(20.0, new_predict_val, label='huber_loss');
plt.grid()
plt.legend();

Here is the implementation of the following given code.

Read: TensorFlow Sparse Tensor

Tensorflow load model with a custom loss function

• In this example, we will learn how to load the model with a custom loss function in Python TensorFlow.
• To perform this particular task we are going to use the mnist.load_data() dataset and split the dataset into the train and test labels.
• Next, we will use the tf.Keras.models.Sequential() function and this function is used to add a linear stack of layers into the Keras model.

Example:

import os
from tensorflow import keras
import tensorflow as tf
(new_train_imag, new_train_label), (test_images, new_test_label) = tf.keras.datasets.mnist.load_data()

train_labels = new_train_label[:2000]
new_test_label = new_test_label[:2000]

new_train_imag = new_train_imag[:2000].reshape(-2, 14 * 14) / 255.0
test_images = test_images[:2000].reshape(-1, 14 * 14) / 255.0
def declare_new_model():
  new_model = tf.keras.models.Sequential([
    keras.layers.Dense(256, activation='relu', input_shape=(784,)),
    keras.layers.Dropout(0.7),
    keras.layers.Dense(12)
  ])

  new_model.compile(optimizer='adam',
                loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
                metrics=[tf.metrics.SparseCategoricalAccuracy()])

  return new_model


new_model = declare_new_model()

new_model.summary()

In the following given code we have used the tf.Keras.models.Sequential() function and within this function we have set the activation and input_Shape() value as an argument. After that, we used the model.compile() and use the tf.losses.SparseCategoricalCrossentropy().

Here is the Screenshot of the following given code.

Read: TensorFlow global average pooling

Tensorflow custom loss function gradient

• In this section, we will discuss how to use the gradient tape in the Tensorflow custom loss function.
• In this example, we are going to use the tf.GradientTape() function and this function is used to generate the gradient using operations in this model.

Syntax:

Let’s have a look at the Syntax and understand the working of the tf.gradients() function in Python TensorFlow

tf.gradients
   (
    ys,
    xs,
    grad_ys=None,
    name='gradients',
    gate_gradients=False,
    aggregation_method=None,
    stop_gradients=None,
   )

• It consists of a few parameters
  • ys: This parameter indicates the list of input tensors to be differentiated.
  • grad_ys: It is an optional parameter and it specifies the list of tensors with the same size as ys and computed each y in ys.
  • name: By default, it takes the value of the gradient and specifies the name of the operation.
  • gate_gradients: If it is true then it will add a tuple and also it will avoid some conditions. By default, it takes a false value.
  • aggregation_method: This function is used to combine gradient values.

Example:

import tensorflow as tf

def new_custom_loss(model, new_tensor):
    def cust_loss(new_true_val,new_pred_val):
        with tf.GradientTape() as t:
            t.watch(new_tensor)
            new_output = model(new_tensor)
        DyDX = t.gradient(new_output, new_tensor)
        new_val = DyDX[:, 5:6]
        new_r_prediction=new_val
        
        new_loss_pde = tf.reduce_mean(tf.square(new_r_prediction))
        return new_loss_pde
    return cust_loss
print(new_custom_loss)

In the above code, we have defined the cust_loss function and assigned the true and predicted value. After that, we created a session with tf.GradientTape() function and set the tensor value to it.

Here is the implementation of the following given code.

Read: Binary Cross Entropy TensorFlow

Tensorflow custom loss function numpy

• In this example, we are going to use the numpy array in the custom loss function.
• To do this task first we will create an array with sample data and find the mean squared value with the numpy() function.
• Next, we will use the tf.keras.Sequential() function and assign the dense value with input shape.

Syntax:

Here is the Syntax of tf.keras.Sequential() function in Python TensorFlow

tf.keras.Sequential
                   (
                    layers=None,
                    name=None
                   )

Example:

import keras
import numpy as np

new_true = np.array([[76.0,65.0]]) #sample data
new_prediction = np.array([[23.0, 56.0]])

new_arr = keras.losses.MSE(new_true, new_prediction)

print(f'Mean squared error {new_arr.numpy()}')
new_model = keras.Sequential([keras.layers.Dense(20, input_shape=(1,), activation='relu'),keras.layers.Dense(1)

])
new_model.compile(loss='mse', optimizer='adam')
new_model.fit(np.array([[80.0],[40.0], [60.0],[90.0],[40.0],[10.0],[70.0], [50.0]]), np.array([4, 8, 12,16,20, 24,28, 30]), epochs=10)

Here is the execution of the following given code

Also, take a look at some more TensorFlow tutorials.

• TensorFlow feed_dict
• TensorFlow Graph – Detailed Guide
• TensorFlow Get Variable + Examples
• TensorFlow Learning Rate Scheduler
• TensorFlow mean squared error
• Tensorflow iterate over tensor

In this Python tutorial, we have learned how to use the custom loss function in Python TensorFlow. Also, we have covered the following topics.

• Custom loss function TensorFlow Keras
• Tensorflow custom loss function multiple outputs
• Tensorflow custom loss function gradient
• Tensorflow custom loss function numpy
• Tensorflow load model with a custom loss function