Tensorflow embedding_lookup

In this Python tutorial, we will learn how to use the tf.nn.embedding_lookup() function in Python TensorFlow. Also, we will cover the following topics.

• Tensorflow embedding_lookup
• Tensorflow embedding_lookup_sparse
• Tensorflow embedding_lookup_unique
• Tensorflow embedding_lookup mask
• Tensorflow embedding_lookup zero padding
• Tensorflow embedding_lookup gather
• Tensorflow embedding_lookup gradient

TensorFlow embedding_lookup

• In this section, we will discuss how to embed the list of tensors in Python TensorFlow.
• To perform this particular task, we are going to use the tf.nn.embedding_lookup() function and this function is used to generate lookups on the list of tensors.
• In simple words, this function defines the operation that recovers the rows of the first argument depending on the index number.

Syntax:

Here is the Syntax of tf.nn.embedding_lookup() function in Python TensorFlow.

tf.nn.embedding_lookup
                      (
                       params,
                       ids,
                       max_norm=None,
                       name=None
                      )

• It consists of a few parameters
  • params: This parameter indicates sharded embedding tensor.
  • ids: By default, it take int32 and int64 bit type and it looked up in params.
  • max_norm: By default, it takes none value and it is larger than its value.
  • name: By default, it takes none value and defines the name of the operation.

Example:

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
tens = [
    [0.2783, 0.8762, -0.2789],
    [0.1923, 0.8823, -0.1892],
    [0.8567, -0.99234, 0.8123],
    [0.156, 0.8567, 0.9562]
]
result = tf.constant(tens, dtype=tf.float32)
 
with tf.compat.v1.Session() as val:
  new_index_val = [0, 2]
  new_output = tf.nn.embedding_lookup(result, new_index_val)
 
  print(val.run(new_output))

In the following given code, we have imported the TensorFlow library and then created a session by importing the tf.compat.v1.disable_eager_execution() function.

After that, we have declared a variable and assigned the index values, and used the tf.nn.embedding_lookup() function.

Here is the implementation of the following given code.

Read: TensorFlow Get Variable

Tensorflow embedding_lookup_sparse

• In this section, we will discuss how to look up embedding for weights and ids in Python TensorFlow.
• To perform this particular task, we are going to use the tf.nn.embedding_lookup_sparse() function and this function will help the user to embed given weights and ids from a list of input tensors.

Syntax:

Let’s have a look at the Syntax and understand the working of tf.nn.embedding_lookup_space in Python TensorFlow.

tf.nn.embedding_lookup_sparse
                             (
                              params,
                              sp_ids,
                              sp_weights,
                              combiner=None,
                              max_norm=None,
                              name=None
                             )

• It consists of a few parameters
  • params: This parameter indicates the list of input tensors and the same shape.
  • sp_ids: This parameter specifies the sparse tensor of int64 ids.
  • sp_weights: In this parameter, we are going to specify all the weights and it must be taken to be 1.
  • combiner: By default, it takes the None value and it generates the weighted sum of the embedding result for each row.
  • max_norm: By default, it takes None value and it is larger than its value before combining.
  • name: By default, it is an optional parameter and it specifies the name of the operation.

Example:

import tensorflow as tf
import numpy as np
tf.compat.v1.disable_eager_execution()
tensor = tf.SparseTensor(indices=[[2], [3], [4]], values=[3, 6, 9], dense_shape=[4])

val_num = 10
val_size = 1
arr = np.array([15.0, 2.0, 5.0, 3.0, 2.0, 11.0, 7.0, 10.0, 15.0, 14.0])
result = tf.Variable(arr)

new_output = tf.nn.embedding_lookup_sparse(result, tensor, None)

with tf.compat.v1.Session() as sess:
    sess.run(tf.compat.v1.initialize_all_variables())
    print(sess.run(new_output))

In the above code, we have used the tf.SparseTensor() function and assign the index values and dense shape as an argument. After that, we have declared variables in which we have mentioned the size of the given input tensor.

Here is the implementation of the following given code.

Read: Python TensorFlow Placeholder

Tensorflow embedding_lookup_unique

• In this section, we will discuss how to get the unique values from an embedded lookup in Python TensorFlow.
• To perform this particular task, we are going to use the tf.contrib.layers.embedding_lookup_unique(), this function only works in Tensorflow 1.15.0 version, and this function avoids duplicates lookups.

Syntax:

Let’s have a look at the Syntax and understand the working of tf.contrib.layers.embedding_lookup_unique()

tf.contrib.layers.embedding_lookup_unique(
    params,
    ids,
    name=None
)

Example:

Let’s take an example and check how to get the unique values from an embedded lookup in Python TensorFlow.

Source Code:

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
tens = [
    [0.256, 0.134, -0.765],
    [0.843, 0.778, -0.281],
    [0.954, -0.1561, 0.1789],
    [0.167, 0.290, 0.178]
]
result = tf.constant(tens, dtype=tf.float32)
 
with tf.compat.v1.Session() as val:
  new_index_val = [0, 2]
  new_output = tf.contrib.layers.embedding_lookup_unique(result, new_index_val)
 
  print(val.run(new_output))

Here is the Output of the following given code.

Read: Tensorflow iterate over tensor

Tensorflow embedding_lookup mask

• In this section, we will discuss how to get the mask values by using embedding_lookup in Python TensorFlow.
• To perform this particular task, we are going to define the CustomEmbedding() function and then we create the input and output dimensions for input tensors.
• Next, we will set the condition self inputs, if it is not then self.mask_zero() return None if not then it will return 0.

Example:

Let’s take an example and check how to get the mask values through embedded lookup in Python TensorFlow.

Source Code:

import tensorflow as tf
import numpy as np
class CustomEmbedding(keras.layers.Layer):
    def __init__(self, new_inp_dim, new_out_dim, mask_zero=False, **kwargs):
        super(CustomEmbedding, self).__init__(**kwargs)
        self.input_dim = new_inp_dim
        self.output_dim = new_out_dim
        self.mask_zero = mask_zero

    def build(self, input_shape):
        self.embeddings = self.add_weight(
            shape=(self.input_dim, self.output_dim),
            initializer="random_normal",
            dtype="float32",
        )

    def call(self, inputs):
        return tf.nn.embedding_lookup(self.embeddings, inputs)

    def compute_mask(self, inputs, mask=None):
        if not self.mask_zero:
            return None
        return tf.not_equal(inputs, 0)


layer = CustomEmbedding(12, 36, mask_zero=True)
new_tens = np.random.random((2, 12)) * 9
new_tens = new_tens.astype("int32")

y = layer(new_tens)
mask = layer.compute_mask(new_tens)

print(mask)

In the following given code, we have created a random value by using the np.random.random() function and then assign the datatype as int32. Once you will execute this code the output displays boolean mask values.

Here is the Screenshot of the following given code.

Read: Convert list to tensor TensorFlow

Tensorflow embedding_lookup zero padding

• In this Program, we will discuss how to pad the zero values in embedding_lookup Python TensorFlow.
• To do this task we are going to use the tf.keras.preprocessing.sequence.pad_sequences() function and this function is used to get the same length of the given tensor.

Syntax:

Let’s have a look at the Syntax and understand the working of tf.keras.preprocessing.sequence.pad_sequences() function in Python Tensorflow.

tf.keras.preprocessing.sequence.pad_sequences(
    sequences,
    maxlen=None,
    dtype='int32',
    padding='pre',
    truncating='pre',
    value=0.0
)

• It consists of a few parameters
  • sequences: This parameter indicates the list of sequences.
  • maxlen: By default, it takes none value and specifies the maximum length of all the sequences. It will check the condition if the sequence is not provided then this parameter will be padded with the same length.
  • dtype: By default, it takes the int32 value and it is an optional parameter with a type of the output sequences.
  • padding: By default, it is a pre-value and it specifies the after or before each sequence.
  • truncating: It is an optional parameter and it removes value from sequences larger than max length.
  • value: By default, its value is 0 and it specifies the padding value.

Example:

Let’s take an example and check how to pad the zero values in embedding_lookup Python TensorFlow.

Source Code:

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

new_tens = [
    [26, 12, 90],
    [167, 211, 976, 245, 912],
    [197, 278, 23, 67, 856, 179],
]

new_outputs = tf.keras.preprocessing.sequence.pad_sequences(
    new_tens, padding="post"
)
print(new_outputs)

In the following given code, we have imported the numpy and TensorFlow library and then created a tensor. After that, we have used the tf.keras.preprocessing.sequence.pad_sequences() function and within this function, we have assigned the padding and tensor as an argument.

Here is the Screenshot of the following given code.

Read: Python TensorFlow expand_dims

Tensorflow embedding_lookup gather

• In this section, we will discuss how to gather the embedding_lookup in Python TensorFlow.
• To perform this particular task, we are going to use the tf.nn.embedding_lookup() function and this function is used to generate lookups on the list of tensors.

Syntax:

tf.nn.embedding_lookup
                      (
                       params,
                       ids,
                       max_norm=None,
                       name=None
                      )

Example:

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
input_tens = [
    [1, 4, -7],
    [5, 2, -8],
    [6, -8, 1],
    [9, 2, 0]
]
result = tf.constant(input_tens, dtype=tf.int32)
 
with tf.compat.v1.Session() as val:
  new_index_val = [2, 3]
  new_output = tf.nn.embedding_lookup(result, new_index_val)
 
  print(val.run(new_output))

In the following given code, we have imported the TensorFlow library and then created a session by importing the tf.compat.v1.disable_eager_execution() function.

After that, we have declared a variable and assigned the index values, and used the tf.nn.embedding_lookup() function.

Here is the implementation of the following given code.

Read: Python TensorFlow one_hot

Tensorflow embedding_lookup gradient

• In this section, we will discuss the gradient in embedding_lookup in Python TensorFlow.
• To do this task, we are going to use the tf.GradientTape() function and this function is used to record the operation for automatic differentiation.

Syntax:

Let’s have a look at the Syntax and understand the working of tf.GradientTape() function in Python TensorFlow.

tf.GradientTape(
    persistent=False, watch_accessed_variables=True
)

• It consists of a few parameters.
  • persistent: By default, it takes a false value and it specifies the persistent gradient tape is declared.
  • watch_accessed_varaibles: This parameter indicates the true meaning gradients and can be represented from any result that is derived from reading a training variable.

Example:

Let’s take an example and check how the gradient in embedding_lookup in Python TensorFlow.

Source Code:

import tensorflow as tf

tens = tf.constant(7.0)
with tf.GradientTape() as m:
  m.watch(tens)
  with tf.GradientTape() as mm:
    mm.watch(tens)
    y = tens * tens
  new_result = mm.gradient(y, tens) 
new_result2 = m.gradient(new_result, tens)  
print(new_result2)

In the following given code, we have created the tensor by using the tf.constant() function, and then we have defined the built-in function tf.GradientTape().

Here is the Screenshot of the following given code.

You may also like to read the following TensorFlow tutorials.

• Python TensorFlow random uniform
• Python TensorFlow reduce_mean
• Python TensorFlow reduce_sum
• TensorFlow Tensor to numpy
• Gradient descent optimizer TensorFlow

In this Python tutorial, we have learned how to use the tf.nn.embedding_lookup() function in Python TensorFlow. Also, we have covered the following topics.

• Tensorflow embedding_lookup
• Tensorflow embedding_lookup_sparse
• Tensorflow embedding_lookup_unique
• Tensorflow embedding_lookup mask
• Tensorflow embedding_lookup zero padding
• Tensorflow embedding_lookup gather
• Tensorflow embedding_lookup gradient