As a developer, I was working on a deep learning project where I needed to understand the fundamental building blocks of TensorFlow. The issue is, many developers jump straight into using TensorFlow without truly understanding what tensors are and how they work. This can lead to confusion when debugging models or optimizing performance.
In this article, I’ll cover everything you need to know about tensors in TensorFlow, from basic concepts to practical implementations.
So let’s get started!
What is a Tensor?
Tensors are the core data structure in TensorFlow. Think of tensors as multi-dimensional arrays that flow through the computational graph of your neural network.
A tensor is defined by three key attributes:
- Rank: The number of dimensions (0D = scalar, 1D = vector, 2D = matrix, 3D+ = tensor)
- Shape: The size of each dimension
- Data Type: The type of data stored (float32, int32, string, etc.)
Tensors allow TensorFlow to optimize computations and efficiently run them on various hardware like CPUs, GPUs, and TPUs.
Read TensorFlow Variable
Methods to Create Tensors in TensorFlow
I will explain methods to create Tensors in TensorFlow.
1 – Use tf.constant()
The simple way to create a tensor is to use the tf.constant() function in Python:
import tensorflow as tf
# Scalar (rank 0)
scalar = tf.constant(7)
# Vector (rank 1)
vector = tf.constant([10, 20, 30, 40])
# Matrix (rank 2)
matrix = tf.constant([[1, 2], [3, 4]])
# 3D tensor (rank 3)
tensor_3d = tf.constant([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
print(f"Scalar shape: {scalar.shape}")
print(f"Vector shape: {vector.shape}")
print(f"Matrix shape: {matrix.shape}")
print(f"3D tensor shape: {tensor_3d.shape}")Output:
Scalar shape: ()
Vector shape: (4,)
Matrix shape: (2, 2)
3D tensor shape: (2, 2, 2)You can see the output in the screenshot below.
When you run this code, you’ll see the shapes of each tensor, helping you understand their dimensions.
2 – Use tf.Variable()
Variables are special tensors that can be modified. They’re essential for parameters that need to be updated during training:
# Creating a variable tensor
weights = tf.Variable([[1.0, 2.0], [3.0, 4.0]])
# Updating variable values
weights.assign([[5.0, 6.0], [7.0, 8.0]])
print(weights)Output:
<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[5., 6.],
[7., 8.]], dtype=float32)>You can see the output in the screenshot below.
I often use variables for model weights and biases that need to be updated during the training process.
Check out Tensorflow Convert String to Int
3 – Use Special Tensor Creators
TensorFlow provides several functions to create tensors with specific patterns:
# Create a tensor of zeros
zeros = tf.zeros([3, 4])
# Create a tensor of ones
ones = tf.ones([2, 3, 4])
# Create a tensor with random normal values
random_normal = tf.random.normal([3, 3], mean=0, stddev=1)
# Create a tensor with random uniform values
random_uniform = tf.random.uniform([2, 2], minval=0, maxval=1)
print("Zeros tensor:\n", zeros)
print("\nRandom normal tensor:\n", random_normal)Output:
Zeros tensor:
tf.Tensor(
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[[0. 0. 0. 0.]
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]], shape=(3, 4), dtype=float32)
Random normal tensor:
tf.Tensor(
[[ 0.21193734 1.763043 0.632623 ]
[ 0.38136902 -1.0868169 -2.2551832 ]
[ 1.638089 -1.6419734 0.27840304]], shape=(3, 3), dtype=float32)You can see the output in the screenshot below.
These functions are extremely useful when initializing neural network layers.
Understand Tensor Properties
Now, I will explain to you the Properties of a Tensor.
Read TensorFlow Fully Connected Layer
Tensor Rank and Shape
The rank and shape of a tensor are fundamental properties that determine how operations can be performed:
# Creating a sample tensor
sample_tensor = tf.constant([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]])
# Get tensor rank
rank = tf.rank(sample_tensor)
# Get tensor shape
shape = sample_tensor.shape
print(f"Tensor rank: {rank}")
print(f"Tensor shape: {shape}")When analyzing complex models, understanding tensor shapes has saved me countless debugging hours.
Tensor Data Types
TensorFlow supports various data types, and choosing the right one can significantly impact performance:
# Different data types
int_tensor = tf.constant(10, dtype=tf.int32)
float_tensor = tf.constant(10.5, dtype=tf.float32)
string_tensor = tf.constant("TensorFlow", dtype=tf.string)
# Convert between types
converted_tensor = tf.cast(float_tensor, dtype=tf.int32)
print(f"Original type: {float_tensor.dtype}")
print(f"Converted type: {converted_tensor.dtype}")For production models, I typically use float32 for most operations, as it offers a good balance between precision and performance.
Tensor Operations
Let me show you some tensor properties.
Check out Batch Normalization TensorFlow
Basic Mathematical Operations
Tensors support all standard mathematical operations:
# Create two tensors
a = tf.constant([[1, 2], [3, 4]])
b = tf.constant([[5, 6], [7, 8]])
# Addition
add_result = a + b # or tf.add(a, b)
# Multiplication
multiply_result = a * b # or tf.multiply(a, b)
# Matrix multiplication
matmul_result = tf.matmul(a, b)
print("Addition result:\n", add_result)
print("\nElement-wise multiplication:\n", multiply_result)
print("\nMatrix multiplication:\n", matmul_result)Understanding the difference between element-wise multiplication (*) and matrix multiplication (tf.matmul) is crucial for the correct implementation of neural networks.
Check out Binary Cross-Entropy TensorFlow
Reshaping Tensors
Reshaping is a common operation when preparing data for different neural network layers:
# Create a tensor
original = tf.constant([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
# Reshape to a 4x3 matrix
reshaped = tf.reshape(original, [4, 3])
# Reshape to a 2x2x3 tensor
reshaped_3d = tf.reshape(original, [2, 2, 3])
print("Original tensor:", original)
print("\nReshaped to 4x3:\n", reshaped)
print("\nReshaped to 2x2x3:\n", reshaped_3d)I often need to reshape tensors when transitioning between convolutional layers and fully connected layers in CNNs.
Tensors and NumPy Integration
TensorFlow tensors and NumPy arrays work seamlessly together, which is incredibly useful:
import numpy as np
# Convert NumPy array to tensor
numpy_array = np.array([[1, 2], [3, 4]])
tensor_from_np = tf.convert_to_tensor(numpy_array)
# Convert tensor to NumPy array
tensor = tf.constant([[5, 6], [7, 8]])
numpy_from_tensor = tensor.numpy()
print("NumPy to tensor:\n", tensor_from_np)
print("\nTensor to NumPy:\n", numpy_from_tensor)This interoperability has been invaluable when I need to use NumPy for data preprocessing before feeding it into a TensorFlow model.
Use Tensors in Real-World Applications
Let’s look at a practical example, implementing a simple neural network to classify images of clothing items from the Fashion MNIST dataset:
# Load the Fashion MNIST dataset
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
# Normalize the images to [0,1]
train_images = train_images / 255.0
test_images = test_images / 255.0
# Build the model
model = tf.keras.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(10)
])
# Compile the model
model.compile(
optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy']
)
# Train the model
model.fit(train_images, train_labels, epochs=5)
# Evaluate accuracy
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print(f"\nTest accuracy: {test_acc}")In this example, tensors flow through each layer of the neural network, transforming from the initial 28×28 input images to the final 10-class output predictions.
Read Build an Artificial Neural Network in Tensorflow
Advanced Tensor Concepts
Let me explain to you the advanced tensor concepts with an example.
Broadcasting
Broadcasting allows operations between tensors of different shapes:
# Create tensors of different shapes
matrix = tf.constant([[1, 2], [3, 4], [5, 6]]) # Shape: (3, 2)
vector = tf.constant([10, 20]) # Shape: (2,)
# Broadcasting adds the vector to each row of the matrix
result = matrix + vector # Shape: (3, 2)
print("Matrix shape:", matrix.shape)
print("Vector shape:", vector.shape)
print("Result shape:", result.shape)
print("Result:\n", result)Broadcasting has simplified my code significantly, eliminating the need for explicit loops when performing operations on tensors of compatible shapes.
Gradient Computation
Automatic differentiation is central to training neural networks:
x = tf.Variable(3.0)
with tf.GradientTape() as tape:
y = x**2 # y = x squared
# Compute the gradient of y with respect to x
dy_dx = tape.gradient(y, x)
print(f"Value of x: {x.numpy()}")
print(f"Value of y = x²: {y.numpy()}")
print(f"Gradient dy/dx at x=3: {dy_dx.numpy()}") # Should be 2x = 6Understanding how TensorFlow computes gradients using tensors helped me debug training issues in complex models.
Read Training a Neural Network in TensorFlow
Tensor Performance Optimization
Now, I will explain the performance optimization of the tensor.
Use the Right Data Types
Using smaller data types can significantly reduce memory usage and speed up computation:
# Using float32 (standard precision)
tensor_f32 = tf.constant([1.0, 2.0, 3.0], dtype=tf.float32)
# Using float16 (half precision)
tensor_f16 = tf.constant([1.0, 2.0, 3.0], dtype=tf.float16)
print(f"float32 size: {tensor_f32.numpy().nbytes} bytes")
print(f"float16 size: {tensor_f16.numpy().nbytes} bytes")In production, I’ve seen up to 2x performance improvements by carefully selecting appropriate tensor data types.
GPU Acceleration
TensorFlow automatically uses available GPUs, but you can explicitly control tensor placement:
# Check if GPU is available
print("GPU Available:", tf.config.list_physical_devices('GPU'))
# Create a tensor on the CPU
with tf.device('/CPU:0'):
cpu_tensor = tf.constant([[1.0, 2.0], [3.0, 4.0]])
print(f"CPU tensor device: {cpu_tensor.device}")
# Create a tensor on the GPU (if available)
if tf.config.list_physical_devices('GPU'):
with tf.device('/GPU:0'):
gpu_tensor = tf.constant([[1.0, 2.0], [3.0, 4.0]])
print(f"GPU tensor device: {gpu_tensor.device}")For large models, properly managing tensor placement across devices can make the difference between waiting hours versus minutes for training.
These concepts form the foundation of all deep learning models in TensorFlow, from simple neural networks to complex architectures like transformers. By mastering tensors, you’ll be better equipped to build, debug, and optimize your machine learning models.
If you have any questions or suggestions, kindly leave them in the comments below.
Other Python articles you may also like:
- Tensorflow Gradient Descent in Neural Network
- Tensorflow Activation Functions
- Use TensorFlow’s get_shape Function
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
