Recently, I was working on a machine learning project that involved a large dataset with mostly zero values. When I tried to process this data using regular NumPy arrays, my computer almost crashed due to memory limitations. That’s when I discovered the power of sparse matrices in SciPy, particularly the LIL (List of Lists) matrix format.
In this article, I’ll share everything I’ve learned about working with lil_matrix in SciPy, from creating them to converting between formats and performing operations.
So let’s dive in!
What is a Lil_Matrix and When to Use It?
A LIL (List of Lists) matrix is one of several sparse matrix formats available in SciPy’s sparse module. Sparse matrices are specialized data structures designed to efficiently store and manipulate matrices that have mostly zero values.
The lil_matrix format stores non-zero elements in row-based linked lists, making it particularly efficient for incremental matrix construction and modifications. This is in contrast to other formats like CSR (Compressed Sparse Row), which excel at matrix operations and arithmetic.
Here’s when you should consider using lil_matrix:
- When building a matrix incrementally (adding elements one by one)
- When you need to frequently modify matrix values
- When working with very large datasets that are mostly zeros
- When memory efficiency is crucial for your application
Create a Lil_Matrix in SciPy
Let’s look at various ways to create a lil_matrix in SciPy:
Method 1: Create an Empty Lil_Matrix
The simplest way to create a lil_matrix is by specifying its dimensions:
import numpy as np
from scipy.sparse import lil_matrix
# Create an empty 5x5 lil_matrix
A = lil_matrix((5, 5))
print(A.toarray())Output:
[[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]]I executed the above example code and added the screenshot below.
This creates a 5×5 matrix filled with zeros. The toarray() method converts it to a dense NumPy array for display purposes, though you typically wouldn’t do this with large sparse matrices in practice.
Method 2: Set Values Individually
Once you’ve created an empty matrix, you can set individual elements just like you would with a NumPy array:
# Create an empty matrix
A = lil_matrix((5, 5))
# Set individual values
A[0, 0] = 1 # Set element at row 0, column 0 to 1
A[1, 1] = 2 # Set element at row 1, column 1 to 2
A[2, 3] = 3 # Set element at row 2, column 3 to 3
print(A.toarray())Output:
[[1. 0. 0. 0. 0.]
[0. 2. 0. 0. 0.]
[0. 0. 0. 3. 0.]
[0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0.]]I executed the above example code and added the screenshot below.
This is where lil_matrix shines compared to other sparse formats, it’s optimized for this type of incremental construction.
Method 3: Create from a Dense Array
You can directly create a lil_matrix from an existing NumPy array:
# Create a dense array
dense_array = np.array([[1, 0, 0],
[0, 2, 0],
[0, 0, 3]])
# Convert to lil_matrix
B = lil_matrix(dense_array)
print(B.toarray())Output:
[[1 0 0]
[0 2 0]
[0 0 3]]I executed the above example code and added the screenshot below.
This is useful when you already have your data in a NumPy array and want to convert it to a sparse format to save memory.
Check out Python Scipy Leastsq
Method 4: Use Random Data
When testing or creating example matrices, it’s often useful to generate random sparse matrices:
from scipy import sparse
# Create a random sparse matrix with approximately 1% non-zero elements
C = sparse.random(1000, 1000, density=0.01, format='lil')
# Print the number of non-zero elements
print(f"Matrix shape: {C.shape}")
print(f"Number of non-zero elements: {C.nnz}")
print(f"Percentage of non-zero elements: {C.nnz / (C.shape[0] * C.shape[1]) * 100:.2f}%")The sparse.random() function is extremely useful for creating test matrices. You can control:
- The dimensions of the matrix (here: 1000×1000)
- The density (proportion of non-zero elements, here: 0.01 or 1%)
- The format (here: ‘lil’)
Read Python Scipy Odeint
Method 5: Create from Other Sparse Formats
You can also create a lil_matrix by converting from other sparse matrix formats:
from scipy.sparse import csr_matrix
# Create a CSR matrix
csr = csr_matrix(([1, 2, 3], ([0, 1, 2], [0, 1, 2])), shape=(3, 3))
# Convert to lil_matrix
D = csr.tolil()
print(D.toarray())This is particularly useful when you receive data in another sparse format but need the flexibility of lil_matrix for modifications.
Check out Python SciPy Interpolate
Convert Between Matrix Formats
While lil_matrix is great for building matrices; other formats are better for specific operations. SciPy makes it easy to convert between formats:
from scipy.sparse import csr_matrix, csc_matrix, dok_matrix, coo_matrix
# Create a lil_matrix
A = lil_matrix((5, 5))
A[0, 0] = 1
A[1, 1] = 2
A[4, 3] = 3
# Convert to other formats
csr_A = A.tocsr() # Compressed Sparse Row - good for row slicing and matrix-vector products
csc_A = A.tocsc() # Compressed Sparse Column - good for column slicing
dok_A = A.todok() # Dictionary of Keys - another format good for incrementally building matrices
coo_A = A.tocoo() # COOrdinate format - good for quickly constructing matrices
# Convert back to lil_matrix if needed
A_again = csr_A.tolil()It’s important to note that you should convert lil_matrix to other formats (usually CSR or CSC) before performing operations like matrix multiplication or solving linear systems, as these operations are much faster in those formats.
Read Python SciPy Optimize Root
Efficient Operations with Lil_Matrix
Let’s explore some common operations with lil_matrix:
import numpy as np
from scipy.sparse import lil_matrix
# Create matrices
A = lil_matrix((5, 5))
A[0, 0] = 1
A[1, 1] = 2
A[2, 2] = 3
A[3, 3] = 4
A[4, 4] = 5
B = lil_matrix((5, 5))
B[0, 4] = 5
B[1, 3] = 4
B[2, 2] = 3
B[3, 1] = 2
B[4, 0] = 1
# Basic properties
print(f"Shape: {A.shape}")
print(f"Data type: {A.dtype}")
print(f"Number of non-zeros: {A.nnz}")
# Convert to CSR for efficient operations
A_csr = A.tocsr()
B_csr = B.tocsr()
# Matrix addition
C_csr = A_csr + B_csr
C = C_csr.tolil() # Convert back to LIL if needed
# Matrix multiplication
D_csr = A_csr @ B_csr # Or use A_csr.dot(B_csr)
D = D_csr.tolil()
# Element-wise multiplication
E_csr = A_csr.multiply(B_csr)
E = E_csr.tolil()
# Slicing
row_slice = A[1:3, :]
col_slice = A[:, 2:4]
# Find non-zero elements
non_zeros = A.nonzero()A key thing to remember is that lil_matrix is efficient for building and modifying matrices, but not for arithmetic operations. Always convert to CSR or CSC format before performing calculations.
Check out Python SciPy Fcluster
Real-World Application: Text Analysis with TF-IDF
Here’s a practical example of using lil_matrix for text analysis with a dataset of customer reviews from a US-based e-commerce site:
from scipy.sparse import lil_matrix
import numpy as np
from sklearn.feature_extraction.text import TfidfTransformer
# Sample customer reviews
reviews = [
"This product exceeded my expectations. Great quality for the price.",
"Shipping was fast, but the product quality was disappointing.",
"Amazing customer service when I had questions about the product.",
"The price was good, but shipping took forever to arrive.",
"Product broke after just two weeks of normal use."
]
# Create a vocabulary
unique_words = set()
for review in reviews:
unique_words.update(review.lower().split())
vocab = {word: idx for idx, word in enumerate(unique_words)}
# Create a lil_matrix for word counts (document-term matrix)
dtm = lil_matrix((len(reviews), len(vocab)))
# Fill the matrix with word counts
for i, review in enumerate(reviews):
for word in review.lower().split():
j = vocab[word]
dtm[i, j] += 1
# Convert to CSR for efficient TF-IDF calculation
dtm_csr = dtm.tocsr()
# Calculate TF-IDF
tfidf_transformer = TfidfTransformer()
tfidf_matrix = tfidf_transformer.fit_transform(dtm_csr)
# Find most important words in first review
first_review_weights = tfidf_matrix[0].toarray()[0]
important_indices = np.argsort(first_review_weights)[::-1][:3] # Top 3 words
important_words = [word for word, idx in vocab.items() if idx in important_indices]
print(f"Most important words in first review: {important_words}")This example shows how lil_matrix can be used in a text analysis pipeline, where we first build a sparse document-term matrix and then convert it to CSR for efficient TF-IDF calculation.
Read Python SciPy Spatial Distance Cdist
Common Use Cases and Performance Tips
Here are some practical scenarios where lil_matrix shines:
- Large-scale machine learning – When processing high-dimensional data with sparse features
- Recommender systems – User-item interaction matrices are typically very sparse
- Network/graph analysis – Adjacency matrices for large graphs have mostly zeros
- Image processing – Certain transforms produce sparse matrices
Performance tips when working with lil_matrix:
- Always convert to CSR or CSC before performing mathematical operations
- Pre-allocate the matrix size if you know the dimensions in advance
- Consider using
dok_matrixinstead if you need dictionary-like access - Use
lil_matrixfor building matrices, but switch to more efficient formats for storage - For very large matrices, use NumPy’s advanced indexing to set multiple values at once
# Efficient way to set multiple values at once
rows = [0, 1, 2, 3]
cols = [1, 2, 3, 4]
values = [10, 20, 30, 40]
A = lil_matrix((5, 5))
A[rows, cols] = valuesI hope you found this guide helpful for working with lil_matrix in SciPy. Sparse matrices are incredibly powerful for dealing with large-scale data problems, and understanding when and how to use them effectively can dramatically improve your code’s performance and memory efficiency. If you have any questions or suggestions, feel free to share them in the comments below.
Other Python articles you may also like:
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
