While cleaning a dataset for a project, I ran into an issue where my Python list had a mix of numbers and NaN values. If you’ve worked with real-world data in the US, say, sales reports, healthcare data, or survey results, you know how common missing values can be.
The problem is simple: NaN (Not a Number) values can break calculations and give unexpected results. Unfortunately, Python lists don’t have a built-in method to directly filter them out.
I’ve tried different approaches, and in this tutorial, I’ll show you the most effective methods I use to remove NaN values from a list in Python.
Methods to Remove NaN Values from a List in Python
Let me show you the methods to remove NaN values from a list in Python.
1: Use the For Loop in Python
The for loop in Python is a common and effective way to remove NaN values from a list.
original_data = [1, 2, 3, 4, float('nan'), 6]
cleaned_data = []
for x in original_data:
if x == x:
cleaned_data.append(x)
print(cleaned_data)In this code, we iterate through each element in the original_data list in Python. The condition if x == x checks if the element is not NaN (since NaN does not equal itself)
If the condition is true, we append the element to the cleaned_data list in Python.
# When it iterates to NaN == NaN, then it will return False
if x == x:
cleaned_data.append(x)Output
[1, 2, 3, 4, 6]You can refer to the screenshot below to see the output.
This method removes NaN values from a list by using a for loop and the condition x == x, which filters out NaN since it is not equal to itself.
2: Use List Comprehension in Python
List comprehension is a concise and elegant way to create lists in Python. It offers a more compact syntax compared to loops. In the context of removing NaN values from a list.
employee_age = [42, 35, float('nan'), 26, float('nan'), 28]
filtered_list = [x for x in employee_age if x == x]
print(filtered_list)We use the same logic we used in the previous example, using list comprehension in Python with fewer lines of code and a faster approach.
Output
[42, 35, 26, 28]You can refer to the screenshot below to see the output.
This method removes NaN values efficiently using list comprehension, providing a shorter and faster alternative to a traditional loop.
3: Use the Filter Method in Python
The filter() method is a built-in function of Python that applies a specified function to each item of the collection and returns an iterator containing only the items for which the function evaluates to True.
Code
week_sales = [5000, 8000, float('nan'), 2000, float('nan'), 3000, 9000]
filtered_list = list(filter(lambda x: x == x, week_sales ))
print(filtered_list)The filter() method in Python removes NaN values from the week_sales list by applying a lambda function that checks each element for equality with itself.
Output
[5000, 8000, 2000, 3000, 9000]You can refer to the screenshot below to see the output.
Since NaN does not equal itself, the lambda function in Python filters out NaN values. Then, filtered_list is converted back to a list for further processing.
4. Use the isnan() math Module in Python
To remove NaN values from the Python list, we can use the isnan() function, which is an inbuilt method of the math module in Python.
It will check all the elements and return True, where it will find NaN.
from math import isnan
original_list = [1, 2, float('nan'), 4, float('nan'), 6]
cleaned_list = []
for i in original_list:
if not isnan(i):
cleaned_list+=[i]
print(cleaned_list)I’ve initialized the original_list list in Python with numerical values, including NaN values that float(‘nan’) represents.
Then, iterate through each element i in original_list. The isnan() function in Python is used to check if i is not a NaN value.
Output
[1, 2, 4, 6]You can refer to the screenhsot below to see the output.
This method uses Python’s math.isnan() for a clear and reliable way to detect and remove NaN values from a list.
5: Remove NaN from the Numpy array
To remove NaN values from a list from Python NumPy array, we will use the isnan() method of the numpy library to check whether the element is NaN or not, and it will return True or False based on whether the element is NaN.
If an element is NaN in the array, it will return True.
Code
import numpy as np
from numpy import nan
data = np.array([5, 12, nan, 7,nan,9])
filtered_data = data[np.logical_not(np.isnan(data))]
print(filtered_data)We used NumPy’s isnan() function in Python to identify NaN values within the array data. The np.logical_not() function is used to negate this result.
Output
[ 5. 12. 7. 9.]You can refer to the screenshot below to see the output.
This method efficiently removes NaN values from a NumPy array using np.isnan() with np.logical_not() for clean, numeric-only data.
6. Remove NaN from the List in Pandas
To remove NaN from a list using Pandas, there is one inbuilt function in Python called dropna(), which will directly remove the NaN values from the series in Python, and then you can convert it to a list using the tolist() method.
Here is an instance to remove NaN values from a list in Python using the pandas library:
Code
import pandas as pd
original_list = [1, 2, float('nan'), 4, float('nan'), 6]
series = pd.Series(original_list)
cleaned_list = series.dropna().tolist()
print(cleaned_list)We’ve used the dropna() function in Python Pandas to remove any NaN values from the series of original lists, resulting in a cleaned_list. Finally, the tolist() function in Python converts the cleaned series into a list format.
Output
[1.0, 2.0, 4.0, 6.0]You can refer to the screenshot below to see the output.
This method removes NaN values from a list using Pandas’ dropna() and converts the cleaned series back into a list with tolist().
Here, I’ve covered all the different ways to remove the NaN value from the list in Python using for loop, list comprehension, filter method, isnan() method from math module, isnan() method from numpy, and dropna() method from Pandas. Each method offers its advantages and can be chosen based on specific requirements.
You may like to read other Python tutorials:
- Find the Mean of a List in Python
- Sum a List in Python Without the Sum Function
- Python Sort List Alphabetically
- Sum Elements in a List in Python
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
