How To Remove Duplicates From A List In Python?

As a data scientist working with large datasets of US demographic information, I often encounter lists containing duplicate values that need to be cleaned up before analysis. We often come across this situation when working with real-world data. In this tutorial, I will explain how to remove duplicates from a list in Python with examples.

Table of Contents

Remove Duplicates from a List in Python

To remove duplicates from a list in Python, you can convert the list to a set, which will automatically eliminate any duplicate elements. For example, if you have a list names = ['John', 'Emma', 'John', 'Olivia', 'Emma'], you can remove the duplicates by converting it to a set: unique_names = list(set(names)), resulting in ['John', 'Emma', 'Olivia'].

Let’s get into several effective methods to eliminate duplicates from Python lists, with examples using common American names.

Read How to Find Duplicates in a Python List?

Method 1: Convert a List to a Set

The simplest way to remove duplicates from a Python list is by converting the list into a set. A set is an unordered collection of unique elements, so it will automatically remove any duplicate values. Here’s how to do it:

names = ['John', 'Emma', 'Michael', 'Emma', 'William', 'John', 'Olivia', 'Olivia', 'Sophia']
unique_names = list(set(names))
print(unique_names)

Output:

['Michael', 'William', 'Emma', 'John', 'Olivia', 'Sophia']

You can refer to the below screenshot to see the output.

As you can see, the duplicates ‘John’, ‘Emma’, and ‘Olivia’ have been removed, and we are left with a list of unique names. However, note that the original order of the elements is not preserved when using this method.

Preserve Order While Removing Duplicates

If maintaining the original order of the Python list is important, you can use a dictionary to remove duplicates while preserving order, as explained by DataCamp:

names = ['John', 'Emma', 'Michael', 'Emma', 'William', 'John', 'Olivia', 'Olivia', 'Sophia']
unique_names = list(dict.fromkeys(names))
print(unique_names)

Output:

['John', 'Emma', 'Michael', 'William', 'Olivia', 'Sophia']

Here, we create a dictionary using dict.fromkeys(), which automatically removes duplicates because dictionaries cannot have duplicate keys. We then convert the dictionary keys back into a list, preserving the original order.

Check out How to Divide Each Element in a List by a Number in Python?

Method 2: Use a For Loop

Another way to remove duplicates is by using a for loop to iterate through the list and add unique elements to a new list in Python. Here’s how it works:

names = ['Liam', 'Noah', 'Oliver', 'Liam', 'Ava', 'Charlotte', 'Oliver', 'Mia']
unique_names = []

for name in names:
    if name not in unique_names:
        unique_names.append(name)

print(unique_names)

Output:

['Liam', 'Noah', 'Oliver', 'Ava', 'Charlotte', 'Mia']

You can refer to the below screenshot to see the output.

How to Remove Duplicates from a List in Python

This method checks each element in the original list and appends it to the unique_names list only if it is not already present, effectively removing duplicates while maintaining the original order.

Read How to Find the Largest Number in a List Using Python?

Method 3: Use a List Comprehension

Python list comprehensions provide a concise way to create new lists based on existing lists. We can leverage this to remove duplicates as follows:

names = ['Elijah', 'Amelia', 'Benjamin', 'Isabella', 'Lucas', 'Amelia', 'Henry', 'Mia', 'Alexander']
unique_names = []
[unique_names.append(name) for name in names if name not in unique_names]
print(unique_names)

Output:

['Elijah', 'Amelia', 'Benjamin', 'Isabella', 'Lucas', 'Henry', 'Mia', 'Alexander']

You can refer to the below screenshot to see the output.

Remove Duplicates from a List in Python list comprehension

This one-liner accomplishes the same result as the for loop method, appending unique elements to the unique_names list using a list comprehension.

Check out How to Remove None Values from a List in Python?

Method 4: Use the Collections Module

Python’s built-in collections module provides a convenient way to remove duplicates while keeping track of the order in which elements first appear. Here’s an example using the OrderedDict class:

from collections import OrderedDict

names = ['Harper', 'Mason', 'Evelyn', 'Ella', 'Jayden', 'Harper', 'Evelyn', 'Abigail']
unique_names = list(OrderedDict.fromkeys(names))
print(unique_names)

Output:

['Harper', 'Mason', 'Evelyn', 'Ella', 'Jayden', 'Abigail']

The OrderedDict.fromkeys() method creates an ordered dictionary with the list elements as keys, automatically removing duplicates. We then convert the ordered dictionary keys back into a list to get the unique names in the order they first appeared.

Read How to Add Elements to an Empty List in Python?

Conclusion

In this tutorial, I explained how to remove duplicates from a list in Python. I discussed four methods, such as converting a list to a set, using a for loop, using a list comprehension and using the Collection module.