How to Remove Duplicates from a List in Python

A lot of answers on how to remove duplicates from a list in Python still loop with if x not in result before every append. It works, but it rescans the result each time.

The better answer is list(dict.fromkeys(items)). It’s one line, it keeps the original order, and it makes a single pass over the list.

In this article, I’ll show you five ways to do it, including arrays, when each one fits, and how to handle lists that can’t be turned into a set. I ran every snippet on Python 3.12.5.

The sample list we’ll clean up

Every example uses the same list of cities from this week’s orders:

orders = ["Austin", "Denver", "Austin", "Tampa", "Boise", "Denver", "Austin"]

• Austin shows up three times and Denver twice
• Tampa and Boise appear once, so they’re already unique
• Seven orders in total, but only four different cities

The goal is a list of those four cities. If you only want to know which values repeat, see finding duplicates in a Python list instead.

Method 1: Remove duplicates from a list with dict.fromkeys (Recommended)

This is the method I reach for first. Dictionary keys must be unique, and since Python 3.7 a dictionary remembers the order its keys were added in.

Here’s the code, using the seven orders from above:

orders = ["Austin", "Denver", "Austin", "Tampa", "Boise", "Denver", "Austin"]

unique = list(dict.fromkeys(orders))
print(unique)
print(f"{len(orders)} orders -> {len(unique)} cities")

Output:

['Austin', 'Denver', 'Tampa', 'Boise']
7 orders -> 4 cities

That gives four cities, and Austin stays first because it appeared first.

How does this code work?

dict.fromkeys(orders) builds a dictionary with each city as a key. Adding Austin a second time doesn’t create a new key, so repeats simply vanish. If dictionaries are new to you, see dictionary comprehensions in Python.

Then list() turns the keys back into a list. The values in that dictionary are all None, but we never use them.

Pro Tip: You can write list({city: None for city in orders}) to get the same result, but fromkeys says what you mean in fewer characters.

Method 2: Remove duplicates with set() when order doesn’t matter

A set stores each value once, so list(set(orders)) is the shortest way to remove duplicates. The trade-off is that a set doesn’t promise any particular order.

Same seven orders here. Let me show you what comes back:

orders = ["Austin", "Denver", "Austin", "Tampa", "Boise", "Denver", "Austin"]

unique = list(set(orders))
print(sorted(unique))            # sorted here only so the output is stable
print(len(unique), "cities")
print(unique == list(dict.fromkeys(orders)))   # is the order the same as the original order?

Output:

['Austin', 'Boise', 'Denver', 'Tampa']
4 cities
False

How does this code work?

set(orders) keeps one copy of each city, and list() turns it back into a list. I sorted the result before printing because a set’s order can change between runs.

The differences between these types are laid out in comparing lists, tuples, sets and dictionaries.

The last line compares the set’s order with the first-seen order. If that prints False, the set shuffled your cities.

So here’s the catch: use a set when you only care about which values exist, such as counting distinct cities. If you need the original order, go back to Method 1.

Method 3: Remove duplicates with a loop and a seen set

Sometimes you need your own rule for what counts as a duplicate. Say “Austin” and “austin” should be treated as the same city. A short loop gives you that control.

I added two more orders, “austin” and “TAMPA”, to show it:

orders = ["Austin", "Denver", "Austin", "Tampa", "Boise", "Denver", "Austin"]

orders = orders + ["austin", "TAMPA"]       # same cities, different capitalization

seen = set()
unique = []
for city in orders:
    key = city.lower()                      # compare in lowercase, keep the original text
    if key not in seen:
        seen.add(key)
        unique.append(city)
print(unique)

Output:

['Austin', 'Denver', 'Tampa', 'Boise']

How does this code work?

The loop walks through the orders once. For each city, it builds a lowercase key and checks whether that key is already in the seen set.

If it isn’t, the key goes into seen and the original city goes into unique. Checking a set is fast, so this is still a single pass. For more on loops like this one, see for i in range in Python.

The result keeps the first spelling of each city: Austin, Denver, Tampa and Boise. The extra “austin” and “TAMPA” are dropped.

Method 4: Remove duplicates from a list of lists or dictionaries

Sets and dictionary keys need hashable values. Lists and dictionaries aren’t hashable, so the earlier methods raise an error on them. Here are some rows with a repeat, plus customer records:

rows = [["Austin", 59.99], ["Denver", 20.00], ["Austin", 59.99]]

try:
    list(dict.fromkeys(rows))
except TypeError as err:
    print("TypeError:", err)

unique = [list(t) for t in dict.fromkeys(tuple(r) for r in rows)]   # tuples are hashable
print(unique)

customers = [{"id": 1, "email": "emily@example.com"},
             {"id": 2, "email": "james@example.com"},
             {"id": 3, "email": "emily@example.com"}]
by_email = {}
for c in customers:
    by_email.setdefault(c["email"], c)       # keeps the first customer for each email
print(list(by_email.values()))

Output:

TypeError: unhashable type: 'list'
[['Austin', 59.99], ['Denver', 20.0]]
[{'id': 1, 'email': 'emily@example.com'}, {'id': 2, 'email': 'james@example.com'}]

How does this code work?

The first attempt fails with TypeError: unhashable type: 'list', because a list can change and so it can’t be a dictionary key.

Tuples can’t change, so tuple(r) fixes it. After fromkeys removes the repeat, I turn each tuple back into a list.

For records, pick the field that defines a duplicate. I used the email, and setdefault keeps only the first customer for each one.

Sorting these lists is covered in sorting a list of lists in Python.

Method 5: Remove duplicates from an array in Python

Python has no built-in array type like C or Java. When people say “array”, they usually mean a list, the array module, or a NumPy array. Everything above already works on a plain list.

The other two need a small change. Here’s how I’d remove duplicates from an array in Python in each case:

from array import array
import numpy as np

counts = array("i", [5, 3, 5, 8, 3, 5])        # the built-in array module
print(array("i", dict.fromkeys(counts)))        # keeps first-seen order

nums = np.array([5, 3, 5, 8, 3, 5])             # a NumPy array
print(np.unique(nums))                          # sorted unique values
values, first = np.unique(nums, return_index=True)
print(nums[np.sort(first)])                     # unique values in original order

Output:

array('i', [5, 3, 8])
[3 5 8]
[5 3 8]

How does this code work?

array("i", ...) accepts any iterable of integers, so passing dict.fromkeys(counts) rebuilds the array from the unique values in their original order.

np.unique() returns the distinct values already sorted. To keep the original order instead, ask for return_index=True, sort those first positions, and use them to pick values from the array.

For more on NumPy arrays, see NumPy 3D arrays in Python.

If you need the sorted-array interview version that edits in place, see removing duplicates from a sorted array.

Pro Tip: For one-dimensional numeric data, np.unique() is the natural choice when you already use NumPy. For a plain list, dict.fromkeys needs no install.

Why is the old not-in-result loop slow?

Now the fun part. I timed the old loop and three better ways on 20,000 values with 2,000 distinct ones:

import timeit

items = [i % 2_000 for i in range(20_000)]       # 20,000 values, only 2,000 distinct

def old_way():
    result = []
    for x in items:
        if x not in result:                      # rescans the whole result every time
            result.append(x)
    return result

def seen_loop():
    seen, result = set(), []
    for x in items:
        if x not in seen:
            seen.add(x)
            result.append(x)
    return result

for label, fn in [("not in result", old_way),
                  ("seen set", seen_loop),
                  ("dict.fromkeys", lambda: list(dict.fromkeys(items))),
                  ("set", lambda: list(set(items)))]:
    t = min(timeit.repeat(fn, number=3, repeat=3)) / 3
    print(f"{label:<14} {t * 1000:9.2f} ms")

Output:

not in result      83.97 ms
seen set            0.51 ms
dict.fromkeys       0.31 ms
set                 0.26 ms

How does this code work?

x not in result checks a list, and a list is searched from the start every time. As the result grows, each check takes longer, so the total time grows with the square of the list size.

The other three use hashing, which finds a value in about the same time however big the collection is. That’s why they finish in a small fraction of the time.

On seven cities you won’t notice a difference. On a few hundred thousand rows, you will. To check how much a list shrank, compare its length, as in finding the size of a list.

Picking the right way to remove duplicates

Method	Keeps order	Use it when
dict.fromkeys	Yes	Almost always; short, fast and keeps order
set()	No	You only need the distinct values
Loop with a seen set	Yes	You need a custom rule, like ignoring case
Tuples or a key field	Yes	Rows or dictionaries that can’t be hashed
if x not in result loop	Yes	Only on tiny lists (slow)

If you just want the distinct values without cleaning the original, see getting unique values from a list in Python.

Things to Keep in Mind When Removing Duplicates

• A set can shuffle your list. Use dict.fromkeys when order matters.
• Lists and dictionaries are unhashable and raise a TypeError. Convert rows to tuples, or key your records on one field.
• “Austin” and “austin” are different strings. Compare lowercase keys if capitalization shouldn’t matter.
• Don’t remove duplicates that carry meaning, like quantities in a cart. Count them instead.
• None of these change the original list; they build a new one, so assign the result to a variable.

Working with lists? These guides pair well with this one:

• Merge two lists without duplicates in Python
• Flatten a list of lists in Python

Frequently asked questions