How to Find Duplicates in a Python List

The easiest way to find duplicates in a Python list is collections.Counter: count every value, then keep the ones that appear more than once. You get each duplicate and how many times it shows up:

dupes = [x for x, n in Counter(items).items() if n > 1]

The examples use US state codes, customer emails and order rows, and they ran on Python 3.12.5.

You’ll see Counter, a set-based loop, a quick yes/no check, duplicates in rows and dictionaries, and why list.count() gets slow.

Find duplicates in a Python list with Counter

Counter builds a value-to-count mapping in one pass. If you’d rather have each duplicate listed once in the order it first repeats, track what you’ve seen in a set:

from collections import Counter

states = ["TX", "CA", "NY", "TX", "FL", "CA", "TX", "WA"]

counts = Counter(states)
dupes = {state: n for state, n in counts.items() if n > 1}
print(dupes)                                   # value -> how many times

seen, repeats = set(), []
for s in states:                               # first-seen order, each duplicate once
    if s in seen and s not in repeats:
        repeats.append(s)
    seen.add(s)
print(repeats)

Output:

{'TX': 3, 'CA': 2}
['TX', 'CA']

To delete the repeats rather than report them, see removing duplicates from a list in Python.

How do you check if a list has duplicates?

When you only need yes or no, compare the length of the list with the length of a set built from it. A loop with early exit is faster when a repeat is likely near the start:

emails = ["emily@example.com", "james@example.com", "olivia@example.com", "emily@example.com"]

print("has duplicates:", len(emails) != len(set(emails)))

def has_duplicates(items):
    seen = set()
    for item in items:                       # stops at the first repeat
        if item in seen:
            return True
        seen.add(item)
    return False

print("has_duplicates():", has_duplicates(emails), has_duplicates(["a", "b"]))

Output:

has duplicates: True
has_duplicates(): True False

Sets only keep unique values, which is why the length drops. Going back and forth is covered in converting a set to a list in Python.

Find duplicates in a list of lists or dictionaries

Counter and sets need hashable values, so lists inside a list fail. Convert rows to tuples, or count a key such as a lowercased email:

from collections import Counter, defaultdict

rows = [["Austin", 59.99], ["Denver", 20.00], ["Austin", 59.99]]
try:
    Counter(rows)
except TypeError as err:
    print("TypeError:", err)
print(Counter(tuple(r) for r in rows))          # tuples are hashable

customers = [
    {"id": 1, "email": "emily@example.com"},
    {"id": 2, "email": "james@example.com"},
    {"id": 3, "email": "EMILY@example.com"},
]
positions = defaultdict(list)
for i, c in enumerate(customers):
    positions[c["email"].lower()].append(i)   # compare emails case-insensitively
print({email: idx for email, idx in positions.items() if len(idx) > 1})

Output:

TypeError: unhashable type: 'list'
Counter({('Austin', 59.99): 2, ('Denver', 20.0): 1})
{'emily@example.com': [0, 2]}

defaultdict(list) collects every position for each key, which is handy when you need to fix the records later. For more counting patterns, see counting with a Python dictionary.

Why is list.count() slow for finding duplicates?

items.count(x) scans the whole list each time, so calling it for every item does a full scan per element. Counter walks the list once:

import random
import timeit
from collections import Counter

random.seed(1)
ids = [random.randint(1, 5_000) for _ in range(10_000)]

counter_way = lambda: [x for x, n in Counter(ids).items() if n > 1]
count_way = lambda: list({x for x in ids if ids.count(x) > 1})

print("same result:", sorted(counter_way()) == sorted(count_way()))
for name, fn in [("Counter", counter_way), ("list.count()", count_way)]:
    t = min(timeit.repeat(fn, number=1, repeat=3))
    print(f"{name:<13} {t * 1000:>9.1f} ms for {len(ids):,} items")

Output:

same result: True
Counter             0.5 ms for 10,000 items
list.count()      606.6 ms for 10,000 items

count() is still perfect for counting one specific value; see counting occurrences in a Python list.

Methods for finding duplicates compared

I reach for Counter first because it answers both questions at once: which values repeat and how often. The same idea works on arrays; see removing duplicates from an array in Python.

Need	Use
Duplicates with counts	Counter(items)
Each duplicate once, in order	Seen-set loop
Just yes or no	len(items) != len(set(items))
Rows or dicts	Tuples, or a key with defaultdict(list)

These guides pair well with this one:

• Merge two lists without duplicates
• Remove duplicate values from a dictionary
• Count occurrences in Python arrays
• Create an empty list in Python

Frequently asked questions