Remove Duplicates from a Dictionary in Python

Recently, while working on a project in Python, I faced an issue where my Python dictionary contained duplicate values.

In many real-world data cleaning tasks, duplicate values in a dictionary can cause reporting errors or skewed results.

So, in this tutorial, I’ll show you four simple methods to remove duplicates from a dictionary in Python. These are the same methods I use in my projects, and they are easy to follow even if you’re just starting with Python.

Table of Contents

Remove Duplicates from a Python Dictionary

When working with data in Python, dictionaries are one of the most powerful structures. But duplicates can lead to problems:

Repeated values may give wrong counts in analytics.
Exporting data to JSON or CSV becomes messy.
APIs might reject duplicate values.

For example, imagine you’re building a Python script to store customer ZIP codes in the USA. If your dictionary has multiple customers mapped to the same ZIP code, you may want to remove duplicates to simplify lookups.

Method 1 – Use a For Loop in Python

The simplest way I learned to remove duplicates from a dictionary in Python is by using a for loop. This method works by iterating through the dictionary and storing unique values in a new dictionary.

# Python program to remove duplicates from a dictionary using a loop

customer_data = {
    "John": "10001",
    "Alice": "90001",
    "Bob": "10001",
    "Emma": "30301",
    "Sophia": "90001"
}

unique_dict = {}
seen_values = set()

for key, value in customer_data.items():
    if value not in seen_values:
        unique_dict[key] = value
        seen_values.add(value)

print("Original Dictionary:", customer_data)
print("Dictionary without duplicates:", unique_dict)

You can refer to the screenshot below to see the output.

remove duplicates from dictionary python

When I run this code, it keeps only the first occurrence of each value and removes the duplicates. This is a clean and beginner-friendly approach.

Method 2 – Use Python Dictionary Comprehension

Another efficient way is to use dictionary comprehension. This method reduces the code and makes it more readable.

# Python program to remove duplicates using dictionary comprehension

customer_data = {
    "John": "10001",
    "Alice": "90001",
    "Bob": "10001",
    "Emma": "30301",
    "Sophia": "90001"
}

unique_dict = {}
[unique_dict.setdefault(v, k) for k, v in customer_data.items()]
unique_dict = {v: k for k, v in unique_dict.items()}

print("Original Dictionary:", customer_data)
print("Dictionary without duplicates:", unique_dict)

You can refer to the screenshot below to see the output.

python remove duplicates from dictionary

I personally use this method when I need a quick one-liner solution in Python. It’s compact but still effective.

Method 3 – Use Python’s dict.fromkeys()

Python provides a handy function dict.fromkeys() that can also help in removing duplicates.

Here, we reverse the dictionary to remove duplicate values.

# Python program to remove duplicates using dict.fromkeys()

customer_data = {
    "John": "10001",
    "Alice": "90001",
    "Bob": "10001",
    "Emma": "30301",
    "Sophia": "90001"
}

unique_dict = dict.fromkeys(customer_data.values())
unique_dict = {v: k for k, v in unique_dict.items()}

print("Original Dictionary:", customer_data)
print("Dictionary without duplicates:", unique_dict)

You can refer to the screenshot below to see the output.

python remove duplicates from list of dictionaries

This trick is useful when you want a shorter solution without writing loops. But keep in mind that it may overwrite keys if multiple keys share the same value.

Method 4 – Use Python Pandas

When working with large datasets in Python, I often use pandas to clean dictionaries. This method is especially powerful if your dictionary is converted into a DataFrame.

# Python program to remove duplicates from a dictionary using pandas

import pandas as pd

customer_data = {
    "John": "10001",
    "Alice": "90001",
    "Bob": "10001",
    "Emma": "30301",
    "Sophia": "90001"
}

# Convert dictionary to DataFrame
df = pd.DataFrame(list(customer_data.items()), columns=["Name", "Zip"])

# Drop duplicate Zip values
df = df.drop_duplicates(subset="Zip", keep="first")

# Convert back to dictionary
unique_dict = dict(zip(df["Name"], df["Zip"]))

print("Original Dictionary:", customer_data)
print("Dictionary without duplicates:", unique_dict)

I recommend this method when dealing with real-world datasets like customer records, sales data, or survey results in Python. It’s more flexible and works great if you’re already using pandas.

Which Method Should You Use?

For beginners → Use the for loop method.
For clean, short code → Use dictionary comprehension.
For quick tricks → Use dict.fromkeys().
For large datasets → Use pandas.

In my own projects, I often switch between these depending on the situation.

Conclusion

While Python dictionaries don’t allow duplicate keys, duplicate values are still possible. Removing duplicates helps keep your data clean, accurate, and ready for analysis.

I’ve shown you four methods that I personally use: loop, comprehension, dict.fromkeys(), and pandas. Try them out and see which one fits your project best.

If you’re working with US-based datasets like ZIP codes, customer IDs, or product SKUs, these methods will save you a lot of time.