While working on a data-cleaning project for a client in New York, I came across a massive dataset full of repeated entries. The challenge? Identify and print all the duplicate elements in a Python array efficiently.
If you’ve worked with real-world data in Python, whether it’s from a CSV file, an API, or a database, you’ve probably noticed that duplicates sneak in quite often. They can distort your analysis, inflate counts, and sometimes even break your logic if not handled properly.
In this tutorial, I’ll walk you through different ways to print duplicate elements in an array using Python.
Method 1 – Use Loops to Print Duplicate Elements in Python
When I first started with Python, I preferred using loops because they gave me full control.
It’s a simple and beginner-friendly way to understand how duplicates occur in an array.
Here’s how you can do it:
# Sample array of numbers
numbers = [5, 3, 8, 3, 9, 1, 5, 7, 9, 2, 3]
# Create an empty list to store duplicates
duplicates = []
# Loop through each element
for num in numbers:
if numbers.count(num) > 1 and num not in duplicates:
duplicates.append(num)
# Print the duplicate elements
print("Duplicate elements in the array are:", duplicates)You can see the output in the screenshot below.
This method checks how many times each element appears in the array using the count() function. If the count is greater than one, it gets added to the duplicates list (avoiding repetition).
While this works well for small arrays, it can be slow for large datasets because count() it scans the list every time. So, it’s great for learning, but not the most efficient method in Python.
Method 2 – Use Python Sets to Print Duplicate Elements
Once I got comfortable with Python, I started using sets because they make the code cleaner and faster. Sets automatically remove duplicates, which makes them perfect for this task.
Here’s how I do it:
numbers = [10, 15, 20, 15, 30, 40, 10, 50, 30, 60]
# Create two sets
seen = set()
duplicates = set()
# Loop through the list
for num in numbers:
if num in seen:
duplicates.add(num)
else:
seen.add(num)
# Print duplicates
print("Duplicate elements in the array are:", list(duplicates))You can see the output in the screenshot below.
In this code, I maintain two sets, one for elements I’ve already seen and another for duplicates. Whenever an element appears again, it’s added to the duplicates set.
This approach is much faster than using count() because set lookups in Python are O(1) on average. It’s ideal when working with large arrays, such as customer IDs or product SKUs in a retail dataset.
Method 3 – Use Python Collections.Counter
If you want a more Pythonic and professional solution, the collections.Counter class is your best friend. It’s fast, clean, and built exactly for this kind of frequency counting.
Here’s an example:
from collections import Counter
# Example array with repeating elements
cities = ["New York", "Chicago", "Dallas", "New York", "Miami", "Dallas", "Boston", "Chicago"]
# Use Counter to count occurrences
city_counts = Counter(cities)
# Print only duplicates
duplicates = [city for city, count in city_counts.items() if count > 1]
print("Duplicate elements in the array are:", duplicates)You can see the output in the screenshot below.
Counter automatically counts how many times each element appears in the Python array. Then, I filter out the ones that appear more than once.
This is one of my go-to methods when dealing with real-world datasets, especially when analyzing customer data or log files in Python. It’s both elegant and efficient.
Method 4 – Use List Comprehension and Python Sets
List comprehensions are one of Python’s most powerful and concise features. You can use them to find duplicates in just one line of code.
Here’s how:
numbers = [2, 4, 6, 8, 2, 10, 4, 12, 6, 14, 8]
# Find duplicates using list comprehension
duplicates = list(set([num for num in numbers if numbers.count(num) > 1]))
print("Duplicate elements in the array are:", duplicates)You can see the output in the screenshot below.
This method uses a list comprehension to check which elements appear more than once. Then, I convert it to a set to remove repeated duplicates.
It’s a neat, one-line solution, though not as efficient as using Counter or sets directly for very large arrays. Still, it’s a great option for quick scripts or small to medium datasets.
Method 5 – Use NumPy Arrays (for Numerical Data)
If you’re working with numerical data in Python, chances are you’re already using NumPy. NumPy provides efficient array operations and can help identify duplicates quickly.
Here’s how I use it:
import numpy as np
# Create a NumPy array
data = np.array([1, 2, 3, 4, 3, 2, 5, 6, 1, 7, 8, 2])
# Use NumPy unique with return_counts
unique, counts = np.unique(data, return_counts=True)
# Extract duplicates
duplicates = unique[counts > 1]
print("Duplicate elements in the array are:", duplicates)The np.unique() function returns all unique elements and their counts. By filtering where the count is greater than one, we easily get all duplicates.
This method is extremely fast and memory-efficient, making it perfect for large numerical datasets, for example, analyzing sensor data or financial records in Python.
Method 6 – Use Dictionary Comprehension (Advanced Python Trick)
For those who love writing concise and efficient Python code, dictionary comprehensions can also do the trick. It’s a bit advanced, but it gives you a lot of flexibility.
Here’s how:
numbers = [11, 22, 33, 44, 22, 55, 11, 66, 77, 11]
# Create a dictionary with element counts
count_dict = {num: numbers.count(num) for num in set(numbers)}
# Extract duplicates
duplicates = [num for num, count in count_dict.items() if count > 1]
print("Duplicate elements in the array are:", duplicates)This method uses a dictionary to map each element to its frequency. Then, it filters out elements that appear more than once.
It’s not the most performance-optimized way, but it’s a great demonstration of how flexible Python can be. I often use this when I want to quickly inspect both unique and duplicate counts.
Handle Case Sensitivity in Python Arrays
When dealing with strings, case sensitivity can cause unexpected duplicates. For example, “New York” and “new york” are treated differently in Python.
Here’s how to handle that:
cities = ["New York", "Chicago", "new york", "Boston", "CHICAGO", "Miami"]
# Convert all to lowercase before finding duplicates
normalized = [city.lower() for city in cities]
duplicates = [city for city in set(normalized) if normalized.count(city) > 1]
print("Duplicate elements (case-insensitive) are:", duplicates)By converting all items to lowercase, I ensure that duplicates are detected correctly regardless of case. This technique is especially useful when cleaning user-input data in Python applications.
Real-World Example: Detect Duplicate Customer IDs
Let’s look at a real-world scenario. Imagine you’re analyzing customer data from a U.S.-based e-commerce system, and you need to find duplicate customer IDs.
Here’s how I would do it:
from collections import Counter
customer_ids = [
"CUST001", "CUST002", "CUST003", "CUST001", "CUST004",
"CUST005", "CUST002", "CUST006", "CUST007", "CUST003"
]
# Use Counter to find duplicates
id_counts = Counter(customer_ids)
duplicates = [cid for cid, count in id_counts.items() if count > 1]
print("Duplicate customer IDs found:", duplicates)This approach helps ensure data integrity before performing analytics or merging datasets.
It’s one of the most common use cases I’ve seen in real-world Python projects.
So, these are some of the most effective ways to print duplicate elements in an array using Python. From basic loops to advanced tools like collections.Counter and NumPy, Python provides multiple ways to handle duplicates efficiently.
In my experience, the Counter method is the best balance of readability and performance for most use cases. However, if you’re working with large numerical datasets, NumPy is unbeatable in speed.
You may also like to read the other Python articles:
- Remove a Key Without an Error in the Python Dictionary
- Python Dictionary KeyError: None
- Python Dictionary Key Error
- Python Dictionary Comprehension
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
