As a Python developer working on a project for my clients, I recently faced an issue where I needed to identify and extract duplicate values from a list. After researching and experimenting with various methods, I discovered several effective techniques to accomplish this task. In this tutorial, I will explain how to find duplicates in a Python list with suitable examples.
Find Duplicates in a Python List
When working with lists in Python, it’s common to encounter situations where you need to find and handle duplicate elements. Duplicates can occur due to data inconsistencies, user input errors, or merging multiple data sources. Identifying and dealing with duplicates is crucial to ensuring data integrity and performing various data manipulation tasks efficiently.
Read How to Create an Empty List in Python?
Method 1: Use the count() Function
One simple approach to find duplicates in a list is by using the Python built-in count() function. This method iterates through each element in the list and checks if its count is greater than one. If an element appears more than once, it is considered a duplicate.
Here’s an example:
def find_duplicates(lst):
duplicates = []
for item in lst:
if lst.count(item) > 1 and item not in duplicates:
duplicates.append(item)
return duplicates
# Example usage
cities = ["New York", "Los Angeles", "Chicago", "Houston", "New York", "Chicago"]
duplicate_cities = find_duplicates(cities)
print("Duplicate cities:", duplicate_cities)Output:
Duplicate cities: ['New York', 'Chicago']I executed the above example code and added the screenshot below.
In this example, we define a function called find_duplicates() that takes a list lst as input. We iterate through each item in the list using a for loop. For each item, we use the count() function to determine its frequency in the list. If the count is greater than one and the item is not already present in the duplicates list, we append it to duplicates. Finally, we return the list of duplicate elements.
Check out How to Get the Index of an Element in a List in Python?
Method 2: Use a Dictionary
Another efficient way to find duplicates in a Python list is by utilizing a dictionary. We can create a dictionary where the keys are the unique elements from the list, and the values represent their frequencies. By iterating through the dictionary, we can easily identify the elements with a frequency greater than one.
Here’s an example:
def find_duplicates(lst):
frequency = {}
for item in lst:
if item in frequency:
frequency[item] += 1
else:
frequency[item] = 1
duplicates = [item for item, count in frequency.items() if count > 1]
return duplicates
# Example usage
names = ["John", "Emma", "Oliver", "Emma", "Liam", "Olivia", "John"]
duplicate_names = find_duplicates(names)
print("Duplicate names:", duplicate_names)Output:
Duplicate names: ['John', 'Emma']I executed the above example code and added the screenshot below.
In this approach, we create an empty dictionary called frequency. We iterate through each item in the list using a for loop. If the item already exists as a key in the dictionary, we increment its corresponding value by one. If the item is encountered for the first time, we add it to the dictionary with an initial value of one.
After building the frequency dictionary, we use a list comprehension to create a new list called duplicates. We iterate over the dictionary items and include only the items whose count is greater than one. Finally, we return the list of duplicate elements.
Read How to Remove Duplicates from a List in Python?
Method 3: Use the collections.Counter Class
Python’s collections module provides a convenient class called Counter that can be used to count the frequencies of elements in a list. By leveraging the Counter class, we can easily find the duplicates in a list.
Here’s an example:
from collections import Counter
def find_duplicates(lst):
counter = Counter(lst)
duplicates = [item for item, count in counter.items() if count > 1]
return duplicates
# Example usage
fruits = ["apple", "banana", "orange", "grape", "apple", "banana", "kiwi"]
duplicate_fruits = find_duplicates(fruits)
print("Duplicate fruits:", duplicate_fruits)Output:
Duplicate fruits: ['apple', 'banana']I executed the above example code and added the screenshot below.
In this method, we import the Counter class from the collections module. We create an instance of Counter called counter and pass the list lst as an argument. The Counter class automatically counts the frequencies of each element in the list.
Similar to the previous method, we use a list comprehension to create a new list called duplicates. We iterate over the counter items and include only the items whose count is greater than one. Finally, we return the list of duplicate elements.
Check out How to Convert String to List in Python?
Handle Duplicates
Once you have identified the duplicate elements in a list, you can perform various actions based on your specific requirements. Some common scenarios include:
- Removing Duplicates: If you want to remove the duplicate elements and keep only the unique values, you can convert the list to a set and then back to a list. For example:
lst = [1, 2, 3, 2, 4, 1, 5]
unique_lst = list(set(lst))
print("List with duplicates removed:", unique_lst)Output:
List with duplicates removed: [1, 2, 3, 4, 5]- Count Duplicates: If you need to count the occurrences of each duplicate element, you can use the
Counterclass as demonstrated in Method 3. For example:
from collections import Counter
lst = ["apple", "banana", "orange", "grape", "apple", "banana", "kiwi"]
counter = Counter(lst)
for item, count in counter.items():
if count > 1:
print(f"{item}: {count} occurrences")Output:
apple: 2 occurrences
banana: 2 occurrences- Filter Duplicates: If you want to filter out duplicate elements based on certain criteria, you can use conditional statements or list comprehensions. For example:
lst = [("John", 25), ("Emma", 30), ("Oliver", 35), ("Emma", 28), ("Liam", 40)]
filtered_lst = [item for item in lst if item[0] != "Emma"]
print("Filtered list:", filtered_lst)Output:
Filtered list: [('John', 25), ('Oliver', 35), ('Liam', 40)]In this example, we filter out the tuples where the first element is “Emma”.
Read Convert String to List in Python Without Using Split
Conclusion
In this tutorial, I explained how to find duplicates in a Python list. I discussed their methods to achieve this task, such as using the count() function, using a dictionary and using the collections.Counter Class. I also discussed how to handle duplicates.
You may read:
- How to Iterate Through a List in Python?
- How to Write a List to a File in Python?
- How to Select Items from a List in Python?
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
