Working with data in Python often involves combining information from multiple sources. Over my ten years of developing software, I’ve frequently had to merge lists while ensuring I don’t end up with redundant entries.
Whether you are consolidating customer names from different states or merging product inventories, keeping your data clean is essential.
In this tutorial, I will show you exactly how to merge lists in Python while automatically removing duplicates.
Merge Lists Without Duplicates Matters
When I first started coding, I would often use the + operator to combine lists. However, I quickly realized that this approach retains all duplicate values, which can compromise data analysis.
For instance, if you are merging a list of tech hubs in California with a list from Washington, you don’t want “Seattle” appearing twice if it was accidentally included in both datasets.
Cleaning your data at the moment of merging saves you a lot of debugging time later.
Use the Set() Function (The Fastest Way)
Sets are built-in Python collections that inherently do not allow duplicate elements. When you convert a list to a set, Python automatically handles the deduplication for you.
Let’s say you have two lists of popular grocery chains in the United States, and some overlap between the two lists.
# List of grocery chains in the Northeast
northeast_stores = ["Whole Foods", "Trader Joe's", "Wegmans", "Stop & Shop"]
# List of grocery chains in the West Coast
west_coast_stores = ["Safeway", "Trader Joe's", "Whole Foods", "Vons"]
# Merging and removing duplicates using set()
merged_list = list(set(northeast_stores + west_coast_stores))
print(merged_list)
# Output: ["Whole Foods", "Trader Joe's", "Wegmans", "Stop & Shop", "Safeway", "Vons"]I executed the above example code and added the screenshot below.
I prefer this method because it is incredibly fast. However, keep in mind that sets do not maintain the original order of the items.
Preserve Order with Dictionary Keys
Sometimes, the order of your data matters. I’ve worked on several projects where the sequence of the entries was crucial for the final report.
If you need to merge lists and keep the original order while removing duplicates, using dict.fromkeys() is a trick I use often.
# Top visited US National Parks - Group A
parks_group_a = ["Yellowstone", "Yosemite", "Zion", "Grand Canyon"]
# Top visited US National Parks - Group B
parks_group_b = ["Grand Canyon", "Acadia", "Yellowstone", "Rocky Mountain"]
# Merging while preserving order
merged_ordered_list = list(dict.fromkeys(parks_group_a + parks_group_b))
print(merged_ordered_list)
# Output: ['Yellowstone', 'Yosemite', 'Zion', 'Grand Canyon', 'Acadia', 'Rocky Mountain']I executed the above example code and added the screenshot below.
This method is my “go-to” when I want the first appearance of an item to define its position in the final list.
Use a Loop for More Control
While built-in functions are great, there are times when I need more control over how the merging happens.
Using a simple for loop allows you to add custom logic, such as checking for case sensitivity or filtering out specific values during the merge.
# List of domestic airlines
domestic_airlines = ["Delta", "United", "American", "Southwest"]
# List of international airlines (operating in the US)
international_airlines = ["Delta", "Lufthansa", "United", "Emirates"]
# Create a copy of the first list
final_airlines = domestic_airlines.copy()
# Loop through the second list and add only new items
for airline in international_airlines:
if airline not in final_airlines:
final_airlines.append(airline)
print(final_airlines)
# Output: ['Delta', 'United', 'American', 'Southwest', 'Lufthansa', 'Emirates']I executed the above example code and added the screenshot below.
I find this approach helpful when the lists are relatively small, and readability is the priority for the team.
The List Comprehension Approach
If you are a fan of concise code, list comprehensions are a powerful tool in the Python arsenal.
I often use this when I want to merge lists within a single line of code without sacrificing too much readability. It combines the logic of a loop into a compact syntax.
# Major tech companies in Silicon Valley
silicon_valley = ["Google", "Apple", "Meta", "Nvidia"]
# Major tech companies in Austin (Silicon Hills)
austin_tech = ["Tesla", "Oracle", "Google", "Meta"]
# Merge and remove duplicates in one line
merged_tech = silicon_valley + [company for company in austin_tech if company not in silicon_valley]
print(merged_tech)
# Output: ['Google', 'Apple', 'Meta', 'Nvidia', 'Tesla', 'Oracle']I executed the above example code and added the screenshot below.
This method is elegant because it appends the first list with only the unique items from the second list.
Use the Union Operator (Python 3.9+)
With the release of Python 3.9, we gained some cool new features for merging dictionaries, but we can also use set unions for lists quite effectively.
The | operator represents a union in set theory. It is a very “Pythonic” way to express that you want everything from both collections without repeats.
# US Ivy League Schools - Set 1
ivy_set_1 = {"Harvard", "Yale", "Princeton", "Columbia"}
# US Ivy League Schools - Set 2
ivy_set_2 = {"Dartmouth", "Yale", "Cornell", "Brown"}
# Use the union operator
all_ivies = list(ivy_set_1 | ivy_set_2)
print(all_ivies)I use this specifically when I am already working with set objects in my code, as it makes the math-like logic of the merge very clear to anyone reading the script.
Merge Multiple Lists at Once
In real-world scenarios, you rarely just have two lists. You might have five or ten lists coming from different API endpoints.
When I face this, I don’t want to write a dozen merge statements. Instead, I use itertools.chain or a nested approach.
import itertools
# Lists of US State Capitals by region
northeast = ["Albany", "Trenton", "Harrisburg"]
south = ["Austin", "Tallahassee", "Atlanta", "Austin"]
midwest = ["Springfield", "Columbus", "Madison"]
# Merge all and remove duplicates
all_capitals = list(set(itertools.chain(northeast, south, midwest)))
print(all_capitals)Using itertools is highly memory-efficient, which is something I always look for when handling larger datasets.
Performance Considerations
After a decade of Python development, I’ve learned that the “best” way often depends on the size of your data.
If you have millions of records (like US Census data), using set() is significantly faster than using a loop with if x not in list. This is because checking for an item in a list takes longer as the list grows, whereas checking a set takes the same amount of time regardless of size.
However, if your list only has a few hundred items, any of the methods above will work perfectly.
Which Method Should You Choose?
I generally follow these rules of thumb:
- Use set() if you don’t care about the order and want maximum speed.
- Use dict.fromkeys() if you need to keep the order of the items.
- Use a For Loop if you need to add complex conditional logic during the merge.
- Use itertools.chain if you are merging a large number of lists.
Every project has different requirements, so I recommend testing a few of these to see which fits your specific coding style.
I hope you found this tutorial helpful! Merging lists is a fundamental skill, and doing it efficiently will make your Python code much cleaner.
You may also like to read the other Python articles:
- Count Occurrences in Python List
- Remove Multiple Items From a List in Python
- Remove Brackets From List in Python
- ValueError: Could Not Convert String to Float in Python
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
