Python merge two lists without duplicates

In this Python tutorial, we will discuss how Python merge two lists without duplicates. Here we will understand how to merge two given lists in Python without duplicates.

In Python, we can merge two lists without duplicates using the following 6 methods.

  • Using set() function
  • Using extend() method
  • Using or operator
  • Using np.unique() method
  • Using collections.Counter()
  • Using loop & append()

Python merge two lists without duplicates

Here we will discuss all 6 methods to merge two lists without duplicates in Python. Let us start with the first method of using the set() function.

Method 1: Python merge two lists without duplicates using set()

One of the common ways to merge two lists with duplicates in Python is by following the given steps:

  • First, combining all list elements into one list
  • Next, covert the list with all the elements into a set using the set() function
  • The set will hold only the unique values and it will remove all the duplicates from it.
  • In the last, convert the set back into a list using the list() function.

Here is an example of this approach in Python.

# Defining two lists with dupliactes
day1_attendees = ["Alice", "Bob", "Charlie", "David"]
day2_attendees = ["Charlie", "David", "Emily", "Frank"]

# Merging lists together
all_attendees = day1_attendees + day2_attendees

# Printing merged list
print("Complete list:", all_attendees)

# Removing duplicates
unique_attendees = list(set(all_attendees))

# Printing merged list without duplicates
print("List without duplicates:", unique_attendees)

Once we execute the above Python program, we will get the following result.

Complete list: ['Alice', 'Bob', 'Charlie', 'David', 'Charlie', 'David', 'Emily', 'Frank']
List without duplicates: ['Alice', 'Charlie', 'Emily', 'David', 'Frank', 'Bob']

Read: Linked Lists in Python

Method 2: Python merge two lists without duplicates using extend()

Another way to merge two lists without duplicates in Python is by using the extend() method.

Like in the append() method, we pass only a single value to add to the last of the list. In the same way, the extend() method also allows to insert elements to the last of the list. However, the extend() method accepts an iterable instead of a single value.

So, within the extend() method, we will use the for loop and if statement to insert only unique values to the list without any duplicates.

Here is the code for this example in Python.

# Defining two lists with dupliactes
emp_1 = ["Alice", "Bob", "Charlie", "David"]
emp_2 = ["Charlie", "David", "Alice", "Frank"]

# Creating a copy
merged_list = list(emp_1)

# Using extend() method
merged_list.extend(y for y in emp_2 if y not in merged_list)

# Printing merged list without duplicates
print("List without duplicates:", merged_list)

In the above example, we defined 2 lists – emp_1 and emp_2. After this, we created a copy of the first list named merged_list. Then utilized the extend() method with the loop and if statement on the merged_list.

In this way, we will get a merged list with no duplicates in Python.

List without duplicates: ['Alice', 'Bob', 'Charlie', 'David', 'Frank']

Read: Python Extend Vs Append

Method 3: Python merge two lists without duplicates using or operator

Another quick way to merge two lists without any duplicates is by using the or operator with the set() function in Python.

In this method, again we will use the set() function to convert a list to a set and to merge both sets using the or operator. In the last, we will gain convert the result of or operator back into a list.

# Defining two lists with dupliactes
states_1 = ["California", "Texas", "Florida"]
states_2 = ["Florida", "California", "Georgia", "Washington"]

# Using set() and or operator
unique_states = list(set(states_1) | set(states_2))

# Printing merged list without duplicates
print("List without duplicates:", unique_states)

In the above example, we have taken two different lists in Python that consists of duplicate values. After this, we used the set() function to convert those lists to sets and used the or operator between them to fetch only unique values.

Once we execute the above program, we will get the following list without any duplicates.

List without duplicates: ['Texas', 'Washington', 'California', 'Florida', 'Georgia']

Read: Concatenate multiple lists in Python

Method 4: Python merge two lists without duplicates using np.unique()

The np.unique() is a NumPy function in Python that returns the sorted unique elements of an array.

So, in this method, we will concatenate both the Python lists using the + operator. And then use the np.unique() method to create an array of only unique values. In the last, we will convert that array back into a list using the list() function in Python.

# Importing numpy 
import numpy as np

# Defining two lists with dupliactes
num_1 = [21, 31, 41, 51, 61]
num_2 = [32, 21, 42, 61, 51]

# Using np.unique()
unique_nums = list(np.unique(num_1 + num_2))

# Printing merged list without duplicates
print("List without duplicates:", unique_nums)

In the above example, we have merged num_1 and num_2 lists using the np.unique() and list() functions in Python.

The result of the above Python program is shown below.

List without duplicates: [21, 31, 32, 41, 42, 51, 61]

Read: How to add string to list Python

Method 5: Python merge two lists without duplicates using collections

In the collections module, we have the Counter built-in class that takes an iterable as input and return a dictionary containing the count of each value. In this dictionary, the key is the unique value and the value is the count of each value.

So, by utilizing its keys from the Counter class, we can merge two lists without duplicates. An example of this approach is given below.

# Importing Counter
from collections import Counter

# Defining two lists with dupliactes
num_1 = [21, 31, 41, 51, 61]
num_2 = [32, 21, 42, 61, 51]

# Creating a Counter object for each list
counter_num_1 = Counter(num_1)
counter_num_2 = Counter(num_2)

# Add the Counter objects, but only unique records from num_2
combined_counter = counter_num_1 + Counter(ele for ele in counter_num_2 if ele not in counter_num_1)
 
# Convert the Counter object to list
final_list = list(combined_counter.elements())
 
# Printing merged list without duplicates
print("List without duplicates:", final_list)

In the above example, we have used the Counter class from the collections module and created a Counter object using each list. After this, we merged both Counter objects by using for loop and if statement.

In the end, once we get the Counter object with all unique values, we will convert the Counter object back into a list.

The result of the above example is shown below.

List without duplicates: [21, 31, 41, 51, 61, 32, 42]

Read: Python program to loop through a list

Method 6: Merge two lists Python without duplicates using a loop & append()

  • In this method, we will simply use the for loop to iterate over each item in one list.
  • After this, we will use the if statement to check if the item given in one list also exists in another one or not.
  • In the last, we will use the append() method to add the unique list element from one list to another. This will result in forming a single list with all unique values.

An example of this approach is given below.

# Defining two lists with dupliactes
num_1 = [21, 31, 41, 51, 61]
num_2 = [32, 21, 42, 61, 51]

# Creating a copy
merged_list = list(num_1)


# Uisng for loop and append()
for element in num_2:
    if element not in merged_list:
        merged_list.append(element)
      

# Printing merged list without duplicates
print("List without duplicates:", merged_list)

Once we execute the above Python program, we will get the following result.

Python merge two lists without duplicates
Merging two lists without duplicates in Python

You may also like to read the following Python tutorials.

Conclusion

So, in this Python tutorial, we have understood how to merge two lists in Python using 5 different methods. Moreover, for each method in Python, we will discuss each method using examples.

Here is the list of topics that we have covered.

  • Using set() function
  • Using extend() method
  • Using or operator
  • Using np.unique() method
  • Using collections.Counter()
  • Using loop & append()