A payroll script I built for a small U.S. company received employee badge IDs from several office locations. The combined data was already sorted, but some IDs appeared more than once. Those duplicates caused incorrect employee counts and extra payroll validation work.
To remove duplicates from a sorted array in Python, I used the two-pointer technique. It scans the array once, keeps each unique value, and avoids unnecessary memory usage.
This guide explains that approach step by step. You will also learn when a new list, Python array, or NumPy array makes more sense.
What Does Removing Duplicates From a Sorted Array Mean?
A sorted array stores values in ascending or descending order. In Python, developers commonly use a list when they informally refer to an array.
Consider these sorted employee badge IDs:
employee_ids = [101, 101, 102, 103, 103, 103, 104]
The value 101 appears twice, while 103 appears three times. After removing duplicates, the result should contain one copy of each ID:
[101, 102, 103, 104]
Sorting matters because equal values appear next to each other. You only need to compare each item with the last unique item.
With an unsorted list, the same value might appear in several distant positions. You would need extra tracking, sorting, or membership checks. If your data is unsorted, review how to sort an array in Python before applying the in-place solution.
You can also read about the differences between common Python containers in this guide to lists, tuples, sets, and dictionaries.
Remove Duplicates From a Sorted Array in Python In Place
The two-pointer technique provides the best general solution for this problem. A pointer is simply a variable that stores an array index.
This approach uses:
- A write pointer that marks where the next unique value belongs.
- A read pointer that scans every remaining value.
- One comparison between the current value and the last unique value.
The algorithm changes the original list. This behavior makes it an in-place operation because it does not create another full-size list.
Create the Sample Sorted Array
I will use sorted transaction category codes from a U.S. accounting export:
category_codes = [1001, 1001, 1002, 1002, 1005, 1008, 1008]
print(category_codes)
Output:
[1001, 1001, 1002, 1002, 1005, 1008, 1008]
I executed the above example code and added the screenshot below.
The list contains seven items but only four unique category codes. If you are new to this data structure, see how to create arrays in Python.
Handle an Empty Array First
An empty list has no first element. Accessing index 0 would raise an IndexError.
Check the list before starting the pointer logic:
category_codes = []
if not category_codes:
print("The array is empty.")
else:
print("The array contains data.")
Output:
The array is empty.
I executed the above example code and added the screenshot below.
The expression not category_codes returns True when the list contains no items. You can explore more empty-array checks in this guide on how to check if an array is empty in Python.
Write the Two-Pointer Function
A function is a reusable block of code that performs one specific task. The following function removes adjacent duplicates and returns the number of unique values:
def remove_sorted_duplicates(values):
if not values:
return 0
write_index = 1
for read_index in range(1, len(values)):
if values[read_index] != values[write_index - 1]:
values[write_index] = values[read_index]
write_index += 1
return write_index
category_codes = [1001, 1001, 1002, 1002, 1005, 1008, 1008]
unique_count = remove_sorted_duplicates(category_codes)
unique_codes = category_codes[:unique_count]
print("Unique count:", unique_count)
print("Unique codes:", unique_codes)
Output:
Unique count: 4
Unique codes: [1001, 1002, 1005, 1008]
I executed the above example code and added the screenshot below.
The function starts write_index at 1 because the first value always counts as unique. The for loop starts reading from the second item.
When the current value differs from the last unique value, the function copies it into the write position. It then moves write_index forward.
The function returns write_index, which also represents the number of unique items. For more background on reusable code, see how to define a function in Python.
Pro Tip: In my experience, most errors happen when developers compare against
values[read_index - 1]. Compare againstvalues[write_index - 1]instead. That position always holds the last accepted unique value.
Understand How the Pointers Move
Consider this smaller array:
numbers = [10, 10, 20, 20, 30]
The pointer movement looks like this:
| Read index | Current value | Last unique value | Action |
|---|---|---|---|
| 1 | 10 | 10 | Skip duplicate |
| 2 | 20 | 10 | Write 20 at index 1 |
| 3 | 20 | 20 | Skip duplicate |
| 4 | 30 | 20 | Write 30 at index 2 |
After the loop, the internal list looks like this:
numbers = [10, 10, 20, 20, 30]
unique_count = remove_sorted_duplicates(numbers)
print(numbers)
print(numbers[:unique_count])
Output:
[10, 20, 30, 20, 30]
[10, 20, 30]
The values after unique_count no longer matter. The algorithm only guarantees that the first section contains unique values.
This detail often surprises beginners. The function rearranges useful values at the front, but it does not automatically shorten the list.
Delete the Unused Values
You can remove the leftover section with del:
numbers = [10, 10, 20, 20, 30]
unique_count = remove_sorted_duplicates(numbers)
del numbers[unique_count:]
print(numbers)
Output:
[10, 20, 30]
The slice numbers[unique_count:] represents everything after the unique section. The del statement removes that entire range.
You can also assign a sliced copy back to the variable:
numbers = [10, 10, 20, 20, 30]
unique_count = remove_sorted_duplicates(numbers)
numbers = numbers[:unique_count]
print(numbers)
Output:
[10, 20, 30]
Slicing creates a new list, while del shortens the existing list. See this detailed guide to slicing lists in Python for more examples.
Remove Duplicates From a Sorted Array and Return a New List
An in-place update is efficient, but it changes the original data. That behavior may create problems when another part of your program still needs the complete array.
A new-list approach often looks clearer in business scripts. It uses additional memory, but the code remains easy to read and test.
def get_unique_sorted_values(values):
if not values:
return []
unique_values = [values[0]]
for value in values[1:]:
if value != unique_values[-1]:
unique_values.append(value)
return unique_values
invoice_numbers = [5001, 5001, 5002, 5004, 5004, 5005]
unique_invoices = get_unique_sorted_values(invoice_numbers)
print("Original:", invoice_numbers)
print("Unique:", unique_invoices)
Output:
Original: [5001, 5001, 5002, 5004, 5004, 5005]
Unique: [5001, 5002, 5004, 5005]
The function adds the first value to unique_values. It then compares every remaining item with the last stored item.
This version preserves the input list and returns a clean result. I prefer it for small automation scripts where clarity matters more than saving a little memory.
The append() method adds each accepted value to the end of the list. You can practice that operation with these examples for adding elements to an empty list.
Remove Duplicates From a Sorted Array Using itertools.groupby
The built-in itertools module contains tools for working with iterators. An iterator supplies values one at a time instead of loading an additional collection immediately.
The groupby() function places consecutive equal values into groups. Because the array is sorted, all copies of a value belong to the same group.
from itertools import groupby
state_codes = ["CA", "CA", "FL", "NY", "NY", "TX", "TX"]
unique_states = [state for state, group in groupby(state_codes)]
print(unique_states)
Output:
['CA', 'FL', 'NY', 'TX']
Each groupby() result contains a key and a group iterator. The list comprehension keeps only each group’s key.
This solution is concise and does not modify the original list. However, the two-pointer version makes the underlying logic easier to understand during a coding interview or beginner Python lesson.
Remove Duplicates From a Sorted Python array.array
Python includes an array module for storing values of one fixed data type. Unlike a regular list, an array.array cannot mix integers, strings, and other unrelated types.
The same two-pointer function works because array.array supports indexing and assignment:
from array import array
warehouse_ids = array("i", [12, 12, 15, 18, 18, 21])
unique_count = remove_sorted_duplicates(warehouse_ids)
del warehouse_ids[unique_count:]
print(warehouse_ids.tolist())
Output:
[12, 15, 18, 21]
The "i" type code tells Python to store signed integers. The tolist() method converts the result into a regular list for display.
Use array.array when you need compact storage for many values of the same basic type. A standard list remains more convenient for most beginner scripts and small command-line applications.
You can learn the setup details in this guide on how to initialize an array in Python.
Remove Duplicates From a Sorted NumPy Array
NumPy is commonly used for numerical analysis and large datasets. Its unique() function returns the distinct values from an array.
import numpy as np
sales_regions = np.array([1, 1, 2, 2, 3, 4, 4])
unique_regions = np.unique(sales_regions)
print(unique_regions)
Output:
[1 2 3 4]
This approach creates a new NumPy array. It provides short, readable code when your project already depends on NumPy.
Do not add NumPy only for this single operation. The built-in list solution avoids another dependency and works well for normal automation tasks.
If you already process numerical data, review these