How to Remove Duplicates from a Sorted Array in Python

A payroll script I built for a small U.S. company received employee badge IDs from several office locations. The combined data was already sorted, but some IDs appeared more than once. Those duplicates caused incorrect employee counts and extra payroll validation work.

To remove duplicates from a sorted array in Python, I used the two-pointer technique. It scans the array once, keeps each unique value, and avoids unnecessary memory usage.

This guide explains that approach step by step. You will also learn when a new list, Python array, or NumPy array makes more sense.

What Does Removing Duplicates From a Sorted Array Mean?

A sorted array stores values in ascending or descending order. In Python, developers commonly use a list when they informally refer to an array.

Consider these sorted employee badge IDs:

employee_ids = [101, 101, 102, 103, 103, 103, 104]

The value 101 appears twice, while 103 appears three times. After removing duplicates, the result should contain one copy of each ID:

[101, 102, 103, 104]

Sorting matters because equal values appear next to each other. You only need to compare each item with the last unique item.

With an unsorted list, the same value might appear in several distant positions. You would need extra tracking, sorting, or membership checks. If your data is unsorted, review how to sort an array in Python before applying the in-place solution.

You can also read about the differences between common Python containers in this guide to lists, tuples, sets, and dictionaries.

Remove Duplicates From a Sorted Array in Python In Place

The two-pointer technique provides the best general solution for this problem. A pointer is simply a variable that stores an array index.

This approach uses:

• A write pointer that marks where the next unique value belongs.
• A read pointer that scans every remaining value.
• One comparison between the current value and the last unique value.

The algorithm changes the original list. This behavior makes it an in-place operation because it does not create another full-size list.

Create the Sample Sorted Array

I will use sorted transaction category codes from a U.S. accounting export:

category_codes = [1001, 1001, 1002, 1002, 1005, 1008, 1008]

print(category_codes)

Output:

[1001, 1001, 1002, 1002, 1005, 1008, 1008]

I executed the above example code and added the screenshot below.

The list contains seven items but only four unique category codes. If you are new to this data structure, see how to create arrays in Python.

Handle an Empty Array First

An empty list has no first element. Accessing index 0 would raise an IndexError.

Check the list before starting the pointer logic:

category_codes = []

if not category_codes:
    print("The array is empty.")
else:
    print("The array contains data.")

Output:

The array is empty.

I executed the above example code and added the screenshot below.

The expression not category_codes returns True when the list contains no items. You can explore more empty-array checks in this guide on how to check if an array is empty in Python.

Write the Two-Pointer Function

A function is a reusable block of code that performs one specific task. The following function removes adjacent duplicates and returns the number of unique values:

def remove_sorted_duplicates(values):
    if not values:
        return 0

    write_index = 1

    for read_index in range(1, len(values)):
        if values[read_index] != values[write_index - 1]:
            values[write_index] = values[read_index]
            write_index += 1

    return write_index


category_codes = [1001, 1001, 1002, 1002, 1005, 1008, 1008]

unique_count = remove_sorted_duplicates(category_codes)
unique_codes = category_codes[:unique_count]

print("Unique count:", unique_count)
print("Unique codes:", unique_codes)

Output:

Unique count: 4
Unique codes: [1001, 1002, 1005, 1008]

I executed the above example code and added the screenshot below.

The function starts write_index at 1 because the first value always counts as unique. The for loop starts reading from the second item.

When the current value differs from the last unique value, the function copies it into the write position. It then moves write_index forward.

The function returns write_index, which also represents the number of unique items. For more background on reusable code, see how to define a function in Python.

Pro Tip: In my experience, most errors happen when developers compare against values[read_index - 1]. Compare against values[write_index - 1] instead. That position always holds the last accepted unique value.

Understand How the Pointers Move

Consider this smaller array:

numbers = [10, 10, 20, 20, 30]

The pointer movement looks like this:

After the loop, the internal list looks like this:

numbers = [10, 10, 20, 20, 30]

unique_count = remove_sorted_duplicates(numbers)

print(numbers)
print(numbers[:unique_count])

Output:

[10, 20, 30, 20, 30]
[10, 20, 30]

The values after unique_count no longer matter. The algorithm only guarantees that the first section contains unique values.

This detail often surprises beginners. The function rearranges useful values at the front, but it does not automatically shorten the list.

Delete the Unused Values

You can remove the leftover section with del:

numbers = [10, 10, 20, 20, 30]

unique_count = remove_sorted_duplicates(numbers)
del numbers[unique_count:]

print(numbers)

Output:

[10, 20, 30]

The slice numbers[unique_count:] represents everything after the unique section. The del statement removes that entire range.

You can also assign a sliced copy back to the variable:

numbers = [10, 10, 20, 20, 30]

unique_count = remove_sorted_duplicates(numbers)
numbers = numbers[:unique_count]

print(numbers)

Output:

[10, 20, 30]

Slicing creates a new list, while del shortens the existing list. See this detailed guide to slicing lists in Python for more examples.

Remove Duplicates From a Sorted Array and Return a New List

An in-place update is efficient, but it changes the original data. That behavior may create problems when another part of your program still needs the complete array.

A new-list approach often looks clearer in business scripts. It uses additional memory, but the code remains easy to read and test.

def get_unique_sorted_values(values):
    if not values:
        return []

    unique_values = [values[0]]

    for value in values[1:]:
        if value != unique_values[-1]:
            unique_values.append(value)

    return unique_values


invoice_numbers = [5001, 5001, 5002, 5004, 5004, 5005]

unique_invoices = get_unique_sorted_values(invoice_numbers)

print("Original:", invoice_numbers)
print("Unique:", unique_invoices)

Output:

Original: [5001, 5001, 5002, 5004, 5004, 5005]
Unique: [5001, 5002, 5004, 5005]

The function adds the first value to unique_values. It then compares every remaining item with the last stored item.

This version preserves the input list and returns a clean result. I prefer it for small automation scripts where clarity matters more than saving a little memory.

The append() method adds each accepted value to the end of the list. You can practice that operation with these examples for adding elements to an empty list.

Remove Duplicates From a Sorted Array Using itertools.groupby

The built-in itertools module contains tools for working with iterators. An iterator supplies values one at a time instead of loading an additional collection immediately.

The groupby() function places consecutive equal values into groups. Because the array is sorted, all copies of a value belong to the same group.

from itertools import groupby

state_codes = ["CA", "CA", "FL", "NY", "NY", "TX", "TX"]

unique_states = [state for state, group in groupby(state_codes)]

print(unique_states)

Output:

['CA', 'FL', 'NY', 'TX']

Each groupby() result contains a key and a group iterator. The list comprehension keeps only each group’s key.

This solution is concise and does not modify the original list. However, the two-pointer version makes the underlying logic easier to understand during a coding interview or beginner Python lesson.

Remove Duplicates From a Sorted Python array.array

Python includes an array module for storing values of one fixed data type. Unlike a regular list, an array.array cannot mix integers, strings, and other unrelated types.

The same two-pointer function works because array.array supports indexing and assignment:

from array import array

warehouse_ids = array("i", [12, 12, 15, 18, 18, 21])

unique_count = remove_sorted_duplicates(warehouse_ids)
del warehouse_ids[unique_count:]

print(warehouse_ids.tolist())

Output:

[12, 15, 18, 21]

The "i" type code tells Python to store signed integers. The tolist() method converts the result into a regular list for display.

Use array.array when you need compact storage for many values of the same basic type. A standard list remains more convenient for most beginner scripts and small command-line applications.

You can learn the setup details in this guide on how to initialize an array in Python.

Remove Duplicates From a Sorted NumPy Array

NumPy is commonly used for numerical analysis and large datasets. Its unique() function returns the distinct values from an array.

import numpy as np

sales_regions = np.array([1, 1, 2, 2, 3, 4, 4])

unique_regions = np.unique(sales_regions)

print(unique_regions)

Output:

[1 2 3 4]

This approach creates a new NumPy array. It provides short, readable code when your project already depends on NumPy.

Do not add NumPy only for this single operation. The built-in list solution avoids another dependency and works well for normal automation tasks.

If you already process numerical data, review these Python NumPy array examples and this guide to NumPy array indexing.

Verify That the Input Is Sorted

The two-pointer approach assumes that duplicates sit next to each other. It produces the wrong result when equal values appear in separate positions.

For example:

employee_ids = [101, 103, 101, 104]

unique_count = remove_sorted_duplicates(employee_ids)

print(employee_ids[:unique_count])

Output:

[101, 103, 101, 104]

The second 101 remains because it does not appear beside the first one.

You can validate the order before removing duplicates:

def is_sorted(values):
    return all(
        values[index] <= values[index + 1]
        for index in range(len(values) - 1)
    )


employee_ids = [101, 103, 101, 104]

if is_sorted(employee_ids):
    print("Ready to remove duplicates.")
else:
    print("Sort the array first.")

Output:

Sort the array first.

For trusted data pipelines, this validation may add unnecessary work. For user-provided files or command-line input, the check can prevent silent data errors.

Compare the Available Approaches

Each solution has a different balance of clarity, memory use, and dependencies.

Time complexity describes how runtime grows as the input grows. O(n)O(n) means the algorithm processes each item once.

Space complexity describes the additional memory an algorithm needs. The two-pointer approach uses constant extra space because it only creates index variables.

For a sorted list with millions of values, I would choose two pointers. For a short local script, I would usually return a new list because the intent stays obvious.

Things to Keep in Mind

• Confirm the sort order: The two-pointer algorithm only removes adjacent duplicates. Sort or validate unknown input before processing it.
• Handle empty arrays: Always return early for an empty collection. Otherwise, accessing the first element causes an IndexError.
• Choose whether to modify the input: In-place updates save memory but can surprise other parts of your program. Return a new list when the original data still matters.
• Use the unique count correctly: After an in-place scan, only values[:unique_count] contains the valid result. Values beyond that boundary are leftover data.
• Test different data types: The comparison works with integers, strings, dates, and custom objects when equal values compare consistently.
• Avoid repeated removal calls: Calling remove() inside a loop shifts later elements repeatedly. That pattern performs poorly on large arrays.

Frequently Asked Questions

You learned how to remove duplicates from a sorted array in Python using two pointers, a new list, itertools, and NumPy. For most large sorted arrays, the in-place two-pointer approach offers the best balance of speed and memory usage. I hope you found this article helpful.

You May Also Like

• Remove duplicates from an array in Python
• Remove duplicates from a Python list
• Get unique values from a list in Python
• Convert an array to a set in Python
• Count occurrences in Python arrays

Bijay Kumar

Bijay Kumar is an experienced Python and AI professional who enjoys helping developers learn modern technologies through practical tutorials and examples. His expertise includes Python development, Machine Learning, Artificial Intelligence, automation, and data analysis using libraries like Pandas, NumPy, TensorFlow, Matplotlib, SciPy, and Scikit-Learn. At PythonGuides.com, he shares in-depth guides designed for both beginners and experienced developers. More about us.

enjoysharepoint.com/