Streaming the file in pieces with pandas chunksize is the reliable way to read large CSV files in Python: only one chunk sits in memory at a time. Loading just the columns and types you need helps too. Here’s the core pattern:
for chunk in pd.read_csv("sales_2026.csv", chunksize=250_000):
process(chunk) # each chunk is a normal DataFrameIn this tutorial, we’ll test every approach on one generated sales file, timed and measured on Python 3.12.5, pandas 3.0.6, pyarrow 26.0.0 and polars 2.1.0. I ran each script myself, so the numbers below come straight from those runs.
Build a large sales CSV to test with
You need a file big enough to feel slow. This script writes two million US orders for 2026, with dollar amounts, to sales_2026.csv. If you’ve never written one before, see how to create a CSV file in Python.
import os
import time
import numpy as np
import pandas as pd
rows = 2_000_000
rng = np.random.default_rng(2026)
states = np.array(["CA", "TX", "FL", "NY", "IL", "PA", "OH", "GA", "NC", "WA"])
products = np.array(["Laptop", "Monitor", "Keyboard", "Mouse", "Headset", "Webcam"])
prices = np.array([899.99, 229.50, 79.99, 39.95, 119.00, 64.99])
start = time.perf_counter()
pick = rng.integers(0, len(products), rows)
sales = pd.DataFrame({
"order_id": np.arange(100001, 100001 + rows),
"order_date": pd.Timestamp("2026-01-01") + pd.to_timedelta(rng.integers(0, 300, rows), unit="D"),
"state": states[rng.integers(0, len(states), rows)],
"product": products[pick],
"quantity": rng.integers(1, 6, rows),
"unit_price": prices[pick],
})
sales["amount"] = (sales["quantity"] * sales["unit_price"]).round(2)
sales.to_csv("sales_2026.csv", index=False, date_format="%Y-%m-%d", float_format="%.2f")
size_mb = os.path.getsize("sales_2026.csv") / 1024**2
print(f"Wrote {rows:,} rows to sales_2026.csv ({size_mb:.1f} MB) in {time.perf_counter() - start:.1f} s")Output:
Wrote 2,000,000 rows to sales_2026.csv (86.9 MB) in 6.1 s
On my machine the file came out at 86.9 MB. Save the script as make_sales_csv.py, open Command Prompt in the same folder and run python make_sales_csv.py. You’ll need pip install pandas pyarrow polars first.
What happens when pandas reads the whole large CSV?
A plain pd.read_csv() loads every row and column at once. That’s fine until the DataFrame outgrows your RAM, so measure it with memory_usage(deep=True). My read_csv guide for pandas covers the everyday options.
import time
import pandas as pd
start = time.perf_counter()
sales = pd.read_csv("sales_2026.csv")
seconds = time.perf_counter() - start
print(f"Rows: {len(sales):,} Read time: {seconds:.2f} s")
print(f"Memory: {sales.memory_usage(deep=True).sum() / 1024**2:.1f} MB")
print(sales.dtypes)Output:
Rows: 2,000,000 Read time: 1.60 s
Memory: 142.1 MB
order_id int64
order_date str
state str
product str
quantity int64
unit_price float64
amount float64
dtype: object
The 86.9 MB file became a 142.1 MB DataFrame. Text columns cost the most, and the dates stay as plain strings. Before a full load, it’s worth peeking at a few rows with nrows=5.
Read a large CSV in chunks with chunksize
Passing chunksize returns a reader you loop over. Each chunk is a DataFrame, so you aggregate per chunk and combine the partial results. Memory stays at roughly one chunk, however big the file grows.
import pandas as pd
totals = pd.Series(dtype="float64")
rows = 0
with pd.read_csv("sales_2026.csv", chunksize=250_000) as reader:
for i, chunk in enumerate(reader, start=1):
rows += len(chunk)
totals = totals.add(chunk.groupby("state")["amount"].sum(), fill_value=0)
if i <= 2:
print(f"Chunk {i}: {len(chunk):,} rows, {chunk.memory_usage(deep=True).sum() / 1024**2:.1f} MB")
print(f"Processed {rows:,} rows in {i} chunks")
print("Top 3 states by revenue:")
for state, amount in totals.nlargest(3).items():
print(f" {state}: ${amount:,.2f}")
print(f"Total revenue: ${totals.sum():,.2f}")Output:
Chunk 1: 250,000 rows, 17.8 MB
Chunk 2: 250,000 rows, 17.8 MB
Processed 2,000,000 rows in 8 chunks
Top 3 states by revenue:
GA: $144,185,696.75
CA: $143,811,939.31
WA: $143,532,819.99
Total revenue: $1,434,494,131.28
Each chunk used about 17.8 MB here. Pick a chunk size that fits comfortably in memory; 100,000 to 500,000 rows is a sensible start. Splitting plain lists works the same way, as shown in splitting a Python list into chunks.
Chunks also let you cut a huge file down to a smaller one. This writes only Texas laptop orders, appending each filtered chunk:
import pandas as pd
out = "tx_laptops_2026.csv"
first = True
kept = 0
for chunk in pd.read_csv("sales_2026.csv", chunksize=250_000):
match = chunk[(chunk["state"] == "TX") & (chunk["product"] == "Laptop")]
match.to_csv(out, mode="w" if first else "a", header=first, index=False, float_format="%.2f")
first = False
kept += len(match)
print(f"Saved {kept:,} Texas laptop orders to {out}")
print(pd.read_csv(out, nrows=3))Output:
Saved 33,316 Texas laptop orders to tx_laptops_2026.csv
order_id order_date state product quantity unit_price amount
0 100039 2026-05-02 TX Laptop 5 899.99 4499.95
1 100128 2026-01-14 TX Laptop 2 899.99 1799.98
2 100286 2026-10-01 TX Laptop 2 899.99 1799.98The trick is writing the header once, then switching to mode="a". More export options are in saving a pandas DataFrame to CSV.
Cut pandas memory with usecols and dtype
Often you don’t need every column. usecols skips the rest while parsing, and the category dtype stores repeated text like state codes once:
import pandas as pd
MB = 1024**2
full = pd.read_csv("sales_2026.csv")
slim = pd.read_csv(
"sales_2026.csv",
usecols=["order_date", "state", "product", "amount"],
dtype={"state": "category", "product": "category"},
parse_dates=["order_date"],
)
full_mb = full.memory_usage(deep=True).sum() / MB
slim_mb = slim.memory_usage(deep=True).sum() / MB
print(f"All 7 columns, default types: {full_mb:.1f} MB")
print(f"4 columns, category + datetime: {slim_mb:.1f} MB ({slim_mb / full_mb:.0%} of the full read)")
print(slim.dtypes)Output:
All 7 columns, default types: 142.1 MB
4 columns, category + datetime: 34.3 MB (24% of the full read)
order_date datetime64[us]
state category
product category
amount float64
dtype: object
That slim read used 24% of the memory, and the dates are real datetime64 values now. For date handling details, see converting a pandas column to datetime.
Speed up read_csv with the pyarrow engine
pandas can hand parsing to Apache Arrow with engine="pyarrow", which uses several CPU cores. I timed it against the default C engine:
import time
import pandas as pd
tests = [
("C engine (default)", {}),
("pyarrow engine", {"engine": "pyarrow"}),
("pyarrow engine + pyarrow types", {"engine": "pyarrow", "dtype_backend": "pyarrow"}),
]
for label, options in tests:
start = time.perf_counter()
df = pd.read_csv("sales_2026.csv", **options)
seconds = time.perf_counter() - start
mb = df.memory_usage(deep=True).sum() / 1024**2
print(f"{label:<31} {seconds:5.2f} s {mb:7.1f} MB order_date: {df['order_date'].dtype}")Output:
C engine (default) 1.67 s 142.1 MB order_date: str
pyarrow engine 0.18 s 184.1 MB order_date: object
pyarrow engine + pyarrow types 0.10 s 100.1 MB order_date: date32[day][pyarrow]
The surprise: the pyarrow engine alone turned order_date into Python date objects, so it used more memory than the C engine. Adding dtype_backend="pyarrow" keeps it as compact date32, giving you speed and memory together.
One limit: the pyarrow engine doesn’t support chunksize. For files bigger than RAM, stick with the C engine and chunks.
Stream rows with Python’s csv module
If pandas isn’t available, the built-in csv module reads one row at a time with almost no memory. It’s slower, but fine for simple filters. Using Decimal keeps the dollar totals exact; reading a file line by line in Python explains the same streaming idea.
import csv
from decimal import Decimal
total = Decimal("0.00")
orders = 0
with open("sales_2026.csv", newline="", encoding="utf-8") as f:
for row in csv.DictReader(f):
if row["state"] == "TX" and row["order_date"].startswith("2026-03"):
total += Decimal(row["amount"])
orders += 1
print(f"Texas orders in March 2026: {orders:,}")
print(f"Revenue: ${total:,.2f}")Output:
Texas orders in March 2026: 20,817
Revenue: $14,846,936.03Query a large CSV lazily with Polars
Polars’ scan_csv() builds a lazy query, so it reads only the columns the query touches and runs on all cores. Nothing happens until collect():
import time
import polars as pl
start = time.perf_counter()
top = (
pl.scan_csv("sales_2026.csv")
.filter(pl.col("product") == "Lapto