In this tutorial, I will explain how to download files from URLs using Python. As a developer based in the USA, I recently needed to download a large dataset from a government website for a data analysis project. I found that using Python to automate the file download process was incredibly efficient and saved me a lot of time. In this article, I’ll walk you through the different methods and libraries you can use to download files from URLs with Python, providing clear examples.
Download Files from URLs Using Python
Before getting into the code, it’s essential to understand the basics of downloading files from URLs. In Python, there are several libraries available that make downloading files simple and efficient.
Read How to Copy Files to Another Directory in Python?
Method 1: Use the urllib Library
The urllib library is a built-in Python module that provides a simple interface for downloading files from URLs. Here’s an example of how to use urllib.request.urlretrieve() to download a file:
import urllib.request
url = "https://www.census.gov/data/datasets/time-series/demo/popest/2010s-counties-total.html"
filename = "usa_county_population.csv"
urllib.request.urlretrieve(url, filename)
print(f"File {filename} downloaded successfully!")I executed the above example code and added the screenshot below.
In this example, we import the urllib.request module and specify the URL of the file we want to download, along with the desired filename. We then use urllib.request.urlretrieve() to download the file and save it locally. This is one of the most commonly used methods for downloading files in Python.
Check out How to Open a File in Python?
Method 2: Use the requests Library
Another popular library for downloading files in Python is requests. It provides a more user-friendly interface compared to urllib. Here’s an example:
import requests
url = "https://www.bls.gov/oes/current/oes_nat.htm"
filename = "usa_occupational_employment.xlsx"
response = requests.get(url)
with open(filename, 'wb') as file:
file.write(response.content)
print(f"File {filename} downloaded successfully!")I executed the above example code and added the screenshot below.
In this example, we use requests.get() to send a GET request to the specified URL. We then open a file in write binary mode (‘wb’) and write the content of the response to the file. This method is more readable and easier to use compared to urllib.
Read Python Get Filename From Path
Method 3: Use the wget Library
The wget library is another option for downloading files in Python. It provides a simple interface and supports features like resuming interrupted downloads. Here’s an example:
import wget
url = "https://www.ers.usda.gov/webdocs/DataFiles/48747/Unemployment.xls"
filename = "usa_unemployment_data.xls"
wget.download(url, filename)
print(f"File {filename} downloaded successfully!")In this example, we use the wget module to download an Excel file containing unemployment data from the USDA website. The wget.download() function takes the URL and the desired filename as arguments and downloads the file.
Check out Python file Does Not Exist Exception
Conclusion
In this article, I have explained how to download files from URLs using Python. I discussed mainly three methods to accomplish this task such as using urllib library, using the request library, using the wget library.
You can also read:
- How to Copy File and Rename in Python
- How to Check If a File Exists and Create It If Not in Python?
- How to Get the File Size in MB using Python?
I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.
