How to Convert String to UTF-8 in Python

In some situations, you may have the strings that you want to convert into UTF-8 in Python so that you can ensure correct rendering in a web application.

Python provides some built-in functions to convert strings to UTF-8, and they are pretty easy and simple to use.

In this article, I will show you a couple of methods for converting strings to UTF8 in Python along with examples and screenshots.

Character Encoding in Python

Before getting into code, let’s establish what we’re talking about when we mention UTF-8 encoding.

Text in computers is stored as numbers, and character encodings define how characters map to those numbers. UTF-8 is a variable-width character encoding that can represent every character in the Unicode standard while remaining backward compatible with ASCII.

UTF-8 has become the dominant encoding on the web, with over 97% of websites using it. When working with international text, proper UTF-8 handling is crucial.

Read How to Reverse a String in Python?

Convert String to UTF-8 in Python

Let me explain to you the important methods to convert string to UTF-8 in Python along with examples.

1. Use the .encode(‘utf-8’)

In Python, strings are stored as Unicode by default. Sometimes, you may need to convert them into UTF-8 bytes. The .encode('utf-8') method takes your string and converts it into a series of bytes using the UTF-8 encoding format.

text = "Hello, world!"
utf8_encoded = text.encode('utf-8')

print(utf8_encoded)
print(type(utf8_encoded))

Output:

b'Hello, world!'
<class 'bytes'>

I executed the above example code and added the screenshot below.

"Hello, world!" is a regular Python string, encode('utf-8') turns it into bytes then you’ll see the b in front.

Check out Convert String to Float with 2 Decimal Places in Python

2. Use the bytes() constructor

In Python, the bytes() constructor can also be used to convert a string into UTF-8 encoded bytes. It works just like .encode('utf-8') but you pass the string and encoding as arguments to the bytes() function.

text = "Python is fun"
utf8_bytes = bytes(text, encoding='utf-8')

print(utf8_bytes)
print(type(utf8_bytes))

Output:

b'Python is fun'
<class 'bytes'>

I executed the above example code and added the screenshot below.

This method is beneficial when you want to show both the input string and the encoding used clearly.

Read Convert String of 1s and 0s to Binary in Python

3. Use .encode() with error handling

When converting a Python string to UTF-8, some characters might not be supported or cause errors. To avoid the program crashing, you can add error handling using the errors parameter in .encode().

text = "Café ☕"

# Convert to UTF-8 bytes with error handling
utf8_bytes = text.encode('utf-8', errors='ignore')  # or use 'replace'

print(utf8_bytes) 
print(type(utf8_bytes))

Output:

b'Caf\xc3\xa9 \xe2\x98\x95'
<class 'bytes'>

I executed the above example code and added the screenshot below.

The string contains special characters: é and a coffee emoji ☕, .encode('utf-8', errors='ignore') tries to convert everything, but skips anything it can’t handle.

I have listed some error parameter options, they are:

Check out How to Compare Strings in Python?

Convert Strings in Web Applications

When building web applications, you’ll often need to handle UTF-8 encoding for form submissions, API responses, and more. Let’s look at a practical example using Flask:

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/process', methods=['POST'])
def process_text():
    # Get text from the request
    text = request.form.get('text', '')

    # Ensure it's properly encoded as UTF-8
    utf8_text = text.encode('utf-8').decode('utf-8')

    # Process the text (e.g., count characters)
    char_count = len(utf8_text)

    return jsonify({
        'original_text': utf8_text,
        'character_count': char_count
    })

if __name__ == '__main__':
    app.run(debug=True)

This approach ensures that text inputs from web forms are correctly handled and safely processed in UTF-8 format.

Read How to Split a String into Equal Parts in Python?

Deal with File I/O and UTF-8

When reading from or writing to files, it is important to specify the encoding. Here’s how to properly handle UTF-8 encoding in file operations:

# Writing UTF-8 content to a file
def write_utf8_file(filename, content):
    with open(filename, 'w', encoding='utf-8') as f:
        f.write(content)

# Reading UTF-8 content from a file
def read_utf8_file(filename):
    with open(filename, 'r', encoding='utf-8') as f:
        return f.read()

# Example usage
sample_text = "This is a test with unicode: 你好, world!"
write_utf8_file("sample.txt", sample_text)
read_back = read_utf8_file("sample.txt")
print(read_back)  # Should match original text

This approach ensures that all text is properly encoded and decoded as UTF-8, preventing those frustrating encoding errors that can plague file I/O operations.

Check out How to Insert a Python Variable into a String?

Convert Strings in Python 2 vs Python 3

If you’re working with legacy code in Python 2, it’s important to note the differences in string handling:

# Python 2 (not runnable in Python 3)
# In Python 2, you'd use unicode objects
unicode_string = u"Hello, 世界!"
utf8_string = unicode_string.encode('utf-8')

# Python 3
# In Python 3, all strings are Unicode by default
normal_string = "Hello, 世界!"
utf8_bytes = normal_string.encode('utf-8')

While Python 2 reached its end of life in 2020, some organizations still maintain legacy codebases. If you’re working with Python 2 code, remember that strings and Unicode handling are significantly different from Python 3.

Solve Common UTF-8 Encoding Problems

Now I will explain some common UTF-8 encoding problems and solutions to them.

Problem 1: ASCII Decoding Errors

One common issue occurs when a Python server receives UTF-8 characters but tries to decode them as ASCII. Here’s how to fix it:

# Problem scenario
def handle_query_string(query):
    # In some environments, this might try to use ASCII by default
    # Leading to errors with non-ASCII characters

    # Solution: explicitly decode from UTF-8
    decoded_query = query.encode('latin1').decode('utf-8')
    return decoded_query

# Example
weird_query = "search=caf%C3%A9"  # URL-encoded 'café'
print(handle_query_string(weird_query))

Explicitly decoding input using UTF-8 helps avoid unexpected ASCII-related errors, especially when dealing with special or international characters.

Read How to Split a String by Index in Python?

Problem 2: Database Interactions

When working with databases, proper encoding is difficult. Here’s a pattern I use with SQLite:

import sqlite3

# Create a database connection that properly handles UTF-8
conn = sqlite3.connect('mydb.sqlite')

# Create a table with text that will store UTF-8
conn.execute('''
CREATE TABLE IF NOT EXISTS messages (
    id INTEGER PRIMARY KEY,
    content TEXT
)
''')

# Insert data with UTF-8 characters
message = "Hello from New York! 你好纽约！"
conn.execute("INSERT INTO messages (content) VALUES (?)", (message,))
conn.commit()

# Retrieve data
cursor = conn.execute("SELECT content FROM messages")
for row in cursor:
    print(row[0])

conn.close()

SQLite handles UTF-8 well by default in Python 3, but explicit encoding might be needed with other database systems.

Check out Convert Binary to Decimal in Python

Advanced UTF-8 Conversion Techniques in Python

Let me show you some advanced UTF-8 techniques for conversion in Python.

Detect Encoding

Sometimes, you may receive byte data from an external source—like a file or web response—without knowing its encoding. In such cases, the chardet library can help guess the encoding, allowing you to safely decode the content.

import chardet

# Some sample data with unknown encoding
mysterious_bytes = b'\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82'

# Detect the encoding
detection = chardet.detect(mysterious_bytes)
print(f"Detected encoding: {detection['encoding']} with confidence {detection['confidence']}")

# Decode using detected encoding
decoded = mysterious_bytes.decode(detection['encoding'])
print(f"Decoded text: {decoded}")

Using chardet makes your program more flexible when dealing with unknown or mixed-encoding data.

Work with Binary Data and Text

In Python, it’s important to understand the difference between binary data (bytes) and human-readable text.

# Text to bytes (encoding)
text = "Hello, world!"
bytes_data = text.encode('utf-8')
print(f"Text to bytes: {bytes_data}")

# Bytes to text (decoding)
decoded_text = bytes_data.decode('utf-8')
print(f"Bytes to text: {decoded_text}")

# Handling binary data that isn't text
binary_data = bytes([0x00, 0x01, 0x02, 0x03])
hex_representation = binary_data.hex()
print(f"Binary data as hex: {hex_representation}")

By using .encode() and .decode() methods, you can easily switch between text and binary formats, making your code robust when handling files, APIs, or hardware data streams.

Read Python Split Regex

In this article, I have explained how to convert string to UTF-8 in Python. I discussed three methods to accomplish this task such as using the .encode('utf-8), using the bytes() constructor, and .encode('utf-8) with error handling. I covered converting strings in web applications, dealing with file I/O and UTF-8, converting strings in Python 2 vs Python 3, Solving common UTF-8 encoding problems, and advanced UTF-8 conversion techniques in Python.

Related tutorials you may like to read:

• How to Split a Sentence into Words in Python?
• How to Split a String and Get the Last Element in Python?
• How to Split a String into an Array in Python?

Bijay Kumar

I am Bijay Kumar, a Microsoft MVP in SharePoint. Apart from SharePoint, I started working on Python, Machine learning, and artificial intelligence for the last 5 years. During this time I got expertise in various Python libraries also like Tkinter, Pandas, NumPy, Turtle, Django, Matplotlib, Tensorflow, Scipy, Scikit-Learn, etc… for various clients in the United States, Canada, the United Kingdom, Australia, New Zealand, etc. Check out my profile.

enjoysharepoint.com/