How to Convert a DataFrame to a Nested Dictionary in Python [4 Methods]

Are you interested in converting a dataframe to a nested dictionary? In this Pandas article, I will explain how to convert a dataframe to a nested dictionary in Python using different methods with examples.

I will also explain how to convert a dataframe to a dictionary with multiple values in Python.

To convert a DataFrame to a nested dictionary in Python, one can utilize several methods like to_dict() with the orient parameter for simple column-to-dictionary conversions, groupby() and apply() for aggregating data into structured formats, list comprehension for creating custom dictionary layouts, and the json module for converting to and from JSON format. Also, we have used groupby method along with to_dict function to convert a dataframe to a dictionary with multiple values in Python.

Convert a DataFrame to a Nested Dictionary in Python

There are four different ways to convert a dataframe to a nested dictionary in Python:

  1. Using to_dict() with orient parameter
  2. Using groupby() and apply()
  3. Using List Comprehension
  4. Using json module

Let’s see them one by one using some demonstrative examples:

Note: To confirm that the output we are getting is a nested dictionary we can use the isinstance() method, like this:

print(isinstance(nested_dict, dict) and all(isinstance(item, dict) for item in nested_dict.values()))

1. Pandas dataframe to nested dictionary using to_dict() with orient parameter

The to_dict() with orient parameter method is the most straightforward approach provided by Pandas. This converts a dataframe to a nested dictionary in Python, where we can specify the structure of the output dictionary using the orient parameter. It’s great for simple, direct conversions.

import pandas as pd

df = pd.DataFrame({
    'State': ['California', 'Texas', 'Florida'],
    'Capital': ['Sacramento', 'Austin', 'Tallahassee']
})

nested_dict = df.to_dict(orient='index')
print(nested_dict)

print(isinstance(nested_dict, dict) and all(isinstance(item, dict) for item in nested_dict.values()))

Output:

{0: {'State': 'California', 'Capital': 'Sacramento'}, 1: {'State': 'Texas', 'Capital': 'Austin'}, 2: {'State': 'Florida', 'Capital': 'Tallahassee'}}
True

Below is a screenshot taken following the implementation of the code in the Pycharm editor.

pandas dataframe to dict

2. Dataframe to nested dictionary using groupby() and apply()

The groupby() and apply() methods are ideal for more complex DataFrames where we need to group data based on certain criteria before converting a dataframe to a nested dictionary in Python. It allows for customized grouping and structuring, making it useful for specialized needs.

import pandas as pd

df = pd.DataFrame({
    'State': ['California', 'California'],
    'City': ['Los Angeles', 'San Francisco'],
    'Population': [3979576, 883305]
})

grouped_dict = df.groupby('State').apply(lambda x: x.to_dict(orient='records')).to_dict()
print(grouped_dict)

print(isinstance(grouped_dict, dict) and all(isinstance(item, list) for item in grouped_dict.values()))

Output:

{'California': [{'State': 'California', 'City': 'Los Angeles', 'Population': 3979576}, {'State': 'California', 'City': 'San Francisco', 'Population': 883305}]}
True

Following the execution of the code in Pycharm, the resulting screenshot is displayed below.

convert dataframe to dictionary with one column as key and other as value in Python

3. Dataframe to dictionary using list comprehension

This method involves manually constructing through a dataframe to a nested dictionary in Python using list comprehension. It offers high flexibility and control over the resulting dictionary’s structure, especially when the built-in methods don’t fit the specific structural requirements.

import pandas as pd

df = pd.DataFrame({
    'State': ['California', 'Texas', 'Florida'],
    'Abbreviation': ['CA', 'TX', 'FL'],
    'Capital': ['Sacramento', 'Austin', 'Tallahassee']
})

nested_dict = {row['Abbreviation']: {'Capital': row['Capital']} for _, row in df.iterrows()}
print(nested_dict)

print(isinstance(nested_dict, dict) and all(isinstance(item, dict) for item in nested_dict.values()))

Output:

{'CA': {'Capital': 'Sacramento'}, 'TX': {'Capital': 'Austin'}, 'FL': {'Capital': 'Tallahassee'}}
True

Following the execution of the code within the Pycharm editor, the subsequent screenshot displays the output.

Convert a DataFrame to a Nested Dictionary in Python

4. Python to dataframe nested dictionary using json module

This method involves converting the DataFrame to a JSON string using the to_json() method and then parsing it back into a dictionary with the json module. It’s a two-step process that is useful when dealing with data interchange formats or when needing a JSON representation.

import json
import pandas as pd

df = pd.DataFrame({
    'Park': ['Yellowstone', 'Yosemite', 'Grand Canyon'],
    'State': ['Wyoming', 'California', 'Arizona']
})

json_str = df.to_json(orient='records')
nested_dict = json.loads(json_str)
print(nested_dict)

print(isinstance(nested_dict, list) and all(isinstance(item, dict) for item in nested_dict))

Output:

[{'Park': 'Yellowstone', 'State': 'Wyoming'}, {'Park': 'Yosemite', 'State': 'California'}, {'Park': 'Grand Canyon', 'State': 'Arizona'}]
True

The screenshot provided below illustrates the results post-execution of the code in the Pycharm editor.

convert dataframe to nested dictionary in Python pandas

Convert a dataframe to a dictionary with multiple values in Python

Converting a DataFrame to a dictionary with multiple values in Python can be done using the Pandas library’s groupby method along with to_dict function. This approach is particularly useful when we want to group by a specific column and then map each unique value in that column to a list of associated values from other columns.

Let’s demonstrate this with an example:

import pandas as pd

df = pd.DataFrame({
    'State': ['California', 'California', 'Texas', 'Texas', 'Florida'],
    'City': ['Los Angeles', 'San Francisco', 'Houston', 'Austin', 'Miami']
})

state_cities_dict = df.groupby('State')['City'].apply(list).to_dict()
print(state_cities_dict)

print(type(state_cities_dict))

Output:

{'California': ['Los Angeles', 'San Francisco'], 'Florida': ['Miami'], 'Texas': ['Houston', 'Austin']}
<class 'dict'>

Upon running the code in Pycharm, the resulting output is captured in the screenshot presented below.

python pandas convert dataframe to dictionary with multiple values

Conclusion

Here, I have explained various methods to convert a DataFrame into a nested dictionary in Python, using practical examples. These methods include the use of to_dict() with the orient parameter for straightforward conversions, groupby() and apply() for complex group-based structures, list comprehension for custom dictionary formats, and the json module for JSON-oriented transformations.

I have also explained how to convert a dataframe to a dictionary with multiple values in Python using groupby method along with to_dict function.

Each technique offers unique advantages, catering to different data manipulation needs and scenarios in Python.

You may also like to read: