How to Iterate Over Rows in a DataFrame in Pandas

Python Pandas Dataframe Loops

Pandas is a popular Python library used for data manipulation and analysis. One common task in data analysis is to iterate over the rows of a DataFrame to perform specific operations or calculations. In this article, we will explore different methods to iterate over rows in a Pandas DataFrame and understand how to access the values in each row by column names.

Method 1: Using the iterrows() Method

The iterrows() method in Pandas allows us to iterate over the rows of a DataFrame. This method returns an iterator that yields both the index of each row and a Series object containing the row values.

Example:


import pandas as pd

# Create a DataFrame
data = {'c1': [10, 11, 12], 'c2': [100, 110, 120]}
df = pd.DataFrame(data)

# Iterate over the rows using the iterrows() method
for index, row in df.iterrows():
    print(row['c1'], row['c2'])

In this example, we create a DataFrame called df with two columns 'c1' and 'c2'. We then use the iterrows() method to iterate over each row of the DataFrame. Inside the loop, we access the values in each row by using the column names as keys of the row object.

Note that the values returned by the iterrows() method are in the form of a Series object, which is similar to a Pandas DataFrame but contains only a single row of data. Therefore, we can access the values within each row using the column names as keys of the row Series.

Method 2: Using the itertuples() Method

The itertuples() method is another efficient way to iterate over the rows of a DataFrame in Pandas. This method returns an iterator that yields a named tuple for each row, where each attribute of the named tuple corresponds to a column in the DataFrame.

Example:


import pandas as pd

# Create a DataFrame
data = {'c1': [10, 11, 12], 'c2': [100, 110, 120]}
df = pd.DataFrame(data)

# Iterate over the rows using the itertuples() method
for row in df.itertuples():
    print(row.c1, row.c2)

In this example, we create a DataFrame called df with the same columns as the previous example. We then use the itertuples() method to iterate over each row of the DataFrame. Inside the loop, we access the values in each row by using the attributes of the named tuple, which correspond to the column names.

The itertuples() method returns a named tuple with the following structure: namedtuple('Pandas', ['Index', 'c1', 'c2', ...]). The first attribute, Index, represents the index of each row, and the subsequent attributes correspond to the column names in the DataFrame.

Although the itertuples() method may be slightly faster than the iterrows() method, it provides less flexibility, as the values in each row can only be accessed by their attribute names instead of the column names.

Method 3: Using the apply() Method

The apply() method in Pandas can also be used to iterate over the rows of a DataFrame. This method applies a function to each row or column of the DataFrame and returns a new Series or DataFrame, depending on the function used.

Example 1: Using a Lambda Function


import pandas as pd

# Create a DataFrame
data = {'c1': [10, 11, 12], 'c2': [100, 110, 120]}
df = pd.DataFrame(data)

# Iterate over the rows using the apply() method with a lambda function
df.apply(lambda row: print(row['c1'], row['c2']), axis=1)

In this example, we create a DataFrame called df with the same columns as the previous examples. We then use the apply() method with a lambda function to iterate over each row of the DataFrame. Inside the lambda function, we access the values in each row using the column names as keys of the row object.

Note that we need to specify the axis=1 parameter in order to apply the lambda function to each row instead of each column.

Example 2: Using a Custom Function


import pandas as pd

# Create a DataFrame
data = {'c1': [10, 11, 12], 'c2': [100, 110, 120]}
df = pd.DataFrame(data)

# Define a custom function to be applied to each row
def print_row_values(row):
    print(row['c1'], row['c2'])

# Iterate over the rows using the apply() method with a custom function
df.apply(print_row_values, axis=1)

In this example, we create a custom function called print_row_values(), which takes a row as input and prints the values of 'c1' and 'c2'. We then use the apply() method with the custom function to iterate over each row of the DataFrame.

Using the apply() method with a custom function provides more flexibility, as we can define any desired operations or calculations within the custom function.

Choosing the Right Method

When choosing a method to iterate over rows in a Pandas DataFrame, consider the following factors:

  • Performance: itertuples() is the fastest method, followed by iterrows() and then apply(). However, the performance difference may be negligible for small to medium-sized DataFrames.
  • Flexibility: iterrows() and apply() provide more flexibility in accessing and manipulating the values in each row, while itertuples() requires using the attribute names of the named tuple.
  • Memory Usage: itertuples() and apply() may use more memory compared to iterrows(), as they create additional data structures.

It is generally recommended to use the most appropriate method based on the specific requirements of your analysis.

Related Questions