Applying a Function to Each Partition in a DataFrame - .foreachPartition() ​

Overview ​

The foreachPartition() function is used to apply a function to each partition of a DataFrame. It is a higher-order function that allows you to perform custom operations on batches of data within each partition, rather than processing individual rows. The foreachPartition() function does not return a new DataFrame; instead, it is typically used for side effects, such as writing data to an external storage system, establishing a connection to an external database, or performing bulk operations on the data.

Example ​

from pyspark.sql import SparkSession

# Create a SparkSession (if not already created)
spark = SparkSession.builder.appName("ForeachPartitionExample").getOrCreate()

# Sample data as a list of dictionaries
data = [{"name": "Alice", "age": 30},
        {"name": "Bob", "age": 25},
        {"name": "Charlie", "age": 35},
        {"name": "David", "age": 28},
        {"name": "Eva", "age": 32}]

# Create a DataFrame
df = spark.createDataFrame(data)


# Define a function to apply to each partition
def process_partition(iter):


# This function will be executed for each partition of the DataFrame
for row in iter:
    name = row["name"]
age = row["age"]
print(f"Processing row: Name: {name}, Age: {age}")

# Apply the function using 'foreachPartition()'
df.foreachPartition(process_partition)

Output:

Processing row: Name: Alice, Age: 30
Processing row: Name: Bob, Age: 25
Processing row: Name: Charlie, Age: 35
Processing row: Name: David, Age: 28
Processing row: Name: Eva, Age: 32

In this example, the foreachPartition() function is used to apply the process_partition() function to each partition of the DataFrame. The function processes rows in batches within each partition, which can be more efficient than processing individual rows one by one. The foreachPartition() function is useful for tasks that involve side effects or bulk operations on data, as it allows you to work with data in larger batches within each partition.