PySpark - How to create and use Broadcast Variables

Introduction

In distributed computing environments like Apache Spark, efficient data handling is critical for performance. One useful feature for optimizing computations is broadcast variables. Broadcast variables allow you to share large read-only data across all nodes in a Spark cluster without duplicating the data for each task. In this tutorial, we'll explore what broadcast variables are, why they're useful, and how to create and use them in PySpark.

What Are Broadcast Variables?

In Spark, each task running on a worker node gets a copy of the variables used in the task. For large datasets, this can cause performance issues, as the data is replicated across all worker nodes. This duplication can be avoided by using broadcast variables, which ensure that the data is sent to each worker only once. Each worker can then read from the broadcasted data locally, reducing network overhead and memory consumption.

Broadcast variables are particularly useful when:

You have large lookup tables, configuration settings, or static data that need to be used in multiple tasks.
You want to avoid sending the same data repeatedly across the network.
You want to share read-only data among all tasks without recalculating it.

Benefits of Using Broadcast Variables

Efficiency: Reduce network I/O and memory overhead by broadcasting large datasets to all nodes instead of sending them with each task.
Consistency: Ensure that the same data is available to all tasks consistently, without recalculating it repeatedly.
Performance: Broadcast variables allow the cluster to operate more efficiently by making large datasets accessible without unnecessary shuffling or replication.

Import Libraries

First, we import the following python modules:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

Create SparkSession

Before we can work with Pyspark, we need to create a SparkSession. A SparkSession is the entry point into all functionalities of Spark.

In order to create a basic SparkSession programmatically, we use the following command:

spark = SparkSession \
    .builder \
    .appName("Python PySpark Example") \
    .getOrCreate()

Create PySpark DataFrame

Next, we create the PySpark DataFrame with some example data from a list. To do this, we use the method createDataFrame() and pass the data and the column names as arguments.

column_names = ["language", "framework", "users"]
data = [
    ("Python", "Django", 20000), 
    ("Python", "FastAPI", 9000), 
    ("Java", "Spring", 7000), 
    ("JavaScript", "ReactJS", 5000)
]
df = spark.createDataFrame(data, column_names)
df.show()