Data Engineer

Data Engineer

As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.

96 posts
PySpark - Extract a Substring from a DataFrame Column
Academy Membership PySparkPython

PySpark - Extract a Substring from a DataFrame Column

Introduction When dealing with large datasets in PySpark, it's common to encounter situations where you need to manipulate string data within your DataFrame columns. One such common operation is extracting a portion of a string—also known as a substring—from a column. In this tutorial, we will...

PySpark - Count Rows and Columns of a DataFrame
Academy Membership PySparkPython

PySpark - Count Rows and Columns of a DataFrame

Introduction In data processing and analysis with PySpark, it's often important to know the structure of your data, such as the number of rows and columns in a DataFrame. This is crucial for various operations, including data validation, transformations, and general exploration. In this tutorial, we'll...

PySpark - Count Distinct Values of a DataFrame Column
Academy Membership PySparkPython

PySpark - Count Distinct Values of a DataFrame Column

Introduction In this tutorial, we want to count the distinct values of a PySpark DataFrame column. In order to do this, we use the distinct().count() method and the  countDistinct() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql....

PySpark - How to create and use Broadcast Variables
Academy Membership PySparkPython

PySpark - How to create and use Broadcast Variables

Introduction In distributed computing environments like Apache Spark, efficient data handling is critical for performance. One useful feature for optimizing computations is broadcast variables. Broadcast variables allow you to share large read-only data across all nodes in a Spark cluster without duplicating the data for each task. In this tutorial,...

You’ve successfully subscribed to Deep Learning Nerds | The ultimate Learning Platform for AI and Data Science
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Success! Your email is updated.
Your link has expired
Success! Check your email for magic link to sign-in.