As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction In this tutorial, we want to drop duplicates from a PySpark DataFrame. In order to do this, we use the the dropDuplicates() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need...
Introduction In this tutorial, we want to filter specific rows from a PySpark DataFrame based on specific conditions. In order to do this, we use the the filter() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can...
Introduction In this tutorial, we want to sort a PySpark DataFrame by specific columns. In order to do this, we use the the orderBy() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we...
Introduction In this tutorial, we want to drop columns from a PySpark DataFrame. In order to do this, we use the the drop() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need...
Introduction In this tutorial, we want to add columns to a PySpark DataFrame. In order to do this, we use the the withColumn() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions import lit, col, sum, when from...
Introduction In this tutorial, we want to select specific columns from a PySpark DataFrame. In order to do this, we use the select() method of PySpark in different variants. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions import col Create...
Introduction In this tutorial, we want to rename a PySpark DataFrame column. In order to do this, we use the the withColumnRenamed() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need to...
Introduction In this tutorial, we want to convert a PySpark DataFrame into a Pandas DataFrame with a specific schema. In order to do this, we use the the toPandas() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we...
Introduction In this tutorial, we want to convert a Pandas DataFrame into a PySpark DataFrame with a specific schema. In order to do this, we use the the createDataFrame() function of PySpark. Import Libraries First, we import the following python modules: import pandas as pd from pyspark.sql import...