As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction In this tutorial, we will explore how to read a Delta table into a PySpark DataFrame. Goal A Delta table stored in a lakehouse should be read into a PySpark DataFrame. Prequestisies ☑️ Notebook created We have already created the notebook "dlnerds_notebook". If you want to know...
Introduction In this tutorial, we will walk through the steps required to write a PySpark DataFrame to a Delta table in Microsoft Fabric. We'll start by creating an example DataFrame, then demonstrate how to write it to a Delta table. Goal An example PySpark DataFrame should be created...
Introduction In Microsoft Fabric, notebooks can interact very closely with lakehouses. In this tutorial, we will explain step-by-step how to connect a lakehouse and a notebook in Microsoft Fabric. Goal A lakehouse and a notebook should be connected. Prequestisies ☑️ Notebook created We have already created the notebook "dlnerds_notebook&...
Introduction In this tutorial, we will show you how to get a randomly sampled subset of a PySpark DataFrame. In order to do this, we will use the sample() function of PySpark. What is the sample() Function? The sample() function in PySpark is used to create a new DataFrame by...
Introduction In Microsoft Fabric, there are several activities that can be added to a data pipeline. One import activity is the Notebook activity. In this tutorial, we will explain step-by-step how to add a Notebook activity to a data pipeline using the Data Factory user interface. Goal A CSV File...
Introduction In Microsoft Fabric, there are several activities that can be added to a data pipeline. One import activity is the Copy activity. In this tutorial, we will explain step-by-step how to add a Copy activity to a data pipeline using the Data Factory user interface. Goal A CSV File...
Introduction In this tutorial, we will show you how to concatenate multiple string columns of a PySpark DataFrame into a single column. In order to do this, we will use the functions concat() and concat_ws() of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a notebook. In this tutorial, we will explain step-by-step how to ingest data into a Fabric lakehouse using a notebook....
Introduction In this tutorial, we will show you how to remove the leading and trailing whitespaces from a string column of a PySpark DataFrame. In order to do this, we will use the functions trim(), ltrim() and rtrim() of PySpark. Import Libraries First, we import the following python modules: from...