As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction In Microsoft Fabric, there are several activities that can be added to a data pipeline. One import activity is the Copy activity. In this tutorial, we will explain step-by-step how to add a Copy activity to a data pipeline using the Data Factory user interface. Goal A CSV File...
Introduction In this tutorial, we will show you how to concatenate multiple string columns of a PySpark DataFrame into a single column. In order to do this, we will use the functions concat() and concat_ws() of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a notebook. In this tutorial, we will explain step-by-step how to ingest data into a Fabric lakehouse using a notebook....
Introduction In this tutorial, we will show you how to remove the leading and trailing whitespaces from a string column of a PySpark DataFrame. In order to do this, we will use the functions trim(), ltrim() and rtrim() of PySpark. Import Libraries First, we import the following python modules: from...
Introduction In this tutorial, we will explain step-by-step how o read an Excel file into a PySpark DataFrame in Databricks. Configure Cluster First, install on a Databricks cluster the spark-excel library (also referred as com.crealytics.spark.excel). To do this, select your Databricks cluster in the "Compute"...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a dataflow. In this tutorial, we will explain step-by-step how to ingest data into a Fabric warehouse using a dataflow....
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a data pipeline. In this tutorial, we will explain step-by-step how to ingest data into a Fabric warehouse using a...
Introduction One fundamental part of Microsoft Fabric is transforming data. Whether filtering, joining, merging or grouping data, there are several options available in Fabric to perform these operations. In this tutorial, we will explain step-by-step how to group data and apply an aggregation function using a dataflow. Goal A delta...
Introduction In this tutorial, we will show you how to group and concatenate strings in a PySpark DataFrame. In order to do this, we will use the groupBy() method in combination with the functions concat_ws(), collect_list() and array_distinct() of PySpark. Import Libraries First, we import the following...