Introduction In distributed computing environments like Apache Spark, efficient data handling is critical for performance. One useful feature for optimizing computations is broadcast variables. Broadcast variables allow you to share large read-only data across all nodes in a Spark cluster without duplicating the data for each task. In this tutorial,...
Introduction Installing Python is a straightforward process that gives you access to one of the most powerful programming languages available today. Whether you're running macOS, Windows or Linux, Python can be easily installed through official channels so you can start programming quickly. This tutorial will show you how...
Introduction In this tutorial, we will explore how to read a Delta table into a PySpark DataFrame. Goal A Delta table stored in a lakehouse should be read into a PySpark DataFrame. Prequestisies ☑️ Notebook created We have already created the notebook "dlnerds_notebook". If you want to know...
Introduction In this tutorial, we will walk through the steps required to write a PySpark DataFrame to a Delta table in Microsoft Fabric. We'll start by creating an example DataFrame, then demonstrate how to write it to a Delta table. Goal An example PySpark DataFrame should be created...
Introduction In this tutorial, we will show you how to get a randomly sampled subset of a PySpark DataFrame. In order to do this, we will use the sample() function of PySpark. What is the sample() Function? The sample() function in PySpark is used to create a new DataFrame by...
Introduction In today's data-driven world, understanding text data is crucial across various domains, from data analysis to engineering and architecture. However, dealing with text data often requires converting it into numerical representations for machine learning models to process efficiently. This is where embedding vectors come into play, offering...
Introduction In this tutorial, we will show you how to concatenate multiple string columns of a PySpark DataFrame into a single column. In order to do this, we will use the functions concat() and concat_ws() of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql...
Introduction In this tutorial, we will show you how to remove the leading and trailing whitespaces from a string column of a PySpark DataFrame. In order to do this, we will use the functions trim(), ltrim() and rtrim() of PySpark. Import Libraries First, we import the following python modules: from...
Introduction In this tutorial, we will explain step-by-step how o read an Excel file into a PySpark DataFrame in Databricks. Configure Cluster First, install on a Databricks cluster the spark-excel library (also referred as com.crealytics.spark.excel). To do this, select your Databricks cluster in the "Compute"...