As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction FastAPI is a modern web framework that empowers developers to build web APIs quickly and efficiently using Python. A GET endpoint allows clients to retrieve data from a server. In a GET request, query parameters allow passing additional information to an endpoint via the URL's query string,...
Introduction FastAPI is a modern web framework that empowers developers to build web APIs quickly and efficiently using Python. A GET endpoint allows clients to retrieve data from a server. In a GET request, path parameters can be used to pass values directly in the URL to retrieve specific resources....
Introduction In data processing and analysis with PySpark, it's often important to know the structure of your data, such as the number of rows and columns in a DataFrame. This is crucial for various operations, including data validation, transformations, and general exploration. In this tutorial, we'll...
Introduction The Requests module offers a simple and user-friendly way of making HTTP requests in Python. Whether you are retrieving data from a web page, interacting with an API, or sending information to a server, the Requests module makes the task straightforward. In this tutorial, we will explore how to...
Introduction In this tutorial, we want to count the distinct values of a PySpark DataFrame column. In order to do this, we use the distinct().count() method and the countDistinct() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql....
Introduction When working with complex data, you often come across nested JSON structures that are stored in a single column. These nested structures can be difficult to analyze in their raw form, so flattening the JSON column becomes essential for data preparation. Dataflows in Microsoft Fabric provide a powerful way...
Introduction In distributed computing environments like Apache Spark, efficient data handling is critical for performance. One useful feature for optimizing computations is broadcast variables. Broadcast variables allow you to share large read-only data across all nodes in a Spark cluster without duplicating the data for each task. In this tutorial,...
Introduction Microsoft Fabric offers a gallery of sample datasets that can be used to perform and practice Real-Time Analytics. These sample data sets offer a great opportunity to gain experience and familiarize yourself with the technologies and services in the Real-Time Intelligence Experience. In this tutorial we will show you...
Introduction In this tutorial, we will explore how to read a Delta table into a PySpark DataFrame. Goal A Delta table stored in a lakehouse should be read into a PySpark DataFrame. Prequestisies ☑️ Notebook created We have already created the notebook "dlnerds_notebook". If you want to know...