As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction When working with data in Python, two of the most popular tools available are Pandas and PySpark. Both offer powerful data manipulation capabilities but are designed for different use cases. Pandas is well-suited for small to moderately large datasets, offering an intuitive API for data analysis. PySpark, on the...
Introduction Microsoft Fabric provides robust security controls for managing data access. For the DP-600 certification exam, mastering file-level access control is essential to restrict access to specific folders within a Fabric Lakehouse. In this tutorial, you'll learn how to implement file-level access control using OneLake data access roles...
Introduction Microsoft Fabric provides robust security controls for managing data access. For the DP-600 certification exam, mastering Column-Level Security (CLS) is essential to restrict access to specific columns within a table in a Microsoft Fabric Warehouse. In this tutorial, you'll learn how to implement column-level access control to...
Introduction Microsoft Fabric provides robust security controls for managing data access. For the DP-600 certification exam, mastering Object-Level Security (OLS) is essential to restrict access to specific objects in a Microsoft Fabric Warehouse, such as schemas, tables, and views. In this tutorial, you'll learn how to implement object-level...
Introduction Microsoft Fabric provides powerful security features to control data access at a granular level. In the DP-600 certification exam, mastering how to implement row-level access control is essential for ensuring users see only the data relevant to them while maintaining governance and compliance. In this tutorial, you'll...
Introduction When working with PySpark DataFrames, handling missing or empty values is a common task in data preprocessing. In many cases, empty strings ("") should be treated as null values for better compatibility with Spark operations, such as filtering, aggregations, and machine learning workflows. In this tutorial, we’ll...
Introduction In dbt, one of the most essential commands is dbt test. In this tutorial, we’ll dive into the dbt test command, exploring its syntax, functionality, and practical usage. This tutorial is also a valuable resource for those studying for the dbt Analytics Engineering Certification Exam. Explanation of dbt...
Introduction When working with data in PySpark, you might often encounter scenarios where a single column contains multiple pieces of information, such as a combination of names, categories, or attributes. In such cases, it is essential to split these values into separate columns for better data organization and analysis. In...
Introduction Data Transformation is a crucial step in any data project, ensuring that raw data is turned into accurate, consistent, and usable data. One tool that has become especially valuable in this process is dbt (Data Build Tool). In this tutorial, we’ll explore what dbt is, how it simplifies...