As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
📘 Introduction When you run a PySpark job, Spark doesn’t immediately execute each transformation. Instead, it constructs something called a DAG (Directed Acyclic Graph) — a roadmap of all the operations that need to happen. This DAG is the heart of Spark’s execution engine. It tells Spark how your data...
📘 Introduction In this hands-on dbt tutorial, you'll learn how to overwrite project variables at runtime — a powerful feature that lets you dynamically change your dbt model behavior without modifying your code or dbt_project.yml. This is especially useful when you need to run the same transformation logic...
📘 Introduction When working with large datasets in PySpark, joins can easily become performance bottlenecks. This happens because Spark needs to shuffle data across the cluster to match rows between DataFrames — a costly operation when both datasets are big. If one of your DataFrames is small, though, there’s a faster...
📘Introduction In this tutorial, you'll learn how GraphQL and REST differ and why developers choose one over the other. Whether you’re building modern web apps or managing large datasets, understanding these differences is essential for efficient, scalable applications. 🟣 GraphQL GraphQL (Graph Query Language) is a query language...
📘 Introduction In this hands-on dbt tutorial, you'll learn how to use project variables to make your SQL models more dynamic, flexible, and reusable. Instead of hardcoding values directly into your SQL logic, dbt allows you to define variables in your dbt_project.yml file. These variables can be...
📘 Introduction In this hands-on dbt tutorial, you’ll learn how to use Jinja variables inside your dbt models to make your SQL code more dynamic, flexible, and reusable. Instead of hardcoding values directly into your SQL logic, Jinja allows you to define variables that can be adjusted at runtime or...
📘 Introduction In this hands-on dbt tutorial, you’ll learn step-by-step how to create a singular test to validate custom business rules using SQL. Unlike built-in generic tests (like unique or not_null), singular tests allow you to write fully customizable SQL logic to enforce strict data quality checks. ✅ Prerequisites ☑️ A...
📘 Introduction In this hands-on dbt tutorial, we’ll walk you step-by-step through how to set up data warehouse grants in dbt, ensuring the right users have access to the right layers and models. 📌 This is a must-know topic for the dbt Analytics Engineering Certification Exam, so mastering it now puts...
📘Introduction In this hands-on dbt tutorial, we’ll walk you step-by-step through how to make your SQL cleaner and more maintainable by using CTEs (Common Table Expressions) instead of deeply nested subqueries. ✅ Prerequisites ☑️ A dbt project set up ☑️ Source data loaded into your data warehouse ☑️ Source configurations defined in sources....