Data Engineer

Data Engineer

As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.

97 posts
Overview of essential dbt commands

Overview of essential dbt commands

Introduction dbt empowers data professionals to transform raw data into clean, analytics-ready datasets. Mastering its essential commands is key to unlocking its full potential. In this tutorial, we’ll explore the key dbt commands, providing both the syntax and a short description for each command. This tutorial is also a...

PySpark - Extract a Substring from a DataFrame Column
Academy Membership PySparkPython

PySpark - Extract a Substring from a DataFrame Column

Introduction When dealing with large datasets in PySpark, it's common to encounter situations where you need to manipulate string data within your DataFrame columns. One such common operation is extracting a portion of a string—also known as a substring—from a column. In this tutorial, we will...

PySpark - Count Rows and Columns of a DataFrame
Academy Membership PySparkPython

PySpark - Count Rows and Columns of a DataFrame

Introduction In data processing and analysis with PySpark, it's often important to know the structure of your data, such as the number of rows and columns in a DataFrame. This is crucial for various operations, including data validation, transformations, and general exploration. In this tutorial, we'll...

PySpark - Count Distinct Values of a DataFrame Column
Academy Membership PySparkPython

PySpark - Count Distinct Values of a DataFrame Column

Introduction In this tutorial, we want to count the distinct values of a PySpark DataFrame column. In order to do this, we use the distinct().count() method and the  countDistinct() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql....

You’ve successfully subscribed to Deep Learning Nerds | The ultimate Learning Platform for AI and Data Science
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Success! Your email is updated.
Your link has expired
Success! Check your email for magic link to sign-in.