ITVersity, Inc.

Getting Started with Databricks Community Edition

Setup and Validate Spark Cluster using Databricks Community Edition

Overview of Databricks Notebooks

Overview of Magic Commands in Databricks Notebooks

Recreating Clusters using Databricks Community Edition

Limitations using Databricks Community Edition

Introduction to Basics of Spark and Spark APIs

Overview of Spark and Distributed Computing

Review Data Sets for Word Count

Quick Revision of Python Collections for Spark RDDs

YouTube - Python for Data Engineering Courses

Overview of Spark RDDs

Overview of Spark Dataframes and Datasets

YouTube - Recommend other videos in description

Overview of APIs to create Spark RDDs

Read Text Data from Files into Spark RDDs

YouTube - Review Videos

Overview of APIs or Functions on Spark RDDs

Previewing the Data in Spark RDDs using Actions

Important Concepts related to Transformations in Spark

Filter Data in Spark RDDs

Row Level Transformations using map on Spark RDDs

Understand the requirements for Word Count

Recap of String Functions in Python

YouTube - Course Recommendations

Overview of Aggregations

Using reduce function on Spark RDDs

Flatten lists in Spark RDDs using flatMap

Understand Concept of Shuffling with Example

Word Count using reduceByKey on Spark RDDs

Sort Word Count Results using sortByKey on Spark RDDs

Save Spark RDDs to Text Files

Final Logic to perform Word Count using Spark RDDs

Review Spark Driver logs and Spark UI

Overview of Lazy Evaluation and DAGs in Spark

Next Steps for Data Engineering using Spark

YouTube - Promote Guided Programs

Review Databricks Datasets using fs commands.cmproj

Getting Started with Spark Dataframes using Pyspark.cmproj

Overview of Apache Spark.cmproj

Compute Item Revenue using Pyspark.cmproj

Aggregations using Pyspark.cmproj

Restart Databricks Cluster using Community Edition.cmproj

DBFS Commands to Manage Files and Folders.cmproj

Writing Processed Data to DBFS using Spark.cmproj

Getting Started with Spark SQL using Databricks Community Edition.cmproj

Review Files and Data using fs Commands.cmproj

Create External Table using Spark SQL.cmproj

Review Online Retail Data Set to Practice Spark SQL.cmproj

Compute Item Revenue using Spark SQL.cmproj

Compute Invoice Revenue using GROUP BY in Spark SQL.cmproj

Revision of SQL Syntax.cmproj

Views and CTEs in Spark SQL.cmproj

Create Spark Metastore Table for Processed Data.cmproj

Populate Data into Spark Tables using INSERT.cmproj

Data Engineering Pipeline using Spark SQL.cmproj

Introduction to Word Count using Spark Dataframe APIs

Preview Data in Spark Dataframes

Using split and explode on Spark Dataframes

Revision of Standard Transformations

Using groupBy and orderBy for Aggregations using Spark

Final Code for Word Count using Spark Dataframe APIs

Review Driver Logs and Spark UI

Conclusion and Next Steps

Introduction to Word Count using Spark SQL

Create External Table using Spark SQL

Standard Transformations and Functions in Spark SQL

Using split and explode in Spark SQL

GROUP BY and CTE using Spark SQL

Write Data into DBFS using Spark SQL

Final Code for Word Count using Spark SQL

Conclusion and Next Steps