Data Engineer

DATB-PO : Databricks Performance Optimization

Course Duration

4 Hours

Delivery

ILT/VILT

Overview

Overview Course Curriculum Training Schedule Exam & Certification FAQs

In this course, you’ll learn how to optimize workloads and physical layout with Spark and Delta Lake and and analyze the Spark UI to assess performance and debug applications. We’ll cover topics like streaming, liquid clustering, data skipping, caching, photons, and more.

What You'll Learn

Spark Architecture
Designing the Foundation File Explosion
Code Optimization
Fine-Tuning - Choosing the Right Cluster

Who Should Attend

Data engineers, performance engineers and platform administrators focused on tuning and optimizing Apache Spark workloads on the Databricks platform.
Professionals addressing issues such as data skew, spills, shuffles, serialization inefficiencies, and sub-optimal file layouts with Delta Lake.
Individuals responsible for diagnosing performance bottlenecks using the Spark UI, selecting appropriate cluster instance types, and implementing optimisations such as data skipping, liquid clustering and caching.
Practitioners experienced with PySpark and Delta Lake who want to deepen their technical skillset to ensure high-performance, scalable data workflows.
Teams tasked with improving throughput, reducing cost and supporting production workloads by optimising cluster configurations, query performance and data layout in the Databricks Lakehouse.

Prerequisites

The content was developed for participants with these skills/knowledge/abilities:

Ability to perform basic code development tasks using Databricks (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)
Intermediate programming experience with PySpark
Extract data from a variety of file formats and data sources
Apply a number of common transformations to clean data
Reshape and manipulate complex data using advanced built-in functions
Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)

Learning Journey

Coming Soon...

Module 1. Spark Architecture

Spark UI Introduction

Module 2. Designing the Foundation File Explosion

Introduction to Designing Foundation
File Explosion
Data Skipping and Liquid Clustering
Lab: Data Skipping and Liquid Clustering

Module 3. Code Optimization

Skew
Shuffle
Spill
Lab: Exploding Join
Serialization
User-Defined Functions

Module 4. Fine-Tuning - Choosing the Right Cluster

Fine-Tuning: Choosing the Right Cluster
Pick the Best Instance Types

Frequently Asked Questions (FAQs)

None

Keep Exploring

Course Curriculum

Training Schedule

Exam & Certification

FAQs

Frequently Asked Questions

Improve yourself and your career by taking this course.

Enroll Now

More Courses By Databricks

Browse All Courses

IBM Analytics3_8g102g-ibm-security-guardium-data-protection-foundations-training

Data Engineer

Ready to Take Your Business from Great to Awesome?

Level-up by partnering with Trainocate. Get in touch today.

Name

Phone

I'm inquiring for

Myself

Company

Inquiry Details

By submitting this form, you consent to Trainocate processing your data to respond to your inquiry and provide you with relevant information about our training programs, including occasional emails with the latest news, exclusive events, and special offers.

You can unsubscribe from our marketing emails at any time. Our data handling practices are in accordance with our Privacy Policy.