Data Engineer

DATB-SLFDP : Databricks Streaming and Lakeflow Declarative Pipelines

Course Duration

3 Hours

Delivery

ILT/VILT

Overview

Overview Course Curriculum Training Schedule Exam & Certification FAQs

This course provides a comprehensive understanding of Spark Structured Streaming and Delta Lake, including computation models, configuration for streaming read, and maintaining data quality in a streaming environment.

What You'll Learn

Introduction to Streaming
Aggregations, Time Windows, Watermarks
Streaming Joins (Optional)
Streaming ETL Patterns with Lakeflow Declarative Pipelines

Who Should Attend

Data engineers or real-time analytics developers building streaming pipelines on the Databricks Lakehouse Platform.
Professionals responsible for ingesting, transforming and delivering continuous data flows using Spark Structured Streaming, Lakeflow Declarative Pipelines, streaming tables and materialised views.
Individuals applying concepts like event-time processing, watermarking, late data handling, incremental updates and Change Data Capture (CDC) in their pipelines.
Practitioners with experience in SQL, Python and working with the Databricks workspace who want to deepen their skills in declarative streaming pipeline design and orchestration.
Teams transitioning from batch-only workflows to hybrid or fully streaming architectures and aiming to build maintainable, observable, scalable streaming pipelines using the Lakeflow framework on Databricks.

Prerequisites

The content was developed for participants with these skills/knowledge/abilities:

Ability to perform basic code development tasks using the Databricks Data Engineering and Data Science workspace (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)
Intermediate programming experience with PySpark
Extract data from a variety of file formats and data sources
Apply a number of common transformations to clean data
Reshape and manipulate complex data using advanced built-in functions
Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)
Beginner experience configuring and scheduling data pipelines using the Lakeflow Spark Declarative Pipelines UI
Beginner experience defining Lakeflow Spark Declarative Pipelines using PySpark
Ingest and process data using Auto Loader and PySpark syntax
Process Change Data Capture feeds with APPLY CHANGES INTO syntax
Review pipeline event logs and results to troubleshoot Declarative Pipeline syntax

Learning Journey

Coming Soon...

Module 1. Introduction to Streaming

Streaming Data Concepts
Introduction to Structured Streaming
Reading from a Streaming Query
Streaming from Delta Lake
Streaming Query

Module 2. Aggregations, Time Windows, Watermarks

Aggregations, Time Windows, Watermarks
Event Time + Aggregatios over Time Windows
Stream Aggregation
Windowed Aggregation with Watermark

Module 3. Streaming Joins (Optional)

Streaming Joins (Optional)
Stream/Stream Joins (Options)

Module 4. Streaming ETL Patterns with Lakeflow Declarative Pipelines

Data Ingestion Patterns
Auto Load to Bronze
Stream from Multiplex Bronze
Data Quality Enforcement
Streaming ETL

Frequently Asked Questions (FAQs)

None

Keep Exploring

Course Curriculum

Training Schedule

Exam & Certification

FAQs

Frequently Asked Questions

Improve yourself and your career by taking this course.

Enroll Now

More Courses By Databricks

Browse All Courses

IBM Analytics3_8g102g-ibm-security-guardium-data-protection-foundations-training

Data Engineer

Ready to Take Your Business from Great to Awesome?

Level-up by partnering with Trainocate. Get in touch today.

Name

Phone

I'm inquiring for

Myself

Company

Inquiry Details

By submitting this form, you consent to Trainocate processing your data to respond to your inquiry and provide you with relevant information about our training programs, including occasional emails with the latest news, exclusive events, and special offers.

You can unsubscribe from our marketing emails at any time. Our data handling practices are in accordance with our Privacy Policy.