Data Engineering Pipeline Management with Apache Airflow

سرفصل های دوره

Take a deeper dive into data engineering pipeline management using Apache Airflow. In this course, certified Google cloud architect and data engineer Janani Ravi guides you through using Apache Airflow to complete your data engineering pipeline management workflows. Learn how to work with role-based access control, including creating users with different roles, executing a branching DAG and a SQL DAG, recalling actions that users with different roles can perform, and more. Go over how to manage SLAs and schedule DAGs with datasets. Find out how to work with AirFlow plugins and explore the CSV reader plugin. Plus, discover how to scale Apache Airflow, set up a data transformation pipeline, execute tasks, and more.

This course was created by Janani Ravi. We are pleased to host this training in our library.

01 - Introduction

01 - Features for data engineering pipeline management

02 - 1. Working with Role-Based Access Control

01 - Prerequisites

02 - Quick install overview

03 - Creating an admin user and exploring roles

04 - Creating users with different roles

05 - Executing a simple branching DAG

06 - Executing a simple SQL DAG

07 - The public and viewer roles

08 - The user role

09 - The op role

10 - Actions, resources, and permissions

11 - Adding permissions to the public role

12 - Creating and configuring a custom role

03 - 2. Managing SLAs

01 - Configuring emails for SLA management

02 - Configuring task-level SLAs

03 - Triggering and viewing SLA misses

04 - Configuring DAG-level SLAs

05 - Configuring DAG failed action

04 - 3. Scheduling DAGs with Datasets

01 - Dataset producer pipeline

02 - Dataset consumer pipeline

03 - Data-aware scheduling

04 - Purchases producer pipeline and join pipeline

05 - Data-aware scheduling with multiple datasets

05 - 4. Working with Airflow Plugins

01 - Introducing plugins

02 - Adding menu items using plugins

03 - Exploring the CSV reader plugin

04 - Implementing the CSV reader plugin

06 - 5. Scaling Airflow

01 - Scaling Apache Airflow

02 - Basic setup for the transformation pipeline

03 - DAG for the transformation pipeline

04 - Install RabbitMQ on macOS and Linux

05 - Set up an admin user for RabbitMQ

06 - Configuring the CeleryExecutor for Airflow

07 - Executing tasks on a single Celery worker

08 - Executing tasks on multiple Celery workers

09 - Assigning tasks to queues

07 - Conclusion

01 - Summary and next steps

139,000 تومان

خرید اشتراک افزودن به سبد خرید

خرید دانلودی فوری

در این روش نیاز به افزودن محصول به سبد خرید و تکمیل اطلاعات نیست و شما پس از وارد کردن ایمیل خود و طی کردن مراحل پرداخت لینک های دریافت محصولات را در ایمیل خود دریافت خواهید کرد.

تولید کننده: LinkedIn Learning (Lynda)