Azure Databricks end to end project with Unity Catalog CICD

سرفصل های دوره

Azure Databricks Mastery: Hands-on project with Unity Catalog , Delta lake, CI/CD implementing Medallion Architecture

1. Introduction

1. Course Introduction

2. Project Architecture and Concepts

3. Course prerequisites and benefits

4.1 cicd.zip

4.2 CICD.zip

4.3 notebooks.zip

4.4 notebooks.zip

4. Project Complete Code.html

2. Environment Setup

1. Section Introduction

2. Creating a budget for project

3. Creating an Azure Databricks Workspace

4. Creating an Azure Datalake Storage Gen2

5. Walkthough on databricks Workspace UI

3. Azure Databricks - An Introduction

1. Section Introduction

2. Introduction to Distributed Data Processing

3. What is Azure Databricks

4. Azure Databricks Architecture

5. Cluster types and configuration

6. Behind the scenes when creating cluster

7.1 Login page.html

7.2 Signup page.html

7. Sign up for Databricks Community Edition

8.1 01. understanding notebook and markdowns.zip

8. Understanding notebook and Markdown basics

9.1 02. magic commands.zip

9. Notebook - Magic Commands

10.1 03. dbutis - file system utilities.zip

10. DBUitls -File System Utilities

11.1 04. dbuitls - widget utilities.zip

11. DBUitls -Widget Utilities

12.1 dbutils - notebooks utils - child - para.zip

12.2 dbutils - notebooks utils - child.zip

12.3 dbutils notebook utils - parent - para.zip

12.4 dbutils notebook utils - parent.zip

12. DBUtils - Notebook Utils

13. Basic knowledge on Azure databricks.html

4. Delta lake

1. Section Intro

2. Drawbacks of Azure Datalake

3. What is delta lake

4. Understanding Lakehouse Architecture

5. Creating databricks workspace and ADLS for delta lake

6.1 Countries1.csv

6.2 Grant the service principal access to Azure Data Lake Storage Gen2.html

6.3 test access.zip

6. Accessing Datalake storage using service principal

7.1 01. drawbacks of adls.zip

7. Drawbacks of ADLS - practical

8.1 01. drawbacks of adls - delta.zip

8. Creating Delta lake

9. Understanding the delta format

10.1 02. understanding the transaction log.zip

10. Understanding Transaction Log

11.1 03. creating delta tables.zip

11. Creating delta tables using SQL Command

12.1 04. creating delta tables using pyspark.zip

12. Creating Delta table using PySpark Code

13.1 SchemaLessCols.csv

13.2 SchemaMoreCols.csv

13. Uploading files for next lectures

14.1 05. schema enforce.zip

14.2 SchemaLessCols.csv

14.3 SchemaManagementDelta.csv

14.4 SchemaMoreCols.csv

14. Schema Enforcement

15.1 05. schema evolution.zip

15. Schema Evolution

16.1 06. versioning and time travel.zip

16.2 SchemaManagementDelta.csv

16. Time Travel and Versioning

17.1 07. vacuum command.zip

17. Vacuum Command

18.1 08. convert to delta.zip

18. Convert to Delta

19. Understanding Optimize Command - Demo

20.1 09. optimize command.zip

20. Optimize Command - Practical

21.1 10. upsert using merge.zip

21. UPSERT using MERGE

22. Test your Delta lake knowledge.html

5. Unity Catalog

1. Section Introduction

2. What is Unity Catalog

3. Creating Access Connector for Databricks

4. Creating Metastore in Unity Catalog

5. Unity Catalog Object Model

6. Roles in Unity Catalog

7. Creating users in Azure Entra ID

8. User and groups management Practical

9.1 Cluster Policy Project Defaults.txt

9.2 Compute Policy Definition.html

9. Cluster Policies

10. What are cluster pools

11.1 Cluter policy for pool.txt

11. Creating Cluster Pool

12. Creating a Dev Catalog

13.1 unity catalog privileges.zip

13. Unity Catalog Privileges

14. Understanding Unity Catalog

15. Creating and accessing External location and storage credentials

16.1 managed vs external tables.zip

16. Managed and External Tables in Unity Catalog

17. Check your knowledge on Unity Catalog.html

6. Spark Structured Streaming

1. Section Introduction

2. Spark Structured Streaming - basics

3. Understanding micro batches and background query

4. Supported Sources and Sinks

5.1 01 - streaming basics.zip

5. WriteStream and checkpoints

6. Community Edition Drop databases

7.1 02. outputmodes.zip

7. Understanding outputModes

8.1 03. triggers.zip

8. Understanding Triggers

9. Autoloader - Intro

10.1 04. autoloader basics.zip

10. Autoloader - Schema inference

11. Schema Evolution - Demo

12.1 05. autoloader - schema evolution.zip

12. Schema Evolution - Practical

13. Check your knowledge on Streaming.html

7. Project Overview

1. Section Introduction

2. Typical Medallion Architecture

3. Project Architecture

4. Understanding the dataset

8. Project Setup

1. Section Introduction

2. Expected Setup

3. Creating containers and External Locations

4.1 01. project setup.zip

4. Creating all schemas dynamically

5.1 01. project setup.zip

5. Creating bronze Tables Dynamically

9. Ingestion to Bronze

1. Section Introduction

2. Ingesting data to bronze layer - Demo

3.1 Sample files.zip

3. Ingesting raw traffic data to bronze table

4. Assignment to get the raw roads data to bronze table.html

5.1 02. load to bronze.zip

5. Ingesting raw roads data to bronze Table

6. To prove autoloader handles incremental loading

10. Silver Layer Transformations

1. Section Introduction

2.1 03. silver - traffic transformations.zip

2. Transforming Silver Traffic data

3. To prove only incremented records were being transformed

4.1 04. common.zip

4. Creating a common Notebook

5. Run one notebook from another notebook

6.1 05. silver - roads transformation.zip

6. Transforming Silver Roads data

11. Loading to Gold Layer

1. Section Introduction

2. Getting data to Gold Layer

3.1 06. gold - final transformations and loading.zip

3. Gold Layer Transformations and loading

12. Orchestrating with Workflows

1. Section Introduction

2.1 notebooks.zip

2. Adding run for common notebook in all notebooks

3. Creating Jobs and executing end to end flow

4. Attaching trigger to workflows

13. Reporting with Power BI

1. Installing Power BI Desktop

2.1 final report.zip

2. Reporting data to Power BI

14. Continuous Integration and Continuous Deployment (CICD)

1. Section Introduction

2. Expected Setup

3. Understanding Continuous Integration

4. Understanding Continuous Deployment

5. Creating Required resources for UAT

6. Configuring storage containers and external locations for UAT

7. Login and create repository in Azure DevOps

8. Integrating Azure Devops with Databricks

9. Creating feature branch and pull request to main branch

10. Creating pull request as new user

11.1 Code.zip

11. Uploading and understanding YAML Files for CICD

12. Creating CI pipeline to have live folder

13. Permissions to see Live Folder

14.1 For CD Pipeline.zip

14. Creating Deployment pipeline and deploying

15. End to end test CICD pipeline

16. Running notebooks in UAT

15. Delta Live Tables (DLT)

1. Section Intro

2. Origin of Delta live tables

3. Considerations in Lakehouse Architecture

4. Understanding Declarative ETL

5. Limitations of Delta Live Tables

6. Defining Tables from datasets

7. Creating DLT Pipeline

8.1 dlt.zip

8. End to end DLT Pipeline

9. Deleting cluster by DLT pipeline

16. Conclusion

1.1 My LinkedIn profile.html

1. Course completion

2. My other Data Engineering Courses.html

45,900 تومان

خرید اشتراک افزودن به سبد خرید

خرید دانلودی فوری

در این روش نیاز به افزودن محصول به سبد خرید و تکمیل اطلاعات نیست و شما پس از وارد کردن ایمیل خود و طی کردن مراحل پرداخت لینک های دریافت محصولات را در ایمیل خود دریافت خواهید کرد.

تولید کننده: Udemy