Snowflake cloud database with ELT(Airflow+Python+Talend)

سرفصل های دوره

Learn to integrate ETL tools with Snowflake and leverage Airflow for ELT with snowflake.Process 250+ GB data.ELT flow.

1. Module 1

1. Module 1 Intro

2. prerequisites

1.1 course ddl stmts.zip

1. Setup snowflake environment

2.1 Setting up virtual machine.docx

2.2 Virtual machine download alternate link 2.html

2.3 Virtual box download.html

2.4 Virtual machine download.html

2.5 Virtual machine download alternate link.html

2. Virtual machine download

3.1 airflow.zip

3. Mount local folder to vm and start vm

4.1 ssh pwd.zip

4.2 Visual studio code download.html

4. Setup Vs code for VM

5. Configure connection details

6. Github link.html

3. Airflow intro

1. Airflow introduction

2. Airflow architecture

3. First Dag

4.1 Postgress airflow metadata db connection details.docx

4.2 postgress airflow metadata db connection details.zip

4. Introduction to metadata DB

5. Airflow scheduler

6. Scheduling in airflow

7. Executors in airflow

8. Airflow webserver

9. Handle failures

4. Snowflake airflow ELT

1. Introduction

2. Load to snowflake example 1

3. [Demo] Load to snowflake example 1

4. Load to snowflake example 2

5. Load to snowflake example 3

5. Snowflake python api intro -- Basics

1.1 Python API.html

1.2 python api.zip

1.3 Python API.xlsx

1. Introduction

2. Connection test 1

3. Connection test 2

4. execute method

5. cursor attributes

6. execute many

7. execute async part 1

8. execute async part 2

9. execute async part 3

10. Formatting pyformat

11. Formatting qmark

12. Result metadata

13. Type code

6. Snowflake python api intro -- Advanced

1. Fetch pandas arrow

2. Context manager

3. Result batch

4. Get result batch

5. Fetch arrow batches

6. Fetch pandas batches

7. To pandas arrow

8. Execute stream

9. Execute string

10. Cursor dict

7. Snowflake python ELT

1. Copy data s3 to snowflake

2. Copy data s3 to snowflake ( parallel )

3. Execute ELT query Python way

4. Schedule airflow job For python code

8. Talend introduction

1. Talend as open source ETL tool.

2.1 Installing Talend.docx

2. Download and install talend.

3. Talend workspace introduction.

4.1 Installing Talend.docx

4.2 Section 2 lec 4 emp data.zip

4.3 section 2 lec 4 emp ddl.zip

4.4 Snowflake project.zip

4.5 TOS DI-20181026 1147-V7.1.1.zip

4. Build first job and configure talend

5. Build sample job Level 1

6. Build sample job Level 2

7. Build sample job Level 3

8. Building sample job Level4

9. Build sample job Level5

10. Build sample job Schedule using airflow

11. Summary

9. Module 2

1. Module 2 intro

10. Create generic components

1. Analyst requirement.

2.1 S3snowcp.zip

2. S3 to snowflake copy app demo

3. Requirement to process data.

4.1 Eltjobdemo.zip

4. Elt app demo

5. Scheduling job.

6. Writing generic dag code part1

7. Writing generic dag code part2

8. Writing generic dag code part3

11. Building workflow

1. Introduction

2.1 Data for this section.html

2. High level design

3.1 Data for this section.html

3. Copy data from s3

4. Parsing data

5. Audit src records

6. Archive and create view part 1

7. Archive and create view part 2

8. Score records part 1

9. Score records part 2

10. prepare dag code

11. Execute airflow job

12. Improve the approach

13. Parameter registry

14. Summary

12. Nyc traffic data demo

1. Introduction.

2.1 Nyc website.html

2.2 Nyc data volume.xlsx

2.3 Nyc data.7z

2. Nyc traffic data overview

3. Load sample data

4. Nyc full data load

5.1 Queries.zip

5. Snow sight Dashboard

13. Bonus sections starts

1. Bonus section.html

14. Generic jar logic

1.1 create stage obj audit tbls.zip

1.2 Creating integration object.docx

1. Building talend generic copy job.

2. Evaluating audit tables.

3. ELT transform job demo.

4.1 elt audit tbls ddl.zip

4. ELT Generic job code demo.

15. Bonus section Exporting talend jobs as jar apps

1. Introduction.

2. Managers requirement.

3. Solution step 1 preparing query.

4. Solution step 2 create talend job.

5. Solution step 3 parameterise talend job.

6.1 manager requirement1.zip

6. Solution step 4 create jar application.

7. Managers requirement 2.

8. Solution step 1 creating talend job.

9.1 managers requirement2.zip

9. Solution step 2 using context groups.

10. Developer problem

11.1 cost warning.zip

11. Cost warning app

“Generative AI for DevOps Engineers” به سبد خرید شما اضافه شد. مشاهده سبد خرید

139,000 تومان

خرید اشتراک افزودن به سبد خرید

خرید دانلودی فوری

در این روش نیاز به افزودن محصول به سبد خرید و تکمیل اطلاعات نیست و شما پس از وارد کردن ایمیل خود و طی کردن مراحل پرداخت لینک های دریافت محصولات را در ایمیل خود دریافت خواهید کرد.

تولید کننده: Udemy