Azure Data Engineer Course

By completing this module, you will acquire

1. Cloud computing concepts
2. An overview of Azure services
3. Essential knowledge about managing and governing resources

By completing this module, you will gain proficiency in

1. Setting up and managing Azure SQL Database
2. Understanding the cost and performance implications of different service tiers
3. Implementing access control measures
4. Securely storing secrets in Azure Key Vault
5. Mastering basic SQL concepts and best practices
6. Working with dynamic SQL in the context of stored procedures

By completing this module, you will gain practical skills in

1. 1. Setting up and configuring Azure Data Factory
   2. Creating and managing linked services, triggers, and datasets
   3. Performing metadata and data copy activities between different data sources
   4. Exploring dataflow concepts for data transformation
   5. Implement strategies for both full and incremental data loads using the watermark concept.

By completing this module, you will gain proficiency in

1. Creating more dynamic and flexible data pipelines and dataflows in Azure Data Factory.
2. Specifically, you'll be able to use parameterized datasets and dataflows to create reusable components and incorporate dynamic content to make your pipelines and dataflows adaptive.
3. implement error-handling strategies for robust and resilient data processing.

By completing this module, you will gain practical skills in

1. Building automation workflows using Azure Logic App. Specifically, you will learn how to connect to external APIs, extract, and parse data (particularly in JSON format), store data in Azure Storage
2. Schedule logic app workflows for regular execution and integrate email services for notifications or reporting.

By completing this module, you will gain practical skills in

1. Building a data warehousing solution using Azure SQL and Azure Data Factory.
2. How to schedule triggers for regular data processing
3. How to extract data from APIs
4. How to securely manage secrets
5. Perform data transformations using ADF activities
6. Implement upsert operations using stored procedures, handle errors, set up email notifications, and maintain an audit trail for data changes in SQL tables.

By completing this module, you will gain practical skills in

1. Working with Azure Synapse Analytics, including creating and configuring instances,
2. Navigating the Synapse Studio
3. Setting up Apache Spark Pools, using notebooks for data analysis
4. Managing access and roles and interacting with data stored in Azure Data Lake Storage Gen2.

The ETL project is designed to ensure universal comprehension of data engineering tools, their limitations, and the practical implementation of course-covered techniques in real-world ETL workflows for all participants.