IIRF Online > IT & Software > Other IT & Software > Microsoft Azure > Azure Data Engineer Technologies for Beginners [Bundle]
Azure Data Engineer Technologies for Beginners [Bundle] by Udemy
Microsoft Azure SQL Database, Data Lake, Data Factory, Synapse Analytics, Cosmos DB, Databricks, HDInsight, storage
Course Highlights
- You will be able to identify right Azure SQL Server deployment option, purchasing model and service tier according to requirements and will be able to deploy it in cloud.
- You will able to deploy Azure Synapse Analytics (formerly known as Azure SQL Data warehouse) in Azure Cloud environment.
- You will have good internal MPP architecture understanding, and so you will be able to analyse your on-premises data warehouse, and migrate data to Azure Data Warehouse.
- You will be able to create Azure Data Lake Gen1 storage account, populate it will data and analyze it using U-SQL Language.
- You will understand Azure Data Factory key components and advantages. You will be able to create, schedule and monitor simple pipelines.
- We will understand how HDInsight makes Hadoop easy, and we will go through simple demo where we’ll fetch data from Data Lake, process it through Hive and later will store data in SQL Server.
Skills you will learn!
Curriculum
6 Topics
Course Overview and Set Expectations
Similar courses? which one should I take?
QAs related to course - Imp Note
How to keep using Azure Portal for FREE after 12 months?
How to get FREE credits for Azure Portal
Before you start...
14 Topics
Download PPT of this Module
Introduction
Create Azure free subscription
Azure Portal overview
Azure Services Overview
Managed and Unmanaged Service
Resource Management group and Subscription
Resource Groups
Resource Provider
Tagging
Access Control
Demo - Create Azure SQL Database service
Delete resources and set budget
Conclusion
5 Topics
Download PPT of this Module
Introduction
Data Engineer role and responsibility
Introduction to Data Engineer technologies
Conclusion
56 Topics
Module Introduction
Download PPT - Azure SQL Database
**** SECTION **** - Introduction to Azure SQL Database
Introduction
Why choosing SQL Server in Azure
Azure IaaS vs PaaS database offerings
Iaas vs Managed Instance
SQL Server PaaS deployment options
Further References from Microsoft Docs
Practice Test
**** SECTION **** - Azure SQL Server in Virtual Machine (IaaS)
Introduction
SQL Server in Azure Virtual Machine
Demo - SQL Server in Azure Virtual Machine
Further References from Microsoft Docs
Practice Test
**** SECTION **** - Azure Single Database
Introduction
Demo - Azure Single Database
Purchasing models and Service Tier
Azure Database vs Azure Data warehouse
Further References from Microsoft Docs
Practice Test
**** SECTION **** - Elastic Database Pool
Introduction
Azure Elastic Database
Demo - Azure Elastic Database
Further References from Microsoft Docs
Practice Test
**** SECTION **** - Managed Instance Database
Introduction
Azure Managed Instance Database
Difference between on-premises and Managed Instance
Migration options for Managed Instance
Service tiers for Managed Instance
Management operations
Demo - Managed Instance
Further References from Microsoft Docs
Practice Test
**** SECTION **** - Azure Database Security
Introduction
Azure Database Security options
Azure Managed Instance advance security
Encrypting Data at Rest and Motion
Dynamic Data Masking
High Availability vs Disaster Recovery
RTO vs RPO
Azure SQL Database High Availability and Disaster Recovery options
Azure SQL Database Backup and Restore
Azure SQL Database Scaling
Further References from Microsoft Docs
Practice Test
Monitoring through portal
Monitoring through external tools
Intelligent performance and Automatic Tuning
Query Performance Insight
71 Topics
IMPORTANT Updates on new changes - PLEASE READ
Data Warehouse - crash course
Module Introduction
Azure Synapse - PPT and Demo Resources
**** UPDATED SECTION **** - Introduction to Azure Synapse Analytics Service
Introduction
Why warehousing in cloud
Traditional vs Modern Warehouse architecture
What is Synapse Analytics Service
Demo: Create Dedicated SQL Pool
Demo: Connect SQL Pool with SSMS
Demo: Create Azure Synapse Analytics Workspace
Demo: Explore Synapse Studio V2
Demo: Create Dedicated SQL Pool and Spark Pool from inside Synapse Studio
Demo: Analyse Data using Dedicated SQL Pool
Analyse Data using Apache Spark Notebook
Demo: Analyse Data using Serverless SQL Pool
Demo: Data Factory Copy Tool from Synapse Integrate Tab
Demo: Monitor Synapse Analytics Studio
Azure Synapse a Game changer
Azure Synapse Benefits
Synpase unified experience
Summary
Practice Test
**** SECTION **** - Azure Synapse Internal and Architecture
Introduction
Azure Synapse MPP Architecture
Storage and Sharding patterns
Data Distribution and Distributing Keys
Data Types and Table Types
Partitioning
Best Practices for Fact and Dimension tables
Demo - Analyse data distribution before migration to azure
Summary
Practice Test
**** SECTION **** - Data Migration to Azure Synapse Data Warehouse
Introduction
5 best practices for Data Load
DIfferent loading methods
Loading with SSIS vs PolyBase
Demo - Loading with SSIS
Demo - Loading with Polybase
Demo - Loading with Data Factory
Summary
Practice Test
Assignment Question - Lab: Lake to Synapse
Assignment Solution - Lab: Lake to Synapse
**** SECTION **** - Security layers in Azure Synapse Service
Introduction
Advance Data Security
Auditing
Network Security
Transparent data encryption
Dynamic Data Masking
Access Management
Summary
Practice Test
**** SECTION **** - Configuring and Optimizing Azure Synapse Service
Introduction
Configuration Options
Performing other tasks
Backup and Restore
Scaling
Price Optimization
Managing Workloads
Monitoring
Support and Troubleshooting
Delete Resources
Summary
Practice Test
DP-200 Quick Notes/cheat sheet
49 Topics
Data Lake - PPT and Demo Resources
Introduction
Problem statement
What is Data Lake? and how it can solve this problem?
Data Lake vs Hadoop
How Data Lake Gen2 evolved?
Blob Storage vs Data Lake
Hierarchical namespace
Demo - Create Azure Data Lake Gen2 Account
Demo - Data Lake Gen2 account overview
Data Lake Gen2 other features
**** SECTION **** - Data Lake Ingestion through diff tools
Explore different tools to Ingest Data in Azure Data Lake Gen2
Demo - On-Premises to Data Lake Gen2 using portal and storage explorer
Demo - On-Premises to Data Lake Gen2 using azcopy
Demo - Azure Blob to Data Lake Gen 2 using Data Factory
Demo - Azure SQL Database to Data Lake Gen2 using Data Factory
Demo - Amazon S3 to Data Lake Gen2 using Data Factory
**** SECTION **** - Data flow around Data Lake
Data flow around Data Lake
Data Lake and Transient clusters
Data Processing using HDInsight
**** SECTION **** - Data Lake Processing through Databricks
Imp Note about this Demo
Demo overview
Provision Databricks Clusters and workbook
Mount Data Lake to Databricks DBFS
Explore Analyze Clean Transform
**** SECTION **** - Data Lake Security Layers
Introduction
Storage Access Keys
SAS - Shared Access Signature
Azure Active Directory
Role Based Access Control (RBAC)
Access Control list (ACL)
Firewalls and Virtual Networks
Encryption in Transit
Encryption at Rest
Advanced threat protection
**** SECTION **** - Data Lake Monitoring and Optimization
Activity Log
Demo: Activity Logs
Metrics
Demo: Metrics
Demo: Insights
Demo: Alerts
Diagnostic settings
Demo: Diagnostic settings
Optimization
12 Topics
PPT and Demo Resources
Introduction to NoSQL
SQL vs NoSQL
4 types of NoSQL Databases
NoSQL Offerings by Microsoft Azure
Azure Storage Introduction
Demo: Part 1 Provision Azure Storage Account
Demo Part 2: Azure Storage Replication
Demo: Part 3 Continue Provisioning Azure Storage account
Azure Blob Storage
Managing the Data LifeCycle
Azure Storage - HA and DR Options
28 Topics
Cosmos DB - PPT and Demo Resources
Problem Statement How Cosmos DB Evolved
Cosmos DB Features
Cosmos DB - Multi Model 5 APIs
Azure Table Storage vs Cosmos DB Table API
Provision Cosmos DB Account
Cosmos DB - Database Containers and items
Cosmos DB - Throughput and Request units
Cosmos DB - Horizontally Scalable
Cosmos DB - Partition and Partitioning key
Cosmos DB - Dedicated vs Shared throughput
Cosmos DB - Avoiding hot partitions
Cosmos DB - Single partition vs Cross partition
Cosmos DB - Composite Key
Cosmos DB - Partition key best practice
Cosmos DB - Automatic Indexing
Demo Insert and query data in your cosmos DB
Cosmos DB - Time to Live feature
Cosmos DB - Globally Distribution feature
Cosmos DB - Multi Master feature
Cosmos DB - Manual vs Automatics Failover
Cosmos DB - 5 consistent levels
Cosmos DB - Azure CLI
Cosmos DB - Pricing
Cosmos DB - Monitoring through AMS (Azure Monitor Service)
Cosmos DB - Monitoring through Cosmos DB Portal
Cosmos DB - Security
Cosmos DB - High Availability and Disaster Recovery Option
12 Topics
PPT and Demo Resources
What is Live Event Processing?
Introduction to Azure Streaming Analytics Service
Streaming Analytics - What is time windowing
Streaming Analytics - Tumbling Window
Streaming Analytics - Hopping Window
Streaming Analytics - Sliding Window
Streaming Analytics - Session Window
Streaming Analytics - Processing Blob Storage Input
Streaming Analytics - Processing IOT Hub Input
Stream Analytics - Monitoring
Stream Analytics - Optimization
12 Topics
PPT and Demo Resources
Spark Basics
Why Spark is difficult? Why Databricks Evolved?
Why Databricks in Cloud? Introduction to Azure Databricks
How to save Databricks demo Cost
Demo overview
Demo: Provision Databricks Clusters and workbook
Demo: Mount Data Lake to Databricks DBFS
Demo :Explore Analyze Clean Transform and Load Data in Databricks
Azure Databricks Clusters
Azure Databricks other Important Components
Databricks - Monitoring
19 Topics
PPT and Demo Resources
What is Data Factory
Data Factory in Azure Ecosystem
Provision Azure Data Factory Instance
Data Factory Components
Data Factory - Pipeline and Activities
Data Factory - Linked Service and Datasets
Data Factory - Integration runtime
Data Factory - Triggers
Data Factory - Copy Data Activity Wizard
Data Factory - Copy Data Activity using Author
Data Factory - Secure Input/Output Property
Data Factory - User Properties
Data Factory - Parameters
Data Factory - Data Flow Concept
Data Factory - Mapping Data Flow
Data Factory - Wrangling Data Flow
Data Factory - Monitoring
Data Factory - Monitoring through Metrics Alerts Diagnostic Settings
4 Topics
PPT and Demo Resources
Why Monitoring is Important
Introduction to Azure Monitor Service
Demo: Azure Monitor Service
20 Topics
PPT - Hadoop and HDInsight
Hadoop - crash course
HDInsight overview
Why Hadoop is Hard
How HDInsight make Hadoop Easy
Important aspects of HDInsight
HDInsight cluster types
HDInsight Architecture
**** SECTION **** - Demo - HDInsight
Demo overview
Create Azure Data Lake Storage Gen 2 (source) and SQL Server (destination)
What is Managed Identity
Add Managed Identity to Gen2 and Database accounts
Create HDInsight Interactive Query Cluster
Ambari overview and UI
Ingest dataset in to DataLake storage
Data Extraction with Hive
Data transformation with Hive
Data Export using Sqoop
Summary
1 Topic
Crash Courses for absolute beginners
12 Topics
Download PPT of this Module
Introduction
What is Data Warehouse
Why we need Data Warehouse
Ideal Data Warehouse Solution
Responsibilities of Data Warehouse designer
SQL Server vs Data Warehouse
Dimensional Modeling
Facts and Fact Table
Dimensions and Dimension table
Star vs Snowflake
Summary
7 Topics
Download PPT
Hadoop overview
Why we need distribute computing?
Two ways to build system
Introducing Hadoop
Hadoop vs RDBMS
Hadoop Summary
2 Topics
Practice Test 1
Practice Test 2
1 Topic
Bonus Section
Azure Data Engineer Technologies for Beginners [Bundle]
Related Microsoft Azure Courses
.webp)
Thank You!
Your review has been submitted successfully.