Azure Data Factory (Adf): Build Scalable Data Pipelines

Build, Orchestrate, and Secure Data Pipelines with Azure Data Factory for Cloud, Hybrid, and Multi-Cloud Integration

What you'll learn

Explain the fundamentals of Azure Data Factory (ADF) and its role within the Azure cloud ecosystem.

Understand cloud computing concepts, services, and data types relevant to enterprise data integration.

Identify and configure core components of ADF including linked services, datasets, pipelines, and triggers.

Perform data ingestion and migration tasks such as copying data across Azure Blob Storage, ADLS Gen2, SQL databases, and on-premises sources.

Analyze and manage copy activity behavior to optimize data transfer efficiency and reliability.

Implement parameterization in ADF pipelines using linked services, datasets, and variables for reusability and dynamic configurations.

Execute advanced copy operations such as bulk data transfers, file count–based activities, and multi-file ingestion scenarios.

Leverage stored procedures and SQL queries within ADF pipelines to transform and manage data flows.

Convert and transform structured data formats (e.g., CSV to JSON) using ADF data flows.

Apply security best practices by integrating Azure Key Vault for secrets management and secure credential handling.

Design and execute different data loading strategies including full loads, incremental (delta) loads, and hybrid approaches.

Integrate APIs and external services within ADF pipelines to extend data movement and transformation capabilities.

Orchestrate hybrid and multi-cloud data workflows by connecting ADF with AWS, Google Cloud, and on-premises systems.

Schedule, monitor, and automate pipelines using triggers and scheduling features in ADF.

Apply transformation logic using data flows, including join operations, to prepare data for analytics and downstream systems.

Requirements

Enthusiasm and determination to make your mark on the world!

Description

A warm welcome to the Azure Data Factory (ADF): Build Scalable Data Pipelines course by Uplatz.What is Azure Data FactoryAzure Data Factory (ADF) is Microsoft’s cloud-based ETL (Extract, Transform, Load) and data integration service. It enables organizations to move, transform, and orchestrate data from multiple sources, whether on-premises, in the cloud, or across different platforms.It serves as the data pipeline service within Azure, allowing data to be connected, cleaned, and delivered to systems such as data lakes, data warehouses, business intelligence platforms, and machine learning pipelines.How Azure Data Factory WorksAzure Data Factory follows a workflow approach with four main stages:1. Connect to Data Sources (Extract)ADF connects to more than 100 data sources using linked services, such as SQL Server, Azure Blob Storage, Amazon S3, Google Cloud Storage, Salesforce, and SAP. Data is ingested either in batches or real time.2. Prepare and Transform Data (Transform)ADF uses Data Flows (a visual, no-code transformation interface) or custom activities such as SQL scripts, Spark jobs, Databricks notebooks, and stored procedures. Transformations may include joins, filtering, aggregations, format conversions (CSV to JSON, Parquet, etc.), and data cleansing.3. Move and Load Data (Load)Data is loaded into target systems including Azure SQL Database, Azure Synapse Analytics, Azure Data Lake, Cosmos DB, or external storage systems. It supports full loads, incremental (delta) loads, and streaming ingestion.4. Orchestrate and Monitor PipelinesWorkflows are organized into pipelines that contain one or more activities. Triggers allow scheduling or event-based execution. ADF includes built-in monitoring and logging to track performance, identify errors, and analyze throughput.Core Components of ADFPipelines: Logical groups of activities that define a workflowActivities: Individual steps such as copy, transform, or execute stored procedureDatasets: References to data structures such as tables or filesLinked Services: Connection details to data sourcesData Flows: Visual interface to build transformation logicIntegration Runtime (IR): The compute engine that executes data movement and transformations, available as cloud or self-hostedWhy Use Azure Data FactoryFully managed and serverless with automatic scalingSupports hybrid and multi-cloud data integrationLow-code/no-code development experience with option for advanced codingEnterprise-grade security and governance through Azure Key Vault and RBACPrepares data pipelines for advanced analytics, reporting, and machine learning workloadsAzure Data Factory - Course CurriculumTopic 1: Foundations of Azure & ADFSession 1 – Introduction to Azure Data FactorySession 2 – Cloud Computing Part-1Session 3 – Cloud Computing Part-2Session 4 – Cloud ServicesSession 5 – Types of DataTopic 2: Core Components of ADFSession 6 – Top Components of ADF PART-1Session 7 – Top Components of ADF PART-2Topic 3: Data Copy & Migration BasicsSession 8 – Case Study-1: Copying the Data from Blob Storage to Blob StorageSession 9 – Azure BLOB Storage to ADLS Gen2 CopySession 10 – Copy Multiple Files from Azure BLOB Storage to ADLS Gen2Session 11 – Copy Data from Azure Blob to SQL DBSession 12 – Copy Data ToolSession 13 – Copy Activity Behaviour PART-1Session 14 – Copy Activity Behaviour PART-2Topic 4: Parameterization in ADFSession 15 – Parameterized Linked Services PART-1Session 16 – Parameterized Linked Services PART-2Session 17 – Parameterized Dataset and PipelineTopic 5: Advanced Copy OperationsSession 18 – Copy Bulk Data from SQL Database to Blob Storage PART-1Session 19 – Copy Bulk Data from SQL Database to Blob Storage PART-2Session 20 – Copy Activity on the Basis of File Counts in SourceTopic 6: Stored Procedures & TransformationsSession 21 – Understanding Stored Procedure on Azure CloudSession 22 – Copy of the Data Using Stored Procedure and SQL QuerySession 23 – Conversion of CSV to JSON Using ADF PART-1Session 24 – Conversion of CSV to JSON Using ADF PART-2Session 25 – Copy File (JSON to CSV)Topic 7: Security & Key ManagementSession 26 – Azure Key Vault ServiceTopic 8: Loading StrategiesSession 27 – Full Load and Delta Load PART-1Session 28 – Full Load and Delta Load PART-2Session 29 – Full Load and Delta Load PART-3Topic 9: Hybrid Data IntegrationSession 30 – Copy Data from On-Premise to Cloud in ADF PART-1Session 31 – Copy Data from On-Premise to Cloud in ADF PART-2Topic 10: API Integration & VariablesSession 32 – Integration of API with ADF PART-1Session 33 – Integration of API with ADF PART-2Session 34 – Pipeline Variable PART-1Session 35 – Pipeline Variable PART-2Topic 11: Multi-Cloud IntegrationsSession 36 – Integration of AWS with AzureSession 37 – Integration of ADF with Google Cloud StorageTopic 12: Scheduling & OrchestrationSession 38 – Triggers in ADFSession 39 – Schedule Trigger in Azure Data Factory PART-1Session 40 – Schedule Trigger in Azure Data Factory PART-2Topic 13: Data TransformationsSession 41 – Join Transformation Using Data Flows in ADF

Overview

Section 1: Foundations of Azure Data Factory (ADF)

Lecture 1 Introduction to Azure Data Factory (ADF)

Lecture 2 Cloud Computing

Lecture 3 Cloud Deployment Model

Lecture 4 Cloud Services

Lecture 5 Types of Data

Section 2: Core Components of ADF

Lecture 6 Part 1 - Top Components of ADF

Lecture 7 Part 2 - Top Components of ADF

Section 3: Data Copy & Migration Basics

Lecture 8 Copy Data from Blob Storage to Blob Storage

Lecture 9 Part 1 - Copy from Azure BLOB Storage to ADLS Gen2

Lecture 10 Part 2 - Copy from Azure BLOB Storage to ADLS Gen2

Lecture 11 Copy Data from Azure Blob to SQL DB

Lecture 12 Copy Data Tool

Lecture 13 Part 1 - Copy Activity Behavior

Lecture 14 Part 2 - Copy Activity Behavior

Section 4: Parameterization in ADF

Lecture 15 Part 1 - Parameterized Linked Services

Lecture 16 Part 2 - Parameterized Linked Services

Lecture 17 Parameterized Dataset and Pipeline

Section 5: Advanced Copy Operations

Lecture 18 Part 1 - Copy Bulk Data from SQL Database to Blob Storage

Lecture 19 Part 2 - Copy Bulk Data from SQL Database to Blob Storage

Lecture 20 Copy Activity on the Basis of File Counts in Source

Section 6: Stored Procedures & Transformations

Lecture 21 Understanding Stored Procedure on Azure Cloud

Lecture 22 Copy of the Data Using Stored Procedure and SQL Query

Lecture 23 Part 1 - Conversion of CSV to JSON using ADF

Lecture 24 Part 2 - Conversion of CSV to JSON using ADF

Lecture 25 Copy File - JSON to CSV

Section 7: Security & Key Management

Lecture 26 Azure Key Vault Service

Section 8: Loading Strategies

Lecture 27 Part 1 - Full Load and Delta Load

Lecture 28 Part 2 - Full Load and Delta Load

Lecture 29 Part 3 - Full Load and Delta Load

Section 9: Hybrid Data Integration

Lecture 30 Part 1 - Copy Data from On-Premise to Cloud in ADF

Lecture 31 Part 2 - Copy Data from On-Premise to Cloud in ADF

Section 10: API Integration & Variables

Lecture 32 Part 1 - Integration of API with ADF

Lecture 33 Part 2 - Integration of API with ADF

Lecture 34 Part 1 - Pipeline Variable

Lecture 35 Part 2 - Pipeline Variable

Section 11: Multi-Cloud Integrations

Lecture 36 Integration of AWS with Azure

Lecture 37 Integration of ADF with Google Cloud Storage

Section 12: Scheduling & Orchestration

Lecture 38 Triggers in ADF

Lecture 39 Schedule Trigger in Azure Data Factory

Lecture 40 Dataflows Filter

Section 13: Data Transformations

Lecture 41 Join Transformation using Data Flows in ADF

Data Engineers & ETL Developers – who want to design, build, and manage scalable data pipelines on Azure.,Cloud Engineers & Solution Architects – seeking to integrate on-premises and multi-cloud data sources into Azure-based analytics platforms.,Database Administrators (DBAs) – transitioning into cloud data management and automation using ADF.,Business Intelligence (BI) Developers & Analysts – who need to prepare, move, and transform data for reporting and analytics.,Big Data & Analytics Professionals – looking to orchestrate data workflows across heterogeneous systems.,Software Developers – who want to extend their skills into data integration and cloud-based data flows.,IT Professionals & System Integrators – involved in enterprise data migration, modernization, and cloud adoption projects.,Students & Beginners in Cloud Data Engineering – who wish to gain hands-on exposure to Azure Data Factory as a stepping stone into data engineering and cloud analytics careers.