Secure AWS to Azure Data Pipeline via Workload Identity Federation (OIDC)
Sector: Banking / Financial Services
Architecture Style: Event-Driven Multi-Cloud (AWS + Azure)
Bank transaction logs (CSV/JSON) are pushed to the AWS S3 landing bucket from external mainframe systems.
Azure Data Factory assumes an AWS IAM role via Workload Identity Federation (OIDC). No long-lived access keys are stored.
ADF Pipeline copies the delta files from AWS S3 into Azure Blob Storage (Raw Zone).
Blob creation fires an Event Grid event to an Azure Function, which performs lightweight schema validation and triggers Databricks.
Databricks mounts Blob, cleanses data, runs ML-based fraud detection models, and structures the output.
Cleaned, scored transactions are upserted into Azure Cosmos DB (NoSQL API) for high-concurrency, low-latency querying.
Power BI connects to Cosmos DB to provide real-time dashboards for the bank's fraud analytics team.
All component logs, pipeline metrics, and security events are centralized in Log Analytics.
# AWS Provider & S3 Landing Zone
resource "aws_s3_bucket" "banking_landing" {
bucket = "gh-bank-landing-dev-centralus"
}
# AWS IAM Role for OIDC (Azure AD / Entra ID Trust)
resource "aws_iam_role" "azure_adf_role" {
name = "AzureADFAccessRole"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRoleWithWebIdentity"
Effect = "Allow"
Principal = {
Federated = "arn:aws:iam::123456789012:oidc-provider/login.microsoftonline.com/TENANT_ID/v2.0"
}
Condition = {
StringEquals = {
"login.microsoftonline.com/TENANT_ID/v2.0:sub" = "api://AzureADFAppID"
}
}
}]
})
}
# Azure Resource Group & Storage
resource "azurerm_resource_group" "rg" {
name = "rg-banking-poc-dev-centralus"
location = "Central US"
}
resource "azurerm_storage_account" "raw" {
name = "stbankrawdevcentralus"
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
account_tier = "Standard"
account_replication_type = "LRS"
is_hns_enabled = true
}
# Azure Data Factory & Managed Identity
resource "azurerm_user_assigned_identity" "adf_mi" {
name = "mi-adf-banking-dev"
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
}
resource "azurerm_federated_identity_credential" "aws_trust" {
name = "fic-aws-trust"
resource_group_name = azurerm_resource_group.rg.name
audience = ["api://AzureADFAppID"]
issuer = "https://login.microsoftonline.com/TENANT_ID/v2.0"
parent_id = azurerm_user_assigned_identity.adf_mi.id
subject = "system:serviceaccount:default:adf-sa"
}
resource "azurerm_cosmosdb_account" "cosmos" {
name = "cosmos-banking-dev-centralus"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
offer_type = "Standard"
kind = "GlobalDocumentDB"
capabilities {
name = "EnableServerless"
}
consistency_policy {
consistency_level = "Session"
}
geo_location {
location = azurerm_resource_group.rg.location
failover_priority = 0
}
}
Environment: dev | Region: centralus | Workload: ~10GB Data/mo, 100k requests.
| Cloud Component | SKU / Tier (Dev) | Estimated Cost (USD) |
|---|---|---|
| AWS S3 Bucket | Standard (10 GB) + API calls | $ 0.50 |
| AWS IAM | Global / Free | $ 0.00 |
| Azure Data Factory | Data Pipeline (DIU) - Scheduled runs | $ 15.00 |
| Azure Blob Storage (ADLS Gen2) | Standard LRS (10 GB) | $ 0.40 |
| Azure Event Grid & Functions | Consumption Tier (Free grant applies) | $ 0.50 |
| Azure Databricks | Compute (Dev clusters, auto-terminate) | $ 60.00 |
| Azure Cosmos DB | Serverless (Pay per RU) | $ 5.00 |
| Azure Monitor / Log Analytics | Pay-As-You-Go (Data Ingestion ~5GB) | $ 14.00 |
| Total Estimated Monthly Cost (Dev Environment) | $ 95.40 | |