Enable Azure Data Factory (ADF) and cloud-based jobs in IBM Workload Scheduler to support multi-region failover

Currently, IBM Workload Scheduler Azure Data Factory (ADF) job definitions are tightly coupled to a single Azure region. The primary or initial job definition does not support automatic or easy failover to another Azure location in the event of a regional outage or service degradation.

To support Azure datacenter failover scenarios effectively, customers require a solution that involves no or minimal changes at the job definition level. This is especially critical for enterprise environments where large numbers of ADF jobs are managed centrally through IWS.

Observations

From analysis of Azure ADF configurations across regions, the following parameters remain unchanged:

• Client ID
• Client Secret
• Tenant ID
• Subscription ID

The only region-specific parameters that differ are:

• Azure Resource Group
• Azure Data Factory name

This indicates that regional dependencies could be abstracted or parameterized within IWS, allowing a single job definition to operate across multiple Azure regions.

Requested Enhancement

Enhance the IWS Azure Data Factory (ADF) plugin to support built-in multi-region failover, allowing jobs to automatically retry or fail over to a secondary Azure location when the primary region is unavailable.

This could be achieved by:

• Allowing multiple Azure Resource Groups / ADF names to be defined at the connection or job level
• Supporting location-based priority or fallback logic
• Minimizing or eliminating the need to duplicate or redeploy job definitions for each region

Benefits

• Improved resiliency and disaster recovery for Azure-based workloads
• Reduced operational overhead and manual intervention during regional outages
• Consistent job definitions across environments and regions
• Increased adoption and value of IWS for cloud-native and multi-region Azure deployments

This enhancement would significantly benefit all customers using IBM Workload Scheduler with Azure Data Factory, enabling them to fully leverage Azure’s high-availability architecture while continuing to use IWS’s enterprise-grade scheduling capabilities.

Use Case

In the event of an Azure regional outage:

1. ADF jobs scheduled via IWS fail in the primary region.
2. IWS seamlessly fails over execution to a secondary Azure region without requiring job definition changes.
3. Business-critical data pipelines continue to run with minimal disruption.