Retry Capability on Network Gateway Issues for Azure Datafactory jobs

This change is intended to address one of the issues we are currently experiencing with ADF jobs. Due to transient cloud-related issues, TWS ADF plugin jobs occasionally fail with errors such as the one shown below.

This typically occurs after the job has been successfully triggered and TWS is polling for the pipeline’s completion. If a transient error occurs during the polling phase, the job fails on the TWS side, even though the corresponding ADF pipeline continues to run and remains in a running state.

To prevent these scenarios, a retry mechanism should be implemented during the polling process. This would help avoid polling failures caused by temporary cloud issues. In addition, it would be beneficial to evaluate whether a similar retry mechanism can be applied during the initial job trigger phase, where TWS attempts to start a job but fails due to transient cloud errors. Care must be taken to ensure that this does not result in the same pipeline being triggered multiple times.

Implementing this enhancement would help eliminate many intermittent failures and aligns with recommendations provided by Microsoft. This improvement would be beneficial for all teams and organizations using the ADF plugin.

Example error:

AWKADF021E Error while triggering the pipeline
GatewayTimeout: The gateway did not receive a response from 'Microsoft.DataFactory' within the specified time period