Azure Rate Limiting (AzureRL) — Azure Provider

Overview

Azure enforces limits on the number of API calls it can handle in a given period. When those limits are exceeded, Azure rejects the call with HTTP 429 (Too Many Requests). Previously, during large operations such as large reservations or bulk pool teardowns, HyWorks could send calls faster than Azure would accept them, and the rejected calls were not retried, so the operation failed and had to be rerun manually.

Azure Rate Limiting (AzureRL) addresses this in two ways:

• Retry — when Azure returns 429 or a transient 503, HyWorks waits and retries the call instead of failing the operation.
• Pacing — HyWorks limits how many Azure calls of each type can run simultaneously, so bursts are spread out, and Azure is less likely to reject them.

Large-scale provisioning and teardown, therefore, complete without manual reruns.

Note

The feature is enabled by default. In most deployments, no configuration change is required.

Scope

Supported on HyWorks, HyLabs and AVD deployments using the Azure Provider, and applied to all Azure operations the provider performs — VM, disk, snapshot, image, network and resource group operations, AVD host pools and session hosts, and Entra ID directory lookups.

Read, write, delete, directory, and sign-in operations are paced independently, so reads never block writes. Limits apply per Azure subscription, and per tenant for directory and sign-in calls, so a busy subscription does not slow down another.

Configuration Settings

Configured in MMC > Advanced / Global Settings > Azure.

Setting	Purpose	Default	Range
AzureRateLimitEnabled	Master on/off switch for the whole feature.	true	true / false
AzureRateLimitRetryCount	How many times a call is retried when Azure returns 429/503.	2	0–30
AzureRateLimitWaitTime	Seconds to wait between retries when Azure does not specify its own wait time.	5	0–60
AzureRateLimitReadPerMin	Azure read limit, requests per minute.	1500	0 = use default
AzureRateLimitWritePerMin	Azure write limit, requests per minute.	600	0 = use default
AzureRateLimitDeletePerMin	Azure delete limit, requests per minute.	600	0 = use default
AzureRateLimitGraphPerMin	Microsoft Graph (directory) limit, requests per minute.	6000	0 = use default
AzureRateLimitAuthPerMin	Entra sign-in / token limit, requests per minute.	1200	0 = use default

Enter Azure's documented limit for your subscription, or a lower value if you want headroom. A value of 0 means "use the built-in default", not "disable" — only AzureRateLimitEnabled = false disables the feature.

Retry behaviour: HyWorks waits for the period Azure specifies (capped at 60 seconds) or, if Azure specifies none, for AzureRateLimitWaitTime, and retries up to AzureRateLimitRetryCount times. Errors that are not throttling — invalid credentials, bad request, resource not found — fail immediately and are never retried.

Note

Changes take effect without a service restart; worker/RMS nodes apply them within one heartbeat. Lowering a per-minute value will extend the time taken by clone, recompose, and delete operations, so restore production values after any test.

Logging

Log entries are written only when the feature is enabled and DEBUG logging is on. To enable, set the root log level to DEBUG in the log4net.config file alongside the running Controller service executable and restart the service. Entries appear in Logs\DebugEDController.txt — filter for [AzureRL].

Entry	Meaning	What to do with it
QUEUED	A call arrived and is about to be sent to Azure.	Normal. Shows the operation type and subscription.
READY	The call was allowed to proceed. waited=0ms means no delay.	Occasional waiting is normal under load. Consistently long waits mean the per-minute value is limiting throughput.
DONE	The call completed, with response time (http=) and status (status=).	status=429 means Azure rejected the call.
USAGE	A once-per-second summary of calls currently running.	Shows which operation type is the bottleneck during a large run.
QUOTA	Capacity Azure reports as remaining.	Values falling toward 0 mean throttling is approaching. Informational only.
STATS	A periodic summary per subscription and operation, including how many calls waited.	Compare with 429 counts before raising a limit.
RETRY	Warning-level. Azure rejected a call and HyWorks retried it.	Definitive proof of Azure throttling. Frequent entries mean the configured limits are above what Azure allows.

Warning

DEBUG logging significantly increases log volume during large operations. Revert to the normal level after diagnosis.