Configuring ASO

Configuration of ASO is done primarily through a secret in the azureserviceoperator-system namespace called aso-controller-settings. This secret contains both details about the (optional) global credential as well as other operator pod options.

The supported options are:

AZURE_SUBSCRIPTION_ID

Azure subscription the operator will use for ARM communication if no more specific credential is specified at the per-resource or per-namespace scope.

Format: GUID

Example: 00000000-0000-0000-0000-000000000000

Required: True

This may be set to empty string to configure no global credential.

AZURE_TENANT_ID

Azure tenantID the operator will use for ARM communication if no more specific credential is specified at the per-resource or per-namespace scope.

Format: GUID

Example: 00000000-0000-0000-0000-000000000000

Required: True

This may be set to empty string to configure no global credential.

AZURE_CLIENT_ID

Azure clientID the operator will use for ARM communication if no more specific credential is specified at the per-resource or per-namespace scope.

Format: GUID

Example: 00000000-0000-0000-0000-000000000000

Required: True

This may be set to empty string to configure no global credential.

AZURE_CLIENT_SECRET

The secret associated with the client to use if no more specific credential is specified at the per-resource or per-namespace scope.

Format: String

Required: False

AZURE_CLIENT_CERTIFICATE

AzureClientCertificate is a PEM or PKCS12 certificate string including the private key for Azure Credential Authentication. If the certificate is password protected, use AZURE_CLIENT_CERTIFICATE_PASSWORD for the password.

Format: String

Required: False

AZURE_CLIENT_CERTIFICATE_PASSWORD

The password used to protect the AZURE_CLIENT_CERTIFICATE.

Format: String

Required: False

AZURE_SYNC_PERIOD

AZURE_SYNC_PERIOD is the frequency at which resources are re-reconciled with Azure when there have been no triggering changes in the Kubernetes resources. This sync exists to detect and correct changes that happened in Azure that Kubernetes is not aware about. BE VERY CAREFUL setting this value low - even a modest number of resources can cause subscription level throttling if they are re-synced frequently. If nil or empty (""), sync period defaults to 1h.

Specify the special value "never" to stop syncing.

Format: duration string

Example: "1h", "15m", or "60s". See ParseDuration for more details.

Required: False

AZURE_OPERATOR_MODE

AZURE_OPERATOR_MODE determines whether the operator should run watchers, webhooks or both (default). An empty string, or any unrecognized value, means both.

Format: webhooks|watchers|both

Examples: "webhooks", "watchers" or "both"

Required: False

AZURE_TARGET_NAMESPACES

AZURE_TARGET_NAMESPACES lists the namespaces the operator will watch for Azure resources (if the mode includes running watchers). If it’s empty the operator will watch all namespaces.

Spaces after ,’s and at the start and end of the string are ignored.

Format: string (comma-separated namespace names)

Example: ns1,ns2

Required: False

USE_WORKLOAD_IDENTITY_AUTH

USE_WORKLOAD_IDENTITY_AUTH boolean is used to determine if we’re using Workload Identity authentication for global credential.

Format: true|false

Example: "true" or "false"

Required: False

AZURE_AUTHORITY_HOST

AZURE_AUTHORITY_HOST is the URL of the AAD authority. If not specified, the default is the AAD URL for the public cloud: https://login.microsoftonline.com/. See https://docs.microsoft.com/azure/active-directory/develop/authentication-national-cloud

Format: string

Example: "https://login.chinacloudapi.cn"

Required: False

AZURE_RESOURCE_MANAGER_ENDPOINT

AZURE_RESOURCE_MANAGER_ENDPOINT is the Azure Resource Manager endpoint. If not specified, the default is the Public cloud resource manager endpoint. See https://docs.microsoft.com/cli/azure/manage-clouds-azure-cli#list-available-clouds for details about how to find available resource manager endpoints for your cloud. Note that the resource manager endpoint is referred to as “resourceManager” in the Azure CLI.

Format: string

Example: "https://management.chinacloudapi.cn"

Required: False

AZURE_RESOURCE_MANAGER_AUDIENCE

AZURE_RESOURCE_MANAGER_AUDIENCE is the Azure Resource Manager AAD audience. If not specified, the default is the Public cloud resource manager audience https://management.core.windows.net/. See https://docs.microsoft.com/cli/azure/manage-clouds-azure-cli#list-available-clouds for details about how to find available resource manager audiences for your cloud. Note that the resource manager audience is referred to as “activeDirectoryResourceId” in the Azure CLI.

Format: string

Example: "https://management.core.chinacloudapi.cn/"

Required: False

AZURE_USER_AGENT_SUFFIX

AZURE_USER_AGENT_SUFFIX is appended to the default User-Agent for Azure HTTP clients.

Format: string

Example: "my-user-agent"

Required: False

MAX_CONCURRENT_RECONCILES

MAX_CONCURRENT_RECONCILES is the number of threads/goroutines dedicated to reconciling each resource type. If not specified, the default is 1.

IMPORTANT: Having MAX_CONCURRENT_RECONCILES set to N does not mean that ASO is limited to N interactions with Azure at any given time, because the control loop yields to another resource while it is not actively issuing HTTP calls to Azure. Any single resource only blocks the control-loop for its resource-type for as long as it takes to issue an HTTP call to Azure, view the result, and make a decision. In most cases the time taken to perform these actions (and thus how long the loop is blocked and preventing other resources from being acted upon) is a few hundred milliseconds to at most a second or two. In a typical 60s period, many hundreds or even thousands of resources can be managed with this set to 1.

MAX_CONCURRENT_RECONCILES applies to every registered resource type being watched/managed by ASO.

Format: int

Example: 2

Required: False

RATE_LIMIT_MODE

RateLimitMode configures the internal rate-limiting mode.

  • disabled (default): No ASO-controlled rate-limiting occurs. ASO will attempt to communicate with Azure and kube-apiserver as much as needed based on load. It will back off based on throttling from either kube-apiserver or Azure, but will not artificially limit its throughput.

  • bucket: Uses a token-bucket algorithm to rate-limit reconciliations. Note that this limits how often the operator performs a reconciliation, but not every reconciliation triggers a call to kube-apiserver or Azure (though many do). Since this controls reconciles it can be used to coarsely control throughput and CPU usage of the operator, as well as the number of requests that the operator issues to Azure. Keep in mind that the Azure throttling limits (defined at https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling) differentiate between request types. Since a given reconcile for a resource may result in polling (a GET) or modification (a PUT) it’s not possible to entirely avoid Azure throttling by tuning these bucket limits.

    We don’t recommend enabling this mode by default.

    If enabling this mode, we strongly recommend doing some experimentation to tune these values to something to works for your specific need.

Format: disabled|bucket

Example: disabled or bucket

Required: False

RATE_LIMIT_QPS

RATE_LIMIT_QPS is the rate (per second) that the bucket is refilled. This value only has an effect if RATE_LIMIT_MODE is ‘bucket’.

Format: float

Example: 5

Required: False

RATE_LIMIT_BUCKET_SIZE

RATE_LIMIT_BUCKET_SIZE is the size of the bucket. This value only has an effect if RATE_LIMIT_MODE is ‘bucket’.

Format: int

Example: 200

Required: False