ASOv1 to ASOv2 migration guide

ASOv1 is deprecated. We strongly recommend migrating to ASOv2 to continue getting new features and support.

Assumptions

  • Migrating a single cluster.
    • If you have ASOv1 installed on multiple clusters, apply these steps to each cluster.
  • ASOv1 is installed in its default namespace azureoperator-system.
    • If it is installed into a different namespace, replace all occurrences of azureoperator-system in this document with the namespace it’s installed into.
  • ASOv2 is not yet installed in the cluster.
  • ASOv1 is managing a single subscription, so ASOv2 will also be managing that same single subscription via a single global credential.
    • See ASOv2 credential scope for other credential scopes that allow for managing multiple subscriptions and/or using multiple identities.

Performing the migration

Check the version of cert-manager

Ensure the version of cert-manager in your cluster supports v1 resources.

kubectl api-resources --api-group cert-manager.io`

should show v1 resources.

Install ASOv2

Follow the standard instructions. We recommend you use the same credentials as ASOv1 is currently using.

Stop ASOv1 reconciliation

Mark each ASOv1 resource with the skipreconcile=true annotation. This annotation ensures that ASOv1 will no longer update or delete the resource in question in Azure.

To annotate a specific resource:

kubectl annotate -n <namespace> storageaccount <name> skipreconcile=true

To annotate all ASOv1 resources in a given namespace:

kubectl annotate $(kubectl api-resources -o name | grep azure.microsoft.com | paste -sd "," - | xargs kubectl get -A -o name) -n <namespace> skipreconcile=true`

Once you have annotated the resources, double-check that they all have the skipreconcile annotation.

Use asoctl to import the resources from Azure into ASOv2

The commandline tool asoctl is a utility for working with ASO. Here we are going to use it to import your existing Azure resources into ASOv2.

Download the latest version of asoctl.

Export the resources you want to transfer using the asoctl import azure-resource command. import azure-resource imports the referenced resource and all its child resources by default. This means if you want to import the resources from a specific resource group being managed by ASOv1 you can just refer to the resource group.

When running this command it’s recommended you use at least asoctl v2.7.0 and pass the -namespace (or -n) and -annotation (or -a) arguments to import the resources to a particular namespace with the serviceoperator.azure.com/reconcile-policy: skip annotation.

asoctl import azure-resource /subscriptions/<subid>/resourceGroups/<rg> -o resources.yaml -namespace <namespace> -annotation serviceoperator.azure.com/reconcile-policy=skip

If you have a large number of resources you may want to also use the --output-folder option to render a single resource per file, rather than one file with all resources.

This should produce a file similar to the one shown here (this is a simplified file showing just a resource group and a single resource for the sake of brevity. It’s likely that your file is much larger):

---
apiVersion: resources.azure.com/v1api20200601
kind: ResourceGroup
metadata:
  name: aso-test
  namespace: ns1
  annotations:
    serviceoperator.azure.com/reconcile-policy: skip
spec:
  azureName: aso-test
  location: westus
---
apiVersion: storage.azure.com/v1api20230101
kind: StorageAccount
metadata:
  name: aso-test-storage1
  namespace: ns1
  annotations:
    serviceoperator.azure.com/reconcile-policy: skip
spec:
  accessTier: Hot
  allowBlobPublicAccess: false
  allowCrossTenantReplication: false
  azureName: asoteststorage1
  encryption:
    keySource: Microsoft.Storage
    services:
      blob:
        enabled: true
        keyType: Account
      file:
        enabled: true
        keyType: Account
  kind: StorageV2
  location: westus
  minimumTlsVersion: TLS1_0
  networkAcls:
    bypass: AzureServices
    defaultAction: Allow
  owner:
    name: aso-test
  sku:
    name: Standard_RAGRS
    tier: Standard
  supportsHttpsTrafficOnly: true

Examine resources.yaml to ensure it has the resources you expect

Once you have resources.yaml locally, examine it to ensure that it has the resources you expect.

Configure resources.yaml to export the secrets you need from Azure

These are secrets such as storage account keys which ASOv1 has automatically exported into a Kubernetes secret.

ASOv1 exports secrets from Azure according to the Secret naming rules. ASOv1 always exports these secrets, there is no user configuration required on either the name of the secret or its values. In contrast, ASOv2 requires the user to opt-in to secret export, via the spec.operatorSpec.secrets property.

You may need to configure resouces.yaml to export corresponding secrets. To determine if, for a given namespace, there are secrets being written by ASOv1 and consumed by your applications, check the following two things:

  1. Are there secrets in the namespace owned by ASO?
    • Use kubectl get secrets -n ns1 -o json | jq -r '.items[] | select(.metadata.ownerReferences[]? | select(.apiVersion | contains("azure.microsoft.com")))' to find ASOv1 owned secrets.
  2. Do you have pods or services consuming those secrets? If they aren’t being consumed anywhere, you don’t need them and they can be ignored/discarded.

Here’s an example of secrets created by ASOv1 for various resources:

kubectl get secrets -n ns1
NAME                                         TYPE     DATA   AGE
azuresqlserver-azuresql-migration-sample-1   Opaque   5      106m
cosmosdb-cosmosdb-sample-1                   Opaque   10     6h42m
eventhub-eventhub-sample-1                   Opaque   7      101m
rediscache-rediscache-sample-1               Opaque   2      80m
storageaccount-cutoverteststorage1           Opaque   5      105m

Secrets owned by ASOv1 will have an ownerReferences section like this:

  ownerReferences:
  - apiVersion: azure.microsoft.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Eventhub
    name: eventhub-sample-1
    uid: 0a034f50-3060-4114-98e6-b5085326da0d

Once you’ve identified the set of secrets which are exported by ASOv1 and which are being consumed by your applications, configure ASOv2 to export similar secrets by using the spec.operatorSpec.secrets field. See examples for examples of various resource types.

Configure resources.yaml to source secret values from Kubernetes

Some Azure services require user-supplied secrets, for example: Azure SQL Server requires an administrator password. ASOv1 generated these passwords for you, but in ASOv2 you control these passwords and their lifecycle (including rollover).

asoctl does not automatically include secret password fields in its output. Instead, you must supply a Kubernetes secret containing the password which ASOv2 will supply to Azure.

The following ASOv1 resources automatically generated usernames and passwords which you will need to take ownership of when migrating to ASOv2:

  • Azure SQL Server
  • Azure SQL User
  • My SQL Server
  • My SQL User
  • PostgreSQL SQL Server
  • PostgreSQL SQL User
  • VM
  • VMSS

The process for each of these resources is the same:

  1. Find the ASOv1 managed secret which contains the current password.
  2. Create a new secret containing the same password.
  3. Update the ASOv2 resources.yaml to supply the new password in the correct location.

See examples for examples of how to do this with various resource types.

Configure resources.yaml to have annotation serviceoperator.azure.com/reconcile-policy: skip

This can be done automatically by asoctl during resource import with the -annotation serviceoperator.azure.com/reconcile-policy=skip argument or manually by modifying resources.yaml afterward.

Apply resources.yaml to your cluster

This will create the ASOv2 resources. At this point you should be in a state where:

  • ASOv1 is not managing the resources because they have all been annotated with skipreconcile=true.
  • ASOv2 is not managing the resources because they have all been annotated with serviceoperator.azure.com/reconcile-policy=skip.

Check that the ASOv2 resources are in the correct namespaces and that you’re seeing the secrets and configmaps you expect (these will be exported by ASOv2 even when annotated with serviceoperator.azure.com/reconcile-policy=skip).

Remove the serviceoperator.azure.com/reconcile-policy=skip from the ASOv2 resources (one at a time if desired) and swap existing deployments to source data from the ASOv2 secrets/configmaps.

Rolling back

If at any point something isn’t working, you can roll back by marking the ASOv2 resource with the serviceoperator.azure.com/reconcile-policy=skip annotation and removing the skipreconcile=true annotation from the ASOv1 resource. Don’t forget to swap your deployments to use the secrets from ASOv1 if they had been updated to rely on the ASOv2 secrets.

Cleaning up

Once you’ve migrated the ASOv1 resources to ASOv2 and been running successfully for a while, you can delete the ASOv1 resources in Kubernetes with kubectl delete.

Examples

Gotchas

  • resourcegroup is the Kubernetes “short name” for both ASOv1 and ASOv2 resource groups. This means that running kubectl get resourcegroups is ambiguous. Kubernetes will use ASOv1’s resourcegroup asuming it was installed first. If ASOv1 and ASOv2 are installed into the same cluster, we recommend using the fully specified name for querying resourcegroups from ASOv2: kubectl get resourcegroup.resources.azure.com.
  • ASOv2 does not support keyVault secrets.
  • ASOv2 does not support MySQL single server or PostgreSQL single server, as these are being deprecated. See MySQL and PostgreSQL for more details.

Helpful commands

View all ASOv1 resources in the cluster:

kubectl api-resources -o name | grep azure.microsoft.com | paste -sd "," - | xargs kubectl get -A

View all ASOv2 resources in the cluster:

kubectl api-resources -o name | grep azure.com | paste -sd "," - | xargs kubectl get -A

Annotate a single ASOv1 resource to skip all reconciliation:

kubectl annotate -n ns1 storageaccount <name> skipreconcile=true

Annotate existing ASOv1 resources in namespace ns1 to skip all reconciliation:

kubectl annotate $(kubectl api-resources -o name | grep azure.microsoft.com | paste -sd "," - | xargs kubectl get -A -o name) -n ns1 skipreconcile=true

Find secrets owned/created by ASOv1:

kubectl get secrets -n ns1 -o json | jq -r '.items[] | select(.metadata.ownerReferences[]? | select(.apiVersion | contains("azure.microsoft.com")))'

Annotate existing ASOv2 resources in namespace ns1 to skip all reconciliation:

kubectl annotate $(kubectl api-resources -o name | grep azure.com | paste -sd "," - | xargs kubectl get -A -o name) -n ns1 serviceoperator.azure.com/reconcile-policy=skip

Clear the serviceoperator.azure.com/reconcile-policy=skip annotation:

kubectl annotate $(kubectl api-resources -o name | grep azure.com | paste -sd "," - | xargs kubectl get -A -o name) -n ns1 serviceoperator.azure.com/reconcile-policy-