Adding a new code generated resource to ASO v2

This document discusses how to add a new resource to the ASO v2 code generation configuration. Check out this PR if you’d like to see what the end product looks like.

What resources can be code generated?

Any ARM resource can be generated. If you’re not sure if the resource you are interested in is an ARM resource, check if it is defined in a resource-manager folder in the Azure REST API specs repo. If it is, it’s an ARM resource.

Determine a resource to add

There are three key pieces of information required before adding a resource to the code generation configuration file, and each of them can be found in the Azure REST API specs repo. We will walk through an example of adding Azure Synapse Workspace.

Note: In many cases there will be multiple API versions for a given resource, you can see this in the Synapse folder for example, there are 4 API versions as of April 2023. It is strongly recommended that you use the latest available non-preview api-version when choosing the version of the resource to add.

The three key pieces of information needed to code generate a resource are:

  1. The name of the resource.

    You usually know this going in. In our example above, the name of the resource is workspaces. If you’re not sure, look in the Swagger/OpenAPI specification file for the service and find the documented PUT for the resource you’re interested in. The resource name will be the second to last section of the URL.

  2. The group the resource is in.

    This is named after the Azure service, for example resources or documentdb. In our example entry from above, this is synapse (from Microsoft.Synapse, the provider documented in the resource URL).

  3. The api-version of the resource.

    This is usually a date, sometimes with a -preview suffix. In our example entry from above, this is 2021-06-01.

Adding the resource to the code generation configuration file

The code generation configuration file is located here. To add a new resource to this file, find the objectModelConfiguration section of the file.

Find the configuration for the group you want; if it’s not there, create a new one, inserting it into the existing list in alphabetical order. Within the group, find the version you want; again, create a new one if it’s not already there.

Add your new resource to the list for that version, including the directive $export: true nested beneath. You must also include the $supportedFrom: annotation. This should be the next release which will contain support for the resource in question. You can determine the name of the next ASO release by looking at our milestones.

The final result should look like this:

<group>:
  <version>:
    <resource name>: # singular, typically just remove the trailing "s"
      $export: true
      $supportedFrom: <the upcoming release>

For example, taking the Azure Synapse Workspace sample from above:

synapse:
 2021-06-01:
   Workspace:
     $export: true
     $supportedFrom: v2.0.0

If ASO was already configured to generate resources from this group (or version), you will need to add your new configuration around the existing values.

Run the code generator

Follow the steps in the contributing guide to set up your development environment. Once you have a working development environment, run the task command to run the code generator.

Fix any errors raised by the code generator

<Resource> looks like a resource reference but was not labelled as one

Example:

Replace cross-resource references with genruntime.ResourceReference:
[“github.com/Azure/azure-service-operator/v2/apis/containerservice/v1api20210501/PrivateLinkResource.Id” looks like a resource reference but was not labelled as one.

To fix this error, determine whether the property in question is an ARM ID or not, and then update the objectModelConfiguration section in the configuration file.

Find the section you added earlier, adding your property with an $armReference: declaration nested below.

If the property is an ARM ID, use $armReference: true to flag that property as a reference:

network:
  2020-11-01:
    NetworkSecurityGroup:
      $export: true
      PrivateLinkResource: 
        $armReference: true # the property IS an ARM reference

If the property is not an ARM ID, use $armReference: false instead:

network:
  2020-11-01:
    NetworkSecurityGroup:
      $export: true
      PrivateLinkResource: 
        $armReference: false # the property IS NOT an ARM reference

$export specified for type but not consumed

This error is produced when you’ve added configuration for a new resource to the objectModelConfiguration section of the configuration file, but that configuration had no effect. (ASO prefers to make configuration errors visible rather than silently continue while doing the wrong thing).

The most likely cause of this error is a typo in the name of the group, version, or kind specified. ASO will try to help by listing the closest match - for example

version 2021-05-01-preview not seen (did you mean 2018-05-01-preview?)

Double check the names you’ve used and correct any typos.

If you’re importing a preview version of a resource, you may need to modify the typeFilters section at the top of the file. Early in the development of ASO we discovered that some preview versions are poorly formed - filtering them out was a straightforward way to avoid problems.

Type filters are applied in order, with the first matching filter being used. This one prunes all preview versions:

  - action: prune
    version: '*preview'
    because: preview SDK versions are excluded by default (they often do very strange things)

To allow a specific preview version, add a new filter to the list and make sure it appears before the prune filter (so it’s applied first):

  - action: include
    group: keyvault
    version: v*20210401preview
    because: We want to support keyvault which is only available in preview version

Be sure to give a good reason for including the preview version so that other maintainers know why it’s there.

If you have multiple preview versions for a single group, you can (and should) combine them together into a single filter. All of the filter fields allow using semicolons (;) to separate multiple values.

  - action: include
    group: servicebus
    version: v*20210101preview;v*20221001preview
    because: We want to export these particular preview versions

TODO: expand on other common errors

Examine the generated resource

After running the generator, the new resource you added should be in the apis directory.

Have a look through the files in the directory named after the group and version of the resource that was added. In our Workspaces example, the best place to start is v2/api/synapse/v1api20210601/workspace_types_gen.go There may be other resources that already exist in that same directory - that’s expected if ASO already supported some resources from that provider and API version.

Starting with the workspace_types_gen.go file, find the struct representing the resource you just added. It should be near the top and look something like this:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].status"
// +kubebuilder:printcolumn:name="Severity",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].severity"
// +kubebuilder:printcolumn:name="Reason",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].reason"
// +kubebuilder:printcolumn:name="Message",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].message"
// Generator information:
// - Generated from: /synapse/resource-manager/Microsoft.Synapse/stable/2021-06-01/workspace.json
// - ARM URI: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}
type Workspace struct {
metav1.TypeMeta   `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec              Workspace_Spec   `json:"spec,omitempty"`
Status            Workspace_STATUS `json:"status,omitempty"`
}

Look over the Spec and Status types and their properties (and the properties of their properties and so-on). The Azure REST API specs which these types were derived from are not perfect. Sometimes they mark a readonly property as mutable or have another error or mistake.

Common issues

This is a non-exhaustive list of common issues which may need to be fixed in our configuration file.

Properties that should have been marked read-only but weren’t

There is a section in the config detailing a number of these errors which have been fixed. Look for “Deal with properties that should have been marked readOnly but weren’t”.

Here’s an example, where DocumentDB missed setting the provisioningState property to read-only. We override it in the config:

  - group: documentdb
    name: Location  # This type is subsequently flattened into NamespacesTopics_Spec
    property: ProvisioningState
    remove: true
    because: This property should have been marked readonly but wasn't.

Types that can’t be found

If you get an error indicating the generator can’t find a type, but you’re sure it exists:

E1214 10:34:15.476761   95884 gen_kustomize.go:111] 
Error during code generation:
failed during pipeline stage 23/67 [filterTypes]: 
Apply export filters to reduce the number of generated types: 
group cdn: version 2021-06-01: 
type DodgyResource not seen (did you mean ResourceReference?): 
type DodgyResource: $exportAs: ReputableResource not consumed

It’s possible the submodule v2/specs/azure-rest-api-specs is out of date. Try running git submodule update --init --recursive to update the submodule.

Debugging

Sometimes it is useful to see what each stage of the generator pipeline has changed. To write detailed debug logs detailing internal stage after each stage of the pipeline has run, use the --debug flag to specify which type definitions to include.

To see debug output including all types in the network group, run:

PS> aso-gen gen-types azure-arm.yaml --debug network
I0622 12:28:01.913420    5572 gen_types.go:49] Debug output will be written to the folder C:\...\Temp\aso-gen-debug-1580100077
... elided ...
I0622 12:29:15.836643    5572 gen_types.go:53] Debug output is available in folder C:\...\Temp\aso-gen-debug-1580100077

The volume of output is high, so the logs are written into a new temp directory. As shown, the name of this directory is included in the output of the generator twice, once at the start and again at the end.

A separate output file is generated after each pipeline stage runs, allowing for easy comparison to identify what each stage achieves.

Debug Output

In this screenshot, I’m comparing the output after stage 44 with the output after stage 52, reviewing the results of all the intervening stages.

Stage diffs

The debug flag accepts a variety of values:

  • A single group: --debug network
  • Multiple groups: --debug network;compute
    (Use a semicolon to separate groups)
  • A specific version of a group: --debug network/v1api20201101
    (Use a slash to separate group and version; versions are specified as package names)
  • Multiple groups and versions: --debug network/v1api20201101;network/v1api20220701 (Again, use a semicolon to separate)
  • Wildcards to match multiple groups: --debug db*

Determine if the resource has secrets generated by Azure

Some resources, such as microsoft.storage/accounts and microsoft.documentdb/databaseAccounts, have keys and endpoints generated by Azure. Unfortunately, there is no good way to automatically detect these in the specs which the code generator runs on.

You must manually identify if the resource you are adding has keys (or endpoints) which users will want exported to a secret store. If the resource in question does have Azure generated secrets or endpoints, identify those endpoints in the configuration file by specifying $azureGeneratedSecrets.

Our example resource above does not have any Azure generated secrets. As mentioned above, microsoft.documentdb/databaseAccounts has Azure generated secrets. Here is the snippet from the configuration file showing how they were configured.

  documentdb:
    2021-05-15:
      DatabaseAccount:
        $export: true
        $azureGeneratedSecrets:
          - PrimaryMasterKey
          - SecondaryMasterKey
          - PrimaryReadonlyMasterKey
          - SecondaryReadonlyMasterKey
          - DocumentEndpoint

Since these properties are manually configured, they must also be retrieved from Azure manually. This can be done using the extension framework supported by the operator. Use the documentdb secrets extension as a template for authoring an extension for your resource.

Determine if the resource has secrets for input

Some resources take secrets as input from the user, for example when creating a virtual machine the user may supply a pasword. This is a string field but because it represents a secret value we don’t want to just stick it on the Spec as a string. It needs to be a genruntime.SecretReference (a reference to a secret) instead, so that users can safely supply their secret to the operator without it being accessible via plain-text.

In most cases, the Swagger/OpenAPI document will annotate these properties with an extension x-ms-secret, which indicates that value is a secret. The ASO code-generator uses that annotation to automatically transform the annotated field to a genruntime.SecretReference.

In some cases, the resources Swagger specification is missing this annotation. If that happens, you can use an override in our azure-arm.yaml to force a particular property to be treated as a secret. For example, under synapse:

WorkspaceProperties:
  SqlAdministratorLoginPassword:
    $isSecret: true

Write a CRUD test for the resource

The best way to do this is to start from an existing test and modify it to work for your resource. It can also be helpful to refer to examples in the ARM templates GitHub repo.

Every new resource should have a handwritten test as there is always the possibility that the way a particular resource provider behaves will change with a new version.

Given that we don’t want to have to maintain tests for every version of every resource, and each additional test makes our CI test suite take longer, consider removing tests for older versions of resources when we add tests for newer versions. This is a judgment call, and we recommend discussion with the team first.

As an absolute minimum, we want to have tests for

  • the latest stable version of the resource;
  • the prior stable version of the resource; and
  • the latest preview version of the resource.

These tests live in the v2/internal/controllers folder and should follow the following naming convention:

<group>_<subject>_<scenario>_<version>_test.go

More information on the naming convention can be found in that folders README.

Record the test passing

See the code generator test README for how to run tests and record their HTTP interactions to allow replay.

Add a new sample

The samples are located in the samples directory. There should be at least one sample for each kind of supported resource. These currently need to be added manually. It’s possible in the future we will automatically generate samples similar to how we automatically generate CRDs and types, but that doesn’t happen today.

Run test for added sample and commit the recording

The added new sample needs to be tested and recorded.

If a recording for the test already exists, delete it.
Look in the recordings directory for a file with the same name as your new test.
Typically these are named Test_<GROUP>_<VERSION_PREFIX>_CreationAndDeletion.yaml. For example, if we’re adding sample for NetworkSecurityGroup resource, check for Test_Network_v1beta_CreationAndDeletion.yaml

Run the test and record it:

TEST_FILTER=Test_Samples_CreationAndDeletion task controller:test-integration-envtest

Some Azure resources take longer to provision or delete than the default test timeout of 15m, so you may need to add the TIMEOUT environment variable to the command above. For example, to give your test a 60m timeout, use:

TIMEOUT=60m TEST_FILTER=Test_Samples_CreationAndDeletion task controller:test-integration-envtest

Send a PR

You’re all done!