Adding a new code generated resource to ASO v2

This document discusses how to add a new resource to the ASO v2 code generation configuration. Check out this PR if you’d like to see what the end product looks like.

What resources can be code generated?

Any ARM resource can be generated. There are a few ways to determine if a resource is an ARM resource:

  1. If the resource is defined in the ARM template JSON schema, or the auto-generated ARM template JSON schema it is an ARM resource.
  2. If the resource is defined in a resource-manager folder in the Azure REST API specs repo, it is an ARM resource.

If the resource is not in either of the above places and cannot be deployed via an ARM template, then it’s not an ARM resource and currently cannot be code generated.

Determine a resource to add

There are three key pieces of information required before adding a resource to the code generation configuration file, and each of them can be found in the ARM template JSON schema or the auto-generated ARM template JSON schema. To get started, find the entry for the resource in one of the two templates mentioned. For example, the entry for Azure Network Security Groups looks like this: { "$ref": "https://schema.management.azure.com/schemas/2020-11-01/Microsoft.Network.json#/resourceDefinitions/networkSecurityGroups" },

Note: In many cases there will be multiple entries for the same resource, each with a different api-version. It is strongly recommended that you use the latest available non-preview api-version.

  1. The name of the resource.

    You usually know this going in. In our example above, the name of the resource is networkSecurityGroups.

  2. The group the resource is in.

    This is usually named after the Azure service, for example resources or documentdb. In our example entry from above, this is network.

  3. The api-version of the resource.

    This is usually a date, sometimes with a -preview suffix. In our example entry from above, this is 2020-11-01.

Adding the resource to the code generation configuration file

The code generation configuration file is located here. To add a new resource to this file, find the objectModelConfiguration section of the file.

Find the configuration for the group you want; if it’s not there, create a new one, inserting it into the existing list in alphabetical order. Within the group, find the version you want; again, create a new one if it’s not already there.

Add your new resource to the list for that version, including the directive $export: true nested beneath. The final result should look like this:

<group>:
  <version>:
    <resource name>: # singular, typically just remove the trailing "s"
      $export: true

For example, taking the Azure Network Security Groups sample from above:

network:
  2020-11-01:
    NetworkSecurityGroup:
      $export: true

If ASO was already configured to generate resources from this group (or version), you will need to add your new configuration around the existing values.

Run the code generator

Follow the steps in the contributing guide to set up your development environment. Once you have a working development environment, run the task command to run the code generator.

Fix any errors raised by the code generator

<Resource> looks like a resource reference but was not labelled as one

Example:

Replace cross-resource references with genruntime.ResourceReference: [“github.com/Azure/azure-service-operator/hack/generated/_apis/containerservice/v1alpha1api20210501/PrivateLinkResource.Id” looks like a resource reference but was not labelled as one.

To fix this error, determine whether the property in question is an ARM ID or not, and then update the objectModelConfiguration section in the configuration file.

Find the section you added earlier, adding your property with an $armReference: declaration nested below.

If the property is an ARM ID, use $armReference: true to flag that property as a reference:

network:
  2020-11-01:
    NetworkSecurityGroup:
      $export: true
      PrivateLinkResource: 
        $armReference: true # the property IS an ARM reference

If the property is not an ARM ID, use $armReference: false instead:

network:
  2020-11-01:
    NetworkSecurityGroup:
      $export: true
      PrivateLinkResource: 
        $armReference: false # the property IS NOT an ARM reference

TODO: expand on other common errors

Examine the generated resource

After running the generator, the new resource you added should be in the apis directory.

Have a look through the files in the directory named after the group and version of the resource that was added. In our NetworkSecurityGroups example, the best place to start is /v2/api/network/v1alpha1api20201101/network_security_group_types_gen.go There may be other resources that already exist in that same directory - that’s expected if ASO already supported some resources from that provider and API version.

Starting with the network_security_group_types_gen.go file, find the struct representing the resource you just added. It should be near the top and look something like this:

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:storageversion
// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].status"
// +kubebuilder:printcolumn:name="Reason",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].reason"
// +kubebuilder:printcolumn:name="Message",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].message"
//Generated from: https://schema.management.azure.com/schemas/2020-11-01/Microsoft.Network.json#/resourceDefinitions/networkSecurityGroups
type NetworkSecurityGroup struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec              NetworkSecurityGroups_Spec                                           `json:"spec,omitempty"`
	Status            NetworkSecurityGroup_Status_NetworkSecurityGroup_SubResourceEmbedded `json:"status,omitempty"`
}

Look over the Spec and Status types and their properties (and the properties of their properties and so-on). The Azure REST API specs and ARM template JSON schemas which these types were derived from are not perfect. Sometimes they mark a readonly property as mutable or have another error or mistake.

Common issues

This is a non-exhaustive list of common issues which may need to be fixed in our configuration file.

Properties that should have been marked read-only but weren’t

There is a section in the config detailing a number of these errors which have been fixed. Look for “Deal with properties that should have been marked readOnly but weren’t”.

Here’s an example, where DocumentDB missed setting the provisioningState property to read-only. We override it in the config:

  - group: documentdb
    name: Location  # This type is subsequently flattened into NamespacesTopics_Spec
    property: ProvisioningState
    remove: true
    because: This property should have been marked readonly but wasn't.

Debugging

Sometimes it is useful to see what each stage of the generator pipeline has changed. To write detailed debug logs detailing internal stage after each stage of the pipeline has run, use the --debug flag to specify the group (or groups) to include.

PS> aso-gen gen-types azure-arm.yaml --debug network
I0622 12:28:01.913420    5572 gen_types.go:49] Debug output will be written to the folder C:\...\Temp\aso-gen-debug-1580100077
... elided ...
I0622 12:29:15.836643    5572 gen_types.go:53] Debug output is available in folder C:\...\Temp\aso-gen-debug-1580100077

The volume of output is high, so the logs are written into a new temp directory. As shown, the name of this directory is included in the output of the generator twice, once at the start and again at the end.

A separate output file is generated after each pipeline stage runs, allowing for easy comparison to identify what each stage achieves.

Debug Output

In this screenshot, I’m comparing the output after stage 44 with the output after stage 52, reviewing the results of all the intervening stages.

Stage diffs

Normal use of the --debug flag is to specify a single output group (e.g. --debug network) but you can also specify multiple groups using semicolons (e.g. --debug network;compute) or wildcards (e.g. --debug db*).

Determine if the resource has secrets generated by Azure

Some resources, such as microsoft.storage/accounts and microsoft.documentdb/databaseAccounts, have keys and endpoints generated by Azure. Unfortunately, there is no good way to automatically detect these in the specs which the code generator runs on.

You must manually identify if the resource you are adding has keys (or endpoints) which users will want exported to a secret store. If the resource in question does have Azure generated secrets or endpoints, identify those endpoints in the configuration file by specifying $azureGeneratedSecrets.

Our example resource above does not have any Azure generated secrets. As mentioned above, microsoft.documentdb/databaseAccounts has Azure generated secrets. Here is the snippet from the configuration file showing how they were configured.

  documentdb:
    2021-05-15:
      DatabaseAccount:
        $export: true
        $azureGeneratedSecrets:
          - PrimaryMasterKey
          - SecondaryMasterKey
          - PrimaryReadonlyMasterKey
          - SecondaryReadonlyMasterKey
          - DocumentEndpoint

Since these properties are manually configured, they must also be retrieved from Azure manually. This can be done using the extension framework supported by the operator. Use the documentdb secrets extension as a template for authoring an extension for your resource.

Write a CRUD test for the resource

The best way to do this is to start from an existing test and modify it to work for your resource. It can also be helpful to refer to examples in the ARM templates GitHub repo.

Run the CRUD test for the resource and commit the recording

See the code generator README for how to run recording tests.

Add a new sample

The samples are located in the samples directory. There should be at least one sample for each kind of supported resource. These currently need to be added manually. It’s possible in the future we will automatically generate samples similar to how we automatically generate CRDs and types, but that doesn’t happen today.

Run test for added sample and commit the recording

The added new sample needs to be tested and recorded. To perform that, follow the steps below:

  1. If a recording for the test already exists, delete it.
    Look in the recordings directory for a file with the same name as your new test.
    Typically these are named Test_<GROUP>_<VERSION_PREFIX>_CreationAndDeletion.yaml. For example, If we’re adding sample for NetworkSecurityGroup resource, check for Test_Network_v1beta_CreationAndDeletion.yaml
  2. if recording exists, delete it. Else move to next step
  3. run TEST_FILTER=Test_Samples_CreationAndDeletion task controller:test-integration-envtest to re-record the test.

Send a PR

You’re all done!