Adding a new code generated resource to ASO v2
This document discusses how to add a new resource to the ASO v2 code generation configuration. Check out this PR if you’d like to see what the end product looks like.
What resources can be code generated?
Any ARM resource can be generated. There are a few ways to determine if a resource is an ARM resource:
- If the resource is defined in the ARM template JSON schema, or the auto-generated ARM template JSON schema it is an ARM resource.
- If the resource is defined in a
resource-manager
folder in the Azure REST API specs repo, it is an ARM resource.
If the resource is not in either of the above places and cannot be deployed via an ARM template, then it’s not an ARM resource and currently cannot be code generated.
Determine a resource to add
There are three key pieces of information required before adding a resource to the code generation configuration file, and each of them can be found in the
ARM template JSON schema or the auto-generated ARM template JSON schema. To get started, find the entry for the resource in one of the two templates mentioned. For example, the entry for Azure Network Security Groups looks like this:
{ "$ref": "https://schema.management.azure.com/schemas/2020-11-01/Microsoft.Network.json#/resourceDefinitions/networkSecurityGroups" },
Note: In many cases there will be multiple entries for the same resource, each with a different api-version
. It is strongly recommended that you use the latest available non-preview api-version
.
-
The
name
of the resource.You usually know this going in. In our example above, the name of the resource is
networkSecurityGroups
. -
The
group
the resource is in.This is usually named after the Azure service, for example
resources
ordocumentdb
. In our example entry from above, this isnetwork
. -
The
api-version
of the resource.This is usually a date, sometimes with a
-preview
suffix. In our example entry from above, this is2020-11-01
.
Adding the resource to the code generation configuration file
The code generation configuration file is located here. To add a new resource to this file, find the objectModelConfiguration
section of the file.
Find the configuration for the group
you want; if it’s not there, create a new one, inserting it into the existing list in alphabetical order. Within the group, find the version
you want; again, create a new one if it’s not already there.
Add your new resource to the list for that version, including the directive $export: true
nested beneath. The final result should look like this:
<group>:
<version>:
<resource name>: # singular, typically just remove the trailing "s"
$export: true
For example, taking the Azure Network Security Groups sample from above:
network:
2020-11-01:
NetworkSecurityGroup:
$export: true
If ASO was already configured to generate resources from this group (or version), you will need to add your new configuration around the existing values.
Run the code generator
Follow the steps in the contributing guide to set up your development environment.
Once you have a working development environment, run the task
command to run the code generator.
Fix any errors raised by the code generator
<Resource> looks like a resource reference but was not labelled as one
Example:
Replace cross-resource references with genruntime.ResourceReference: [“github.com/Azure/azure-service-operator/hack/generated/_apis/containerservice/v1alpha1api20210501/PrivateLinkResource.Id” looks like a resource reference but was not labelled as one.
To fix this error, determine whether the property in question is an ARM ID or not, and then update the objectModelConfiguration
section in the configuration file.
Find the section you added earlier, adding your property with an $armReference:
declaration nested below.
If the property is an ARM ID, use $armReference: true
to flag that property as a reference:
network:
2020-11-01:
NetworkSecurityGroup:
$export: true
PrivateLinkResource:
$armReference: true # the property IS an ARM reference
If the property is not an ARM ID, use $armReference: false
instead:
network:
2020-11-01:
NetworkSecurityGroup:
$export: true
PrivateLinkResource:
$armReference: false # the property IS NOT an ARM reference
TODO: expand on other common errors
Examine the generated resource
After running the generator, the new resource you added should be in the apis directory.
Have a look through the files in the directory named after the group
and version
of the resource that was added.
In our NetworkSecurityGroups
example, the best place to start is /v2/api/network/v1alpha1api20201101/network_security_group_types_gen.go
There may be other resources that already exist in that same directory - that’s expected if ASO already supported some resources from that provider and API version.
Starting with the network_security_group_types_gen.go
file, find the struct representing the resource you just added. It should be near the top and look something like this:
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:storageversion
// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].status"
// +kubebuilder:printcolumn:name="Reason",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].reason"
// +kubebuilder:printcolumn:name="Message",type="string",JSONPath=".status.conditions[?(@.type=='Ready')].message"
//Generated from: https://schema.management.azure.com/schemas/2020-11-01/Microsoft.Network.json#/resourceDefinitions/networkSecurityGroups
type NetworkSecurityGroup struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec NetworkSecurityGroups_Spec `json:"spec,omitempty"`
Status NetworkSecurityGroup_Status_NetworkSecurityGroup_SubResourceEmbedded `json:"status,omitempty"`
}
Look over the Spec
and Status
types and their properties (and the properties of their properties and so-on).
The Azure REST API specs and ARM template JSON schemas which these types were derived from are not perfect. Sometimes they mark a readonly
property as mutable or have another error or mistake.
Common issues
This is a non-exhaustive list of common issues which may need to be fixed in our configuration file.
Properties that should have been marked read-only but weren’t
There is a section in the config detailing a number of these errors which have been fixed. Look for “Deal with properties that should have been marked readOnly but weren’t”.
Here’s an example, where DocumentDB missed setting the provisioningState
property to read-only. We override it in the config:
- group: documentdb
name: Location # This type is subsequently flattened into NamespacesTopics_Spec
property: ProvisioningState
remove: true
because: This property should have been marked readonly but wasn't.
Types that can’t be found
If you get an error indicating the generator can’t find a type, but you’re sure it exists:
E1214 10:34:15.476761 95884 gen_kustomize.go:111]
Error during code generation:
failed during pipeline stage 23/67 [filterTypes]:
Apply export filters to reduce the number of generated types:
group cdn: version 2021-06-01:
type DodgyResource not seen (did you mean ResourceReference?):
type DodgyResource: $exportAs: ReputableResource not consumed
It’s possible the submodule v2/specs/azure-rest-api-specs
is out of date. Try running git submodule update --init --recursive
to update the submodule.
Debugging
Sometimes it is useful to see what each stage of the generator pipeline has changed. To write detailed debug logs detailing internal stage after each stage of the pipeline has run, use the --debug
flag to specify the group (or groups) to include.
PS> aso-gen gen-types azure-arm.yaml --debug network
I0622 12:28:01.913420 5572 gen_types.go:49] Debug output will be written to the folder C:\...\Temp\aso-gen-debug-1580100077
... elided ...
I0622 12:29:15.836643 5572 gen_types.go:53] Debug output is available in folder C:\...\Temp\aso-gen-debug-1580100077
The volume of output is high, so the logs are written into a new temp directory. As shown, the name of this directory is included in the output of the generator twice, once at the start and again at the end.
A separate output file is generated after each pipeline stage runs, allowing for easy comparison to identify what each stage achieves.
In this screenshot, I’m comparing the output after stage 44 with the output after stage 52, reviewing the results of all the intervening stages.
Normal use of the --debug
flag is to specify a single output group (e.g. --debug network
) but you can also specify multiple groups using semicolons (e.g. --debug network;compute
) or wildcards (e.g. --debug db*
).
Determine if the resource has secrets generated by Azure
Some resources, such as microsoft.storage/accounts
and microsoft.documentdb/databaseAccounts
, have keys and endpoints generated by
Azure. Unfortunately, there is no good way to automatically detect these in the specs which the code generator runs on.
You must manually identify if the resource you are adding has keys (or endpoints) which users will want exported to a secret store.
If the resource in question does have Azure generated secrets or endpoints, identify those endpoints in the configuration file
by specifying $azureGeneratedSecrets
.
Our example resource above does not have any Azure generated secrets. As mentioned above, microsoft.documentdb/databaseAccounts
has
Azure generated secrets. Here is the snippet from the configuration file showing how they were configured.
documentdb:
2021-05-15:
DatabaseAccount:
$export: true
$azureGeneratedSecrets:
- PrimaryMasterKey
- SecondaryMasterKey
- PrimaryReadonlyMasterKey
- SecondaryReadonlyMasterKey
- DocumentEndpoint
Since these properties are manually configured, they must also be retrieved from Azure manually. This can be done using the extension framework supported by the operator. Use the documentdb secrets extension as a template for authoring an extension for your resource.
Write a CRUD test for the resource
The best way to do this is to start from an existing test and modify it to work for your resource. It can also be helpful to refer to examples in the ARM templates GitHub repo.
Run the CRUD test for the resource and commit the recording
See the code generator README for how to run recording tests.
Add a new sample
The samples are located in the samples directory. There should be at least one sample for each kind of supported resource. These currently need to be added manually. It’s possible in the future we will automatically generate samples similar to how we automatically generate CRDs and types, but that doesn’t happen today.
Run test for added sample and commit the recording
The added new sample needs to be tested and recorded. To perform that, follow the steps below:
- If a recording for the test already exists, delete it.
Look in the recordings directory for a file with the same name as your new test.
Typically these are namedTest_<GROUP>_<VERSION_PREFIX>_CreationAndDeletion.yaml
. For example, If we’re adding sample for NetworkSecurityGroup resource, check forTest_Network_v1beta_CreationAndDeletion.yaml
- if recording exists, delete it. Else move to next step
- run
TEST_FILTER=Test_Samples_CreationAndDeletion task controller:test-integration-envtest
to re-record the test.
Send a PR
You’re all done!