2023-02: Adoption Policy
Resources which exist in Azure prior to being created in Kubernetes are “adopted” by the operator when the resource
is created in Kubernetes. Today this adoption process is automatic and implicit. As a user there isn’t an easy way
to tell if a resource you’re about to
kubectl apply already exists in Azure (will be adopted) or does not exist in
Azure (will be created).
We already have support for reconcile-policy which
is related to adoption. Some of
reconcile-policy’s primary use cases are for when the user does not want ASO to
fully manage an adopted resource.
The following are principles we should follow related to resource adoption:
The operator should adopt resources by default: This is standard goal-seeking/desired-state behavior. If you create a resource in Kubernetes that says: “I expect there to be a Virtual Machine that looks like this at location X” and that Virtual Machine already exists, then we are in the desired state. The alternative would be to error if the resource already exists, which goes against the Kubernetes model.
Users must be able to instruct the operator to stop making changes to the backing Azure resource: This is enabled by
This is useful for the following scenarios:
- The operator is doing something wrong and the user needs a way to stop the operator from acting.
- Pausing the operator temporarily while they make some manual fix via the Azure portal.
- Creating a resource in ASO purely to get ownership references to work. The resource in question is not managed by ASO at all. It was created and managed some other way (portal, terraform, az-cli, etc).
Users must be able to instruct the operator to skip deletion of the backing resource: This is useful for scenarios such as moving ownership of resources from ASO in cluster A to ASO in cluster B, or as a safety net to protect stateful resources such as databases from accidental deletion. This is enabled by
Open question: what should the default
reconcile-policy be for resources which are adopted?
There are two main options.
Option 1: The default
reconcile-policy for adopted resources is
manage is also the default
reconcile-policy for resources ASO creates, so there is no difference between
creation and adoption with this option.
This means ASO automatically adopts, updates, and will delete backing Azure resources as normal unless the user
- It’s easy to explain what the behavior is. Simplistic users don’t have to understand
reconcile-policy(they won’t ever see it unless they set it themselves).
- Matches ARM/BICEP, raw REST, and SDK behavior, as well as some of our competitors (Crossplane and Google Config Connector) as spelled out below.
- Is the current default
- Users have no good way to know their resource existed previously. They may
kubectl deleteit without realizing that other things depend on this resource. Deletion (and some mutation) is destructive in Azure so lost data, etc cannot easily be recovered.
This sort-of matches the behavior of ARM templates and SDKs. If you call the
DELETE API the resource is gone.
Some discussion of this topic has happened over in ACK.
Crossplane doesn’t support any notion of
reconcile-policy at all.
kubectl delete always deletes the backing resource no matter what.
GCP does it this way:
When you create a resource, Config Connector creates the resource if it doesn’t exist. If a Google Cloud resource already exists with the same name, then Config Connector acquires the resource and manages it.
GCP also supports a
reconcile-policy-like annotation, so doing it like this would basically be doing it exactly like GCP.
Kubernetes has a similar concept for
reclaimPolicy, whose default
Delete for dynamically provisioned PVs, but
Retain for manually created PVs.
Option 2: The default
reconcile-policy for adopted resource is
ASO detects that a resource previously existed and sets
reconcile-policy automatically to
- Defaults to a more conservative
reconcile-policyto protect users from accidentally deleting things they may not have wanted to delete.
- Higher complexity. Users will be exposed to
reconcile-policyas set by the operator (when a resource is adopted) so need to understand it.
- Bias towards safety may cause resource user thinks was deleted to continue existing (and billing) in Azure. Not a data loss though.
reconcile-policywould have split ownership between user and operator, which means deciding if we should trigger reconciliation events on its change is tricky (we don’t want our defaulting of it to trigger an event).
There’s some discussion of a similar feature over in ACK. Jay Pipes had an interesting comment:
In the case of ACK and the user experience of resource deletion, I posit that the behaviour of our interface has no surprises and is actually hard to misuse. It doesn’t have surprises because our interface to delete a resource is to call kubectl delete on that resource. This is as non-surprising as it gets. It is hard to misuse because calling kubectl delete is an explicit action the user has to take and it’s not a confusing verb or semantic. “delete” means to remove the resource.
It’s important to note that ACK does adoption differently than ASO though (see example), so adoption is much more explicit for them. We don’t want to have such explicit adoption though as covered in principle #1: The operator should adopt resources by default
CAPZ does something sort-of like this using Azure
tags, but only for a select set of resources which they support
“BYO-resource” for. Things like VNET. Since some child resources don’t support tags they apply the parent tag to
all of the children automatically, which makes sense for their use-case but doesn’t make as much sense for ASO.
Open question: Should the adoption behavior be configurable at the operator (global) or per-resource level?
Some users (such as CAPZ) may want adoption semantics different than the defaults we choose. Even on the PR that added this design proposal different users disagreed on the behavior they wanted.
Option 3: No configuration at the operator or per-resource level
- Simple - value of this cannot be understated. The more configuration users have to understand the harder the operator becomes to adopt or understand.
- Fails to cater to some legitimate use-cases.
Option 4: Allow configuration at the operator or per-resource level
This option would just be configuring the behavior of the operator in case it found an existing resource in Azure. The behavior if ASO created the resource is unchanged.
The 3 scenarios here are, if the resource already exists in Azure when ASO goes to create it:
- Fully adopt the resource, mutate it if its configuration is different in ASO than in Azure, and delete
kubectl deleteis run.
- Partially adopt the resource. Mutations of the resource would be performed by ASO but
kubectl deletewill only delete the resource in Kubernetes and not in Azure.
- Use the existing resource and don’t make any changes to it to make it match the ASO shape, and do not delete it
kubectl deleteis run.
This is discussed in #2703
Option 4.1: adoption-policy annotation
A new property
ADOPTION_POLICY in the operator global secret, and an annotation on each
serviceoperator.azure.com/adoption-policy would allow users to control this behavior.
As with our other configurations, the more specific configuration would win.
adopt-if-exists: Corresponding to scenario 1 above.
detach-on-delete-if-exists: Corresponding to scenario 2 above.
skip-if-exists: Corresponding to scenario 3 above.
This would exist alongside the existing
reconcile-policy annotation. The most restrictive of the two options would
The full matrix of possibilities (where there is overlap) is:
||Resource already exists in Azure?||Behavior|
|Unspecified||Unspecified||Yes||Fully manages the resource¹|
|Unspecified||skip-if-exists||Yes||Skips managing the resource|
|Unspecified||skip-if-exists||No||Fully manages the resource|
|detach-on-delete||skip-if-exists||Yes||Skips managing the resource|
|detach-on-delete||skip-if-exists||No||Manages the resource, detaches on delete|
|skip||skip-if-exists||Yes||Skips managing the resource|
|skip||skip-if-exists||No||Skips managing the resource|
|Unspecified||detach-on-delete-if-exists||Yes||Manages the resource, detaches on delete|
|Unspecified||detach-on-delete-if-exists||No||Fully manages the resource|
|detach-on-delete||detach-on-delete-if-exists||Yes||Manages the resource, detaches on delete|
|detach-on-delete||detach-on-delete-if-exists||No||Manages the resource, detaches on delete|
|skip||detach-on-delete-if-exists||Yes||Skips managing the resource|
|skip||detach-on-delete-if-exists||No||Skips managing the resource|
¹: Actual behavior depends on earlier sections of this design
- Users need to understand this complicated table to figure out what is actually happening to their resource, it’s not immediately obvious based on the annotations.
- There’s no way to tell which option was actually chosen for users.
- We need to somehow remember which option we chose so we don’t think that a resource we created needs to be adopted after a crash or pod restart. That further increases the number of annotations used by this feature…
Option 4.2: reconcile-policy-if-exists annotation
A new property
RECONCILE_POLICY_IF_EXISTS in the operator global secret, and an annotation on each
serviceoperator.azure.com/reconcile-policy-if-exists would allow users to control this behavior.
reconcile-policy-if-exists’s are the same as the existing
manage: Corresponding to scenario 1 above.
detach-on-delete: Corresponding to scenario 2 above.
skip: Corresponding to scenario 3 above.
The implementation of
reconcile-policy-if-exists would basically be a delayed defaulting of
when reconciliation ran for the first time. The operator would check if the resource already exists and write the corresponding
reconcile-policy annotation to the resource if the resource existed.
- Uses a concept we already have.
- Possibility of collision between user specified
reconcile-policywritten by the operator because of
- We can mitigate this issue by rejecting some configurations with a webhook, although we have to be careful how we craft
that webhook because the operator itself needs to be able to set both. The easiest way to accomplish this would be with
a webhook that rejects requests that include both
reconcile-policy-if-existsannotations if the finalizer has not been added yet. Once the finalizer has been set, changing
reconcile-policy-if-existshas no effect, and all changes to the policy should flow via the
- An alternative mitigation would be that
reconcile-policyif the resource exists. This has the advantage of allowing configurations like
reconcile-policy-if-exists: manage, where a resource wouldn’t be created if it didn’t exist, but would be adopted and managed if it did already exist. This would also allow
reconcile-policy-if-exists: skip. Possible downsides here include us modifying user-specified annotations. It’s possible that this will cause problems with GitOps workflows as the users checked-in spec will have
reconcile-policy: <value>, but that may have been overwritten by the operator dynamically. Subsequent applications of the resource would likely overwrite
reconcile-policyback to its original value.
- We can mitigate this issue by rejecting some configurations with a webhook, although we have to be careful how we craft that webhook because the operator itself needs to be able to set both. The easiest way to accomplish this would be with a webhook that rejects requests that include both
- Excess events generated by us setting
reconcile-policy, which is an annotation we watch for updates. We can deal with this:
- If the
skipthis is safe as we don’t have to trigger reconciliation if the only modification to the resource is that the
reconcile-policyis being set to
skip. This helps avoid a situation where we reconcile once, issue a GET to Azure, see that the resource exists and update the
reconcile-policyannotation to be
skip, which then triggers another reconciliation and another GET to Azure.
- If the
detach-on-deletewe’re safe as those events only matter for delete so also don’t trigger reconciliation.
- If the
An implementation note on fault tolerance
If we choose either flavor of option 4 we need to ensure that we are fault-tolerant. We must avoid a situation where the operator:
- Checks if the resource in Azure exists and determines it does not
- Creates the resource in Azure
- Crashes or is restarted and does not ever persist to etcd the fact the resource didn’t already exist
- Restarts and thinks that the resource already existed, triggering the adoption logic on a resource ASO itself just created
This means there must be an etcd write after checking if the resource exists, before we actually issue a PUT to Azure.
Claim step in the operator reconcile path already caters to exactly this problem.
Unfortunately because ARM resources don’t universally support
If-Match headers +
etag we are open to a race where
the resource is created between the GET and the following PUT. There’s nothing we can do about that without optimistic
concurrency support on the server side.
I propose we choose option 1: Default
As mentioned in the discussion of that topic, it’s the default we currently have and so is non-breaking to adopt, and is how
ARM templates/BICEP, raw REST and the SDKs work today already. Extra protection against adoption/updating a resource that
already exists can be achieved with some of those technologies, but it’s not the default and requires the user to opt-in
Alongside that, I propose we choose option 4.2:
to allow users to configure this behavior at the operator level or at the per-resource level. This is the other side of the
coin, if we’re going to default to adopting we should give users who do not want that as the default a way to configure
the behavior they do want.
As part of option 4.2, we should also implement one of the mitigations for the issue of
reconcile-policy. The obvious choice is the webhook rejecting overlap. While this limits certain scenarios
reconcile-policy-if-exists: manage +
reconcile-policy: skip, there’s no known use-case for these scenarios.
The advantage of rejecting things now is that we can always open it up and be less restrictive in the future if there is
customer need. If instead we allowed both to be set then it’s hard to take it away.