Introduction

AAZDev is a tool that helps Azure CLI developers generate Atomic CLI commands from REST API Specifications.

Background

What is Azure CLI?

The full name of Azure CLI is Azure command-line interface. It’s a set of commands used to create and manage Azure resources.

Below is a common CLI command that will create an azure virtual network.

az network vnet create --name my-vnet --resource-group my-resource-group

Every CLI command consists of 2 parts:

• Command name: There are thousands of commands in CLI, which are organized into a tree. Its nodes are command groups, and its leaves are commands.

  
      az (root)
       |> network (group)
             |> vnet  (group)
                  |> create (command)
  

  We call network vnet the command group name and create the command operation name.

• Argument: CLI user can pass in various types of arguments. In addition to the primitive types, users can also pass object, array and dict. For more information, please review CLI/Argument.

There are two repos to maintain Azure CLI commands：

• Azure CLI
• Azure CLI Extension

What is REST API Specifications?

Azure uses swagger(OpenAPI) or TypeSpec to define REST API specifications. Most of files are maintained in the following two repos:

• azure-rest-api-specs
• azure-rest-api-specs-pr

Swagger Specification (or known as OpenAPI Specification) is an API description format for REST APIs. An OpenAPI file, can be written in YAML or json, allows developers to describe the entire API.

TypeSpec (tsp) is a language for defining cloud service APIs and models. It is a highly extensible language with primitives that can describe APIs using REST, gRPC, and other protocols.

The definition of API in swagger or TypeSpec is required before using AAZDev tool.

What is Atomic CLI Command?

An Atomic CLI Command operate on one azure resource without dependencies on other things, such as SDK or other commands.

Azure api is designed to be restful. So usually one Atomic CLI command operates on one API. But there is an exception, there are multiple APIs that list the same kind of resource in difference scopes. In that case, one list Atomic command operates on multiple APIs.

Atomic CLI commands do not depend on the SDK or other commands. This brings several advantages to CLI:

• State of truth: An Atomic CLI command has all required information in its model and code, so its current state tells how it works without the influence by the change in SDK or other commands. This feature can be used for breaking change detection or other command analysis tasks.
• Flexible: Atomic CLI commands are decoupled from others commands, so it’s easy to add, modify, upgrade and release an Atomic CLI command without influence other commands. And the change can be refined down to the API level instead of SDK level.

The degree of coupling between commands relay on SDK can be seen from the following PR. The SDK packages a batch of APIs, and when one API has new change and is released in a new SDK version, we have to test and update the commands that use the whole batch of APIs in SDK instead of the one API we care about. Each time we bump up an SDK, hundreds of tests need to rerun in live and their recording files need to be updated, we also need to fix other commands that are broken by new SDK. It wasted a lot of time and created a lot of hidden problems. By applying Atomic CLI commands, we can avoid them.

Overview

AAZDev Tool consists of 4 parts:

• API Translators: They are responsible for translating the API specification into a command model. We’ve implemented both swagger 2.0 and Typespec translator which can support to translate the API specs in azure-rest-api-specs repo and azure-rest-api-specs-pr repo.
• Model Editors: They are used to edit command models. Currently, Workspace Editor is implemented. More details are introduced in Workspace paragraph.
• Command Models: The command models generated from translators and modified in editors will be persisted in a repo called AAZ. The persistence of models can be useful in many ways. More details are introduced in AAZ Repo paragraph.
• Code Generators: They are used to generator code from command models. The CLI Generator is implemented to generate code into Azure CLI repo and CLI Extension Generator is for Azure CLI Extension

For CLI Developers, they edit command models in Model Editors and select command models for Code Generators.

Swagger V2 Translator

A Resource can be translated into a single command or multiple commands under a command group.

Group part naming

Swagger V2 translator uses the valid part of url to generate the group part of a command.

The valid part of url is the url suffix starts by its Resource Provider with parameter placeholders replaced by {} and locations segment removed.

The table below show some examples with valid part highlighted.

Each segments of valid part corresponds to a group name. The parameter segments are ignored by default. And Microsoft. prefix in resource provider segment is ignored too. The singular values of the remaining segments are converted to snake case with - as separator. For example Microsoft.Network/virtualNetworks/{}/subnets/{} has three segments: Microsoft.Network, virtualNetworks, subnets and they correspond to network, virtual-network and subnet. So the full command group name will be network virtual-network subnet

Operation part naming

Swagger V2 translator use the HTTP methods to generate the operation part of a command.

The table below show the mapping relationship between the common command operation name and the resource HTTP methods.

For example if a resource has the following properties:

resource_id: '/subscriptions/{}/resourcegroups/{}/providers/microsoft.edgeorder/addresses/{}'
methods:
    - GET
    - DELETE
    - PUT
    - PATCH

This resource will be translated into four commands under az edge-order address command group:

az data-bricks workspace:
    - show
    - delete
    - create
    - update

The show, delete, create commands call GET, DELETE, PUT methods respectively.

The update command will either use a combination of GET+PUT methods or a single PATCH method. When all three methods exist in a resource, Generic First operation will use GET+PUT combination, and Patch First will use PATCH method.

Translators will also detect a GET method should be named as show or list based on its url and response schema. For example the following two resources will generate list command of az edge-order address.

resource_id: '/subscriptions/{}/providers/microsoft.edgeorder/addresses'
methods:
    - GET

resource_id: '/subscriptions/{}/resourcegroups/{}/providers/microsoft.edgeorder/addresses'
methods:
    - GET

Translators will automatically merge them into one list commands if their response schemas are the same. If their response schemas are different, a special suffix will be added after list operation name to distinguish two commands. And the name can be renamed in editor.

The POST method is special. If a resource has POST method only and the last segment of valid part is not a parameter segment, that segment will be used as operation name, else a temporary name will be generated, which can be renamed in editor later.

TypeSpec API Translator - typespec-aaz

typespec-aaz translates an interface under a specific namespace from typespec specification into a resource url aligned with swagger api, with defined data models translated into the input and output parameters.

The data models defined in typespec will be translated into aaz models. For example,

The model value defined as below

  @doc("The name of the Community Training Resource")
  @pattern("^[a-zA-Z0-9-]{3,24}$")
  @key("communityTrainingName")
  @segment("communityTrainings")
  @path
  name: string;

would be tranlsated into a parameter object below:

{
  description: "The name of the Community Training Resource",
  format: {pattern: "^[a-zA-Z0-9-]{3,24}$"},
  pattern: "^[a-zA-Z0-9-]{3,24}$",
  name: "communityTrainingName",
  required: true,
  type: "string"
}

typespec-aaz translates the typespec api into aaz models and then the following steps keep the same as swagger specifications.

Workspace

Before developers finish customizing the command models and export them in AAZ repo for persistence, the draft is saved in a workspace. Workspaces are like containers, they are isolated so that changes in one do not affect the others. Therefore, developers can create as many workspaces as needed for different purposes.

It’s possible to add resources from different resource providers, but they should be in the same plane. Currently, we only support Management plane. Another note is that a workspace don’t allow to add a resource multiple times in different versions. For example, if virtual network resource(‘/subscriptions/{}/resourcegroups/{}/providers/microsoft.network/virtualnetworks/{}’) of version 2021-05-01 is added in a workspace, it’s not allowed to add other versions of this resource in this workspace.

Please jump to Workspace Editor for more details.

AAZ Repo

The name AAZ comes from the abbreviation of Atomic Azure. This repo is to maintain Atomic Azure Cli command models. You can access this repo by link.

There are two folders in the root of AAZ:

• Commands: This folder provides an index of command models available in AAZ. They are organized in a command tree.
• Resources: This folder save command models in xml files which is called command configurations. Those files are separated by resources.

Command Tree

The data of command tree is in Commands/tree.json. Json files are hard to read, so the data is also rendered as Markdown files in a tree hierarchy.

Each folder represents each node of the tree and also represents a command group. Its readme.md file shows the summary, subgroups and commands of this group. And the links in the file help us to browse quickly.

Each Markdown files starts with ‘_’ represents each leaf of the tree which also represents a command. Its contains the summary, versions and examples of this command. And each version links to a command configuration in Resources folder.

Command Configuration

The format of command configuration file path is:

Resources/{plane}/{base64(resource_id)}/{api_version}.xml

A command configuration file contains more than one commands, which are generated from the same resource.

The diagram below show the structure of command configuration file.

The commands in a configuration file organized in hierarchy.

There are three sections in every command:

• Argument Section: The arguments defined in argument section and grouped by Argument Group.
• Operation Section: The operations performed by the command are defined in Operation Section. Currently, there are two kinds of operations:
  • HTTP Operations: This kind of operations are used to generate HTTP request.
  • Instance Operations: This kind of operations are used to change the data of resource instance.
• Output Section: The output transformations are defined in Output Section. The data flow from Argument Section to Operation Section and then to Output Section. The arguments are linked with operations by their variant name. The operations are generated by swagger and do not support customization. The Operation Section in diagram is from a classic update command which use get+put methods. In model editors the modification of arguments and special output formats are supported.

Verify Metadata

To verify the “complicated” structure above, AAZ repo itself provides a verification workflow based on GitHub Actions. When you raise a pull request, a workflow will be automatically triggered, and we don’t need to pay much attention on that.

But when the workflow fails, we should know how to fix it. A typical failure is as follows:

When you click to view the details, you will see:

What does “Please pull the latest main branch, and Export your command model again.” mean? Actually, there is a set of Git commands behind that:

1. Run git reset --hard to revert all your previous changes.
2. Run git pull <upstream> main to pull the latest main branch.
3. Click Export button on the workspace editor to export your target model.
4. Now you can commit and push your current changes.

If the above steps still don’t solve your problem, you can click to expand Verify data consistency. There must be something wrong with your current structure, please check the exception part carefully.

CLI Generator and CLI Extension Generator

Command Module

CLI commands are separated into different command modules in Azure CLI repo or Azure CLI extension repo. So it’s required to select a specific module for generators.

Profile

Azure CLI uses profiles to support Azure Stack. There are five profiles:

• latest
• 2020-09-01-hybrid
• 2019-03-01-hybrid
• 2018-03-01-hybrid
• 2017-03-09-profile

The latest profile contains a full set of commands and the rest profiles contains a sub set of commands from the latest. One command may call different api versions in different profiles. So its arguments and output schema may vary from profile to profile.

Please jump to CLI Generator Usage for more details.