Terraform Contribution Flow

High-level contribution flow


---
config:
  nodeSpacing: 20
  rankSpacing: 20
  diagramPadding: 50
  padding: 5
  flowchart:
    wrappingWidth: 300
    padding: 5
  layout: elk
  elk:
    mergeEdges: true
    nodePlacementStrategy: LINEAR_SEGMENTS
---

flowchart TD
  A(1 - Fork the module source repository)
    click A "/Azure-Verified-Modules/contributing/terraform/terraform-contribution-flow/#1-fork-the-module-source-repository"
  B(2 - Setup your Azure test environment)
    click B "/Azure-Verified-Modules/contributing/terraform/terraform-contribution-flow/#2-prepare-your-azure-test-environment"
  C(3 - Implement your contribution)
    click C "/Azure-Verified-Modules/contributing/terraform/terraform-contribution-flow/#3-implement-your-contribution"
  D{4 - Pre-commit<br>checks successful?}
    click D "/Azure-Verified-Modules/contributing/terraform/terraform-contribution-flow/#4-run-pre-commit-checks"
  E(5 - Create a pull request to the upstream repository)
    click E "/Azure-Verified-Modules/contributing/terraform/terraform-contribution-flow/#5-create-a-pull-request-to-the-upstream-repository"
  A --> B
  B --> C
  C --> D
  D -->|yes|E
  D -->|no|C

GitFlow for contributors

The GitFlow process outlined here depicts and suggests a way of working with Git and GitHub. It serves to synchronize the forked repository with the original upstream repository. It is not a strict requirement to follow this process, but it is highly recommended to do so.

---

config:
  logLevel: debug
  gitGraph:
    rotateCommitLabel: false
---

gitGraph LR:
  commit id:"fork"
  branch fork/main
  checkout fork/main
  commit id:"checkout feature" type: HIGHLIGHT
  branch feature
  checkout feature
  commit id:"checkout fix"
  branch fix
  checkout main
  merge feature id: "Pull Request 'Feature'" type: HIGHLIGHT
  checkout fix
  commit id:"Patch 1"
  commit id:"Patch 2"
  checkout main
  merge fix id: "Pull Request 'Fix'" type: HIGHLIGHT
Tip

When implementing the GitFlow process as described, it is advisable to configure the local clone of your forked repository with an additional remote for the upstream repository. This will allow you to easily synchronize your locally forked repository with the upstream repository. Remember, there is a difference between the forked repository on GitHub and the clone of the forked repository on your local machine.

UpstreamToForkAndSourceRepository UpstreamToForkAndSourceRepository

Note

Each time in the following sections we refer to ‘your xyz’, it is an indicator that you have to change something in your own environment.

Prepare your developer environment

1. Fork the module source repository

Important

Each Terraform AVM module will have its own GitHub repository in the Azure GitHub Organization as per SNFR19.

This repository will be created by the Module owners and the AVM Core team collaboratively, including the configuration of permissions as per SNFR9

Module contributors are expected to fork the corresponding repository and work on a branch from within their fork, before then creating a Pull Request (PR) back into the source repository’s main branch.

To do so, simply navigate to your desired repository, select the 'Fork' button to the top right of the UI, select where the fork should be created (i.e., the owning organization) and finally click ‘Create fork’.

Note

If the module repository you want to contribute to is not yet available, please get in touch with the respective module owner which can be tracked in the Terraform Resource Modules index see PrimaryModuleOwnerGHHandle column.

Optional: The usage of local source branches

For consistent contributors but also Azure-org members in general it is possible to get invited as collaborator of the module repository which enables you to work on branches instead of forks. To get invited get in touch with the module owner since it’s the module owner’s decision who gets invited as collaborator.

2. Prepare your Azure test environment

AVM performs end-to-end (e2e) test deployments of all modules in Azure for validation. We recommend you to perform a local e2e test deployment of your module before you create a PR to the upstream repository. Especially because the e2e test deployment will be triggered automatically once you create a PR to the upstream repository.

  1. Have/create an Azure Active Directory Service Principal with at least Contributor & User Access Administrator permissions on the Management-Group/Subscription you want to test the modules in. You might find the following links useful:

    # Linux/MacOs
    export ARM_SUBSCRIPTION_ID=$(az account show --query id --output tsv) # or set <subscription_id>
    export ARM_TENANT_ID=$(az account show --query tenantId --output tsv) # or set <tenant_id>
    export ARM_CLIENT_ID=<client_id>
    export ARM_CLIENT_SECRET=<service_principal_password>
    
    # Windows/Powershell
    $env:ARM_SUBSCRIPTION_ID = $(az account show --query id --output tsv) # or set <subscription_id>
    $env:ARM_TENANT_ID = $(az account show --query tenantId --output tsv) # or set <tenant_id>
    $env:ARM_CLIENT_ID = "<client_id>"
    $env:ARM_CLIENT_SECRET = "<service_principal_password>"
  2. Change to the root of your module repository and run ./avm docscheck (Linux/MacOs) / avm.bat docscheck (Windows) to verify the container image is working as expected or needs to be pulled first. You will need this later.

    PullLatestAzterraformContainerImage PullLatestAzterraformContainerImage

3. Implement your contribution

To implement your contribution, we kindly ask you to first review the Terraform specifications and composition guidelines in particular to make sure your contribution complies with the repository’s design and principles.

Tip

To get a head start on developing your module, consider using the tooling recommended per spec TFNFR37. For example you can use the newres tool to help with creating variables.tf and main.tf if you’re developing a module using Azurerm provider.

4. Run Pre-commit Checks

Important

Make sure you have Docker installed and running on your machine.

Note

To simplify and help with the execution of commands like pre-commit, pr-check, docscheck, fmt, test-example, etc. there is now a simplified avm script available distributed to all repositories via terraform-azurerm-avm-template which combines all scripts from the avm_scripts folder in the tfmod-scaffold repository using avmmakefile.

The avm script also makes sure to pull the latest mcr.microsoft.com/azterraform:latest container image before executing any command.

4.1. Run pre-commit and pr-check

The following commands will run all pre-commit checks and the pr-check.

# Running all pre-commit checks
# `pre-commit` runs depsensure fmt fumpt autofix docs
# `pr-check` runs fmtcheck tfvalidatecheck lint unit-test

## Linux/MacOs
./avm pre-commit
./avm pr-check

## Windows
avm.bat pre-commit
avm.bat pr-check

4.2 Run e2e tests

Currently you have two options to run e2e tests:

Note

With the help of the avm script and the commands ./avm test-example (Linux/MacOs) / avm.bat test-example (Windows) you will be able to run it in a more simplified way. Currently the test-example command is not completely ready yet and will be released soon. Therefore please use the below docker command for now.

  1. Run e2e tests with the help of the azterraform docker container image.

    # Linux/MacOs
    
    docker run --rm -v $(pwd):/src -w /src -v $HOME/.azure:/root/.azure -e TF_IN_AUTOMATION -e AVM_MOD_PATH=/src -e AVM_EXAMPLE=<example_folder> -e ARM_SUBSCRIPTION_ID -e ARM_TENANT_ID -e ARM_CLIENT_ID -e ARM_CLIENT_SECRET mcr.microsoft.com/azterraform:latest make test-example
    
    # Powershell
    
    docker run --rm -v ${pwd}:/src -w /src -v $HOME/.azure:/root/.azure -e TF_IN_AUTOMATION -e AVM_MOD_PATH=/src -e AVM_EXAMPLE=<example_folder> -e ARM_SUBSCRIPTION_ID -e ARM_TENANT_ID -e ARM_CLIENT_ID -e ARM_CLIENT_SECRET mcr.microsoft.com/azterraform:latest make test-example

    Make sure to replace <client_id> and <service_principal_password> with the values of your service principal as well as <example_folder> (e.g. default) with the name of the example folder you want to run e2e tests for.

  2. Run e2e tests with the help of terraform init/plan/apply.

    Simply run terraform init and terraform apply in the example folder you want to run e2e tests for. Make sure to set the environment variables ARM_SUBSCRIPTION_ID, ARM_TENANT_ID, ARM_CLIENT_ID and ARM_CLIENT_SECRET before you run terraform init and terraform apply or make sure you have a valid Azure CLI session and are logged in with az login.

5. Create a pull request to the upstream repository

Once you are satisfied with your contribution and validated it, submit a pull request to the upstream repository and work with the module owner to get the module reviewed by the AVM Core team, by following the initial module review process for Terraform Modules, described here. This is a prerequisite for publishing the module. Once the review process is complete and your PR is approved, merge it into the upstream repository and the Module owner will publish the module to the HashiCorp Terraform Registry.

5.1 Create the Pull Request [Contributor]

These steps are performed by the contributor:

  1. Navigate to the upstream repository and click on the Pull requests tab.
  2. Click on the New pull request button.
  3. Ensure the base repository is set to the upstream AVM repo.
  4. Ensure the base branch is set to main.
  5. Ensure your head repository and compare branch are set to your fork and the branch you are working on.
  6. Click on the Create pull request button.

5.2 Review the Pull Request [Owner]

  1. IMPORTANT: The module owner must first check for any malicious code or changes to workflow files. If they are found, the owner should close the PR and report the contributor.
  2. Review the changes made by the contributor and determine whether end to end tests need to be run.
  3. If end to end tests do not need to be run (e.g. doc changes, small changes, etc) then so long as the static analysis passes, the PR can be merged to main.
  4. If end to end tests do need to be run, then follow the steps in 5.3.

5.3 Release Branch and Run End to End Tests [Owner]

  1. IMPORTANT: The module owner must first check for any malicious code or changes to workflow files. If they are found, the owner should close the PR and report the contributor.
  2. Create a release branch from main. Suggested naminmg convention is release/<description-of-change>.
  3. Open the PR created by the contributor and click Edit at the top right of the PR.
  4. Change the base branch to the release branch you just created.
  5. Wait for the PR checks to run, validate the code looks good and then merge the PR into the release branch.
  6. Create a new PR from the release branch to the main branch of the AVM module.
  7. The end to end tests should trigger and you can approve the run.
  8. Once the end to end tests have passed, merge the PR into the main branch.
  9. If the end to end tests fail, investigate the failure. You have two options:
    1. Work with the contributor to resolve the issue and ask them to submit a new PR from their fork branch to the release branch.
      1. Re-run the tests and merge to main. Repeat the loop as required.
    2. If the issue is a simple fix, resolve it directly in the release branch, re-run the tests and merge to main.

Common mistakes to avoid and recommendations to follow

  • If you contribute to a new module then search and update TODOs (which are coming with the terraform-azurerm-avm-template) within the code and remove the TODO comments once complete
  • terraform.lock.hcl shouldn’t be in the repository as per the .gitignore file
  • Update the support.md file
  • \_header.md needs to be updated
  • support.md needs to be updated
  • Exclude terraform.tfvars file from the repository

Subsections of Contribution Flow

Terraform Owner Contribution Flow

This section describes the contribution flow for module owners who are responsible for creating and maintaining Terraform Module repositories.

Important

This contribution flow is for Module owners only.

As a Terraform Module Owner you need to be aware of the AVM contribution process overview & Terraform specifications (including Interfaces) as as these need to be considered during pull request reviews for the modules you own.

Info

Make sure module authors/contributors tested their module in their environment before raising a PR. The PR uses e2e checks with 1ES agents in the 1ES subscriptions. At the moment their is no read access to the 1ES subscription. Also if more than two subscriptions are required for testing, that’s currently not supported.

1. Owner Activities and Responsibilities

Familiarise yourself with the responsibilities as Module Owner outlined in Team Definitions & RACI and in the TF Issue Triage.

  1. Watch Pull Request (PR) and issue (questions/feedback) activity for your module(s) in your repository and ensure that PRs are reviewed and merged in a timely manner as outlined in SNFR11.
Info

Make sure module authors/contributors tested their module in their environment before raising a PR. Also because once a PR is raised a e2e GitHib workflow pipeline is required to be run successfully before the PR can be merged. This is to ensure that the module is working as expected and is compliant with the AVM specifications.

2. GitHub repository creation and configuration

Once your module has been approved and you are ready to start development, you need to request that a new repository be created for your module.

You do that by adding a comment to the issue with the #RFRC tag. The  Status: Ready For Repository Creation 📝  label will then be applied. This will trigger the creation of the repository and the configuration of the repository with the required settings.

Info

If you need your repository to be created urgently, please message the AVM Core team in the AVM Teams channel.

Once your module is ready for development, the  Status: Repository Created 📄  label will be added to the issue and you’ll be notified it is ready.

3. Module Development Activities

You can now start developing your module, following standard guidance for Terraform module development.

Some useful things to know:

Pull Request

You can raise a pull request anytime, don’t wait until the end of the development cycle. Raise the PR after you first push your branch.

You can then use the PR to run end to end tests, check linting, etc.

Once readdy for review, you can request a review per step 4.

Grept

Grept is a linting tool for repositories, ensures predefined standards, maintains codebase consistency, and quality.
It’s using the grept configuration files from the Azure-Verified-Modules-Grept repository.

You can see here which files are synced from the terraform-azurerm-avm-template repository.

Set environment variables and run Grept:

export GITHUB_REPOSITORY_OWNER=Azure
export GITHUB_REPOSITORY=Azure/terraform-azurerm-avm-res-<RP>-<modulename>"

./avm grept-apply
$env:GITHUB_REPOSITORY_OWNER="Azure"
$env:GITHUB_REPOSITORY="Azure/terraform-azurerm-avm-res-<RP>-<modulename>"

./avm grept-apply

Custom Variables and Secrets for end to end tests

The respoitory has an environment called test, it has have approvals and secrets applied to it ready to run end to end tests.

  • In the unusual cicumstance that you need to use your own tenant and subscription for end to end tests, you can override the secrets by setting ARM_TENANT_ID_OVERRIDE, ARM_SUBSCRIPTION_ID_OVERRIDE, and ARM_CLIENT_ID_OVERRIDE secrets.
  • If you need to supply additional secrets or variables for your end to end tests, you can add them to the test environment. They must be prefixed with TF_VAR_, otherwise they will be ignored.

4. Review the module

Once the development of the module has been completed, get the module reviewed from the AVM Core team by following the AVM Review of Terraform Modules process here which is a pre-requisite for the next step.

5. Publish the module

Once a module has been reviewed and the PR is merged to main. Follow the below steps to publish the module to the HashiCorp Registry.

Ensure your module is ready for publishing:

  1. Create a release with a new tag (e.g. 0.1.0) via Github UI.

    • Go to the releases tab and click on Draft a new release.
    • Ensure that the Target is set to the main branch.
    • Select Choose a tag and type in a new tag, such as 0.1.0 Make sure to create the tag from the main branch.
    • Generate the release notes using the Generate release notes button.

    DeploymentProtectionRules DeploymentProtectionRules

  2. Elevate your respository access using the Open Source Management Portal.

  3. Sign in to the HashiCorp Registry using GitHub.

  4. Publish a module by selecting the Publish button in the top right corner, then Module

  5. Select the repository and accept the terms.

Info

Once a module gets updated and becomes a new version/release it will be automatically published with the latest published release version to the HashiCorp Registry.

Important

When an AVM Module is published to the HashiCorp Registry, it MUST follow the below requirements:

  • Resource Module: terraform-<provider>-avm-res-<rp>-<ARM resource type> as per RMNFR1
  • Pattern Module: terraform-<provider>-avm-ptn-<patternmodulename> as per PMNFR1

Terraform Core Team Repository Creation Process

This section describes the process for AVM core team members who are responsible for creating Terraform Module repositories.

Important

This contribution flow is for AVM Core Team members only.

1. Find Issues Ready for Repository Creation

  1. When a module owner is ready to start development, they will add the  Status: Ready For Repository Creation 📝  label to the proposal via a comment issue.
  2. To find issues that are ready for repository creation, click this link
  3. Open one of the issues to find the details you need.
    1. Module name: This will be in the format avm-<type>-<name>. e.g. avm-res-network-virtualnetwork
    2. Module owner GitHub handle: This will be in the content of the issue
    3. Module description: If this does not exist, then create one. The description will automtically be prefixed with Terraform Azure Verified <module-type> Module for ..., where <module-type> is either Resource, Pattern, or Utility

2. Create the repository

  1. Open a PowerShell terminal

  2. Clone the https://github.com/Azure/terraform-azure-modules repository and navigate to the repository_creation_helper folder

    git clone "https://github.com/Azure/terraform-azure-modules"
    cd ./terraform-azure-modules/repository_creation_helper
  3. Install the GitHub CLI if you don’t already have it installed: https://cli.github.com

  4. Login to GitHub CLI

    gh auth login -h "github.com" -w -p "https" -s "delete_repo" -s "workflow" -s "read:user" -s "user:email"

    Follow the prompts to login to your GitHub account.

  5. Run the following command, replacing the values with the details you collected in step 1

    $moduleProvider = "azurerm" # Only change this if you know why you need to change it :)
    $moduleName = "<module name>" # Replace with the module name (do not include the "terraform-azurerm" prefix)
    $moduleDescription = "<module description>" # Replace with a short description of the module
    $moduleOwner = "<github user handle>" # Replace with the GitHub handle of the module owner
    
    ./New-Repository.ps1 `
        -moduleProvider $moduleProvider `
        -moduleName $moduleName `
        -moduleDescription $moduleDescription `
        -moduleOwner $moduleOwner

    For example:

    $moduleProvider = "azurerm" # Only change this if you know why you need to change it :)
    $moduleName = "avm-res-network-virtualnetwork" # Replace with the module name (do not include the "terraform-azurerm" prefix)
    $moduleDescription = "Virtual Networks" # Replace with a short description of the module
    $moduleOwner = "jaredfholgate" # Replace with the GitHub handle of the module owner
    
    ./New-Repository.ps1 `
        -moduleProvider $moduleProvider `
        -moduleName $moduleName `
        -moduleDescription $moduleDescription `
        -moduleOwner $moduleOwner
  6. The script will stop and prompt you to fill out the Microsoft Open Source details,

  7. Open the Open Source Portal using the link in the script output.

  8. Click Complete Setup, then use the following table to provide the settings:

    QuestionAnswer
    Classify the repositoryProduction
    Assign a Service tree or Opt-outAzure Verified Modules / AVM
    Direct ownersAdd the module owner and yourself as direct owners. Add the avm-team-module-owners as security group.
    Is this going to ship as a public open source licensed projectYes, creating an open source licensed project
    What type of open source will this beSample code
    What license will you be releasing withMIT
    Did your team write all the code and create all of the assets you are releasing?Yes, all created by my team
    Does this project send any data or telemetry back to Microsoft?Yes, telemetry
    Does this project implement cryptographyNo
    Project nameAzure Verified Module (Terraform) for ‘module name
    Project version1
    Project descriptionAzure Verified Module (Terraform) for ‘module name’. Part of AVM project - https://aka.ms/avm
    Business goalsCreate IaC module that will accelerate deployment on Azure using Microsoft best practice.
    Will this be used in a Microsoft product or service?This is open source project and can be leveraged in Microsoft service and product.
    Adopt security best practice?Yes, use just-in-time elevation
    Maintainer permissionsLeave empty
    Write permissionsLeave empty
    Repository templateUncheck
    Add .gitignoreUncheck
  9. Click Finish setup + start business review to complete the setup

  10. Wait for it to process and then click View repository

  11. If you don’t see the Elevate your access button, then refresh the browser window

  12. Click Elevate your access and follow the prompts to elevate your access

  13. Now head back over to the terminal and type yes and hit enter to complete the repository configuration

  14. Open the new repository in GitHub.com and verify it all looks good.

    1. On the home page
      1. The name is correct
      2. The description is correct
      3. The Terraform registry url looks good
      4. The repository has the template files in it
    2. In Setting
      1. The repository is public
      2. The Collaborators and teams are correct

3. Request the GitHub App Install

  1. Create a new issue at https://github.com/microsoft/github-operations/issues/new?template=GitHub-App-Installation-Request.md

  2. Update the issue with the following details:

    1. Title: [GitHub App] Installation Request - Azure Verified Modules

    2. Body - replace <repository url> with the URL of the repository you created in step 2:

      > __Note:__ If the app is listed on the [Auto-Approved list](https://docs.opensource.microsoft.com/github/apps/approvals/), you do not need to complete this form.
      
      You complete these steps:
      
      - [x] Confirm the app is not in the [Auto-Approved list](https://docs.opensource.microsoft.com/github/apps/approvals/)
      - [x] Fill out and verify the information in this form
      - [x] Update the title to reflect the org/repo and/or app name
      - [x] Submit the native request within the GitHub user interface
      
      Operations will help complete these steps:
      
      - [ ] Approve the app if already requested on GitHub natively
      - [ ] Close this issue
      
      Finally, you'll complete any configuration with the app or your repo that is required once approved.
      
      # My request
      
      - GitHub App name: Azure Verified Modules
      
      - GitHub organization in which the app would be installed: Azure
      
      - Is this an app created by you and/or your team?
      
        - [x] Yes, this is an app created by me and/or my team
        - [ ] No, this is a Microsoft 1st-party app created by another team
        - [ ] No, this is a 3rd-party marketplace app
      
      - If this __is an app created by you and/or your team__, please provide some ownership information in case future questions come up:
      
        - Service Tree ID: our service tree ID is: Unchanged
        - A few specific individuals at Microsoft if we have questions (corporate email list):Unchanged
        - An optional team discussion list: Unchanged
      
      - Is this an app you/your team created to address [reduced PAT lifetimes](https://aka.ms/opensource/tsg/pat)?
        - [x] Yes
        - [ ] No
      
      - Are you looking for this app to be installed on individual repos or all repos in an organization?
      
        - [x] Individual repos: <repository url>
        - [ ] All repos in an organization
      
      - Does this app have any side-effects if it is installed into all repos in an organization? Side effects can include creating labels, issues, pull requests, automatic checks on PRs, etc.
      
        - [ ] Yes, it has side effects and you should be careful if installing to all repos in an org
        - [x] No side effects
      
      - Please provide a description of the app's functionality and what are you trying to accomplish by utilizing this app:
      
        Unchanged
      
      - For any major permissions (org admin, repo admin, etc.), can you explain what they are and why they are needed?
      
        Unchanged
      
      - Any other notes or information can you provide about the app?
  3. Submit the issue

4. Notify the Module Owner and Update the Issue Status

  1. Add a comment to the issue you found in step 1 to let the module owner know that the repository has been created and is be ready for them to start development.

    @<module owner> The module repository has now been created. You can find it at <repository url>.
    
    The final step of repository configuration is still in progress, but you will be able to start developing your code immediately.
    
    The final step is to create the environment and credentials require to run the end to end tests. If the environment called `test` is not available in 48 hours, please let me know.
    
    Thanks
  2. Add the  Status: Repository Created 📄  label to the issue

  3. Remove the  Status: Ready For Repository Creation 📝  label from the issue