kubernetes-hackfest

Delivering modern cloud-native applications with open source technologies on Azure Kubernetes Service

View the Project on GitHub Azure/kubernetes-hackfest

Lab: CI/CD with Brigade and Helm

This workshop will guide you through building a Continuous Integration (CI) and Continuous Deployment (CD) pipeline using the open source platform Brigade. The pipeline will utilize Azure Container Registry to build the images and Helm for application updating.

Prerequisites

Instructions

The general workflow/result will be as follows:

Setup Github Repo

In order to trigger this pipeline you will need your own Github account and forked copy of this repo. Log into Github in the browser and get started.

  1. Broswe to https://github.com/azure/kubernetes-hackfest and click “Fork” in the top right.

    Brigade GitHub Fork

  2. Grab your clone URL from Github which will look something like: https://github.com/thedude-lebowski/kubernetes-hackfest.git

    Brigade GitHub Clone

  3. Clone your repo in Azure Cloud Shell.

    Note: If you have cloned the repo in earlier labs, the directory name will conflict. You can either delete the old one or just rename it before this step.

     git clone https://github.com/<your-github-account>/kubernetes-hackfest.git
    
     cd kubernetes-hackfest
    

Setup Brigade

  1. Update helm repo

     helm repo add brigade https://brigadecore.github.io/charts
    
  2. Apply the cluster role binding for the brigade-worker service account
     kubectl apply -f labs/cicd-automation/brigade/brigade-rbac.yaml
    
  3. Install brigade chart into it’s own namespace

     kubectl create ns brigade
    
     helm install brigade brigade/brigade --namespace brigade --set brigade-github-app.enabled=true --set brigade-github-app.service.type=LoadBalancer
    
     kubectl get pod,svc -n brigade
    
     NAME                                              READY   STATUS    RESTARTS   AGE
     pod/brigade-brigade-api-77d8c6cd59-grlcw          0/1     Running   0          8s
     pod/brigade-brigade-ctrl-5885799bcd-zmxjx         1/1     Running   0          8s
     pod/brigade-brigade-github-app-597b756478-dkx84   1/1     Running   0          8s
     pod/brigade-kashti-7d46999bb9-zgf7t               1/1     Running   0          8s
    
     NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
     service/brigade-brigade-api          ClusterIP      10.0.85.60    <none>        7745/TCP       8s
     service/brigade-brigade-github-app   LoadBalancer   10.0.189.63   <pending>     80:31744/TCP   8s
     service/brigade-kashti               ClusterIP      10.0.52.160   <none>        80/TCP         8s
    

Setup Brigade Project

Brigade uses projects to define the configuration for pipelines. Brigade Projects are also installed with a Helm chart. In this section, we will create a YAML file to configure the brigade project Helm chart.

  1. Create a brigade project YAML file.

    • Create a file called brig-proj-hackfest.yaml Keep this file local on the cloud shell. This is a sensitive file and should not be pushed to Github.

      Note: In the latest Azure Cloud Shell, there is a built-in editor to allow easy file editing without leaving the shell.

    • Add the contents below to start your file

        project: 
        repository: 
        cloneURL: 
        sharedSecret: 
        github:
            token: 
        secrets:
            acrServer: 
            acrName: 
            appId: 
            password: 
            tenant: 
      
    • Edit the values from above to match your Github account (example below)
      • project: thedude-lebowski/kubernetes-hackfest
      • repository: github.com/thedude-lebowski/kubernetes-hackfest
      • cloneURL: https://github.com/thedude-lebowski/kubernetes-hackfest.git
      • sharedSecret: useSomethingSuperSecretForThis!
    • Create a Github token and update the brig-proj-hackfest.yaml
      • In your Github, click on Settings and Developer settings
      • Select Personal sccess tokens
      • Select Generate new token Brigade GitHub Settings

      • Enter brigade-project for the description and give access to the repo Brigade GitHubToken

      Note: More details on Brigade and Github integration are here: https://github.com/Azure/brigade/blob/master/docs/topics/github.md

      • Be sure to copy the access token value and add it to your project YAML file.
    • Gather your ACR info from the Azure portal. Edit the brig-proj-hackfest.yaml for these values
      • acrServer (something like myacr.azurecr.io)
      • acrName (something like myacr)
    • Create an Azure service principal with rights to your ACR Resource Group.
      • You can get your subscription ID in the Azure portal or running az account list -o table

          export AZSUBID="471d33fd-a776-405b-947c-467c291dc741"
          export RGNAME=kubernetes-hackfest
        
          az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/$AZSUBID/resourceGroups/$RGNAME"
        
          # mock values:
          {
              "appId": "11pp69f2-2d9a-4c46-921c-99058df3738z",
              "displayName": "azure-cli-2018-01-10-21-20-18",
              "name": "http://azure-cli-2018-01-10-21-20-18",
              "password": "11111fe4-07dd-4adb-b984-d25f030f7a92",
              "tenant": "99f999bf-99f1-41af-99ab-2d7cd011ab12"
          }
        
      • Be sure to grant the service principal rights to the RG where your ACR is located.
      • Use the output to set the values in the brig-proj-hackfest.yaml file
    • After the above steps, your file will look like the below (values are not valid for realz)

        project: thedude-lebowski/kubernetes-hackfest
        repository: github.com/thedude-lebowski/kubernetes-hackfest
        cloneURL: https://github.com/thedude-lebowski/kubernetes-hackfest.git
        sharedSecret: useSomethingSuperSecretForThis!
        github:
            token: 1yyy8a4d1c08004rrttt685980814d3f358e5b0z
        secrets:
            acrServer: myacr.azurecr.io
            acrName: myacr
            appId: 11pp69f2-2d9a-4c46-921c-99058df3738z
            password: 11111fe4-07dd-4adb-b984-d25f030f7a92
            tenant: 99f999bf-99f1-41af-99ab-2d7cd011ab12
      
  2. Create your brigade project

     # from the directory where your file from step #1 was created
    
     helm install brig-proj-hackfest brigade/brigade-project -f brig-proj-hackfest.yaml --namespace brigade
    

    Note: There is a brig CLI client that allows you to view your brigade projects. More details here: https://github.com/Azure/brigade/tree/master/brig

Setup Brigade Pipeline

To save time, we will only deploy the web-ui application in this lab.

  1. In the Azure cloud shell, cd ~/kubernetes-hackfest and create a file called brigade.js

  2. Edit brigade.js in cloud shell

  3. Paste the contents from the sample brigade.js file in this file

  4. Review the pipeline steps in the javascript

  5. Commit the new file to your Github repository

     git add .
     git add -A
     git commit -m "added brigade pipeline script"
     git push
    

    Note that we are using the master branch here. Normally we would use other branches and PR’s. For simplicity, we are using master just for this lab.

Configure Github Webhook

  1. Get a URL for your Brigade Gateway

     kubectl get service brigade-brigade-github-app -n brigade
    
     NAME                 TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
     brigade-brigade-gw   LoadBalancer   10.0.45.233   13.67.129.228   7744:30176/TCP   4h
    
     # use these commands to create the full URL
    
     export GH_WEBHOOK=http://$(kubectl get svc brigade-brigade-github-app -n brigade -o jsonpath='{.status.loadBalancer.ingress[0].ip}')/events/github
    
     echo $GH_WEBHOOK
    

    The webhook URL should look something like: http://13.67.129.228/events/github You will use this in the next step.

  2. In your forked Github repo, click on Settings

  3. Click Webhooks

  4. Click Add webhook

  5. Set the Payload URL to the URL created in step 1

  6. Set the Content type to application/json

  7. Set the Secret to the value from your brig-proj-hackfest.yaml called “sharedSecret”

  8. Set the Which events... to Let me select individual events and check Push and Pull request

    Brigade GitHub Webhook

  9. Click the Add webhook button

Test the CI/CD Pipeline

  1. Make a code change in the web-ui application source code.

  2. Push the update to Github and validate the build in brigade.

     kubectl get pod -n brigade
    
     NAME                                                READY     STATUS      RESTARTS   AGE
     brigade-brigade-api-789bf79dbd-t2p8g                1/1       Running     0          1d
     brigade-brigade-ctrl-5d85d9f5bc-txwz4               1/1       Running     0          3h
     brigade-brigade-github-gw-65f45c69c7-8r2qw          1/1       Running     0          3h
     brigade-worker-01cjzdd4dnssjp50chw2fps6h3           0/1       Completed   0          31m
     brigade-worker-01cjzdn7d650jff10a04292pjs           0/1       Completed   0          1m
     job-runner-acr-builder-01cjzdd4dnssjp50chw2fps6h3   0/1       Completed   0          31m
     job-runner-acr-builder-01cjzdn7d650jff10a04292pjs   0/1       Completed   0          1m
     job-runner-helm-01cjzdd4dnssjp50chw2fps6h3          0/1       Completed   0          31m
     job-runner-helm-01cjzdn7d650jff10a04292pjs          0/1       Completed   0          25s
    
     kubectl logs job-runner-helm-01cjzdn7d650jff10a04292pjs -n brigade
     # Note: On the above command you can add the -f flag to 'follow' the output (i.e. to keep watching it as it rolls)
    
  3. Check the service-tracker-ui application pods and ensure they were updated with the new imageTag created in the build.

  4. If it worked, celebrate and go get a beer.

Add Kashti Web Dashboard (Optional)

The Kashti Web Dashboard was deployed as part of the Brigade Helm chart. By default the dashboard service is set as ClsuterIP. You can edit the service and change the type to LoadBalancer to view the dashboard.

kubectl edit svc brigade-kashti -n brigade

Troubleshooting / Debugging

Docs / References

Next Lab: Networking