Skip to content

Module 8 - ACA Monitoring and Observability with Application Insights

Module Duration

60 minutes

Objective

In this module, we will accomplish four objectives:

  1. Learn how Azure Container Apps integrate with Application Insights to examine application telemetry.
  2. Configure Application Insights for the microservices.
  3. Deploy updated Background Processor, API, and UI projects to Azure.
  4. Understand how telemetry data is visualized.

Module Sections

  • From the VS Code Terminal tab, open developer command prompt or PowerShell terminal in the project folder TasksTracker.ContainerApps (root):

    1
    cd ~\TasksTracker.ContainerApps

  • Restore the previously-stored variables by executing the local script. The output informs you how many variables have been set.

    1
    .\Variables.ps1

1. Azure Container Apps & Application Insights

In this module, we will explore how we can configure ACA and ACA Environment with Application Insights which will provide a holistic view of our container apps health, performance metrics, logs data, various telemetries and traces. ACA does not support Auto-Instrumentation for Application Insights, so in this module, we will be focusing on how we can integrate Application Insights into our microservice application.

1.1 Application Insights Overview

Application Insights is an offering from Azure Monitor that empowers us to monitor all ACAs under the same Container App Environment and collect telemetry about the workload services. Furthermore, it supports us in understanding the usage of the services and users' engagement via integrated analytics tools.

The term Telemetry refers to the information gathered to monitor our application, which can be classified into three distinct groups:

  1. Distributed Tracing: Distributed Tracing allows for visibility into the communication between services participating in distributed transactions. For instance, when the Frontend Web Application interacts with the Backend API Application to add or retrieve information. An application map of how calls are flowing between services is very important for any distributed application.

  2. Metrics: This offers a view of a service's performance and its use of resources. For instance, it helps in monitoring the CPU and memory usage of the Backend Background Processor, and identifying when it is necessary to scale up the number of replicas.

  3. Logging: provides insights into how code is executing and if errors have occurred.

In module 1 we have already provisioned a Workspace-based Application Insights Instance and configured it for the ACA environment by setting the property --dapr-instrumentation-key. We presume that you have already set up an instance of Application Insights that is available for use across the three container apps.

2. Installing Application Insights SDK Into the Three Microservices Apps

2.1 Install the Application Insights SDK Using NuGet

Our next step is to incorporate the Application Insights SDK into the three microservices, which is a uniform procedure.

Note

While we will outline the process of configuring Application Insights for the Backend API service, the identical steps must be followed for the other two services.

To incorporate the SDK, use the NuGet reference below in the csproj file of the Backend API project. You may locate the csproj file in the project directory TasksTracker.TasksManager.Backend.Api:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="7.0.13" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
  </ItemGroup>

</Project>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="7.0.13" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
  </ItemGroup>

</Project>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
  </ItemGroup>

</Project>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="9.0.0" />
  </ItemGroup>

</Project>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="9.0.0" />
  </ItemGroup>

</Project>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.22.0" />
  </ItemGroup>

</Project>

2.2 Set RoleName Property in All the Services

For each of the three projects, we will add a new file to each project's root directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;

namespace TasksTracker.TasksManager.Backend.Api
{
    public class AppInsightsTelemetryInitializer : ITelemetryInitializer
    {
        public void Initialize(ITelemetry telemetry)
        {
            if (string.IsNullOrEmpty(telemetry.Context.Cloud.RoleName))
            {
                //set custom role name here
                telemetry.Context.Cloud.RoleName = "tasksmanager-backend-api";
            }
        }
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;

namespace TasksTracker.TasksManager.Backend.Svc
{
    public class AppInsightsTelemetryInitializer : ITelemetryInitializer
    {
        public void Initialize(ITelemetry telemetry)
        {
            if (string.IsNullOrEmpty(telemetry.Context.Cloud.RoleName))
            {
                //set custom role name here
                telemetry.Context.Cloud.RoleName = "tasksmanager-backend-processor";
            }
        }
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.Extensibility;

namespace TasksTracker.TasksManager.Frontend.Ui
{
    public class AppInsightsTelemetryInitializer : ITelemetryInitializer
    {
        public void Initialize(ITelemetry telemetry)
        {
            if (string.IsNullOrEmpty(telemetry.Context.Cloud.RoleName))
            {
                //set custom role name here
                telemetry.Context.Cloud.RoleName = "tasksmanager-frontend-webapp";
            }
        }
    }
}

RoleName property for three services

The only difference between each file on the 3 projects is the RoleName property value.

Application Insights will utilize this property to recognize the elements on the application map. Additionally, it will prove beneficial for us in case we want to filter through all the warning logs produced by the Backend API service. Therefore, we will apply the tasksmanager-backend-api value for filtering purposes.

Next, we need to register this AppInsightsTelemetryInitializer class in Program.cs in each of the three projects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
using Microsoft.ApplicationInsights.Extensibility;
using TasksTracker.TasksManager.Backend.Api.Services;
using TasksTracker.TasksManager.Backend.Api;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) => {
    o.TelemetryInitializers.Add(new AppInsightsTelemetryInitializer());
});
builder.Services.AddDaprClient();
builder.Services.AddSingleton<ITasksManager, TasksStoreManager>();
//builder.Services.AddSingleton<ITasksManager, FakeTasksManager>();
builder.Services.AddControllers();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
using Microsoft.ApplicationInsights.Extensibility;
using TasksTracker.TasksManager.Backend.Svc;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) => {
    o.TelemetryInitializers.Add(new AppInsightsTelemetryInitializer());
});
builder.Services.AddControllers().AddDapr();
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.UseCloudEvents();

app.MapControllers();

app.MapSubscribeHandler();

app.Run();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
using Microsoft.ApplicationInsights.Extensibility;
using TasksTracker.TasksManager.Frontend.Ui;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) => {
    o.TelemetryInitializers.Add(new AppInsightsTelemetryInitializer());
});

builder.Services.AddRazorPages();

builder.Services.AddDaprClient();

builder.Services.AddHttpClient("BackEndApiExternal", httpClient =>
{
    var backendApiBaseUrlExternalHttp = builder.Configuration.GetValue<string>("BackendApiConfig:BaseUrlExternalHttp");

    if (!string.IsNullOrEmpty(backendApiBaseUrlExternalHttp)) {
        httpClient.BaseAddress = new Uri(backendApiBaseUrlExternalHttp);
    } else {
        throw new("BackendApiConfig:BaseUrlExternalHttp is not defined in App Settings.");
    }
});

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();

app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapRazorPages();

app.Run();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
using Microsoft.ApplicationInsights.Extensibility;
using TasksTracker.TasksManager.Backend.Api.Services;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) => {
    o.TelemetryInitializers.Add(new TasksTracker.TasksManager.Backend.Api.AppInsightsTelemetryInitializer());
});

builder.Services.AddSingleton<ITasksManager, FakeTasksManager>();
builder.Services.AddControllers();
// Learn more about configuring OpenAPI at https://aka.ms/aspnet/openapi
builder.Services.AddOpenApi();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
using Microsoft.ApplicationInsights.Extensibility;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) => {
    o.TelemetryInitializers.Add(new TasksTracker.TasksManager.Backend.Svc.AppInsightsTelemetryInitializer());
});

builder.Services.AddControllers().AddDapr();
builder.Services.AddControllers();
// Learn more about configuring OpenAPI at https://aka.ms/aspnet/openapi
builder.Services.AddOpenApi();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.UseCloudEvents();

app.MapControllers();

app.MapSubscribeHandler();

app.Run();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
using Microsoft.ApplicationInsights.Extensibility;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.Configure<TelemetryConfiguration>((o) => {
    o.TelemetryInitializers.Add(new TasksTracker.TasksManager.Frontend.Ui.AppInsightsTelemetryInitializer());
});

builder.Services.AddRazorPages();

builder.Services.AddDaprClient();

builder.Services.AddHttpClient("BackEndApiExternal", httpClient =>
{
    var backendApiBaseUrlExternalHttp = builder.Configuration.GetValue<string>("BackendApiConfig:BaseUrlExternalHttp");

    if (!string.IsNullOrEmpty(backendApiBaseUrlExternalHttp)) {
        httpClient.BaseAddress = new Uri(backendApiBaseUrlExternalHttp);
    } else {
        throw new("BackendApiConfig:BaseUrlExternalHttp is not defined in App Settings.");
    }
});

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}

app.UseHttpsRedirection();

app.UseRouting();

app.UseAuthorization();

app.MapStaticAssets();
app.MapRazorPages()
   .WithStaticAssets();

app.Run();

2.3 Set the Application Insights Instrumentation Key

In the previous module, we've used Dapr Secret Store to store connection strings and keys. In this module we will demonstrate how we can use another approach to secrets in Container Apps.

We need to set the Application Insights Instrumentation Key so that the projects are able to send telemetry data to the Application Insights instance. We are going to set this via secrets and environment variables once we redeploy the Container Apps and create new revisions. Locally, we can set it in each appsettings.json file. Obtain the key from the variable:

1
$APPINSIGHTS_INSTRUMENTATIONKEY

1
2
3
4
5
6
{
  // Configuration removed for brevity
  "ApplicationInsights": {
    "InstrumentationKey": "<Application Insights Key here for local development>"
  }
}

With this step completed, we have done all the changes needed. Let's now deploy the changes and create new ACA revisions.

3. Deploy Services to ACA and Create New Revisions

3.1 Add Application Insights Instrumentation Key As a Secret

Let's create a secret named appinsights-key on each Container App which contains the value of the Application Insights instrumentation key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
az containerapp secret set `
--name $BACKEND_API_NAME `
--resource-group $RESOURCE_GROUP `
--secrets "appinsights-key=$APPINSIGHTS_INSTRUMENTATIONKEY "

az containerapp secret set `
--name $FRONTEND_WEBAPP_NAME `
--resource-group $RESOURCE_GROUP `
--secrets "appinsights-key=$APPINSIGHTS_INSTRUMENTATIONKEY "

az containerapp secret set `
--name $BACKEND_SERVICE_NAME `
--resource-group $RESOURCE_GROUP `
--secrets "appinsights-key=$APPINSIGHTS_INSTRUMENTATIONKEY "

3.2 Build New Images and Push Them to ACR

As we did before, we are required to build and push the images of the three applications to ACR. By doing so, they will be prepared to be deployed in ACA.

To accomplish this, continue using the same PowerShell console and paste the code below (make sure you are on the following directory TasksTracker.ContainerApps):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Build Backend API on ACR and Push to ACR
az acr build `
--registry $AZURE_CONTAINER_REGISTRY_NAME `
--image "tasksmanager/$BACKEND_API_NAME" `
--file 'TasksTracker.TasksManager.Backend.Api/Dockerfile' .

# Build Backend Service on ACR and Push to ACR
az acr build `
--registry $AZURE_CONTAINER_REGISTRY_NAME `
--image "tasksmanager/$BACKEND_SERVICE_NAME" `
--file 'TasksTracker.Processor.Backend.Svc/Dockerfile' .

# Build Frontend Web App on ACR and Push to ACR
az acr build `
--registry $AZURE_CONTAINER_REGISTRY_NAME `
--image "tasksmanager/$FRONTEND_WEBAPP_NAME" `
--file 'TasksTracker.WebPortal.Frontend.Ui/Dockerfile' .

3.3 Deploy New Revisions of the Services to ACA and Set a New Environment Variable

We need to update all three container apps with new revisions so that our code changes are available for end users.

Tip

Notice how we used the property --set-env-vars to set new environment variable named ApplicationInsights__InstrumentationKey. Its value is a secret reference obtained from the secret appinsights-key we added in step 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Update Backend API App container app and create a new revision
az containerapp update `
--name $BACKEND_API_NAME  `
--resource-group $RESOURCE_GROUP `
--revision-suffix v$TODAY-5 `
--set-env-vars "ApplicationInsights__InstrumentationKey=secretref:appinsights-key"

# Update Frontend Web App container app and create a new revision
az containerapp update `
--name $FRONTEND_WEBAPP_NAME  `
--resource-group $RESOURCE_GROUP `
--revision-suffix v$TODAY-5 `
--set-env-vars "ApplicationInsights__InstrumentationKey=secretref:appinsights-key"

# Update Backend Background Service container app and create a new revision
az containerapp update `
--name $BACKEND_SERVICE_NAME `
--resource-group $RESOURCE_GROUP `
--revision-suffix v$TODAY-5 `
--set-env-vars "ApplicationInsights__InstrumentationKey=secretref:appinsights-key"

Success

With those changes in place, you should start seeing telemetry coming to the Application Insights instance provisioned. Let's review Application Insights' key dashboards and panels in Azure portal.

4. Visualizating Telemetry Data

4.1 Distributed Tracing Via Application Map

Application Map will help us spot any performance bottlenecks or failure hotspots across all our services of our distributed microservice application. Each node on the map represents an application component (service) or its dependencies and has a health KPI and alerts status.

distributed-tracing-ai

Looking at the image above, you will see for example how the Backend Api with could RoleName tasksmanager-backend-api is depending on the Cosmos DB instance, showing the number of calls and average time to service these calls. The application map is interactive so you can select a service/component and drill down into details.

For example, when we drill down into the Dapr State node to understand how many times the backend API invoked the Dapr Sidecar state service to Save/Delete state, you will see results similar to the image below:

distributed-tracing-ai-details

Note

It will take some time for the application map to fully populate.

4.2 Monitor Production Application Using Live Metrics

This is one of the key monitoring panels. It provides you with near real-time (1-second latency) status of your entire distributed application. We have the ability to observe both the successes and failures of our system, monitor any occurring exceptions and trace them in real-time. Additionally, we can monitor the live servers (including replicas) and track their CPU and memory usage, as well as the number of requests they are currently handling.

These live metrics provide very powerful diagnostics for our production microservice application. Check the image below and see the server names and some of the incoming requests to the system.

live-metrics-ai

Transaction search in Application Insights will help us find and explore individual telemetry items, such as exceptions, web requests, or dependencies as well as any log traces and events that we've added to the application.

For example, if we want to see all the event types of type Request for the cloud RoleName tasksmanager-backend-api in the past 24 hours, we can use the transaction search dashboard to do this. See how the filters are set and the results are displayed nicely. We can drill down on each result to have more details and what telemetry was captured before and after. A very useful feature when troubleshooting exceptions and reading logs.

transaction-search-ai

4.4 Failures and Performance Panels

The failure panel enables us to assess the frequency of failures across various operations, which assists us in prioritizing our efforts towards the ones that have the most significant impact.

failures-ai

The Performance panel displays performance details for the different operations in our system. By identifying those operations with the longest duration, we can diagnose potential problems or best target our ongoing development to improve the overall performance of the system.

failures-ai

  • Navigate to the root and persist the module to Git.

    1
2
    git add .
git commit -m "Add Module 8"

Review

In this module, we have accomplished four objectives:

  1. Learned how Azure Container Apps integrate with Application Insights to examine application telemetry.
  2. Configured Application Insights for the microservices.
  3. Deployed updated Background Processor, API, and UI projects to Azure.
  4. Understood how telemetry data is visualized.