Observability

Home

Table Of Contents

• Logging
• Runtime diagnostics output
• Available metrics
• IoT Edge Metrics collector
• OpenTelemetry
• Prometheus
  • Setup
  • View Prometheus dashboard
  • View Grafana dashboard

Logging

Logging can be configured using the --ll command line argument. It allows to set a desired log level. More fine grained logging can be configured using an appsettings.json file as per .net documentation.

It is also possible to enable more detailed logging of the OPC UA stack using the --sl option.

To get the logs the iotedge command line utility can be used. Run the following from the local command line:

iotedge logs -f <container name>

These logs contain the current state of operation and can be used to narrow down an issue.

Additionally, logs from edge modules can be fetched via direct methods describes how to fetch and upload logs from Edge modules.

Runtime diagnostics output

The OPC Publisher emits metrics through its Prometheus endpoint (/metrics). To learn more about how to create a local metrics dashboard for OPC Publisher V2.7, refer to the tutorial here.

To measure the performance of OPC Publisher, the di parameter can be used to print metrics to the log in the interval specified (in seconds). Using --pd=Events the diagnostics information can instead be emitted to a topic or to IoT Hub. The routing or topic can be configured using the --dtt command line option.

The following table describes the instruments that are collected per writer group and subsequently emitted:

(*) Not exposed through the API

Available metrics

By convention, all instrument names start with “iiot”, e.g., iiot_<component>_<metric>. Metrics are collected through the .net Observability infrastructure. For backwards compatibility and to support IoT Edge metrics collector, metrics from OPC Publisher are exposed in Prometheus format on path /metrics on the default HTTP server port (see -p command line argument. When binding the HTTPS port to 9072 on the host machine, the URL becomes https://localhost:9072/metrics.

One can combine information from multiple metrics to understand and paint a bigger picture of the state of the publisher. The following table describes the individual default metrics which are in Prometheus format.

You can use the IoT Edge Metrics Collector module to collect the metrics.

Edge modules would be instrumented with Prometheus metrics. Each module would expose the metrics on a pre-defined port. The metricscollector module would use the configuration settings to scrape metrics in a defined interval. It would then push the scraped metrics to Log Analytics Workspace. Using Azure Data Explorer (ADX), we can then query our metrics from workspace. We are creating and deploying an Azure Workbook which would provide insights into the edge modules. This would act as our primary monitoring system for edge modules.

IoT Edge Metrics collector

The metrics collector module pulls metrics exposed by OPC Publisher and pushes them to the Log Analytics workspace.

The default configuration to enable scraping metrics is:

{
    "schemaVersion": "1.0",
    "scrapeFrequencySecs": 120,
    "metricsFormat": "Json",
    "syncTarget": "AzureLogAnalytics",
    "endpoints": {
        "opcpublisher": "http://opcpublisher/metrics"
    }
}

This assumes the http server in OPC Publisher has not been disabled. Disabling the internal HTTP server also disables the Prometheus endpoint.

OpenTelemetry

It is possible to export all metrics, traces and logs to an OpenTelemetry (OTEL) collector. To specify the OTLP GRPC endpoint to export to, use the --oc command line argument.

An example setup using the OpenTelemetry (OTEL) Contrib Collector and Grafana stack can be found in the /deploy/docker folder.

Prometheus

OPC Publisher module also exposes all metrics to a Prometheus scraper (at the standard /metrics path). For troubleshooting and debugging, a local Grafana dashboard can be configured on Azure IoT Edge.

It is recommended to use the OpenTelemetry collector and expose metrics from the collector to Prometheus.

This section shows you how to setup and view OPC Publisher and EdgeHub metrics as bird’s eye view and how to quickly drill down into the issues in case of failures.

In this tutorial two pre-configured docker images (for Prometheus and Grafana) must be created and deployed as Azure IoT Edge Modules. The Prometheus Module scrapes the metrics from OPC Publisher and EdgeHub, while Grafana is used for visualization of metrics.

Setup

1. Create and configure Azure Container Registry to have an admin account enabled. Registry name and password are needed for pushing the docker images of Prometheus and Grafana.

   

2. Create and push images to this ACR:

   • Download the zip (metricsdashboard.zip) located in this folder and extract it.

   • Navigate to prometheus folder as shown:

     

   • Edit prometheus.yml if needed. It defaults to scraping EdgeHub metrics at 9600 and OPC Publisher metrics at 9702. If only OPC Publisher metrics are needed, then remove the scraping config of EdgeHub.

   • Run buildimage.bat and enter the registryname, password and tagname when prompted. It will push the edgeprometheus image to container registry.

     

   • Navigate back to the grafana folder and run buildimage.bat located in this folder. Enter the same information to push the edgegrafana image to ACR.

3. Now go to the portal and verify that the images are present, as shown in the image below.

   

4. IoT Edge should now be configured to expose metrics. Navigate to the IoT Edge device to be used and perform the following steps:

   • Select Set Modules to update the configuration

   • Enter the registry information in the Container Registry Credentials so that it can authenticate with the Azure Container Registry.

   

   • Optional: Enable metrics of edgeHub (Note: As of release 1.0.10, metrics are automatically exposed by default on http port 9600 of the EdgeHub)

     • Select the Runtime Settings option and for the EdgeHub and Agent, make sure that the stable version is selected with version 1.0.9.4+

       

     • Adjust the EdgeHub Create Options to include the Exposed Ports directive as shown above.

     • Add the following environment variables to the EdgeHub. (Not needed for version 1.0.10+)

       ExperimentalFeatures__Enabled : true

       ExperimentalFeaturesEnable__Metrics : true

     • Save the dialog to complete the runtime adjustments.

   • Expose port on publisher module:

     • Navigate to the publisher and expose port 9702 to allow the metrics to be scraped, by adding the exposed ports option in the Container Create Options in the Azure portal.

       

     • Select Update to complete the changes made to the OPC publisher module.

5. Now add new modules based on the Prometheus and Grafana images created previously.

   • Prometheus:

     • Add a new “IoT Edge Module” from the “+ Add” drop down list.

     • Add a module named prometheus with the image uploaded previously to the Azure Container Registry.

       

     • In the Container Create Options expose and bind the Prometheus port 9090.

       {
           "Hostname": "prometheus",
           "ExposedPorts": {
               "9090/tcp": {}
           },
           "HostConfig": {
               "PortBindings": {
                   "9090/tcp": [
                       {
                           "HostPort": 9090
                       }
                   ]
               }
           }
       }
       

     • Click Update to complete the changes to the Prometheus module.

     • Select Review and Create and complete the deployment by choosing Create.

   • Grafana:

     • Add a new “IoT Edge Module” from the “+ Add” drop down list.

     • Add a module named grafana with the image uploaded previously to the Azure Container Registry.

       

     • Update the environment variables to control access to the Grafana dashboard.

       • GF_SECURITY_ADMIN_USER - the user name you want to use

       • GF_SECURITY_ADMIN_PASSWORD - a password of your choice

       • GF_SERVER_ROOT_URL - http://localhost:3000

         

     • In the Container Create Options expose and bind the Grafana port 3000.

       {
           "Hostname": "grafana",
           "ExposedPorts": {
               "3000/tcp": {}
           },
           "HostConfig": {
               "PortBindings": {
                   "3000/tcp": [
                       {
                           "HostPort": 3000
                       }
                   ]
               }
             }
        }
       

     • Click Update to complete the changes to the Grafana module.

     • Select Review and Create and complete the deployment by choosing Create.

   • Verify all the modules are running successfully:

     

View Prometheus dashboard

• Navigate to the Prometheus dashboard through a web browser on the IoT Edge’s host on port 9090.

  • http://{edge host IP or name}:9090/graph
  • Note: When using a VM, make sure to add an inbound rule for port 9090

• Check the target state by navigating to /targets

  

• Metrics can be viewed as shown:

  

• 

View Grafana dashboard

• Navigate to the Grafana dashboard through a web browser against the edge’s host on port 3000.
  • http://{edge host IP or name}:3000
  • When prompted for a user name and password enter the values entered in the environment variables
  • Note: When using a VM, make sure to add an inbound rule for port 3000

• Prometheus has already been configured as a data source and can now be directly accessed. Prometheus is scraping EdgeHub and OPC Publisher metrics.

• Select the dashboards option to view the available dashboards and select “Publisher” to view the pre-configured dashboard as shown below.

  

• One could easily add/remove more panels to this dashboard as per the needs.