ExpressRoute Gateway

The presented resiliency recommendations in this guidance include ExpressRoute Gateway and associated ExpressRoute Gateway settings.

Summary of Recommendations

Recommendations Details

ERGW-1 - Connect the ExpressRoute Gateway to two or more circuits from different peering locations for higher resiliency

Category: Availability

Impact: High

Guidance

Connect each ExpressRoute Gateway to a minimum of two circuits, with each circuit connecting from a diverse peering location compared to the other.

Resources

Resource Graph Query

// Azure Resource Graph Query
// Provides a list of ExpressRoute Gateways that are not connected to two or more ExpressRoute Circuits. Baremetal circuits are excluded from consideration
//This query assumes that the running entity has visibilty to the gateway, connection, and circuit scopes.
//Start with a full list of gateways
(resources
| where type == "microsoft.network/virtualnetworkgateways"
| where properties.gatewayType == "ExpressRoute"
| extend exrGatewayId = tolower(tostring(id))
| join kind=inner(
resources
| where type == "microsoft.network/virtualnetworkgateways"
| where properties.gatewayType == "ExpressRoute"
| extend exrGatewayId = tolower(tostring(id))
| join kind=leftouter(
//connections joined with circuit peer info
resources
| where type == "microsoft.network/connections"
| extend connectionType = properties.connectionType
| extend exrGatewayId = tolower(tostring(properties.virtualNetworkGateway1.id))
| extend peerId = tolower(tostring(properties.peer.id))
| extend connectionId = tolower(tostring(id))
| where connectionType == "ExpressRoute"
| join kind=leftouter(
  resources
  | where type == "microsoft.network/expressroutecircuits"
    //should this be location instead of peeringLocation
  | extend circuitId = tolower(tostring(id))
  | extend peeringLocation = tostring(properties.serviceProviderProperties.peeringLocation)
  | extend peerId = tolower(id)
) on peerId ) on exrGatewayId
//remove bare metal services connections/circuits
| where not(isnotnull(connectionId) and isnull(sku1))
//group by gateway ID's and peering locations
| summarize by exrGatewayId, peeringLocation
//summarize to connections with fewer than two unique connections
| summarize connCount = count() by exrGatewayId
| where connCount < 2) on exrGatewayId
| project recommendationId = "ergw-1", name, id, tags, param1 = "twoOrMoreCircuitsConnectedFromDifferentPeeringLocations: false")
| union
(
resources
| where type == "microsoft.network/virtualnetworkgateways"
| where properties.gatewayType == "ExpressRoute"
| extend exrGatewayId = tolower(tostring(id))
| join kind=leftouter(
//connections joined with circuit peer info
resources
| where type == "microsoft.network/connections"
| extend connectionType = properties.connectionType
| extend exrGatewayId = tolower(tostring(properties.virtualNetworkGateway1.id))
| extend peerId = tolower(tostring(properties.peer.id))
| extend connectionId = tolower(tostring(id))
| where connectionType == "ExpressRoute") on exrGatewayId
| where isnull(connectionType)
| project recommendationId = "ergw-1", name, id, tags, param1 = "twoOrMoreCircuitsConnectedFromDifferentPeeringLocations: false", param2 = "noConnectionsOnGateway: true"
)



ERGW-2 - Use Zone-redundant gateway SKUs

Category: Availability

Impact: High

Guidance

Azure ExpressRoute gateway provides different SLAs when it’s deployed in a single availability zone and when it’s deployed in two or more availability zones. For information about all Azure SLAs, see Service Level Agreements (SLA) for Online Services. To automatically deploy your virtual network gateways across availability zones, you can use zone-redundant virtual network gateways. With zone-redundant gateways, you can benefit from zone-resiliency to access your mission-critical, scalable services on Azure

Resources

Resource Graph Query

// Azure Resource Graph Query
// For all VNGs of type ExpressRoute, show any that do not have AZ in the SKU tier
resources
| where type =~ "Microsoft.Network/virtualNetworkGateways"
| where properties.gatewayType == "ExpressRoute"
| where properties.sku.tier !contains 'AZ'
| project recommendationId = "ergw-2", name, id, tags, param1= strcat("sku-tier: " , properties.sku.tier), param2=location
| order by id asc



ERGW-3 - Configure an Azure Resource lock for ExpressRoute Gateway to prevent accidental deletion

Category: Availability

Impact: Medium

Guidance

Configure an Azure Resource lock for ExpressRoute Gateway to prevent accidental deletion. As an administrator, you can lock an Azure subscription, resource group, or resource to protect them from accidental user deletions and modifications. The lock overrides any user permission.

Resources

Resource Graph Query

// under-development



ERGW-4 - Monitor gateway health

Category: Monitoring

Impact: High

Guidance

Setup monitoring using Network Insights for ExpressRoute Gateway availability, performance, and scalability.

Configure alerts for availability metrics for routes advertised, routes learned and number of VMs based on the supported amounts for the ExpressRoute Gateway SKU in use. Configure alerts for frequency of routes changed based on the customer environment.

Configure alerts for performance metrics for bits in, bits out and CPU utilization according to ExpressRoute Gateways | Azure Monitor Baseline Alerts. Configure alerts for packets per second based on the supported amount for the ExpressRoute Gateway SKU in use and based on the customer environment.

Configure alerts for scalability metrics for active flows based on the supported amounts for the ExpressRoute Gateway SKU in use and the expected number of flows for the customer environment, and for max flows per second for when this value exceeds a historical baseline for the customer environment.

Resources

Resource Graph Query

//The KQL files for this test are distributed into 7 different files.
//Make sure and run all 7 for complete coverage
.



ERGW-6 - Avoid using ExpressRoute circuits for VNet to VNet communication

Category: Networking

Impact: Medium

Guidance

By default, connectivity between virtual networks is enabled when you link multiple virtual networks, each with an ExpressRoute Gateway, to the same ExpressRoute circuit. However, Microsoft advises against using your ExpressRoute circuit for communication between virtual networks and instead use other techniques such as VNet peering, routing in a VNet hub via Azure Firewall, NVA and/or Azure Route Server, site-to-site VPN within Azure, the use of virtual WAN, or the use of SD-WAN.

For more information about why VNet-to-VNet connectivity isn’t recommended over ExpressRoute, see: Connectivity between virtual networks over ExpressRoute | Microsoft Learn

Resources

Resource Graph Query

// under-development



ERGW-7 - Configure customer-controlled gateway maintenance - In Preview

Category: Availability

Impact: High

Guidance

ExpressRoute virtual network gateways undergo regular updates to enhance functionality, reliability, performance, and security. Configuring and scheduling customer-controlled maintenance will minimize the impact of these updates and align the update schedule to best fit your maintenance windows.

Resources

Resource Graph Query

// Azure Resource Graph Query
// Find all Virtual Network Gateways without Maintenance Configurations

resources
| where type =~ "Microsoft.Network/virtualNetworkGateways"
| extend resourceId = tolower(id)
| join kind=leftouter  (
    maintenanceresources
    | where type =~ "Microsoft.Maintenance/configurationAssignments"
    | project JsonData = parse_json(properties)
    | extend maintenanceConfigurationId = tolower(tostring(JsonData.maintenanceConfigurationId))
    | join kind=inner (
        resources
        | where type =~ "Microsoft.Maintenance/maintenanceConfigurations"
        | project maintenanceConfigurationId=tolower(id)
    ) on maintenanceConfigurationId
    | project maintenanceConfigurationId, resourceId=tolower(tostring(JsonData.resourceId))
) on resourceId
| where isempty(maintenanceConfigurationId)
| project recommendationId = "ergw-7", name, id, tags, param1= strcat("sku-tier: " , properties.sku.tier), param2=location