ExpressRoute Direct
The presented resiliency recommendations in this guidance include ExpressRoute Direct and associated resources and settings.
Summary of Recommendations
| Recommendation | Category | Impact | State | ARG Query Available |
|---|---|---|---|---|
| ERD-1 - The Admin State of both Links of an ExpressRoute Direct should be in Enabled state | Availability | High | Verified | No |
| ERD-2 - Ensure you do not over-subscribe an ExpressRoute Direct | System Efficiency | High | Verified | No |
| ERD-3 - Enable rate-limiting to help optimize network performance by controlling the traffic volume across all your ExpressRoute Direct based circuits - In Preview | System Efficiency | Medium | Verified | No |
Recommendations Details
ERD-1 - The Admin State of both Links of an ExpressRoute Direct should be in Enabled state
Category: Availability
Impact: High
Recommendation/Guidance
In Azure ExpressRoute Direct, the “Admin State” refers to the administrative status of the ExpressRoute layer 1 links. It essentially indicates whether a particular link is enabled or disabled, in other words the physical port is on or off; and is required to pass traffic across the ExpressRoute Direct connection. Admin State is a crucial setting because it determines the operational status of your ExpressRoute Direct, affecting connectivity between your on-premises network and Azure services.
Resources
Resource Graph Query
// Azure Resource Graph Query
// Find all Express Route Directs that do not have Admin State of both Links Enabled
resources
| where type == "microsoft.network/expressrouteports"
| where properties['links'][0]['properties']['adminState'] == "Disabled" or properties['links'][1]['properties']['adminState'] == "Disabled"
| project recommendationId = "erd-1", name, id, tags, param1 = strcat("Link1AdminState: ", properties['links'][0]['properties']['adminState']), param2 = strcat("Link2AdminState: ", properties['links'][1]['properties']['adminState'])
ERD-2 - Ensure you do not over-subscribe an ExpressRoute Direct
Category: System Efficiency
Impact: High
Recommendation/Guidance
You can provision logical ExpressRoute circuits on top of your selected ExpressRoute Direct resource of 10-Gbps or 100-Gbps up to the subscribed Bandwidth of 20-Gbps or 200-Gbps. From a resiliency perspective this is not recommended. If one of the ExpressRoute Direct ports goes down, and your ExpressRoute circuits are already consuming 100% of the 10-Gbps or 100-Gbps, the second ExpressRoute Direct port wouldn’t have bandwidth enough to support any additional load. One reason a port may be down would be during a maintenance event. The remaining port would support all traffic during the maintenance event, up to the 10-Gbps or 100-Gbps capacity. Unless you use rate limiting for ExpressRoute Direct circuits (Preview) to limit the bandwidth of non-production connections, you should not over-subscribe your ExpressRoute Direct ports being used for production workloads.
Resources
Resource Graph Query
// Azure Resource Graph Query
// Find all Express Route Directs that are over subscribed
resources
| where type == "microsoft.network/expressrouteports"
| where toint(properties['provisionedBandwidthInGbps']) > toint(properties['bandwidthInGbps'])
| project recommendationId = "erd-2", name, id, tags, param1 = strcat("provisionedBandwidthInGbps: ", properties['provisionedBandwidthInGbps']), param2 = strcat("bandwidthInGbps: ", properties['bandwidthInGbps'])
ERD-3 - Enable rate-limiting to help optimize network performance by controlling the traffic volume across all your ExpressRoute Direct based Circuits - In Preview
Category: System Efficiency
Impact: Medium
Recommendation/Guidance
Rate limiting is a feature that enables you to control the traffic volume between your on-premises network and Azure over an ExpressRoute Direct circuit. It applies to the traffic over either private or Microsoft peering of the ExpressRoute circuit. This feature helps distribute the port bandwidth evenly among the circuits, ensures network stability, and prevents network congestion. This document outlines the steps to enable rate limiting for your ExpressRoute Direct circuits.
Resources
Resource Graph Query
// under-development