Author: Nicholas Russo
Border Gateway Protocol (BGP) is an enormous protocol with a nearly endless list of features, knobs and capabilities. BGP’s mechanism for choosing the best path is complex but also well known. You should brush up on that algorithm if you’re out of practice. This blog explores a lesser-known but powerful traffic engineering technique using BGP communities and the local-preference value. You should have basic familiarity with BGP and IP routing in general to realize the most value from this blog.
Consider the following topology. R1 is a small business with two Internet uplinks, one through R2 and one through R4. R2 provides transit service for R1 into AS 65345, a large regional carrier. R1’s link to R4 is much slower (or higher latency, higher monetary cost, lower reliability, etc.) and should only be used as a backup. This policy decision should be implemented bidirectionally, applying to traffic egressing from R1 and for traffic ingressing to R1. I’m a big IPv6 fan and this network is IPv6-only. All the same concepts apply to IPv4 and dual-stacked networks as well. The full configurations for each device are at the end of this blog for those interested. Last, I’m using the RFC 3849 IPv6 documentation prefix to avoid any accidental overlap with production networks.
Handling R1’s egress traffic is straightforward and is easily achieved using local-preference applied inbound. As an example, R1 can apply a value of 100 (the default on Cisco routers) from R4 and apply a value of 200 from R2. Since higher values are preferred, all routers in AS 65001 will prefer R2 for egressing traffic towards AS 65345 and beyond. We won’t explore this configuration in-depth today.
Handling R1’s ingress is more complicated. Traditionally, there are three common solutions.
Use the Multi Exit Discriminator (MED)
This represents a BGP “metric” and hints at the internal IGP topology to a peer AS of the advertiser’s AS. In reality, many carriers ignore the MED value, and even if they honor it, it’s evaluated quite late in the BGP best-path algorithm. Worse still, it typically isn’t compared between different AS numbers, so the AS 65345 routers would not compare MED values from R1 and R2 directly. MED is not the correct tool for R1 in this scenario, as illustrated by the diagram below.
Use AS-path prepending
Slightly more effective than MED, this allows an advertiser to artificially lengthen the AS-path, making a prefix less desirable across a given link. In a simple case, this could solve R1’s problem, as R1 could just prepend 65001 on its advertisements towards R4 so that AS 65345 prefers R3 as the egress point towards R1 with R2 as an intermediate hop. However, this is not guaranteed. Suppose R2 is prepending AS numbers towards R3 to make R4 the preferred egress point from AS 65345! That would foil R1’s plan quite nicely. Also, consider a case where instead of a single AS, such as AS 65002, there is a large number of AS hops between AS 65001 and AS 65345. This would have the same effect on the AS-path length and leave R1 guessing as to the correct number of prepend actions. The diagram below illustrates a failed attempt by R1 to use this technique.
Use longest-match routing
Advertising more specific prefixes, such as a single /127 via the backup link and two /128s via the primary link, is a reliable technique to guarantee ingress, as illustrated below. Upstream carriers seldom aggregate or filter customer prefixes (unless they are allocated from a provider aggregate range, of course), but there is a drawback. For IPv4, /24 is the shortest prefix allowed on the Internet, and for IPv6, /48 is the shortest. If all you have is an IPv6 /48, you can’t carve it into two /49s without a special arrangement with your upstream ISPs (good luck with that). Let’s assume that this technique is not currently available to Globomantics.
So, what is our customer to do? This is a common issue and has vexed many businesses. In response to this, service providers have developed an alternative solution that relies on BGP communities. Communities are like sticky notes; they can be attached to various routes to signal arbitrary attributes of a route. A service provider can match a given community for a received route, then set the local-preference to a corresponding value. In plain English, the customer says, “I want you to set local-preference X on this prefix, so I’ve attached community Y to signal that”. Implementations will inevitably vary between vendors, but broadly speaking, a giant if/then statement is evaluated to map community values to local-preference values.
Speaking from personal experience, I once consulted for a global service provider where we implemented this exact technique. Our policy contained four local-preference options and the first step is to configure the community-lists. These are the community matching constructs used in Cisco IOS. Each list will match exactly one community value, and these are arbitrary; they can be any value you prefer. Many service providers will use a format where the first value is their local AS, which is 65345 in our demo. The second value is often the desired local-preference value, making the community value self-documenting. Service providers typically publish these communities on their support portals for quick reference; customers don’t even need to ask permission before leveraging them!
# R4
ip community-list standard CL_LPREF_80 permit 65345:80
ip community-list standard CL_LPREF_90 permit 65345:90
ip community-list standard CL_LPREF_110 permit 65345:110
ip community-list standard CL_LPREF_120 permit 65345:120
Here’s a Cisco IOS example route-map that illustrates the conditional logic. Assuming a default local-preference of 100, customers can choose from values of 80, 90, 110 or 120. Therefore, customers have two options to make a prefix less preferred (80 and 90) and two options to make a prefix more preferred (110 and 120). Note that the policy should have a catch-all clause that handles routes without any communities attached. In this example, I’m permitting all routes with no modification, but this default action may vary between environments. R4 will apply this policy inbound on the BGP session with R1.
# R4
route-map RM_CUSTOMER_COMM_LPREF_IN permit 10
match community CL_LPREF_80
set local-preference 80
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 20
match community CL_LPREF_90
set local-preference 90
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 30
match community CL_LPREF_110
set local-preference 110
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 40
match community CL_LPREF_120
set local-preference 120
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 1000
!
router bgp 65345
address-family ipv6
neighbor 2001:DB8:1:4::1 route-map RM_CUSTOMER_COMM_LPREF_IN in
The final step is to enable the transmission of communities from R1 to R4, at a minimum. Cisco IOS does not send communities to BGP peers by default. This step also includes applying the desired community value outbound. To keep this blog and the subsequent configurations as simple as possible, I’m not matching specific prefixes for this policy application. Feel free to apply more precise matching criteria in your production environments as necessary. R1 must apply either 65345:80 or 65345:90 in this design, signaling that AS 65345 should not prefer R4 as an egress point towards R1.
# R1
route-map RM_BGP_COMM_R4_OUT permit 10
set community 65345:80
!
router bgp 65001
address-family ipv6
network 2001:DB8::1/128
neighbor 2001:DB8:1:4::4 send-community
neighbor 2001:DB8:1:4::4 route-map RM_BGP_COMM_R4_OUT out
Now, let’s verify everything. On R4, let’s check the BGP table for 2001:db8::1/128, the prefix from R1. R4 has two paths. The eBGP path from R1 has a community of 65345:80, a local-preference of 80, and it is not the best path. The iBGP path from R2 has no community, a local-preference of 100 (default) and it is the best path. This iBGP path from R2 even has a giant AS-path, which R1 successfully overrode by signaling to AS 65345 that the egress point of R4 should be less preferred. The most important part of this output is the “localpref 80” text which indicates a successful policy implementation.
R4#show bgp ipv6 unicast 2001:db8::1/128
BGP routing table entry for 2001:DB8::1/128, version 21
Paths: (2 available, best #1, table default)
Advertised to update-groups:
4
Refresh Epoch 6
65002 65002 65002 65002 65001
2001:DB8::3 (metric 10) from 2001:DB8::3 (192.0.2.3)
Origin IGP, metric 0, localpref 100, valid, internal, best
rx pathid: 0, tx pathid: 0x0
Refresh Epoch 8
65001
2001:DB8:1:4::1 (FE80::1) from 2001:DB8:1:4::1 (192.0.2.1)
Origin IGP, metric 0, localpref 80, valid, external
Community: 65345:80
rx pathid: 0, tx pathid: 0
Since R4 learned its best path from R3, let’s head there next. There’s only one BGP route available, which is the best path via R2 using eBGP. Again, the giant AS-path doesn’t matter since the local-preference set by R4 is less than the local-preference set by R3. BGP routers can only advertise their best paths, so none of the other AS 65345 routers currently know about R4’s backup path. Designing a BGP fast-failover backup plan is a problem for another day.
R3#show bgp ipv6 unicast 2001:db8::1/128
BGP routing table entry for 2001:DB8::1/128, version 27
Paths: (1 available, best #1, table default)
Advertised to update-groups:
3
Refresh Epoch 5
65002 65002 65002 65002 65001
2001:DB8:2:3::2 (FE80::2) from 2001:DB8:2:3::2 (192.0.2.2)
Origin IGP, localpref 100, valid, external, best
rx pathid: 0, tx pathid: 0x0
We can continue our network verification by heading to R5. R5 is only aware of the egress path via R3 using iBGP. Again, without using the community/local-preference technique just described, R1 would be left performing trial-and-error AS-path prepending (not ideal).
R5#show bgp ipv6 unicast 2001:db8::1/128
BGP routing table entry for 2001:DB8::1/128, version 20
Paths: (1 available, best #1, table default)
Advertised to update-groups:
2
Refresh Epoch 6
65002 65002 65002 65002 65001
2001:DB8::3 (metric 10) from 2001:DB8::3 (192.0.2.3)
Origin IGP, metric 0, localpref 100, valid, internal, best
rx pathid: 0, tx pathid: 0x0
Let’s quickly confirm that traffic from R6, a device beyond AS 65345, can reach R1 via R2 instead of the direct R1-R4 link. Using traceroute, we target R1’s loopback sourced from R6’s loopback, and the routed path is R5-R3-R2-R1 as expected.
R6#traceroute 2001:db8::1 source 2001:db8::6
Type escape sequence to abort.
Tracing the route to 2001:DB8::1
1 2001:DB8:5:6::5 6 msec 5 msec 5 msec
2 2001:DB8::3 4 msec 6 msec 5 msec
3 2001:DB8:2:3::2 2 msec 6 msec 5 msec
4 2001:DB8:1:2::1 2 msec 2 msec 5 msec
To summarize: If you find yourself wanting to influence ingress traffic towards your AS but the traditional methods of MED, AS-path prepending, or longest-match routing are inadequate or unavailable to you, check your service provider’s website. You might be pleasantly surprised! If you enjoyed this blog and my no-nonsense, tech-focused style of training, you’ll love my Cisco Advanced Routing series on Pluralsight.
Reference configurations
hostname R1
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
ipv6 address 2001:DB8::1/128
!
interface Ethernet0/0
ipv6 address FE80::1 link-local
ipv6 address 2001:DB8:1:4::1/64
!
interface Ethernet0/1
ipv6 address FE80::1 link-local
ipv6 address 2001:DB8:1:2::1/64
!
router bgp 65001
bgp router-id 192.0.2.1
no bgp default ipv4-unicast
neighbor 2001:DB8:1:2::2 remote-as 65002
neighbor 2001:DB8:1:4::4 remote-as 65345
!
address-family ipv6
network 2001:DB8::1/128
neighbor 2001:DB8:1:2::2 activate
neighbor 2001:DB8:1:2::2 route-map RM_BGP_LPREF_R2_IN in
neighbor 2001:DB8:1:4::4 activate
neighbor 2001:DB8:1:4::4 send-community
neighbor 2001:DB8:1:4::4 route-map RM_BGP_COMM_R4_OUT out
exit-address-family
!
ip bgp-community new-format
!
route-map RM_BGP_LPREF_R2_IN permit 10
set local-preference 200
!
route-map RM_BGP_COMM_R4_OUT permit 10
set community 65345:80
hostname R2
ipv6 unicast-routing
ipv6 cef
!
interface Ethernet0/0
ipv6 address FE80::2 link-local
ipv6 address 2001:DB8:2:3::2/64
!
interface Ethernet0/1
ipv6 address FE80::2 link-local
ipv6 address 2001:DB8:1:2::2/64
!
router bgp 65002
bgp router-id 192.0.2.2
no bgp default ipv4-unicast
neighbor 2001:DB8:1:2::1 remote-as 65001
neighbor 2001:DB8:2:3::3 remote-as 65345
!
address-family ipv6
neighbor 2001:DB8:1:2::1 activate
neighbor 2001:DB8:2:3::3 activate
neighbor 2001:DB8:2:3::3 route-map RM_BGP_ASPREP_R3_OUT out
exit-address-family
!
ip bgp-community new-format
!
route-map RM_BGP_ASPREP_R3_OUT permit 10
set as-path prepend 65002 65002 65002
hostname R3
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
ipv6 address 2001:DB8::3/128
ospfv3 1 ipv6 area 0
!
interface Ethernet0/0
ipv6 address FE80::3 link-local
ipv6 address 2001:DB8:2:3::3/64
!
interface Ethernet0/1
ipv6 address FE80::3 link-local
ospfv3 1 ipv6 area 0
ospfv3 1 ipv6 network point-to-point
!
interface Ethernet0/2
ipv6 address FE80::3 link-local
ospfv3 1 ipv6 area 0
ospfv3 1 ipv6 network point-to-point
!
router ospfv3 1
router-id 192.0.2.3
!
address-family ipv6 unicast
exit-address-family
!
router bgp 65345
bgp router-id 192.0.2.3
no bgp default ipv4-unicast
neighbor 2001:DB8::4 remote-as 65345
neighbor 2001:DB8::4 update-source Loopback0
neighbor 2001:DB8::5 remote-as 65345
neighbor 2001:DB8::5 update-source Loopback0
neighbor 2001:DB8:2:3::2 remote-as 65002
!
address-family ipv6
neighbor 2001:DB8::4 activate
neighbor 2001:DB8::4 next-hop-self
neighbor 2001:DB8::5 activate
neighbor 2001:DB8::5 next-hop-self
neighbor 2001:DB8:2:3::2 activate
exit-address-family
!
ip bgp-community new-format
hostname R4
ip cef
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
ipv6 address 2001:DB8::4/128
ospfv3 1 ipv6 area 0
!
interface Ethernet0/0
ipv6 address FE80::4 link-local
ipv6 address 2001:DB8:1:4::4/64
!
interface Ethernet0/1
ipv6 address FE80::4 link-local
ospfv3 1 ipv6 area 0
ospfv3 1 ipv6 network point-to-point
!
interface Ethernet0/3
ipv6 address FE80::4 link-local
ospfv3 1 ipv6 area 0
ospfv3 1 ipv6 network point-to-point
!
router ospfv3 1
router-id 192.0.2.4
!
address-family ipv6 unicast
exit-address-family
!
router bgp 65345
bgp router-id 192.0.2.4
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 2001:DB8::3 remote-as 65345
neighbor 2001:DB8::3 update-source Loopback0
neighbor 2001:DB8::5 remote-as 65345
neighbor 2001:DB8::5 update-source Loopback0
neighbor 2001:DB8:1:4::1 remote-as 65001
!
address-family ipv6
neighbor 2001:DB8::3 activate
neighbor 2001:DB8::3 next-hop-self
neighbor 2001:DB8::5 activate
neighbor 2001:DB8::5 next-hop-self
neighbor 2001:DB8:1:4::1 activate
neighbor 2001:DB8:1:4::1 route-map RM_CUSTOMER_COMM_LPREF_IN in
exit-address-family
!
ip bgp-community new-format
ip community-list standard CL_LPREF_80 permit 65345:80
ip community-list standard CL_LPREF_90 permit 65345:90
ip community-list standard CL_LPREF_110 permit 65345:110
ip community-list standard CL_LPREF_120 permit 65345:120
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 10
match community CL_LPREF_80
set local-preference 80
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 20
match community CL_LPREF_90
set local-preference 90
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 30
match community CL_LPREF_110
set local-preference 110
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 40
match community CL_LPREF_120
set local-preference 120
!
route-map RM_CUSTOMER_COMM_LPREF_IN permit 1000
hostname R5
ipv6 unicast-routing
ipv6 cef
!
interface Loopback0
ipv6 address 2001:DB8::5/128
ospfv3 1 ipv6 area 0
!
interface Ethernet0/0
ipv6 address FE80::5 link-local
ipv6 address 2001:DB8:5:6::5/64
!
interface Ethernet0/2
ipv6 address FE80::5 link-local
ipv6 address 2001:DB8:3:5::5/64
ospfv3 1 ipv6 area 0
ospfv3 1 ipv6 network point-to-point
!
interface Ethernet0/3
ipv6 address FE80::5 link-local
ospfv3 1 ipv6 area 0
ospfv3 1 ipv6 network point-to-point
!
router ospfv3 1
router-id 192.0.2.5
!
address-family ipv6 unicast
exit-address-family
!
router bgp 65345
bgp router-id 192.0.2.5
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 2001:DB8::3 remote-as 65345
neighbor 2001:DB8::3 update-source Loopback0
neighbor 2001:DB8::4 remote-as 65345
neighbor 2001:DB8::4 update-source Loopback0
neighbor 2001:DB8:5:6::6 remote-as 65006
!
address-family ipv6
neighbor 2001:DB8::3 activate
neighbor 2001:DB8::3 next-hop-self
neighbor 2001:DB8::4 activate
neighbor 2001:DB8::4 next-hop-self
neighbor 2001:DB8:5:6::6 activate
exit-address-family
!
ip bgp-community new-format