In today’s world, network attacks occurs everyday, targeting all organisations, from the smallest to the biggest.
Denial of Services attacks (DoS or DDoS) are innumerable, and their scale increases from month to month.
These attacks can disrupt services, overwhelm network infrastructure, and cause significant financial and reputational damage to organizations.
However, network administrators have a powerful ally in their arsenal to combat such threats : the Border Gateway Protocol (BGP).
In this article, we will explore how BGP can be leveraged to mitigate DoS attacks, focusing on key techniques such as Remote Triggered Black Hole Filtering (RTBH), Source-Based RTBH, and BGP Flow Specification (FlowSpec).
Border Gateway Protocol – The technique
BGP is well known as being the Internet routing protocol.
Despite it has been adapted for more and more “internal networks” use-cases, its main principle is not to route “between routers” (IGPs are more adapted to this use-case) but to route “between ASes” (Autonomous Systems).
One of the key features that makes Border Gateway Protocol (BGP) a powerful tool for mitigating Denial of Service (DoS) attacks is its ability to announce routes with arbitrary next-hops. BGP allows routers to advertise routes to destinations for which they are not the next-hop, effectively influencing the path that traffic takes through the network.
This capability is only provided by BGP, while common routing protocols (IGP) do not transport any next-hop information on their updates : this information is computed by the router which receives the update, which automatically considers the announcing router as the next-hop.
This next-hop manipulation capability of BGP, permitted by the fact that the next-hop information is part of the path attributes (see below) forms the basis for techniques such as Remote Triggered Black Hole (RTBH) filtering and its variant, Source-Based Remote Triggered Black Hole (Source RTBH) filtering.
It allows network administrators to strategically announce black hole routes, diverting malicious traffic away before it reaches the target network, thus preserving bandwidth and protecting critical services from disruption.
Remote-Triggered Black Hole
RTBH is the first interesting feature (or let’s say “technique”) of BGP as a security tool within service providers networks we will discuss in this article.
This filtering method involves announcing a route for the target IP address of the victim under attack with the next-hop set to a specific black hole value (agreed with the ISP). It is very useful for DDoS (Distributed Denial of Service) attacks, as it permits to drop traffic only for the specific attacked IP, while not stopping the announcement of the full company subnet.
Essentially, this instructs routers in the ISP network to drop any traffic destined for the targeted IP address, effectively mitigating the impact of the attack.
The announcement of the black hole route triggers routers to discard traffic destined for the victim, preventing it from reaching its intended destination.
Depending of the way the black hole routes propagation is configured on the ISP network, it can even avoid the offending traffic to enter the ISP backbone, as we’ll see below.
This announcement for black-holing the victim’s subnet (or more specific IPs on this subnet) can be done using a bogus next-hop value (generally in the 192.0.2.0/24 range, called “Test-Net” which is used for tests and documentation, and more specifically using the 192.0.2.1 address).
A recursive route to this IP (pointing to the null0 interface) has to be pre-installed in the ISP’s edge routers so that the announced RTBH route will make the ISP routers dropping the concerned traffic before it reaches the victim’s interconnexion link : RTBH is a set of well-coordinated configurations across multiple routers. Announcing a black hole route (with the specific next-hop value) will have no impact if the ISP’s discard-routes for this next-hop does not exists.
Those black hole announcements should include the “no-export” well-known community to make sure they are not re-announced by the ISP to its peers AS : this specific route should only reach the edge routers of the ISP, which are responsible for dropping the concerned traffic.
It is also agreed to add a specific community to the announced prefix, which permits to add a second safety precaution to avoid it to be redistributed beyond the ISP’s network, and also adds some filtering scope options.
As specified on RFC 7999, the blackhole communities need to be chosen with the two lowest-order octets being BLACKHOLE (= 0xFFFF029A) (666 converted to decimal) : each community used for blackholing (in what is called the “nex-format” is then in the X:666 format.
When the attack is over, the customer needs to send a withdrawal for the black holed prefix in order to receive the associated traffic again.
For information, see below a small list of the RTBH community of some well-known providers. (You’ll notice that only a few of them use the well-known community value of 666).
| Provider | RTBH Community |
|---|---|
| AT&T | 7018:86 |
| GTT / TiNet | 3257:2666 |
| Hurricane Electric | 6939:666 |
| Level3 | 3356:9999 |
| MTS Allstream | 15290:9999 |
| Qwest | 209:2 |
| Sprint | 1239:66 |
| Verizon / MCI | 701:9999 |
RTBH announce trigger
Black hole route announcement can be triggered by different sources, each having different pros and cons :
Customer-router black hole initiation (as shown above)
The customer router eventually detects the attack and initiates the announcement of the black hole route to the ISP network (manually or automatically).
– Pros : the customer maintains full control on the black hole announcement, and is fully autonomous on the attack mitigation without relying on any ISP intervention or coordination.
– Cons : the customer may have limited visibility into the overall network topology and attack traffic, potentially leading to suboptimal mitigation decisions. Also, the customer might not be able to know when the attack is over (DDoS traffic stopped, in the current scenario), and thus will probably have to rely on ISP’s information to remove the black hole route announcement. If the customer router becomes overwhelmed by the attack traffic, it may struggle to initiate black hole route announcements effectively.
Customer triggering-host black hole initiation
In this method, a specific triggering host is located on the customer-side network and attached to the ISP network through another BGP peering. This host sole function could be to detect the attack and announce the black hole route (again manually or automatically).
– Pros : same than with the previous method. It also addresses the issue where the CE router is overwhelmed by the attack and cannot be used for the black hole route announcement.
– Cons : This black hole triggering host has to be carefully managed / monitored as it could become a single point of failure during the attack. With or without having this host redundant, this method adds some complexity to the customer network.
Triggering-host on ISP network
On this third scenario, the RTBH triggering host is part of the ISP network, and allows for centralized detection and initiation of black hole routes announcement.
– Pros : Placing the triggering host within the ISP network centralizes control and coordination of black hole route announcements, potentially leading to more efficient and effective mitigation. Also, the ISP’s network operations team may have greater expertise and resources for detecting and mitigating attacks compared to individual customers (well, that’s not always true ^^).
– Cons : Granting the ISP access to customer traffic for detection purposes may raise privacy and security concerns, requiring careful consideration of data protection measures. Customers become reliant on the ISP for initiating black hole route announcements, reducing autonomy, reactivity and control over mitigation actions.
RTBH : Example
Here is a quick extract of part of the configuration needed to put in place the RTBH example seen above, with the customer router being used for the announcement of the black hole route.
PE routers configuration :
router bgp 65001
neighbor 10.10.10.10 remote-as 65001
neighbor 10.10.10.10 update-source Loopback0
neighbor 10.10.10.10 send-community
neighbor CUSTOMER send-community
!
address-family ipv4
neighbor 10.10.10.10 activate
neighbor CUSTOMER activate
neighbor CUSTOMER route-map RTBH-in in
ip route 192.2.0.1/32 Null0 name RTBH
ip community-list RTBH permit 100:666
route-map RTBH-in permit 10
match community RTBH
set community no-export
set ip next-hop 192.2.0.1
route-map RTBH-in permit 20The RTBH-in route-map matches the customer-announced routes having the “100:666” community, and sets the 192.2.0.1 value as next-hop for those routes : this is the value matched by the recursive routes on all other PEs, routing to null0.
Note also that the “no-advertise” well-known community is also added to the route, avoiding it to be redistributed beyond the ISP AS.
The neighbor “CE” is added to the “CUSTOMER” peer-group on PE4.
Then, here’s the configuration of the CE router :
router bgp 65666
router-id 100.100.100.100
neighbor 192.168.1.1 remote-as 65001
neighbor 192.168.1.1 send-community
!
address-family ipv4
neighbor 192.168.1.1 activate
neighbor 192.168.1.1 route-map RTBH out
network 128.66.0.0/24
redistribute static
route-map RTBH permit 10
match tag 666
set community 100:666
route-map RTBH permit 20The RTBH route-map (applied for outgoing announcements to the ISP peer) matches the tag value 666 on any announced route (can be any administrator-defined value), and adds the RTBH ISP community on those routes before announcement.
In a nominal state, we can see the following routing table on the PE routers :
Gateway of last resort:
B I 0.0.0.0/0 [200/0] via 100.66.1.1, Ethernet1
C 1.1.1.1/32 is directly connected, Loopback0
O 2.2.2.2/32 [110/30] via 100.66.1.1, Ethernet1
O 3.3.3.3/32 [110/30] via 100.66.1.1, Ethernet1
O 4.4.4.4/32 [110/30] via 100.66.1.1, Ethernet1
O 10.10.10.10/32 [110/20] via 100.66.1.1, Ethernet1
C 100.66.1.0/30 is directly connected, Ethernet1
B I 128.66.0.0/24 [200/0] via 100.66.1.1, Ethernet1
S 192.2.0.1/32 is directly connected, Null0
O 192.168.1.0/30 [110/30] via 100.66.1.1, Ethernet1Now let’s imagine that the IP 128.66.0.13 is victim of a DDoS attack.
The customer just needs to add a null0 route for this IP on the CE router customer-router, with the tag value “666”.
This tag will be matched by the outgoing route-map for the PE neighbor, which will add the “100:666” community value (agreed with the ISP).
On PE side, this community will be matched, and will cause the next-hop value to be replaced by 192.2.0.1, and the “no-export” community to be added to avoid announcing this specific route outside of the ISP AS :
ce(config)#ip route 128.66.0.13/32 Null0 tag 666Now let’s take a look at the PEs routing table, for example PE1 here :
Gateway of last resort:
B I 0.0.0.0/0 [200/0] via 100.66.1.1, Ethernet1
C 1.1.1.1/32 is directly connected, Loopback0
O 2.2.2.2/32 [110/30] via 100.66.1.1, Ethernet1
O 3.3.3.3/32 [110/30] via 100.66.1.1, Ethernet1
O 4.4.4.4/32 [110/30] via 100.66.1.1, Ethernet1
O 10.10.10.10/32 [110/20] via 100.66.1.1, Ethernet1
C 100.66.1.0/30 is directly connected, Ethernet1
B I 128.66.0.13/32 is directly connected, Null0
B I 128.66.0.0/24 [200/0] via 100.66.1.1, Ethernet1
S 192.2.0.1/32 is directly connected, Null0
O 192.168.1.0/30 [110/30] via 100.66.1.1, Ethernet1We can immediately see the Null0 route for 128.66.0.13/32 added to the PE routers routing table (because of the recursive route for 192.0.2.1), thus dropping all incoming traffic for the DDoS attack directly at the edge of the ISP network.
The full 128.66.0.0/24 network is still joinable from the “rest of the world”, while the 128.66.0.13 IP address does not receive any traffic as long as this black hole prefix remains announced.
Source-based Remote Triggered Black Hole
A very interesting variation of RTBH permits (in the case of a specific and identified source of attack) to blackhole the source (and not the target) of this attack. Which is of course not applicable to DDoS attacks, but can have its scope of application.
This mechanism is based on another feature, named uRPF (Unicast Reverse Path Forwarding), which is used by routers to validate the source of a given traffic : an incoming packet pass the uRPF check only if the source IP address of the packet matches a valid route in the router’s routing table, ensuring that packets are not arriving on an interface that would not be used to forward packets back to the source.
This feature can be enabled on most platforms by using the following command on the edge connections of the routers :
interface eth1
ip verify unicast sourceCombining bogus next-hop announcement with uRPF permits to create a route for the source subnet on the ISP routers, so that traffic received from this source is not passing the uRPF check.
Let’s take the following example, of the same customer using network 128.66.0.0/24.
This time, it is not experiencing a DDoS attack, but an attack sourced by an identified range, 128.70.0.0/24.
While the customer want to block this annoying traffic, the goal is to keep its full public network joinable from the rest of the world.
By configuring uRPF on the PE routers, the same logic than before can be used, this time not to black hole the destination of the traffic, but the source.
By announcing a black hole route for 128.70.0.0/24, the destination network (128.66.0.0/24) will remain joinable for legitimate traffic (from 128.72.0.0/24, for example), but all traffic sourced from 128.70.0.0/24 will be dropped at the edge of the ISP network.
Source RTBH : Example
The PE routers configuration is changed so that uRPF is performed on access interfaces.
Example on PE 1 :
interface eth2
ip verify unicast source reachable-via rxLet’s then announce the black hole route from CE :
ip route 128.70.0.0/24 null0 tag 666Taking a look again to the PE1 routing table :
Gateway of last resort:
B I 0.0.0.0/0 [200/0] via 100.66.1.1, Ethernet1
C 1.1.1.1/32 is directly connected, Loopback0
O 2.2.2.2/32 [110/30] via 100.66.1.1, Ethernet1
O 3.3.3.3/32 [110/30] via 100.66.1.1, Ethernet1
O 4.4.4.4/32 [110/30] via 100.66.1.1, Ethernet1
O 10.10.10.10/32 [110/20] via 100.66.1.1, Ethernet1
C 100.66.1.0/30 is directly connected, Ethernet1
B I 128.66.0.0/24 [200/0] via 100.66.1.1, Ethernet1
B I 128.70.0.0/24 is directly connected, Null0
B I 128.72.0.0/24 [200/0] via 100.66.1.1, Ethernet1
S 192.2.0.1/32 is directly connected, Null0
O 192.168.1.0/30 [110/30] via 100.66.1.1, Ethernet1Any packet received from Ethernet2 on PE1 from 128.70.0.0/24 will cause an uRPF failure (as the route to this network is pointing to Null0, and not to Ethernet2), which will cause the traffic to be immediately dropped.
At the same time, the customer network (128.66.0.0/24) will be perfectly joinable from any other IP range.
Of course, there is almost no use-case where an ISP will allow its customers to use source-RTBH announcing the black hole by itself. Indeed, any black hole announcement from any customer will cause the black holed range to be completely “disconnected” from the ISP network, and thus not able to reach any of the other ISP’s customers.
This kind of black holing is thus generally performed by the ISP itself, using the “Triggering-host on ISP network” scenario we discussed above.
BGP as a security helper : to the next level, with sFlow & FlowSpec
BGP Flow Specification (FlowSpec) is a powerful extension to the Border Gateway Protocol (BGP) that enables fine-grained traffic filtering and manipulation at the network layer.
Unlike traditional BGP, which focuses on exchanging routing information, FlowSpec allows network operators to specify traffic filtering rules based on various criteria such as source/destination IP addresses, protocols, port numbers, and packet attributes. Each rule gets assigned a traffic-rate, which permits to limit, or drop (if traffic-rate is set to 0) the matched traffic. Those filtering rules are then propagated by BGP using a specific SAFI (ie : FlowSpec-IPv4 is AFI 1 / SAFI 133), and enables rapid and precise mitigation of denial-of-service (DoS) attacks, traffic engineering, and traffic optimization.
On the other hand, sFlow (or sampled-Flow) is a network monitoring technology that provides real-time visibility into traffic flows within the network. sFlow-enabled devices sample packets and generate sFlow datagrams containing information about traffic volumes, flow characteristics, and network performance metrics. sFlow data can be collected and analyzed by sFlow collectors to gain insights into network behavior, identify anomalies, and troubleshoot performance issues.
Combining those technologies offers huge possibilities for network security and traffic management :
- FlowSpec rules can be dynamically updated based on insights gained from sFlow analysis. For example, anomalous traffic patterns detected by sFlow can trigger the creation or modification of FlowSpec rules to block or rate-limit suspicious traffic flows in real-time
- Automated integration between FlowSpec and sFlow allows for rapid and automated response to security incidents and performance issues. When anomalies or threats are detected by sFlow analysis, corresponding FlowSpec rules can be automatically deployed to mitigate the impact and protect the network.
Several tools are able of this integration, such as Netscout Arbor, Radware DefensePro, Cisco Stealthwatch…
For simple use-cases or labs, or just for those of us who like to tinker and play with APIs, inMon sFlow-RT (from the inventors of sFlow themselves !) is a good one.
Let’s see how it can help, back to our test use-case already used above in this article.
We added an sFlow-RT instance on the ISP network (could be added on customer side if the ISP allows for FlowSpec announcements), and added the “flow-spec IPv4” address-family on the ISP routers peerings.
(Note that of course, route-reflectors are generally used on such networks, but it does not changes the principle here).
The sFlow-RT server both receives sFlow samples analytics from the PE router, and is also peered with the PE router for flow-spec IPv4 announces (neighbouring between 172.60.1.101 – 172.60.1.200).
Note that sFlow-RT can also be peered for the IPv4 address-family, and used for RTBH (source or destination) announcements, as we used on previous sections.
Config extract (sFlow-RT IP is 172.60.1.200) :
sflow sample 500
sflow polling-interval 10
sflow destination 172.60.1.200
sflow source 172.60.1.101
sflow run
!
router bgp 65001
router-id 4.4.4.4
neighbor 172.60.1.200 remote-as 65666
neighbor 172.60.1.200 send-community extended
neighbor 192.168.1.2 remote-as 65666
neighbor 192.168.1.2 send-community extended
neighbor 10.10.10.10 remote-as 65001
neighbor 10.10.10.10 send-community extended
!
address-family flow-spec ipv4
neighbor 172.60.1.200 activate
neighbor 10.10.10.10 activate
!
address-family ipv4
neighbor 192.168.1.2 activate
neighbor 192.168.1.2 route-map RTBH out
!We can see both the BGP flow-spec ipv4 peering with the 172.60.1.200 (sFlow-RT), and the sFlow configuration sending sampled analytics to this same IP.
1 / 500 packet is sampled, with a polling-interval of 10 seconds (analytics sent to the sFlow collector at this interval). Please note that those are really low values which are not adapted for production.
Let’s add some hosts on the networks connected to the edge PE routers, and simulate a DDoS DNS amplification attack with hping.
hping3 --flood --udp -k -s 53 128.66.0.13Thanks to the sFlow analytics it receives, sFlow-RT is able to detect and classify (depending of the configured thresholds) the flow and to build a FlowSpec filtering rule accordingly.
After a few seconds (depending of the DDoS attack trigger configuration), we can see a BGP UPDATE message containing the FlowSpec NLRI information :
Looking at the PE BGP flow-spec ipv4 table, we can see the rule announced by sFlow-RT that has been implemented to mitigate the attack :
pe#show bgp flow-spec ipv4 det
BGP Flow Specification rules for VRF default
Router identifier 4.4.4.4, local AS number 65001
BGP Flow Specification Matching Rule for 128.66.0.13/32;*;IP:17;SP:53;
Rule identifier: 3880907448
Matching Rule:
Destination Prefix: 128.66.0.13/32
Source Prefix: *
IP Protocol: 17
Source Port: 53
Paths: 1 available
65666
from 192.168.1.2 (100.100.100.100)
Origin IGP, metric -, localpref 100, weight 0, valid, external, best
Actions: DropThis rule matches all UDP traffic (IP Protocol 17), from any source, destined to 128.66.0.13/32, which source-port is 53 (typical for DNS amplification attacks).
The associated action is “Drop” : the rule has been configured with a traffic-rate value of 0.
We can see that the rule is propagated to the other PE routers, and “installed” on the matching interfaces :
PE1(config)# show flow-spec ipv4
Flow specification rules for VRF default
Applied on: Ethernet2
Flow-spec rule: 128.66.0.13/32;*;IP:17;SP:53;
Rule identifier: 3880907448
Matches:
Destination Prefix: 128.66.0.13/32
Source Prefix: *
IP Protocol: 17
Source Port: 53
Actions:
Drop
Status:
Installed: yes
Counter: 31283 packets
The attack is then immediately mitigated, blocked at the edge of the ISP network, thus completely freeing the link between the ISP and the customer (as well as the ISP network itself) from this unwanted traffic.
All other traffic (not matched by the filtering rule) remains allowed, thus avoiding any denial of service of the targeted server due to attack mitigation.
By adding other sFlow receivers at the different edges of the ISP network and some automation, it is then possible to automate the removal of the flow-spec filtering rule as soon as the attack stops.
Anecdotes and past real scenarios
The different methods we discussed here are of course used everyday by Internet-peered customers as well as ISP to mitigate attacks and implement traffic engineering.
However, there are some interesting use situations where those techniques have been use (more or less efficiently) in the past :
The Pakistanese YouTube black-holing :
In 2008, Pakistan Telecom (PTCL) inadvertently caused a global Internet routing incident when they attempted to censor YouTube in their country by announcing a more specific BGP route for YouTube’s IP addresses with a next-hop of null0.
This misconfiguration caused major disruption to YouTube’s services worldwide, as the announcement propagated across the Internet, leading to traffic destined for YouTube being dropped into a black hole.
The incident highlighted the importance of proper BGP route filtering and validation to prevent unintended consequences and emphasized the potential impact of RTBH filtering on global Internet routing (remember the importance of making sure that black-hole routes are added the “no-advertise” community to avoid uncontrolled propagation).
Link
August 30th 2020 – The outage of CenturyLink / Level3 :
In August 2020, CenturyLink, one of the main telecommunications provider, experienced a widespread Internet outage caused by an erroneous FlowSpec rule deployment.
The incident occurred when CenturyLink inadvertently announced a FlowSpec rule to its BGP peers, instructing them to drop all BGP traffic (the announced FlowSpec rule was dropping TCP 179, which is the flow used to announce the rule itself…. see the issue ?).
As a result, critical BGP routing information necessary for the functioning of the Internet was disrupted, leading to widespread service outages and disruptions for numerous online services and networks.
The incident highlighted the potential risks associated with misconfigured or improperly implemented FlowSpec rules, emphasizing the importance of rigorous testing, validation, and monitoring procedures to prevent such incidents in the future.
Link
In summary
BGP offers a robust and flexible framework for mitigating DoS attacks, providing network administrators with powerful tools to defend against malicious actors and safeguard critical infrastructure.
By leveraging techniques such as RTBH, Source RTBH, and BGP FlowSpec, organizations can enhance their resilience against a wide range of DoS attack vectors while minimizing disruption to legitimate traffic.
Finally, find below a sum-up of the different techniques we discussed on this article and their specificities :
| Technique | RTBH (Remote Triggered Black Hole) | Source-Based RTBH | FlowSpec with sFlow |
|---|---|---|---|
| Overview | Mitigates DDoS attacks by blackholing traffic to specific destinations | Mitigates DDoS attacks by blackholing traffic from specific sources | Dynamically filters and manipulates traffic based on specified criteria using BGP with a specific address-family for filtering rules propagation |
| Triggering Mechanism | Customer-initiated or automated detection triggers announcement of black hole routes | Customer-initiated or automated detection triggers announcement of black hole routes with source IP addresses | Analysis of sFlow data triggers creation or modification of FlowSpec rules (FlowSpec rule can also manually created) |
| Flexibility | Less granular, typically used for broad traffic black-holing | More granular, allows for targeted black-holing of traffic from specific sources | Highly granular, enables fine-tuned traffic filtering and manipulation based on various criteria |
| Use-Cases | Mitigating volumetric DDoS attacks targeting specific destinations | Mitigating DoS attacks originating from specific sources or subnets | Dynamic traffic filtering, DDoS mitigation, traffic engineering, and policy enforcement |
| Deployment | Typically deployed at the network edge or border routers | Deployed at the network edge or border routers, often in conjunction with RTBH | Implemented within BGP-enabled routers and sFlow-enabled devices throughout the network |