Stateless DDoS protection via Vector Packet Processing

Criminal groups target Internet Service Providers with Distributed Denial of Service attacks. They often demand ransom to stop the attack and threaten to launch a more devastating one if the ransom is not paid.

Traditional methods of combatting such attacks rely on stateful firewalls, which are prone to state table exhaustion, thereby letting malicious traffic through or causing a DoS themselves. This blog post goes over building your own stateless Scrubbing Center, which can withstand various types of attacks preventing a denial of service for customers.

A Scrubbing center within of AS cannot solve link saturation that is happening outside of it. If the purchased upstream capacity is lower than the attack launched by the adversary, the upstream provider will drop traffic before it can ever be filtered, resulting in a DoS.

Vector Packet Processing

VPP is a Linux foundation backed project, part of the FD.io (Fast Data) project, providing the functionality of network switches and routers. The design of VPP is hardware and deployment (bare metal, VM, container) agnostic as it runs completely in user space, providing a kernel bypass.

Practically, VPP takes a whole vector of packets and pushes it through a graph node, before pushing it onto another node. Remaining packet vectors get processed even faster, as the instruction cache learns from the first vector packet. This is opposed to scalar processing, which is the standard method of packet processing in Linux, where each of these packets are pushed through nodes individually.

Without VPP, the process of receiving packets is as follows: NIC receives a packet -> calls an interrupt -> CPU stops work -> switches to kernel space -> kernel copies the packet over -> switches to user space. This context switching consumes a massive amount of CPU cycles. With VPP: NIC receives a packet -> Direct Memory Access is used to write packets to user space RAM. Instead of having to wait for interrupts, it can process packets immediately.

What are we protecting against

The efficiency of a DDoS attack lies in amplification. The bigger the ratio between sent and received packet sizes, the less bandwidth the attacker needs to cause damage. This is often done through reflection and use of protocols which produce much larger packet replies than what the original incoming packet was.

CLDAP requests can reach 70x amplification. This means that to generate a 1.4 Tbps attack, the attacker only needs 20 Gbps of traffic. However, CLDAP traffic can be cut off without causing DoS for many customers, so although it provides astronomical amplifications, in reality it does not cause much trouble, at least not in comparison with more standard protocols.

TCP was designed to operate on unreliable networks, meaning a single SYN request can trigger several SYN-ACKs in rapid succession. There is not much amplification to be gained from singular TCP packets (up to 10%). However, attackers can achieve greater amplification by targeting IP space that is allocated, but is unoccupied. This prevents the source of the SYN-ACK from receiving a RST packet. The source will in turn send up to seven SYN-ACK packets before halting. That means that, in the worst case, a 40 byte request generated 308 bytes of packets arriving towards the target's infrastructure. That's an 7.7x amplification.

Scrubbing Center

Network traffic scrubbing is one of the DDoS mitigation techniques. It consists of passing traffic through high capacity networks with "traffic scrubbing" filters.

A Scrubbing center contains one or more scrubbing servers, which are dedicated machines that receives network traffic and attempts to filter good traffic from bad. Ideally, the scrubbing server will only forward non-DDoS packets.

It can either be constructed locally, which has benefits of having almost no latency, or a cloud-based solution can be utilized. In any case, it is an off-ramp solution which is inactive if there is no attack and routes through it only activate when needed.

Building an effective Scrubbing Center requires a ton of knowledge of the underlying protocols. It also requires hardware capable of digesting traffic volume of at least what the upstream provider grants, which can be hundred of Gigabits per second. If one does not have any DDoS protection whatsoever, they should weight in the possibility of opting for a cloud-based solution before they can develop their own local one.

Topology of the demo

The demo consists of three interconnected debian servers with public IPv4 addresses.

There is a web server running on rtr-3 with the private ip of 10.100.100.1 accessible from rtr-1 over a GRE tunnel going through the internet. Traffic is symmetrical without the use of the scrubbing center. However, in a filtering mode with the Scrubbing Center active, traffic to rtr-3 goes through rtr-2 where it gets processed and healthy bits are passed to rtr-3 which replies through the tunnel, creating asymmetry.

Demonstration

Let's setup our own scrubbing center now. If you would like to follow along in your own lab, make sure your topology resembles the one mentioned above. I will assume all the interfaces are configured and that rtr-1 can access a webserver running on rtr-3 with a single hop and that there is a way to get to rtr-3 through rtr-2.

Prepare BGP for diversion of traffic

First, we need a mechanism that dynamically pulls traffic away from the main link and into our Scrubbing Center during an attack. We will use the BIRD Internet Routing Daemon for this. Let's install it on both rtr-1 and rtr-2:

apt install bird2

Configuring rtr-1 (ISP Router)

This router receives routes from our Scrubbing Center. I only have one in this topology, but you can repeat the following configuration for as many as you'd like to simulate it better. Assign bigger metric to the routes in the kernel. This is so BGP gets priority when activated. 10.0.12.1 is the local IP address on an interface connecting this node to the Scrubbing Center and 10.0.12.2 is the IP address of the interface on the Scrubbing Center the node is connected with.
/etc/bird/bird.conf on rtr-1:

log syslog all;

router id 10.0.12.1;

protocol device { scan time 10; }
  
# Kernel route has a metric higher than 10
protocol kernel {
    ipv4 { import all; export all; };
    learn;
    metric 10;
}

protocol bgp scrubber {
    local 10.0.12.1 as 65001;
    neighbor 10.0.12.2 as 65002;
    ipv4 { 
        import all;
        export none;
    };
}

Configuring rtr-2 (Scrubbing Center)

Here we define the static route 10.100.100.0/24 via 10.99.99.2 that will be advertised to the rest of the network during an attack. The IP address 10.99.99.2 acts as the gateway into VPP, which we will configure next.
/etc/bird/bird.conf on rtr-2:

log syslog all;

router id 10.0.12.2;

protocol device { scan time 10; }
protocol kernel { ipv4 { import all; export all; }; }

protocol static hijack {
    ipv4;
    route 10.100.100.0/24 via 10.99.99.2;
}

protocol bgp isp {
    local 10.0.12.2 as 65002;
    neighbor 10.0.12.1 as 65001;
    ipv4 {
        import none;
# none - diversion inactive
# all - diversion active
        export all; 
    };
}

Setting up VPP

For the purpose of this demo, we will activate the scrubbing center manually and that by changing "none" to "all" in rtr-2 BIRD config and watch the route change propagate to our ISP nodes. Notice the address 10.99.99.2, it will be our scrubbing center running on rtr-2, to perform a kernel bypass and avoid having to work with scalar packets we must avoid any bridges made by Linux interfaces, but for the purpose of this simplified demo we will set it up that way.

apt install vpp vpp-plugin-core vpp-plugin-dpdk

VPP operates completely isolated bypassing the Linux kernel. To get data into it and back out, we must create dedicated interfaces. First, we hijack the physical network card that connects to the victim on rtr-3 (in my case it is ens20) and assign it an IP address that resides in VPP:

vppctl create host-interface name ens20
vppctl set interface ip address host-ens20 10.0.23.1/30
vppctl set interface state host-ens20 up

Next, we need to build a bridge between Linux (where BGP sends the diverted traffic) and VPP. We create a virtual tap0 interface inside of VPP with the IP 10.99.99.2:

vppctl create tap id 0 host-if-name vpp-scrub
vppctl set interface ip address tap0 10.99.99.2/30
vppctl set interface state tap0 up

On the Linux side, we bring this bridge interface up and assign it an IP address in the same subnet:

ip link set vpp-scrub up
ip addr add 10.99.99.1/30 dev vpp-scrub

The rules are now done, anything the Linux kernel routes to 10.99.99.2 will drop into VPP for filtering.

Routing clean traffic

VPP ignores Linux routing tables, so we need to manually tell it how to forward the cleaned traffic to the victim. We add a static route pointing out through the host-ens20 interface:

vppctl ip route add 10.100.100.1/32 via 10.0.23.2 host-ens20

So far we set up a network topology which can divert traffic through a Scrubbing Center (rtr-2) if needed. We have not yet added any rules into our Scrubbing Center, meaning the scrubbing does not scrub anything out of the network traffic. Let's do that now.

Simulating a single IP attack

hping3 -S -A -p 80 --flood 10.100.100.1

You should see a spike in CPU usage on rtr-3 as an observation of the attack going through. If you then activate the following rule on rtr-2 to statelessly drop SYN-ACK packets (TCP flags 18):

vppctl set acl-plugin acl "deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 6 tcpflags 18 mask 18, permit src 0.0.0.0/0 dst 0.0.0.0/0"
vppctl set acl-plugin interface tap0 input acl 0

Where the zero at the end is the index of the ACL. You can see your ACL rules with vppctl show acl-plugin acl

vppctl show int

You should see tap0 rx packets going sharply up, while host-ens20 stays unchanged. If you then start a regular ping from rtr-1 targeting rtr-3, while keeping the flood going, you should see tx packets under host-ens20 going up. If you run vppctl show errors you will see exactly why the packets are being dropped.

If the attack still goes through and packets are not being dropped, make sure there's export all in BIRD configuration on rtr-2

Simulating an IP range attack

Targeting a single IP quickly triggers standard rate-limiting defenses and attackers know it. To bypass this, they spread the attack across the entire /24 subnet. They will target IP space that is allocated, but is unoccupied (Dark space). Stateful firewall would try to open a connection state for every packet hitting these random IPs, quickly running out of RAM.

hping3 -S -A -p 80 --flood 10.100.100.55

VPP is completely stateless, it does not track connections or care if the destination IP exists. It matches the header against the ACL and drops it. After running vppctl show errors, you can see ACL deny packets counter going up on rtr-2. All the flood packets are instantly dropped.

Conclusion

This single ACL rule blocks only a very specific kind of traffic and it is most likely not enough to prevent a sophisticated DDoS attack with attackers throwing random protocols around. You can dig into VPP's documentation and add more rules and filters, but it does not change the fact that setting up your own Scrubbing Center requires a lot of proactiveness to keep up with the variety of attack vectors, but even if you opt for a cloud-based protection, it never hurts to understand how one is build in practice.