Peering LANs 2.0 – EVPN rollout on the DE-CIX Apollon platform

Q&A With Dr Thomas King, CTO at DE-CIX 

We are currently working on implementing EVPN at all our locations where we run the DE-CIX Apollon platform (all locations except India). This upgrade will make our peering LANs even more robust and safer for our customers. 

EVPN stands for “Ethernet Virtual Private Network” and using it in large networks has many benefits. We enhanced the EVPN standard together with Nokia and other partners to implement features for the use-case of Internet Exchanges. These improvements are standardized in the RFC 9161 titled “Operational Aspects of Proxy ARP/ND in Ethernet Virtual Private Networks”. DE-CIX is the first Internet Exchange operator to implement this enhanced version of EVPN across all locations, increasing the resilience of connections for our customers. Dr. Thomas King, CTO at DE-CIX, explains what the benefits are and why we are implementing this change.

What is EVPN?

EVPN, or an “Ethernet Virtual Private Network”, will help to eliminate unnecessary network “noise” on peering LANs at Internet Exchanges. This network noise is generated by the Address Resolution Protocol (ARP) in IPv4 and the Neighbor Discovery Protocol (NDP) in IPv6, resulting in lot of requests being flooded to all customer routers within the peering LAN. The bigger the peering LAN, the more flooding occurs. As a result, EVPN is especially useful for suppressing ARP and NDP network “noise” in large peering LANs at Internet Exchanges such as DE-CIX Frankfurt, DE-CIX New York, and DE-CIX Madrid.

What are the issues with current peering LANs?

When a router wants to talk to another router, it sends out an ARP or NDP request for determining the MAC address of the other router. However, the ARP or NDP request is flooded to all the routers that are part of the peering LAN. Heavy loads of this traffic can consume a lot of resources (e.g. CPU and RAM) on some routers, requiring additional configuration changes to mitigate the issue or causing operational issues such as triggering alarms. The bigger the peering LAN, the louder the network noise of ARP and NDP traffic, and in Frankfurt for example, some smaller out-of-the-box routers are unable to keep the BGP session running as a result.

How will it work on the DE-CIX platform?

Instead of flooding the peering LAN with ARP and NDP requests and each customer router having to process them, with the enhanced EVPN version we are implementing, all requests are snooped by the DE-CIX routers. The DE-CIX routers either respond to the requests or discard them, therefore significantly suppressing the level of ARP and NDP requests each customer router needs to process.

What are the benefits of EVPN?

A router answering ARP and NDP requests for addresses that do not belong to its own interfaces can interfere with and disturb the traffic flow between the intended neighboring routers. For instance, a proxy server running on a customer router in a peering LAN – also known as Proxy ARP – can offer its own MAC address as the destination, routing the traffic away from the intended destination. Despite permanent monitoring, we can only act reactively, after this has already happened on our peering LANs. EVPN mitigates this attack vector completely, making connections more resilient. Another benefit for customers is that we can provision new peering services faster. Currently, when you order a new peering service, we will put it first on “quarantine” to test it and make sure everything works as intended (e.g., Proxy ARP is not activated on the customer router). After the EVPN implementation on the peering LANs, we will be able to immediately provision new peering services without this delay, as EVPN eliminates critical attack vectors as described above.

So, Peering LAN 2.0 is just EVPN?

We are not simply implementing EVPN on the peering LANs. We are also changing how the underlying MPLS network functions. For this, we will implement RSVP-TE, which allows us to introduce advanced traffic engineering features. These traffic engineering features are especially beneficial for products like GlobePEER Remote, DirectCLOUD, and the Microsoft Azure Peering Service, as they will enable us to steer in a fine-grained manner traffic flows between two connected DE-CIX locations (e.g., between DE-CIX Frankfurt and DE-CIX New York). So, Peering LAN 2.0 is the combination of EVPN and RSVP-TE.

How is the Peering LAN 2.0 project being rolled out? When will it be done?

We will implement the changes progressively, in one location at a time. We will start with the smaller Internet Exchanges in October, and then move on to larger and more complex locations with many routers. We expect to finish the migration of all locations by the end of the year at the latest. Announcements about the upcoming migration work will be shared with our customers ahead of time.

Will there be any impact on our customers?

The implementation is basically a software configuration on our routers. The implementation is being executed during maintenance windows, so there are no to-dos for our customers. If the migration goes smoothly, customers will only experience a minimal downtime – a few seconds – during the migration. And as mentioned, we will inform customers about the migration in advance.