netlink filtering to avoid clostly FNHE table dumps on Linux

[Tomas Hlavacek]

Hi!

This is not a direct follow-up on the thread "BIRD trying to reinsert
existing kernel routes, netlink issue?" (
https://www.mail-archive.com/bird-users@network.cz/msg05429.html ) but
I think that the cause of my issue can be the same.

My scenario: I have a server in LAB that has a public IP on a GRE
tunnel interface and the tunnel goes to a router in the Internet. BIRD
receives full BGP feed over the GRE tunnel. The tunnel is terminated
in a VRF and BIRD exports the BGP feed to the kernel RT associated
with the VRF. I observed pretty much the same set of issues as Sasha
and others described in the linked thread two years ago: BIRD starts
normally, BGP session comes up but after 3 or 5 minutes it max-out one
CPU, mostly in sys (doing the netlink I/O). Then BIRD stops responding
normally over birdc - each command takes several minutes to complete
and I also saw the log messages like:

Jan  2 00:00:35 lab bird: Netlink: File exists
Jan  2 00:00:35 lab bird: Netlink: File exists
Jan  2 00:00:35 lab bird: ...
Jan  2 00:00:35 lab bird: I/O loop cycle took 73127 ms for 11 events

I realized that BIRD actually transfers huge amounts of data from
kernel over netlink (several gigabytes of messages per minute or so,
which translates to hundreds of millions of FIB records) in the
beginning of each sync and it ultimately stalls BIRD I/O long enough
to miss BGP keepalive deadlines and the sessions start flapping, which
makes the situation hard to understand using the conventional
profiling tools, tcpdump for netlink dump etc. FYI I am attaching my
quick and dirty patch to BIRD that I used to collect stats from
netlink interactions to understand the problem and finally to add
experimental PoC fix - see  nl_request_rt_dump() in the patch.

The large table that BIRD pulled from the kernel was a FNHE table
where Linux collects PMTU records for *all* destination IPs that are
routed to the tunnel (which does not seem to be right and I will
discuss it in LKML shortly). These records have (default) 600s
expiration time and in my scenario I happen to receive some
backscatter traffic that in most cases gets ICMP or TCP reset
responses that could ultimately create millions of these records in a
few minutes.

The reason why this problem occured only in Linux ~5.2+ lies in the
patch  https://patchwork.ozlabs.org/project/netdev/patch/8d3b68cd37fb5fddc470904cdd6793fcf480c6c1.1561131177.git.sbrivio@redhat.com/
that changed the semantics of netlink dump requests. Now the kernel
dumps the FIB Next Hop Exceptions table (previously known as route
cache) alongside the RT unless the requester sets sockopt
NETLINK_GET_STRICT_CHK and clear the flag RTM_F_CLONED in the dump
request. BIRD does not apply the filters so the kernel dumps
everything. But iproute2 and other programs that use netlink utilize
the filters, so no similar performance issue occurs unless I
explicitly dump the FNHE table (ip route show cache).

I believe that many different types of Linux tunnels create the PMTU
records for all packets transmitted over the tunnel as well. And it
works like that for a long time - the code that creates the route
cache (at that time, now it is FNHE table) records has been introduced
in Linux 3.10 (https://elixir.bootlin.com/linux/v3.10/source/net/ipv4/ip_tunnel.c#L591).

Regardless of what may or may not happen on the kernel side I think
that implementing the netlink filter in BIRD to avoid the described
situation makes sense. I am almost certain that my experimental fix
breaks other things (most likely OSPF) but I would be glad to help
make it right.

What do you think?

Best regards,
Tomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-EXPERIMENT-Add-stats-print-and-strict-dump-over-NL.patch
Type: application/x-patch
Size: 9133 bytes
Desc: not available
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20220108/00d8ac9d/attachment.bin>