BIRD drops specific IPv6 session for no reason

[Stavros Konstantaras]

Hi Stefan (and the rest of the community),

To give you an update, two days before we upgraded one of our Route servers with Kernel version 4.9 and at the same time we upgraded Bird to version 1.6.8

Situation became worse, the customer who reported the bug, 6 hours after the maintenance he complained for severe flapping. After checking the logs, we could clearly see that his IPv6 BGP session was flapping 3-4 times per hour. Always the same “Hold timer expired” reason. 


We tried:
 - sysctl -w net.ipv6.route.max_size=65536

Apparently didn’t work much and the flapping continued. Then we tried:
 - ethtool -K eth1 tx-gso-partial off. (per Stefan’s recommendation)


That seemed to bring some stability for few hours. But during night it started crashing again and we also observed some flapping during the morning. 

Now we applied the "net.ipv6.route.max_size = 2147483647” in an attempt to make it more stable, after digging around the net and discovering the following thread:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861115 <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861115>


But this is really weird and not much in the logs to figure out the root cause. But thank’s for the tip and sorry for the late reply.



Best regards,

Stavros Konstantaras | Sr. Network Engineer | AMS-IX 
M +31 (0) 620 89 51 04 | T +31 20 305 8999
ams-ix.net


> On 13 Mar 2020, at 09:28, Stefan Jakob <tinysammy at gmail.com> wrote:
> 
> Hi Stavros,
> 
> Try to disable all those low level NIC driver thingies like TCP Segment Offloading (TSO).
> 
> F.e.
> 
> ethtool eno1 -K tso off
> 
> gso off
> 
> 
> Rgds, SJ
> 
> Stavros Konstantaras <stavros.konstantaras at ams-ix.net> schrieb am Fr., 28. Feb. 2020, 12:43:
> Hi Bird community,
> 
> We are investigating a weird customer issue regarding our Bird Route Servers (version 1.6.3) and a specific IPv6 session. Customer reports a sudden drop of his IPv6 session and -until now- we could not relate those drops with any issue or instability. Everything seems normal and no other customer complained at the moment of the incident. 
> 
> 
> 
> After some packet capturing at the moment of the event, we discovered that BIRD does not send a response messages to the customer’s BGP keepalive messages (see attached picture), which result to the BGP hold timer to expire and the sessions to be dropped. We observed this anomaly with both RSs but at different time slots and the tcpdump capture was running at the Interface were Bird is sending all BGP traffic for customers. At the moment of the event, we didn’t do any maintenance or other RS related work.
> 
> Has any of you experienced this in the past? If yes, how did you solve this?
> Any related feedback is welcomed. 
> 
> 
> 
> 
> 
> 
> 
> 
> Best regards,
> 
> Stavros Konstantaras | Sr. Network Engineer | AMS-IX 
> M +31 (0) 620 89 51 04 | T +31 20 305 8999
> ams-ix.net
> 
> <image-2020-02-21-10-13-53-698.png><image-2020-02-21-10-13-53-698.png>





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20200319/c06890b7/attachment.htm>