1 out of 2 BGP sessions down, all routes become unreachable

[Yannis Nikolopoulos]

Hello again,

On 4/10/21 10:00 PM, Alexander Zubkov wrote:
> Hi,
>
> Please show also:
> show interface
lo up (index=1)
     MultiAccess AdminUp LinkUp Loopback Ignored MTU=65536
     127.0.0.1/8 (Preferred, scope host)
     a.b.c.1/32 (scope univ)
     a.b.c.2/32 (scope univ)
     2a02:x:2::53/128 (Preferred, scope univ)
     2a02:x:1:2::53/128 (scope univ)
     ::1/128 (scope host)
eth0 up (index=2)
     MultiAccess Broadcast Multicast AdminUp LinkUp MTU=8950
     192.168.0.6/24 (Preferred, scope site)
     fe80::x:x:x:x/64 (Preferred, scope link)
eth1 up (index=3)
     MultiAccess Broadcast Multicast AdminUp LinkUp MTU=9000
     a.b.71.70/27 (Preferred, scope univ)
     2a02:x:y:33::70/64 (Preferred, scope univ)
     fe80::y:y:y:y/64 (Preferred, scope link)
eth2 up (index=4)
     MultiAccess Broadcast Multicast AdminUp LinkUp MTU=9000
     10.20.18.69/28 (Preferred, scope site)
     2a02:x:x:53::69/64 (Preferred, scope univ)
     fe80::z:z:z:z/64 (Preferred, scope link)



> show route all

unfortunately, I didn't take this output during the timeframe the 
problem occured

>
> So when you have both bgp_v6_a and bgp_v6_b established - you have
> normal routes. And when you have only bgp_v6_a established - you have
> that unreachable route only?

correct on both counts

> Your bgp sessions are internal (iBGP), so they may probably have
> bgp_nex_hop different from the router's address.

correct

> And also because of
> recursive resolution they may depend on other routes you have in your
> bird.

that, I don't follow

regards,
Yannis
>
> On Sat, Apr 10, 2021 at 8:02 PM Dez C <dez at otenet.gr> wrote:
>> Hi,
>>
>> all external routes were unreachable (apart from the local loopback
>> which was imported via direct protocol). master6 consisted solely of
>> entries like:
>>
>> 2a02:xxxx:d00d::/45 unreachable [bgp_v6_a 19:38:23.222 from 2a02:xxxx:yyyy:53::a] * (100) [ASxxxxx1i]
>>
>> No entries whatsoever from the other bgp session (bgp_v6_b), like in
>> normal operation:
>>
>> 2a02:xxxx:d00d::/45   unicast [bgp_v6_a 2021-04-07] * (100) [ASxxxxx1i]
>>           via 2a02:xxxx:yyyy:53::a on eth2
>>                        unicast [bgp_v6_b 2021-04-07] (100) [ASxxxxx1i]
>>           via 2a02:xxxx:yyyy:53::b on eth2
>>
>>
>>
>> regards,
>> Yannis
>>
>>
>> On 4/9/21 11:22 PM, Alexander Zubkov wrote:
>>> Hi,
>>>
>>> And what do you see in your bird's console? "show route" etc.
>>>
>>> On Fri, Apr 9, 2021 at 7:10 PM Dez C <dez at otenet.gr> wrote:
>>>> Hello,
>>>>
>>>> First message here, although a long-time bird user (not an expert in any
>>>> way). We're using bird as a bgp speaker for all our VMs that either need
>>>> to announce routes or import routes from VRFs etc.
>>>>
>>>> The setup is pretty straight forward I'd guess. The VM has 2 IPv6 BGP
>>>> sessions with 2 L3 switches (one per switch). The VM announces one /128
>>>> route and imports all routes learned from the BGP sessions. The same
>>>> routes are learned from both sessions, so whenever I do a 'ip -6 r' I
>>>> see 2 entries for each route (with different next hops) like:
>>>> 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::a dev eth2 proto bird metric
>>>> 32 pref medium
>>>> 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::b dev eth2 proto bird metric
>>>> 32 pref medium
>>>>
>>>> At some point, both BGP sessions went down and one of them did not
>>>> recover for an hour. Normally, this should not be a problem but in this
>>>> specific case, I could no longer see 2 routes in the kernel table, one
>>>> reachable and another unreachable, just an unreachable one, so even
>>>> though the anycast route was being announced and traffic was reaching
>>>> the VM, it couldn't send the packets to their destination as all routes
>>>> were unreachable in the kernel table:
>>>> unreachable 2a02:xxxx:d00d::/45 dev lo proto bird metric 32 error -113
>>>> pref medium .
>>>>
>>>> I'm running 2.0.7 (upgrading soon) on centos7 and here's the relevant
>>>> part of my config:
>>>>
>>>> protocol device {
>>>> }
>>>> protocol direct direct_v6_lo {
>>>>            interface "lo*";  # announce loopback
>>>>            ipv6;
>>>> }
>>>> protocol direct direct_v6_eth {
>>>>            interface "eth*";
>>>>            ipv6;
>>>> }
>>>> protocol bgp bgp_v6_a {
>>>>            local 2a02:xxxx:yyyy:53::37 as 66666
>>>>            neighbor 2a02:xxxx:yyyy:53::a as 66666;  # switch A
>>>>            ipv6 {
>>>>                   export where proto = "direct_v6_lo";
>>>>            };
>>>> }
>>>> protocol bgp bgp_v6_b {
>>>>            local 2a02:1388:4000:53::37 as 66666;
>>>>            neighbor 2a02:1388:4000:53::b as 66666;  # switch B
>>>>            ipv6 {
>>>>                    export where proto = "direct_v6_lo";
>>>>            };
>>>> }
>>>> protocol kernel {
>>>>        ipv6 {
>>>>            export filter {
>>>>                if proto = "direct_v6_eth" then reject;
>>>>                accept;
>>>>            };
>>>>        };
>>>>       merge paths limit 2;
>>>> }
>>>>
>>>> any ideas?
>>>> thanks in advance and regards,
>>>> Yannis