Best practices for redundant iBGP/eBGP route distribution? [bird 2.0.7]

[Ondrej Zajicek]

On Mon, Dec 16, 2019 at 03:18:33PM +0100, Nico Schottelius wrote:
> >> Question 1: Is "direct;" is the right protocol for all links?
> >>
> >> As all links are layer 2 connections, we have configured all links to be
> >> direct. However this causes the "Invalid NEXT_HOP attribute" in various
> >> situations.
> >
> > Generally, BGP assumes you already have all internal routes - either from
> > OSPF, or equivalent static/direct routes. These are necessary for recursive
> > next hop resolution, but can be avoided in direct next hop resolution.
> 
> To put a bit more detail into this: we have a transfer/peering vlan that
> utilises 2a0a:e5c0:1:8::/64. So every
> router/switch/whatever-needs-to-speak-bgp is in this network.

Both IBGP and EBGP? Or do you have your upstream links in separate networks?

> Because every speaker is in this network, my assumption is that it
> should be possible to (recursive) resolve every route. Am I wrong on this?

Yes, if you have direct route 2a0a:e5c0:1:8::/64 in BIRD routing table
(e.g. from direct protocol).


> > If you do not use IGP, then direct IBGP sessions may have some advantages,
> > namely they can react to link-down events.
> >
> > Note that direct/multihop session and direct/recursive gateway resolution
> > are two options, although the second by default depends on the first. So
> > it is e.g. possible to configure direct session with recursive resolution.
> 
> Can you elaborate a bit more on the effects?
> 
> i.e. if looking only at the effects of the first setting
> (direct/multihop) without the adjusted defaults or only at the second
> setting (direct/recursive), what are the exact outcome of the 4 combinations
> 
> direct/gateway direct
> direct/gateway recursive
> multihop/gateway direct
> multihop/gateway recursive

direct / multihop specifies connection / TCP behavior for BGP session. In
first case connection is tightly bound to interface and is single-hop,
while in the multihop case it is regular routed TCP connection. Does not
have direct effect on next hop resolution.

gateway option specifies behavior of next hop resolution.


> >> Question 2: Is "next hop self ebgp;" the correct answer to the Invalid
> >> NEXT_HOP attribute?
> >
> > If you use direct next hop resolution, then received NEXT_HOP is supposed
> > to be directly reachable, ideally on the same interface. So in this case
> > you should use 'next hop self;'.
> 
> I am puzzled by this statement a bit, because of the following
> documentation on the bird website:
> 
> "
> gateway direct|recursive
> 
> For received routes, their gw (immediate next hop) attribute is computed
> from received bgp_next_hop attribute. This option specifies how it is
> computed. Direct mode means that the IP address from bgp_next_hop is
> used if it is directly reachable, otherwise the neighbor IP address is
> used.
> "
> 
> So if I read it here correct, in the direct/direct configuration, the
> gateway should be the peer's IPv6 address, if the next hop address is
> not directly reachable. However this is *not* what I experience in the
> bird 2.0.7 setup. However this *is* what I experience in the bird 1.6
> setup.

This seeems like a bug in documentation (left from BIRD 1).
The "otherwise the neighbor IP address is used." shold not be here.


> There is another problem which renders "next hop self" tricky: as all
> routers are paired with every other router, we are hiding the origin,
> i.e. we announce networks from the dc1 in dc2 and this results in
> routing from dc1->dc2->dc1.

No, because IBGP does not propagate routes received from IBGP.


> > Variant 'next hop self ebgp;' is more for cases where you use recursive
> > resolution, but your IGP/internal routes do not cover border/inter-AS
> > links.
> 
> That sounds a bit similar to the actual situation we have.

The difference probably does not matter in your case. Depends on which
routes do you propagate using IBGP.

If you propagate static route 2a0a:e5c1:111::/48 with next hop on different
network through IBGP with 'next hop self ebgp' and direct mode, then BGP
NEXT_HOP would be original next hop and unresolvable on the other side.


> >> Question 3: Is "not direct" (aka multiphop) the right thing for iBGP?
> >
> > As i wrote above, you would need static/direct/IGP routes for internal
> > networks (e.g. from 'direct' protocol).
> >> So our dcs are directly connected vi layer 2, but the default for iBGP
> >> is multihop. If we omit the "direct" keyword, the result is that no
> >> routes are in the end imported from the other DC and that we get various
> >> warnings like the following in syslog:
> >>
> >> Dec 16 00:58:35 router2 daemon.warn bird: Next hop address 2a0a:e5c0:1:8::5 resolvable through recursive route for 2a0a:e5c0:1:8::/64
> >
> > You probably do not want to export your internal routes
> > (2a0a:e5c0:1:8::/64) through IBGP.
> 
> This is actually directly the result from the "direct" protocol that we
> tried to implement as you described :-)

You should configure IBGP to not propagate routes from 'direct' protocol.


> I just reconfigured the test routers to use the combination
> direct/gateway recursive and it seems to mostly work, however there is
> at least one route that it does not resolve: a route that the router in
> dc2 routes to another (non-bgp) router.
> ...
> It is shown as unreachable in dc1, while it clearly can be reached and
> routed in dc2.
> ...
> 
> Shouldn't this resolve via direct/gateway recursive mode?

You would need static/IGP route for 2a0a:e5c0:2:2::/64 in DC1 router
directed to 2a0a:e5c0:1:8::5.

Perhaps you have one received from IBGP, but that is not allowed for
recursive route resolution. BIRD allows just one level of indirection
in recursive mode (so it should not be really called 'recursive').

So in short, BIRD assumes that full set of routes for all your local
networks from IGP/static/direct are available on all your BGP routers
as a base level and on top of that recursive next hops from IBGP are
resolved. Or alternatively,  'gateway direct' and 'next hop self' are
used to avoid recursive next hop resolution.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santiago at crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."