Best practices for redundant iBGP/eBGP route distribution? [bird 2.0.7]

[Nico Schottelius]

Good morning,

I was wondering, what is the best practice is for running 2 data
centers, each with one uplink and a dark fiber in between them?

I think it's pretty much a standard situation, but will show our setup
and assumptions as well:

upstream 1        upstream 2
 |                |
dc1--darkfiber----dc 2

Additional each DC has 2 routers:

dc1:            dc2:
|---router1 ------- router 1
|              /
|             /
|   /--------/
|  router2---------router2
|                     |
|---------------------|

(both routers are connected to each other and both routers are connected
to each upstream)

The objective is to stay online in each DC as long as possible, so in
theory:

- upstream 1 can die or
- upstream 2 can die or
- the darkfiber can die

And in each of the situations, both DCs should still be reachable from
outside and both be reaching itself.

Question 1: Is "direct;" is the right protocol for all links?

As all links are layer 2 connections, we have configured all links to be
direct. However this causes the "Invalid NEXT_HOP attribute" in various
situations.

Question 2: Is "next hop self ebgp;" the correct answer to the Invalid
NEXT_HOP attribute?

This way we rewrote routes from upstreams and generally speaking it
seems to work. However, this still fails iBGP routes:

router1.dc1 cannot reach a network in dc2 that is configured to be
statically routed to another host in dc2. My assumption is that this is
the problem of the "gateway direct" resolution, even though the
documentation kind of indicates that this should be resolved via the
next hop of the dc2.

Originally I'd have thought that not even "next hop self ebgp" is
required, as the next hop attribute is coming from the peering router,
so the route should be directly reachable.

Question 3: Is "not direct" (aka multiphop) the right thing for iBGP?

So our dcs are directly connected vi layer 2, but the default for iBGP
is multihop. If we omit the "direct" keyword, the result is that no
routes are in the end imported from the other DC and that we get various
warnings like the following in syslog:

Dec 16 00:58:35 router2 daemon.warn bird: Next hop address 2a0a:e5c0:1:8::5 resolvable through recursive route for 2a0a:e5c0:1:8::/64


So at this stage it's a bit unclear to me, how proper iBGP
redistribution / next hop resolution should work with bird.

Question 4: How to (not) announce umbrella networks?

So both data centers are reachable within the 2a0a:e5c0::/29
prefix and all routers have a statement

        route 2a0a:e5c0::/29 unreachable;

in the static protocol. However if the dark fiber goes down, it is not
clear that / how the upstreams should / will propagate the smaller (per
DC) /48s.

If we remove the full /29 announcement, we might run into the problem
that a packet is being sent out through one upstream and returns back to
us and thus creates a routing loop.

While the last question is eBGP specific,
the main question of this thread still is: how do you correctly
distribute internal and external routes via iBGP in your DCs?

Thanks for any hints in the right direction.

Best regards,

Nico

--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch