Improving Anycast routing with Bird

Kyle Drake kyle at kyledrake.net
Thu Sep 22 06:44:08 CEST 2016


Thanks for the detailed information, this is very informative. I realize
now that this question was probably off topic for the mailing list and I
apologize to everyone for that.

After talking with someone about it, I realized that the problem was that
my BGP provider (http://bgp.he.net/AS20473) was indeed direct peering with
Comcast in a few datacenters but not all their datacenters, causing a heavy
traffic bias to a few locations. I used one of their BGP communities to AS
prepend Comcast to discourage direct peering and I'll see if that helps
spread the traffic more evenly (my guess is that it will, the IP transit
routes tend to be doing much better).

Thanks again!

This is how my bird.conf ended up looking:

log syslog all;

router id **MY_IP**;

protocol bgp vultr {
  local as 395409;
  source address **MY_IP**;
  import none;
  export all;
  graceful restart on;
  multihop 2;
  neighbor **BGP_IP** as 64515;
  password "**OBSCURED**";
  export filter {
    bgp_community = -empty-;
    bgp_community.add((64603,7922)); # 64603 = Prepend 3x to Comcast ASN
(7922)
    accept;
  };
}

protocol static {
  route 198.51.233.0/24 via **MY_IP**;
}

protocol device {
  scan time 5;
}

protocol direct {
  interface "neocities*";
  import all;
}

On Wed, Sep 21, 2016 at 4:41 PM, Keenan Tims <ktims at gotroot.ca> wrote:

> These decisions are entirely made by Comcast's equipment - you have very
> little influence over their routing choices, especially because you do not
> peer directly with them. They do seem a bit nonsensical - Comcast's
> internal metrics should know that SEA is closer to PDX than LAX is, and
> should route accordingly, but this may be done for capacity reasons facing
> Choopa in Seattle or on the PDX-SEA links, an observation that the majority
> of Choopa traffic is destined for LA, and Seattle would be a detour, or who
> knows why else. You could ask Vultr to raise this with Comcast and see if
> you can get an adjustment or answer, some providers are responsive to this
> kind of request, but only from a direct customer/peer.
>
> There is one mechanism built into BGP for influencing route metrics - MED,
> but this is only transitive to the neighbour AS, so Comcast wouldn't see
> your MEDs, and it's very common to ignore them anyway.
>
> The other mechanism for influencing this kind of thing is through BGP
> community signalling, though this method is usually pretty blunt. Typically
> you can suppress announcements toward a specific AS entirely, or prepend
> toward them (lengthening the AS path and affecting BGP route selection);
> usually you can also modify the localpref of your announcement within your
> peer's network. The goal being, more or less, to encourage remote networks
> to avoid the problem AS/link entirely. If you're trying to avoid Comcast's
> messed up internal routing, your only real option is not to announce toward
> Comcast at all, and hope that Comcast will hand it off quickly to someone
> that routes it better. I wouldn't really count on it. Prepending might
> work, if Choopa and Comcast are peers, but if Choopa is a customer, Comcast
> will likely local-pref their routes above their peering/transit, and AS
> path length can't defeat this.
>
> Vultr indicates they support some BGP communities to do this kind of
> manipulation:
>
> https://www.vultr.com/docs/as20473-bgp-customer-guide
>
> Comcast might also support something similar, though it is common to strip
> communities before passing the routes upstream. You might be able to work
> with Vultr to have them signal to Comcast that your announcements are
> backup only or the like, or do so yourself and the Vultr/Choopa routers
> pass this along. But again, this has the same effect as not announcing to
> them at all - they will choose a *different* route that doesn't connect
> directly to Choopa, but it may not be better.
>
> More or less you're at their mercy.
>
> K
>
>
> Quoting Kyle Drake <kyle at kyledrake.net>:
>
>
>>>
>>> Normally you would use anycast to get you to a DNS server (which doesn't
>>> have to be that near), then a geographic DNS server to get you to the
>>> right
>>> CDN element.
>>>
>>>
>>> That's what I was doing previously, but I need to control the IPs for the
>> CDN, and I only have the budget for one /24, so I'm trying to make the
>> best
>> of it. Aside from some occasional weird routing, the network has worked
>> really well. State has not been an issue for what we're doing (short-lived
>> HTTP connections).
>>
>> I'm just trying to see what the extent of my powers to control weird
>> routing are. It's odd to see Comcast cold-potatoing connections to the
>> wrong routes, sometimes on the other side of the continent (
>> https://gist.github.com/kyledrake/7a4cd36ea276ec3134b4a51a42a37f48). I'm
>> wondering if there is a way to configure Bird to help steer these sorts of
>> things a little better, even if it's on a case-by-case or region-by-region
>> level.
>>
>> My apologies if these are all dumb questions. Again, not much anycast
>> documentation out there (I'm planning to improve this later by putting
>> together a web resource for people doing this).
>>
>> -Kyle
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20160921/479e0974/attachment.html>


More information about the Bird-users mailing list