FreeBSD, multiple routing tables, "KRT: Error sending route 192.222.249.232/29 to kernel: File exists"

[Dan Mahoney]

> On Mar 12, 2022, at 9:39 PM, Marek Zarychta <zarychtam at plan-b.pwste.edu.pl> wrote:
> 
> W dniu 12.03.2022 o 16:48, Dan Mahoney pisze:
>> Hey there,
>> 
>> We've got a fairly simple setup where we have two fibs.  Interface routes are in both fibs the way BSD does it.  (Table 1 is the second fib, if it matters).
> 
> No, by default interface route is added only in interface fib for FreeBSD 13.0 and later. Setting "net.add_addr_allfibs=1" brings back old behavior.
> 
>> trying
>> log syslog all;
>> 
>> protocol kernel {
>>     scan time 20;
>>     ipv4 {
>>         import all;
>>         export all;
>>         table master4;
>>         };
>>     kernel table 1;
>>     }
>> 
>> protocol kernel {
>>     ipv6 {
>>         import all;
>>         export all;
>>         table master6;
>>         };
>>     kernel table 1;
>>     }
>> 
>> 
>> protocol device {
>>         scan time 10;
>>         }
>> 
>> setfib 1 netstat -nr:
>> 
>> Routing tables (fib: 1)
>> 
>> Internet:
>> Destination        Gateway            Flags     Netif Expire
>> default            192.222.249.233    UGS        bge0
>> 127.0.0.1          link#5             UH          lo0
>> 127.0.0.2          link#6             UH          lo1
>> 192.222.249.232/29 link#1             U          bge0
>> 
>> fib 0 would be the same.
>> 
>> An old thread seemed to imply this was because the "direct" protocol was missing.
>> 
>> Adding the "direct" protocol doesn't stop this error every 20 seconds in syslog (which seems to be related to the scan time).
>> 
>> Mar 12 15:24:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
>> Mar 12 15:24:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
>> Mar 12 15:24:51 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
>> Mar 12 15:25:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
>> Mar 12 15:25:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
>> 
>> Is it possible bird isn't differentiating routes learned from each fib and is learning from both, separately, and not properly deduplicating those routes?
>> 
>> -Dan
> 
> What FreeBSD version are you running? FreeBSD routing stack has been significantly rewritten for 13.0-RELEASE, moreover this rewritten routing stack in stable/13 was polished a bit, so if you are encountering this on 13.0-RELEASE, then please upgrade to 13.1-BETA.

We're on 12.3.  We're not ready to run a .0 OS release in production.

> 
> You can also probably either:
> 
> 1. Filter this route to prevent it from being added.

That's an option, yes.  We'd have to write some automation to exclude this specific route, but still allow the export of all the rest of our BGP routes.  I'd still like to understand why bird *feels the need* to send this route to the kernel.  Is it learning it from fib 0 and attempting to update fib 1 or something?  Is there a log I can enable that would confirm that's happening?

> 
> 2. Disable aforementioned  "net.add_addr_allfibs".

Thanks, these look promising, even if we do have to wait for the roll to 13.0 or 13.1.  I'd still ideally like to find a solution that works on 12.x, which is a production release for a while now.

-Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20220313/3b19ef38/attachment.htm>