Races in birdc causing daemon to crash on low spec routers
Yifei Sun
yifei.sun at inria.fr
Tue Jun 2 11:09:49 CEST 2026
Hello all,
I'm hitting crash in BIRD 3.3.0 on several routers.
The machines are running
- NixOS x86_64-linux 7.0.10
- 2-5 BGP IPv4/IPv6 peers per affected host
- prometheus-bird-exporter 1.4.5, scraping every 30s
prometheus-bird-exporter runs `show protocols all` over
/run/bird/bird.ctl every ~30s, and on hosts with active BGP churn that
crashes bird inside bgp_show_proto_info(). systemd restarts it, the next
scrape kills it again, and NRestarts climbs ~120/hr...
toompea and timah took 14 and 13 cores in their last ~20 minutes of
uptime, highline took 4 over a slower span. The other seven routers
(butte, cradle, baldy, kongo, lantau, roraima, rysy) run the same build
and exporter with no cores (most likely because these machines are only
getting default routes). The crash needs concurrent BGP state changes.
For example:
```console
$ colmena exec --verbose --on @router -- 'sudo coredumpctl list || true'
timah |
highline |
kongo |
toompea |
butte |
cradle |
baldy |
rysy |
lantau |
roraima |
rysy | No coredumps found.
rysy | Succeeded
toompea | TIME PID UID GID SIG COREFILE EXE SIZE
toompea | Thu 2026-05-28 15:41:25 UTC 1185401 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 98.9M
toompea | Thu 2026-05-28 15:41:53 UTC 1265107 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.3M
toompea | Thu 2026-05-28 15:42:22 UTC 1265317 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.9M
toompea | Thu 2026-05-28 15:42:54 UTC 1265529 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 106.1M
toompea | Thu 2026-05-28 15:43:23 UTC 1265740 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 101.8M
toompea | Thu 2026-05-28 15:43:54 UTC 1265952 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 105.9M
toompea | Thu 2026-05-28 15:44:22 UTC 1266165 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 90.2M
toompea | Thu 2026-05-28 15:44:53 UTC 1266380 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.6M
toompea | Thu 2026-05-28 19:45:24 UTC 1266590 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 102.9M
toompea | Thu 2026-05-28 19:45:53 UTC 1355754 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 88.6M
toompea | Thu 2026-05-28 19:46:22 UTC 1355959 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 87.4M
toompea | Thu 2026-05-28 19:46:53 UTC 1356167 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.8M
toompea | Thu 2026-05-28 19:47:22 UTC 1356371 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 88.8M
toompea | Thu 2026-05-28 19:47:54 UTC 1356389 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 104.8M
toompea | Succeeded
butte | No coredumps found.
butte | Succeeded
timah | TIME PID UID GID SIG COREFILE EXE SIZE
timah | Thu 2026-05-28 15:36:51 UTC 1268745 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 96.9M
timah | Thu 2026-05-28 15:37:17 UTC 1268960 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 98.6M
timah | Thu 2026-05-28 15:37:52 UTC 1268974 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 95.9M
timah | Thu 2026-05-28 15:38:18 UTC 1269382 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 89.2M
timah | Thu 2026-05-28 15:38:53 UTC 1269397 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 95.4M
timah | Thu 2026-05-28 19:39:27 UTC 1269828 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 131.6M
timah | Thu 2026-05-28 19:39:48 UTC 1359330 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 76.2M
timah | Thu 2026-05-28 19:40:20 UTC 1359346 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 99.8M
timah | Thu 2026-05-28 19:40:49 UTC 1359574 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 94.1M
timah | Thu 2026-05-28 19:41:20 UTC 1359832 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 98.9M
timah | Thu 2026-05-28 19:41:51 UTC 1360097 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 93.9M
timah | Thu 2026-05-28 19:42:22 UTC 1360320 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 88.9M
timah | Thu 2026-05-28 19:42:48 UTC 1360526 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 78.8M
timah | Succeeded
highline | TIME PID UID GID SIG COREFILE EXE SIZE
highline | Thu 2026-05-28 11:34:31 UTC 1203740 999 999 SIGSEGV present /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 99.4M
highline | Thu 2026-05-28 11:37:02 UTC 1204447 999 999 SIGSEGV present /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 81.9M
highline | Thu 2026-05-28 11:40:00 UTC 1205349 999 999 SIGSEGV present /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 108.9M
highline | Thu 2026-05-28 12:15:56 UTC 1220240 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 80.3M
highline | Succeeded
baldy | No coredumps found.
baldy | Succeeded
kongo | No coredumps found.
kongo | Succeeded
roraima | No coredumps found.
roraima | Succeeded
lantau | No coredumps found.
lantau | Succeeded
cradle | No coredumps found.
cradle | Succeeded
| All done!
```
```console
$ sudo coredumpctl info
PID: 1356389 (bird)
UID: 993 (bird)
GID: 991 (bird)
Signal: 11 (SEGV)
Timestamp: Thu 2026-05-28 19:47:42 UTC (1h 37min ago)
Command Line: /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird -c /etc/bird/bird.conf
Executable: /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird
Control Group: /system.slice/bird.service
Unit: bird.service
Slice: system.slice
Boot ID: fd4c001e44dc44e1a05cbc5d60440ecb
Machine ID: 77293224076f4ff7845c2358cb35a4c0
Hostname: toompea
Storage: /var/lib/systemd/coredump/core.bird.993.fd4c001e44dc44e1a05cbc5d60440ecb.1356389.1779997662000000.zst (present)
Size on Disk: 104.8M
Message: Process 1356389 (bird) of user 993 dumped core.
Module /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird without build-id.
Module libz.so.1 without build-id.
Module libssh.so.4 without build-id.
Stack trace of thread 1356389:
#0 0x000059d538758d08 bgp_show_proto_info.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0xdcd08)
#1 0x000059d538706fb2 proto_cmd_show (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x8afb2)
#2 0x000059d5387077c1 proto_apply_cmd.isra.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x8b7c1)
#3 0x000059d5386a2de8 cf_parse.isra.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x26de8)
#4 0x000059d5386ac462 cli_parse (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x30462)
#5 0x000059d5386f96e5 cli_command (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x7d6e5)
#6 0x000059d5386f9980 cli_event.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x7d980)
#7 0x000059d5386dff35 ev_run_list_limited (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x63f35)
#8 0x000059d538693c8c main (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x17c8c)
#9 0x00007ae37602b285 __libc_start_call_main (libc.so.6 + 0x2b285)
#10 0x00007ae37602b338 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2b338)
#11 0x000059d538694885 _start (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x18885)
Stack trace of thread 1356391:
#0 0x00007ae3760a6922 __syscall_cancel_arch (libc.so.6 + 0xa6922)
#1 0x00007ae37609a00c __internal_syscall_cancel (libc.so.6 + 0x9a00c)
#2 0x00007ae37609a084 __syscall_cancel (libc.so.6 + 0x9a084)
#3 0x00007ae37611778e __poll (libc.so.6 + 0x11778e)
#4 0x000059d5387b3bfe bird_thread_main.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x137bfe)
#5 0x00007ae37609dd53 start_thread (libc.so.6 + 0x9dd53)
#6 0x00007ae37612563c __clone3 (libc.so.6 + 0x12563c)
ELF object binary architecture: AMD x86-64
```
I temporarily worked around this by removing the prefix count showed
when invoking `birdc show proto all`. But I’m not sure if this type of patches
are acceptable for upstreaming (please see reasoning in the commit
message in the patch):
https://raw.githubusercontent.com/stepbrobd/inc/refs/heads/master/pkgs/bird3/fix.patch
Best,
Yifei
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20260602/c31c0054/attachment.htm>
More information about the Bird-users
mailing list