kernel: BIRD kernel syncer protocol issues on Linux
Sergey Popovich
popovich_sergei at mail.ru
Thu Sep 26 11:21:02 CEST 2013
Helo, BIRD developers!
I have found few issues with kernel syncer on Linux with enabled
route learning.
There are few problems in sysdep/linux/rtnetlink.c with netlink
code:
* Use after free when accessing nl_table_map[] in nl_parse_route().
* Possible race condition between rx buffer allocation nl_async_rx_buffer in
for async socket and nl_open_async() and it's usage in nl_async_hook().
* Socket descriptor leakage in error path in nl_open_async() when
bind(2) call fails.
These problems addressed with patch
bird-1.3.11-fix-shutdow~ath-in-rtnetlink.patch
Additionaly, patch bird-1.3.11-fix-building-with-DBG-on.patch fixes
building when -DGLOBAL_DEBUG or -DLOCAL_DEBUG enabled (enables building
lots of debugging code available with DBG() macro).
Information to reproduce problem with use after free in nl_parse_route().
==========================================
BIRD configuration
------------------------
# Configure logging
log stderr all;
# More chances to trigger crash.
log syslog all;
router id 172.16.1.1;
protocol device devices {
scan time 120;
}
protocol kernel kernel10 {
debug all;
persist no;
scan time 120;
learn yes;
device routes no;
kernel table 10;
import all;
export none;
}
Start BIRD, compiled with "-DGLOBAL_DEBUG" in debugging mode
------------------------------------------------------------------------------------
# bird -d
olock: init
Parsing configuration file `/etc/bird.conf'
Protocol preconfig: Direct Static Pipe BGP Kernel Device
Protocol postconfig: devices kernel10
sysdep_commit
global_commit
rt_commit
rt_commit:
master: created
Allocating FIB hash of order 10: 1024 entries, 0 low, 4096 high
done
protos_commit
protos_commit:
26-09-2013 10:57:45 <TRACE> kernel10: Initializing
done
Protocol start
...
Kicking kernel10 up
26-09-2013 10:57:45 <TRACE> kernel10: Starting
Allocating FIB hash of order 10: 1024 entries, 0 low, 4096 high
kernel10 reporting state transition HUNGRY/DOWN -> */UP
kernel10: Scheduling meal
Connecting protocol kernel10 to table master
26-09-2013 10:57:45 <TRACE> kernel10: Connected to table master
26-09-2013 10:57:45 <TRACE> kernel10: State changed to feed
do_commit finished with 0 obstacles remaining
26-09-2013 10:57:45 <INFO> Started
Entering I/O loop.
Feeding protocol devices
Feeding protocol devices continued
Announcing routes to new protocol devices
Protocol devices up and running
Feeding protocol kernel10
Feeding protocol kernel10 continued
Announcing routes to new protocol kernel10
26-09-2013 10:57:45 <TRACE> kernel10: State changed to up
Protocol kernel10 up and running
KRT: Scanning routing table
26-09-2013 10:57:45 <TRACE> kernel10: Scanning routing table
KRT: Got 192.168.10.0/24, type=6, oif=-1, table=10, prid=3, proto=kernel10
26-09-2013 10:57:45 <TRACE> kernel10: 192.168.10.0/24: [alien] created
KRT: Got 192.168.20.0/24, type=6, oif=-1, table=20, prid=3, proto=(none)
...
KRT: Pruning table master
26-09-2013 10:57:45 <TRACE> kernel10: Pruning table master
KRT: Pruning inherited routes
26-09-2013 10:57:45 <TRACE> kernel10: Pruning inherited routes
192.168.10.0/24: announcing (metric=0)
26-09-2013 10:57:45 <TRACE> kernel10 > added [best] 192.168.10.0/24 blackhole
26-09-2013 10:57:45 <TRACE> kernel10 < rejected by protocol 192.168.10.0/24
blackhole
After startup BIRD learns route from KRT 10
--------------------------------------------------------
# birdc 'show route table master all'
BIRD 1.3.11 ready.
192.168.10.0/24 blackhole [kernel10 10:57] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
Change KRT ID to 10 ("kernel table 10" configuration option)
and tell bird about configuration changes
----------------------------------------------------
# birdc configure
BIRD 1.3.11 ready.
Reading configuration from /etc/bird.conf
Reconfiguration in progress
Debug output after reconfiguration
---------------------------------------------
Parsing configuration file `/etc/bird.conf'
Protocol preconfig: Direct Static Pipe BGP Kernel Device
Protocol postconfig: devices kernel10
26-09-2013 10:57:54 <INFO> Reconfiguring
sysdep_commit
global_commit
rt_commit
rt_commit:
master: same
done
protos_commit
protos_commit:
devices: same
26-09-2013 10:57:54 <INFO> Restarting protocol kernel10
+++ adding obstacle 1
Kicking kernel10 down
26-09-2013 10:57:54 <TRACE> kernel10: Shutting down
KRT: Flushing kernel routes
26-09-2013 10:57:54 <TRACE> kernel10: Flushing kernel routes
kernel10 reporting state transition HAPPY/UP -> */DOWN
kernel10: Scheduling flush
26-09-2013 10:57:54 <TRACE> kernel10: State changed to flush
Pruning neighbors
done
Protocol start
do_commit finished with 1 obstacles remaining
Pruning route table master
26-09-2013 10:57:54 <TRACE> kernel10 > removed [sole] 192.168.10.0/24
blackhole
Pruning route table master
Flushing protocol kernel10
26-09-2013 10:57:54 <TRACE> kernel10: State changed to down
Protocol kernel10 down
kernel10 has shut down for reconfiguration
+++ deleting obstacle 1
26-09-2013 10:57:54 <TRACE> kernel10: Initializing
Kicking kernel10 up
26-09-2013 10:57:54 <TRACE> kernel10: Starting
Allocating FIB hash of order 10: 1024 entries, 0 low, 4096 high
kernel10 reporting state transition HUNGRY/DOWN -> */UP
kernel10: Scheduling meal
Connecting protocol kernel10 to table master
...
KRT: Scanning routing table
26-09-2013 10:57:54 <TRACE> kernel10: Scanning routing table
KRT: Got 192.168.10.0/24, type=6, oif=-1, table=10, prid=3, proto=kernel10
26-09-2013 10:57:54 <TRACE> kernel10: 192.168.10.0/24: [alien] created
KRT: Got 192.168.20.0/24, type=6, oif=-1, table=20, prid=3, proto=kernel10
26-09-2013 10:57:54 <TRACE> kernel10: 192.168.20.0/24: [alien] created
**************************************************************************
And route from KRT ID 10 is seen and accepted in nl_parse_route() at
sysdep/linux/rtnetlink.c.
**************************************************************************
...
KRT: Pruning table master
26-09-2013 10:57:54 <TRACE> kernel10: Pruning table master
KRT: Pruning inherited routes
26-09-2013 10:57:54 <TRACE> kernel10: Pruning inherited routes
192.168.10.0/24: announcing (metric=0)
26-09-2013 10:57:54 <TRACE> kernel10 > added [best] 192.168.10.0/24 blackhole
26-09-2013 10:57:54 <TRACE> kernel10 < rejected by protocol 192.168.10.0/24
blackhole
192.168.20.0/24: announcing (metric=0)
26-09-2013 10:57:54 <TRACE> kernel10 > added [best] 192.168.20.0/24 blackhole
26-09-2013 10:57:54 <TRACE> kernel10 < rejected by protocol 192.168.20.0/24
blackhole
26-09-2013 10:57:54 <INFO> Reconfigured
Feeding protocol kernel10
Feeding protocol kernel10 continued
Announcing routes to new protocol kernel10
26-09-2013 10:57:54 <TRACE> kernel10 < rejected by protocol 192.168.10.0/24
blackhole
26-09-2013 10:57:54 <TRACE> kernel10 < rejected by protocol 192.168.20.0/24
blackhole
26-09-2013 10:57:54 <TRACE> kernel10: State changed to up
Protocol kernel10 up and running
After reconfiguration finishes, we see route from KRT ID 10 in BRT "master"
table,
to which syncer is attached.
------------------------------------------------------------------------------------------------
# birdc 'show route table master all'
BIRD 1.3.11 ready.
192.168.10.0/24 blackhole [kernel10 10:57] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
192.168.20.0/24 blackhole [kernel10 10:57] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
And adding routes to KRT ID 10 when BIRD uses KRT ID 20 causes them to be
stored in "master" table:
---------------------------------------------------------------------------------------------------
# ip -4 route add blackhole 192.168.30.0/24 table 10
# birdc 'show route table master all'
BIRD 1.3.11 ready.
192.168.10.0/24 blackhole [kernel10 10:57] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
192.168.20.0/24 blackhole [kernel10 10:57] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
192.168.30.0/24 blackhole [kernel10 10:58] * (10)
Type: inherit unicast univ
Kernel.source: 3
Kernel.metric: 0
And debugging output from BIRD
------------------------------------------
KRT: Received async route notification (24)
KRT: Got 192.168.30.0/24, type=6, oif=-1, table=10, prid=3, proto=kernel10
26-09-2013 11:15:24 <TRACE> kernel10: 192.168.30.0/24: [alien async] created
krt_learn_async: distributing change
26-09-2013 11:15:24 <TRACE> kernel10 > added [best] 192.168.30.0/24 blackhole
26-09-2013 11:15:24 <TRACE> kernel10 < rejected by protocol 192.168.30.0/24
blackhole
There also possible to trigger crash, infinite loop (spot on bird 1.3.9 on the
other
system), but I have no instructions to reproduce this reliable. At least
logging to
syslog or to a file among with logging to stderr and changing name of the
protocol
(kernel10->kernel20 etc.) after route learned from other KRT, or wait for
periodic
KRT scan, possibly followed by protocol name change).
--
SP5475-RIPE
Sergey Popovich
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bird-1.3.11-fix-building-with-DBG-on.patch
Type: text/x-patch
Size: 465 bytes
Desc: not available
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20130926/210b5440/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bird-1.3.11-fix-shutdown-path-in-rtnetlink.patch
Type: text/x-patch
Size: 1172 bytes
Desc: not available
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20130926/210b5440/attachment-0001.bin>
More information about the Bird-users
mailing list