On the possibility of updating BGP passwords without network disruption.

[Alexander Zubkov]

Hello,

Do you have graceful restart enabled for your session from both ends?
But anyway, I'm not sure that bird uses graceful session restart when
it restarts protocols due to reconfiguring. Maybe someone tested it
already or Bird developers will tell for sure.

On Mon, Aug 8, 2022 at 5:06 PM Calvin Zachman <calvin.zachman at ibm.com> wrote:
>
> Hi BIRD users,
>
>
>
> Does anyone know whether a BGP shared secret can be rotated without incurring any network downtime? I did some testing with the BGP password functionality offered and it appears that any update to the BGP password configuration incurs a brief network outage with both existing/new connections. It seems like something about the way BIRD is restarting is leading to it pulling down learned routes immediately as opposed to letting them live according to the timeout setting. Does BIRD flush all routes it has learned when this configuration changes? Here is a brief excerpt to demonstrate the outage. Take note that the network disruption precisely matches the timestamp at which BIRD is reconfigured:
>
>
>
> # Logs from calico-node-4w4bv (10.95.14.104)
>
> 2022-06-29 18:26:18.836 [INFO][60] confd/client.go 1415: Trigger to recheck BGP peers following possible password update
>
> 2022-06-29 18:26:18.836 [INFO][60] confd/client.go 250: Recompute v1 BGP peerings
>
> 2022-06-29 18:26:18.836 [INFO][60] confd/client.go 949: Recompute BGP peerings:
>
> bird: Reconfiguration requested by SIGHUP
>
> 2022-06-29 18:26:18.844 [INFO][60] confd/resource.go 278: Target config /etc/calico/confd/config/bird.cfg has been updated due to change in key: /calico/bgp/v1/global
>
> bird: Reconfiguring
>
> bird: device1: Reconfigured
>
> bird: direct1: Reconfigured
>
> bird: Restarting protocol Mesh_10_95_14_105
>
> bird: Mesh_10_95_14_105: Shutting down
>
> bird: Mesh_10_95_14_105: State changed to stop
>
> bird: Restarting protocol Mesh_10_95_14_110
>
> bird: Mesh_10_95_14_110: Shutting down
>
> bird: Mesh_10_95_14_110: State changed to stop
>
> bird: Mesh_10_95_14_105: State changed to down
>
> bird: Mesh_10_95_14_105: Initializing
>
> bird: Mesh_10_95_14_105: Starting
>
> bird: Mesh_10_95_14_105: State changed to start
>
> bird: Mesh_10_95_14_110: State changed to down
>
> bird: Mesh_10_95_14_110: Initializing
>
> bird: Mesh_10_95_14_110: Starting
>
> bird: Mesh_10_95_14_110: State changed to start
>
> bird: Reconfigured
>
> bird: Mesh_10_95_14_110: Connected to table master
>
> bird: Mesh_10_95_14_110: State changed to feed
>
> bird: Mesh_10_95_14_110: State changed to up
>
> bird: Mesh_10_95_14_105: Connected to table master
>
> bird: Mesh_10_95_14_105: State changed to feed
>
> bird: Mesh_10_95_14_105: State changed to up
>
> 2022-06-29 18:26:36.079 [INFO][58] monitor-addresses/autodetection_methods.go 117: Using autodetected IPv4 address 10.95.14.104/26 on matching interface eth0
>
> 2022-06-29 18:26:54.537 [INFO][62] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m2.1s: avg=5ms longest=21ms (resync-filter-v4,resync-nat-v4)
>
>
>
> # Connection Tester Daemonset (hitting an echoserver twice a second or so)
>
>
>
> Wed Jun 29 18:26:17 UTC 2022
>
> Successful echo server connection
>
> Wed Jun 29 18:26:17 UTC 2022
>
>
>
> Wed Jun 29 18:26:18 UTC 2022
>
> Successful echo server connection
>
> Wed Jun 29 18:26:18 UTC 2022
>
>
>
> Wed Jun 29 18:26:18 UTC 2022
>
> curl: (28) Connection timed out after 300 milliseconds
>
> Failed to connect to echo server
>
> Wed Jun 29 18:26:19 UTC 2022
>
>
>
> Wed Jun 29 18:26:19 UTC 2022
>
> curl: (28) Connection timed out after 300 milliseconds
>
> Failed to connect to echo server
>
> Wed Jun 29 18:26:20 UTC 2022
>
>
>
> Wed Jun 29 18:26:20 UTC 2022
>
> Successful echo server connection
>
> Wed Jun 29 18:26:20 UTC 2022
>
>
>
> # For peer /host/10.95.14.81/ip_addr_v4
>
> protocol bgp Mesh_10_95_14_81 from bgp_template {
>
>   neighbor 10.95.14.81 as 64512;
>
>   source address 10.95.14.82;  # The local address we use for the TCP connection
>
>   graceful restart time 1800; # This parameter seems to make no difference when changing BGP passwords
>
>   password "LJiKASiglY+KafEwEn/cSmkiok0zHgpQq5EtYhYgoDcSQwKIpX22Tz7jOzX+";
>
> }
>
>
>
>
>
> I have perused the RFCs for both BGP Graceful Restart (4724) & Secure BGP Sessions (2385) but haven't found a solid answer yet. When the password is changed it makes complete sense that any peers with the new password will refuse to accept any NEW routes received from peers using the old password and vice versa. I don't see the fundamental reason why TCP segments arriving with some unexpected hash necessitates that previously learned routes from that peer need to be flushed with no TTL, but the observance of the outage suggests that is what is happening. One would think that, in principle, it could wait to tear down existing routes until a configurable timeout (say the graceful restart) expires, providing a window in which we can change the password and maintain stable routing.
>
>
>
> I am relatively new to BGP and am using BIRD indirectly via Calico for container networking inside Kubernetes. I will of course take things up with the guys behind Calico, but is there anything in the BGP spec/BIRD implementation which fundamentally prevents network disruption free secret rotation? Let me know if there is any place I should look for more information on this or any debug logs which would be helpful.
>
>
>
> Thanks,
>
> Calvin
>
>