BIRD memory usage

[Pavel Tvrdík]

Hi Justin,

On 2016-09-05 16:21, Justin Cattle wrote:
> Hi,
> 
> A colleague of mine reported a memory usage issue with the bird daemon
> last year, which resulted in a request for a core dump, but we never
> followed it up.
> I'd like to re-open this discussion and see if anything can be done to
> fix it.
> 
> I'll provide some information regarding a production environment,
> where the problem is most obvious.  But any further details and
> diagnostics will have to come from our lab environment.
> Please note, in production we mostly run 1.5, but in the lab we are on
> 1.6, however we see the same symptoms in both environments on both
> versions.
> 
> The symptoms are twofold, but potentially related -  greater than
> expected memory usage reported by the bird daemon itself for the
> number of routes, but also massively more memory actually used by the
> daemon process.
> 
> When the process is started, we see "normal" memory usage, which then
> seems to grow indefinitely in distinct steps, separated by a period of
> a few hours.
> 
> In production, this consumes most of the 32G of memory until the
> kernel oom-killer to intervenes.
> 
>> Production:
>> 
>> BIRD 1.5.0 ready.
>> 
>> bird> show memory
>> 
>> BIRD memory usage
>> 
>> Routing tables:   1405 MB
>> 
>> Route attributes:   84 kB
>> 
>> ROA tables:        192  B
>> 
>> Protocols:          45 kB
>> 
>> Total:            1405 MB
>> 
>> bird> show route count
>> 
>> 2273 of 2273 routes for 1142 networks
> 
>> # ps u  -p 3441
>> 
>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME
>> COMMAND
>> 
>> bird      3441  0.1 55.4 18275124 18241540 ?   Ssl  Aug10  73:39
>> /usr/sbin/bird -f -u bird -g bird
> 
> ..so that's ~1.4G reported by bird, and ~18G actually consumed by the
> process.
> 
>> Lab:
>> 
>> BIRD 1.6.0 ready.
>> 
>> bird> show mem
>> 
>> BIRD memory usage
>> 
>> Routing tables:    693 MB
>> 
>> Route attributes:   28 kB
>> 
>> ROA tables:        192  B
>> 
>> Protocols:          41 kB
>> 
>> Total:             693 MB
>> 
>> bird> show route count
>> 
>> 175 of 175 routes for 91 networks
> 
>> # ps u -p 29085
>> 
>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME
>> COMMAND
>> 
>> bird     29085  0.0 14.9 4994852 4915032 ?     Ssl  Aug05  19:41
>> /usr/sbin/bird -f -u bird -g bird

Thanks for this report. I successfully simulated this weird behavior 
too. The setting of kernel protocol with some export filter will cause 
memory leak bug. I prepared fixing commits in branch 
`krt-export-filtr-fix'

https://gitlab.labs.nic.cz/labs/bird/commits/krt-export-filtr-fix

Can you please download it and confirm, that the bug is fixed?

Best,
Pavel

> 
> ..so that's ~ 0.7G reported by bird, and ~5G actually consumed by the
> process.
> 
> I also attached the bird config from the lab.
> 
> Any help is much appreciated!
> Thanks.
> 
> Cheers,
> Just
> Notice:  This email is confidential and may contain copyright material
> of members of the Ocado Group. Opinions and views expressed in this
> message may not necessarily reflect the opinions and views of the
> members of the Ocado Group.
> 
> If you are not the intended recipient, please notify us immediately
> and delete all copies of this message. Please note that it is your
> responsibility to scan this message for viruses.
> 
> Fetch and Sizzle are trading names of Speciality Stores Limited and
> Fabled is a trading name of Marie Claire Beauty Limited, both members
> of the Ocado Group.
> 
> References to the “Ocado Group” are to Ocado Group plc (registered
> in England and Wales with number 7098618) and its subsidiary
> undertakings (as that expression is defined in the Companies Act 2006)
> from time to time.  The registered office of Ocado Group plc is Titan
> Court, 3 Bishops Square, Hatfield Business Park, Hatfield, Herts. AL10
> 9NE.