Out of memory crash of 3000 routes in BGP

[Justin Cattle]

I must admit, after deploying 1.6.2 to production, I am still seeing more
memory usage than expected.
I was going to take some time to gather some more stats - but seeing this
made me jump early :)

Memory usage is still better than it was originally, but I see 2G in some
cases, which seems pretty excessive for 2500 routes.

The other strange this is, I see a lot of variance across hosts with the
same route count, config, and similar process run times.
Some node processes are using 12Mg - 16Mg, while some are using up
to 2228Mg after just one day.

The bird.conf has changed a little, so I'll attach it again here.
Madhu - perhaps you should provide your bird.conf as well ?


Also - some basic info:


# birdc
BIRD 1.6.2 ready.
bird>
bird> show route count
2484 of 2484 routes for 1245 networks
bird> show mem
BIRD memory usage
Routing tables:   2235 MB
Route attributes:   87 kB
ROA tables:        192  B
Protocols:          41 kB
Total:            2235 MB



If you need any further info - please let me know.





Cheers,
Just

On 11 October 2016 at 02:21, Madhu <informmadhu at gmail.com> wrote:

> I have 3000 routes with 64 path ecmp. It is crashing . I don't see the
> coredump . Is there anyway to fix using trace ?
>
> Madhu
>
> Sent from my iPhone
>
> On Oct 10, 2016, at 6:08 PM, Jonathan Stewart <jonathan.stewart at gmail.com>
> wrote:
>
> It is not a general issue, many here run hundreds of thousands of routes.
>
> It must be something specific to your environment.
>
> Jonathan
>
> On 10 Oct 2016 4:40 p.m., "Madhu" <informmadhu at gmail.com> wrote:
>
>> Looks like the problem is still there. I used 1.6.0 and also 1.6.2.
>>
>>
>>
>> On Mon, Oct 10, 2016 at 10:56 AM, james machado <hvgeekwtrvl at gmail.com>
>> wrote:
>>
>>> Madhu check out the following thread.  http://bird.network.cz/piperm
>>> ail/bird-users/2016-September/010578.html
>>>
>>>
>>>
>>> On Mon, Oct 10, 2016 at 10:02 AM, Madhu <informmadhu at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>>   I am seeing a out of memory crash with 3000 routes in BGP. Is this a
>>>> known issue? I m seeing this issue in 1.6.0 and 1.6.2 .
>>>>
>>>>
>>>> [ 1098.912753] Call Trace:
>>>> [ 1098.915481]  [<ffffffff81b96521>] dump_stack+0x64/0x82
>>>> [ 1098.921225]  [<ffffffff81b91a46>] dump_header+0x7f/0x1f1
>>>> [ 1098.927163]  [<ffffffff810b3e76>] ? put_online_cpus+0x56/0x80
>>>> [ 1098.933587]  [<ffffffff811110ac>] ? rcu_oom_notify+0xcc/0xf0
>>>> [ 1098.939914]  [<ffffffff811a2735>] oom_kill_process+0x205/0x360
>>>> [ 1098.946435]  [<ffffffff814595b5>] ? security_capable_noaudit+0x15/
>>>> 0x20
>>>> [ 1098.953733]  [<ffffffff811a2ee2>] out_of_memory+0x492/0x4d0
>>>> [ 1098.959962]  [<ffffffff811a8f30>] __alloc_pages_nodemask+0xa00/0xb60
>>>> [ 1098.967067]  [<ffffffff811eb1fa>] alloc_pages_vma+0x9a/0x160
>>>> [ 1098.973393]  [<ffffffff811cd5c8>] handle_mm_fault+0xd38/0x1080
>>>> [ 1098.979914]  [<ffffffff811d4b74>] ? change_protection+0x594/0x850
>>>> [ 1098.986726]  [<ffffffff8109f5ae>] __do_page_fault+0x19e/0x560
>>>> [ 1098.993149]  [<ffffffff811d4f81>] ? mprotect_fixup+0x151/0x290
>>>> [ 1098.999669]  [<ffffffff8109f9a1>] do_page_fault+0x31/0x70
>>>> [ 1099.005704]  [<ffffffff81ba40e8>] page_fault+0x28/0x30
>>>>
>>>> Madhu
>>>>
>>>
>>>
>>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20161011/8a670def/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bird.conf
Type: application/octet-stream
Size: 1437 bytes
Desc: not available
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20161011/8a670def/attachment.obj>