Core Router London 1 - Memory Exhaustion

Major incident Core Network East Router1 Switch1 Switch2 Switch3 DNS Authorative1 DNS Recursive2
2024-11-03 19:17 GMT · 49 minutes

Updates

Resolved

The router has been rebooted.
Optimal routing has been restored.
All xDSL/FTTc/p customer connections have returned.
Free router memory has returned to an expected level.

And as a minor by product, the router firmware was updated to a level which is (according to Cisco) no lower troubled by memory leaks.
I wait to be convinced!

Apologies for the inconvenience.

November 3, 2024 · 20:05 GMT
Monitoring

Full routing tables have reloaded and all bar 6 xDSL customer connections have returned. Routing is still propagating around our core.

Apologies for the inconvenience.

November 3, 2024 · 19:37 GMT
Investigating

The router has returned to service. Routing flaps will occur once again as the tables reload and optimum routes are restored.

Presently around 80 xDSL/SoGEA/FTTC and P customer sessions have not returned, although are popping back as I type.

A customer router reboot might be required if these do not return in the next few minutes.

November 3, 2024 · 19:33 GMT
Issue

Monitoring has reported free memory has dropped below a critical level.

We are about to commence a router reboot to avoid crashing into a none routing state which threatens to black hole some traffic.

A major routing update will occur across our core as providers are flushed from the routing table. This will inevitably cause poor traffic flows for a period while routes re-establish.

Apologies for the short notice, but it makes more sense to attend to this issue this evening than it potentially affect a working weekday.

November 3, 2024 · 19:17 GMT

← Back