TalkTalk & BTW Leased Lines

TalkTalk finally closed the fault case, stating all services have been restored.
We noted no further interruption to managed services throughout the afternoon, and therefore will be closing our internal case.
We can only apologise for the inconvenience, and will of course communicate an Incident Report as soon as the supplier provides one.

Although all services have now been stable for a couple of hours, TalkTalk still have the case open and are pending further updates.

Therefore the services must remain “at risk” until they provide an all clear notice.

Services are still unstable, so we’ve chased the supplier for an update.
Presently we are being advised works are ongoing until approx 3pm.

As TalkTalk have continued to work through the issues this morning, we’ve seen a gradual return to service, and as I type, a flap of a number of services occurred, returning all but one circuit into service.

We do not have a formal clear notice from TTB as yet, so services should still be considered At Risk

Although all services were restored at 00:19, proactive monitoring alerted to a further number of circuits which lost service once again.

TalkTalk are aware, and are presently investigating.

TTB Update - All serviced restored as of 00:19.

We do not have an update from the supplier as yet - however can see that all monitored services have returned.

Further updates and Incident report to be published in the morning.

Our apologies for the inconvenience.

Latest Update
Talk Talk engineers will be taking further steps to resolve this issue by restarting a process on device NPE001.THN the out of hours planned works window, 00:01hrs onwards 08/06/201
Further information will be provided within the next update due time.

TalkTalk have advised:

-Severity Change- This incident has been upgraded to severity P1.

Investigations are underway with our engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until our engineers have completed further diagnostics.

TalkTalk have updated as follows:

Fault Description: Network monitoring has indicated that there is a loss of service for some customers connected due to a crashed files on the network.

-Impact Analysis -
Impact Type: TLOS
Impacted Exchanges:
Potential Number of customers affected:
B2B:80

RCA and Resolution details will follow shortly.

The outage is presently still under supplier investigation.
We are advised further updates are due in 10-15 mins.

Monitoring has alerted to a significant number of TTB and BTW Leased Lines down spread around the country.
There is no obvious location/pattern to the loss of service at present, and we are raising case with the supplier.