VirginMedia Outage, Middle England

VM 00.00

Dear Customer,

The service has been restored and a full RCA will be shared in due course - in the meantime should you experience and further issues please contact our service desk with you circuit ID.
Thank you
VM02

VM 20.00

Hello

Our engineer is at our Seacroft headend completing further tests on the core fibres to locate the remaining fibre damage. We are now awaiting further updates.

Kind regards,
The Fault Management Centre

VM 18:00

The ERS splicer has completed an audit of the 160f cable.
Engineers are now asessing the remaining impact, once completed investigations will continue for the remaining circuits still at fault.

VM 17:00

The ERS Splicer is continuing to audit the previously spliced 160F cable and currently completed 16 of 20 tubes. Once complete any remaining impact will be assessed. Some impact has come up when the recent tubes were completed.

Kind regards,
VMB

While it would appear all bar one of our circuits have returned, clearly Virgin are still working on the issue, so services should continue to be considered “at risk” at the present moment.

The ERS Splicer is contiuning to audit the previously spliced 160F cable and currently completed 9 of 20 tubes.
At 13:46 the remaining residential impacting faults restored
MIM remains engaged with all Support Teams working towards full restoration of services.

Kind regards,
VMB

VM.

As suspected they’ve still got problems.

Engineering teams have arrived at site and started investigating the fibres. There are potentially 2 tubes crossed, but this is being confirmed. A Conference call has been arranged for 9:30 to discuss the current plan and estimation of work.

VM advise that all Splicing is complete.

From a service delivery overview, we can still see a couple of services that have not returned, and are working on making contact with the customers to establish if this is due to site local issues ot ongoing Virgin troubles.

VM 02:00

Update 17

The 1st ERS Splicer has completed splicing the B-end of the 160F cable (1 of 3).

The 2nd ERS Splicer is working to splice the remaining 160F cable (3 of 3)

Splicing progress as follows:

160f 3 of 3
A-end
20 tubes of 20 completed
B-end
6 tubes of 20 completed (Splicing in progress)

160f 1 of 3 - Splicing completed
160f 2 of 3 - Splicing completed
48f 1 of 2 - Splicing completed
48f 2 of 2 - Splicing completed
16f 1 of 1 - Splicing completed

VM. Midnight.

Update 16
The VMO2 Network Engineer has confirmed that the 2nd ERS Splicer has arrived on site as of 23:00, and splicing is about to commence for the 160F cable (3 of 3)

The 1st ERS Splicer continue to splice the B-end of the 160F cable (1 of 3).

VM 22.30

ERS Splicer continue to splice the B-end of the 160F cable (1 of 3).

The NOM has confirmed that an additional Splicer will arrive on-site at 22:00, and splice the remaining 20 tubes at B-end of the 160F cable (3 of 3

VM 20:00

Update 14
ERS Splicers continue to splice the B-end of the 160F cable (1 of 3).

The Splicer working on the 48F cables finished splicing and has been stood down due to H&S issues.

The NOM is working to arrange an additional Splicer to splice the remaining 20 tubes at B-end of the 160F cable (3 of 3)

VM 17:31

ERS Splicers are currently recreating the 1st 160f cable joint enclosures and will then commence with preparing the fibres. Alarms are being reviewed to determine which services are restored and the impact will be updated accordingly. ERS Splicers are due to handover to replacement field resources this evening to complete the remaining 160f cables. Once the 48f cables are completed, the VMO2 Splicers will leave site.

Kind regards,
VMB

VM: 15:24

Not much of an update I’m afraid, but this is what they’ve offered.

All Engineering teams remain on site working towards restoration of services.

Kind regards,
VMB

VM:

ERS Splicers are continuing to splice the 3x 160f cables.
VMO2 Day Splicers are currently preparing the 2x 48f cables for splicing.
Once the 48f cables have been restored, a Splicer will complete the 16f cable.

VM 11:58

Splicers continue to work on site, they are aiming for services to start restoring soon. All support teams remain engaged and alarms being monitored to see any restoration.

Kind regards,

VM Update: 10:40

ERS Splicers (3 of 3) are currently working to splice the 3x160f cables.

4 Day Splicers are currently onsite as of 09:45 and are working to prepare and then splice the 3x48f cables.

Monitoring of alarms for restoration is ongoing.

Splicing continues at a steady pace and we are seeing further circuits back in service

VM Update.

Current Situation

The ERS Splicers remain working towards splicing the first 160f cable at both ends and have spliced 3 trays so far.

Support teams will continue to monitor before confirming service restoration.

VM Update 4.

Current Situation

The Network Engineer has advised that 15m of 3x160f, 2x48f & 16f straps have been pulled between the new pits.

2 ERS Splicers arrived on site and have started preparing the 1st 160F cable at both ends.

VM Update 3

Current Situation
Update 3
MIM engaged the Network Engineer on site who confirmed that the ERS Civils remain working to build the 2 new pits outside of [removed], to house the new joints.

An ETR of 20:00 has been provided for completion of this work.

Once the 2 new pits have been built the 2 ERS Splicers will arrive on site. The ERS supervisor has also confirmed that a further Splicer will arrive on site during the night to assist with ongoing restorations.

Plan

• Replacement sections of fibre cable to be introduced and spliced into new joints either side of the damaged section [removed]

Resources Onsite:

• 1 Network Engineer
• 1 ERS Supervisor
• ERS Civils

Resources Requested:

• ERS Cable Gang
• 2 ERS Splicers

VM Latest Update.

Please note the ETA is for the building of the jointing chambers and following this, the fibre splicing will commence.

ERS Civil exposed the break location and the cables were assessed. 6x cables have been completely severed with 3x partially damaged and the remaining cables undamaged. ERS Civils are currently building 2x new pits to house the new joints. ERS have provided an ETA of 20:00 to complete the new pits.
ERS Supervisor is currently arranging for ERS resources to be available until service is restored.

VM Update:

Network Engineers, Civils Crew and ERS engineers are continuing work on site working towards restoration.

Kind regards,
VMB

Update from Virgin:

External civils were engaged and advised that while creating a trench VMO2 ducting was damaged.
An ERS Supervisor is currently onsite as of 13:59 to provide ERS resources. The existing damaged ducting & sub-ducting will be removed and the cables exposed to allow for the field resources to assess the condition of each cable.
Once the assessment has been completed, the ERS Supervisor will request replacement sections of each fibre cable type.

The damaged section of each cable will be removed and a replacement joints introduced either side of the damage. 2 new pit locations will be created either side of the damage for the new joints to be placed into.

Apologies for the ongoing troubles. The MSO incident is at highest priority at Virgin, but we have yet to receive an ETA for resolution.

We are actively chasing Virgin for updates, but despite them acknowledging the issue, they have not yet diagnosed the cause of the problem.
We will continue to apply pressure for an ETA of resolution.

At 11:24, we became aware of a large number of VirginMedia Ethernet leased lines going out of service.
Upon investigation it has also become apparent some areas of TTB Ethernet are affected.
This is presently affecting customers in Yorkshire, Derbyshire, Nottinghamshire and Greater Manchester areas.

Virgin are aware, and further updates to follow