Emergency reboot of FPC's in City Reach, London [RESOLVED]

July 1, 2021, 4:11 PM - Europe/London

We will be rebooting a few MPC's in City Reach, London as they are producing errors, high latency and packet loss. Unfortunately we have exhausted all methods to fix this without having to reboot.

During the reboots it may cause higher packet loss and latency for 5-10 minutes.

We apologize for the inconvenience this may cause.

Locations affected
DDCL / City Reach - London, UK
Resolved
We have resolved the issue. RFO will follow in another update.

July 2, 2021, 10:23 AM - Europe/London

Update
We have brought the new LAG online and the packet loss has now reduced to 0 between rt0 and rt1.

We are sorry for the time this took to be resovled. We will continue investigations as to why there was an issue between routers on compatible hardware.

July 1, 2021, 7:34 PM - Europe/London

Update
A new LAG is being ran between rt0 and rt1 on different cards. We will bring this up as soon as its complete and tear down the old one to avoid a full outage.

Currently you will see fluctuating latency and packet loss.

July 1, 2021, 6:52 PM - Europe/London

Update
Unfortunately the new card did not resolve the issue. We are now replacing MPC's in rt0-cr.

There are no obvious errors on any card and this seems to be extremely obscure.

July 1, 2021, 6:31 PM - Europe/London

Update
New card has been installed and fibre moved over. We will bring the LAG up shortly.

July 1, 2021, 5:23 PM - Europe/London

Update
Ports on the MPC in FPC 3 have been downed a new card is being installed into FPC 4. We will then move the ports over.

July 1, 2021, 4:36 PM - Europe/London

Update
FPC 3 32x10G line card will now be replaced with a new spare unit we have onsite.

July 1, 2021, 4:28 PM - Europe/London

Update
FPC 3 is back online on rt0-cr

July 1, 2021, 4:21 PM - Europe/London

Update
Restart commencing on FPC 11 on rt1-cr and FPC 3 on rt0-cr

July 1, 2021, 4:16 PM - Europe/London

back to all