Emergency reboot of FPC's in City Reach, London [RESOLVED]

July 1, 2021, 4:11 PM - Europe/London

We will be rebooting a few MPC's in City Reach, London as they are producing errors, high latency and packet loss. Unfortunately we have exhausted all methods to fix this without having to reboot.

During the reboots it may cause higher packet loss and latency for 5-10 minutes.

We apologize for the inconvenience this may cause.

Locations affected
DDCL / City Reach - London, UK
We have resolved the issue. RFO will follow in another update.

July 2, 2021, 10:23 AM - Europe/London

We have brought the new LAG online and the packet loss has now reduced to 0 between rt0 and rt1.

We are sorry for the time this took to be resovled. We will continue investigations as to why there was an issue between routers on compatible hardware.

July 1, 2021, 7:34 PM - Europe/London

A new LAG is being ran between rt0 and rt1 on different cards. We will bring this up as soon as its complete and tear down the old one to avoid a full outage.

Currently you will see fluctuating latency and packet loss.

July 1, 2021, 6:52 PM - Europe/London

Unfortunately the new card did not resolve the issue. We are now replacing MPC's in rt0-cr.

There are no obvious errors on any card and this seems to be extremely obscure.

July 1, 2021, 6:31 PM - Europe/London

New card has been installed and fibre moved over. We will bring the LAG up shortly.

July 1, 2021, 5:23 PM - Europe/London

Ports on the MPC in FPC 3 have been downed a new card is being installed into FPC 4. We will then move the ports over.

July 1, 2021, 4:36 PM - Europe/London

FPC 3 32x10G line card will now be replaced with a new spare unit we have onsite.

July 1, 2021, 4:28 PM - Europe/London

FPC 3 is back online on rt0-cr

July 1, 2021, 4:21 PM - Europe/London

Restart commencing on FPC 11 on rt1-cr and FPC 3 on rt0-cr

July 1, 2021, 4:16 PM - Europe/London

back to all