February 5, 2021, 10:31 PM - Europe/London
Hello,
We're currently investigating a fault with the rdp process on a couple of our Juniper routers that had failed at approximately 22:00 GMT this evening.
The cause of this is currently unknown but this did cause a cascade of problems to neighbouring routers with high volumes of BGP updates being sent out.
We're currently reviewing the log files from these routers and will provide more information as soon as possible.
Apologies for any inconvenience caused, and thank you for your patience.
- HydraCom NOC
Locations affected
DDCL / City Reach - London, UKTelehouse North Two - London, UKInterxion PAR-5 - Paris, FR
- Resolved
- Yesterday (05/02/2021) at 21:55 PM GMT the routing process on 3 routers (rt0-par5, rt0-thn2 and rt1-cr) crashed and core dumped before restarting and re-initialising all routing protocol services. This subsequently caused connection issues for a large portion of customers for 10-20 minutes.
After further investigation, the issue appears to have been caused by a prefix-list being replaced with the same name but using a different address family (inet6 to inet). Even with the commit passing checks in it's current state and there being nothing wrong with the configuration, it appears to have identified a potential bug with Junos which caused the routing sub system to crash.
We'd like to try and re-create the scenario on a router not in production to see if the same thing occurs to be 100% sure, but from what we've observed within the system log files we're confident this was the trigger.February 6, 2021, 7:45 PM - Europe/London