Previous incidents

Our monitoring systems recorded a sharp increase in the w_await indicator on some servers of the RX cluster. This indicator reflects the response time of NVMe drives during write operations.

Since we use RAID1 on our servers to increase the reliability of data storage, its speed depends on the "slowest disk" itself. If at least one disk from the array starts working incorrectly, this is reflected in the entire array.

During an internal investigation, we found that all the problematic drives...

Since we did not receive a clear response from CloudFlare, what exactly is the reason for this incident, we decided to replace the IP addresses with others. The replacement was successful, this subnet has been decommissioned, and customers have no problems working with the CloudFlare network.

The issue affecting system performance due to a bug in the BillManager software has been resolved. The abnormal CPU usage has returned to normal levels, and service response times have stabilized. We continue to monitor the situation closely.

The root cause has been identified as a microcode bug in BillManager, and the incident is under further analysis by ISPsystem. No further impact is expected at this time.

We apologize for the inconvenience caused and thank you for your understanding.

At the moment, the incident has been resolved, no repeats are expected.