Previous incidents

The software update helped to rectify the situation and at the moment the load on the software part has been reduced to a minimum.

However, we are still considering replacing the Juniper stack with another one in the near future.

After a more thorough inspection, our engineer identified a faulty motherboard. It has been replaced, and the server is currently active — no recurrence is expected.

The faulty motherboard will be sent to the vendor for further analysis to help prevent similar incidents in the future.

Our monitoring systems recorded a sharp increase in the w_await indicator on some servers of the RX cluster. This indicator reflects the response time of NVMe drives during write operations.

Since we use RAID1 on our servers to increase the reliability of data storage, its speed depends on the "slowest disk" itself. If at least one disk from the array starts working incorrectly, this is reflected in the entire array.

During an internal investigation, we found that all the problematic drives...

Since we did not receive a clear response from CloudFlare, what exactly is the reason for this incident, we decided to replace the IP addresses with others. The replacement was successful, this subnet has been decommissioned, and customers have no problems working with the CloudFlare network.

The issue affecting system performance due to a bug in the BillManager software has been resolved. The abnormal CPU usage has returned to normal levels, and service response times have stabilized. We continue to monitor the situation closely.

The root cause has been identified as a microcode bug in BillManager, and the incident is under further analysis by ISPsystem. No further impact is expected at this time.

We apologize for the inconvenience caused and thank you for your understanding.

At the moment, the incident has been resolved, no repeats are expected.