Root Cause Analysis – EMEA1 Service Degradation
Summary
On May 5, 2026, Fenergo experienced a service degradation in the EMEA1 production environment during a planned maintenance activity. The maintenance was part of a standard automated rolling update process. During the update, an underlying data store did not complete a recovery transition as expected, resulting in a temporary period of degraded availability for services dependent on write operations.
Service stability was restored at 12:53 PM CEST.
Type of Event
Unplanned service degradation during planned maintenance.
Customer Impact
Customers in EMEA1 may have experienced intermittent failures, delays, or temporary unavailability for affected services during the incident window.
Root Cause
The incident occurred during a planned automated update to an underlying data store. As part of the update process, the system was expected to transition safely between nodes. This transition did not complete as expected, causing the data store to enter a degraded recovery state. During this period, some application services were temporarily unable to complete write-dependent operations. A contributing factor was that the automated maintenance process did not stop early enough when the transition did not complete successfully.
Corrective Actions
Fenergo is implementing the following improvements:
Corrective Action | Target |
Enhanced maintenance controls: strengthen automated safeguards so maintenance activities pause or stop if expected readiness checks do not pass. | End of May 2026 |
Improved service validation: expand health and readiness checks before, during, and after maintenance activities. | End of May 2026 |
Resilience and scheduling review: review update sequencing, timing, and underlying resilience controls to further reduce customer impact. | End of May 2026 |
Closing Summary
This incident was triggered during a planned maintenance activity when an underlying data store did not complete a recovery transition as expected.
Fenergo has identified the contributing factors and is implementing additional safeguards to improve maintenance controls, readiness validation, and resilience during future updates.
Preliminary RCA
During a planned platform update today, an underlying data store experienced an unexpected restart event. As a safeguard to protect data consistency, the application temporarily entered read-only mode while the service recovered. Normal service has now been restored, and we are completing a detailed review of the update process and recovery behaviour to identify any further preventive actions.
Full RCA will be provided over coming days
11:50am UK Time
Issue is reporting resolved and Fenergo have tested.
We are actively monitoring to ensure stability
No remediation required.
Preliminary RCA and Full RCA will be provided on this page.
11:30am UK Time
Our team has identified root cause and are actively working to remediate.
We are still estimating resolution by 12.00 UK time
11:00am UK Time
Our team has identified root cause and are actively working to remediate.
We are still estimating resolution by 11.30 UK time
10:30am UK Time
Fenergo monitoring has alerted us to an issue with the Production Tenants on EMEA1
This is resulting in service degradation and multiple errors
Our team has identified root cause and are actively working to remediate.
Estimating resolution by 11.30 UK time
Next Update at 11.00 UK Time