18.10pm UK time
Incident Summary
All tenants experienced a period of degraded service, with the platform operating in read-only mode. The issue was caused by a failure in an automated credential rotation process for a backend database component.
Root Cause
Our platform performs regular automated rotation of database credentials as part of our security posture. During a recent rotation cycle, a failure occurred in one step of the process — specifically, the updated credentials were not propagated correctly — resulting in the application being unable to establish a full read/write connection to the database. The system defaulted to read-only mode, preserving data integrity throughout the incident.
Resolution
The credential rotation process was completed manually, restoring full connectivity and read/write access for all tenants.
Data Integrity
No data loss occurred. The read-only failsafe behaviour performed as intended, protecting the integrity of all tenant data throughout the incident.
Contributing Factors
This automated rotation process has been running successfully for over a year. The failure represents an isolated deviation from expected behaviour, the precise cause of which is under investigation.
Next Steps
A full review of the rotation process logs is underway to identify the specific point of failure.
The automated solution will be reviewed and hardened to prevent recurrence.
Additional monitoring and alerting will be evaluated to ensure any future failures in this process are detected and remediated more rapidly.
16.10pm UK Time
Issue has been resolved and all Fenergo monitoring is reporting healthy systems. There has been no data loss or corruption.
Preliminary RCA will be provided within next 24hrs on this incident page with Full RCA to be provided shortly afterwards
16.00pm UK Time
Root cause has been identified and Fenergo teams are working hard to resolve.
No ETA can be shared at this moment as we continue our investigations and resolution.
Next Update: 16:30pm UK
15:20pm UK Time
Fenergo monitoring has alerted us to an issue with the EventStore service in the APAC 2 region within AWS. Command API's are currently not working which would result in customers being unable to progress journeys or make calls to command API's.
Application is up however is in Read-Only mode,
Our team has identified root cause and are actively working to remediate.