04 December 2025
RCA
Detailed Event Narrative
On the morning of December 3rd, the Portal service in the EMEA1b environment began intermittently timing out for some customers. Our engineering teams responded immediately, identified the cause, and implemented corrective measures. The system returned to normal operation within approximately four hours, and customer impact remained limited to Portal.
Event Timeline
Dec 3rd: 09.19am
Dec 3rd: 01.35pm
Services Impacted
Portal
Root Cause Analysis
Outdated internal test data created conditions that triggered an unusually large number of requests to be made, resulting in timeouts & throttling
Impact
Portal onboarding was impacting with many onboarding journeys unable to initiate or proceed.
Corrective Actions
We introduced improvements to internal test-data hygiene to ensure outdated test data does not accumulate - Completed
Modify Portal domains configuration of AWS CloudWatch integration - implement asynchronous publish method for sending client metrics - Completed
----------------------------
Preliminary RCA
Fenergo identified an infrastructure configuration issue in our monitoring services that created contention in a shared component and resulted in sporadic 504 timeouts, temporarily affecting some users in specific portal flows.
03.Dec.2025 @ 14.45 GMT
System has fully recovered and we are no longer seeing any errors.
Full RCA will be published on this page in following days
Fenergo apologise for any inconvenience and appreciate client's patience as we resolved issue
03.Dec.2025 @ 14.00 GMT
System has reported fully recovered for last 25 minutes. We will continue to monitor system to ensure full restoration
Next update at 15.00GMT
03.Dec.2025 @ 13.00 GMT
We believe we have identified and remediated the root cause of the issue. We are seeing a gradual reduction in errors as the system restores.
We are continuing to monitor until we have confidence issue is resolved and sytem is stable.
Next update at 14.00 GMT
03.Dec.2025 @ 12.00 GMT
Issue is still ongoing and our teams are still actively investigating this issue.
We have engaged our infrastructural partner AWS to support in the investigation.
Next update at 13.00 GMT if not sooner
03.Dec.2025 @ 11.00 GMT
Issue is still ongoing. We have identified intermittant errors and seeing Magic Link creation errors also.
Our teams are still investigating trying to determine root cause.
Next update at 12.00 GMT if not sooner
03.Dec.2025 @ 10.00 GMT
Fenergo have identified alerting issue highlighting an ongoing performance degradation on EMEA1b Portals. Our investigations are ongoing and updates will be provided here