Root Cause Analysis
Incident Overview
On April 9, Fenergo experienced a high-severity incident impacting association queries and related processing in the EMEA1B environment.
The issue affected non-production (SDLC) tenants.
Customer Impact
Between approximately 09:30 UTC and 11:25 UTC, customers may have experienced:
Delays or failures in association queries
Intermittent errors in journey processing
Degraded system performance
Service was fully restored once the issue was resolved.
Root Cause
The incident was caused by unexpectedly high system load.
A large volume of activity from multiple tenants occurred at the same time, exceeding system capacity. Existing retry behaviour and connection handling increased system pressure during this period, prolonging the impact.
Resolution
The issue resolved once system load reduced and stability was restored.
Actions taken included:
Scaling infrastructure
Increasing system capacity
Introducing safeguards to limit traffic spikes
Corrective Actions
We are implementing improvements to prevent recurrence, including:
Enhancing retry logic with controlled backoff
Optimising system performance under load
Strengthening connection and capacity management
Improving monitoring and alerting
Summary
This incident was driven by unexpected load exceeding system capacity.
We have stabilised the platform and are implementing further improvements to ensure greater resilience and consistent performance going forward.
Preliminary RCA
This incident was driven by unusually high activity in a small number of client test environments, which placed pressure on shared platform resources and was further amplified by application connection handling under load. We have since strengthened platform resilience through both infrastructure upgrades and application-level improvements. Further details will be shared as part of the formal RCA.
14.45 UK Time
Fenergo monitoring is showing system stability in non production tenants in EMEA1B.
RCAs will be provided on this page over the coming days
13.45 UK Time
Please be aware that Non Production tenants are still recovering and may show continued degradation as we process events from earlier in the day.
12.10 UK Time
Fenergo have resolved issues on Non-Production tenants. We are currently monitoring to ensure system stability
12.00pm UK Time
Fenergo have identified root cause and have begun the restoration of service.
ETA as yet not fully confirmed but will advise on this channel.
Next update 12.30 UK Time
11:15am UK Time
Fenergo monitoring has alerted us to an issue with the Non-Production Tenants on EMEA1B
This is resulting in service degradation and multiple errors
Our team has identified root cause and are actively working to remediate.
Next Update at 12.00 UK Time