RCA
Detailed Event Narrative
On the morning of October 29th, the Emea1b environment experienced an issue that caused multiple journeys to get stuck during task transitions. The root cause was the exhaustion of the AWS Step Functions StartExecution API quota due to a high number of concurrent executions triggered by Distributed Maps and Access Layer inheritance calculations. This resulted in blocked journeys as Step functions responsible for task pre-processing were paused while waiting for available quota.
TimeLine:
29/10/2025
9:07 AM: Raised P2 alarm from stuck journeys monitor
10:55 AM: Root cause for the issue was found
11:30 AM: No more journeys are getting stuck
11:43 AM: issue resolved
Root Cause Analysis
Exhaustion of AWS Step Functions StartExecution quota caused by high concurrency.
Impact
Journeys were unable to start or stuck in transitions between tasks
Resolution
The issue was resolved by stopping two busy Step Functions that were consuming the majority of the StartExecution quota. This immediately released quota capacity and allowed other Step Functions to proceed normally. No rollback or code deployment was needed.
Corrective Actions:
Action Items
Create an additional P1 alarm for large number of stuck journeys detected.
Move into the new Access layers inheritance logic that is not using any Step functions quota.