AWS’s Tokyo fault ran 5 hours 45 minutes; some launches ‘succeeded’, then terminated

01A launch that said yes, then terminated

Auto Scaling, ECS and RDS, which start instances through the same EC2 API, were affected, and ECS and Fargate tasks failed to launch.¹ The status checks were unreliable too: EC2 and EBS health data went missing or arrived late, CloudWatch alarms for the zone moved to INSUFFICIENT_DATA, and AWS told customers to “rely on your own health telemetry”, warning that alarms set to fire on missing data might do so.¹

02The second fault: network changes that did not arrive

Once the API recovered, AWS found that new instances, and running instances whose network configuration was modified, had intermittent connectivity. It traced this to delays in an internal subsystem that propagates instance networking changes, “related to the underlying root cause”. ECS, EKS, Elastic Load Balancing and NAT Gateway were hit, along with new clusters, gateways and VPC gateway endpoints. AWS asked customers not to attach or detach network interfaces or gateways, or change VPC peering, in the zone. Traffic to existing resources and between instances was not affected.¹

03What running in several zones did and did not cover

Spreading across zones protected what was already running: existing instances were not affected, and AWS pointed new launches at the two healthy zones. It did not cover everything. The API errors were reported for the region, and in its 16:40 UTC post AWS saw low levels of launch failures in other zones as well.¹ And apne1-az1 is a zone ID: AWS maps zone names such as ap-northeast-1a to physical zones differently for each account, so a team has to look up its own mapping to know which of its subnets sat in the affected zone.² Google’s six-hour London outage showed a related limit: Google named “control-plane dependencies across zonal boundaries” among its causes.³

AWS has not published a root cause or said how many customers were affected.