On October 9th of 2019 we had a network partition on our AWS infrastructure. This triggered a failover on one shard of our data layer infrastructure at 7:59 UTC without issues. This layer is composed of different shards, and normally the failover occurs without any downtime. However in that case, 15 minutes after the failover (at 8:14 UTC), due to a memory issue, new shard master started to synchronize again from disk to memory, and during the next 7 minutes we had a downtime on Service Mamangement API, backend was not accepting requests, returning 5XX respondes code because of data layer issue.

Timeline:

• Oct 9, 2019 07:59 UTC - network partition on one shard, master failover.
• Oct 9, 2019 08:14 UTC - 15 minutes after failover (new shard master was working OK), new shard master synchronized again from disk to memory.
• Oct 9, 2019 08:21 UTC - new shard master full synchronized again (on memory).

SLA:

• Oct 9, 2019 08:14 UTC - Oct 9, 2019 08:21 UTC - backend not accepting requests - 5XX response codes returned because of data layer issue.
• Oct 9, 2019 08:21 UTC - back to 100% traffic.

Root cause:

The initial trigger was a network partition failure in the AWS infrastructure which led to failover of one of the server instances in our database system.

Prevention actions:

Review failover process to shorten failover time.