On July 24, 2017, we experienced an elevated number of errors in our Service Management API. The outage was caused by an automated database failover which has failed during a maintenance task, corrupting the data on one of the shards in our database cluster.

To reestablish the service our team had to restore the database from a backup file. Due to the size of the backup file, the restore operation has taken several minutes to be accomplished.

After the backup restoration, the database has been successfully restored, and all the errors in our Service Management API have been resolved.