We are currently experiencing a service disruption in our 3scale APIs (Account Management API, Analytics API, Billing API)
Incident Report for Red Hat 3scale
Postmortem

On Feb 28th, 2019 a partial outage of 13 minutes on 3scale APIs. Service Management API for auth services of API traffic was not affected. This was due to missing pagination on the 3scale Management API.

Timeline:

Feb 28, 2019 08:41:11 UTC - 3scale APIs not accepting requests - 5XX response code returned

Feb 28, 2019 08:44:11 UTC - 3scale APIs back to 100% traffic after 3 minutes

Feb 28, 2019 09:01:11 UTC - 3scale APIs not accepting requests - 5XX response code returned

Feb 28, 2019 09:05:11 UTC - 3scale APIs back to 100% traffic after 4 minutes

Feb 28, 2019 09:11:11 UTC - 3scale APIs not accepting requests - 5XX response code returned

Feb 28, 2019 09:14:11 UTC - 3scale APIs back to 100% traffic after 3 minutes

Feb 28, 2019 09:22:11 UTC - 3scale APIs not accepting requests - 5XX response code returned

Feb 28, 2019 09:25:11 UTC - 3scale APIs back to 100% traffic after 3 minutes

Root cause:

A call to 3scale Management API is loading all usage limits of the application plan. There is no pagination in this API so rendering the response can be really slow. This blocks the web servers and under a high load makes other requests fail.

Preventive Actions:

Issue opened to add pagination to this API.

Posted 3 months ago. Feb 28, 2019 - 13:18 CET

Resolved
This incident has been resolved.
Posted 3 months ago. Feb 28, 2019 - 11:15 CET
Update
We are continuing to monitor for any further issues.
Posted 3 months ago. Feb 28, 2019 - 11:01 CET
Monitoring
Service is back to normal operation, we keep monitoring.
Posted 3 months ago. Feb 28, 2019 - 10:52 CET
Identified
The issue has been identified.
Posted 3 months ago. Feb 28, 2019 - 10:44 CET
Investigating
Our operations team is working to identify the root cause and implement a solution.
Posted 3 months ago. Feb 28, 2019 - 09:49 CET
This incident affected: Account Management API, Analytics API, Billing API.