Administration UI Service Degradation
Incident Report for Red Hat 3scale
Postmortem

Due to a specific API call over the 3scale API, some long running query's has been generated in the Administration Portal Database, which, was blocking a table thus blocking some of the further requests, causing intermittent issues in our 3scale APIS, Developer Portal UI and Admin portal.

The affected components were:

  • Admin Portal UI
  • Developer Portal UI
  • 3scale APIs (Account Management API, Analytics API, Billing API)

Preventative Measures:

We have identified the affected tables and applied the needed corrections on the queries to mitigate such degradations in the future.

Summary:

  • Incident started at Jan 23, 2018 - 16:05 UTC
  • Incident resolved on Jan 23, 2018 - 18:25 UTC
Posted Jan 24, 2018 - 16:49 CET
Resolved
The incident has been resolved.

Summary:
* Incident started at: Jan 23, 2018 - 17:05 CET
* Incident resolved at: Jan 23, 2018 - 19:25 CET
Posted Jan 23, 2018 - 19:25 CET
Monitoring
A fix has been implemented and our team is monitoring the results.
Posted Jan 23, 2018 - 19:11 CET
Investigating
Our operations team is working to identify the root cause and implement a solution.
Posted Jan 23, 2018 - 18:43 CET
Monitoring
A fix has been implemented and our team is monitoring the results.
Posted Jan 23, 2018 - 17:43 CET
Identified
The issue has been identified, and our team is working on a fix.
Posted Jan 23, 2018 - 17:05 CET
← Current Status Powered by Atlassian Statuspage