tag:status.3scale.net,2005:/historyRed Hat 3scale Status - Incident History2024-03-29T08:52:43+01:00Red Hat 3scaletag:status.3scale.net,2005:Incident/198593372024-02-13T11:20:59+01:002024-02-13T11:21:00+01:00Database maintenance tasks<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>11:20</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>11:14</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>11:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>17:28</var> CET</small><br><strong>Scheduled</strong> - We want to make you aware that on Tuesday, February 13th, between 10:00 UTC and 11:00 UTC there will be a scheduled service disruption of about 15 minutes due to database maintenance.<br /><br />To ensure uptime and stability, we are performing this upgrade in our infrastructure during the less busy period of the platform.<br /><br />The affected services are:<br />- Admin Portal UI<br />- Developer Portal UI<br />- Account Management API, Analytics API, Billing API<br />- ActiveDocs Proxy<br /><br />We appreciate your patience and understanding. As always, if you have questions or issues, please contact our support team.</p>tag:status.3scale.net,2005:Incident/190339372023-11-20T11:30:37+01:002023-11-20T11:30:37+01:00Database maintenance tasks<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>11:30</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>11:26</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>11:02</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>13:02</var> CET</small><br><strong>Scheduled</strong> - We want to make you aware that on Monday, November 20th, between 10:00 UTC and 11:00 UTC there will be a scheduled service disruption of about 15 minutes due to database maintenance.<br /><br />To ensure uptime and stability, we are performing this upgrade in our infrastructure during the less busy period of the platform.<br /><br />The affected services are:<br />- Admin Portal UI<br />- Developer Portal UI<br />- Account Management API, Analytics API, Billing API<br />- ActiveDocs Proxy<br /><br />We appreciate your patience and understanding. As always, if you have questions or issues, please contact our support team.</p>tag:status.3scale.net,2005:Incident/190447482023-11-07T12:41:24+01:002023-11-07T17:43:50+01:00We are currently experiencing a service disruption in our Service Management API<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>12:41</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved, a postmortem will be available shortly.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>12:06</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/173499722023-05-24T11:29:41+02:002023-05-24T16:37:39+02:00Service disruption affecting all APIs<p><small>May <var data-var='date'>24</var>, <var data-var='time'>11:29</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>11:07</var> CEST</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/173499712023-05-24T11:06:57+02:002023-05-24T11:06:58+02:00We are currently experiencing a service disruption in our Echo API<p><small>May <var data-var='date'>24</var>, <var data-var='time'>11:06</var> CEST</small><br><strong>Resolved</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/160912582023-02-10T16:21:54+01:002023-02-10T16:21:54+01:00We are currently experiencing a service disruption in our Echo API<p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>16:21</var> CET</small><br><strong>Resolved</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/160912492023-02-10T16:21:24+01:002023-02-10T16:21:24+01:00We are currently experiencing a service disruption in our Echo API<p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>16:21</var> CET</small><br><strong>Resolved</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/111422142022-09-29T07:40:02+02:002022-09-29T07:40:03+02:00Database maintenance tasks<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>07:40</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>07:21</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>07:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>17:03</var> CEST</small><br><strong>Scheduled</strong> - We want to make you aware that on Thursday, September 29th, between 5:00 UTC and 6:00 UTC there will be a scheduled service disruption of about 15 minutes due to database maintenance.<br /><br />To ensure uptime and stability, we are performing this upgrade in our infrastructure during the less busy period of the platform.<br /><br />The affected services are:<br />- Admin Portal UI<br />- Developer Portal UI<br />- Account Management API, Analytics API, Billing API<br />- ActiveDocs Proxy<br /><br />We appreciate your patience and understanding. As always, if you have questions or issues, please contact our support team.</p>tag:status.3scale.net,2005:Incident/112985392022-09-28T13:32:57+02:002022-09-28T13:32:57+02:00We are currently experiencing a service disruption in our 3scale APIs (Account Management API, Analytics API, Billing API)<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>13:32</var> CEST</small><br><strong>Resolved</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/101149542022-05-30T12:09:00+02:002022-05-31T15:43:49+02:00Account Management API, Analytics API and Billing API - Service disruption<p><small>May <var data-var='date'>30</var>, <var data-var='time'>12:09</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved, details can be found in the Postmortem.</p>tag:status.3scale.net,2005:Incident/89119752021-12-22T14:49:22+01:002021-12-22T18:49:15+01:00Service Management API - Service disruption<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>14:49</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.<br /><br />A postmortem will be published soon.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>14:06</var> CET</small><br><strong>Update</strong> - The service affectation is related to an ongoing AWS outage. We are working on a mitigation.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>13:12</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/85491652021-12-14T08:36:54+01:002021-12-14T08:36:55+01:00Database maintenance tasks<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>08:36</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>08:29</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>08:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>12:19</var> CET</small><br><strong>Update</strong> - UPDATE: The maintenance window on December 14th has been moved two hours earlier than initially scheduled. The window will start at 7:00 UTC and end at 9:15 UTC.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>11:52</var> CET</small><br><strong>Scheduled</strong> - We want to make you aware that on Tuesday, December 14th, between 7:00 UTC and 9:15 UTC there will be a scheduled service disruption of about 45 minutes due to database maintenance.<br /><br />To ensure uptime and stability, we are performing this upgrade in our infrastructure during the less busy period of the platform.<br /><br />The affected services are:<br />- Admin Portal UI<br />- Developer Portal UI<br />- Account Management API, Analytics API, Billing API<br />- ActiveDocs Proxy<br /><br />We appreciate your patience and understanding. As always, if you have questions or issues, please contact our support team.</p>tag:status.3scale.net,2005:Incident/84211662021-11-08T11:33:28+01:002021-11-08T16:21:32+01:00Service Management API - Service disruption<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:33</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.<br /><br />Several incidents were opened and closed during the service disruption due to the fact that our status page is updated automatically based on HTTP probe results in order to give users quick feedback on the status of the service. As a 10% of the requests were still being correctly processed, some probes were successful during the incident, causing the status page incidents to be resolved.<br /><br />A postmortem will be published soon.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:55</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/81130772021-09-29T19:09:10+02:002021-10-08T13:21:49+02:00Portal UI, Account Management API, Analytics API and Billing API - Intermittent service disruption<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>19:09</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved. The postmortem will be published shortly.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>18:15</var> CEST</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/75263282021-07-15T10:00:00+02:002021-07-19T15:53:58+02:00Service Management API - Degraded service<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved, details can be found in the Postmortem.</p>tag:status.3scale.net,2005:Incident/74350202021-07-01T19:30:00+02:002021-07-07T16:00:05+02:00Billing jobs not executed<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>19:30</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved, details can be found in the Postmortem.</p>tag:status.3scale.net,2005:Incident/73548132021-06-28T21:48:35+02:002021-06-28T21:48:35+02:00We are currently experiencing a service disruption in our 3scale APIs (Account Management API, Analytics API, Billing API)<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>21:48</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>21:47</var> CEST</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/72703162021-06-17T12:28:23+02:002021-06-17T13:02:46+02:00We are currently experiencing a service disruption in our Admin Portal UI<p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>12:28</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>17</var>, <var data-var='time'>12:21</var> CEST</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/64354852021-03-02T14:05:49+01:002021-03-02T14:53:25+01:00We are currently experiencing a service disruption in our Echo API<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>14:05</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>14:04</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/62809122021-02-16T06:08:58+01:002021-02-18T18:54:46+01:00We are currently experiencing a service disruption in our ActiveDocs Proxy<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>06:08</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>06:08</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/62664642021-02-14T12:52:55+01:002021-02-18T18:54:15+01:00We are currently experiencing a service disruption in our ActiveDocs Proxy<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>12:52</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>12:52</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/59528692021-01-05T15:23:45+01:002021-01-05T15:23:46+01:00We are currently experiencing a service disruption in our Echo API<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:23</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>15:23</var> CET</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/50771842020-09-12T22:38:19+02:002020-09-15T12:49:10+02:00We are currently experiencing a service disruption in our 3scale APIs (Account Management API, Analytics API, Billing API)<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>22:38</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>22:37</var> CEST</small><br><strong>Investigating</strong> - Our operations team is working to identify the root cause and implement a solution.</p>tag:status.3scale.net,2005:Incident/47690812020-08-02T02:30:00+02:002020-08-03T11:53:37+02:00Service Management API - Degraded performance<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>02:30</var> CEST</small><br><strong>Resolved</strong> - This Sunday, August 2nd, at 00:30 AM UTC we detected a latency spike caused by a number of requests timing out before completion. The root cause was a cloud provider hardware failure and was resolved at 01:00 AM UTC. We are working on improving the resilience of the affected layer to avoid further affectations and improved the documentation to enable a faster reaction.</p>tag:status.3scale.net,2005:Incident/46388842020-07-29T11:11:56+02:002020-07-29T11:11:56+02:00Database maintenance tasks<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>11:11</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>10:54</var> CEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>10:51</var> CEST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>10:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>11:06</var> CEST</small><br><strong>Scheduled</strong> - We want to make you aware that on Wednesday, July 29th, between 8:00 UTC and 09:15 UTC there will be a scheduled service disruption of about 45 minutes due to a database maintenance.<br /><br />To ensure uptime and stability of the platform, we are performing this upgrade in our infrastructure during​ the less busy period of the platform.<br /><br />The affected services are: <br />- Admin Portal UI <br />- Developer Portal UI <br />- 3scale APIs (Account Management API, Analytics API, Billing API)<br />- ActiveDocs Proxy<br /><br />We appreciate your patience and understanding. As always, if you have questions or issues, please contact our support team.</p>