A crontab command-line issue in our production servers caused the periodic background jobs to fail being triggered. This was a side effect of library upgrades to fix for CVE-2020-36327.

‌As a consequence of the failure, background actions such as daily purging of stale database records, cleanup of log audits, internal PDF reporting and, most importantly, billing did not run during the period between 16/06/2021 and 02/07/2021.

Our notification systems did not catch the failure.

‌On 01/07/2021 at 17:43:58 UTC and 02/07/2021 at 10:10:00 UTC, the execution of background jobs for billing was triggered manually, ensuring events of the billing process that only happen at the beginning of each month were successfully performed.

On 02/07/2010 at 10:04 UTC, we applied a hot fix re-establishing auto-triggering of background jobs by system's crontab to normality. In light of the event, our team has as well accelerated already planned measures of improvement in the system's background jobs triggering mechanism. These latest improvements have been recently finished and were deployed to production on 07/07/2021 at 12:28 UTC.