SQL demand high
Incident Report for Codecov
Postmortem

For weeks we have had a very steady database load. See graphic here.

Boom! We had a sudden spike in database loads that caused the system to cripple.

To mitigate the issue we decided to throttle the backend jobs to allow all web traffic to be received. This caused a large queue of reports. These reports have now been processed, but took hours to complete. See graphic here.

We have a much larger database ready for promotion. A scheduled emergency event will occur to promote the database.

Thank you all for your patience! Codecov is growing rapidly: hiring new staff, growing to new servers, and taking on new bold features. We are dedicated to you - thank you for your time :)

Best, Steve

Posted over 1 year ago. Jun 29, 2017 - 06:33 UTC

Resolved
This incident has been resolved.
Posted over 1 year ago. Jun 29, 2017 - 06:22 UTC
Update
Continuing to work on an immediate database migration. Web service is performing well. Backend processing is throttled.
Posted over 1 year ago. Jun 28, 2017 - 19:28 UTC
Update
We are scaling out backend process to accept frontend traffic. Expect report delays.
Posted over 1 year ago. Jun 28, 2017 - 17:36 UTC
Identified
We are upgrading our database to large machine. Thank you for your patience.
Posted over 1 year ago. Jun 28, 2017 - 16:58 UTC
Investigating
We are working through a high demand on our SQL server.
Posted over 1 year ago. Jun 28, 2017 - 16:47 UTC