SQL queries backed up which crippled the system. The first query to cause a snowball effect concerned retrieving large report data.
We reviewed logs and watched queries to identify the culprit repositories. This was, however, painful because the log details were not specific enough to query properly.
Once we identified the culprit repositories we first, momentarily, blocked the uploading and processing of reports. We then changed the storage strategy to utilize a new scaling technique we have been working on.
We appreciate the love and respect from the community. Above all your patience humbles us.
Thank you for the #hugops :)
<3 The Codecov Team