Degraded upload and processing performance

Incident Report for Codecov

Resolved

We experienced degraded upload and processing performance due to an issue that impacted auto-scaling our Redis cluster. This issue caused the cluster to not scale up automatically to allocate more RAM for processing jobs. Since it did not scale, some previous jobs were evicted / new jobs could not be created, leading to processing issues for commits uploaded between 15:00 UTC and 18:00 UTC
Processing limits have since moved below the current ceiling on our Redis memory, and we're investigating ways to ensure that auto-scaling doesn't fail in the future.

Additionally, increased processing volume led to network congestion between our bash uploader and upload storage proxies, we will increase the maximum throughput of those proxies to ensure the issue with storage does not happen again

Posted Sep 30, 2019 - 15:00 UTC