SSL Errors when Uploading Reports
Incident Report for Codecov
Postmortem

The core issue was caused by the expiry of two intermediate certs in our cert chain. The root and end were valid, but the intermediate certs were invalid. These intermediate certs expire every 20 years (see: https://news.ycombinator.com/item?id=23360624 and https://twitter.com/dnsimple/status/1266729739915591681).

Since an intermediate cert failed, SSL still appeared to be functioning properly in web browsers (Chrome loaded https://codecov.io just fine), but other applications -- such as cURL, which our bash uploader depends on -- did not (see: https://github.com/codecov/codecov-bash/issues/309).

Many of codecov's uploaders rely on cURL in some form or another, thus the expiry of our intermediate certs led to many of uploaders malfunctioning, and users who attempted to upload coverage reports at this time were very likely not able to do so. We expect that this issue lasted from 12:01a CDT on June 1, 2020 to approximately 2:30p CDT on June 1 2020.

We resolved this issue by removing the expired intermediate certs from our chain and updating our certificate usage in our production environment accordingly. Therefore, no users should be experiencing SSL related errors or issues when uploading to codecov. If you are experiencing this or any other error, please let us know.

Posted May 30, 2020 - 19:42 UTC

Resolved
This issue has been identified and a fix has been pushed.

The core issue was caused by the expiry of two intermediate certs in our cert chain. The root and end were valid, but the intermediate certs were invalid. These intermediate certs expire every 20 years (see: https://news.ycombinator.com/item?id=23360624 and https://twitter.com/dnsimple/status/1266729739915591681).
Posted May 30, 2020 - 19:41 UTC
Update
We are continuing to investigate this issue.
Posted May 30, 2020 - 18:27 UTC
Investigating
We are seeing some issues with reports uploading due to SSL errors and we are currently investigating.
Posted May 30, 2020 - 17:43 UTC
This incident affected: Frontend.