Not pointing fingers, but there was a project (maybe another) that uploaded many reports to the same commit. The results were massive reports that crushed our web services. Looking into better techniques to resolve this. Meanwhile, we are limiting uploads to a single commit.
Posted Feb 22, 2017 - 00:28 UTC
Monitoring
Working through the remaining build queue. Thank you all for your patience!
We are patching the server to prevent this issue again in the future.
Posted Feb 22, 2017 - 00:18 UTC
Update
Slowly starting processing agents. Reports are coming through. Large delay.
Posted Feb 22, 2017 - 00:13 UTC
Update
Uploads are still being accepted. Processing is paused.
Posted Feb 21, 2017 - 23:47 UTC
Identified
One or more reports uploaded huge content. We have paused our upload jobs to identify the culprit.
Posted Feb 21, 2017 - 23:46 UTC
Investigating
We are seeing high error rates on web requests associated with database loads.