Everything looks better now. We're going to close the incident while we continue to investigate the root cause.
Posted Feb 12, 2020 - 11:41 PST
Update
We were still seeing errors, so we're turning off the optimizations again.
Posted Feb 12, 2020 - 11:24 PST
Monitoring
403 errors are back below 0.5%, but as predicted, latencies are up. We're going to slowly bring back some optimizations.
Posted Feb 12, 2020 - 08:43 PST
Update
Seeing spikes up to 9% of requests, from a baseline of 0.5-1%. Rolling back to an earlier build, and disabling experimental optimizations. Bigger accounts may experience greater latencies.
Posted Feb 12, 2020 - 08:03 PST
Investigating
Investigating reports of sporadic 403 errors for valid tokens, and possible slowness.