At 4AM EDT this morning, we were alerted to high analysis latency. This alert resolved quickly, but after investigation we discovered that it was the symptom of a long-running incident that we were not aware of: a portion of our backend services had stopped running on Friday afternoon. Due to a variety of monitoring gaps, we were not aware of this until the issue corrected and the delayed work ultimately triggered our latency alarms.
Because of this, a percentage of customer analysis builds triggered between 2PM EDT Friday and 4AM EDT Sunday did not start and were eventually cancelled. We will identify and notify impacted customers on Monday. We apologize for this inconvenience, we've already corrected some of the monitoring gaps that led to this incident and will perform an internal analysis to identify and correct more.
Jun 18, 10:32 EDT
