WHAT HAPPENED?
On May 26, 2021, an outage with a third-party tool caused Business Intelligence Connector (BIC) ETLs to not complete. To resolve the failure, some customers were moved to a slower legacy ETL system, which caused delays. We returned to the new system before the third-party tool was fully fixed, which caused impartial data to be delivered on June 1st and June 2nd.
TIMELINE
May 26, 2021
13:19 UTC: Greenhouse team notices that BIC ETLs will not complete on time due to a third-party tool outage.
19:31 UTC: Greenhouse moves ETLs to a legacy system to resolve the issue.
May 28, 2021
19:40 UTC: Third-party notifies Greenhouse that the issue is resolved.
20:40 UTC: Greenhouse restores the new ETLs.
June 1, 2021
20:01 UTC: Greenhouse team is made aware that Silo 1 customers are missing data in BIC. This was caused by an incomplete resolution to the May 26 outage.
June 2, 2021
02:53 UTC: Greenhouse moves Silo 1 customers to the legacy ETL to mitigate impact.
12:00 UTC: Greenhouse team is made aware of delivery delays on Silo 2, 3, and 4 with 23 out of 71 tables still in progress.
20:13 UTC: ETLs on Silo 2, 3, and 4 complete.
June 3, 2021
15:07 UTC: The incident is fully resolved and data delivery is restored.
WHAT WAS THE EFFECT?
On May 26, Silo 1 customers experienced delays with the ETL, resulting in partial data availability.
On June 1, Silo 1 customers experienced incomplete data.
On June 2, Silo 2, 3, and 4 customers experienced delays with the ETL, resulting in partial data availability.
Here’s how to check which Silo your organization is on.
WHAT WAS THE CAUSE?
The cause was an outage from our third-party ETL provider.
WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?
If you have any questions or concerns, please reach out via our support center.