On May 26, 2021, an outage with a third-party tool caused Business Intelligence Connector (BIC) ETLs to not complete. To resolve the failure, some customers were moved to a slower legacy ETL system, which caused delays. We returned to the new system before the third-party tool was fully fixed, which caused impartial data to be delivered on June 1st and June 2nd.
May 26, 2021
13:19 UTC: Greenhouse team notices that BIC ETLs will not complete on time due to a third-party tool outage.
19:31 UTC: Greenhouse moves ETLs to a legacy system to resolve the issue.
May 28, 2021
19:40 UTC: Third-party notifies Greenhouse that the issue is resolved.
20:40 UTC: Greenhouse restores the new ETLs.
June 1, 2021
20:01 UTC: Greenhouse team is made aware that Silo 1 customers are missing data in BIC. This was caused by an incomplete resolution to the May 26 outage.
June 2, 2021
02:53 UTC: Greenhouse moves Silo 1 customers to the legacy ETL to mitigate impact.
12:00 UTC: Greenhouse team is made aware of delivery delays on Silo 2, 3, and 4 with 23 out of 71 tables still in progress.
20:13 UTC: ETLs on Silo 2, 3, and 4 complete.
June 3, 2021
15:07 UTC: The incident is fully resolved and data delivery is restored.
WHAT WAS THE EFFECT?
On May 26, Silo 1 customers experienced delays with the ETL, resulting in partial data availability.
On June 1, Silo 1 customers experienced incomplete data.
On June 2, Silo 2, 3, and 4 customers experienced delays with the ETL, resulting in partial data availability.
WHAT WAS THE CAUSE?
The cause was an outage from our third-party ETL provider.
WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?
If you have any questions or concerns, please reach out via our support center.