BIC: Delayed Processing Of Tables

Incident Report for Greenhouse

Postmortem

Greenhouse Business Intelligence Connector Postmortem

WHAT HAPPENED?

Since January 1, 2022, we have missed the 9am EST BIC data delivery time 15 times with most of those delays concentrated in two windows of time:

March 19 - April 13 (8 delayed days)
June 30 - July 13 (5 delayed days)

WHAT WAS THE CAUSE?

The March/April incidents were the result of occasional issues with the processing of the data by our third-party service provider. When these issues occur, it requires human intervention to ensure that the process completes successfully.

The June 30th, July 2nd, and July 7th incidents were caused by additional issues with database resources that we did not resolve quickly enough.

The delays of July 12th and July 13th were a result of an upgrade that our cloud provider made to our data warehouse. The bug that they introduced made our process progressively slower causing all our queries to take longer and longer. We worked closely with our cloud provider to find a solution but the first attempt did not resolve the issue causing the second day of delays. After further consultation with our cloud provider, we reverted their changes and have been able to return to our normal completion times.

WHAT HAVE WE BEEN DOING TO MITIGATE THESE ISSUES?

In April, we implemented some optimizations that took our normal completion time from around 6:30 am EST to around 3:00 am EST.
1. The optimizations have normally allowed us and our third-party service the opportunity to resolve any issues before 9am.
2. From April 14 - June 29, we completed the process before 9 am every day.
On June 30th, July 2nd, and July 7th there were some issues with database resources that we did not resolve quickly enough.
1. We have resolved the underlying database resources issue.
2. We also adjusted our alerting from 5am to 4am so that we can intervene more quickly.
3. Our third-party provider has also been adding (and is continuing to add) more alerting so they can intervene as soon as an issue is detected.
For the incidents of July 12th and 13th, after working with our cloud provider to revert changes, we have set our databases to trail behind one major version which will allow our cloud provider to find and resolve these types of issues before we upgrade to newer versions.

We are sorry for the pain that this may have caused each of you and will continue working on optimizing our process.

Posted Jul 18, 2022 - 14:17 UTC

Resolved

This incident has been resolved.

Posted Jul 13, 2022 - 23:46 UTC

Update

We are continuing to monitor for any further issues.

Posted Jul 13, 2022 - 23:42 UTC

Monitoring

The fix has been applied to the BIC infrastructure and it is now back online. The data is being caught up with the latest exports.

Posted Jul 13, 2022 - 21:33 UTC

Update

We are applying a fix to Business Intelligence Connector, which will take BIC data offline for about 60 minutes. This fix should prevent long delays, like the one we are having today.

Posted Jul 13, 2022 - 19:57 UTC

Identified

The vendor has identified the root cause of BIC data delays. We are working with them on a plan to address the root cause to prevent future delays. In the meantime, the BIC data is being refreshed on a slower schedule. We will notify when the refresh is complete.

Posted Jul 13, 2022 - 17:51 UTC

Update

The vendor is still working through the BIC delay issue. We will update next as soon as the cause is identified.

Posted Jul 13, 2022 - 16:34 UTC

Investigating

We are experiencing delays in updates to the Business Intelligence Connector data. A large subset of tables have not yet been updated. Customers using BIC will not have up-to-date data available in these tables. We are currently investigating with our vendor and will provide updates shortly.

Posted Jul 13, 2022 - 13:58 UTC

This incident affected: Greenhouse Business Intelligence Connector (Silo 1, Silo 2, Silo 3, Silo 4, Silo 5, Silo 6, Silo 101).