Greenhouse Business Intelligence Connector Postmortem

WHAT HAPPENED?

Since January 1, 2022, we have missed the 9am EST BIC data delivery time 15 times with most of those delays concentrated in two windows of time:

• March 19 - April 13 (8 delayed days)
• June 30 - July 13 (5 delayed days)

WHAT WAS THE CAUSE?

The March/April incidents were the result of occasional issues with the processing of the data by our third-party service provider. When these issues occur, it requires human intervention to ensure that the process completes successfully.

The June 30th, July 2nd, and July 7th incidents were caused by additional issues with database resources that we did not resolve quickly enough.

The delays of July 12th and July 13th were a result of an upgrade that our cloud provider made to our data warehouse. The bug that they introduced made our process progressively slower causing all our queries to take longer and longer. We worked closely with our cloud provider to find a solution but the first attempt did not resolve the issue causing the second day of delays. After further consultation with our cloud provider, we reverted their changes and have been able to return to our normal completion times.

WHAT HAVE WE BEEN DOING TO MITIGATE THESE ISSUES?

1. In April, we implemented some optimizations that took our normal completion time from around 6:30 am EST to around 3:00 am EST.

   1. The optimizations have normally allowed us and our third-party service the opportunity to resolve any issues before 9am.
   2. From April 14 - June 29, we completed the process before 9 am every day.

2. On June 30th, July 2nd, and July 7th there were some issues with database resources that we did not resolve quickly enough.

   1. We have resolved the underlying database resources issue.
   2. We also adjusted our alerting from 5am to 4am so that we can intervene more quickly.
   3. Our third-party provider has also been adding (and is continuing to add) more alerting so they can intervene as soon as an issue is detected.

3. For the incidents of July 12th and 13th, after working with our cloud provider to revert changes, we have set our databases to trail behind one major version which will allow our cloud provider to find and resolve these types of issues before we upgrade to newer versions.

We are sorry for the pain that this may have caused each of you and will continue working on optimizing our process.