WHAT HAPPENED?

Around 50% of the applications submitted through Greenhouse Job Boards between 4:30PM EDT on Wednesday, June 7th and 11AM EDT on Thursday, June 8th were not immediately processed by Greenhouse Recruiting. By 2PM EDT on Friday, June 9th, all application appeared normally in Greenhouse Recruiting.

WHAT WAS THE EFFECT?

No application data was lost, but candidates who submitted delayed applications received emails informing them that Greenhouse was unable to process their applications. If the "Thank you for Applying" email was configured on the Job Post candidates used to apply, they received those messages when their applications were finally processed on Friday, June 9th.

WHO WAS AFFECTED?

This disruption affected all customers using hosted or embedded job boards. Any career sites built using the Job Board API were not affected.

WHAT WAS THE CAUSE?

On Wednesday, we rolled out infrastructure changes to improve the speed and reliability of Greenhouse Job Boards. Unfortunately, due to a testing configuration oversight, we did not correctly validate the entire application flow. As a result, the Greenhouse Recruiting application was unable to process any of the files created within the new job board infrastructure. While attempting to access these files, we uncovered a separate issue that required moving them from one AWS region to another, which took some time to remedy.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We will be modifying our new service rollout strategy in order to mitigate the chances of testing configuration issues. We will also be improving our monitoring over application processing failures to provide us with an earlier indication that something is wrong.

We take the reliability of our software very seriously, and are committed to making changes to prevent similar issues from occurring again. Please accept our apologies for any inconvenience caused. If you have any questions, please reach out to your Account Manager or support@greenhouse.io.