WHAT HAPPENED?
On Wednesday June 22nd at 10:46pm EDT, the Greenhouse Recruiting application experienced unplanned downtime due to an infrastructure issue with the data store that contains background job processing data.
WHAT WAS THE EFFECT?
The Greenhouse Recruiting application was unavailable for a period of approximately 42 minutes until 11:28pm EDT. Job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected and remained 100% available. There was no loss of data during this incident.
WHO WAS AFFECTED?
This disruption affected all of our customers using the Greenhouse Recruiting application. Other Greenhouse applications were not affected. Job applicants browsing or applying to job postings on job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected.
WHAT WAS THE CAUSE?
On Wednesday evening, one of the servers that houses Redis, a data store used for background job processing data, became unstable. As a consequence the Greenhouse Recruiting application was unable to perform write operations to the data store resulting in error pages for most, if not all, users of the Recruiting application.
By 11:10pm EDT our engineers diagnosed the cause of the issue and began working on a fix. This solution was implemented at 11:28pm EDT, immediately bringing the Greenhouse Recruiting application back online.
WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?
Although we had added safeguards to try to prevent the application from being dependent on our Redis infrastructure, we failed to anticipate this particular kind of error. As such we have immediately changed the application code to anticipate this and other failure modes.
We are also taking several steps on the infrastructure side to prevent this from happening again:
We take the availability of our software very seriously, and sincerely apologize to you and your team for the disruption that this downtime caused.