Greenhouse Recruiting Outage

Incident Report for Greenhouse

Postmortem

WHAT HAPPENED?

On Thursday June 2nd at 10:07am EDT, the Greenhouse Recruiting application experienced unplanned downtime due to an unexpectedly high number of requests.

WHAT WAS THE EFFECT?

The Greenhouse Recruiting application was unavailable for a period of approximately 5 minutes. Job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected and remained 100% available.

WHO WAS AFFECTED?

This disruption affected all of our customers using the Greenhouse Recruitment application. Job applicants browsing or applying to job postings on job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected.

WHAT WAS THE CAUSE?

On Thursday morning, a bug in an internal monitoring application caused an extremely high volume of requests to be made to a page in the admin system of the Greenhouse Recruiting application. After a short time, this caused the application to crash and become unavailable to our customers.

By 10:11am EDT, Greenhouse engineers were able to stop the requests coming from the monitoring application and bring the Greenhouse Recruiting application back online.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We are taking steps to ensure that high request volume in our admin system cannot affect the performance of the rest of the Greenhouse Recruiting application.

We take the availability of our software very seriously, and sincerely apologize to you and your team for the disruption that this downtime caused.

Posted Jun 03, 2016 - 16:47 UTC

Resolved

The issue has been resolved. We have identified the root cause and will be posting a post-mortem shortly.
Posted Jun 02, 2016 - 16:54 UTC

Monitoring

Greenhouse Recruiting experienced approximately 5 minutes of unplanned downtime beginning at 10:08am EDT. The app is back online and we are monitoring the situation. Job boards were unaffected by the outage.
Posted Jun 02, 2016 - 14:15 UTC