Unplanned downtime

Incident Report for Greenhouse

Postmortem

WHAT HAPPENED?

On Friday February 5th at 6:55pm EST, the Greenhouse Recruiting application experienced unplanned downtime due to unexpectedly high database load.

WHAT WAS THE EFFECT?

The Greenhouse Recruiting application was unavailable for a period of approximately 10 minutes. Job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected and remained 100% available.

WHO WAS AFFECTED?

This disruption affected all of our customers using the Greenhouse Recruitment application. Job applicants browsing or applying to job postings on job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected.

WHAT WAS THE CAUSE?

On Friday evening a user of our admin system ran an operation which consumed more resources than the system was designed to handle, causing increasingly slow page response times, eventually resulting in the Greenhouse Recruiting application being unavailable to our customers.

By 7:05pm EST, we were able to cancel the operation, at which point the app began functioning normally. During this time, all job boards were fully operational and unaffected by the outage.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We are taking steps to reduce the impact of the operation, and to prevent the internal tool from being used in the same way again.

We take the availability of our software very seriously, and sincerely apologize to you and your team for the disruption that this downtime caused.

Posted Feb 10, 2016 - 16:32 UTC

Resolved

We are back up and the issue appears to be resolved. We will investigate the issue and post a postmortem as soon as we learn more. Please note that job boards were not affected during this incident.
Posted Feb 06, 2016 - 00:07 UTC

Investigating

We are currently experiencing downtime and are investigating. More details to follow.
Posted Feb 05, 2016 - 23:57 UTC