WHAT HAPPENED?

The Greenhouse Recruitment application suffered performance degradation for 90 seconds ending at 4:28pm EST on 2/23/2016. This was due to issues that occurred during a routine deployment.

WHAT WAS THE EFFECT?

The Greenhouse Recruiting application was either unavailable or performed very poorly for a period of approximately 90 seconds. Other parts of the Greenhouse platform, such as Greenhouse Onboarding, Greenhouse Analytics and job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected and remained 100% available.

WHO WAS AFFECTED?

This disruption affected all of our customers using the Greenhouse Recruitment application. Job applicants browsing or applying to job postings on job boards hosted by Greenhouse or implemented using the Greenhouse Job Boards API were not affected.

WHAT WAS THE CAUSE?

A detailed investigation by our team revealed that, at the time of the deployment, a database connection was "hung" in an idle in transaction (aborted) error state. Unfortunately, this connection had acquired a ROW EXCLUSIVE database lock on an important database table. The type of lock would not usually prevent data being read from, or inserted into, this table.

However, the deployment included a schema change that required an ACCESS EXCLUSIVE lock. This lock conflicts with all other locks and could not be obtained due the aforementioned "hung" connection. In addition to this, any further reads or updates to this table were queued behind this request for the ACCESS EXCLUSIVE lock, causing any queries on this database table to also hang.

Initially, this only affected customers using this database table. Over time however, these hanging queries significantly affected performance of the rest of Greenhouse Recruiting application.

Once we aborted the deployment, the previously "hung" queries were unblocked, and performance returned to normal.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We are investigating ways in which we can close any "hung" connections prior to a deployment. Doing this will ensure that schema migrations that require locks such as ACCESS EXCLUSIVE will run quickly, minimizing the risk to our customers.

We take the availability of our software very seriously, and look forward to making changes to prevent this kind of downtime happening again.