Overview

On 10/21/2015 the Greenhouse web application suffered a brief interruption of service from 6:15am ET lasting until 8am the same day, a duration of approximately 1h45m.

Who was affected?

During this brief outage, the Greenhouse web application was unavailable to all customers.

What happened?

At approximately 6:15am ET on 10/21/15, our team received alerts that the Greenhouse web application was not responding to routine health checks. By 6:20am, we had identified that infrastructure under our immediate control was functioning and immediately reached out to Heroku, our hosting provider, asking them to investigate. Heroku responded, letting us know that they were starting to see platform-wide errors affecting their US-based infrastructure. Shortly afterwards, they updated their status page to reflect these issues.

Heroku resolved their platform issues at approximately 8am the same day (10/21/15), and the Greenhouse web application came back online.

Next steps for Greenhouse

Even though this particular event was outside of our direct control, we take these outages incredibly seriously. As Greenhouse users ourselves, we understand how disruptive they are to a critical part of our customers’ organizations. As such, we are doing everything we can to mitigate their effects on the service we provide. Earlier this year, we began the process of migrating away from Heroku to Amazon Web Services. In light of last week’s incident, we are examining ways to reduce the timeline of that migration. In addition, we are taking immediate action to reduce our dependency on Heroku’s US infrastructure, reducing the risk of extended downtime occurring in the future. We will also soon be upgrading our job board infrastructure so that it will be available even if the main Greenhouse web application is unavailable.

We are taking every step within our power to reduce the frequency of application stability issues. We view each and every incident as a learning opportunity with an eye towards self improvement: for each incident we conduct postmortems and detail each failure-mode which contributed to the issue (people, process, and technology); and we are consistently actioning both short- and long-term mitigations to ensure that the same class of issue does not arise again. We sincerely apologize for the negative impact that these recent issues have had on your team.