WHAT HAPPENED?
At approximately 2:57 PM UTC (10:57 AM ET) on Thursday, June 23, Greenhouse engineers noted a problem with our sessions manager. The session manager was creating keys at an elevated rate vs our expected session rate.
While trying to diagnose this problem, at approximately 3:08 PM UTC, a Greenhouse engineer ran a command to list all the sessions, which brought down the main session manager with a failure that cascaded to follower machines as well.
Greenhouse alerting triggered an incident approximately a minute later at 3:09 PM UTC.
At about 3:15 PM UTC, Greenhouse had assessed the scope of the outage and put up a status page. The scope was that Silo 1 was completely inaccessible and many users who were not already logged into other Silos were unable to login because of the architecture of our login system. We were also unable to communicate alternate login procedures due to chat support being inaccessible because of its reliance on Silo 1.
By 3:25 PM, Greenhouse engineers had assessed what happened and began working on parallel paths to make Silo 1 accessible. One group of engineers was working to bring up a new session cluster, while another group of engineers was working to restore the old cluster.
At 3:49 PM, Greenhouse engineers succeeded in bringing a new session cluster online and assigning it to Silo 1. This restored access for customers.
At 4:01 PM UTC, Greenhouse resolved the incident.
WHAT WAS THE EFFECT?
During this incident, all customers on Silo 1 were unable to access Greenhouse in any way. Most users were unable to login to Greenhouse on other silos and, due to the nature of our chat support integration, we could not direct these customers to alternate login methods because our chat support team could not authenticate.
WHO WAS AFFECTED?
Customers trying to log in to Greenhouse and Silo 1 customers. Job boards and application submissions were unaffected by this outage. Customers who were already logged in to other silos were also unaffected by the outage.
WHAT WAS THE CAUSE?
Greenhouse engineers released a change several days earlier which had caused an increase in sessions. During normal investigation of this issue, a command was run that brought down Silo 1. We have since reverted the original change.
WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?
The primary cause of this outage was the use of a dangerous command during the normal investigation of what was a non-critical issue. Greenhouse is working to remove access to this command given the consequences of running it.
Greenhouse is also working to reduce the time from the diagnoses of the issue to bringing a new session manager back online. This process took up the bulk of the incident downtime and we are attempting to shorten this time frame so, should something like this occur again, we can recover much more quickly.
We apologize for the inconvenience this incident has caused. We take the reliability of our application seriously and are actively working to prevent similar incidents like this one from occurring in the future. If you have any questions or concerns, please reach out via: https://support.greenhouse.io/hc/en-us/requests/new.