On Thursday October 29th at approximately 1:55pm, Greenhouse engineers performed a routine deployment. However, the new version of the application failed to come online smoothly, and remained offline until our engineers were able to rectify the situation at 2:50pm.
The application was unavailable for a period of 55 minutes.
This disruption affected all of our customers.
In order to minimize disruption when we deploy a new version of our application, servers with the new code are brought online before the servers with the old code are shut down. During this time, only old code is being served to customers whilst the new servers are booting up. Once the new servers are fully operational, they begin serving web requests and the old servers are spun down. This is known as “pre-boot”. During all other deployments, “pre-boot” has not caused any issues. However, during Thursday’s deployment, the “pre-boot” functionality unexpectedly consumed all available database connections, taking the application offline. We are still investigating the root cause of this, focusing our efforts on:
We will update this postmortem as we learn more.
As you may have noticed, in the last couple of weeks Greenhouse has fallen short of our own high expectations for both site reliability and performance. Sadly, these shortcomings have no doubt caused quite a bit of frustration for you and your team. Your expectations for us should remain high, and it is on us to meet those expectations consistently. As a first order of business, I want to apologize to you. We can and we will do better.
In order to address both the issues we're seeing, and to ensure performance and reliability stay high as we grow, we have created a performance and reliability task force within Greenhouse. Many of our most experienced engineers are now deployed towards achieving consistently great performance and uptime. In practice, this means dozens of small projects, and a few larger projects that will help us complete our transition from the scrappy upstart we were in 2013 to a high-performance, reliable SaaS provider in 2016.
Many of the things we're working on are highly technical, so I won't get into too much detail here, but if you are interested in learning more, please feel free to reach out to your Greenhouse Account Manager. That said, I do want to highlight a few projects currently in flight which will improve our reliability and performance in the near term:
This service will ensure that our customers' career pages and job boards stay up even if the main application is failing. Automated disaster recovery has been baked into its design from day one; once released, we expect to have the most reliable job board service in the business.
We have quite simply outgrown our current cloud PaaS provider. The prep work for our migration over to AWS has been in flight for some time, and as of next week, our new platform will formally enter the QA process. We expect to substantially migrate off of our PaaS provider within the next three months and onto AWS. This change will allow us to set up better disaster recovery, a globally distributed infrastructure, and improve overall performance and reliability.
Though most customers experience very good application performance, some of our larger customers have experienced slow page-load times. Though we have made dozens of small improvements along the way, we are now refocusing significant resources on providing consistently great performance to customers at every scale of business.
To address these issues for the long-term, the executive team at Greenhouse has begun to create formal performance and reliability benchmarks, and I will make hitting those benchmarks a consistent priority for both the product and engineering teams.
We are deeply thankful that we have such a loyal, passionate, and energized customer base, and by no means do we take any of that for granted. Thank you for all of your support and for your patience.
Best,
Daniel Chait
CEO & Co-founder