Increased error rates
Incident Report for Greenhouse
Postmortem

2021-03-03 Greenhouse Recruiting Increased Errors Post-Mortem

WHAT HAPPENED?

A Greenhouse Recruiting update was released on March 3, 2021 at 18:56 UTC that led to increased error rates for Greenhouse Recruiting customers until 20:44 UTC. The incident was fully resolved on March 3, 2021 at 21:40 UTC.

TIMELINE

March 3, 2021

18:56 UTC: Increased errors started to occur for Greenhouse Recruiting

20:21 UTC: The upgrade was rolled back for Greenhouse Recruiting silo 2

20:31 UTC: The upgrade was rolled back for all silos

21:40 UTC: Incident was fully resolved

WHAT WAS THE EFFECT?

Some requests to the Greenhouse Recruiting application failed and resulted in a ‘Something went wrong” error message.

WHO WAS AFFECTED?

All Greenhouse Recruiting silos.

Learn how to identify which silo your organization is on.

WHAT WAS THE CAUSE?

We upgraded a dependency used for some of our front end (ajax) requests. This caused requests that require a specific security token (csrf) to fail.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

  • We are requiring the security token as part of our automated tests, so that future issues with the token will be detected.
  • We are setting up Support (chat/ticket) alerts to more quickly identify when a large number of customers experience errors.
  • We are improving our front end error monitoring to alert on this type of error.

If you have any questions or concerns, please reach out via our Support center.

Posted Mar 08, 2021 - 14:47 EST

Resolved
This incident has been resolved.
Posted Mar 03, 2021 - 16:40 EST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 03, 2021 - 15:44 EST
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 03, 2021 - 15:35 EST
Investigating
We're investigating reports from some users receiving errors when using Greenhouse Recruiting.
Posted Mar 03, 2021 - 15:07 EST
This incident affected: Greenhouse Recruiting (Silo 1, Silo 2, Silo 3, Silo 4).