WHAT HAPPENED?
A Greenhouse Job Boards system update caused some application attachments to fail to upload between 19:05 UTC on September 8, 2020 and 14:02 UTC on September 9, 2020. Attachments failed for applications routed to servers on the West Coast. The change was reverted after customers reported failures to process attachments posted to the Activity Feed. After reverting, we reprocessed attachments for all impacted applications. No data loss occurred.
TIMELINE
On September 8, 2020 at 19:05 UTC, a Greenhouse Job Boards system update was released that was later determined to have caused attachment processing issues.
Beginning on September 9, 2020 at 13:28 UTC, the issue with application attachments failing was identified. By 14:02 UTC, the update was rolled back, preventing additional attachment failures.
On September 11, 2020 at 19:18 UTC, all attachments were reprocessed and restored to candidate profiles, fully resolving the incident. All impacted candidates were tagged with the candidate tag “Greenhouse- Reprocessed Attachments Sept 2020”. No data loss occurred.
WHAT WAS THE EFFECT?
Attachments on applications failed to process if they were routed to the USW2 Amazon servers between 19:05 UTC on September 8, 2020 and 14:02 UTC on September 9, 2020. Any attachments that failed to process were recorded with a failure error on the candidate’s Activity Feed in Greenhouse Recruiting. Impacted applications appeared without attachments until they were fully restored on September 11, 2020.
WHO WAS AFFECTED?
Greenhouse Recruiting customers who had applications to the USW2 servers. We informed all operational contacts at impacted customers via email.
WHAT WAS THE CAUSE?
A Greenhouse Job Boards system update was released on September 8, 2020 at 19:05 UTC that changed the format of generated S3 URLs for the USW2 region. The Greenhouse Recruiting system received “Unauthorized” errors when retrieving application attachments from the Greenhouse Job Boards system’s USW2 S3 bucket.
WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?
We apologize for the inconvenience this incident has caused. We take the reliability of our application seriously and are actively working to prevent similar incidents like this one from occurring in the future. If you have any questions or concerns, please reach out via: https://support.greenhouse.io/hc/en-us/requests/new.