Failure to process attachments for some applications
Incident Report for Greenhouse
Postmortem

WHAT HAPPENED?

A Greenhouse Job Boards system update caused some application attachments to fail to upload between 19:05 UTC on September 8, 2020 and 14:02 UTC on September 9, 2020. Attachments failed for applications routed to servers on the West Coast. The change was reverted after customers reported failures to process attachments posted to the Activity Feed. After reverting, we reprocessed attachments for all impacted applications. No data loss occurred.

TIMELINE

On September 8, 2020 at 19:05 UTC, a Greenhouse Job Boards system update was released that was later determined to have caused attachment processing issues.

Beginning on September 9, 2020 at 13:28 UTC, the issue with application attachments failing was identified. By 14:02 UTC, the update was rolled back, preventing additional attachment failures.

On September 11, 2020 at 19:18 UTC, all attachments were reprocessed and restored to candidate profiles, fully resolving the incident. All impacted candidates were tagged with the candidate tag “Greenhouse- Reprocessed Attachments Sept 2020”. No data loss occurred.

WHAT WAS THE EFFECT?

Attachments on applications failed to process if they were routed to the USW2 Amazon servers between 19:05 UTC on September 8, 2020 and 14:02 UTC on September 9, 2020. Any attachments that failed to process were recorded with a failure error on the candidate’s Activity Feed in Greenhouse Recruiting. Impacted applications appeared without attachments until they were fully restored on September 11, 2020.

WHO WAS AFFECTED?

Greenhouse Recruiting customers who had applications to the USW2 servers. We informed all operational contacts at impacted customers via email. 

WHAT WAS THE CAUSE?

A Greenhouse Job Boards system update was released on September 8, 2020 at 19:05 UTC that changed the format of generated S3 URLs for the USW2 region. The Greenhouse Recruiting system received “Unauthorized” errors when retrieving application attachments from the Greenhouse Job Boards system’s USW2 S3 bucket.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

  • We’re updating application code to be more flexible when handling S3 url formats.
  • We are improving internal alerting on the volume of attachment failures.
  • We are investigating building a tool that will allow us to reprocess applications more efficiently in the future.
  • We are investigating clearer in-app alerting in Application Review when a candidate has submitted an attachment but it has failed to process.

We apologize for the inconvenience this incident has caused. We take the reliability of our application seriously and are actively working to prevent similar incidents like this one from occurring in the future. If you have any questions or concerns, please reach out via: https://support.greenhouse.io/hc/en-us/requests/new.

Posted Sep 21, 2020 - 10:39 EDT

Resolved
We have completed reprocessing attachments.

Candidates who were impacted and reprocessed have been tagged with “Greenhouse- Reprocessed Attachments Sept 2020”.

To quickly find these candidates in the app, you can click the “All Candidates” top level navigation link and then open the “Profile Details” dropdown to filter by “Candidate Tag”.
Posted Sep 11, 2020 - 15:18 EDT
Update
We’ve reprocessed attachments for 99.5% of impacted applications. We’re continuing to reprocess the remaining few attachments. Candidates that were impacted and reprocessed have been tagged with “Greenhouse- Reprocessed Attachments Sept 2020”.

To quickly find these candidates in the app, you can click the “All Candidates” top level navigation link and then open the “Profile Details” dropdown to filter by “Candidate Tag”.
Posted Sep 11, 2020 - 11:59 EDT
Update
We are continuing to work on reprocessing attachments that previously failed to upload.
Posted Sep 10, 2020 - 09:38 EDT
Monitoring
New applications with attachments are now processing as expected. A small number of applications that were begun by the applicant prior to the fix may fail to include attachments. We are continuing to work on reprocessing attachments that previously failed to upload.
Posted Sep 09, 2020 - 12:55 EDT
Identified
We've identified the issue and have reduced the rate of attachment processing failures. We're investigating the small amount of new failures. We are currently reprocessing previous failures, which will restore the attachment to the candidate application in Greenhouse.
Posted Sep 09, 2020 - 11:25 EDT
Investigating
We're investigating failures to import attachments when processing some applications.
Applications impacted have a note on their activity feed with a failure message.
We're investigating reprocessing failed application attachments.
Posted Sep 09, 2020 - 10:01 EDT
This incident affected: Greenhouse Recruiting (Silo 1).