WHAT HAPPENED?

Greenhouse Job Boards and the Job Board API were unavailable for 25 minutes from 3:45pm EST on December 6th 2016. This was due to issues with our internal DNS infrastructure.

WHAT WAS THE EFFECT?

Greenhouse Job Boards and the Job Board API were unavailable for a period of approximately 25 minutes. Other parts of the Greenhouse platform, such as Greenhouse Recruiting, Greenhouse Onboarding, and Greenhouse Analytics were not affected and remained 100% available. No external candidate applications could be submitted during this period.

WHO WAS AFFECTED?

This disruption affected all of our customers using hosted or embedded job boards, as well as any career sites built using the Job Board API.

WHAT WAS THE CAUSE?

One of the two DNS servers supporting Greenhouse Job Boards became unavailable due to an issue with the underlying infrastructure. In these situations, the Job Boards should fall back on the secondary DNS server, but an error in the fallback process caused the system to continue using the inactive server instead of the functioning one.

As soon as we identified the cause of the downtime, we restarted the failing DNS server, bringing Job Boards and the Job Board API back online.

WHAT ARE WE DOING TO PREVENT THIS FROM OCCURRING AGAIN?

We have identified optimizations that we can make to our server configuration to ensure that the DNS failover works as intended. These changes should also reduce the likelihood of this class of issue causing downtime in the future.

We take the availability of our software very seriously, and look forward to making changes to prevent this kind of downtime happening again.