Delay in sending emails

Incident Report for Greenhouse

Resolved

Email delivery to Microsoft managed domains is operating normally. We are closing this incident, but feel free to reach out to our Support team in case you are still observing bounce-back type notifications. We greatly appreciate your patience with us as we have worked through this incident.
Posted Apr 25, 2024 - 17:31 UTC

Update

We can confirm that Microsoft has lifted the rate-limiting of our IP addresses. Delivery of emails to Microsoft managed domains is no longer delayed. Out of abundance of caution, we are continuing to monitor email delivery and are leaving the redelivery logic in place. We are also expecting to complete the warmup of the new IP addresses to have them on hand in case we experience similar issues in the future. We hope to fully resolve the incident tomorrow.
Posted Apr 24, 2024 - 21:10 UTC

Monitoring

The automated resending process has been operating as expected. In addition, we are observing a decrease in rate-limiting by Microsoft. We will continue to monitor both aspects. The warmup of the new IP addresses also continues and we expect it to take at least the rest of this week to complete. We appreciate your patience as we continue to work through this incident.
Posted Apr 23, 2024 - 21:22 UTC

Update

We have been monitoring the automated resending process along with the failure rates due to rate limits over the past few days. As mentioned in our last update, some emails that fail to be delivered to Microsoft-managed domains once will be delayed as we attempt to redeliver them. The warm-up of the IP address has also been progressing as expected. We appreciate your patience as we continue to work through this incident.
Posted Apr 22, 2024 - 21:22 UTC

Update

We have released and are monitoring the automated resending of emails that failed delivery due to rate-limiting. Starting tomorrow you should observe a decrease in the "Too Old" bounce notifications. As we are resending messages while still experiencing rate-limiting by Microsoft, some emails going to Microsoft managed domains will be delayed in reaching the recipients. In parallel, while the warm up of the new IP addresses continues, we are working with our email provider to make the case to Microsoft to lift the rate-limiting.
We appreciate your patience as we continue to work through this incident.
Posted Apr 19, 2024 - 19:51 UTC

Update

The warm up of the new IP addresses continues. We are also testing the automated resending of emails that failed delivery due to rate-limiting. If the tests are successful we will roll out the functionality to all our customers. We appreciate your patience as we continue to work through this incident.
Posted Apr 18, 2024 - 21:00 UTC

Update

We continue to monitor the warm up of the new IP addresses. We have also made good progress on the automated resending of emails that fail delivery due to rate-limiting. We are targeting the end of this week to go live with this enhancement. We appreciate your patience as we continue to work through this incident.
Posted Apr 17, 2024 - 21:42 UTC

Update

The ramp up of the email traffic to the new IP addresses is proceeding. In addition to this effort we are exploring a way to automatically resend messages that fail to get delivered due to rate-limiting. We appreciate your patience as we continue to work through this incident.
Posted Apr 16, 2024 - 20:31 UTC

Update

We continue to work with our email provider to ramp up the traffic being directed to the IP addresses. We appreciate your patience as we work through this incident.
Posted Apr 15, 2024 - 22:46 UTC

Update

We have started the use of the new IP addresses to alleviate the current rate-limiting issues and will gradually increase the traffic being directed to the IP addresses. We expect to see gradual improvements as a result of these changes. We will continue to work with Microsoft and our email provider to monitor and resolve the situation.
Posted Apr 12, 2024 - 18:08 UTC

Update

We will begin the IP address migration process tomorrow morning to adjust to the throttling limits. We plan to gradually increase the use of the new IP addresses over the coming weeks and will be sending regular updates in the meantime. We appreciate your patience while we address the delivery issues.
Posted Apr 10, 2024 - 21:45 UTC

Update

We will be adding new IP addresses to mitigate the throttling by Microsoft on Thursday. All customers are encouraged to add the new IP addresses to their DNS records as needed, per instructions from Greenhouse in an email sent on Friday, April 5. Please reach out to the support team with any questions.
Posted Apr 08, 2024 - 21:02 UTC

Update

We are adding new IP addresses to our email infrastructure to help with the email delivery rate limits imposed by Microsoft. Please ensure your IT team reviews the email sent by Greenhouse with instructions, or reach out to our Support team if you did not receive this email. We apologize for the inconvenience and will provide a full post-mortem once this is resolved.
Posted Apr 05, 2024 - 18:04 UTC

Update

Greenhouse is continuing to work with our email provider to resolve this ongoing issue. We have identified a path forward with our email provider and will be emailing the operational contacts for all customers tomorrow with specific next steps.
Posted Apr 04, 2024 - 21:21 UTC

Update

We appreciate your patience as we continue working with our email provider and Microsoft to mitigate the email rate-limiting issue. We will provide further updates on Thursday including the next steps to reduce the impact.
Posted Apr 03, 2024 - 21:22 UTC

Update

We are still working with our email provider on the email rate-limiting issue and we are searching for ways to mitigate it with Microsoft.
Posted Apr 02, 2024 - 20:51 UTC

Update

We are continuing to work with our email provider on a resolution and attempting to get more information from Microsoft on the root cause that led to rate-limiting.
Posted Apr 01, 2024 - 20:24 UTC

Update

We are continuing to work with our email provider to find a resolution to the issues with sending email to Microsoft email addresses. We will provide another update on Monday.
Posted Mar 29, 2024 - 20:12 UTC

Update

We appreciate your patience as we continue to work with our email provider and Microsoft to identify a resolution.
We are also exploring other avenues to remove the rate limiting.
Posted Mar 28, 2024 - 20:56 UTC

Update

We are continuing to work with our email provider and Microsoft to identify a resolution.
Users who receive a bounce-back email with the message “Too old” are encouraged to retry sending the referenced email to ensure successful delivery.
Posted Mar 27, 2024 - 19:31 UTC

Identified

We have identified that one of our email providers has encountered a rate limiting issue with Microsoft, causing delays and occasional failures in sending to Outlook and Hotmail email accounts. We are working with this email provider to find a resolution.
Posted Mar 26, 2024 - 20:48 UTC

Investigating

We are seeing delays in sending emails to Microsoft-based email providers (e.g. outlook, hotmail). We are currently investigating the issue.
Posted Mar 26, 2024 - 19:46 UTC
This incident affected: Greenhouse Recruiting (Silo 1, Silo 2, Silo 3, Silo 4, Silo 5, Silo 6, Silo 7, Silo 8, Silo 101).