Monitoring - The automated resending process has been operating as expected. In addition, we are observing a decrease in rate-limiting by Microsoft. We will continue to monitor both aspects. The warmup of the new IP addresses also continues and we expect it to take at least the rest of this week to complete. We appreciate your patience as we continue to work through this incident.
Apr 23, 2024 - 17:22 EDT
Update - We have been monitoring the automated resending process along with the failure rates due to rate limits over the past few days. As mentioned in our last update, some emails that fail to be delivered to Microsoft-managed domains once will be delayed as we attempt to redeliver them. The warm-up of the IP address has also been progressing as expected. We appreciate your patience as we continue to work through this incident.
Apr 22, 2024 - 17:22 EDT
Update - We have released and are monitoring the automated resending of emails that failed delivery due to rate-limiting. Starting tomorrow you should observe a decrease in the "Too Old" bounce notifications. As we are resending messages while still experiencing rate-limiting by Microsoft, some emails going to Microsoft managed domains will be delayed in reaching the recipients. In parallel, while the warm up of the new IP addresses continues, we are working with our email provider to make the case to Microsoft to lift the rate-limiting.
We appreciate your patience as we continue to work through this incident.
Apr 19, 2024 - 15:51 EDT
Update - The warm up of the new IP addresses continues. We are also testing the automated resending of emails that failed delivery due to rate-limiting. If the tests are successful we will roll out the functionality to all our customers. We appreciate your patience as we continue to work through this incident.
Apr 18, 2024 - 17:00 EDT
Update - We continue to monitor the warm up of the new IP addresses. We have also made good progress on the automated resending of emails that fail delivery due to rate-limiting. We are targeting the end of this week to go live with this enhancement. We appreciate your patience as we continue to work through this incident.
Apr 17, 2024 - 17:42 EDT
Update - The ramp up of the email traffic to the new IP addresses is proceeding. In addition to this effort we are exploring a way to automatically resend messages that fail to get delivered due to rate-limiting. We appreciate your patience as we continue to work through this incident.
Apr 16, 2024 - 16:31 EDT
Update - We continue to work with our email provider to ramp up the traffic being directed to the IP addresses. We appreciate your patience as we work through this incident.
Apr 15, 2024 - 18:46 EDT
Update - We have started the use of the new IP addresses to alleviate the current rate-limiting issues and will gradually increase the traffic being directed to the IP addresses. We expect to see gradual improvements as a result of these changes. We will continue to work with Microsoft and our email provider to monitor and resolve the situation.
Apr 12, 2024 - 14:08 EDT
Update - We will begin the IP address migration process tomorrow morning to adjust to the throttling limits. We plan to gradually increase the use of the new IP addresses over the coming weeks and will be sending regular updates in the meantime. We appreciate your patience while we address the delivery issues.
Apr 10, 2024 - 17:45 EDT
Update - We will be adding new IP addresses to mitigate the throttling by Microsoft on Thursday. All customers are encouraged to add the new IP addresses to their DNS records as needed, per instructions from Greenhouse in an email sent on Friday, April 5. Please reach out to the support team with any questions.
Apr 08, 2024 - 17:02 EDT
Update - We are adding new IP addresses to our email infrastructure to help with the email delivery rate limits imposed by Microsoft. Please ensure your IT team reviews the email sent by Greenhouse with instructions, or reach out to our Support team if you did not receive this email. We apologize for the inconvenience and will provide a full post-mortem once this is resolved.
Apr 05, 2024 - 14:04 EDT
Update - Greenhouse is continuing to work with our email provider to resolve this ongoing issue. We have identified a path forward with our email provider and will be emailing the operational contacts for all customers tomorrow with specific next steps.
Apr 04, 2024 - 17:21 EDT
Update - We appreciate your patience as we continue working with our email provider and Microsoft to mitigate the email rate-limiting issue. We will provide further updates on Thursday including the next steps to reduce the impact.
Apr 03, 2024 - 17:22 EDT
Update - We are still working with our email provider on the email rate-limiting issue and we are searching for ways to mitigate it with Microsoft.
Apr 02, 2024 - 16:51 EDT
Update - We are continuing to work with our email provider on a resolution and attempting to get more information from Microsoft on the root cause that led to rate-limiting.
Apr 01, 2024 - 16:24 EDT
Update - We are continuing to work with our email provider to find a resolution to the issues with sending email to Microsoft email addresses. We will provide another update on Monday.
Mar 29, 2024 - 16:12 EDT
Update - We appreciate your patience as we continue to work with our email provider and Microsoft to identify a resolution.
We are also exploring other avenues to remove the rate limiting.
Mar 28, 2024 - 16:56 EDT
Update - We are continuing to work with our email provider and Microsoft to identify a resolution.
Users who receive a bounce-back email with the message “Too old” are encouraged to retry sending the referenced email to ensure successful delivery.
Mar 27, 2024 - 15:31 EDT
Identified - We have identified that one of our email providers has encountered a rate limiting issue with Microsoft, causing delays and occasional failures in sending to Outlook and Hotmail email accounts. We are working with this email provider to find a resolution.
Mar 26, 2024 - 16:48 EDT
Investigating - We are seeing delays in sending emails to Microsoft-based email providers (e.g. outlook, hotmail). We are currently investigating the issue.
Mar 26, 2024 - 15:46 EDT