The issue has been resolved, and we have implemented internal measures to avoid such service disruptions in the future by specifying better monitoring for our workflow engine fleet to ensure that elevated delivery failures will be marked for both EU and US clusters.

Between the 7th and 12th of August, we have observed an elevated rate of random delivery failures that were caused by a downstream provider rate limiting our API Calls. Notifications in this period marked as failed in the activity feed were not delivered, our retry engine has recovered some of the jobs, and those will be marked as successful in the activity feed.