Incidents reported on status page for Novu https://novustatus.com/ https://d1lppblt9t2x15.cloudfront.net/logos/8c177e1189ee341a20147261e3b64ff4.png https://novustatus.com/ en https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. https://novustatus.com/incident/533234 Sun, 23 Mar 2025 22:12:18 -0000 https://novustatus.com/incident/533234#41719d433fb573499eee7b7ad3c5b9b0a404e92dfbc21c22f2e31d448ddce0ef Webhooks service recovered. https://novustatus.com/incident/533234 Sun, 23 Mar 2025 22:10:32 -0000 https://novustatus.com/incident/533234#946f70bfd9f8b2d58347e9cc225a75ff85e2ef79b92cb37d5ecd43c6de2ae27d Webhooks service went down. https://novustatus.com/incident/511069 Tue, 11 Feb 2025 13:31:00 -0000 https://novustatus.com/incident/511069#c0e968007b4253588fee3e8168a6c347280d4ded164a3c2cd141214d87120119 We are seeing an increase in response times on our API Services. The team is actively looking in resolving the bottlenecks. https://novustatus.com/incident/453844 Thu, 31 Oct 2024 10:53:00 -0000 https://novustatus.com/incident/453844#3159b180bb127055adbda96b0265b9a380e8232864bc51c4cb91c7a4b0d79aab We've noticed increased delays in message deliveries. https://novustatus.com/incident/441238 Tue, 08 Oct 2024 13:10:00 -0000 https://novustatus.com/incident/441238#9ec5ccdbfdf7bce1b53d38a4fdd17aa93aba5ac4beb14c0736fb42d5df5e054e The US API at api.novu.co experienced 504 gateway timeout errors during a routine deployment, causing downtime in several regions between UTC 12:44 to 13:04. We immediately rolled back to the previous deployment and systems fully recovered at 13:04. We are continuing to monitor system health and investigate root cause. https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:22:00 -0000 https://novustatus.com/incident/420792#5081fe6a0a91acf16e62815d138355b419df4a9c0a65d4034312643987f8ed2e The issues with errors from FCM are resolved and confirmed. https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:22:00 -0000 https://novustatus.com/incident/420792#5081fe6a0a91acf16e62815d138355b419df4a9c0a65d4034312643987f8ed2e The issues with errors from FCM are resolved and confirmed. https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:14:00 -0000 https://novustatus.com/incident/420792#ff8c470779002cdf1df5ef5675ae2555d403c8818c6e710f4c020986caa7aaca The fix was deployed, we are validating the API responses from FCM https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:14:00 -0000 https://novustatus.com/incident/420792#ff8c470779002cdf1df5ef5675ae2555d403c8818c6e710f4c020986caa7aaca The fix was deployed, we are validating the API responses from FCM https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:44:00 -0000 https://novustatus.com/incident/420792#85a661981ac1428199f7a2d28f34706680874e0451b4680e40132fd78fc4a1ee We have identified the cause for the issue and deploying a fix https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:44:00 -0000 https://novustatus.com/incident/420792#85a661981ac1428199f7a2d28f34706680874e0451b4680e40132fd78fc4a1ee We have identified the cause for the issue and deploying a fix https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:33:00 -0000 https://novustatus.com/incident/420792#5706b11117f883fe8f998e04702c542522341b2faa096750da5853936c336cf7 We are looking into elevated error response from Google's FCM service https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:33:00 -0000 https://novustatus.com/incident/420792#5706b11117f883fe8f998e04702c542522341b2faa096750da5853936c336cf7 We are looking into elevated error response from Google's FCM service https://novustatus.com/incident/420145 Tue, 27 Aug 2024 15:02:00 -0000 https://novustatus.com/incident/420145#d9ee427bbd72a7271ebde675722e50741ff0a92b3773177cf97e5b5d1daa99b9 WebSocket Service DNS health-checks from our Status update provider reporting unreachability. No drop in connection was observed or ability to access the service has been seen from our team. This report didn't affect any operational elements of the websocket service. https://novustatus.com/incident/417442 Mon, 12 Aug 2024 14:51:00 -0000 https://novustatus.com/incident/417442#b45707fb8a5cd1dd5aa10c4495f5654838af95bd1766abaed4e4570128f6b953 The issue has been resolved, and we have implemented internal measures to avoid such service disruptions in the future by specifying better monitoring for our workflow engine fleet to ensure that elevated delivery failures will be marked for both EU and US clusters. https://novustatus.com/incident/417442 Wed, 07 Aug 2024 19:46:00 -0000 https://novustatus.com/incident/417442#c42df717f6559740e4e938173ce0e57002357378899f8b5800aeafa433e3e36c Between the 7th and 12th of August, we have observed an elevated rate of random delivery failures that were caused by a downstream provider rate limiting our API Calls. Notifications in this period marked as failed in the activity feed were not delivered, our retry engine has recovered some of the jobs, and those will be marked as successful in the activity feed. https://novustatus.com/incident/406814 Wed, 31 Jul 2024 15:07:00 -0000 https://novustatus.com/incident/406814#c566f3cdec2a3619bba71e7a4652bb943a67a3cf773cbbdbb299cd793c47728d A fix has been deployed and the system is fully operational. We are continuing to monitor the system for abnormal activity. https://novustatus.com/incident/406814 Wed, 31 Jul 2024 14:37:00 -0000 https://novustatus.com/incident/406814#32541fd25a36df4aac4d523927c345be413610fe097f95f5e5870eaf8cc9e484 We have identified the root cause of the issue and working on a fix. https://novustatus.com/incident/406814 Wed, 31 Jul 2024 14:18:00 -0000 https://novustatus.com/incident/406814#298258635d2b93d36948f034e7eece5ae91602d611736e31fb02e9f2beae51ec We are experiencing elevated 500 errors. We are investigating. https://novustatus.com/incident/307149 Mon, 01 Jan 2024 10:00:00 -0000 https://novustatus.com/incident/307149#c973549e8984d5f601e9ed84357b2e889650d090a03e533c7ba112f7be86be4f API is back to operational state, monitoring the service. https://novustatus.com/incident/307149 Mon, 01 Jan 2024 09:34:00 -0000 https://novustatus.com/incident/307149#5eed70359f364ea8c4f49ae2d9dba8b8fd093471fd7af36454836095a7621e55 We have identifier the issue, as fix is being deployed at the moment. https://novustatus.com/incident/307149 Mon, 01 Jan 2024 09:20:00 -0000 https://novustatus.com/incident/307149#6ef3b80f00d5eaa2dba4acf81582d86754b2292a2e1ffdaaa5362f791bbb47eb Elevated 502 Responses from API Service https://novustatus.com/incident/237837 Thu, 27 Jul 2023 02:38:57 -0000 https://novustatus.com/incident/237837#85d468d57a3e75ef41f2be0f6db1568ed5c10aefee8f43ea89439b18b10d3ecc WebSocket Service recovered. https://novustatus.com/incident/237837 Thu, 27 Jul 2023 02:31:00 -0000 https://novustatus.com/incident/237837#3c4bfaec24102cb0a416b7b057295ed29909e7c472edf87464dce243f2dc1265 A fix was deployed, services fully recovered, monitoring the system behavior. https://novustatus.com/incident/237837 Thu, 27 Jul 2023 01:47:52 -0000 https://novustatus.com/incident/237837#17e1354935df3c869b45c5a2b9cb291814f3d74b8d3d791c5fbfa28cb12a31b5 WebSocket Service went down. https://novustatus.com/incident/237837 Thu, 27 Jul 2023 00:31:00 -0000 https://novustatus.com/incident/237837#225ebb94951e7719bf09536c78a28a1c4fcf3a44bf272bba42e3578ce41f79bb A high memory load on our Redis cluster has caused a cascading error propagating to the API and WebSocket services. Our team is working on a fix. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:49:00 -0000 https://novustatus.com/incident/223993#e1c8642dcb9bcec7d095d13cc3c1473f591124a3f9f8f70783bce6144c0a0dad The job backlog has completed processing messages received prior to the incident. Throughput and response times are back to operational status. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:49:00 -0000 https://novustatus.com/incident/223993#e1c8642dcb9bcec7d095d13cc3c1473f591124a3f9f8f70783bce6144c0a0dad The job backlog has completed processing messages received prior to the incident. Throughput and response times are back to operational status. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:15:00 -0000 https://novustatus.com/incident/223993#75cea8584fee425350e49ddbf679aff9f02dabeb6ce6127182766e506ed71b7d Service is back online and we are monitoring the system behavior while jobs are being processed from the queue. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:15:00 -0000 https://novustatus.com/incident/223993#75cea8584fee425350e49ddbf679aff9f02dabeb6ce6127182766e506ed71b7d Service is back online and we are monitoring the system behavior while jobs are being processed from the queue. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:15:00 -0000 https://novustatus.com/incident/223993#4ed838a27485ec6e5d71d13e1c9fcaff5b4a86aa08e9b5b236b5a157e0b3818e Novu API and system are unresponsive. Our team is monitoring and identifying the root cause. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:15:00 -0000 https://novustatus.com/incident/223993#4ed838a27485ec6e5d71d13e1c9fcaff5b4a86aa08e9b5b236b5a157e0b3818e Novu API and system are unresponsive. Our team is monitoring and identifying the root cause. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:05:00 -0000 https://novustatus.com/incident/223993#43a06cde8f826411fb87a21925d003c7ff0a7c8883b2ffa0927c29a4aeceb27b We have identified the root cause, and a fix is being deployed to the systems right now. https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:05:00 -0000 https://novustatus.com/incident/223993#43a06cde8f826411fb87a21925d003c7ff0a7c8883b2ffa0927c29a4aeceb27b We have identified the root cause, and a fix is being deployed to the systems right now. https://novustatus.com/incident/185630 Thu, 16 Mar 2023 09:51:00 -0000 https://novustatus.com/incident/185630#17b60cac740907fae53c6ec045029040669bd4a738bacfb13d74a81589dcb3bc Response times are back at normal operational rates. https://novustatus.com/incident/185630 Thu, 16 Mar 2023 09:25:00 -0000 https://novustatus.com/incident/185630#79e9ba1eecc040bbcd68c4158df633c1d62508ff0213e5e4de00f9c4f7ad3706 The component causing the degraded performance was found and a patch was issued, monitoring the response times right now. https://novustatus.com/incident/185630 Thu, 16 Mar 2023 09:17:00 -0000 https://novustatus.com/incident/185630#9bfceaf991425dff385b7fdaa933b96c4c5ab71e9ee23c8da8402d31286d3ce2 The management API is currently experiencing a slowdown in response time and timeout errors for some reported customers. https://novustatus.com/incident/152202 Tue, 10 Jan 2023 12:56:00 -0000 https://novustatus.com/incident/152202#4490e5aa79a3005ae772c5458792af60c3d5f52fe3a49b3ecb8b16a46fa4a9bc Resopnse times are stable and returned to normal operation. https://novustatus.com/incident/152202 Tue, 10 Jan 2023 11:58:00 -0000 https://novustatus.com/incident/152202#f34a3cc8c0d1ef7174c954a02618df206c885901988997aa0aedc1d09fc6f378 A caching service upgrade was issued and deployed to help with load on the affected service. https://novustatus.com/incident/152202 Tue, 10 Jan 2023 11:54:00 -0000 https://novustatus.com/incident/152202#8f1da05246d804428cc49ce9136b2fc59a5cbdc90c399fe0b3de5e7fb67967c5 Response times are back to our average standard numbers. The teams keep monitoring API response time. https://novustatus.com/incident/152202 Tue, 10 Jan 2023 11:30:00 -0000 https://novustatus.com/incident/152202#f7857773085a73bc1ba52109fa7932c4b9722f3b233440a889933c75dce25e19 For a percentage of our users, we are seeing an elevated response time for API requests. We are monitoring this right now and working on a fix.