Incidents | Novu Incidents reported on status page for Novu https://novustatus.com/ https://d1lppblt9t2x15.cloudfront.net/logos/8c177e1189ee341a20147261e3b64ff4.png Incidents | Novu https://novustatus.com/ en Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 19:34:00 -0000 https://novustatus.com/incident/743992#9a4af6b64f7c3db76c29c01523c8b5c431602a4411ca6b19e618d2eabd62b016 The slow secondary instance was resolved and query times are back to normal on our API, we are working with our cloud vendor for a full RCA. The incident post mortem is available at: https://www.notion.so/novuhq/Preference-Schema-Indices-Incident-October-14-15-2025-285d03ad838f8024accacb4b589ec277 Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 19:34:00 -0000 https://novustatus.com/incident/743992#9a4af6b64f7c3db76c29c01523c8b5c431602a4411ca6b19e618d2eabd62b016 The slow secondary instance was resolved and query times are back to normal on our API, we are working with our cloud vendor for a full RCA. The incident post mortem is available at: https://www.notion.so/novuhq/Preference-Schema-Indices-Incident-October-14-15-2025-285d03ad838f8024accacb4b589ec277 Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 13:25:00 -0000 https://novustatus.com/incident/743992#c2cb089d806e47042c95321af4142a0c1724c8714ac6b24986c45f9a8b4c863d Response times are returned to normal times, except preference endpoints which still experience slow response times in some scenarios. We are performing database operations on the remaining secondary node. And Working with out database vendor on identifying the root cause. Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 13:25:00 -0000 https://novustatus.com/incident/743992#c2cb089d806e47042c95321af4142a0c1724c8714ac6b24986c45f9a8b4c863d Response times are returned to normal times, except preference endpoints which still experience slow response times in some scenarios. We are performing database operations on the remaining secondary node. And Working with out database vendor on identifying the root cause. Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 11:28:00 -0000 https://novustatus.com/incident/743992#cb16864298b95fbfc285bba58ebf654bb0f45249980b61e5829073f0994577a7 We are still experiencing high load on the secondary nodes and working to restore their operational metrics Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 11:28:00 -0000 https://novustatus.com/incident/743992#cb16864298b95fbfc285bba58ebf654bb0f45249980b61e5829073f0994577a7 We are still experiencing high load on the secondary nodes and working to restore their operational metrics Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 11:05:00 -0000 https://novustatus.com/incident/743992#441b7a1d036094f97e5be0ebe1b031333435b8973abd6fc18d782076dbf1e0a8 Inbox endpoints are now back to operational mode, observing the response times. A DB operation has been performed to roll the affected nodes Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 11:05:00 -0000 https://novustatus.com/incident/743992#441b7a1d036094f97e5be0ebe1b031333435b8973abd6fc18d782076dbf1e0a8 Inbox endpoints are now back to operational mode, observing the response times. A DB operation has been performed to roll the affected nodes Inbox Feed API returning 504 and 502 errors https://novustatus.com/incident/744666 Wed, 15 Oct 2025 10:59:00 -0000 https://novustatus.com/incident/744666#de93a80cbb164df16d26ea5034aa5d6164cd063a2c98dd5d56268e195b3d1d90 We have restored the inbox feed API, altho still experiencing slow response times on preference endpoints. Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 10:53:00 -0000 https://novustatus.com/incident/743992#c583c0e67d6de8bb73643eb7713c7c34cf28d6906f19ecc162fab5d07e105c44 Inbox API endpoints are experiencing a downtime Inbox Feed API returning 504 and 502 errors https://novustatus.com/incident/744666 Wed, 15 Oct 2025 10:53:00 -0000 https://novustatus.com/incident/744666#de800592b954e3918b961683405eb3dca008aacbad25edce6d3f535ba4caf68d Due to overload caused by the incident, inbox feed endpoints experienced an elevated 502 and 504 error rates. Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 10:53:00 -0000 https://novustatus.com/incident/743992#c583c0e67d6de8bb73643eb7713c7c34cf28d6906f19ecc162fab5d07e105c44 Inbox API endpoints are experiencing a downtime Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 10:36:00 -0000 https://novustatus.com/incident/743992#933e2922ddde2236035ae0965ad83707f18ddb067f24894b9f6114f4fd2ee9ad The US API cluster is affected in slow response times and timeouts observed. The team is working on the issue Slow response times observed on US API services https://novustatus.com/incident/743992 Wed, 15 Oct 2025 10:36:00 -0000 https://novustatus.com/incident/743992#933e2922ddde2236035ae0965ad83707f18ddb067f24894b9f6114f4fd2ee9ad The US API cluster is affected in slow response times and timeouts observed. The team is working on the issue Slow response times on US - API observed https://novustatus.com/incident/743636 Wed, 15 Oct 2025 02:15:00 -0000 https://novustatus.com/incident/743636#483171715593c694ccfb2c97df01677305c2e92ca839e9a4706181975b2ca12d API Response times are returning to normal, as well as dashboard loading. We are observing the latency Slow response times on US - API observed https://novustatus.com/incident/743636 Wed, 15 Oct 2025 02:15:00 -0000 https://novustatus.com/incident/743636#483171715593c694ccfb2c97df01677305c2e92ca839e9a4706181975b2ca12d API Response times are returning to normal, as well as dashboard loading. We are observing the latency Slow response times on US - API observed https://novustatus.com/incident/743636 Wed, 15 Oct 2025 00:09:00 -0000 https://novustatus.com/incident/743636#c644cde2792b40342a289009a356a03db5799d5fb8eca417c0b967126ed1af35 We are working with out cloud provider on mitigating the issue. Updates will be posted on the thread. Slow response times on US - API observed https://novustatus.com/incident/743636 Wed, 15 Oct 2025 00:09:00 -0000 https://novustatus.com/incident/743636#c644cde2792b40342a289009a356a03db5799d5fb8eca417c0b967126ed1af35 We are working with out cloud provider on mitigating the issue. Updates will be posted on the thread. Slow response times on US - API observed https://novustatus.com/incident/743636 Tue, 14 Oct 2025 22:09:00 -0000 https://novustatus.com/incident/743636#38ba03e3c9a8d1c53b5f3fa0d135ae8b2bfbdd62663baf4f6b236bb838b509c5 We are seeing slow response times on the API, the team is looking into it Elevated response times on US API service https://novustatus.com/incident/742904 Mon, 13 Oct 2025 20:05:00 -0000 https://novustatus.com/incident/742904#456734d22a86e3e7361efe74776443d3101aa992884604282e390a5dffc2ed24 Services are back to normal response times. Elevated response times on US API service https://novustatus.com/incident/742904 Mon, 13 Oct 2025 20:00:00 -0000 https://novustatus.com/incident/742904#4e0f0890b4010f044646ee6cfa5876ab75c9a9dc92713ed1697cd7f94149db40 Response times are back at normal rates, and we still observing the service performance. Elevated response times on US API service https://novustatus.com/incident/742904 Mon, 13 Oct 2025 19:50:00 -0000 https://novustatus.com/incident/742904#dd6d36a1a5d472148bb190a8d9459e604777a03662b95697363b55c554655cfb We are monitoring increased latency on US API services, a system scale operation has been initialised and we are monitoring the response times. Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 14:52:00 -0000 https://novustatus.com/incident/724593#c2c1b19216fff7f20570f9b971876b0ce4ee6012847c6772ac7c9095e79bb21e Login-requests are back to normal response times. While the 3rd party provider has not reported yet an official resolution, we will keep monitoring the state of the login and session timeout requests Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 14:52:00 -0000 https://novustatus.com/incident/724593#c2c1b19216fff7f20570f9b971876b0ce4ee6012847c6772ac7c9095e79bb21e Login-requests are back to normal response times. While the 3rd party provider has not reported yet an official resolution, we will keep monitoring the state of the login and session timeout requests Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 14:19:00 -0000 https://novustatus.com/incident/724593#d6c259e5f4dd91d0c80380de8db95e4a9f6590c0b003196747c942c5d632f1a9 We are seeing a reduced timeout errors during auth, but still monitoring the situation. To clarify, the delivery ingestion and API is not affected by this. Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 14:19:00 -0000 https://novustatus.com/incident/724593#d6c259e5f4dd91d0c80380de8db95e4a9f6590c0b003196747c942c5d632f1a9 We are seeing a reduced timeout errors during auth, but still monitoring the situation. To clarify, the delivery ingestion and API is not affected by this. Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 14:01:00 -0000 https://novustatus.com/incident/724593#6fe1e628c11e3c069b6df074ab0bbdafb4bf2cdfba6ed5247eb8695c29fdc888 The downstream provider is actively working on resolving the timeout errors observed when logging in. Reloading the dashboard multiple times might still be able to obtain a logged-in session. Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 14:01:00 -0000 https://novustatus.com/incident/724593#6fe1e628c11e3c069b6df074ab0bbdafb4bf2cdfba6ed5247eb8695c29fdc888 The downstream provider is actively working on resolving the timeout errors observed when logging in. Reloading the dashboard multiple times might still be able to obtain a logged-in session. Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 13:43:00 -0000 https://novustatus.com/incident/724593#75a5e30b6d71fb6d1d6ce8dee7dcecbc52aa9573d0132a5ab72f393769b9b5cd We are experiencing errors when accessing the dashboard.novu.co client due to a degraded performance with our downstream auth provider. Our team monitors the situation and will provide updates. Issues with login and dashboard visits https://novustatus.com/incident/724593 Mon, 15 Sep 2025 13:43:00 -0000 https://novustatus.com/incident/724593#75a5e30b6d71fb6d1d6ce8dee7dcecbc52aa9573d0132a5ab72f393769b9b5cd We are experiencing errors when accessing the dashboard.novu.co client due to a degraded performance with our downstream auth provider. Our team monitors the situation and will provide updates. Degraded response times on US API service https://novustatus.com/incident/722566 Thu, 11 Sep 2025 14:51:00 -0000 https://novustatus.com/incident/722566#ff23bc36da047d4308ba014dd83be6abc9009afd0c4fa1e0321733725337fadd Response times are back to normals, we are monitoring the results. Degraded response times on US API service https://novustatus.com/incident/722566 Thu, 11 Sep 2025 14:12:00 -0000 https://novustatus.com/incident/722566#e15a1adaacd9b2c04008c27867a38462c30da2cef3ce4062e3724a987f9e8208 A fix is being deployed on the affected routes Degraded response times on US API service https://novustatus.com/incident/722566 Thu, 11 Sep 2025 13:50:00 -0000 https://novustatus.com/incident/722566#a5af40f0ebd0788fc593637db3a01801f654d8aa613078a36c6d01d909ad8aed We are seeing an elevated response times on our API service, and the team is currently investigating. Elevated error rates on API Service https://novustatus.com/incident/715695 Sun, 31 Aug 2025 07:44:00 -0000 https://novustatus.com/incident/715695#e12721bacdf45cb977c05a274c1b33fc5af486b6cadfca091566a6b871b8b6a1 The response times are back to usual metrics, the team is monitoring Elevated error rates on API Service https://novustatus.com/incident/715695 Sun, 31 Aug 2025 07:16:00 -0000 https://novustatus.com/incident/715695#14bac68cd1ec8e0bf7342291727d9281c88c267b2e0ca26f6e0f8a2fd777166a The team is investigating the issue Elevated error rate on API Calls to our API cluster https://novustatus.com/incident/614123 Fri, 04 Jul 2025 09:47:00 -0000 https://novustatus.com/incident/614123#da87ce8120f4a7ba60166b21d985d3eca354bfd3c55f8e8456c7e43fc2afb5d3 The issue has been resolved, and we are monitoring the service. Elevated error rate on API Calls to our API cluster https://novustatus.com/incident/614123 Fri, 04 Jul 2025 09:41:00 -0000 https://novustatus.com/incident/614123#fbe8309835acdfddf41be2002788bdadd5305c5207a1c57c90a41f8ec52d8ae4 We have identified the root cause and deploying a fix Elevated error rate on API Calls to our API cluster https://novustatus.com/incident/614123 Fri, 04 Jul 2025 09:21:00 -0000 https://novustatus.com/incident/614123#9b15ce68aac15e347c43dd596a6a50ba05504140025a5f14d0d8109056cc3e94 We are seeing some requests to our API Service respond with 502 errors, and investigating the root cause. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 23:35:00 -0000 https://novustatus.com/incident/533236#235bad07a257bddbd6914e298216a6c8461b9806885ce8cef193cc7ad6f77498 DNS Resolution is fully restored across regions. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. DNS resolution failure https://novustatus.com/incident/533236 Sun, 23 Mar 2025 22:19:00 -0000 https://novustatus.com/incident/533236#a62833d62dff0340b5898a79e66d613fb5e2357f125a9233d00f77fb3ad2de51 API and Dashboard access is back to operational mode, we are monitoring the traffic at the moment. Webhooks service is down https://novustatus.com/incident/533234 Sun, 23 Mar 2025 22:12:18 -0000 https://novustatus.com/incident/533234#41719d433fb573499eee7b7ad3c5b9b0a404e92dfbc21c22f2e31d448ddce0ef Webhooks service recovered. Webhooks service is down https://novustatus.com/incident/533234 Sun, 23 Mar 2025 22:10:32 -0000 https://novustatus.com/incident/533234#946f70bfd9f8b2d58347e9cc225a75ff85e2ef79b92cb37d5ecd43c6de2ae27d Webhooks service went down. US API Degraded Response times https://novustatus.com/incident/511069 Tue, 11 Feb 2025 13:31:00 -0000 https://novustatus.com/incident/511069#c0e968007b4253588fee3e8168a6c347280d4ded164a3c2cd141214d87120119 We are seeing an increase in response times on our API Services. The team is actively looking in resolving the bottlenecks. Degraded performance in message deliveries https://novustatus.com/incident/453844 Thu, 31 Oct 2024 10:53:00 -0000 https://novustatus.com/incident/453844#3159b180bb127055adbda96b0265b9a380e8232864bc51c4cb91c7a4b0d79aab We've noticed increased delays in message deliveries. US API Connectivity Issues https://novustatus.com/incident/441238 Tue, 08 Oct 2024 13:10:00 -0000 https://novustatus.com/incident/441238#9ec5ccdbfdf7bce1b53d38a4fdd17aa93aba5ac4beb14c0736fb42d5df5e054e The US API at api.novu.co experienced 504 gateway timeout errors during a routine deployment, causing downtime in several regions between UTC 12:44 to 13:04. We immediately rolled back to the previous deployment and systems fully recovered at 13:04. We are continuing to monitor system health and investigate root cause. Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:22:00 -0000 https://novustatus.com/incident/420792#5081fe6a0a91acf16e62815d138355b419df4a9c0a65d4034312643987f8ed2e The issues with errors from FCM are resolved and confirmed. Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:22:00 -0000 https://novustatus.com/incident/420792#5081fe6a0a91acf16e62815d138355b419df4a9c0a65d4034312643987f8ed2e The issues with errors from FCM are resolved and confirmed. Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:14:00 -0000 https://novustatus.com/incident/420792#ff8c470779002cdf1df5ef5675ae2555d403c8818c6e710f4c020986caa7aaca The fix was deployed, we are validating the API responses from FCM Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 15:14:00 -0000 https://novustatus.com/incident/420792#ff8c470779002cdf1df5ef5675ae2555d403c8818c6e710f4c020986caa7aaca The fix was deployed, we are validating the API responses from FCM Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:44:00 -0000 https://novustatus.com/incident/420792#85a661981ac1428199f7a2d28f34706680874e0451b4680e40132fd78fc4a1ee We have identified the cause for the issue and deploying a fix Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:44:00 -0000 https://novustatus.com/incident/420792#85a661981ac1428199f7a2d28f34706680874e0451b4680e40132fd78fc4a1ee We have identified the cause for the issue and deploying a fix Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:33:00 -0000 https://novustatus.com/incident/420792#5706b11117f883fe8f998e04702c542522341b2faa096750da5853936c336cf7 We are looking into elevated error response from Google's FCM service Elevated FCM provider response rates from Google https://novustatus.com/incident/420792 Wed, 28 Aug 2024 14:33:00 -0000 https://novustatus.com/incident/420792#5706b11117f883fe8f998e04702c542522341b2faa096750da5853936c336cf7 We are looking into elevated error response from Google's FCM service WS Health-check from our monitoring provider https://novustatus.com/incident/420145 Tue, 27 Aug 2024 15:02:00 -0000 https://novustatus.com/incident/420145#d9ee427bbd72a7271ebde675722e50741ff0a92b3773177cf97e5b5d1daa99b9 WebSocket Service DNS health-checks from our Status update provider reporting unreachability. No drop in connection was observed or ability to access the service has been seen from our team. This report didn't affect any operational elements of the websocket service. [Resolved] - Elevated EU Cluster workflow engine delivery failures https://novustatus.com/incident/417442 Mon, 12 Aug 2024 14:51:00 -0000 https://novustatus.com/incident/417442#b45707fb8a5cd1dd5aa10c4495f5654838af95bd1766abaed4e4570128f6b953 The issue has been resolved, and we have implemented internal measures to avoid such service disruptions in the future by specifying better monitoring for our workflow engine fleet to ensure that elevated delivery failures will be marked for both EU and US clusters. [Resolved] - Elevated EU Cluster workflow engine delivery failures https://novustatus.com/incident/417442 Wed, 07 Aug 2024 19:46:00 -0000 https://novustatus.com/incident/417442#c42df717f6559740e4e938173ce0e57002357378899f8b5800aeafa433e3e36c Between the 7th and 12th of August, we have observed an elevated rate of random delivery failures that were caused by a downstream provider rate limiting our API Calls. Notifications in this period marked as failed in the activity feed were not delivered, our retry engine has recovered some of the jobs, and those will be marked as successful in the activity feed. Elevated API errors https://novustatus.com/incident/406814 Wed, 31 Jul 2024 15:07:00 -0000 https://novustatus.com/incident/406814#c566f3cdec2a3619bba71e7a4652bb943a67a3cf773cbbdbb299cd793c47728d A fix has been deployed and the system is fully operational. We are continuing to monitor the system for abnormal activity. Elevated API errors https://novustatus.com/incident/406814 Wed, 31 Jul 2024 14:37:00 -0000 https://novustatus.com/incident/406814#32541fd25a36df4aac4d523927c345be413610fe097f95f5e5870eaf8cc9e484 We have identified the root cause of the issue and working on a fix. Elevated API errors https://novustatus.com/incident/406814 Wed, 31 Jul 2024 14:18:00 -0000 https://novustatus.com/incident/406814#298258635d2b93d36948f034e7eece5ae91602d611736e31fb02e9f2beae51ec We are experiencing elevated 500 errors. We are investigating. Some services are down https://novustatus.com/incident/307149 Mon, 01 Jan 2024 10:00:00 -0000 https://novustatus.com/incident/307149#c973549e8984d5f601e9ed84357b2e889650d090a03e533c7ba112f7be86be4f API is back to operational state, monitoring the service. Some services are down https://novustatus.com/incident/307149 Mon, 01 Jan 2024 09:34:00 -0000 https://novustatus.com/incident/307149#5eed70359f364ea8c4f49ae2d9dba8b8fd093471fd7af36454836095a7621e55 We have identifier the issue, as fix is being deployed at the moment. Some services are down https://novustatus.com/incident/307149 Mon, 01 Jan 2024 09:20:00 -0000 https://novustatus.com/incident/307149#6ef3b80f00d5eaa2dba4acf81582d86754b2292a2e1ffdaaa5362f791bbb47eb Elevated 502 Responses from API Service API and WebSocket Service are down https://novustatus.com/incident/237837 Thu, 27 Jul 2023 02:38:57 -0000 https://novustatus.com/incident/237837#85d468d57a3e75ef41f2be0f6db1568ed5c10aefee8f43ea89439b18b10d3ecc WebSocket Service recovered. API and WebSocket Service are down https://novustatus.com/incident/237837 Thu, 27 Jul 2023 02:31:00 -0000 https://novustatus.com/incident/237837#3c4bfaec24102cb0a416b7b057295ed29909e7c472edf87464dce243f2dc1265 A fix was deployed, services fully recovered, monitoring the system behavior. API and WebSocket Service are down https://novustatus.com/incident/237837 Thu, 27 Jul 2023 01:47:52 -0000 https://novustatus.com/incident/237837#17e1354935df3c869b45c5a2b9cb291814f3d74b8d3d791c5fbfa28cb12a31b5 WebSocket Service went down. API and WebSocket Service are down https://novustatus.com/incident/237837 Thu, 27 Jul 2023 00:31:00 -0000 https://novustatus.com/incident/237837#225ebb94951e7719bf09536c78a28a1c4fcf3a44bf272bba42e3578ce41f79bb A high memory load on our Redis cluster has caused a cascading error propagating to the API and WebSocket services. Our team is working on a fix. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:49:00 -0000 https://novustatus.com/incident/223993#e1c8642dcb9bcec7d095d13cc3c1473f591124a3f9f8f70783bce6144c0a0dad The job backlog has completed processing messages received prior to the incident. Throughput and response times are back to operational status. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:49:00 -0000 https://novustatus.com/incident/223993#e1c8642dcb9bcec7d095d13cc3c1473f591124a3f9f8f70783bce6144c0a0dad The job backlog has completed processing messages received prior to the incident. Throughput and response times are back to operational status. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:15:00 -0000 https://novustatus.com/incident/223993#75cea8584fee425350e49ddbf679aff9f02dabeb6ce6127182766e506ed71b7d Service is back online and we are monitoring the system behavior while jobs are being processed from the queue. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 23:15:00 -0000 https://novustatus.com/incident/223993#75cea8584fee425350e49ddbf679aff9f02dabeb6ce6127182766e506ed71b7d Service is back online and we are monitoring the system behavior while jobs are being processed from the queue. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:15:00 -0000 https://novustatus.com/incident/223993#4ed838a27485ec6e5d71d13e1c9fcaff5b4a86aa08e9b5b236b5a157e0b3818e Novu API and system are unresponsive. Our team is monitoring and identifying the root cause. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:15:00 -0000 https://novustatus.com/incident/223993#4ed838a27485ec6e5d71d13e1c9fcaff5b4a86aa08e9b5b236b5a157e0b3818e Novu API and system are unresponsive. Our team is monitoring and identifying the root cause. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:05:00 -0000 https://novustatus.com/incident/223993#43a06cde8f826411fb87a21925d003c7ff0a7c8883b2ffa0927c29a4aeceb27b We have identified the root cause, and a fix is being deployed to the systems right now. US Cluster API failure https://novustatus.com/incident/223993 Fri, 23 Jun 2023 22:05:00 -0000 https://novustatus.com/incident/223993#43a06cde8f826411fb87a21925d003c7ff0a7c8883b2ffa0927c29a4aeceb27b We have identified the root cause, and a fix is being deployed to the systems right now. Slow response time for the management API are observed https://novustatus.com/incident/185630 Thu, 16 Mar 2023 09:51:00 -0000 https://novustatus.com/incident/185630#17b60cac740907fae53c6ec045029040669bd4a738bacfb13d74a81589dcb3bc Response times are back at normal operational rates. Slow response time for the management API are observed https://novustatus.com/incident/185630 Thu, 16 Mar 2023 09:25:00 -0000 https://novustatus.com/incident/185630#79e9ba1eecc040bbcd68c4158df633c1d62508ff0213e5e4de00f9c4f7ad3706 The component causing the degraded performance was found and a patch was issued, monitoring the response times right now. Slow response time for the management API are observed https://novustatus.com/incident/185630 Thu, 16 Mar 2023 09:17:00 -0000 https://novustatus.com/incident/185630#9bfceaf991425dff385b7fdaa933b96c4c5ab71e9ee23c8da8402d31286d3ce2 The management API is currently experiencing a slowdown in response time and timeout errors for some reported customers. Degraded API Performance for some customers https://novustatus.com/incident/152202 Tue, 10 Jan 2023 12:56:00 -0000 https://novustatus.com/incident/152202#4490e5aa79a3005ae772c5458792af60c3d5f52fe3a49b3ecb8b16a46fa4a9bc Resopnse times are stable and returned to normal operation. Degraded API Performance for some customers https://novustatus.com/incident/152202 Tue, 10 Jan 2023 11:58:00 -0000 https://novustatus.com/incident/152202#f34a3cc8c0d1ef7174c954a02618df206c885901988997aa0aedc1d09fc6f378 A caching service upgrade was issued and deployed to help with load on the affected service. Degraded API Performance for some customers https://novustatus.com/incident/152202 Tue, 10 Jan 2023 11:54:00 -0000 https://novustatus.com/incident/152202#8f1da05246d804428cc49ce9136b2fc59a5cbdc90c399fe0b3de5e7fb67967c5 Response times are back to our average standard numbers. The teams keep monitoring API response time. Degraded API Performance for some customers https://novustatus.com/incident/152202 Tue, 10 Jan 2023 11:30:00 -0000 https://novustatus.com/incident/152202#f7857773085a73bc1ba52109fa7932c4b9722f3b233440a889933c75dce25e19 For a percentage of our users, we are seeing an elevated response time for API requests. We are monitoring this right now and working on a fix.