tag:status.mezmo.com,2005:/historyMezmo Status Page Status - Incident History2024-03-29T05:23:51ZMezmo Status Pagetag:status.mezmo.com,2005:Incident/193119122023-12-04T13:19:26Z2023-12-08T09:43:00ZIntermittent user session timeouts, requiring periodic re-authentication<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>13:19</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved, and no further issues have been observed with user sessions.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>12:13</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix for the user session timeouts on the Web UI, but will continue to monitor the situation closely.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>12:06</var> UTC</small><br><strong>Investigating</strong> - The Web UI is currently encountering user session timeouts, prompting customers to log in every 1-2 minutes. Our team is actively investigating the root cause of this issue, while the remaining aspects of the service remain fully functional.</p>tag:status.mezmo.com,2005:Incident/183120562023-08-30T00:17:53Z2023-09-06T00:40:30ZWeb UI is unresponsive and ingestion of log lines halted<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>00:17</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Update</strong> - The webUI is loading consistently now, but we are still investigating.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>21:01</var> UTC</small><br><strong>Investigating</strong> - Our WebUI is not loading pages consistently. We are investigating. [Reference #3204]</p>tag:status.mezmo.com,2005:Incident/176192662023-06-19T13:58:04Z2023-12-04T13:20:31ZUser sessions are timing out and customers are required to login again<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>13:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:27</var> UTC</small><br><strong>Monitoring</strong> - The fix was implemented and we are now monitoring the user login sessions.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Identified</strong> - The issue has been identified, and a fix is being implemented.</p><p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>11:09</var> UTC</small><br><strong>Investigating</strong> - User sessions to our Web UI are timing out and customers using the UI have to log in every 1-2 minutes. We are investigating why this is happening, but the rest of the service is fully functional. No other components are affected.</p>tag:status.mezmo.com,2005:Incident/171129202023-05-01T20:27:34Z2023-05-08T19:13:21ZThe Web UI is not accessible<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>20:27</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>20:18</var> UTC</small><br><strong>Identified</strong> - The Web UI is not accessible.</p>tag:status.mezmo.com,2005:Incident/160928782023-02-10T18:14:13Z2023-02-21T23:28:00ZSearches are running slowly<p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>18:14</var> UTC</small><br><strong>Resolved</strong> - Searches are running at normal speeds again. All services are fully operational.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>18:11</var> UTC</small><br><strong>Identified</strong> - Searches are running slowly. We have identified the cause and are implementing a fix.<br />(Reference # 3018)</p>tag:status.mezmo.com,2005:Incident/160457792023-02-07T02:07:01Z2023-02-07T23:45:29ZIntermittent delays loading Web UI and running searches<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>02:07</var> UTC</small><br><strong>Resolved</strong> - All data is being returned in search results. All services are fully operational.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>00:16</var> UTC</small><br><strong>Update</strong> - Web UI pages are loading at normal speeds and searches are returning quickly again. A small amount of data (<2%) is not being returned in search results. We are working to restore access to the results.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>22:26</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this incident.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>20:48</var> UTC</small><br><strong>Identified</strong> - We are seeing intermittent delays loading the Web UI and running searches. We are taking remedial action.</p>tag:status.mezmo.com,2005:Incident/114782522022-10-05T16:05:01Z2022-10-12T18:53:39ZDegraded performance for WebUI, Ingestion, Alerting, Searching, Live Tail, Graphing, and Timelines<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All services are fully operational.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>14:58</var> UTC</small><br><strong>Monitoring</strong> - Service is restored but we are still monitoring.</p>tag:status.mezmo.com,2005:Incident/104463882022-06-30T22:11:44Z2022-07-01T22:47:57ZSome customers' logs are not currently being processed<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>22:11</var> UTC</small><br><strong>Resolved</strong> - Newly submitted logs are now being processed and retained. Some logs submitted by some customers during the incident were discarded and not successfully retained. [Reference #2792]</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>21:36</var> UTC</small><br><strong>Investigating</strong> - Some logs submitted to our service in the last 1.5 hours have not been processed. We are taking remedial action now.</p>tag:status.mezmo.com,2005:Incident/101354582022-06-02T22:05:42Z2022-06-03T22:51:10ZLog lines not ingested<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>22:05</var> UTC</small><br><strong>Resolved</strong> - Ingestion has resumed. All services are fully operational.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>21:44</var> UTC</small><br><strong>Identified</strong> - Ingestion of log lines has halted from all sources. We are taking remedial action.</p>tag:status.mezmo.com,2005:Incident/97154182022-04-05T18:32:44Z2022-05-23T17:13:55ZAlerts not triggering<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>18:32</var> UTC</small><br><strong>Resolved</strong> - Alerts of all types have resumed. Alerts are fully operational.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>18:23</var> UTC</small><br><strong>Investigating</strong> - Currently, alerts of all types are not triggering. We are taking remedial action.</p>tag:status.mezmo.com,2005:Incident/94169802022-02-28T18:58:22Z2022-05-23T17:13:55ZIngestion of new logs to our Syslog endpoint – for logs sent using a custom port, only – is intermittently delayed<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>18:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:15</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented for the ingestion of new logs to our Syslog endpoint using a custom port. We will continue to monitor the results.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>19:08</var> UTC</small><br><strong>Update</strong> - Ingestion of new logs to our Syslog endpoint using a Custom Port is still intermittently failing. We are continuing to work on a fix.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>19:51</var> UTC</small><br><strong>Identified</strong> - Ingestion of new logs to our Syslog endpoint using a Custom Port is intermittently failing.</p>tag:status.mezmo.com,2005:Incident/94029712022-02-24T23:43:32Z2022-05-23T17:13:55ZIngestion of new logs — for Syslog only - Partial Outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>23:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. Please reach out to us at support@logdna.com with any additional questions.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>22:45</var> UTC</small><br><strong>Identified</strong> - New logs — from Syslog only -- are intermittently not being ingested by our service. We are working to restore this functionality as soon as possible.</p>tag:status.mezmo.com,2005:Incident/93478412022-02-18T02:15:41Z2022-05-23T17:13:55ZIngestion of new logs — for Syslog only - is intermittently failing<p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>02:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. If your team is still unable to send logs via syslog, please let us know at support@logdna.com</p><p><small>Feb <var data-var='date'>18</var>, <var data-var='time'>01:41</var> UTC</small><br><strong>Investigating</strong> - Ingestion of new logs to our Syslog endpoint is intermittently failing. We are investigating.</p>tag:status.mezmo.com,2005:Incident/93357742022-02-16T21:56:32Z2022-05-23T17:13:55ZSome logs submitted over syslog using custom ports are not being correctly formatted and are not available for Alerting, Searching, Timelines, Graphs, and Live Tail.<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>21:56</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved. If you continue to experience issues with unparsable log lines that show the error “Unidentifiable Syslog Source” and “Unsupported Syslog format”, please contact Support.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>21:17</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>19:58</var> UTC</small><br><strong>Investigating</strong> - Some logs being sent to our service over syslog using custom ports are not being correctly parsed and are not available for Alerting, Searching, Timelines, Graphs, and Live Tail. Unparsable log lines will show the error “Unidentifiable Syslog Source” and “Unsupported syslog format.” Logs being sent over syslog that do not use custom ports are working normally.</p>tag:status.mezmo.com,2005:Incident/93287902022-02-16T17:20:30Z2022-02-16T17:20:30ZPlanned Maintenance of LogDNA Web UI: February 16th, 16:00 UTC to 17:30 UTC<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>17:20</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>02:36</var> UTC</small><br><strong>Scheduled</strong> - Our WebUI will be under maintenance on February 16th, 2022 from 16:00 UTC to 17:30 UTC. Please avoid making any configuration changes during this time period – such as to Views, Boards, Alerts, etc – as they will not be permanent.<br /><br />Additionally, for a five minute period towards the end of the window, alerts will not fire and be discarded.</p>tag:status.mezmo.com,2005:Incident/92655412022-02-08T14:40:57Z2022-05-23T17:13:55ZDelays in Alerting, Searching, Live Tail, Graphing, and Timelines. WebUI intermittently unavailable.<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:25</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring the results. Newly sent logs are being processed again with minimal delays and all services are operational.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>13:55</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue. The web app can be accessible intermittently. Logs can arrive with a delay which will impact searching and alerting.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>13:38</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.mezmo.com,2005:Incident/91690022022-01-26T17:15:55Z2022-05-23T17:13:55ZIngestion, Searching, Live Tail, Alerting, Graphing, and Timelines Delays<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>17:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All services are operational.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring the results. Logs are being ingested again and all services are operational.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>16:23</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>16:10</var> UTC</small><br><strong>Investigating</strong> - Ingestion services are currently halted. Customers will also experience delays with Searching, Live Tail, Alerting, Graphing, and Timelines.</p>tag:status.mezmo.com,2005:Incident/91311702022-01-20T21:38:04Z2022-05-23T17:13:55ZMany services briefly halted due to Cloud Provider incident<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>21:38</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All services are operational.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>21:07</var> UTC</small><br><strong>Monitoring</strong> - Logs are being ingested again without delays. All services are working normally. We will monitor until our Cloud Provider closes their incident.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Investigating</strong> - Our Cloud Provider Equinix is having an incident (see https://status.equinixmetal.com/incidents/gjmh37y6rkjp). For about 5-10 minutes, ingestion was halted and the WebUI was not responsive. Some alerts may have not been triggered. Currently all services are working and there are some delays in processing recently sent logs. We are monitoring Equinix’s incident closely.</p>tag:status.mezmo.com,2005:Incident/91155512022-01-19T01:21:28Z2022-05-23T17:13:55ZWeb UI and Ingestion are intermittently unavailable. Alerting is halted and new Live Tail sessions can't be started.<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>01:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All services are fully operational.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>23:45</var> UTC</small><br><strong>Identified</strong> - We've identified the source of the failure and are taking action to correct it. The WebUI continues to be unavailable at times for intervals of 1-2 minutes each.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Investigating</strong> - Our WebUI is under maintenance and may not load pages consistently. We are working to recover as soon as possible.</p>tag:status.mezmo.com,2005:Incident/89938832022-01-03T20:21:45Z2022-05-23T17:13:56ZLive Tail and Searching Delays<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>20:21</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>19:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring searching and live tail performance.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>19:12</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing delays for live tail and searching of newly ingested log data.</p>tag:status.mezmo.com,2005:Incident/86728252021-11-24T17:00:24Z2022-05-23T17:13:56ZAlerting, Searching, Live Tail, Graphing, and Timelines Delays<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All services are fully operational.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>06:57</var> UTC</small><br><strong>Update</strong> - Our services are recovering and there may be some delays in Alerting, Searching, Live Tail, Graphing, and Timelines. We are monitoring.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>20:03</var> UTC</small><br><strong>Update</strong> - Delays are still being experienced by some customers. We continue to work towards a solution.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>16:42</var> UTC</small><br><strong>Investigating</strong> - Some customers are experiencing delays in Alerting, Searching, Live Tail, Graphing, and Timelines. We are investigating and working to mitigate the issue.</p>tag:status.mezmo.com,2005:Incident/86650222021-11-23T02:04:50Z2022-05-23T17:13:56ZIndexing, Livetail Performance, Search, and Alerting Delays<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>02:04</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. All services are fully operational.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>00:51</var> UTC</small><br><strong>Monitoring</strong> - Our services are mostly back to normal; we are monitoring.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>20:54</var> UTC</small><br><strong>Update</strong> - We are still actively investigating and working on a fix for the issue.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>19:53</var> UTC</small><br><strong>Update</strong> - There continue to be delays with processing newly sent log lines. Additionally, some alerts are not triggering.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>19:01</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing delays in searching for newly ingested log data, livetail, and alerts at this time. We are investigating and working quickly to mitigate the issue.</p>tag:status.mezmo.com,2005:Incident/84272722021-11-09T00:00:00Z2022-05-23T17:13:56ZWeb App is Unreachable<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>Resolved</strong> - We experienced a temporary network issue that affected our web app at 00:00 UTC. Our team has taken remedial action to bring our services back to normal. All services are operational.</p>tag:status.mezmo.com,2005:Incident/83416092021-10-28T23:22:31Z2022-05-23T17:13:56ZEmails Are Not Being Sent<p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>23:22</var> UTC</small><br><strong>Resolved</strong> - Our email alerting feature has been restored to normal operation. All services are fully functional.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>21:02</var> UTC</small><br><strong>Identified</strong> - Our email provider reports that outbound message delivery has resumed but it is not yet fully operational. Our provider will keep unsent emails in their queue and continue to try to send them.</p><p><small>Oct <var data-var='date'>28</var>, <var data-var='time'>18:13</var> UTC</small><br><strong>Investigating</strong> - Our email alerting feature is not working at the moment and customers are not receiving alerts by email in US region. This is due to an ongoing incident with our email provider; see https://status.sparkpost.com/incidents/bwl8dr6gwmts?u=ydzrh5x205pf for more detail. Other types of alerts, such as Slack and webhook, are still working. We are investigating.</p>tag:status.mezmo.com,2005:Incident/81779572021-10-07T18:46:35Z2022-05-23T17:13:56ZSome customers are not able to login to our UI<p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>18:46</var> UTC</small><br><strong>Resolved</strong> - Logins are working for all customers. All services are operational.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>18:02</var> UTC</small><br><strong>Monitoring</strong> - Logins to our UI appear to be working again for all customers. We are monitoring for any further failures.</p><p><small>Oct <var data-var='date'> 7</var>, <var data-var='time'>17:52</var> UTC</small><br><strong>Identified</strong> - Some customers are not able to login to our UI. It appears this is due to an incident with our cloud provider Equinix. See their status page https://status.equinixmetal.com/incidents/wgg6kl862tl6.</p>