Indexing delay
Incident Report for Mezmo Status Page
Resolved
This incident has been resolved.
Posted Mar 15, 2020 - 22:28 UTC
Update
We are continuing to closely monitor the queues to make sure an indexing delay does not return postfix. We will update within a few hours.
Posted Mar 15, 2020 - 16:48 UTC
Update
Update: Archiving delay is actively being reduced and should be back to normal shortly. All indexing, alerting, graphing and screens are back to normal.
Posted Mar 15, 2020 - 04:44 UTC
Update
Update: TCP syslog ports are available as of now. Livetail and alerting is running fine. Indexing (searching) and archiving are still delayed. Infra team is performing corrective procedures to reduce the batches backlog at this time.
Posted Mar 14, 2020 - 18:25 UTC
Update
Update: TCP syslog ports are available as of now. Archiving is slightly delayed. Indexing (search, alerting and graphs) are working correctly at this time. Archiving at this time is still delayed.
Posted Mar 14, 2020 - 15:16 UTC
Update
A fix has been implemented which restored customers who are sending Syslog via TCP to provisioned ports. We will continue to monitor any further issues.
Posted Mar 14, 2020 - 06:54 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 13, 2020 - 21:37 UTC
Monitoring
A fix has been implemented and we are currently monitoring. There is still an indexing delay affecting a subset of customers. This delay will result in the inability to search on recently ingested data and alerts will be slightly delayed as the data is being inserted on the backend. Additionally, customers sending syslog via TCP to provisioned ports, will experience issues with ingestion and possible log loss.

We will be closely monitoring this issue closely and will have a post-mortem written up within 7 business days.
Posted Mar 13, 2020 - 17:12 UTC
Update
We are continuing to work on a fix for this issue.
Posted Mar 13, 2020 - 13:03 UTC
Identified
The issue was not fully mitigated. There is currently an indexing delay greater than 30 minutes or longer at this time which will affect searching, alerting, and graphing.
Posted Mar 13, 2020 - 12:29 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 13, 2020 - 09:45 UTC
Monitoring
Currently, there is an indexing delay greater than 30 minutes present at this time. Customers might see minor delays in search, alerting, graphing. The issue is has been mitigated and we are currently monitoring the backlog.
Posted Mar 13, 2020 - 09:45 UTC
This incident affected: Log Analysis (Log Ingestion (Agent/REST API/Code Libraries), Log Ingestion (Heroku), Log Ingestion (Syslog), Search, Alerting, Livetail, Archiving).