Indexing is back to normal. We will be posting a post-mortem regarding this incident in the near future, detailing what happened, how we addressed it, lessons learned, and how we can improve to better serve our customers. Thank you for your patience.
Posted Jun 13, 2020 - 18:27 UTC
Monitoring
Indexing of current logs has been working as expected for the last several hours. We still have a backlog of logs that were ingested, but not indexed. That should be processed within the next three-to-four hours. Thanks again for your patience.
Posted Jun 13, 2020 - 06:11 UTC
Identified
We are again experiencing indexing delays and working toward resolving the issue as soon as possible. Searches, boards, and screens are affected. Live tail and alerting are both fully functional. We will update as we make progress on this issue.
Posted Jun 12, 2020 - 18:13 UTC
Monitoring
The indexing issue has been resolved and the backlog has been processed. We will continue to monitor and will update this page if there are any changes. In the near future, we'll be posting a public-facing post-mortem, detailing what happened, how we resolved it, what we learned, and where we can improve. Thank you all for your patience!
Posted Jun 12, 2020 - 08:39 UTC
Update
We are still experiencing indexing delays and there is also a backlog of previously ingested logs. We do not have an estimate as to when the issue will be resolved. But we'll update when we have a solution in place, or an estimate for a fix. Thanks for your patience.
Posted Jun 11, 2020 - 23:23 UTC
Update
We are currently experiencing indexing delays as well as working through an indexing backlog. We'll update as we work toward a solution. Do note that live tail and alerting are not affected as a result of the issue.
Posted Jun 11, 2020 - 11:12 UTC
Update
At this point, currently ingested logs are being indexed with no delay. However there's a backlog of logs that were ingested during the previous few hours, but not indexed. That backlog is dropping--but you might still get inaccurate search results, boards and screens until the backlog has been indexed, which will be in several hours. As we have better estimates, we'll update. Thanks again for your patience.
Posted Jun 10, 2020 - 03:27 UTC
Update
We are still working on resolving the indexing delay. As before, search, boards, and screens are affected. Live tail and alerting are fully functional.
We will update when the issue has been resolved.
Posted Jun 09, 2020 - 19:34 UTC
Update
We are still working on resolving the indexing delay. As before, search, boards, and screens are affected. Live tail and alerting are fully functional.
Posted Jun 09, 2020 - 16:19 UTC
Identified
We are again experiencing an indexing delay. As before, search, boards, and screens are affected. Live tail and alerting are fully functional.
Posted Jun 09, 2020 - 14:42 UTC
Monitoring
Indexing backlog has been processed and all functionality has been restored. We will monitor to ensure the issue is addressed.
Posted Jun 09, 2020 - 09:56 UTC
Update
Indexing backlog is reducing. You might see some recently ingested logs indexed and searchable, while logs ingested a few hours ago might not yet be indexed. We will provide an update when we are "caught up" with indexing.
Posted Jun 08, 2020 - 23:23 UTC
Update
We are still working towards a resolution. In addition to search, boards and graphs are affected as well. However, alerting and live tail are fully functional.
Posted Jun 08, 2020 - 22:30 UTC
Update
We are still working towards a resolution. In addition to search, boards and graphs are affected as well. However, alerting and live tail are fully functional.
Posted Jun 08, 2020 - 21:11 UTC
Update
We are still working towards a resolution. In addition to search, boards and graphs are affected as well. However, alerting and live tail are fully functional.
Posted Jun 08, 2020 - 20:07 UTC
Update
We are still working towards a resoultion. In addition to search, boards and graphs are affected as well. However, alerting and live tail are fully functional.
Posted Jun 08, 2020 - 19:05 UTC
Identified
We have identified the issue and working towards a resoultion. In addition to search, boards and graphs are affected as well. However, alerting and live tail are fully functional.
Posted Jun 08, 2020 - 17:24 UTC
Update
We are still investigating the issue. In addition to search, boards and graphs are affected as well. However, alerting and live tail are fully functional.
Posted Jun 08, 2020 - 16:16 UTC
Update
We are still investigating the issue. In addition to search, boards and graphs are affected as well. However, alerting is fully functional.
Posted Jun 08, 2020 - 15:03 UTC
Investigating
We are currently experiencing a delay on searching of newly ingested logline of around 2 hours. We are investigating and working quickly to mitigate the issue.