Update - New logs are appearing in the UI quickly. Live Tail and alerting continue to be halted. Archived logs may appear with delays for some customers.
May 2, 12:58 UTC

Update - New logs are appearing in the UI much quicker than before. Live Tail and alerting are still halted while we are backfilling logs. Archived logs may appear with delays for some customers.
May 2, 05:57 UTC

Update - New logs are now being made available quickly in our UI, with a small number of exceptions (~1%). Live Tail and alerting continue to be halted. We are working on sending archives to their proper destinations.
May 1, 19:17 UTC

Update - New logs are still appearing in the UI with delays. Live Tail and alerting continue to be halted while we are backfilling logs. Some customers may experience delays in excess of 72 hours for archived logs to appear in their archive destination.
May 1, 17:53 UTC

Update - New logs are still appearing in the UI with delays for some customers. Live Tail and alerting have been halted while we are backfilling logs.
May 1, 05:50 UTC

Update - Our investigations suggest this incident began with an update to our messaging bus / parsing pipeline. We've successfully rolled back to an earlier version and the process of backfilling logs for real-time search has begun. To speed up this process, we are continuing to keep Live Tail and alerting halted. Newly submitted logs are appearing, but still with significant delays. Log ingestion continues to work normally and no log lines have been lost.
Apr 30, 21:12 UTC

Update - Live Tail and alerting remain stopped as we solve this incident. New logs are still appearing in the UI but with a significant delay for all customers.
Apr 30, 12:38 UTC

Update - New logs are still appearing in the UI with delays for some customers. Live Tail and alerting have been stopped as we continue to investigate.
Apr 30, 05:39 UTC

Update - As we continue working on this issue, we have stopped Live Tail and alerting. New logs are still appearing in the UI, but with delays.
Apr 29, 18:42 UTC

Update - We continue to work on restoring the service to full operation. Users will still continue to experience delays in when searching views, boards and screens. Some users may not see live tail, neither receive new alerts or getting new logs.
Apr 29, 09:01 UTC

Update - As we continue working on this issue, we have temporarily scaled down live-tail and alerting. Some customers may not see live tail and receive new alerts. New logs still appear in the UI with delays.
Apr 29, 02:43 UTC

Update - We continue to work on restoring the search service to full functionality. All users will continue to see delays of greater than 30 minutes when searching, using views, boards and screens.
Apr 28, 22:31 UTC

Update - We continue to work through completing the aforementioned procedure that is still affecting a small percentage of customers. All users will continue to see delays of greater than 30 minutes when searching, using views, boards and screens.
Apr 28, 19:24 UTC

Update - We’ll shortly perform a procedure that will cause all logs to be unavailable to a small percentage of customers for a short period of time. All users will continue to see delays of greater than 30 minutes when searching, using views, boards and screens.
Apr 28, 16:36 UTC

Update - We continue to work on restoring the service to full operation. At this time users will still continue to experience delays of greater than 30 minutes when searching views, boards and screens.
Apr 28, 15:00 UTC