This incident has been automatically resolved after 24 hours. The None service appears to have recovered or the issue is no longer being actively reported.
Update · UTC
We identified excessive staging events as the cause of the Sentry ingestion disruption and have stopped the main sources of the event flood. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
Update · UTC
We identified excessive staging alerts as the cause of the Sentry ingestion disruption and have stopped the alert flood while restoring error ingestion. The team is working on a lasting fix and monitoring the system.
Update · UTC
We identified excessive staging error events as the cause of the Sentry ingestion disruption and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
Update · UTC
We identified excessive staging alerts as the cause of the Sentry ingestion disruption and have stopped the main sources of the alert flood. The team is working on a lasting fix and monitoring error reporting; production systems were not affected.
Update · UTC
We traced the issue to excessive staging events that exhausted our error-monitoring quota; production was not affected. The event floods have been stopped, and we’re working on a lasting fix while monitoring error reporting.
Update · UTC
We identified excessive staging alerts as the cause of the Sentry ingestion disruption; production was not affected. The alert flood has been stopped, and we’re working on a lasting fix while monitoring error reporting.
Update · UTC
We identified excessive staging alerts as the cause of the Sentry quota issue and have stopped the main sources of the alert flood. The team is working on a lasting fix and monitoring error reporting; production processing remains enabled.
Update · UTC
We identified excessive staging error events as the cause of the Sentry quota issue and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production processing remains enabled.
Update · UTC
We identified excessive staging error events as the cause of the Sentry quota issue and have stopped the main sources of the event flood. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
Update · UTC
We traced the issue to excessive Sentry events from staging systems; production was not affected. The event flood has been stopped, and we’re working on a lasting fix while monitoring error reporting.
Update · UTC
We identified excessive staging events as the cause of the Sentry ingestion disruption and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
Update · UTC
We identified excessive staging events as the cause of the Sentry quota being reached and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production was not affected.
Update · UTC
We identified excessive staging events as the cause of the Sentry quota issue and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production was not affected.
Update · UTC
We identified excessive staging events as the cause of the Sentry quota issue and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production was not affected.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Update · UTC
We are continuing to investigate the issue and will provide updates as available.
Investigating · UTC
*INCIDENT - Hit Sentry quota since yesterday and no errors have been received since*
This is causing us to be system blind, <@U055DA0L5MY> suggestion:
```can someone with sentry admin rate limit or drop those two projects and then bump the on demand budget so we start getting errors again? the loop itself needs a fix too, it should give up on channel_not_found instead of retrying forever. happy to open the ticket.```
cc