ergeon - Service Disruption – Incident details

Service Disruption

ResolvedPartial outage24 hours 27 minutes

Affected

Core Infrastructure

Partial outage from 1:26 AM to 1:54 AM

Public Admin

Partial outage from 1:26 AM to 1:54 AM

  • ResolvedUTC
    This incident has been automatically resolved after 24 hours. The None service appears to have recovered or the issue is no longer being actively reported.
  • UpdateUTC
    We identified excessive staging events as the cause of the Sentry ingestion disruption and have stopped the main sources of the event flood. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
  • UpdateUTC
    We identified excessive staging alerts as the cause of the Sentry ingestion disruption and have stopped the alert flood while restoring error ingestion. The team is working on a lasting fix and monitoring the system.
  • UpdateUTC
    We identified excessive staging error events as the cause of the Sentry ingestion disruption and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
  • UpdateUTC
    We identified excessive staging alerts as the cause of the Sentry ingestion disruption and have stopped the main sources of the alert flood. The team is working on a lasting fix and monitoring error reporting; production systems were not affected.
  • UpdateUTC
    We traced the issue to excessive staging events that exhausted our error-monitoring quota; production was not affected. The event floods have been stopped, and we’re working on a lasting fix while monitoring error reporting.
  • UpdateUTC
    We identified excessive staging alerts as the cause of the Sentry ingestion disruption; production was not affected. The alert flood has been stopped, and we’re working on a lasting fix while monitoring error reporting.
  • UpdateUTC
    We identified excessive staging alerts as the cause of the Sentry quota issue and have stopped the main sources of the alert flood. The team is working on a lasting fix and monitoring error reporting; production processing remains enabled.
  • UpdateUTC
    We identified excessive staging error events as the cause of the Sentry quota issue and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production processing remains enabled.
  • UpdateUTC
    We identified excessive staging error events as the cause of the Sentry quota issue and have stopped the main sources of the event flood. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
  • UpdateUTC
    We traced the issue to excessive Sentry events from staging systems; production was not affected. The event flood has been stopped, and we’re working on a lasting fix while monitoring error reporting.
  • UpdateUTC
    We identified excessive staging events as the cause of the Sentry ingestion disruption and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production services were not affected.
  • UpdateUTC
    We identified excessive staging events as the cause of the Sentry quota being reached and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production was not affected.
  • UpdateUTC
    We identified excessive staging events as the cause of the Sentry quota issue and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production was not affected.
  • UpdateUTC
    We identified excessive staging events as the cause of the Sentry quota issue and have stopped the event floods. The team is working on a lasting fix and monitoring error reporting; production was not affected.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • UpdateUTC
    We are continuing to investigate the issue and will provide updates as available.
  • InvestigatingUTC
    *INCIDENT - Hit Sentry quota since yesterday and no errors have been received since* This is causing us to be system blind, <@U055DA0L5MY> suggestion: ```can someone with sentry admin rate limit or drop those two projects and then bump the on demand budget so we start getting errors again? the loop itself needs a fix too, it should give up on channel_not_found instead of retrying forever. happy to open the ticket.``` cc