Macroscope - Project updates delayed – Incident details

All systems operational

Project updates delayed

ResolvedDegraded performance11 hours 2 minutes

Affected

Third Party

Operational from 3:28 PM to 4:26 PM, Degraded performance from 3:28 PM to 4:41 PM, Operational from 4:35 PM to 2:30 AM, Degraded performance from 4:41 PM to 5:41 PM, Operational from 5:41 PM to 2:30 AM

GitHub API Requests

Operational from 3:28 PM to 4:26 PM, Degraded performance from 4:26 PM to 4:35 PM, Operational from 4:35 PM to 2:30 AM

GitHub Actions

Degraded performance from 3:28 PM to 5:41 PM, Operational from 5:41 PM to 2:30 AM

  • ResolvedUTC

    This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

  • MonitoringUTC

    The degradation has been mitigated. We are monitoring to ensure stability.

  • UpdateUTC

    GitHub Projects update processing has improved, but newly added items may still take longer than expected to appear.

    We're continuing to work on the remaining delays and monitor recovery.

  • UpdateUTC

    Some GitHub Projects updates, including newly added items and changes to issues and pull requests, are still delayed.

    We have seen some recovery, and are working to add more capacity to accelerate the recovery.

  • UpdateUTC

    Some GitHub Projects updates, including newly added items and changes to issues and pull requests, remain delayed.

    We've identified interruptions in update processing and are evaluating mitigations to restore normal operation.

  • UpdateUTC

    Some GitHub Projects updates related to issue changes and merged pull requests are still delayed. We're investigating the remaining delays and working to restore normal processing.

  • UpdateUTC

    Some updates in GitHub Projects are still delayed. We're continuing mitigation work and monitoring.

  • UpdateUTC

    The other affected services have recovered.

    We're continuing to investigate GitHub Projects and verify that normal operation has been restored.

  • UpdateUTC

    We're seeing recovery across the affected services. GitHub Projects is still recovering as queued updates are processed, and we're continuing to monitor for stability.

  • UpdateUTC

    Actions, Issues, Packages and Pull Requests are operating normally.

  • UpdateUTC

    We're seeing recovery after adding capacity, with errors decreasing. Some users may still experience errors loading Issues, Pull Requests and search results on github.com.

    We're continuing to work towards full recovery and monitoring for stability.

  • UpdateUTC

    Some users continue to experience errors loading Issues, Pull Requests, package listings and search results on github.com.

    We're working through multiple mitigation efforts and continuing to investigate.

  • UpdateUTC

    API Requests is operating normally.

  • UpdateUTC

    Packages is experiencing degraded performance. We are continuing to investigate.

  • UpdateUTC

    API Requests is experiencing degraded performance. We are continuing to investigate.

  • UpdateUTC

    Some users continue to experience intermittent errors loading Issues, Pull Requests and search results on github.com.

    We're investigating and evaluating mitigation options to restore normal service.

  • UpdateUTC

    Some users may experience intermittent errors loading Issues, Pull Requests and search results on github.com.

    We're investigating and working to restore normal service.

  • InvestigatingUTC

    We are investigating reports of degraded performance for Actions, Issues and Pull Requests