Macroscope - Delays for Pull Requests and Webhook deliveries – Incident details

All systems operational

Delays for Pull Requests and Webhook deliveries

ResolvedMajor outage4 hours 30 minutes

Affected

Third Party

Operational from 2:52 PM to 3:15 PM, Degraded performance from 3:15 PM to 4:57 PM, Operational from 4:57 PM to 7:22 PM

GitHub Actions

Operational from 2:52 PM to 3:15 PM, Degraded performance from 3:15 PM to 4:57 PM, Operational from 4:57 PM to 7:22 PM

  • ResolvedUTC

    On October 21, 2026, between 13:22 UTC and 19:22 UTC, several GitHub services were delayed due to degradation of our internal event streaming platform.

    Pull request updates, push webhooks, and push-triggered Actions workflows were delayed by up to 33 minutes until 15:36 UTC. Actions status updates were delayed by up to 33 minutes until 16:16 UTC, holding up some merges and deployments that wait on checks. Webhook deliveries and live updates were delayed by up to 25 minutes until 16:52 UTC. Audit log events were delayed by up to two hours.

    This was due to servers in the platform running out of memory caused by client reconnections without sufficient retry backoff. We mitigated the incident by moving the busiest workloads to less-loaded servers and adding capacity.

    We have since added reconnection backoff and added server capacity. We are building a solution for auto-rebalancing connections across servers, and splitting the platform into smaller, isolated clusters to reduce blast radius and time to mitigation of issues like this one in the future.

  • UpdateUTC

    Pull request updates, Actions, and webhook deliveries have recovered. Audit log delays are now below 40 minutes, within the healthy threshold, and continue to decrease as the remaining backlog is processed.

  • UpdateUTC

    Audit log events continue to recover, with delays now below 70 minutes. We’ve applied mitigations to speed recovery and expect audit logs to fully recover in about 40 minutes. Pull request updates, Actions, and webhook deliveries remain recovered. We are monitoring progress and will share further updates.

  • UpdateUTC

    Audit log events are recovering, with delays now below 90 minutes as the remaining backlog is processed. We’ve reduced load on the event-streaming platform and are monitoring recovery. Further updates will follow.

  • UpdateUTC

    Pull request updates, Actions, and webhook deliveries have recovered. Audit log events are recovering and delays are decreasing, but we expect the remaining backlog to take approximately two to three hours to clear. We’re monitoring recovery and will share further updates.

  • UpdateUTC

    Actions and webhook deliveries have recovered. Audit log events are still delayed by about 100 minutes. We are working through the backlog, and full recovery will take some time. We will share more updates as recovery continues.

  • UpdateUTC

    Actions and Webhooks are operating normally.

  • UpdateUTC

    Audit log events continue to be delayed by up to 110 minutes. Pull request updates have recovered. Actions is recovering, and users should no longer experience impact in Actions. Webhook deliveries continue to recover as we work through backlogs. We are monitoring recovery and will share further updates as we learn more.

  • UpdateUTC

    Pull Requests is operating normally.

  • UpdateUTC

    Audit log events are delayed by up to 100 minutes, and Actions workflow status updates continue to experience extended delays. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring recovery and will share further updates as we learn more.

  • UpdateUTC

    Audit log events and Actions workflow status updates continue to be delayed. Pull request updates and webhook deliveries are recovering as we restore event processing and work through backlogs. We’re monitoring the recovery and will share further updates as we learn more.

  • UpdateUTC

    We continue to see delays affecting pull request updates, webhook deliveries, and audit log events. The event streaming platform is showing signs of recovery, and we are monitoring as the impact continues.

  • UpdateUTC

    We are investigating delays affecting pull request updates and webhook deliveries. Increased load on our event streaming platform is contributing to hotspots. We are shifting traffic away from affected areas and adding processing capacity; impact is ongoing while these mitigations take effect.

  • UpdateUTC

    Actions is experiencing degraded performance. We are continuing to investigate.

  • InvestigatingUTC

    We are investigating reports of degraded performance for Actions, Pull Requests and Webhooks