Skip to main content
Skip to content

GitHub Actions failures from a stalled notification task

If GitHub Actions stops working for repositories and organizations that have not used it before, or configuration runs fail while they update GitHub Actions, a background task in one of the GitHub Actions services may have stalled.

Symptoms

You may see one or more of the following problems. Other issues can result in the same problems, so use the checks in Confirming the problem to confirm that a stalled notification task is the cause.

  • In repositories that have not used GitHub Actions before, such as new repositories, workflows do not run. The repository's GitHub Actions settings page, and the page for adding a self-hosted runner, return a 500 error. Repositories that already use GitHub Actions usually keep working.
  • In organizations that have not used GitHub Actions before, the organization's GitHub Actions settings, runners, and runner groups pages return a 500 error. If your enterprise has not used GitHub Actions before, the enterprise runners and runner groups pages can fail in the same way.
  • An upgrade or hotpatch, or another configuration run that updates GitHub Actions, pauses for 15 minutes or more, then fails. The configuration log, /data/user/common/ghe-config.log, contains Failed to update the following host process. This error can also have other causes, especially during an upgrade.
  • The system log gets a new message about every second, saying that a SqlNotificationBackgroundWork task is still running past its interval time. Stalled tasks in job agent processes do not log this message.

ghe-actions-check can report that every service is healthy while this problem is happening, so a healthy result does not rule it out.

Causes

GitHub Actions sets up storage for each repository, user, organization, and enterprise the first time GitHub Actions is used there. This storage holds runner registrations and the other settings that GitHub Actions needs.

The GitHub Actions services run as several processes. Each process runs background tasks that check notification queues in the GitHub Actions databases about once a second, and acknowledge the changes they find. Some operations wait until other processes acknowledge a change.

  • Setting up storage for the first time waits for the two processes of the main GitHub Actions service, actions-api-server and actions-job-agent.
  • A configuration run that updates GitHub Actions waits for the job agent processes, whose names end in -job-agent.

If a background task in one of these processes stops working, because it hangs or because it stops running, that process stops acknowledging changes, so the operations that wait for it fail. Most other GitHub Actions activity does not depend on the queues, so the problem can go unnoticed for days, until a new repository needs GitHub Actions or a configuration run updates it.

In the cases that GitHub has investigated, the task did not recover by itself. Nothing in GitHub Actions restarts it, and ghe-actions-check does not detect the problem. Restarting the GitHub Actions services starts new processes with new background tasks, which fixes it. The reason the task stalls is not known, so the problem can happen again.

Confirming the problem

Each GitHub Actions process logs a heartbeat about every 5 minutes while its notification tasks are working. When a task stalls in a process whose name ends in -api-server, that process also logs a stalled-task message about once a second. Processes whose names end in -job-agent never log stalled-task messages, so for those processes, only the heartbeats show the problem.

  1. SSH into the administrative shell. In a high availability configuration, connect to the primary. In a cluster, run these commands on each node that runs GitHub Actions. To list those nodes, run ghe-cluster-nodes --role actions. For more information, see Acceder al shell administrativo (SSH).

  2. For each GitHub Actions process, count the heartbeats and the stalled-task messages from the last 15 minutes. If you started or restarted GitHub Actions less than 15 minutes ago, wait until 15 minutes have passed first.

    for p in actions-api-server actions-job-agent mps-api-server mps-job-agent token-api-server token-job-agent artifactcache-api-server artifactcache-job-agent; do
      log=$(sudo journalctl -t "$p" --since '15 min ago' --no-pager)
      hb=$(grep -c 'Deployment Notification Heartbeat' <<< "$log")
      case "$p" in
        *-job-agent) st="n/a" ;;
        *) st=$(grep -c 'SqlNotificationBackgroundWork.*still running past its interval' <<< "$log") ;;
      esac
      echo "$p: $hb heartbeats, $st stalled-task messages"
    done
    

    The output looks like this. In this example, the notification task in actions-api-server has stalled.

    actions-api-server: 0 heartbeats, 900 stalled-task messages
    actions-job-agent: 3 heartbeats, n/a stalled-task messages
    mps-api-server: 3 heartbeats, 0 stalled-task messages
    mps-job-agent: 3 heartbeats, n/a stalled-task messages
    token-api-server: 3 heartbeats, 0 stalled-task messages
    token-job-agent: 2 heartbeats, n/a stalled-task messages
    artifactcache-api-server: 3 heartbeats, 0 stalled-task messages
    artifactcache-job-agent: 3 heartbeats, n/a stalled-task messages
    
  3. Use this table to read the result for each process.

    Result for a processWhat it means
    2 or more heartbeats, and no more than a few stalled-task messagesThe process is working. A single slow check can log a few stalled-task messages.
    0 heartbeats, while other processes show 2 or moreThe notification task in this process has stalled.
    About 900 or more stalled-task messages, which is about one a secondThe notification task in this process has stalled.
    1 heartbeat, or more than a few but fewer than about 900 stalled-task messagesNot clear yet. Wait 15 minutes, then run the commands again.
    0 heartbeats for every processGitHub Actions is not running on this node. Check that you are connected to the primary, or to a cluster node that runs GitHub Actions.
    0 heartbeats for only the two artifactcache processesThis is expected if you have disabled the GitHub Actions cache service. For more information, see Utilidades de la línea de comandos.
  4. Check that the stalled processes match your symptoms.

    • The 500 errors happen when actions-api-server or actions-job-agent has stalled.
    • Configuration runs fail when a process whose name ends in -job-agent has stalled.

    If no process has stalled, or the stalled processes do not match your symptoms, a stalled notification task is probably not the cause. See Contacting Soporte de GitHub.

  5. Optionally, to find out when the task in a process whose name ends in -api-server stalled, view its most recent stalled-task message. Replace PROCESS-NAME with the name of the process.

    sudo journalctl -t PROCESS-NAME --since '15 min ago' --no-pager | grep 'SqlNotificationBackgroundWork.*still running past its interval' | tail -n 1
    

    The output looks like this. Each message is one long line, shortened here.

    Mar 10 09:15:42 github-example-com-primary actions-api-server[2817]: {"logType": "ProductTrace", "logData": {"PreciseTimeStamp":"2026-03-10 09:15:42.1234","Level":"2", ... "Message":"Background task adcceb44-740e-44e8-a2cf-95c1d38db76c:Microsoft.TeamFoundation.Framework.Server.NotificationProcessor+SqlNotificationBackgroundWork:9442d144-43fd-4fe3-bdca-e460fdee6c01 is still running past its interval time and thus is delaying its next execution.\nThe next execution was supposed to occur 2.10:34:37.0512746 ago at 03/07/2026 22:41:05.", ... }}
    

    The date and time after ago at are about when the task stalled. They are in UTC, with the date in month/day/year order. The value before them shows how long ago that was, as DAYS.HOURS:MINUTES:SECONDS. The days part only appears if it was one day or more ago.

Resolving the problem

Restart the GitHub Actions services. In a high availability configuration, run every command on the primary. In a cluster, run them on the primary MySQL node, which is typically the node designated as mysql-master in the cluster configuration file. For more information, see Inicializando el clúster.

If a configuration run is in progress, wait for it to finish or fail before you start. A run that this problem blocks fails after 15 minutes or more.

If you plan to contact Soporte de GitHub about this problem, read Contacting Soporte de GitHub before you restart.

Advertencia

Stopping the GitHub Actions services makes GitHub Actions unavailable until you start them again. Jobs that are running when you stop the services can fail. If you can, restart the services when few workflows are running. After the restart, re-run any jobs that failed. Self-hosted runners reconnect on their own.

  1. SSH into the administrative shell.

  2. Check that the GitHub Actions services and their storage are otherwise healthy.

    ghe-actions-check
    ghe-actions-check -s blob
    

    When storage is healthy, the last line of the output from the second command is Blob Storage is healthy!. If either command reports that a service failed to become healthy, or reports Storage tests failed!, do not continue. Contact Soporte de GitHub instead. ghe-actions-start runs the same storage check, so it would fail too.

  3. Stop the GitHub Actions services.

    ghe-actions-stop
    

    For each service, the output includes a line that starts with Attempting to stop. If the output includes Failed to stop, continue to the next step to start the services again, then contact Soporte de GitHub.

  4. Start the GitHub Actions services.

    ghe-actions-start
    

    This can take several minutes. Do not interrupt it. When it finishes, the last line of the output is Actions was enabled!. This means that the services are starting, not that they are ready, so check them in the next step. If the command ends with an error instead, GitHub Actions may stay unavailable. Contact Soporte de GitHub straight away.

  5. Check that the services are healthy. This can take a few minutes.

    ghe-actions-check
    

    When every service is healthy, the output looks like this.

    MSSQL is healthy!
    Mps database is healthy!
    Mps nomad job is healthy!
    Mps AT health api is healthy!
    Token database is healthy!
    Token nomad job is healthy!
    Token AT health api is healthy!
    Actions database is healthy!
    Actions nomad job is healthy!
    Actions AT health api is healthy!
    Artifactcache database is healthy!
    Artifactcache nomad job is healthy!
    Artifactcache AT health api is healthy!
    

    If you have disabled the GitHub Actions cache service, the Artifactcache lines are not shown. While the services are starting, the output can also include lines such as Mps nomad job is not yet healthy (attempt 1 of 20), trying again in 10 seconds....

  6. Because ghe-actions-check cannot detect a stalled task, wait 15 minutes, then run the check in Confirming the problem again. Every process should show 2 or more heartbeats, and no more than a few stalled-task messages. If you have disabled the GitHub Actions cache service, the two artifactcache processes show 0.

  7. Open the GitHub Actions settings page of a repository that was affected, and check that it loads. If maintenance mode is enabled, for example because you are upgrading, do this after you disable maintenance mode. The first load can still fail. If it does, wait a minute and reload the page. If the page still returns a 500 error, contact Soporte de GitHub. For more information, see Administración de la configuración de Acciones de GitHub para un repositorio.

You do not need to clean up storage that failed to set up. The next time GitHub Actions is used in an affected repository, user, organization, or enterprise, GitHub Actions sets up the storage it needs. Workflows that should have run while the problem was happening might not run on their own. If you need them, trigger them again.

Running a failed configuration run again

If a configuration run failed with Failed to update the following host process, and a process whose name ends in -job-agent had stalled, run the configuration run again after you restart the GitHub Actions services. If no job agent process had stalled, do not run it again. Contact Soporte de GitHub instead, because something else probably caused the failure.

If the configuration run was part of an upgrade, do not disable maintenance mode yet. In a high availability configuration, stay in maintenance mode until you have upgraded every replica and replication is current. For more information, see Actualizar con un paquete de actualización.

In a cluster, do not follow these steps. Contact Soporte de GitHub instead.

Advertencia

Running ghe-config-apply disrupts services.

  1. If maintenance mode is not already enabled, enable it. For more information, see Habilitar y programar el modo de mantenimiento.

  2. Apply the configuration.

    ghe-config-apply
    

    If the configuration run fails again, leave maintenance mode enabled and contact Soporte de GitHub.

  3. When the configuration run succeeds, disable maintenance mode. If you are upgrading a high availability configuration, first upgrade every replica and wait until replication is current.

Preventing recurrence

GitHub does not know why the task stalls, so there is no known way to prevent it. You can find it before it affects your users.

  • Before you upgrade or apply a hotpatch, check for a stalled notification task, as described in Confirming the problem. If you find one, restart the GitHub Actions services first. This can stop the stalled task from causing the configuration run to fail.
  • If you forward logs to a monitoring system, alert when a GitHub Actions process does not log Deployment Notification Heartbeat for more than 15 minutes. Also alert when messages that contain both SqlNotificationBackgroundWork and still running past its interval keep arriving for more than 15 minutes. For more information, see Reenvío de logs.

Contacting Soporte de GitHub

Other issues can cause the same 500 errors and configuration failures. If the checks do not find a stalled notification task, if a task stalls again after the restart, if ghe-actions-stop prints Failed to stop, if ghe-actions-start or ghe-actions-check reports an error, or if you are unsure about any of these steps, contact Soporte de GitHub rather than continuing. For more information, see Crear un ticket de soporte.

If you have already opened a ticket, attaching a support bundle helps Soporte de GitHub diagnose the problem faster. Generate an extended support bundle as soon as you notice the problem, before you restart the GitHub Actions services. Logs from the time when the task stalled are the most useful, and they might be several days old. An extended support bundle includes logs from the past eight days. For more information, see Proporcionar datos al soporte técnico de GitHub.

Further reading