Skip to content

Operations

Day-to-day maintenance, health repair, drain/resume, and routine procedures for a live fleet.

Health check

./health.sh               # report: launchd + GitHub state per runner
./health.sh --repair      # also restart any service that is loaded but dead

health.sh exists because no runner plist sets KeepAlive. runsvc.sh supervises the listener (Runner.Listener), so a crashed listener recovers on its own. A crashed or OOM-killed RunnerService.js does not. LaunchD will not revive it, and the only symptom is one repo's jobs queueing forever while every other repo looks fine.

health.sh --repair restarts dead services. It skips draining and drained runners intentionally — a repair that undoes a deliberate drain is worse than no repair.

Scheduled maintenance

Runner LaunchAgents do not set KeepAlive. When RunnerService.js dies, launchd leaves the job loaded but dead — one repo's jobs queue forever while every other repo looks fine. Run repair on a timer so silent deaths close within a minute rather than whenever someone notices:

./healthctl.sh install     # health.sh --repair every 60s (configurable)
./healthctl.sh status
./healthctl.sh logs

healthctl.sh generates its own LaunchAgent plist with absolute paths, a launchd-safe PATH/HOME, and a lock so a slow sweep cannot overlap the next tick. It never touches the GitHub runner plists that register.sh creates. Drained runners stay skipped — repair behaviour is inherited from health.sh.

Set FLEET_HEALTH_INTERVAL in fleet.env before install (default 60), then reinstall if you change it later. See Configuration → Health repair.

Disk cleanup

Run cleanup.sh --apply --auto from a LaunchAgent every 15 minutes. Most runs cost one df and exit. It does real work when free disk is under the pressure line (FLEET_CLEANUP_PRESSURE_GB, default the admission floor plus 20 GB), and once a day otherwise:

  • Host-wide steps wait for an idle fleet: old DerivedData, simulators whose runtime is gone, stale Playwright locks.
  • Per-runner steps skip only the runners that are building. Each runner's ci-<runner> … simulator is erased while shut down once it passes FLEET_CLEANUP_SIM_MAX_GB (3 GB, or 1 GB under pressure); no other simulator is touched. Under pressure, idle runners' Playwright browsers go too.

A weekly run is not enough. On runner-host free disk swung by 20–40 GB within a day while a weekly run reclaimed 1–2 GB, and it refused to run whenever any job was building. When the disk does reach the admission floor, every job is held, so the fleet is idle and the next 15-minute run can take everything it needs.

Install the timer with ./cleanupctl.sh install (join-host.sh --apply does it on a new host). It writes <FLEET_LABEL_PREFIX>.runner-cleanup with the runners' job PATH, bootstraps it into the gui/<uid> domain, and logs to ~/Library/Logs/runner-cleanup.log; ./cleanupctl.sh status shows the last run. Like the runners, it starts at login, so a host without automatic login gets it back when someone logs in.

With one repository mirrored on several hosts, GitHub lists every host's runners for it. cleanup.sh counts only the busy runners that live on this host, so one host's jobs never hold another host's cleanup.

examples/launchd-cleanup.plist is a hand-install template. For health repair without healthctl.sh, examples/launchd-health.plist is a manual fallback — prefer ./healthctl.sh install on production hosts.

Draining a runner for maintenance

Stopping a runner with svc.sh stop kills whatever it is building, leaving a red build on GitHub. drain-runner.sh drains gracefully instead.

scripts/drain-runner.sh project-ios-2 --drain    # stop after current job
scripts/drain-runner.sh project-ios-2             # check drain state
scripts/drain-runner.sh project-ios-2 --resume   # put it back

An idle runner stops immediately. A busy runner is marked draining and stops after its job, which requires the completion hook installed:

scripts/install-hooks.sh --apply

Without the hook, the drain intent is recorded but the service is not stopped automatically — drain the runner again once it is idle.

The dashboard shows draining / drained badges and offers Drain/Resume buttons in the runner drawer.

Taking a whole host out of rotation

Put a .drain file at the fleet root:

echo drained > ~/actions-runners/.drain     # this Mac takes no new runners
rm ~/actions-runners/.drain                 # back into rotation

The agent reports this state on its next heartbeat. Placement will not choose a drained host. Runners already on the host keep working — this stops the fleet putting new work there, not existing work.

Disk cleanup

./cleanup.sh              # dry run: shows what would be deleted
./cleanup.sh --apply      # actually delete

What it cleans: - DerivedData older than 7 days (keyed on mtime, so actively used projects are never candidates) - Unavailable simulators (xcrun simctl delete unavailable) - Runner _diag logs older than 14 days - Each idle runner's warm build caches beside its checkout (_work/<repo>/DerivedData-*, spm-*, .build-cache-*) with nothing written in FLEET_CLEANUP_RUNNER_CACHE_DAYS (7); under disk pressure, all of them. Never a building runner's, never under the runner's own _work/_*, never a git checkout. - The Spotlight marker _work/.metadata_never_index for any runner missing it (register.sh writes it for every new runner) - Playwright browser cache sizes (reported every run) - Stale Playwright __dirlock files older than 6 hours (skipped while an install is in flight)

Refuses to run while any job is in flight. Deleting DerivedData under a live xcodebuild produces a failure that looks like a code problem — the worst kind of CI flake to chase. Stale Playwright locks are only removed when no playwright install is running.

Playwright caches can be several gigabytes per runner. If disk alerts fire repeatedly, check cache sizes in the cleanup dry-run output and confirm workflows export PLAYWRIGHT_BROWSERS_PATH=$RUNNER_TOOL_CACHE/ms-playwright so each runner keeps its own copy.

Simulator lifecycle cleanup

Enable cancellation-safe cleanup for simulator-capable runners in fleet.env:

FLEET_SIMULATOR_CLEANUP=1
FLEET_SIMULATOR_RUNNERS="*-ios *-ios-* *-tvos *-tvos-*"

The job hooks record devices already booted before CI, then shut down only the devices CI introduced after the final overlapping selected job finishes. A detached guardian follows each Runner.Worker, so cancellation, a crash, or a missing completion hook still releases the lease and triggers cleanup. Existing interactive Xcode simulators are preserved.

Backup

cd dashboard
./fleetctl.sh backup

The only thing worth backing up is fleet.db — it holds every run and job ever recorded, and GitHub discards run detail after 90 days. The scripts are in git. The runner registrations are in actions-runners/*/. runner but can be re-registered; the database history cannot be recovered once GitHub's copy ages out.

In PostgreSQL HA mode, managed backups cover current shared control state and npm run migrate:postgres archives every SQLite table with row counts and checksums before cutover. Continue fleetctl.sh backup on each Mac because its SQLite file remains the low-overhead local analytics cache.

Coordinator failover drill

Run this before enabling live autoscaling on the second Mac:

  1. Confirm both /api/health responses name the same leaderId, with exactly one role: leader.
  2. Stop the leader: dashboard/fleetctl.sh stop.
  3. Within 30 seconds, confirm the peer reports role: leader and /api/state has a fresh timestamp.
  4. Trigger an autoscale dry-run and verify one placement row and no duplicate command.
  5. Restart the old leader; it must return as standby.
  6. Temporarily deny one replica access to PostgreSQL. It must stop leading and must not issue actions, while existing runners continue taking GitHub jobs.

Also test PostgreSQL unavailability. Both dashboards should report degraded control state, but runner listeners and in-flight builds must continue because they communicate directly with GitHub.

Logs

cd dashboard && ./fleetctl.sh logs              # tail daemon log (stdout+stderr)
cd dashboard && ./fleetctl.sh logs --lines 200  # last 200 lines
ls ~/actions-runners/project-ios/_diag/         # runner's own diagnostic logs

The runner writes a new _diag/Runner_ log for every started listener. These grow without bound and are cleaned by cleanup.sh.

Deregistering a runner

scripts/deregister.sh project-ios-2             # dry run
scripts/deregister.sh project-ios-2 --apply     # actually remove it

deregister.sh refuses if: - The runner is currently busy (mid-job) - It is the last runner for its repo (removing the last would leave the repo with no runner at all)

To deregister the last runner for a repo, you need to do it from the GitHub UI.

Re-registering a stale runner

A runner whose token has expired or that missed too many GitHub heartbeats shows as offline permanently. Re-register it:

scripts/deregister.sh project-ios --apply    # remove old registration
./register.sh owner/project-ios              # fresh registration

If the runner directory is healthy but the registration is stale, the same command will overwrite it — register.sh detects an existing registration and removes it before creating a new one.

Incident checklist

If a job has been queued for more than a minute:

  1. Open the Runs tab — hover the queued indicator to see the classifier verdict.
  2. Open the Fleet tab — find the runner(s) for that repo. Are they online?
  3. If runner-down: open the runner drawer → recent events show why it stopped. Try ./health.sh --repair.
  4. If label-mismatch: the job's runs-on requires a label no runner has. Register a runner with that label or update the workflow.
  5. If repo-capacity: the queue is longer than the runner count can absorb. Use the Capacity tab to add a runner, or enable autoscaling.
  6. If host-saturation: the host is too busy. Check the load/swap meters.
  7. If github-delay: the runner is ready; GitHub is slow to dispatch. Wait.

If the dashboard is not updating (collector lapsed): 

cd dashboard && ./fleetctl.sh status    # is the daemon alive?
cd dashboard && ./fleetctl.sh logs      # what did it last say?