Cookbook¶

One complete job spec per workload. Every file on this page is included verbatim from collector/examples/jobs/ and validated against job.schema.json by npm test. If one stops validating, CI fails — so what you copy from here is what the collector accepts.

Post any of them to POST /jobs:

curl -X POST http://fleet-host.local:8788/jobs \
  -H 'content-type: application/json' \
  --data-binary @collector/examples/jobs/benchmark.json

About the hashes

Every sha256 below is the SHA-256 of the empty string, used as an obvious placeholder. Replace it with a real artifact hash, with "latest" where the field allows it, or with a ${jobs.<id>.artifact} reference to a dependency's output.

Device workloads¶

`benchmark` — synthetic, fanned out across the shelf¶

examples/jobs/benchmark.json

{
  "schema": 1,
  "job_id": "bench-nightly",
  "workload": "benchmark",
  "executor": "device",
  "backend": "synthetic",
  "params": { "prompt_tokens": 256, "gen_tokens": 64, "warmup_iters": 1, "measure_iters": 3 },
  "targets": { "match": "ram_mb >= 3000" },
  "constraints": { "require_charging": true, "min_battery_pct": 30 },
  "fanout": true
}

fanout: true turns this into one child job per matching device, so a whole-shelf benchmark is one request.

`benchmark` — a real model¶

examples/jobs/benchmark-llama.json

{
  "schema": 1,
  "job_id": "bench-qwen",
  "workload": "benchmark",
  "executor": "device",
  "backend": "llama.cpp",
  "model": { "name": "qwen2.5-0.5b", "format": "gguf", "quant": "Q4_K_M", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "prompt_tokens": 512, "gen_tokens": 128, "warmup_iters": 1, "measure_iters": 3 },
  "targets": { "match": "ram_mb >= 4000 && os ~ 'android'" },
  "constraints": { "require_charging": true }
}

`thermal` — does the cold number survive¶

examples/jobs/thermal.json

{
  "schema": 1,
  "job_id": "thermal-sustained",
  "workload": "thermal",
  "executor": "device",
  "backend": "synthetic",
  "params": { "prompt_tokens": 256, "gen_tokens": 64, "duration_s": 900 },
  "targets": { "device_id": "sm-x930-0d41" },
  "constraints": { "require_charging": true },
  "lease": { "ttl_s": 1800 }
}

`batch`¶

examples/jobs/batch.json

{
  "schema": 1,
  "job_id": "batch-prompts",
  "workload": "batch",
  "executor": "device",
  "backend": "llama.cpp",
  "model": { "name": "qwen2.5-0.5b", "format": "gguf", "quant": "Q4_K_M", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "input_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "max_items": 200, "max_tokens": 256 },
  "targets": { "match": "ram_mb >= 4000" },
  "constraints": { "require_charging": true }
}

`pipeline`¶

examples/jobs/pipeline.json

{
  "schema": 1,
  "job_id": "pipeline-triage",
  "workload": "pipeline",
  "executor": "device",
  "backend": "llama.cpp",
  "model": { "name": "qwen2.5-0.5b", "format": "gguf", "quant": "Q4_K_M", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "topic": "reviews.incoming", "max_tokens": 128 },
  "targets": { "match": "ram_mb >= 4000" },
  "lease": { "ttl_s": 3600 }
}

`vision-eval`¶

examples/jobs/vision-eval.json

{
  "schema": 1,
  "job_id": "plant-id-eval",
  "workload": "vision-eval",
  "executor": "device",
  "backend": "litert",
  "model": { "name": "plantnet-resnet18-int8", "format": "tflite", "quant": "int8", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "input_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "max_items": 120 },
  "targets": { "match": "os ~ 'android' && ram_mb >= 3000" },
  "constraints": { "require_charging": true }
}

The match expression is the one from the plant-ID eval, and it is Android-only for a reason: the model is a .tflite, which an iOS simulator cannot load at all. Under a pool label this job fanned out to three simulators and failed on every one.

`speech-eval`¶

examples/jobs/speech-eval.json

{
  "schema": 1,
  "job_id": "asr-eval",
  "workload": "speech-eval",
  "executor": "device",
  "backend": "whisper.cpp",
  "model": { "name": "whisper-tiny-en", "format": "gguf", "quant": "Q5_1", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "input_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "max_items": 50 },
  "targets": { "match": "ram_mb >= 4000" },
  "constraints": { "require_charging": true }
}

`embed-eval`¶

examples/jobs/embed-eval.json

{
  "schema": 1,
  "job_id": "embed-eval",
  "workload": "embed-eval",
  "executor": "device",
  "params": { "input_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "max_items": 500 },
  "targets": { "match": "ram_mb >= 3000" },
  "constraints": { "require_charging": true }
}

`vantage`¶

examples/jobs/vantage.json

{
  "schema": 1,
  "job_id": "vantage-hourly",
  "workload": "vantage",
  "executor": "device",
  "params": { "urls": ["https://example.com", "https://example.com/pricing"], "repeats": 3 },
  "fanout": { "distinct": "os" }
}

fanout: { "distinct": "os" } is the canary shape — one device per distinct OS, rather than every phone on the shelf.

Host workloads¶

`install`¶

examples/jobs/install.json

{
  "schema": 1,
  "job_id": "install-nightly",
  "workload": "install",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest", "platform": "android" },
  "targets": { "match": "os ~ 'android'", "device_kind": "device" }
}

`ui-test` — Maestro on Android¶

examples/jobs/ui-test.json

{
  "schema": 1,
  "job_id": "ui-nightly",
  "workload": "ui-test",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest" },
  "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" },
  "targets": { "match": "os ~ 'android'", "exclusive": true },
  "lease": { "ttl_s": 1800 },
  "report_to": { "github_status": "you/your-app@0000000000000000000000000000000000000000" }
}

`ui-test` — XCUITest on iOS¶

examples/jobs/ui-test-ios.json

{
  "schema": 1,
  "job_id": "ui-nightly-ios",
  "workload": "ui-test",
  "executor": "host",
  "app": { "name": "your-app-ios", "build": "nightly", "sha256": "latest", "platform": "ios" },
  "suite": { "kind": "xcuitest", "flows": "YourAppUITests-Runner.app" },
  "targets": { "executor": "mac-xcode", "exclusive": true },
  "lease": { "ttl_s": 2400 }
}

Pinned with targets.executor because simctl and devicectl ship with full Xcode, which the Android host deliberately does not have.

`cold-start`¶

examples/jobs/cold-start.json

{
  "schema": 1,
  "job_id": "cold-start-nightly",
  "workload": "cold-start",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest" },
  "params": { "app_id": "com.example.yourapp", "launches": 10, "states": ["cold", "warm", "hot"] },
  "targets": { "match": "os ~ 'android'", "exclusive": true }
}

`app-soak`¶

examples/jobs/app-soak.json

{
  "schema": 1,
  "job_id": "soak-overnight",
  "workload": "app-soak",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest" },
  "params": { "flow": "your-app/browse.yaml", "duration_s": 21600, "interval_s": 300 },
  "targets": { "exclusive": true },
  "lease": { "ttl_s": 28800 },
  "preemptible": true
}

preemptible: true lets a two-minute smoke test interrupt this six-hour run and have it resume, rather than forcing a choice between waiting and throwing the work away.

`a11y-audit`¶

examples/jobs/a11y-audit.json

{
  "schema": 1,
  "job_id": "a11y-nightly",
  "workload": "a11y-audit",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest" },
  "params": { "steps": "your-app/a11y.yaml", "variants": ["default", "largest"], "min_target_pt": 44, "font_scale": 2.0 },
  "targets": { "exclusive": true }
}

`locale-shots`¶

examples/jobs/locale-shots.json

{
  "schema": 1,
  "job_id": "store-shots",
  "workload": "locale-shots",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest" },
  "params": { "flow": "your-app/store-shots.yaml", "locales": ["en-US", "de-DE", "ja-JP", "ar-EG"] },
  "targets": { "exclusive": true },
  "lease": { "ttl_s": 3600 }
}

`web-test`¶

examples/jobs/web-test.json

{
  "schema": 1,
  "job_id": "web-nightly",
  "workload": "web-test",
  "executor": "host",
  "suite": { "kind": "playwright", "flows": "your-site" },
  "params": { "browser": "all" },
  "targets": { "url": "https://example.com" },
  "lease": { "ttl_s": 1800 }
}

`web-shots`¶

examples/jobs/web-shots.json

{
  "schema": 1,
  "job_id": "visual-nightly",
  "workload": "web-shots",
  "executor": "host",
  "suite": { "kind": "playwright", "flows": "your-site" },
  "params": { "browser": ["chromium", "android-device"] },
  "targets": { "url": "https://example.com", "exclusive": true },
  "lease": { "ttl_s": 2400 }
}

android-device expands to one profile per real Android device attached to the claiming executor, and fails its slot if it finds none rather than quietly shrinking the matrix.

`web-audit`¶

examples/jobs/web-audit.json

{
  "schema": 1,
  "job_id": "audit-weekly",
  "workload": "web-audit",
  "executor": "host",
  "targets": { "url": "https://example.com" },
  "lease": { "ttl_s": 3600 }
}

`web-unfurl`¶

examples/jobs/web-unfurl.json

{
  "schema": 1,
  "job_id": "unfurl-weekly",
  "workload": "web-unfurl",
  "executor": "host",
  "suite": { "kind": "playwright", "flows": "your-site" },
  "targets": { "url": "https://example.com" }
}

`drain`¶

examples/jobs/drain.json

{
  "schema": 1,
  "job_id": "drain-weekly",
  "workload": "drain",
  "executor": "host",
  "app": { "name": "your-app-android", "build": "nightly", "sha256": "latest" },
  "params": { "duration_s": 14400, "track": "your-app/commute.gpx" },
  "targets": { "pool": "battery", "exclusive": true },
  "lease": { "ttl_s": 21600 }
}

`archive`¶

examples/jobs/archive.json

{
  "schema": 1,
  "job_id": "archive-reviews",
  "workload": "archive",
  "executor": "host",
  "params": { "source": "play", "app": "com.example.yourapp", "account": "fleet-gsc" }
}

account names a Keychain item on the executor host. The secret itself is never in the spec — POST /jobs is unauthenticated, so a spec is not a place a credential could safely live.

`digest`¶

examples/jobs/digest.json

{
  "schema": 1,
  "job_id": "digest-weekly",
  "workload": "digest",
  "executor": "host",
  "model": { "name": "qwen2.5-3b-instruct", "format": "gguf", "quant": "Q4_K_M", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "app": "com.example.yourapp", "days": 7 },
  "targets": { "match": "ram_mb >= 4000" },
  "lease": { "ttl_s": 7200 }
}

Machine workloads¶

`build`¶

examples/jobs/build.json

{
  "schema": 1,
  "job_id": "build-903",
  "workload": "build",
  "executor": "device",
  "params": {
    "repo": "git@github.com:you/your-app.git",
    "ref": "main",
    "kind": "gradle",
    "task": "assembleRelease",
    "app": "your-app-android"
  },
  "targets": { "match": "capabilities ~ 'build:gradle'" },
  "constraints": { "require_ac": true, "require_idle_s": 300, "max_load": 2 },
  "lease": { "ttl_s": 3600 },
  "report_to": { "github_status": "you/your-app@0000000000000000000000000000000000000000" }
}

The constraints are the point: this waits for a machine on mains that has been idle five minutes, rather than starting an Xcode build while somebody is typing.

A chain: build, install, test¶

examples/jobs/chain-build-install-test.json

{
  "schema": 1,
  "job_id": "ui-903",
  "workload": "ui-test",
  "executor": "host",
  "depends_on": ["build-903", "install-903"],
  "app": {
    "name": "your-app-android",
    "build": "903",
    "sha256": "${jobs.build-903.artifact}"
  },
  "suite": { "kind": "maestro", "flows": "your-app/smoke.yaml" },
  "targets": { "exclusive": true },
  "lease": { "ttl_s": 1800 }
}

Post build.json, install.json and this, and the last two arrive as waiting. ${jobs.build-903.artifact} is substituted at promotion with the real hash the build uploaded. A failed build fails both waiters, naming the cause, rather than leaving them parked forever.

`model-convert`¶

examples/jobs/model-convert.json

{
  "schema": 1,
  "job_id": "convert-resnet18",
  "workload": "model-convert",
  "executor": "device",
  "params": { "input_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "formats": ["tflite", "mlmodelc"], "quant": "int8" },
  "targets": { "match": "capabilities ~ 'model-convert' && ram_mb >= 16000" },
  "constraints": { "require_ac": true },
  "lease": { "ttl_s": 7200 }
}

`dataset-prep`¶

examples/jobs/dataset-prep.json

{
  "schema": 1,
  "job_id": "prep-plantnet",
  "workload": "dataset-prep",
  "executor": "device",
  "params": { "source": "plantnet-300k", "max_items": 120, "split": "held-out" },
  "targets": { "match": "capabilities ~ 'dataset-prep'" },
  "constraints": { "require_ac": true },
  "lease": { "ttl_s": 3600 }
}

`serve`¶

examples/jobs/serve.json

{
  "schema": 1,
  "job_id": "serve-qwen",
  "workload": "serve",
  "executor": "device",
  "model": { "name": "qwen2.5-3b-instruct", "format": "gguf", "quant": "Q4_K_M", "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" },
  "params": { "port": 8081, "bind": "loopback" },
  "targets": { "match": "capabilities ~ 'serve' && ram_mb >= 16000" },
  "constraints": { "require_ac": true },
  "lease": { "ttl_s": 3600 },
  "preemptible": true
}

`shell`¶

examples/jobs/shell.json

{
  "schema": 1,
  "job_id": "shell-maintenance",
  "workload": "shell",
  "executor": "device",
  "params": { "script_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "args": ["--prune"] },
  "targets": { "match": "capabilities ~ 'shell'" },
  "constraints": { "require_ac": true }
}

This one is refused unless the target machine's owner has pinned that exact script_sha256 in a local allowlist file by hand. A machine that has pinned nothing does not declare the capability at all, so it is never offered the job.

`self-check`¶

examples/jobs/self-check.json

{
  "schema": 1,
  "job_id": "self-check-daily",
  "workload": "self-check",
  "executor": "device",
  "targets": { "match": "capabilities ~ 'self-check'" },
  "fanout": true
}

Adding one¶

New example specs go in collector/examples/jobs/, named for the workload they demonstrate, and are picked up by the validator automatically.

Cookbook¶

Device workloads¶

benchmark — synthetic, fanned out across the shelf¶

benchmark — a real model¶

thermal — does the cold number survive¶

batch¶

pipeline¶

vision-eval¶

speech-eval¶

embed-eval¶

vantage¶

Host workloads¶

install¶

ui-test — Maestro on Android¶

ui-test — XCUITest on iOS¶

cold-start¶

app-soak¶

a11y-audit¶

locale-shots¶

web-test¶

web-shots¶

web-audit¶

web-unfurl¶

drain¶

archive¶

digest¶