@async/pipeline

Running Locally

Local runs are the primary workflow. GitHub Actions should invoke the same pipeline command you already use on your machine.

Use async-pipeline when the command should be explicit. Short aliases and smart runner dispatch belong in @async/run.

Common Commands

List jobs, tasks, and sources:

pnpm async-pipeline list

Run a job and its dependencies:

pnpm async-pipeline run verify
pnpm async-pipeline run verify --concurrency 2

Run one task and its dependencies through a synthetic job:

pnpm async-pipeline run-task test
pnpm async-pipeline run-task test --concurrency 1

The scheduler starts ready tasks in deterministic graph order and runs independent tasks in parallel up to the configured concurrency. Use --concurrency 1 to force a sequential run when debugging task interactions.

Explain a task:

pnpm async-pipeline explain build

Render the graph:

pnpm async-pipeline graph --format json
pnpm async-pipeline graph --format dot

Read pipeline metadata without running anything:

pnpm async-pipeline metadata --format json
pnpm async-pipeline metadata --format json --include-sources

List and sync declared sources:

pnpm async-pipeline sources list
pnpm async-pipeline sources sync

Run a source task:

pnpm async-pipeline run-task storefront:test

Generate a GitHub matrix for source tasks in a job:

pnpm async-pipeline matrix verifyImpact --format github

Generate or check the GitHub Actions bootloader:

pnpm async-pipeline github generate
pnpm async-pipeline github check
pnpm async-pipeline sync github generate
pnpm async-pipeline sync github check

Generate or check package-manager tasks from sync.tasks:

pnpm async-pipeline sync tasks generate
pnpm async-pipeline sync tasks check

Use --workflow <path> and --lock <path> when you want to render/check generated files outside .github/, such as in tests.

Run environment checks:

pnpm async-pipeline doctor

Clear local task cache and prune local state:

pnpm async-pipeline cache clear
pnpm async-pipeline gc --keep 20 --cache-days 30

gc keeps the newest run records and prunes task-cache entries unused for --cache-days days. Cache hits refresh their last-used time, and --cache-days 0 disables cache pruning.

Inspect A Run

After async-pipeline run verify, inspect .async/runs:

ls .async/runs
cat .async/runs/<run-id>/summary.md
cat .async/runs/<run-id>/execution.json
ls .async/runs/<run-id>/logs

The execution record includes:

Cache Behavior

Enable cache per task:

task({
  inputs: ["src/**/*.ts", "package.json", "pnpm-lock.yaml"],
  outputs: ["dist/**"],
  cache: "file:local",
  run: sh`pnpm build`
})

cache: true uses the pipeline cache default. Explicit refs such as file:local make task behavior clearer in examples and metadata.

On the next run, the task can be skipped when:

Task cache lives under:

.async/cache/tasks

Input resolution ignores .git/, .async/, and node_modules/ by default. A task’s declared outputs are also excluded from that task’s input files, so dist/** or packages/*/dist/** cannot make a build dirty after it writes its own artifacts.

For file-cache tasks with declared outputs, cache entries include result.json, outputs.json, and copied output files. On a hit, outputs are restored before the task returns cached. Existing result-only entries for output-producing tasks are treated as misses and repopulated on the next successful run. Memory cache cannot restore files; it only honors output-producing hits while the output files from the previous run still exist.

Warm source checkouts live under:

.async/sources

Keep local run state out of git:

.async/
*.tgz
.tmp/

.async/ contains run records, logs, task cache, source checkouts, and repo-local npm cache used by pack checks. *.tgz catches tarballs from pack commands. .tmp/ is a good place for custom github generate --workflow ... --lock ... experiments.

When you deliberately want a clean local pipeline state:

rm -rf .async

Retry And Timeout

Retry a flaky command:

task({
  retry: { attempts: 3, delayMs: 1000 },
  run: sh`pnpm test`
})

Set a timeout:

task({
  timeout: "5m",
  run: sh`pnpm build`
})

Supported timeout units:

ms
s
m
h

Requirements

Declare required tools:

task({
  requires: {
    tools: ["node", "pnpm", "ollama"]
  },
  run: sh`ollama list`
})

The host runner checks tools with command -v. Ollama is an optional requirement declaration, not a package dependency.

Config Files

The CLI searches the project root in this order:

pipeline.ts
pipeline.mjs
pipeline.js

Use pipeline.ts on Node 24. Use pipeline.mjs or pipeline.js for Node 20+.