@async/pipeline

Getting Started

Use this guide to try the repo, add a pipeline to another project, inspect local runs, and wire the same workflow into CI.

Requirements

• pnpm
• Node 24 for pipeline.ts
• Node 20+ for pipeline.mjs or pipeline.js
• Optional: Lima as limactl for programmatic isolated-runner experiments

1. Try This Repo

From the checkout:

cd /Users/patrickjs/code/async-framework/async-pipeline
pnpm install --frozen-lockfile
pnpm build
pnpm async-pipeline run verify

The repo dogfoods its own pipeline in ../pipeline.ts. The verify job expands to:

typecheck -> test -> build -> pack

Run quick inspection commands:

pnpm async-pipeline list
pnpm async-pipeline graph --format json
pnpm async-pipeline explain build
pnpm async-pipeline doctor

Check the generated GitHub bootloader:

pnpm async-pipeline github check

2. Add A Pipeline To A Project

After the package is published:

pnpm add -D @async/pipeline

Create pipeline.ts at the project root:

import { definePipeline, job, sh, task, trigger } from "@async/pipeline";

export default definePipeline({
  name: "web-app",
  cache: "file:local",
  namedInputs: {
    source: [
      "src/**/*.ts",
      "src/**/*.tsx",
      "package.json",
      "pnpm-lock.yaml",
      "tsconfig.json"
    ]
  },
  triggers: {
    pr: trigger.github({ events: ["pull_request"] }),
    main: trigger.github({ events: ["push"], branches: ["main"] }),
    nightly: trigger.cron("17 2 * * *")
  },
  tasks: {
    typecheck: task({
      inputs: ["source"],
      cache: "file:local",
      timeout: "2m",
      run: sh`pnpm typecheck`
    }),
    test: task({
      dependsOn: ["typecheck"],
      inputs: ["source"],
      cache: "file:local",
      retry: { attempts: 2, delayMs: 500 },
      run: sh`pnpm test`
    }),
    build: task({
      dependsOn: ["test"],
      inputs: ["source"],
      outputs: ["dist/**"],
      cache: "file:local",
      run: sh`pnpm build`
    })
  },
  jobs: {
    verify: job({
      target: "build",
      trigger: ["pr", "main"]
    }),
    nightly: job({
      target: "build",
      trigger: ["nightly"]
    })
  }
});

Add scripts:

{
  "scripts": {
    "async-pipeline": "async-pipeline",
    "verify": "async-pipeline run verify"
  }
}

Add local runtime state and scratch output to .gitignore:

.async/
*.tgz
.tmp/

Use .tmp/ only for generated workflow experiments. The real generated GitHub workflow and lock should stay committed.

Run it:

pnpm async-pipeline run verify

Use the explicit async-pipeline command in docs and CI. Short aliases and smart runner dispatch belong in @async/run, not this package.

3. Inspect The Run

Runs write machine-readable records and human-readable summaries under .async/:

ls .async/runs
cat .async/runs/<run-id>/summary.md
cat .async/runs/<run-id>/execution.json
ls .async/runs/<run-id>/logs

The execution record includes task status, attempts, timings, cache keys, cache hit flags, errors, source metadata, and task metadata.

Use metadata commands when you want to inspect a pipeline without running it:

pnpm async-pipeline metadata --format json
pnpm async-pipeline graph --format dot

Metadata reads are safe for planning and automation: they do not clone sources, run prepare, execute tasks, or evaluate deferred shell callbacks.

4. Generate GitHub Actions

GitHub starts workflows from committed YAML, so generate the bootloader from pipeline.ts:

pnpm async-pipeline github generate
# or, when sync.github is configured:
pnpm async-pipeline sync github generate

Commit the generated files:

.github/workflows/async-pipeline.yml
.github/async-pipeline.lock.json

CI then runs:

pnpm async-pipeline github check
pnpm async-pipeline github run
pnpm async-pipeline github run --concurrency 2

github check fails when the committed workflow or lock no longer matches the GitHub-relevant metadata in pipeline.ts.

For tests, render to scratch paths:

pnpm async-pipeline github generate --workflow .tmp/workflow.yml --lock .tmp/lock.json
pnpm async-pipeline github check --workflow .tmp/workflow.yml --lock .tmp/lock.json

5. Sync Package Tasks

Use sync.tasks when pipeline.ts should also own package-manager commands:

sync: {
  tasks: true
}

Then generate or check the package scripts and lock:

pnpm async-pipeline sync tasks generate
pnpm async-pipeline sync tasks check

The distinction is:

tasks     = what can run
jobs      = named entrypoints
triggers  = when jobs should run
sync      = generated files to keep current

Triggers describe when jobs should run. Sync describes which generated files should be kept current.

What To Commit

Commit:

• pipeline.ts, pipeline.mjs, or pipeline.js
• .github/workflows/async-pipeline.yml
• .github/async-pipeline.lock.json
• .async-pipeline/tasks.lock.json when sync.tasks is configured
• package metadata and lockfile changes
• docs that explain the project pipeline

Do not commit:

• .async/
• package tarballs from npm pack
• scratch workflow output from custom --workflow / --lock paths, such as .tmp/
• dist/ unless your project already commits build output

Troubleshooting

If the CLI cannot find a config file, make sure one of these exists at the project root:

pipeline.ts
pipeline.mjs
pipeline.js

If pipeline.ts fails to load on Node 20, use Node 24 or convert the config to pipeline.mjs.

If a task keeps returning a cache hit, check its inputs. A task only becomes dirty when its task config or declared input files change.

If github check fails, rerun:

pnpm async-pipeline github generate

This site is open source. Improve this page.