5 ms·
Parallel and sequential, especially at the command level, are really the wrong abstractions for running scripts. If you have multiple packages, each with builds
by spankalee 8mo ago
Parallel and sequential, especially at the command level, are really the wrong abstractions for running scripts. If you have multiple packages, each with builds, there's a high chance you have dependencies and multiple packages depending on common ones.
What you really want is a way for scripts to describe their dependencies, and then the runner figures out what order to run them in, and cache scripts that don't need to be run because their inputs didn't change.
Wireit[1] is an npm script runner that adds that incrementally on top of package.json. I can't manage an npm monorepo without it now.
Deno started integrating the idea directly into their built-in script runner. I think this is an important enough feature that more runtimes should follow Deno's lead.
[1]: https://github.com/google/wireit https://github.com/google/wireit
- pyrolistical 8mo agoIf only we could make something like that But now we would need each script to independently do their own caching, which isn’t all bad. At least you have more cross runner compatibility and resilience
- spankalee 8mo agoWireit really is that. The script declares dependencies and input, Wireit caches based on the direct inputs and dependency outputs.
- embedding-shape 8mo ago> What you really want is a way for scripts to describe their dependencies, and then the runner figures out what order to run them in, and cache scripts that don't need to be run because their inputs didn't change. DAG + content-addressing, final binary being the target and everything resolved from there. We could have some beautiful build system that just works and is fast, but seems it never magically appears by itself although it seems so elegant. Guess Nix/NixOS is the closest we've gotten so far, works well enough, missing concurrency and parallelism though.
- dpe82 8mo agoGoogle's build system Bazel is what you describe.
- mattnewton 8mo agoI should really give it another look, I usually ended up not adopting it for projects because of the boilerplate and high setup overheard, but those are both things that ai agents can usually be trusted with. Maybe the calculus has changed.
- dpe82 8mo agoYeah Bazel is a PITA to do yourself. I haven't tried but I'm confident Claude et al. would handle it just fine. By similar logic I recently adopted NixOS for a lot of my servers; Claude is perfectly happy to slog through the pain of Nix configs for me.
- hxtk 8mo agoI’m not a fan of generative AI for the use case because it’s rote enough to do deterministically, but deterministic code generation is getting better and better. Gazelle, the BUILD file generator for Go, now supports plugins and several other languages have Gazelle plugins. I’ve used AI to generate BUILD file generators before, though. I had good luck getting it to write a script that would analyze a Java project with circular dependencies and aggregate the cycle participants into a single target.
- hxtk 8mo agoIt’s gotten easier of late because Bazel modules are nice and Gazelle has started support plugins so it can do build file generation for other languages. I don’t like generative AI for rote tasks like this, but I’ve had good luck using generative AI to write deterministic code generators that I can commit to a project and reuse.
- spankalee 8mo agoWireit does both DAG and content-addressing. It figerprints the inputs and outputs of dependencies. And you run scripts externally with plain `npm run` commands. It's really beautiful.
- aziis98 8mo agoWhy didn't I know about this before
- silverwind 8mo ago`make` can do exactly the same.