4 ms·

A problem with this approach is that deleting a file from source/ does not delete it from build/. In my own projects, simply rebuilding the whole site is fast

by jez 3y ago

A problem with this approach is that deleting a file from source/ does not delete it from build/. In my own projects, simply rebuilding the whole site is fast enough, so I opt to remove the whole build folder before a rebuild: https://github.com/jez/jez.github.io/blob/source/Makefile#L10 https://github.com/jez/jez.github.io/blob/source/Makefile#L1... This defeats a big part of why you’d want a build system in the first place (incremental builds), but at least if you know the page you want to regenerate you can still `make` that file directly. If there’s a common workaround for this pattern in makefiles I’d love to learn it.

linkdd 3y ago
Something like this should do the trick: rm/%.html: @rm -f source/%.html build/%.html Run with: $ make rm/page.html
rustybolt 3y ago
> If there’s a common workaround for this pattern in makefiles I’d love to learn it. "make clean"?
- dmd 3y ago
  How does that solve the problem? That forces a total rebuild, which is exactly what he said he didn't want.
  - rustybolt 3y ago
    Yes, I didn't read properly. I guess you could do some magic to delete "unexpected" files, but are there tools which do solve this problem?
    - IshKebab 3y ago
      The cleanest way to do it is essentially "make install". You do all the heavy build steps into a build directory, and then the final stage is to delete the "output" directory and copy all the files you need there. Incremental builds should still be pretty fast since the only repeated action is copying files (and you could link them if you want instead).
      - cratermoon 3y ago
        Conventionally, install puts the outputs in their where they will live for use. It does so in a way that 'make uninstall' will leave things as they were before install. The install target should also run any pre- and post-install commands. There's also a 'make dist' convention, to build a release tarball.
      - Karellen 3y ago
        This is the way, because intermediate build artefacts also end up in `build/`. You don't want those in your `output/` directory, but you also don't want to delete them because they help speed up the incremental builds. Edit: `make install` also protects you against broken builds breaking your live site.
Jhsto 3y ago
I use Nix, so I get incremental builds and your problem goes away.
- schemescape 3y ago
  You’re using Nix to drive your static site generation? If so, please share more details because that sounds intriguing!
  - Jhsto 3y ago
    Here you go: https://juuso.dev/blogPosts/nix-as-a-static-site-generator/nix-as-a-static-site-generator.html https://juuso.dev/blogPosts/nix-as-a-static-site-generator/n...
    - schemescape 3y ago
      Wow, I ask and a blog post just appears! My initial reaction is: I should probably get around to learning about Flakes. I’m not sure I’d want each blog post to pin its templates, but it’s nice to have that choice.
      - __MatrixMan__ 3y ago
        A flake is a function, right? Invoking `nix build` resolves its inputs and computes its outputs. So if the static site is the output, wouldn't the content have to be one of the inputs? Personally, I think I'd use two flakes, one that builds the content into something that's ready to hand-off to code, and a second one that turns it into a usable site. That way you wouldn't end up with a new input for each post, but instead would have a versioned something which represents your content all bundled together, and then the site builder consumes it as a single dependency--but conceptually it's the same. I guess I'm just saying that it's not conventional, but it's a pretty logical conclusion to reach.
- __MatrixMan__ 3y ago
  I was just reading this and thinking that `nix build` would do the same trick even more nicely.
schemescape 3y ago
Not sure if it’s a common pattern, but my solution to this was to always run a command that deletes all “unexpected” files, using GNU Make’s “shell” function to enumerate files and the “filter-out” function to filter out “expected” outputs. Edit: I ensure this runs every time using an ugly hack: running the command as part of variable expansion via the “shell” function. Edit to link my Makefile: https://github.com/jaredkrinke/make-blog/blob/main/Makefile https://github.com/jaredkrinke/make-blog/blob/main/Makefile
mftrhu 3y ago
Not sure if anyone actually uses it, but I would approach the problem with find, comm, and a sprinkle of sed: comm -23 <(find build -type f -iname "*.html" -printf "%P\n" | sed 's/\.html$//' | sort) \ <(find source -type f -iname "*.md" -printf "%P\n" | sed 's/\.md$//' | sort) The find commands get you a list of all the files (and only files - directories will have to be removed in a separate step) in each of the build and source folder, sed chops off the extension, while comm -23 compares them, printing only the files unique to the build folder, which you can then deal with as you see fit (e.g., by feeding them to xargs rm).
- hawski 3y ago
  Using comm was exactly what I did use in my little experiment of a barebones SSG. I did save a list of generated files and compared them. This one liner is the meat of the whole solution: comm -23 <(awk 'NR>1' "$DSTDIR/build-info") <(find "$SRCDIR" -name "*$EXT" -type f -printf "%P\n" | tee >(gen_index) | xargs -n1 "$0" "$SRCDIR" "$DSTDIR" | sort | tee -a "$DSTDIR/build-info.new") | (cd "$DSTDIR" && xargs rm) Full source here: https://gist.github.com/hadrianw/060944011acfcadd889d937b960a5a1f https://gist.github.com/hadrianw/060944011acfcadd889d937b960...
throwaway858 3y ago
The shake build system (a general-purpose build system similar-to/better-than make) has a "prune" feature for exactly this purpose: http://neilmitchell.blogspot.com/2015/04/cleaning-stale-files-with-shake.html http://neilmitchell.blogspot.com/2015/04/cleaning-stale-file... But I think the best solution (that also works with make) is to have a "make dist" target that creates a final .tar.gz archive of the result. If the rule is written properly then it won't contain any stale files. The disadvantage is for large project it may be slow, but you are not supposed to use this rule during development (where it is useless anyway), only for releases (which still can be built incrementally -- only the final .tar.gz needs to be created from scratch)
deleted 3y ago
[deleted]
bogwog 3y ago
I think the best solution is to use something like webpack or vite or whatever. These usually have their own dev server and can watch directories for changes. My personal site is also using a custom make-like ssg, but after spending a disproportionate amount of time writing the bundling/packaging code, I decided to just switch over to one of these tools. It’s a solved problem, and it greatly reduced the complexity of my site.
dredmorbius 3y ago
File deletions and renames are common problems with many revision control / build systems. Other than the nuclear option ("make clean"), another is to have a specific rename / remove make target, so: make rm sourcefile or make mv sourcefile newsourcefile ... which will handle the deletion and renaming of both the original and generated targets. In practice for even fairly large blog and online projects, a make clean / make all cycle is reasonably quick (seconds, perhaps minutes), and is often called for when revising templates or other elements of the site design. If you're operating at a scale where rebuild time is a concern, you probably want to be using an actual CMS (content management system) in which source is managed in a database and generated dynamically on client access.

by jez 3y ago

linkdd 3y ago

Something like this should do the trick: rm/%.html: @rm -f source/%.html build/%.html Run with: $ make rm/page.html

rustybolt 3y ago

> If there’s a common workaround for this pattern in makefiles I’d love to learn it. "make clean"?

dmd 3y ago
How does that solve the problem? That forces a total rebuild, which is exactly what he said he didn't want.
- rustybolt 3y ago
  Yes, I didn't read properly. I guess you could do some magic to delete "unexpected" files, but are there tools which do solve this problem?
  - IshKebab 3y ago
    The cleanest way to do it is essentially "make install". You do all the heavy build steps into a build directory, and then the final stage is to delete the "output" directory and copy all the files you need there. Incremental builds should still be pretty fast since the only repeated action is copying files (and you could link them if you want instead).
    - cratermoon 3y ago
      Conventionally, install puts the outputs in their where they will live for use. It does so in a way that 'make uninstall' will leave things as they were before install. The install target should also run any pre- and post-install commands. There's also a 'make dist' convention, to build a release tarball.
    - Karellen 3y ago
      This is the way, because intermediate build artefacts also end up in `build/`. You don't want those in your `output/` directory, but you also don't want to delete them because they help speed up the incremental builds. Edit: `make install` also protects you against broken builds breaking your live site.

Jhsto 3y ago

I use Nix, so I get incremental builds and your problem goes away.

schemescape 3y ago
You’re using Nix to drive your static site generation? If so, please share more details because that sounds intriguing!
- Jhsto 3y ago
  Here you go: https://juuso.dev/blogPosts/nix-as-a-static-site-generator/nix-as-a-static-site-generator.html https://juuso.dev/blogPosts/nix-as-a-static-site-generator/n...
  - schemescape 3y ago
    Wow, I ask and a blog post just appears! My initial reaction is: I should probably get around to learning about Flakes. I’m not sure I’d want each blog post to pin its templates, but it’s nice to have that choice.
    - __MatrixMan__ 3y ago
      A flake is a function, right? Invoking `nix build` resolves its inputs and computes its outputs. So if the static site is the output, wouldn't the content have to be one of the inputs? Personally, I think I'd use two flakes, one that builds the content into something that's ready to hand-off to code, and a second one that turns it into a usable site. That way you wouldn't end up with a new input for each post, but instead would have a versioned something which represents your content all bundled together, and then the site builder consumes it as a single dependency--but conceptually it's the same. I guess I'm just saying that it's not conventional, but it's a pretty logical conclusion to reach.
__MatrixMan__ 3y ago
I was just reading this and thinking that `nix build` would do the same trick even more nicely.

schemescape 3y ago

Not sure if it’s a common pattern, but my solution to this was to always run a command that deletes all “unexpected” files, using GNU Make’s “shell” function to enumerate files and the “filter-out” function to filter out “expected” outputs. Edit: I ensure this runs every time using an ugly hack: running the command as part of variable expansion via the “shell” function. Edit to link my Makefile: https://github.com/jaredkrinke/make-blog/blob/main/Makefile https://github.com/jaredkrinke/make-blog/blob/main/Makefile

mftrhu 3y ago

Not sure if anyone actually uses it, but I would approach the problem with find, comm, and a sprinkle of sed: comm -23 <(find build -type f -iname "*.html" -printf "%P\n" | sed 's/\.html$//' | sort) \ <(find source -type f -iname "*.md" -printf "%P\n" | sed 's/\.md$//' | sort) The find commands get you a list of all the files (and only files - directories will have to be removed in a separate step) in each of the build and source folder, sed chops off the extension, while comm -23 compares them, printing only the files unique to the build folder, which you can then deal with as you see fit (e.g., by feeding them to xargs rm).

hawski 3y ago
Using comm was exactly what I did use in my little experiment of a barebones SSG. I did save a list of generated files and compared them. This one liner is the meat of the whole solution: comm -23 <(awk 'NR>1' "$DSTDIR/build-info") <(find "$SRCDIR" -name "*$EXT" -type f -printf "%P\n" | tee >(gen_index) | xargs -n1 "$0" "$SRCDIR" "$DSTDIR" | sort | tee -a "$DSTDIR/build-info.new") | (cd "$DSTDIR" && xargs rm) Full source here: https://gist.github.com/hadrianw/060944011acfcadd889d937b960a5a1f https://gist.github.com/hadrianw/060944011acfcadd889d937b960...

throwaway858 3y ago

The shake build system (a general-purpose build system similar-to/better-than make) has a "prune" feature for exactly this purpose: http://neilmitchell.blogspot.com/2015/04/cleaning-stale-files-with-shake.html http://neilmitchell.blogspot.com/2015/04/cleaning-stale-file... But I think the best solution (that also works with make) is to have a "make dist" target that creates a final .tar.gz archive of the result. If the rule is written properly then it won't contain any stale files. The disadvantage is for large project it may be slow, but you are not supposed to use this rule during development (where it is useless anyway), only for releases (which still can be built incrementally -- only the final .tar.gz needs to be created from scratch)

deleted 3y ago

[deleted]

bogwog 3y ago

I think the best solution is to use something like webpack or vite or whatever. These usually have their own dev server and can watch directories for changes. My personal site is also using a custom make-like ssg, but after spending a disproportionate amount of time writing the bundling/packaging code, I decided to just switch over to one of these tools. It’s a solved problem, and it greatly reduced the complexity of my site.

dredmorbius 3y ago

File deletions and renames are common problems with many revision control / build systems. Other than the nuclear option ("make clean"), another is to have a specific rename / remove make target, so: make rm sourcefile or make mv sourcefile newsourcefile ... which will handle the deletion and renaming of both the original and generated targets. In practice for even fairly large blog and online projects, a make clean / make all cycle is reasonably quick (seconds, perhaps minutes), and is often called for when revising templates or other elements of the site design. If you're operating at a scale where rebuild time is a concern, you probably want to be using an actual CMS (content management system) in which source is managed in a database and generated dynamically on client access.