6 ms·

A static site generator written in POSIX shell

76 points by todsacerdoti 9mo ago

mrwh 9mo ago
Love it! My personal sites tend to start hand-written, evolve into a bash script, occasionally advance into python, but mostly just stay in bash, because it's convenient, doesn't need to please anyone else (nor would it!), and is already built for text processing into files. (I write a lot of scrappy shell scripts for my job too. I guess I should start asking an AI to generate whatever I want directly instead. Cobbling together a script is more fun though...)
paseante 9mo ago
[flagged]
- pixelatedindex 9mo ago
  Care to elaborate? I don’t see anything of issue here, in fact it’s pretty good.
- hereonout2 9mo ago
  I don't get this kind of indignation against anything shell related.
- harperlee 9mo ago
  Please have a moment to read the title of this site again :)
riedel 9mo ago
The title seems to be a bit misleading IMHO because it does not really only use `sh` but heavily `sed` it seems, which is a whole programming language well suited for templating. I've in the old days written a Macromedia Dreamweaver compatible template engine using such a scheme, which I personally used quite a long time actually without Dreamweaver because this WordPress madness was even a thing.
lombasihir 9mo ago
why not perl?
- oguz-ismail2 9mo ago
  fell out of popularity lately
- adityaathalye 9mo ago
  Sometimes we are very self-aware, but we still do things just because... - It is a wee publishing system made of pipelined workflows, optionally driven by streams of file events (for the hotreloadin' bits). - It will not surprise a Perl/PHP gentleperson hacker from the last century. - It exists because one whistles silly tunes and shaves yaks. - my site maker's README
- accrual 9mo ago
  Just anecdata, but as someone who first learned HTML/JS/CSS, then PHP, shell, TypeScript, Python, other popular langs... Perl's syntax hurts my brain. :(
  - lombasihir 9mo ago
    yup, been there, even the basic perl sigil might be confusing for some people.
vbezhenar 9mo ago
I wonder what would be a good way to generate a website with minimal software installations, for example in standard github runner image. This example uses comrak tool to process markdown into HTML. I've come up with using Java and XSLT. Java is installed in Github Runner image and there's built-in XSLT support in Java standard library. You can write HTML and use XSLT to add header, footer and do other processings if necessary. So basically I want to generate a website in github runner without accessing network to install something else. I guess one could just `cat header.html content.html footer.html` but that requires a lot of tiny things like extracting title from content and inserting it into header, etc. Nothing that lots of greps and seds couldn't handle, of course...
- lionkor 9mo ago
  A good way to do that with anything, I found, is a Makefile which downloads and compiles the few things it needs :D
  - accrual 9mo ago
    Make seems like a perfect choice for orchestrating a UNIX SSG.
- qznc 9mo ago
  For GitHub just use Jekyll provided by GitHub itself?
  - aashvik 9mo ago
    Jekyll is fine, but it's quite slow, and new versions often introduce breaking changes.
- weitendorf 9mo ago
  We do basically this for our tests in statue: https://github.com/accretional/statue/tree/main/test/hermetic https://github.com/accretional/statue/tree/main/test/hermeti... npm pack builds the file locally, then we expose it to the container filesystem where we do a build and check the outputs. You can move dependencies to bundledDependencies in npm to embed them in the image. However, this is assuming you're rebuilding the static site generator itself every time. If you just want to build a site using an existing static site generator, it's much easier provided that the site generator itself is easy to use (for example, ours has a one-liner to take you all the way from nothing to a local static site running on localhost, see https://statue.dev https://statue.dev) If you aren't changing the SSG code itself between container runs you'd just mount the markdown to the container and pre-install the ssg in the Dockerfile itself. For statue.dev that would just be a Dockerfile almost exactly the same as the one we use already, except you'd use your own script, and RUN this in the Dockerfile itself: yes | npx sv create . --template minimal --types ts --no-add-ons --install npm && npm install statue-ssg && npx statue init && npm install In your script you'd just npm run build then do whatever it is you want to do to send the files somewhere, and wherever starts the script, you'd do something like -v "pathtomymarkdown/foo:/test-package/" - not sure how to do this in github runners. Depending on how interested you/other people are in doing this with statue.dev, we could prob get something like this (where the markdown is parameterized, not the npm package) working by Tuesday. We're building out the sandbox/build features on our cloud platform as we speak, this could be one of the first use cases/defaults.
avian 9mo ago
My first blog was made with NanoBlogger, which was what you would call a static site generator today. It's made in Bash. I remember setting up a few of these back in the day. https://nanoblogger.sourceforge.net/ https://nanoblogger.sourceforge.net/
- hkt 9mo ago
  I've set up a few with that too, it was a great bit of software and a great concept.
msephton 9mo ago
This is fun. I do love a good bit of shell scripting. Also makes me want to make my own little ssg in whatever way I see fit.
- adityaathalye 9mo ago
  Very much pro-this. Mine says: It's job is to help me make my website. Thus, its scope, (mis)feature set, polish will always be production-grade, where production is "works on my machine(s)" :)
- accrual 9mo ago
  > Also makes me want to make my own little ssg in whatever way I see fit. Go for it, it's fun! I made one in TypeScript recently as a way to improve my skills there. Be warned though, completing one SSG may lead to wanting to start another haha.
- msephton 9mo ago
  I'm back! I was nerd-sniped. I made a little ssg for the apps I've created over the last year or two. Being able to generate themed, consistent pages for all my apps is just a huge win. I used node for no real reason other than I've been writing a game in JavaScript recently.
tasuki 9mo ago
Yes, in shell, but requiring 'comrak', whatever that is. I also generate[0] html[1] from markdown for my vimwiki. [0]: https://github.com/tasuki/vitwiki/blob/master/build.sh https://github.com/tasuki/vitwiki/blob/master/build.sh [1]: https://wiki.tasuki.org/ https://wiki.tasuki.org/
- hkt 9mo ago
  https://github.com/kivikakk/comrak https://github.com/kivikakk/comrak - common mark renderer I do somewhat wonder why not pandoc, but it is still an interesting project
  - MallocVoidstar 9mo ago
    pandoc is big (nearly 200MB binary on Debian) and does far more than just Markdown -> HTML
    - tasuki 9mo ago
      Wow I never knew pandoc was 200MB! It's only a 26MB download, but 200MB binary is kind of insane... It is by far the biggest binary I have in `/usr/bin/` (the second being `blender` at 90MB - understandable I guess! - and the third being `stack` at 75MB - haskell again!)
  - aashvik 9mo ago
    Pandoc is awesome, but it has a huge binary and (just tested) is nearly thirty times slower than comrak!
Piraty 9mo ago
i write in markdown, use lowdown + make to build html, push the html to a branch that my hoster serves from : https://piraty.dev/ https://piraty.dev/
austinjp 9mo ago
Lovely. It's gratifying to see this broadly matches my own personal SSG: bash, find, sed, envsubst, and pandoc rather than comrak.
p4bl0 9mo ago
I did something similar a (very) long time ago (15 years back!), a static site generator and blog engine entirely coded in sh (yes, not even Bash) + coreutils. The idea was to use those scripts in git hooks, as they provide a template engine that allows to use the Git repository as a storage backend and publishing method, both for posts content (as file or as commit bodies) and for meta data about posts (author, date, etc). It was fun to build and got a few dozen users at the time, some even contributed small bugfixes and features :). The README is here: https://p4bl0.net/shebang/fugitive-readme.html https://p4bl0.net/shebang/fugitive-readme.html
jimnotgym 9mo ago
Slight tangent, what is a good markdown editor for Linux? I need spell check and a preview at least.
- AlecSchueler 9mo ago
  I just use vim and have a binding for F5 to build the current document then open the result in the browser window.
- accrual 9mo ago
  I sometimes write directly in VSCode and use the preview mode there. I've also used Obsidian but it always felt kinda heavy and distracting for pure writing, though great for managing a large group of .mds in folders.
eemil 9mo ago
Bit of an unknown feature, but tree can output HTML. I've used tree -H to generate directory listings more than once.
- MomsAVoxell 9mo ago
  I use tree almost every single day and I never realized this. Thank you so much for this wonderful factoid, which has simplified my life immensely, seriously. Going to also adopt a mental note “rtfm||gtfo, ffs.”
- adityaathalye 9mo ago
  WAT??? TIL!!! Thank you. Also thank you baker and moore and rocher and sesser and tokoro, you devils you. tree v2.1.1 © 1996 - 2023 by Steve Baker and Thomas Moore HTML output hacked and copyleft © 1998 by Francesc Rocher JSON output hacked and copyleft © 2014 by Florian Sesser Charsets / OS/2 support © 2001 by Kyosuke Tokoro
jrm4 9mo ago
Not quite shell -- but obligatory: I've been using http://zim-wiki.org http://zim-wiki.org for now .. decades? As my main driver for most everything in my life, including my personal and teaching website, custom template I made. jrm4.com
teo_zero 9mo ago
The shell might be POSIX, but sed uses a GNU extension -- namely, the double address "a,b". EDIT: Sorry, I was wrong, the double address is indeed POSIX. I was thinking of the GNU syntax: a,+n
- deng 9mo ago
  From the fine article: Why shell? Well, not really because it’s portable, as despite being a “POSIX script”, most of the date and sed tricks I do don’t work on the BSD versions of those commands, with comrak, additionally, being a dependency.

76 points by todsacerdoti 9mo ago

mrwh 9mo ago

Love it! My personal sites tend to start hand-written, evolve into a bash script, occasionally advance into python, but mostly just stay in bash, because it's convenient, doesn't need to please anyone else (nor would it!), and is already built for text processing into files. (I write a lot of scrappy shell scripts for my job too. I guess I should start asking an AI to generate whatever I want directly instead. Cobbling together a script is more fun though...)

paseante 9mo ago

[flagged]

pixelatedindex 9mo ago
Care to elaborate? I don’t see anything of issue here, in fact it’s pretty good.
hereonout2 9mo ago
I don't get this kind of indignation against anything shell related.
harperlee 9mo ago
Please have a moment to read the title of this site again :)

riedel 9mo ago

The title seems to be a bit misleading IMHO because it does not really only use `sh` but heavily `sed` it seems, which is a whole programming language well suited for templating. I've in the old days written a Macromedia Dreamweaver compatible template engine using such a scheme, which I personally used quite a long time actually without Dreamweaver because this WordPress madness was even a thing.

lombasihir 9mo ago

why not perl?

oguz-ismail2 9mo ago
fell out of popularity lately
adityaathalye 9mo ago
Sometimes we are very self-aware, but we still do things just because... - It is a wee publishing system made of pipelined workflows, optionally driven by streams of file events (for the hotreloadin' bits). - It will not surprise a Perl/PHP gentleperson hacker from the last century. - It exists because one whistles silly tunes and shaves yaks. - my site maker's README
accrual 9mo ago
Just anecdata, but as someone who first learned HTML/JS/CSS, then PHP, shell, TypeScript, Python, other popular langs... Perl's syntax hurts my brain. :(
- lombasihir 9mo ago
  yup, been there, even the basic perl sigil might be confusing for some people.

vbezhenar 9mo ago

I wonder what would be a good way to generate a website with minimal software installations, for example in standard github runner image. This example uses comrak tool to process markdown into HTML. I've come up with using Java and XSLT. Java is installed in Github Runner image and there's built-in XSLT support in Java standard library. You can write HTML and use XSLT to add header, footer and do other processings if necessary. So basically I want to generate a website in github runner without accessing network to install something else. I guess one could just `cat header.html content.html footer.html` but that requires a lot of tiny things like extracting title from content and inserting it into header, etc. Nothing that lots of greps and seds couldn't handle, of course...

lionkor 9mo ago
A good way to do that with anything, I found, is a Makefile which downloads and compiles the few things it needs :D
- accrual 9mo ago
  Make seems like a perfect choice for orchestrating a UNIX SSG.
qznc 9mo ago
For GitHub just use Jekyll provided by GitHub itself?
- aashvik 9mo ago
  Jekyll is fine, but it's quite slow, and new versions often introduce breaking changes.
weitendorf 9mo ago
We do basically this for our tests in statue: https://github.com/accretional/statue/tree/main/test/hermetic https://github.com/accretional/statue/tree/main/test/hermeti... npm pack builds the file locally, then we expose it to the container filesystem where we do a build and check the outputs. You can move dependencies to bundledDependencies in npm to embed them in the image. However, this is assuming you're rebuilding the static site generator itself every time. If you just want to build a site using an existing static site generator, it's much easier provided that the site generator itself is easy to use (for example, ours has a one-liner to take you all the way from nothing to a local static site running on localhost, see https://statue.dev https://statue.dev) If you aren't changing the SSG code itself between container runs you'd just mount the markdown to the container and pre-install the ssg in the Dockerfile itself. For statue.dev that would just be a Dockerfile almost exactly the same as the one we use already, except you'd use your own script, and RUN this in the Dockerfile itself: yes | npx sv create . --template minimal --types ts --no-add-ons --install npm && npm install statue-ssg && npx statue init && npm install In your script you'd just npm run build then do whatever it is you want to do to send the files somewhere, and wherever starts the script, you'd do something like -v "pathtomymarkdown/foo:/test-package/" - not sure how to do this in github runners. Depending on how interested you/other people are in doing this with statue.dev, we could prob get something like this (where the markdown is parameterized, not the npm package) working by Tuesday. We're building out the sandbox/build features on our cloud platform as we speak, this could be one of the first use cases/defaults.

avian 9mo ago

My first blog was made with NanoBlogger, which was what you would call a static site generator today. It's made in Bash. I remember setting up a few of these back in the day. https://nanoblogger.sourceforge.net/ https://nanoblogger.sourceforge.net/

hkt 9mo ago
I've set up a few with that too, it was a great bit of software and a great concept.

msephton 9mo ago

This is fun. I do love a good bit of shell scripting. Also makes me want to make my own little ssg in whatever way I see fit.

adityaathalye 9mo ago
Very much pro-this. Mine says: It's job is to help me make my website. Thus, its scope, (mis)feature set, polish will always be production-grade, where production is "works on my machine(s)" :)
accrual 9mo ago
> Also makes me want to make my own little ssg in whatever way I see fit. Go for it, it's fun! I made one in TypeScript recently as a way to improve my skills there. Be warned though, completing one SSG may lead to wanting to start another haha.
msephton 9mo ago
I'm back! I was nerd-sniped. I made a little ssg for the apps I've created over the last year or two. Being able to generate themed, consistent pages for all my apps is just a huge win. I used node for no real reason other than I've been writing a game in JavaScript recently.

tasuki 9mo ago

Yes, in shell, but requiring 'comrak', whatever that is. I also generate[0] html[1] from markdown for my vimwiki. [0]: https://github.com/tasuki/vitwiki/blob/master/build.sh https://github.com/tasuki/vitwiki/blob/master/build.sh [1]: https://wiki.tasuki.org/ https://wiki.tasuki.org/

hkt 9mo ago
https://github.com/kivikakk/comrak https://github.com/kivikakk/comrak - common mark renderer I do somewhat wonder why not pandoc, but it is still an interesting project
- MallocVoidstar 9mo ago
  pandoc is big (nearly 200MB binary on Debian) and does far more than just Markdown -> HTML
  - tasuki 9mo ago
    Wow I never knew pandoc was 200MB! It's only a 26MB download, but 200MB binary is kind of insane... It is by far the biggest binary I have in `/usr/bin/` (the second being `blender` at 90MB - understandable I guess! - and the third being `stack` at 75MB - haskell again!)
- aashvik 9mo ago
  Pandoc is awesome, but it has a huge binary and (just tested) is nearly thirty times slower than comrak!

Piraty 9mo ago

i write in markdown, use lowdown + make to build html, push the html to a branch that my hoster serves from : https://piraty.dev/ https://piraty.dev/

austinjp 9mo ago

Lovely. It's gratifying to see this broadly matches my own personal SSG: bash, find, sed, envsubst, and pandoc rather than comrak.

p4bl0 9mo ago

I did something similar a (very) long time ago (15 years back!), a static site generator and blog engine entirely coded in sh (yes, not even Bash) + coreutils. The idea was to use those scripts in git hooks, as they provide a template engine that allows to use the Git repository as a storage backend and publishing method, both for posts content (as file or as commit bodies) and for meta data about posts (author, date, etc). It was fun to build and got a few dozen users at the time, some even contributed small bugfixes and features :). The README is here: https://p4bl0.net/shebang/fugitive-readme.html https://p4bl0.net/shebang/fugitive-readme.html

jimnotgym 9mo ago

Slight tangent, what is a good markdown editor for Linux? I need spell check and a preview at least.

AlecSchueler 9mo ago
I just use vim and have a binding for F5 to build the current document then open the result in the browser window.
accrual 9mo ago
I sometimes write directly in VSCode and use the preview mode there. I've also used Obsidian but it always felt kinda heavy and distracting for pure writing, though great for managing a large group of .mds in folders.

eemil 9mo ago

Bit of an unknown feature, but tree can output HTML. I've used tree -H to generate directory listings more than once.

MomsAVoxell 9mo ago
I use tree almost every single day and I never realized this. Thank you so much for this wonderful factoid, which has simplified my life immensely, seriously. Going to also adopt a mental note “rtfm||gtfo, ffs.”
adityaathalye 9mo ago
WAT??? TIL!!! Thank you. Also thank you baker and moore and rocher and sesser and tokoro, you devils you. tree v2.1.1 © 1996 - 2023 by Steve Baker and Thomas Moore HTML output hacked and copyleft © 1998 by Francesc Rocher JSON output hacked and copyleft © 2014 by Florian Sesser Charsets / OS/2 support © 2001 by Kyosuke Tokoro

jrm4 9mo ago

Not quite shell -- but obligatory: I've been using http://zim-wiki.org http://zim-wiki.org for now .. decades? As my main driver for most everything in my life, including my personal and teaching website, custom template I made. jrm4.com

teo_zero 9mo ago

The shell might be POSIX, but sed uses a GNU extension -- namely, the double address "a,b". EDIT: Sorry, I was wrong, the double address is indeed POSIX. I was thinking of the GNU syntax: a,+n

deng 9mo ago
From the fine article: Why shell? Well, not really because it’s portable, as despite being a “POSIX script”, most of the date and sed tricks I do don’t work on the BSD versions of those commands, with comrak, additionally, being a dependency.