4 ms·

Are static site generators only for small sites? E.g. if there is a site with 100,000 pages and the site owner wants to change something in the template (adding

Are static site generators only for small sites? E.g. if there is a site with 100,000 pages and the site owner wants to change something in the template (adding something to the sidebar, or footer, etc.) then does he have to regenerate and redeploy all the 100,000 pages every time when he makes a global change?

hanniabu 7y ago
Depends how it's setup. I remember reading something one time about how you can set it up where each individual build step is cached which would cut down on a lot of build time. So for instance, in this case it would only remind the sidebar.
vages 7y ago
I would guess that depends on whether the change affects the HTML. If it's only a CSS change, you could probably get away with redeploying the CSS file.
mxuribe 7y ago
I can not answer for large sites (nor for makesite.py specifically), but in general to your scenario, yes, all pages would need to be regenerated. I guess you could think of it as a build server. But instead of building code for an app that you would be deploying, you are generating your content for your static site/blog. That being said, you could then create a build server optimized to generate your content, and/or try a static site generator whose forte is the build/generation process (I think hugo fans tend to highlight this feature for that platform?). Maybe instead of generating all content on your local machine, you could keep all your content on a beefier machine, and generate it from there - which could process stuff much faster, etc. Even still, it is probably unlikely that one would have to alter things like sidebars or footers that often, right? Also, worse case scenario, something tells me that even if needing to regenerate that many pages, i mean, would it be ok if it takes just a few minutes - probably not the end of the world, right? (I guess it depends on the timeliness of the single post or footer change that needs to wait for 100,000 pages to get regenerated, eh?) I hope my comments helped a little.
DoctorOW 7y ago
Yes changing the global template would involve regenerating all 100,000 pages in most static site generators. The only exception being those rare ones which offload template rendering to client side (there's a couple that I've seen). Still, some static site generators like Hugo, are built for bigger sites specifically by being able to generate massive amounts of pages very quickly. Even in your 100,000 page example a site probably would be completely rebuilt within a second or two.
earthboundkid 7y ago
It depends on how you do it but generally speaking, the break even point is “will I be generating pages faster than they are accessed?” Eg if I have a site like Wired.com with a long tail of old articles, it might be that by generating it, I refresh old articles 500x daily for the 500 new story creation and edits, even though the average old article gets less than one view per day.
- cameronbrown 7y ago
  Maybe an incremental generator would solve this problem?
softwaredoug 7y ago
Based on our experience (n=1) I would say, yes, it works best for small sites. We have a medium site with on the order of 1000s of pages. Jekyll has become a bit unwieldy to rebuild a full static version of the site (takes about 5 mins for a full rebuild). I would certainly say an order of magnitude higher we’d definitely want server side Wordpress or similar.
- jacquesm 7y ago
  Try Hugo if you're using Jekyll. I switched and never looked back. Blazing fast and very solid, and very near to a drop-in replacement.
  - Tomte 7y ago
    Drop-in? Is there a compatibility layer for the totally different templating systems?
    - jacquesm 7y ago
      Hm, I switched from Jekyll to Hugo years ago and don't remember having to re-work the templates all that much but you're right, that may have been only easy because I never deviated much from the default. If you've done a lot of customization then you may have to do that all over again. Good point.
- jjjbokma 7y ago
  My blog, https://plurrrr.com/ https://plurrrr.com/ consists of 291 pages and is generated within less than 1.5 seconds. I use my own static site generator: https://github.com/john-bokma/tumblelog https://github.com/john-bokma/tumblelog
- dikaio 7y ago
  5 minute+ rebuilds are enough to move you to WordPress? 5 minute rebuilds does seem high but the benefits you get with static site generators (security, push “anywhere”, performance, etc) far outweigh having to manage everything else coming with a wordpress site. I suppose if you and your team are pushing new articles multiple times a day the builds could become annoying but even then I’d build/deploy and go grab a cup of joe; come back all done
z3t4 7y ago
The alternative is to generate the page per request. So it depends on the read/write ratio.
shakna 7y ago
It may require regenerating all pages, but depending on the generator that may not be particularly expensive - they may cache partial assets, so you replace part of the build asset and then the build is basically a bunch of cat-like calls, which tend to be extremely cheap. The largest static site I've managed was only a couple thousand pages (2138 pages from the last build log), but the build time was around 8 seconds (8.1 from the last build log) with cached build assets. I'd be happy using a static generator for most sites.
- dmortin 7y ago
  The build may be cheap, but you still have to upload the 100,000 files every time the global template changes.
  - shakna 7y ago
    Again, not always. You may only have to upload a _diff_ of the changes. And as global changes tend to be similar on every page, that diff may be extremely small for quite a lot of difference.
  - holri 7y ago
    use rsync it only uploads the diff (delta-transfer algorithm)
- dmortin 7y ago
  > It may require regenerating all pages, but depending on the generator that may not be particularly expensive - they may cache partial assets, so you replace part of the build asset and then the build is basically a bunch of cat-like calls, which tend to be extremely cheap. I've been thinking about this, and you can only do quick dependecy checks if the input is in files, right? So if the actual data is in a database then you do have to have some process to dump the changed data into files from the database, so the build system can pick them up. Otherwise, if the build system takes data directly from the db then it doesn't know if the data is changed, so it has to retrieve everything from database which won't be efficient.

hanniabu 7y ago

Depends how it's setup. I remember reading something one time about how you can set it up where each individual build step is cached which would cut down on a lot of build time. So for instance, in this case it would only remind the sidebar.

vages 7y ago

I would guess that depends on whether the change affects the HTML. If it's only a CSS change, you could probably get away with redeploying the CSS file.

mxuribe 7y ago

I can not answer for large sites (nor for makesite.py specifically), but in general to your scenario, yes, all pages would need to be regenerated. I guess you could think of it as a build server. But instead of building code for an app that you would be deploying, you are generating your content for your static site/blog. That being said, you could then create a build server optimized to generate your content, and/or try a static site generator whose forte is the build/generation process (I think hugo fans tend to highlight this feature for that platform?). Maybe instead of generating all content on your local machine, you could keep all your content on a beefier machine, and generate it from there - which could process stuff much faster, etc. Even still, it is probably unlikely that one would have to alter things like sidebars or footers that often, right? Also, worse case scenario, something tells me that even if needing to regenerate that many pages, i mean, would it be ok if it takes just a few minutes - probably not the end of the world, right? (I guess it depends on the timeliness of the single post or footer change that needs to wait for 100,000 pages to get regenerated, eh?) I hope my comments helped a little.

DoctorOW 7y ago

Yes changing the global template would involve regenerating all 100,000 pages in most static site generators. The only exception being those rare ones which offload template rendering to client side (there's a couple that I've seen). Still, some static site generators like Hugo, are built for bigger sites specifically by being able to generate massive amounts of pages very quickly. Even in your 100,000 page example a site probably would be completely rebuilt within a second or two.

earthboundkid 7y ago

It depends on how you do it but generally speaking, the break even point is “will I be generating pages faster than they are accessed?” Eg if I have a site like Wired.com with a long tail of old articles, it might be that by generating it, I refresh old articles 500x daily for the 500 new story creation and edits, even though the average old article gets less than one view per day.

cameronbrown 7y ago
Maybe an incremental generator would solve this problem?

softwaredoug 7y ago

Based on our experience (n=1) I would say, yes, it works best for small sites. We have a medium site with on the order of 1000s of pages. Jekyll has become a bit unwieldy to rebuild a full static version of the site (takes about 5 mins for a full rebuild). I would certainly say an order of magnitude higher we’d definitely want server side Wordpress or similar.

jacquesm 7y ago
Try Hugo if you're using Jekyll. I switched and never looked back. Blazing fast and very solid, and very near to a drop-in replacement.
- Tomte 7y ago
  Drop-in? Is there a compatibility layer for the totally different templating systems?
  - jacquesm 7y ago
    Hm, I switched from Jekyll to Hugo years ago and don't remember having to re-work the templates all that much but you're right, that may have been only easy because I never deviated much from the default. If you've done a lot of customization then you may have to do that all over again. Good point.
jjjbokma 7y ago
My blog, https://plurrrr.com/ https://plurrrr.com/ consists of 291 pages and is generated within less than 1.5 seconds. I use my own static site generator: https://github.com/john-bokma/tumblelog https://github.com/john-bokma/tumblelog
dikaio 7y ago
5 minute+ rebuilds are enough to move you to WordPress? 5 minute rebuilds does seem high but the benefits you get with static site generators (security, push “anywhere”, performance, etc) far outweigh having to manage everything else coming with a wordpress site. I suppose if you and your team are pushing new articles multiple times a day the builds could become annoying but even then I’d build/deploy and go grab a cup of joe; come back all done

z3t4 7y ago

The alternative is to generate the page per request. So it depends on the read/write ratio.

shakna 7y ago

It may require regenerating all pages, but depending on the generator that may not be particularly expensive - they may cache partial assets, so you replace part of the build asset and then the build is basically a bunch of cat-like calls, which tend to be extremely cheap. The largest static site I've managed was only a couple thousand pages (2138 pages from the last build log), but the build time was around 8 seconds (8.1 from the last build log) with cached build assets. I'd be happy using a static generator for most sites.

dmortin 7y ago
The build may be cheap, but you still have to upload the 100,000 files every time the global template changes.
- shakna 7y ago
  Again, not always. You may only have to upload a _diff_ of the changes. And as global changes tend to be similar on every page, that diff may be extremely small for quite a lot of difference.
- holri 7y ago
  use rsync it only uploads the diff (delta-transfer algorithm)
dmortin 7y ago
> It may require regenerating all pages, but depending on the generator that may not be particularly expensive - they may cache partial assets, so you replace part of the build asset and then the build is basically a bunch of cat-like calls, which tend to be extremely cheap. I've been thinking about this, and you can only do quick dependecy checks if the input is in files, right? So if the actual data is in a database then you do have to have some process to dump the changed data into files from the database, so the build system can pick them up. Otherwise, if the build system takes data directly from the db then it doesn't know if the data is changed, so it has to retrieve everything from database which won't be efficient.