7 ms·

I had a website which I was rendering statically using Jekyll. Overtime I had millions of pages, rendering was getting too time consuming and was taking up far

I had a website which I was rendering statically using Jekyll. Overtime I had millions of pages, rendering was getting too time consuming and was taking up far too much of space. I then moved on to PHP. Worked like a charm with $15/mo server on Google cloud. Excellent search performance as well.

throwanem 4y ago
Millions of pages? What was the site?
- KorematsuFredt 4y ago
  It was basically a clone of https://www.gutenberg.org/ https://www.gutenberg.org/.
- robin_reala 4y ago
  It’s not so unusual. My employer has a product line of ~40k products, in ~45 markets, with ~2 languages per market, which gets you to 3.6m product pages immediately, and that’s leaving aside any other pages.
  - throwanem 4y ago
    That's fair, but I bet you're not using a static site generator to produce your PDPs. (Or, if so, I'd love to read an engineering blog post about it! Static PDPs would be a solid optimization in a lot of ways, and while I'm no longer in a role where that could be directly useful, it'd still be interesting to find out what challenges a team overcame and how in the course of getting there.)
    - robin_reala 4y ago
      Most content on our PDPs are statically pre-rendered, yes. Some stuff we obviously can’t of course (e.g. some of the recommendations that are based on your personal behaviour), and those tend to be mini-SPAs. We save from having to re-render everything if, say, a menu item changes, by compositing page fragments at a CDN level and caching the result for a shorter amount of time. This setup is not particularly unusual for a microfrontends environment I don’t believe.
      - throwanem 4y ago
        We did much the same, as I recall. (I didn't spend very much time close to the frontend in that role, so I may misremember, but that description has a familiar ring.) It's probably about as far as the concept can reasonably be taken; in theory I could see something more like "true" SSR with some probably complex hydration, but the engineering time would likely make it uneconomic to pursue what would likely be marginal benefit.
toastercat 4y ago
Yeah, I don't think a site with "millions of pages" is the right fit for SSGs.
di4na 4y ago
I would point out that this is not the hit you think at SSG but at their current build system. You could imagine a distributed build system for this that would build it really fast for competitive price on any ci/lambda/batch system. The fact we barely hear anyone talk about this is the strange twist of history the parent made.
- wtetzner 4y ago
  But at that point, what exactly are you gaining?
  - di4na 4y ago
    Lower cost, better use of hardware, better scalability, easier debugging, more robustness to failure, etc etc
    - wtetzner 4y ago
      It's unclear that any of those are true if you need to maintain a distributed build system.
      - di4na 4y ago
        You don't have to. The tooling can be built to do it. I advise the Build Systems a la Carte for a view of what already exist. You don't maintain jekyll today...
    - account42 4y ago
      Plus, no way for outside actors to trigger the rendering is great for security.

throwanem 4y ago

Millions of pages? What was the site?

KorematsuFredt 4y ago
It was basically a clone of https://www.gutenberg.org/ https://www.gutenberg.org/.
robin_reala 4y ago
It’s not so unusual. My employer has a product line of ~40k products, in ~45 markets, with ~2 languages per market, which gets you to 3.6m product pages immediately, and that’s leaving aside any other pages.
- throwanem 4y ago
  That's fair, but I bet you're not using a static site generator to produce your PDPs. (Or, if so, I'd love to read an engineering blog post about it! Static PDPs would be a solid optimization in a lot of ways, and while I'm no longer in a role where that could be directly useful, it'd still be interesting to find out what challenges a team overcame and how in the course of getting there.)
  - robin_reala 4y ago
    Most content on our PDPs are statically pre-rendered, yes. Some stuff we obviously can’t of course (e.g. some of the recommendations that are based on your personal behaviour), and those tend to be mini-SPAs. We save from having to re-render everything if, say, a menu item changes, by compositing page fragments at a CDN level and caching the result for a shorter amount of time. This setup is not particularly unusual for a microfrontends environment I don’t believe.
    - throwanem 4y ago
      We did much the same, as I recall. (I didn't spend very much time close to the frontend in that role, so I may misremember, but that description has a familiar ring.) It's probably about as far as the concept can reasonably be taken; in theory I could see something more like "true" SSR with some probably complex hydration, but the engineering time would likely make it uneconomic to pursue what would likely be marginal benefit.

toastercat 4y ago

Yeah, I don't think a site with "millions of pages" is the right fit for SSGs.

di4na 4y ago

I would point out that this is not the hit you think at SSG but at their current build system. You could imagine a distributed build system for this that would build it really fast for competitive price on any ci/lambda/batch system. The fact we barely hear anyone talk about this is the strange twist of history the parent made.

wtetzner 4y ago
But at that point, what exactly are you gaining?
- di4na 4y ago
  Lower cost, better use of hardware, better scalability, easier debugging, more robustness to failure, etc etc
  - wtetzner 4y ago
    It's unclear that any of those are true if you need to maintain a distributed build system.
    - di4na 4y ago
      You don't have to. The tooling can be built to do it. I advise the Build Systems a la Carte for a view of what already exist. You don't maintain jekyll today...
  - account42 4y ago
    Plus, no way for outside actors to trigger the rendering is great for security.