4 ms·

Eight minutes!? That’s over 100ms of CPU time per page. I’m baffled and more than a little horrified that you would call that highly optimised. I would honestly

Eight minutes!? That’s over 100ms of CPU time per page. I’m baffled and more than a little horrified that you would call that highly optimised. I would honestly expect anything calling itself highly optimised to be doing that in closer to 8 seconds. 1.7̅ms is ample for something carefully put together, probably even in JavaScript and with a database (fetch everything up front in one or so queries), though it’ll certainly be easier in a language like Rust. I admit I’d be decidedly impressed by a reasonably-featured JavaScript SSG doing 18,000 pages in 8 seconds; but more practically, in a JavaScript ecosystem, I’d be targeting definitely under a minute. I will admit I’ve grown used to Rust code and code that’s been designed to be fast. (Zola’s not that, by the way; it makes many decisions in favour of subjective ergonomics and flexibility at enormous runtime cost, as well as doing a lot of theoretically-unnecessary cloning and such all over the place, which could speed things up a lot if it were sorted out somehow (I haven’t contemplated the matter). For my own website with some admittedly very complex templates, and that’s probably Zola’s slowest part, it’s scaling at about 2ms per page with a warm disk cache. Actually, it does some multithreading, but much of the most expensive stuff in the process isn’t parallelised, so repeated runs average using almost a quarter of my 16-thread laptop. I feel I should multiply that 2ms by something, but I’m not sure quite what!) Edit: Later on the page: “As of October 2020, ElderGuide.com has ~20k pages and builds in 1 minute 22 seconds.” 4ms wall time (16ms if still operating on four cores) is much more encouraging. I guess the page could do with revision. Later still, “Build times on our ~20k page site are routinely less than 10 minutes on a modest VM.” which seems to be the first set of figures again.

nickreese 6y ago
I get what you are saying. In that example most of it was waiting on the remote database. (I’m ans SQL newb). That said, Elder.js makes it easy to see what is causing performance hits with full perf tracking. These days that same site is 1 min 22sec.
- chrismorgan 6y ago
  General rule of thumb with databases is to reduce the number of queries. Fetch large batches of information. This can be difficult to arrange with how such systems are commonly built and/or coded; some sort of intermediary between the consumer and the database can be warranted, which you can tell, “oh btw, these are the items that I’m going to want, d’you mind just fetching them all now so you can hand them directly to the consumers when they ask for them rather than issuing a new query for each?” In a well-done system like this, the database will be a minor component in the performance—templating and other JS-side transformations are much more expensive than fetching easily-identified rows from a database that’s been designed to handle zillions of requests per surprisingly-small-unit-of-time. I’m curious what fraction of the 1m22s it now takes is waiting for the database. Building performance tracking into the product? Good job! That shows me that you’re genuine in caring about it, which is still more encouraging. I’m going to investigate Elder.js.
Keats 6y ago
> much of the most expensive stuff in the process isn’t parallelised Are you sure? Markdown and template rendering of all pages/sections and image processing is done in parallel. I expect those to be the most expensive stuff. The things that are currently done sequentially are taxonomies (an oversight, the pages in them are rendered in parallel) and the search index generation (oversight as well).
- chrismorgan 6y ago
  No, I’m not sure; I apologise for being wrong (since you know what you’re doing). I just went searching for signs of threading and rayon and thought I was only finding it on the earlier parts of the process and image stuff. But I wonder why repeated runs of `zola build --drafts` on my site with no images for it to process average less than 25% of total CPU usage? I would have thought >170ms was long enough for it to hit its stride inside each process.
madacol 6y ago
> Eight minutes!? That’s over 100ms of CPU time per page. Your calculation seems wrong. (8×60×1000)/18000 = 26.666667 milliseconds per page
- chrismorgan 6y ago
  Four CPUs were involved, so multiply that figure by four.

nickreese 6y ago

I get what you are saying. In that example most of it was waiting on the remote database. (I’m ans SQL newb). That said, Elder.js makes it easy to see what is causing performance hits with full perf tracking. These days that same site is 1 min 22sec.

chrismorgan 6y ago
General rule of thumb with databases is to reduce the number of queries. Fetch large batches of information. This can be difficult to arrange with how such systems are commonly built and/or coded; some sort of intermediary between the consumer and the database can be warranted, which you can tell, “oh btw, these are the items that I’m going to want, d’you mind just fetching them all now so you can hand them directly to the consumers when they ask for them rather than issuing a new query for each?” In a well-done system like this, the database will be a minor component in the performance—templating and other JS-side transformations are much more expensive than fetching easily-identified rows from a database that’s been designed to handle zillions of requests per surprisingly-small-unit-of-time. I’m curious what fraction of the 1m22s it now takes is waiting for the database. Building performance tracking into the product? Good job! That shows me that you’re genuine in caring about it, which is still more encouraging. I’m going to investigate Elder.js.

Keats 6y ago

> much of the most expensive stuff in the process isn’t parallelised Are you sure? Markdown and template rendering of all pages/sections and image processing is done in parallel. I expect those to be the most expensive stuff. The things that are currently done sequentially are taxonomies (an oversight, the pages in them are rendered in parallel) and the search index generation (oversight as well).

chrismorgan 6y ago
No, I’m not sure; I apologise for being wrong (since you know what you’re doing). I just went searching for signs of threading and rayon and thought I was only finding it on the earlier parts of the process and image stuff. But I wonder why repeated runs of `zola build --drafts` on my site with no images for it to process average less than 25% of total CPU usage? I would have thought >170ms was long enough for it to hit its stride inside each process.

madacol 6y ago

> Eight minutes!? That’s over 100ms of CPU time per page. Your calculation seems wrong. (8×60×1000)/18000 = 26.666667 milliseconds per page

chrismorgan 6y ago
Four CPUs were involved, so multiply that figure by four.