4 ms·

I'm going to try this out at some point this week, but the one item that makes me pause: the sitemap. Hugo is fast, but it doesn't do incremental builds. Once y

I'm going to try this out at some point this week, but the one item that makes me pause: the sitemap. Hugo is fast, but it doesn't do incremental builds. Once you get to the point where you're publishing 10's of thousands of posts, Hugo starts to get real slow (like 20 minutes to build, even after enabling template caching), so an increase in speed during build time is welcome. But once you get to 50K posts you realize that Hugo doesn't follow Google's convention to break your sitemap into separate files with an index map[0]. Hugo doesn't have a handler for this and as of now I haven't figured out how to modify the sitemap compilation process to be compliant (not like I've tried too hard, I'm not native with Go and the code isn't clear to me where the sitemap is being built). If Blades would handle the sitemap issue for me without needing a plugin or modification, that would be huge for a lot of people on the Hugo community boards. [0] https://support.google.com/webmasters/answer/183668?hl=en https://support.google.com/webmasters/answer/183668?hl=en

gcatalfamo 6y ago
You are not wrong, although I would argue that use cases where you have 50k blog posts (or pages) are not that common...
- chrsstrm 6y ago
  I should have been more clear on the terminology - 50K URLs. Which doesn't always mean 50K posts or pages, but those plus all your taxonomy pages plus all your pagination pages. A 10K post or page site can give you 50K URLs pretty easily if it uses taxonomies and pagination. My case might be unique where the site is being built at the end of a pipeline that sorts through millions of rows of data and gives you the content you want based on your filters, but it's honestly not difficult to get to 50K URLs for many sites that have some age on them.
  - codegeek 6y ago
    I would say a 10k page or post site is also very rare.
    - grtehy 6y ago
      Nah, you'd be wrong. I use a password protected static site as my knowledge dump. I accumulate 1000s of posts in months.
      - Aeolun 6y ago
        You doing it is not the definition of ‘not rare’ :P
        HoolaBoola 6y ago
        I think the conversation here is getting a little sidetracked :) (not indicating Aeolun specifically, just replied to the lowest level)
    - ThePadawan 6y ago
      Seems to me like every newspaper would accumulate 10k pages within a year or two?
      - eloisant 6y ago
        I think static generators are mostly used for personal blogs or pages, but I'd love to hear about newspapers that use them!
chillfox 6y ago
What kind of sites are you building with static site generators that has 10K+ pages of content? I always got the impression that they were more for small blogs/sites.
- chrsstrm 6y ago
  You can take your slow dynamic site and throw some caching in front of it or you can just skip all the moving parts and make the whole thing static from the start and let nginx do what it does best and serve static files super fast. I've done a lot of work for media and news sites that previously ran on WordPress and it is not uncommon to see 10K posts or more. I would argue static site generators are considerably underused for many types of sites that genuinely don't need an active datastore connection.
  - gcatalfamo 6y ago
    This is actually very interesting. Can you tell more on the recommended stack for that kind of websites? You said media but I guess also directories
    - chrsstrm 6y ago
      There are so many options but it genuinely depends on how your content is being generated. If you have everything in a giant CSV or JSON file then there's not much work to do at all. Many shops I worked with were used to WordPress and after WP released the API functionality it became possible to completely decouple the editing & publish function from the front end display. You can still use a securely locked-down WP install for content creation and editing, then just use the (also locked-down) WP API endpoint as your data source that gets fed into your site generator. Hugo has an awesome feature where if you feed it a URL it will go fetch your content for you, so it doesn't even need to exist in the /content dir prior to build. Thanks to the free tiers for Gitlab CI/CD or Github Actions, your repo only contains theme code and with a 10-line CI/CD config file and a container image that has Hugo installed, you just need a trigger to run the build pipeline - then you're left with the static site that you can publish anywhere you like. If you don't need the user roles and editor experience you get from WP then you can feed your generator the content from any source you like as long as the files have the correct frontmatter to turn them into pages/posts. You might need a little glue script here and there to chain systems together, but it's not super difficult to adapt to whatever pipeline you need after you understand where your content comes from.
marosgrego 6y ago
Blades currently doesn't break the sitemap, but it can easily be implemented. I see that 50k pages is a maximum for one sitemap file. Do you think it should break into separate pages only after reaching 50k, or sooner?
- chrsstrm 6y ago
  IMO the easiest implementation would always break into an index and sitemap(s) as needed. 49,999 URLs? 1 index and 1 sitemap. 50K+ URLs? 1 index and sitemaps as needed. This gives you the most consistent output and users who cross the 50K threshold over time will not need to update their sitemap in search tools to be compliant since pointing to the index will already deal with that.
DrBazza 6y ago
Just run my company's internal Hugo build on commodity hardware (4 cores), and I get about 1000 pages per second, i.e. a 3966 page site took "4841ms" - so 20,000 pages assuming linear scaling, would still only 20-30 seconds. Not minutes. Perhaps our theme isn't complex enough?
- chrsstrm 6y ago
  Interesting. Are you using Related [0] content? I always suspected the long build was due to related content but the build profiler isn't specific enough to point that out. [0] https://gohugo.io/content-management/related/ https://gohugo.io/content-management/related/
chopraaa 6y ago
I've faced this issue as well. I run a Python script after the build has completed that reads the entire sitemap and chunks it into parts of 50k. +1 for mentioning incremental builds as well. These two features are often overlooked and people just ask "why do you even have so many pages?" and it ends there because nobody wants to entertain the power user. A third disadvantage I've seen is search. I use fuse to generate a JSON for client side search but it takes a lot of memory to render and is a cause for concern.

gcatalfamo 6y ago

You are not wrong, although I would argue that use cases where you have 50k blog posts (or pages) are not that common...

chrsstrm 6y ago
I should have been more clear on the terminology - 50K URLs. Which doesn't always mean 50K posts or pages, but those plus all your taxonomy pages plus all your pagination pages. A 10K post or page site can give you 50K URLs pretty easily if it uses taxonomies and pagination. My case might be unique where the site is being built at the end of a pipeline that sorts through millions of rows of data and gives you the content you want based on your filters, but it's honestly not difficult to get to 50K URLs for many sites that have some age on them.
- codegeek 6y ago
  I would say a 10k page or post site is also very rare.
  - grtehy 6y ago
    Nah, you'd be wrong. I use a password protected static site as my knowledge dump. I accumulate 1000s of posts in months.
    - Aeolun 6y ago
      You doing it is not the definition of ‘not rare’ :P
      - HoolaBoola 6y ago
        I think the conversation here is getting a little sidetracked :) (not indicating Aeolun specifically, just replied to the lowest level)
  - ThePadawan 6y ago
    Seems to me like every newspaper would accumulate 10k pages within a year or two?
    - eloisant 6y ago
      I think static generators are mostly used for personal blogs or pages, but I'd love to hear about newspapers that use them!

chillfox 6y ago

What kind of sites are you building with static site generators that has 10K+ pages of content? I always got the impression that they were more for small blogs/sites.

chrsstrm 6y ago
You can take your slow dynamic site and throw some caching in front of it or you can just skip all the moving parts and make the whole thing static from the start and let nginx do what it does best and serve static files super fast. I've done a lot of work for media and news sites that previously ran on WordPress and it is not uncommon to see 10K posts or more. I would argue static site generators are considerably underused for many types of sites that genuinely don't need an active datastore connection.
- gcatalfamo 6y ago
  This is actually very interesting. Can you tell more on the recommended stack for that kind of websites? You said media but I guess also directories
  - chrsstrm 6y ago
    There are so many options but it genuinely depends on how your content is being generated. If you have everything in a giant CSV or JSON file then there's not much work to do at all. Many shops I worked with were used to WordPress and after WP released the API functionality it became possible to completely decouple the editing & publish function from the front end display. You can still use a securely locked-down WP install for content creation and editing, then just use the (also locked-down) WP API endpoint as your data source that gets fed into your site generator. Hugo has an awesome feature where if you feed it a URL it will go fetch your content for you, so it doesn't even need to exist in the /content dir prior to build. Thanks to the free tiers for Gitlab CI/CD or Github Actions, your repo only contains theme code and with a 10-line CI/CD config file and a container image that has Hugo installed, you just need a trigger to run the build pipeline - then you're left with the static site that you can publish anywhere you like. If you don't need the user roles and editor experience you get from WP then you can feed your generator the content from any source you like as long as the files have the correct frontmatter to turn them into pages/posts. You might need a little glue script here and there to chain systems together, but it's not super difficult to adapt to whatever pipeline you need after you understand where your content comes from.

marosgrego 6y ago

Blades currently doesn't break the sitemap, but it can easily be implemented. I see that 50k pages is a maximum for one sitemap file. Do you think it should break into separate pages only after reaching 50k, or sooner?

chrsstrm 6y ago
IMO the easiest implementation would always break into an index and sitemap(s) as needed. 49,999 URLs? 1 index and 1 sitemap. 50K+ URLs? 1 index and sitemaps as needed. This gives you the most consistent output and users who cross the 50K threshold over time will not need to update their sitemap in search tools to be compliant since pointing to the index will already deal with that.

DrBazza 6y ago

Just run my company's internal Hugo build on commodity hardware (4 cores), and I get about 1000 pages per second, i.e. a 3966 page site took "4841ms" - so 20,000 pages assuming linear scaling, would still only 20-30 seconds. Not minutes. Perhaps our theme isn't complex enough?

chrsstrm 6y ago
Interesting. Are you using Related [0] content? I always suspected the long build was due to related content but the build profiler isn't specific enough to point that out. [0] https://gohugo.io/content-management/related/ https://gohugo.io/content-management/related/

chopraaa 6y ago

I've faced this issue as well. I run a Python script after the build has completed that reads the entire sitemap and chunks it into parts of 50k. +1 for mentioning incremental builds as well. These two features are often overlooked and people just ask "why do you even have so many pages?" and it ends there because nobody wants to entertain the power user. A third disadvantage I've seen is search. I use fuse to generate a JSON for client side search but it takes a lot of memory to render and is a cause for concern.