3 ms·

15k people, hitting a site that isn't actively changing content, shouldn't be bringing any webserver to it's knees. - They've set the generated page to be unca

15k people, hitting a site that isn't actively changing content, shouldn't be bringing any webserver to it's knees. - They've set the generated page to be uncachable with max-age=0 (https://developers.cloudflare.com/cache/concepts/default-cache-behavior/ https://developers.cloudflare.com/cache/concepts/default-cac...) - nginx is clearly not caching dynamic resources (currently 22s (not ms) to respond) - lots of 3rd party assets loading. Why are you loading stripe before I'm giving you money? - why is there a random 'grey.webp' loading from this michelmeyer.whatever domain? This isn't mastadon or cloudflare, it's a skill issue.

wlonkly 2y ago
> - why is there a random 'grey.webp' loading from this michelmeyer.whatever domain? This got me wondering, and the reason is that they embed a "card" for a link to a similar blog post on michaelmeyer.com (and grey.webp is the imagine in the card). There's a little irony there, I think.
mongonews 2y ago
[flagged]
- deleted 2y ago
  [deleted]
- John23832 2y ago
  [flagged]
rsynnott 2y ago
It shouldn't even be 15k people, unless every person is on a different instance, which is wildly unlikely.
dredmorbius 2y ago
It's not the 15k followers of ItsFOSS who are generating traffic, it is the servers of those 15k accounts own followers who see shares, boosts, or re-shares of content. Given that Mastodon, and the larger Fediverse of which it itself is only a part (though a large share of total activity), are, well, federated (as TFA notes pointedly), sharing links requires each instance to make a request. I'm not particularly clear on Fediverse and Mastodon internals, but my understanding is that an image preview request is only generated once on a per server basis, regardless of how many local members see that link. But, despite some technical work at this, I don't believe there's yet a widely-implemented way of caching and forwarding such previews (which raises its own issues for authenticity and possible hostile manipulation) amongst instances. (There's a long history of caching proxy systems, with Squid being among the best known and most venerable.) Otherwise, I understand that preview requests are now staggered and triggered on demand (when toots are viewed rather than when created) which should mitigate some, but not all, of the issue. The phenomenon is known as a Mastodon Stampede, analagous to what was once called the Slashdot Effect. There's at least one open github issue, #4486, dating to 2017: <https://github.com/mastodon/mastodon/issues/4486 https://github.com/mastodon/mastodon/issues/4486> Some discussion from 2022: <https://www.netscout.com/blog/mastodon-stampede https://www.netscout.com/blog/mastodon-stampede> And jwz, whose love of HN knows no bounds, discusses it as well. Raw text link for the usual reasons, copy & paste to view without his usual love image (see: <https://news.ycombinator.com/item?id=13342590 https://news.ycombinator.com/item?id=13342590>). https://www.jwz.org/blog/2022/11/mastodon-stampede/
yupyup54133 2y ago
Someone asked them in their comments section: > Have you considered switching to a Static Site Generator? Write your posts in Markdown, push the change to your Git repo, have GitHub/GitLab automatically republish your website upon push, and end up with a very cacheable website that can be served from any simple Nginx/Apache. In theory this scales a lot better than a CMS-driven website. Admins Response: > That would be too much of a hassle. A proper CMS allows us to focus on writing.
- em-bee 2y ago
  and i agree with them. static site generators are nice for people who like that kind of workflow, but when you have a team with different personal preferences you need something that is easy to access and doesn't require much learning to get used to. what would be ideal is a CMS that can separate content editing and serving it. iaw a kind of static site generator that is built into a CMS and can push updates to the static site as they happen.
  - mingus88 2y ago
    It’s been a long time since I’ve been in the CMS space but I thought Wordpress had dozens of plugins for caching even 10 years ago. I mean, even just hopping over to host your site in Wordpress.com was a viable option if you were in that middle ground between personal blog and having a dedicated server admin to handle your traffic Hard to believe that you’d be in the business of serving content in 2024 and have to deal with the slashdot effect from 1999 for your blog of articles and images.
    - em-bee 2y ago
      true, but i suspect that wanting to self-host the content is a factor (at least it would be for me). and as others have mentioned, there seem to be issues with the cloudflare setup that should have helped here too, so even with self-hosting, it should be possible to handle this.
    - joshspankit 2y ago
      WordPress does have bunches of caching plugins. Some VPS hosts (shout out SiteGround) also have multiple layers of server-level caching that they will apply automatically. You can take most WordPress websites from multi-second load times to 750ms or less (in fact as a regular exercise I set up fresh WordPress installs on dirt-cheap VPS hosts and see how low I can get them while still having a good site. 250ms to display is not uncommon even without CDNs)
Brian_K_White 2y ago
Ahem, mr skill issue, some information, 15k followers != 15k hits.
jpifvuk0 2y ago
Just as a reminder: the site sees plenty of traffic from various other platforms. It's quite popular, I'm sure, Mastodon is one of the least of their concerns for traffic. If they can handle traffic load on many other various viral/popular traffic, the server is capable enough (even without proper caching). Not every site configuration is perfect, and blaming the site's configuration, and ignoring Mastodon's inherent issue is borderline not practical.
- ehutch79 2y ago
  They went straight for blaming mastodon, without seeming to try and fix or mitigate the issue on their end. Hard to be sympathetic. This isn't a new issue, nor is it unique to mastodon. Reducing server load for sites like this is a very common exercise for many reasons.
  - amiga386 2y ago
    That sounds like victim blaming. There have been other cases, such as where a mobile app developer has hardcoded an image from someone else's website into their app, then millions of users request it every time they open the app. Or where middlebox manufacturers have hardcoded an IP address to use for NTP. Sure, having efficient and well-cached infrastructure setup is good, but there's only so much you can do to "reduce server load" where other people in control of widely-deployed software have made choices that causes millions of devices around the world to hammer _you_ specifically. The people who made those choices don't give a shit, it's not _their_ infrastructure they fucked over. That's why you need to shame them into fixing their botnet and/or block their botnet with extreme prejudice. Mastodon's link preview service is a botnet, Mastodon knows it, and they refuse to fix it.
SunlitCat 2y ago
Maybe it's because it's deemed "too difficult" to change it? Years ago, I was looking into some very popular c++ library and wanted to download the archive (a tar.gz or .zip, can't remember). At that time, they hosted it on sourceforge for download. I was looking for a checksum (md5, sha1 or sha256) and found a mail in their mailing list archive where someone asked for providing said checksums on their page. The answer? It's too complicated to put creating checksums into the process and sourceforge is safe enough. (paraphrased, but that was the gist of the answer) That said, since quite some years they provide checksums for their source archives, but they kinda lost me with that answers years ago.

wlonkly 2y ago

> - why is there a random 'grey.webp' loading from this michelmeyer.whatever domain? This got me wondering, and the reason is that they embed a "card" for a link to a similar blog post on michaelmeyer.com (and grey.webp is the imagine in the card). There's a little irony there, I think.

mongonews 2y ago

[flagged]

deleted 2y ago
[deleted]
John23832 2y ago
[flagged]

rsynnott 2y ago

It shouldn't even be 15k people, unless every person is on a different instance, which is wildly unlikely.

dredmorbius 2y ago

It's not the 15k followers of ItsFOSS who are generating traffic, it is the servers of those 15k accounts own followers who see shares, boosts, or re-shares of content. Given that Mastodon, and the larger Fediverse of which it itself is only a part (though a large share of total activity), are, well, federated (as TFA notes pointedly), sharing links requires each instance to make a request. I'm not particularly clear on Fediverse and Mastodon internals, but my understanding is that an image preview request is only generated once on a per server basis, regardless of how many local members see that link. But, despite some technical work at this, I don't believe there's yet a widely-implemented way of caching and forwarding such previews (which raises its own issues for authenticity and possible hostile manipulation) amongst instances. (There's a long history of caching proxy systems, with Squid being among the best known and most venerable.) Otherwise, I understand that preview requests are now staggered and triggered on demand (when toots are viewed rather than when created) which should mitigate some, but not all, of the issue. The phenomenon is known as a Mastodon Stampede, analagous to what was once called the Slashdot Effect. There's at least one open github issue, #4486, dating to 2017: <https://github.com/mastodon/mastodon/issues/4486 https://github.com/mastodon/mastodon/issues/4486> Some discussion from 2022: <https://www.netscout.com/blog/mastodon-stampede https://www.netscout.com/blog/mastodon-stampede> And jwz, whose love of HN knows no bounds, discusses it as well. Raw text link for the usual reasons, copy & paste to view without his usual love image (see: <https://news.ycombinator.com/item?id=13342590 https://news.ycombinator.com/item?id=13342590>). https://www.jwz.org/blog/2022/11/mastodon-stampede/

yupyup54133 2y ago

Someone asked them in their comments section: > Have you considered switching to a Static Site Generator? Write your posts in Markdown, push the change to your Git repo, have GitHub/GitLab automatically republish your website upon push, and end up with a very cacheable website that can be served from any simple Nginx/Apache. In theory this scales a lot better than a CMS-driven website. Admins Response: > That would be too much of a hassle. A proper CMS allows us to focus on writing.

em-bee 2y ago
and i agree with them. static site generators are nice for people who like that kind of workflow, but when you have a team with different personal preferences you need something that is easy to access and doesn't require much learning to get used to. what would be ideal is a CMS that can separate content editing and serving it. iaw a kind of static site generator that is built into a CMS and can push updates to the static site as they happen.
- mingus88 2y ago
  It’s been a long time since I’ve been in the CMS space but I thought Wordpress had dozens of plugins for caching even 10 years ago. I mean, even just hopping over to host your site in Wordpress.com was a viable option if you were in that middle ground between personal blog and having a dedicated server admin to handle your traffic Hard to believe that you’d be in the business of serving content in 2024 and have to deal with the slashdot effect from 1999 for your blog of articles and images.
  - em-bee 2y ago
    true, but i suspect that wanting to self-host the content is a factor (at least it would be for me). and as others have mentioned, there seem to be issues with the cloudflare setup that should have helped here too, so even with self-hosting, it should be possible to handle this.
  - joshspankit 2y ago
    WordPress does have bunches of caching plugins. Some VPS hosts (shout out SiteGround) also have multiple layers of server-level caching that they will apply automatically. You can take most WordPress websites from multi-second load times to 750ms or less (in fact as a regular exercise I set up fresh WordPress installs on dirt-cheap VPS hosts and see how low I can get them while still having a good site. 250ms to display is not uncommon even without CDNs)

Brian_K_White 2y ago

Ahem, mr skill issue, some information, 15k followers != 15k hits.

jpifvuk0 2y ago

Just as a reminder: the site sees plenty of traffic from various other platforms. It's quite popular, I'm sure, Mastodon is one of the least of their concerns for traffic. If they can handle traffic load on many other various viral/popular traffic, the server is capable enough (even without proper caching). Not every site configuration is perfect, and blaming the site's configuration, and ignoring Mastodon's inherent issue is borderline not practical.

ehutch79 2y ago
They went straight for blaming mastodon, without seeming to try and fix or mitigate the issue on their end. Hard to be sympathetic. This isn't a new issue, nor is it unique to mastodon. Reducing server load for sites like this is a very common exercise for many reasons.
- amiga386 2y ago
  That sounds like victim blaming. There have been other cases, such as where a mobile app developer has hardcoded an image from someone else's website into their app, then millions of users request it every time they open the app. Or where middlebox manufacturers have hardcoded an IP address to use for NTP. Sure, having efficient and well-cached infrastructure setup is good, but there's only so much you can do to "reduce server load" where other people in control of widely-deployed software have made choices that causes millions of devices around the world to hammer _you_ specifically. The people who made those choices don't give a shit, it's not _their_ infrastructure they fucked over. That's why you need to shame them into fixing their botnet and/or block their botnet with extreme prejudice. Mastodon's link preview service is a botnet, Mastodon knows it, and they refuse to fix it.

SunlitCat 2y ago

Maybe it's because it's deemed "too difficult" to change it? Years ago, I was looking into some very popular c++ library and wanted to download the archive (a tar.gz or .zip, can't remember). At that time, they hosted it on sourceforge for download. I was looking for a checksum (md5, sha1 or sha256) and found a mail in their mailing list archive where someone asked for providing said checksums on their page. The answer? It's too complicated to put creating checksums into the process and sourceforge is safe enough. (paraphrased, but that was the gist of the answer) That said, since quite some years they provide checksums for their source archives, but they kinda lost me with that answers years ago.