8 ms·
Please don't share our links on Mastodon
- h2odragon 3y agoMastodon can stomp their webstite through cloudflare? Is this not a massive failure on clouddflare's part? I have a low traffic news site; I wonder If I should share this link or would they prefer not to be troubled by the traffic.
- 1231232131231 3y agoThey can enable a setting that forces users to complete a captcha before accessing the website. That should prevent all the mastodon requests from getting through.
- chrisandchris 3y agoOh, yes, please! Make the web even shittier than it already is!
- bastawhiz 3y agoIf you tell cloudflare not to cache anything you can hardly be surprised that cloudflare doesn't save you any traffic
- iainmerrick 3y agoCloudflare gives you the tools but you have to use them properly. If you don't allow caching, a CDN can't help you.
- paxys 3y agoThe funny thing is that Cloudflare works perfectly well with default out-of-the-box settings. You have to go in and actively sabotage your own site to make it as broken as this one is right now.
- fanf2 3y agoCloudflare does not cache HTML by default https://developers.cloudflare.com/cache/concepts/default-cache-behavior/ https://developers.cloudflare.com/cache/concepts/default-cac...
- kevincox 3y agoIt really doesn't work well by default. It won't cache HTML and it will serve up CAPTCHAs to subresources like JS and IMG that can't be completed by users as well as RSS feeds so that people can't subscribe. The defaults are really quite broken.
- kemotep 3y agoSo how does being linked to on Twitter, Reddit, and Facebook not do the same? Those sites combined clearly would generate an order of magnitude more traffic. Maybe several orders of magnitude more. There has to be more than 15k itsfoss readers between these sites. A link preview with an image is over 100 MB? That sounds insane. And if they mean the total traffic in 5 minutes was 100MB that cannot possibly be bringing Cloudflare to its knees. That is an indictment of Cloudflare’s CDN then!
- ehutch79 3y agoEach of those caches the preview/etc. so they hit it once in a while, and that works for all their users. Each mastadon server is independent and can't share that preview cache. All the issues sound more like a them problem than mastadon or cloudflare. Something's wrong with their setup. Hard to tell as now HN has brought them to their knees
- OJFord 3y ago> Something's wrong with their setup. Hard to tell as now HN has brought them to their knees Agreed, and hardly surprising is it that if the site can't cope with link preview traffic that non-trivial page views would be trouble too!
- kemotep 3y agoCloudflare being unable to handle 100MB of traffic over 5 minutes sounds like itsfoss failed to set something up properly. The link preview is a fraction of the total page that would have to be loaded for each individual user no matter where they came from. ActivityPub itself could more intelligently cache or display the content but something doesn’t add up if Cloudflare can’t handle that kind of traffic let alone Hacker News traffic.
- 1231232131231 3y agoI believe cloudflare is just letting their requests go through and their webserver can't handle it?
- skilled 3y agoLemmy instances do the same. I believe it is over 1,000 separate GET requests in ~2 seconds once you hit publish.
- 2024throwaway 3y agoThe horror.
- paxys 3y agoIronic that their "please don't share our links" post shared on HN also caused their website to crash. It's a 100% static blog post. Use a damn cache, or CDN, or a hundred other ways of handling ~unlimited traffic for free. We are talking about a few thousand hits and tens of megabytes of total data transfer. It isn't 1998. This scale should not be a problem for anyone.
- oefrha 3y agoArchive link here: https://archive.is/ikgco https://archive.is/ikgco I’m entirely baffled why someone savvy enough to produce monitoring graphs and claim to already use Cloudflare can be brought to knees by HN (maybe Reddit too or something?) traffic on a frigging blog. I believe you need to actively sabotage your Cloudflare settings to achieve this.
- tazjin 3y agoCloudflare probably respects your cache headers (I haven't used it, but would expect so). This website disables caching in its headers, so Cloudflare can't actually do much.
- BrandoElFollito 3y agoYes, they should disable caching on index.html only and have the other files update their names on changes
- acdha 3y agoThey should allow everything to be cached and use the Cloudflare API to invalidate the cache when they actually change something. That means that even huge spikes will have no impact on their backend and they’ll still be able to see changes within a second or two of publishing them. This complaint is simply an attempt to shift the blame for their decision to ignore multiple decades of prior art. Even in the 90s we used techniques like caching to avoid this problem, and their page-load times now somehow manage to be worse.
- muglug 3y agoI noticed this last year when an article I wrote was reposted by the creator of Mastodon. Visits from individual instance crawlers vastly outnumbered visits from people reading the content. I had prepared the content for potential virality (hand-written HTML & well-optimised images) but it was still an unwelcome surprise when I checked the server logs and saw all that noise.
- soneca 3y agoI couldn’t understand the scale of this “fediverse effect”. There is this quote: > ” making for a traffic amplification of 36704:1” Does it mean that posting a link on Mastodon generates 36704 requests to that URL?
- skilled 3y agoSeveral thousand, yea. That many, unlikely in a single batch. Favouriting and boosting are actions that amplify how far a Mastodon post is going to spread on a “connected instances” spectrum. But for popular accounts (15k followers as in this case) will definitely spread the link to thousands of instances instantly and cause thousands of individual GET requests. It’s fun to tail access.log and see it happen in real time.
- SideburnsOfDoom 3y ago> Does it mean that posting a link on Mastodon generates 36704 requests to that URL? No, it's a ratio of bytes for traffic . They're counting size of 1 request to post to Mastodon "a single roughly ~3KB POST", vs the total size of content served from GETing the url in that post. IDK how valid this metric is, but that's what they're saying.
- kevincox 3y agoI don't see how the request size to mastodon is at all relevant to the resulting traffic.
- SideburnsOfDoom 3y agoIt's the language of "traffic amplification" as a network attack (1) as used in a DDOS. it's perhaps more relevant when comparing the total bytes in a number of IP Packets of various sizes. I'm not saying that this is the optimal framing, but that's what they were going for: talking about this, correctly or not, as a de facto DDOS. 1) https://www.microsoft.com/en-us/security/blog/2022/05/23/anatomy-of-ddos-amplification-attacks/ https://www.microsoft.com/en-us/security/blog/2022/05/23/ana...
- amiga386 3y agoThere's a straightforward answer. Include all metadata and preview images in the initial post (as data: urls) Don't make every Mastodon instance have to fetch the linked page and all its assets to generate its own previews EDIT: as linked in TFA, it has been nearly 7 years and they're still arguing about it: * https://github.com/mastodon/mastodon/issues/4486 https://github.com/mastodon/mastodon/issues/4486 * https://github.com/mastodon/mastodon/issues/23662 https://github.com/mastodon/mastodon/issues/23662 Dear nincompoops: if you trust the original poster and original server to send you text and images in a toot, and federated instances to pass that around without modification... then you can trust them equally to send you a URL and an image preview. It's arrogance and idiocy that lead you to believe you can trust their images but can't trust their web preview images and you have to verify that yourself by having the Fediverse DDoS the host. This problem will only get worse as the Fediverse expands. Fix it now, don't ignore it because it makes a problem for someone else
- jefftk 3y ago> if you trust the original poster and original server to send you text and images in a toot, and federated instances to pass that around without modification... then you can trust them equally to send you a URL and an image preview I don't think it's that straightforward: normally if I follow @amiga386@mastodon.example you and I both need to trust example.com to accurately report what you say. But if you put a link in to news.example and mastodon.example scrapes and includes a preview I now need to trust mastodon.example to accurately report what news.example is saying. And I might well not! I got into this more with mockups here: https://www.jefftk.com/p/mastodons-dubious-crawler-exemption https://www.jefftk.com/p/mastodons-dubious-crawler-exemption
- amiga386 3y agoI see what you mean, but I also kinda disagree. While people have come to rely on centralised services' link preview generators as some kind of trustworthy source, this shouldn't be taken for granted, and Mastodon users definitely should not give that level of trust. Even on centralised platforms, I've seen endless "screenshots" on Twitter of other web pages and other tweets, aping the form of screenshot-quotations, but actually doctored. The original page or tweet never said what the screenshot claims they did. And I've also seen just the text of tweets claiming that some person said X, when that person did not say X. There can be millions of people affected by this misrepresentation, because they use Twitter-only sources for their information, and don't verify what they see. Then there's the same problem one level up, the screenshot might be a valid screenshot of absolute poppycock published by a partisan news source. This is why I phrased my original statement the way I did... if you trust the original poster and original server. It's quite possible you shouldn't. Trust should be anchored to the individual toot and its poster, and you should distrust them for misrepresenting link previews in the same way you should distrust them for any other misrepresentations they make. Ultimately, you should not trust website preview links any more than you trust the person posting them. You should click through (if you even trust opening the link) to see if the preview matches, and you should use existing tools (blocking, defederation) for posters or servers who abuse your trust. (Edit: and let's also add in the trust that the website represents itself equally to both you and the link preview generator code... a large number of sites become much more responsive and ad-free automatically if you claim to be GoogleBot.. and let's not even mention sites doing A/B testing for virality under the same URL) Why this is a difficult problem to resolve is that the Mastodon wants one thing - "trustworthy" per-server link previews, meaning tens of thousands of servers come thundering around the same time, and this will be millions of servers in future if they don't fix this - at the expense of others, the link targets. Meanwhile, those affected by the Mastodon community's selfish behaviour want them to clean up their act, at the cost of something the Mastodon community thinks is precious and fears losing (trusting link previews). I think solutions need to come from behavioural change, which is to say giving link previews no more trust than the other text or images on a toot, and from that direction it would be much more palatable to have the poster supply the preview, because viewers wouldn't be giving it undue trust.
- ehutch79 3y ago15k people, hitting a site that isn't actively changing content, shouldn't be bringing any webserver to it's knees. - They've set the generated page to be uncachable with max-age=0 (https://developers.cloudflare.com/cache/concepts/default-cache-behavior/ https://developers.cloudflare.com/cache/concepts/default-cac...) - nginx is clearly not caching dynamic resources (currently 22s (not ms) to respond) - lots of 3rd party assets loading. Why are you loading stripe before I'm giving you money? - why is there a random 'grey.webp' loading from this michelmeyer.whatever domain? This isn't mastadon or cloudflare, it's a skill issue.
- wlonkly 3y ago> - why is there a random 'grey.webp' loading from this michelmeyer.whatever domain? This got me wondering, and the reason is that they embed a "card" for a link to a similar blog post on michaelmeyer.com (and grey.webp is the imagine in the card). There's a little irony there, I think.
- mongonews 3y ago[flagged]
- deleted 3y ago[deleted]
- John23832 3y ago[flagged]
- rsynnott 3y agoIt shouldn't even be 15k people, unless every person is on a different instance, which is wildly unlikely.
- dredmorbius 3y agoIt's not the 15k followers of ItsFOSS who are generating traffic, it is the servers of those 15k accounts own followers who see shares, boosts, or re-shares of content. Given that Mastodon, and the larger Fediverse of which it itself is only a part (though a large share of total activity), are, well, federated (as TFA notes pointedly), sharing links requires each instance to make a request. I'm not particularly clear on Fediverse and Mastodon internals, but my understanding is that an image preview request is only generated once on a per server basis, regardless of how many local members see that link. But, despite some technical work at this, I don't believe there's yet a widely-implemented way of caching and forwarding such previews (which raises its own issues for authenticity and possible hostile manipulation) amongst instances. (There's a long history of caching proxy systems, with Squid being among the best known and most venerable.) Otherwise, I understand that preview requests are now staggered and triggered on demand (when toots are viewed rather than when created) which should mitigate some, but not all, of the issue. The phenomenon is known as a Mastodon Stampede, analagous to what was once called the Slashdot Effect. There's at least one open github issue, #4486, dating to 2017: <https://github.com/mastodon/mastodon/issues/4486 https://github.com/mastodon/mastodon/issues/4486> Some discussion from 2022: <https://www.netscout.com/blog/mastodon-stampede https://www.netscout.com/blog/mastodon-stampede> And jwz, whose love of HN knows no bounds, discusses it as well. Raw text link for the usual reasons, copy & paste to view without his usual love image (see: <https://news.ycombinator.com/item?id=13342590 https://news.ycombinator.com/item?id=13342590>). https://www.jwz.org/blog/2022/11/mastodon-stampede/
- lenerdenator 3y agoSo, you're telling me that I shouldn't go to the main Mastodon instance, copy that link into a toot, and @ the main developer account?
- rvz 3y agoThis DDoS'ing problem is due to how the fediverse works. The problem is that federation in general in social networks is extremely inefficient with tons of flaws and this issue one of them.
- jefftk 3y agoThere are two problems here: 1. The "It's Foss" site is designed somewhat carelessly, where many things that could be static aren't, and so it goes under with even a bit of load. 2. Mastodon link preview is badly designed and not spec-compliant. Because the link previews are not triggered by a user request they should respect robots.txt (https://github.com/mastodon/mastodon/issues/21738 https://github.com/mastodon/mastodon/issues/21738), or they should start being triggered by a user request (https://github.com/mastodon/mastodon/issues/23662 https://github.com/mastodon/mastodon/issues/23662).
- johnea 3y agoCould mastedon federate the link preview as well as the link? Some OP advice to "get a damn CDN" seems to be a responce rather than a correction of the issue. Even poorly designed websites shouldn't face DDoS scale access by a social network sharing a link. The social network should mitigate it's massive parallelization.
- ASalazarMX 3y ago> Could mastedon federate the link preview as well as the link? Off-topic, but I love how "mastedon" sounds.
- NotEvil 3y agoThe idea has been discussed. The main problem is that Mastodon is a zero-trust environment, and we can't trust the other server to send us the correct preview.
- abeyer 3y agoDon't trust the server, trust the poster/client... that's really where the preview should be coming from anyway. And if you _don't_ trust the poster, you shouldn't be trusting anything they link you to anyway. It's trivial to lie to these preview generators with a couple of meta tags if the destination isn't trustworthy.
- willcipriano 3y ago
- RobotToaster 3y agoIt sounds like the author's website/cache/whatever isn't configured properly, but it does sound like a terrible design for the preview data to not be included in the federated data.
- amarant 3y agoSeems like what they actually want to say is: please help us, we have no idea how to host a moderately popular website. They must've spelled it wrong...
- kibwen 3y agoWhile I agree with other commenters that a server should be able to gracefully handle the relatively small amount of traffic being generated here, I'm sympathetic to the notion that in this specific instance the problem is exacerbated by all the servers making their requests at almost exactly the same instant, i.e. the Thundering Herd Problem https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem . If Mastodon wanted to address this, servers could add a small amount of random delay before fetching a link preview.
- Benedicht 3y agoOr even share the link preview?
- everforward 3y agoHow would that work without allowing nefarious implementations to federate incorrect previews? Is that even really an issue? It doesn’t seem to be one technically, but I’m not well-versed on the social engineering implications.
- esperent 3y agoMaybe by comparing a filehash?
- everforward 3y agoI'm not sure where it gets the known good hash from without making an HTTP call to the originating URL, though. The trustworthiness of the file hash is the same as the file itself if they have the same origin. I would also consider to what degree such a system ends up looking like a half-baked, distributed Cloudflare anyways. Like yes, I'm sure we could build some kind of incredibly complicated, reputation-based, distributed link preview caching system. Or the host could just fix their Cloudflare (or accept dying under traffic load). My generalized experience has been that untrusted, secure, distributed systems are incredibly difficult to build, and that it's probably not worth doing for something as trivial as URL previews. Just let the request die and swap the preview for a message like "This site was down as of $lastTimeItCheckedForAPreview". Maybe change the message so it shows the URL but doesn't make it an <a> element so people can't just click on it to discourage sending them further traffic. Or worst case, they could try a fallback to reliable, centralized sources for that info. See if Google Search has a cached copy, or archive.org, or whatever else. It's not decentralized, but I also think it should be fine to use non-decentralized features as a fallback for optional features. I've got more confidence that archive.org hasn't tinkered with their version than some random Mastodon instance.
- martinclayton 3y agoPreviews only use the HMTL head - to get OG metadata - and usually an image. Never thought about this before. One of my sites is (a single-page application) 45k HTML and 40k image. Hence about 50% of what is served for previews is wasted. It would be nice if there was a way to recognise an "only html head needed" type of request. Don't think there is?
- jakebasile 3y agoThat'd be the HTTP HEAD verb. https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HE...
- abeyer 3y agoNope, unless things have changed since I last dealt with it, Open Graph metadata is not considered "http-equiv" so does _not_ get served as an http header, only as a meta tag in the html header... so the HEAD verb won't get them for you.
- jakebasile 3y agoAh yeah, that's just the HTTP headers, not the HTML head. Whoops!
- jononor 3y agoSeems like Mastodon is perfect for some basic load testing of a website/blog :)
- ethanholt1 3y agoThat seems like a problem of It’s Foss’ website, not Mastodon. If you’re hosting a presumably static website, a news website no less, it should be able to handle a spike of viewer influx if you make a viral article. Seems like they’re pinning the blame on Mastodon rather than fixing their site.
- aaron695 3y ago[dead]
- captn3m0 3y agoWasn’t there a TLS extension in draft for verifiable archiving (like DKIM for HTTPS responses). Can’t seem to find it now, but that could help with supporting an authenticated link-preview that doesn’t amplify on reach other servers.
- zagrebian 3y agoSounds like web performance optimization is more important than ever.
- pentagrama 3y agoNot the best solution for them, but what about remove the metadata from the pages <head> that let Mastodon and other social media platforms request for the preview images? It will hurt discoverability and clicks maybe, but is better than the site having downtimes.
- nicolas_17 3y agoHow would Mastodon know that the HTML page lacks any metadata in <head> without actually requesting the HTML page?