4 ms·

So how does being linked to on Twitter, Reddit, and Facebook not do the same? Those sites combined clearly would generate an order of magnitude more traffic. Ma

So how does being linked to on Twitter, Reddit, and Facebook not do the same? Those sites combined clearly would generate an order of magnitude more traffic. Maybe several orders of magnitude more. There has to be more than 15k itsfoss readers between these sites. A link preview with an image is over 100 MB? That sounds insane. And if they mean the total traffic in 5 minutes was 100MB that cannot possibly be bringing Cloudflare to its knees. That is an indictment of Cloudflare’s CDN then!

ehutch79 2y ago
Each of those caches the preview/etc. so they hit it once in a while, and that works for all their users. Each mastadon server is independent and can't share that preview cache. All the issues sound more like a them problem than mastadon or cloudflare. Something's wrong with their setup. Hard to tell as now HN has brought them to their knees
- OJFord 2y ago
  > Something's wrong with their setup. Hard to tell as now HN has brought them to their knees Agreed, and hardly surprising is it that if the site can't cope with link preview traffic that non-trivial page views would be trouble too!
- kemotep 2y ago
  Cloudflare being unable to handle 100MB of traffic over 5 minutes sounds like itsfoss failed to set something up properly. The link preview is a fraction of the total page that would have to be loaded for each individual user no matter where they came from. ActivityPub itself could more intelligently cache or display the content but something doesn’t add up if Cloudflare can’t handle that kind of traffic let alone Hacker News traffic.
  - 1231232131231 2y ago
    I believe cloudflare is just letting their requests go through and their webserver can't handle it?
    - vbezhenar 2y ago
      What's the point of using Cloudflare without caching?
      - dabber 2y ago
        Cloudflare has some other features like a Web Application Firewall (WAF) and bot protection/filtering (which seems like it would solve their problems too?) From the article: > Presently, we use Cloudflare as our CDN or WAF, as it is a widely adopted solution. To me, it sounds like the author isn't really familiar with the difference between a CDN and WAF is; or familiar with Cloudflare beyond it being a popular thing they should probably have for that matter.
    - ehutch79 2y ago
      vbezhenar: They've misconfigured something and the page itself is marked max-age=0 which disables cloudflare caching. I'm also betting nginx isn't caching either.
  - ehutch79 2y ago
    They've set their headers to prevent caching by cloudflare. max-age=0 The images arn't the problem, it's their dynamicly generated page, even if the content is pretty static.
OJFord 2y ago
Well all Twitter links (annoyingly) are made to go through t.co, so either Twitter is generating and serving the link preview directly or it's its shortening service that's being hit, probably for a cached response, not the upstream. But I agree this does seem strange, like it shouldn't be unmanageable load at all.
sumtechguy 2y ago
It probably does. The issue is proxy. This is the downside to TLS and letsencrypt. Because man in the middle is a thing. Being a fediverse application the proper idea would be for the clients to P2P the data amongst themselves and lighten the load on the originator. However, in this case a link probably should not do that. So therefore it needs to go get the data itself. It is a DDoS were basically one link can magnify to thousands of other clients. Then until those clients are done they will stomp the site. Perhaps there could be a side channel ask from the client 'hey site are you cool with me showing the user a preview of this'? Or 'hey site is it ok if I share a preview of your page'? That would not totally stop the rush of requests. But would at least keep it from a render of the page for a preview. This method is half baked and something I just came up with and seems wrong though. But auto following links is not exactly in spec either.
- BlueTemplar 2y ago
  How much MitM is even an issue with link previews, rather than actual links ?
  - sumtechguy 2y ago
    If I think like an attacker I would say a preview would be an even juicier target. I can basically run something on thousands of peoples computers with zero input from them.
    - ehutch79 2y ago
      Their site isn't going down from an attack. It's running out of resources for normal operations. It's not even a large amount of traffic. not really.
    - BlueTemplar 2y ago
      Why would it be about running something, rather than just a paragraph of text and a picture ?
      - 2y ago
thinkingtoilet 2y ago
Perhaps because they're not federated they can cache it.
jwalton 2y ago
> A link preview with an image is over 100 MB? I think what’s being implied here is that, when you share a link to Facebook, Facebook will access the page to generate a link preview, so will download a tiny bit of HTML and an image. But when you share a link on mastodon, that link immediately gets propagated to many other mastodon servers, which then propagate it to others, so suddenly many thousands of mastodon instances are simultaneously downloading a little bit of HTML and an image, and the cumulative effect of that in this instance was 100MB over a minute or two. It does seem like a typical static website ought to not have a problem serving that, especially if it’s behind Cloudflare. It seems odd that a single EC2 instance would have a hard time serving that. But given more than one person is complaining about, it also seems like each mastodon instance could very easily delay propagation of the story by a few minutes to soften the blow here.
- iainmerrick 2y ago
  each mastodon instance could very easily delay propagation of the story by a few minutes to soften the blow I liked that idea at first glance, but thinking about it, CDN performance would actually be better with a single huge burst than if they were smeared out (assuming a very short max-age so the site can be updated rapidly).
  - kevincox 2y ago
    It's actually more complicated. With HTTP you don't know requests can be coalesced until you receive the response headers with the Cache-Control and Vary. So if your website takes a few seconds to respond most CDNs will send every single request in that period through. In theory a CDN could optimistically coalesce requests then re-send them when the headers of the first one return. But this is very complex and rarely done in practice. This can also occur on any time the cache gets stale and needs to be refetched.
    - btasker 2y ago
      > most CDNs will send every single request in that period through. I don't think this is true. It certainly isn't for any CDN that I've worked for or on. Cloudflare don't do this either - they use a cache lock - the first request basically acts as a blocker for all the others, leaving the other requests waiting for the response (if it's cacheable they serve that response, if not then they proceed to origin). It's normally configurable, but most sane CDNs do have it enabled by default, precisely because big bursts tend to be sharp in nature and a cache miss can be origin breaking at that point. Just for completeness's sake, Nginx's HTTP proxy module can do it too (the setting's proxy_cache_lock) though it is off by default there.

ehutch79 2y ago

Each of those caches the preview/etc. so they hit it once in a while, and that works for all their users. Each mastadon server is independent and can't share that preview cache. All the issues sound more like a them problem than mastadon or cloudflare. Something's wrong with their setup. Hard to tell as now HN has brought them to their knees

OJFord 2y ago
> Something's wrong with their setup. Hard to tell as now HN has brought them to their knees Agreed, and hardly surprising is it that if the site can't cope with link preview traffic that non-trivial page views would be trouble too!
kemotep 2y ago
Cloudflare being unable to handle 100MB of traffic over 5 minutes sounds like itsfoss failed to set something up properly. The link preview is a fraction of the total page that would have to be loaded for each individual user no matter where they came from. ActivityPub itself could more intelligently cache or display the content but something doesn’t add up if Cloudflare can’t handle that kind of traffic let alone Hacker News traffic.
- 1231232131231 2y ago
  I believe cloudflare is just letting their requests go through and their webserver can't handle it?
  - vbezhenar 2y ago
    What's the point of using Cloudflare without caching?
    - dabber 2y ago
      Cloudflare has some other features like a Web Application Firewall (WAF) and bot protection/filtering (which seems like it would solve their problems too?) From the article: > Presently, we use Cloudflare as our CDN or WAF, as it is a widely adopted solution. To me, it sounds like the author isn't really familiar with the difference between a CDN and WAF is; or familiar with Cloudflare beyond it being a popular thing they should probably have for that matter.
  - ehutch79 2y ago
    vbezhenar: They've misconfigured something and the page itself is marked max-age=0 which disables cloudflare caching. I'm also betting nginx isn't caching either.
- ehutch79 2y ago
  They've set their headers to prevent caching by cloudflare. max-age=0 The images arn't the problem, it's their dynamicly generated page, even if the content is pretty static.

OJFord 2y ago

Well all Twitter links (annoyingly) are made to go through t.co, so either Twitter is generating and serving the link preview directly or it's its shortening service that's being hit, probably for a cached response, not the upstream. But I agree this does seem strange, like it shouldn't be unmanageable load at all.

sumtechguy 2y ago

It probably does. The issue is proxy. This is the downside to TLS and letsencrypt. Because man in the middle is a thing. Being a fediverse application the proper idea would be for the clients to P2P the data amongst themselves and lighten the load on the originator. However, in this case a link probably should not do that. So therefore it needs to go get the data itself. It is a DDoS were basically one link can magnify to thousands of other clients. Then until those clients are done they will stomp the site. Perhaps there could be a side channel ask from the client 'hey site are you cool with me showing the user a preview of this'? Or 'hey site is it ok if I share a preview of your page'? That would not totally stop the rush of requests. But would at least keep it from a render of the page for a preview. This method is half baked and something I just came up with and seems wrong though. But auto following links is not exactly in spec either.

BlueTemplar 2y ago
How much MitM is even an issue with link previews, rather than actual links ?
- sumtechguy 2y ago
  If I think like an attacker I would say a preview would be an even juicier target. I can basically run something on thousands of peoples computers with zero input from them.
  - ehutch79 2y ago
    Their site isn't going down from an attack. It's running out of resources for normal operations. It's not even a large amount of traffic. not really.
  - BlueTemplar 2y ago
    Why would it be about running something, rather than just a paragraph of text and a picture ?
    - 2y ago

thinkingtoilet 2y ago

Perhaps because they're not federated they can cache it.

jwalton 2y ago

> A link preview with an image is over 100 MB? I think what’s being implied here is that, when you share a link to Facebook, Facebook will access the page to generate a link preview, so will download a tiny bit of HTML and an image. But when you share a link on mastodon, that link immediately gets propagated to many other mastodon servers, which then propagate it to others, so suddenly many thousands of mastodon instances are simultaneously downloading a little bit of HTML and an image, and the cumulative effect of that in this instance was 100MB over a minute or two. It does seem like a typical static website ought to not have a problem serving that, especially if it’s behind Cloudflare. It seems odd that a single EC2 instance would have a hard time serving that. But given more than one person is complaining about, it also seems like each mastodon instance could very easily delay propagation of the story by a few minutes to soften the blow here.

iainmerrick 2y ago
each mastodon instance could very easily delay propagation of the story by a few minutes to soften the blow I liked that idea at first glance, but thinking about it, CDN performance would actually be better with a single huge burst than if they were smeared out (assuming a very short max-age so the site can be updated rapidly).
- kevincox 2y ago
  It's actually more complicated. With HTTP you don't know requests can be coalesced until you receive the response headers with the Cache-Control and Vary. So if your website takes a few seconds to respond most CDNs will send every single request in that period through. In theory a CDN could optimistically coalesce requests then re-send them when the headers of the first one return. But this is very complex and rarely done in practice. This can also occur on any time the cache gets stale and needs to be refetched.
  - btasker 2y ago
    > most CDNs will send every single request in that period through. I don't think this is true. It certainly isn't for any CDN that I've worked for or on. Cloudflare don't do this either - they use a cache lock - the first request basically acts as a blocker for all the others, leaving the other requests waiting for the response (if it's cacheable they serve that response, if not then they proceed to origin). It's normally configurable, but most sane CDNs do have it enabled by default, precisely because big bursts tend to be sharp in nature and a cache miss can be origin breaking at that point. Just for completeness's sake, Nginx's HTTP proxy module can do it too (the setting's proxy_cache_lock) though it is off by default there.