12 ms·

I won free load testing

461 points by 0xedb 4y ago

langsoul-com 4y ago
I'm interested in what the fly.io would cost if you didn't work there. I got a fly app and can't proxy it through cloudflare cause that doesn't work.
- julianbuse 4y ago
  I have proxied a fly.io app through cloudflare. The last time I tried, it worked.
  - langsoul-com 4y ago
    Not sure if it's because I'm using a subdomain or not. But I created a cert, added the a and aaaa records to CF, then after it was verified, turned back on the CF proxy. However, my app stopped working until I turned off the proxy
Slaren 4y ago
[dead]
mleonhard 4y ago
I'm working on two projects that could let a stand-alone Rust web server weather a moderate DDoS attack like the author had: Beatrice [0] - A web server with built-in connection limits and thread limits. It's async but supports non-async request handlers. fair-rate-limiter [1] - In theory, one could use this to shed most of the load from DDoS attacking nodes. [0] https://crates.io/crates/beatrice https://crates.io/crates/beatrice [1] https://crates.io/crates/fair-rate-limiter https://crates.io/crates/fair-rate-limiter
2143 4y ago
> Minutes after I posted this article, the attack resumed. Same shit, different AS. Noob question: was that caused by the article getting posted to HN? Or was it really an attack?
mou_sesp_az 4y ago
This guy has some great energy. And humor: >> Because it doesn't return an AddrStream but instead a Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>... >> Gesundheit.
d4a 4y ago
> That lets me answer questions like "what RSS readers (that aren't browsers) is my audience using?" Very interesting... I also use NNW and FreshRSS
fasterthanlime 4y ago
Mirror in case the attack successfully picks up again: https://web.archive.org/web/20220502013024/https://fasterthanli.me/articles/i-won-free-load-testing# https://web.archive.org/web/20220502013024/https://fastertha... As a treat, this is a testament to a logic error I made in the caching code (inserted uncachable versions of pages into the cache for a little while). Enjoy!
- terhechte 4y ago
  I'm spending a lot of time in Rust but I always learn so much new when reading your articles. Particularly, how do you manage find all these interesting crates that you're using. Like arc-swap, or Moka, or color-eyre. Do you actively search for these things or do you have a habit of reading the what's new crates.io feed in the morning with a cup of coffee?
  - nicoburns 4y ago
    Not the OP, but the r/rust subreddit is my other lurking spot besides HN. (And I’m also currently working on a site to document some of these useful crates)
    - fasterthanlime 4y ago
      /r/rust is a great place for that, I also occasionally click through most of the https://lib.rs/ https://lib.rs/ categories, just to see what the names I don't recognize are about. My twitter timeline also has tons of neat little crates pop up every month, and I have to thank my colleagues at Netlify (back when they were doing Rust) and now at fly, for showing me some of those I showcase in the article.
      - no_wizard 4y ago
        Is Netlify not doing rust now? What’s the story there?
        4y ago
the_alchemist 4y ago
What a great post-mortem write-up! Thans for sharing
daenz 4y ago
Out of curiosity, I googled how much a DDoS attack goes for these days. Apparently they can cost as low as $10/hour. I don't know if the shady people will deliver, but that's what the internet says. So apparently it's pretty easy to DDoS anyone and make it difficult to trace back to you.
- alexforster 4y ago
  This is only if you have _no_ idea how to use very basic open-source tools to wreak havoc via some open proxies. The real cost of launching small-scale attacks like this is $5/month on your favorite VPS provider.
  - deleted 4y ago
    [deleted]
  - jorvi 4y ago
    I imagine your VPS provider will gladly give your info to law enforcement whereas a DDoS company might not.
    - alexforster 4y ago
      Unfortunately, this type of abuse is essentially only acted on if you either (a) cause some kind of problem for ops at the VPS company, or (b) the victim tells on you. Even then, the absolute worst thing that will happen to you is your account will be closed. There is just no way on the modern internet to investigate/prosecute these kinds of things. That's why one of our primary goals with the free Cloudflare plan is to invert the problem: make DDoS go away by making it it free to mitigate.
  - fasterthanlime 4y ago
    For anyone reading this: Alex knows about the attack and has my permission to talk about it publicly.
  - FooBarWidget 4y ago
    Open proxies? I heard about this more than a decade ago, but why would anybody in 2022 run an open proxy? Or are these open proxies unintentional, i.e. misconfigured?
minroot 4y ago
Not to criticize, but how the hell do you write this long posts? Why don't others do it too?
- fasterthanlime 4y ago
  Grew up in a very religious, very "literal interpretation of the texts" home. They had everyone hit the books and even write reports. I guess the habit stuck with me after I left the fold.
  - _AzMoo 4y ago
    I assume you're keeping extensive notes as well, right?
- kretaceous 4y ago
  +1. I love Amos' extensive writings.
ReactiveJelly 4y ago
It's more fun in my head if "CloudFlare Attack Mode" allows you to wield CloudFlare as a Black-ICE weapon https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Electronics#blackice https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Elec...
mastax 4y ago
That was fun! Showed some caveats to "I have cloudflare so I'm fine."
ameliaquining 4y ago
Regarding the issue of people sending requests directly to your server, bypassing Cloudflare: While you can use IP allowlisting for this, Cloudflare can also be configured to send requests to your server exclusively over mTLS, with a client certificate chaining up to a root CA which they publish. If I were worried about this kind of attack, I'd probably turn that feature on (and then reject requests that don't present a valid client cert); I don't fully trust IP allowlisting. (On the other hand, I can't say for sure that the performance overhead wouldn't be a problem in a DDoS scenario. But it doesn't seem super likely?)
- dx034 4y ago
  Or you use Cloudflare Tunnel and don't open Ports 80/443 at all.
- kungfufrog 4y ago
  This is a good alternative when security is the priority. Wouldn't verifying an mTLS certs validity be significantly more expensive than whitelisting IP blocks though? Especially when a server is potentially under duress? Can you cache the verification so it wouldn't have to be done each time? All legitimate questions! I don't have the answers but am curious.
  - fasterthanlime 4y ago
    mTLS certainly seems like the the most expensive option here (not that expensive outside attacks, though). S-tier implementations include: firewall rules or a BPF program, or a VPN-based approach (like Cloudflare Tunnel). The way I did it is fine for small-scale attacks like that one, but a large enough attack will have you spend too much time on syscalls and waste valuable kernel resources. I'd love to read a write-up about how these different approaches perform in practice, because this is largely gut feeling / the popular wisdom that "the sooner you block, the better".
  - kevin_nisbet 4y ago
    > Wouldn't verifying an mTLS certs validity be significantly more expensive than whitelisting IP blocks though? I wouldn't expect this to be all that significant if already using TLS. So in the context of only allowing cloudflare to establish connections to the service, and be 100% fronted by cloudflare, mTLS isn't much more expensive than already using TLS, which we all do every day. And my read of the blog post, is most of the expense / optimizations are in the time it takes the server to generate and serve a page. > Especially when a server is potentially under duress? Well, it's a tradeoff. The desired property is only cloudflare can make requests to the actual web server, so compared to IP whitelisting there might be some tradeoffs. I think the biggest one is maintenance, IP whitelisting requires staying ontop of any changes from cloudflare. I'm sure cloudflare is good at pre-announcing new IPs, PoPs, etc, but there is a small risk to missing this, especially if it's a set it and forget approach. Although to be fair, mTLS also has this depending on when the root is set to expire. > Can you cache the verification so it wouldn't have to be done each time? Sort of, but it depends on some support on both the client and server, so it depends on whether both sides have opted into it. The terms you're looking for to do more research are TLS session resumption, which does involve it's own security concerns. More effective at this level, is clients will re-use connections. So as long as the client/server support Connection: keep-alive, you do the mTLS once and run hundreds or thousands of queries in the single TLS verification. My 2 cents is the biggest downside of the mTLS approach is you may still leak the location of the webserver. If the web server is on the same domain or a domain known to be used by the target, some certificate scans may turn it up. IIRC server cert is exchanged before client certificate, so it's possible to leak this without presenting a client certificate. I'd have to double check the spec to be sure though. Although to be fair, this problem also exists in the specific implementation of whitelisting from the OP, as it appears to be code embedded in the web server to drop the connection if not from an approved source. So someone scanning for certs / servers could still learn about the server from the TLS connection, and do a simple flood the server out of existence type attack, or some more limited resource type attacks on number of active connections, etc. Ideally, with this type of whitelisting you would want to do it in iptables or the host firewall, so that you just blackhole the unapproved IPs and reveal nothing about the existence of the server.
jrockway 4y ago
Fun! I tend to be paranoid about exposing things to the Internet, so just put my raw servers behind Envoy. I have tuned that to do rate limiting, circuit breaking (stop sending requests to an upstream when it returns too many errors), idle connection termination, and to shed load when a certain amount of memory is in use, so without any additional configuration for a new service behind the proxy it's somewhat difficult to get the proxy and other services to not respond at all. I'm guessing that in a real attack, the rate limiting service is a weak link. I use a custom rate limit service to aggregate rate limits across a /24 (and hacked that together in an evening), and that is likely the first thing to blow up and erroneously deny service to legitimate users. (I'm sure I have it set up to fail closed, which will be annoying.) I had a hard time ever generating enough load to test any of this for the static serving path. I just set up a mirror of my production environment on my workstation, limited the critical services (Envoy + nginx + rate limit + Redis) to some low amount of CPUs, and then had 31 workers generate synthetic load. I was able to get circuit breakers to open to at least prove that that code works, but I somehow think that I'll run out of network bandwidth before I run out of memory to keep track of open streams. Difficult to load test when the upstream can respond to most requests out of memory. Would be interesting to dig into it more. But for those of you reading this and thinking "I'm going to launch an attack right now", I will just turn off the site if I go over my bandwidth quota. Clone the config repo, host everything locally, run your tests, and send me the results ;)
andrelaszlo 4y ago
I was very surprised to see that your article about golang got flagged here on HN. Never realized that go was such a sensitive topic. I assume this is related? Can anyone in the golang community give some context?
- komuW 4y ago
  I don't think the flagging has anything to do with the Golang community. From my understanding, that came from the HN moderators themselves. @dang made some explanations on one of those threads
  - andrelaszlo 4y ago
    Oh, I missed that! Didn't realize the article was considered so inflammatory. I guess I'm desensitized :) Reading what dang wrote makes it seem really sensible. Links to dang's comments: https://news.ycombinator.com/item?id=31207191 https://news.ycombinator.com/item?id=31207191 https://news.ycombinator.com/item?id=31207126 https://news.ycombinator.com/item?id=31207126
  - dang 4y ago
    The flags came from users. That's almost always the case (see https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html).
    - andrelaszlo 4y ago
      Not sure what happened to my comment but I guess you hid it dang? Thanks! It wasn't my intention to reignite the whole thing.
      - dang 4y ago
        A user marked it offtopic, which downweighted it. A few longstanding HN users have the ability to do this, as part of an experiment we've been running. I've written about it in a few places: https://news.ycombinator.com/item?id=31228950 https://news.ycombinator.com/item?id=31228950 https://news.ycombinator.com/item?id=31227642 https://news.ycombinator.com/item?id=31227642 https://news.ycombinator.com/item?id=31227584 https://news.ycombinator.com/item?id=31227584 https://news.ycombinator.com/item?id=31117569 https://news.ycombinator.com/item?id=31117569 The downweighting is intended for when overly generic or offtopic subthreads are stuck at the top of a page, choking out more interesting or on-topic conversation. (Unfortunately it caused some trouble a while back because one such downweighted subthread was critical of a YC startup, and a much more important rule is that we moderate HN less when YC or YC startups are the topic. But it was a straightforward mistake—the user who marked the thread generic just didn't realize that the topic was YC related.) Since it's an experiment, we want to keep an eye on it. Currently I have the software send an email each time a downweight is applied, so we can review them as part of going through the regular HN inbox. When I saw this one, it was such a perfect example that I came to the thread and auto-collapsed it as well. So that's its current status: downweighted and auto-collapsed. Btw, basically none of these situations get created intentionally. They're a tragedy-of-the-commons thing where each person does what they do innocently but it all adds up to something suboptimal. That's pretty much what moderation exists for: to jiggle the system when it gets stuck in one of its failure modes.
londons_explore 4y ago
34 million requests really isn't many. Bigger sites might handle that number of requests every second. Hand coded and highly optimized services can handle that number of cached small requests every minute on one machine. After all, that's only an egress rate of a few GBits. And your homepage certainly ought to be both cached and small.
- dx034 4y ago
  Depends on the request. 34 million calls to a static page isn't much, even for the smallest VPS. 34 million requests to a data-heavy page with dynamically generated and hard to cache content can be a lot.
  - deleted 4y ago
    [deleted]
- lambda 4y ago
  I take it you didn't read the article? Because that was the author's conclusion, and caching content for logged out users was the main solution.
- efficax 4y ago
  > Bigger sites might handle that number of requests every second. I can think of maybe 10 sites that get anywhere near that amount of traffic and they have millions and millions of dollars of infrastructure behind them. 34 million requests per second is an absurd amount! Nobody even knew how to handle that level of traffic 10 years ago.
komuW 4y ago
The code in this article is great advertisement for Golang. Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>? Gesundheit indeed. Great writing as always.
- viraptor 4y ago
  Not that great. The first outer layer here is pining. It's been discussed for some time now but not yet available in go. It may hit 1.19 maybe... (https://github.com/golang/go/issues/46787 https://github.com/golang/go/issues/46787)
dym_sh 4y ago
does it make you a freeloader xD
tothrowaway 4y ago
OpenResty (Nginx + LuaJIT) can help you limit the damage of unsophisticated DDoS attacks like these. I keep a count of the requests-per-second I'm getting in each nginx worker. I also set a special cookie for every response from the upstream (it could literally be foo=bar). When the RPS goes approve a certain threshold, if the special cookie is not present, I serve a static HTML page (bypassing the upstream) that sets the cookie and reloads the page (Nginx can do 20K+ RPS without breaking a sweat). In my experience, these fly by DDoS attacks never use cookies, so legitimate users can get through, but the bots are blocked. Of course, if you get hit with something slightly more targeted, this defense is worthless.
- krono 4y ago
  Either it's the cost/benefit ratio that been keeping them from handling this, or it simply hadn't crossed their minds until now in which case you might soon need to come up with a new mitigation strategy ;)
  - pythux 4y ago
    Definitely a matter of cost/benefit ratio. DDOS using headless browsers are pretty common.
weird-eye-issue 4y ago
For sites that are mostly read only from non-logged in users Cloudflare can be great. I've got a WordPress content site configured with Cloudflare's APO. Almost 100% of actual human requests for HTML pages or static resources are cached. And there are also a few caching layers at the server level too (setup automatically by Cloudways). The site generates around $15k per month and growing fast on an overprovisioned $50/mo server. I don't think I'll need to spend much more on hosting at even 10x the current traffic due to the caching by Cloudflare. Maybe a bit more for disk space since we store large images but DigitalOcean's Block Storage also makes that incredibly cheap
- stavros 4y ago
  I switched my site to static and it's become hassle-free to deploy ever since. Just copy the files anywhere and you're done. I assume it can't be DDoSed either, though I don't think nobody has ever tried (or they did and I didn't notice).
  - weird-eye-issue 4y ago
    By using Cloudflare APO my site is essentially static. The origin can go down and 99% of pages and all the associated resources can still be served. For our business use case WordPress is a necessity and so switching to a true static solution simple isn't feasible (we acquire and merge other content sites. 99% of content sites being sold are built on WordPress so being in the ecosystem is critical for this reason and many others)
deleted 4y ago
[deleted]
mwcampbell 4y ago
> if you want to skip it, search for "After the storm". > Yes, yes, I know, I should add anchor links for headers. This is a rare case where blind people using screen readers have it (a little) easier. Every serious screen reader I know of has a command to skip to the next heading. It's too bad most sighted web users don't have a similar feature handy.
- bmn__ 4y ago
  An attempt was made: https://news.ycombinator.com/item?id=30463632 https://news.ycombinator.com/item?id=30463632
  - stavros 4y ago
    Oh how I miss this... If Vivaldi has this feature, I might have to switch.
sammy811 4y ago
- 8organicbits 4y ago
  What are you using to generate the traffic? It would be cool to see what's needed on the other end of the pipe.
  - sammy811 4y ago
    - fasterthanlime 4y ago
      Why were some only some requests Range requests? (the status codes included 206 and 200)
      - sammy811 4y ago
        I think whichever PoP india gets served by was very overloaded, resulting in a lot of timeouts, so when wget retries (I set infinite retry attempts), it will do a range request from wherever it left off
        deleted 4y ago
        [deleted]
    - dinedal 4y ago
      Just so I understand, you decided to protest this pricing not by boycotting the service, or making a public statement about it, but instead by targeting an employee's use of the platform with an attack? I'm sorry, I really don't get why you did this or why you thought it would be effective.
    - 8organicbits 4y ago
      How are you connecting from so many different IP addresses?
oefrha 4y ago
> As for fly.io, well, I work there, so, they pay me. Well, that’s nice and all, but if a fly.io customer were attacked with 3.1GB/s throughput, according to the lowest outbound bandwidth price of $0.02/GB [1] they’d be burning at least $3.72/min. 6 times that if attacked from India. That would be a lot less fun. [1] https://fly.io/docs/about/pricing/ https://fly.io/docs/about/pricing/ Edit: They mentioned they waive charges as a result of attacks: https://community.fly.io/t/about-rate-limiting/156/4 https://community.fly.io/t/about-rate-limiting/156/4
- sammy811 4y ago
  - kuschku 4y ago
    This sounds like you're claiming responsibility for the attack?
    - steveklabnik 4y ago
      They do so more explicitly downthread.
      - LegionMammal978 4y ago
        Do you recall what this user said downthread? They seem to have deleted their other comments.
        bronson 4y ago
        The posts were flagged, you can see them if you turn on showdead. Here's sammy811's threadstarter: > I was the one "attacking" the video platform! I saw fly io had insane bandwidth pricing for India, so I spawned a couple of VMs in India to constantly pull the 4k video. Sad the whole ASN got blocked!
        4y ago
junon 4y ago
> The traffic doesn't look like something like headless Chrome was used Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence, what could be the vector for a botnet infection if this were true? Extensions?
- teraku 4y ago
  Could be anything. Could even be just a user with a few beefy machines and a lot of proxies.
- marginalia_nu 4y ago
  Most botnets I've encountered seen seem to consist of enterprise routers and the like, random IoT-stuff. Most of it probably too old and low-powered to even run headless chrome.
- mlyle 4y ago
  > Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence It doesn't look like headless chrome, or headed chrome, or any kind of chrome... because there's just requests for one file and no other resources. Chrome would do a lot of other stuff. This is a wget loop or some other minimal agent with a changed user agent string.
floor_ 4y ago
A security audit of my small 6 person office found a Digital Ocean machine trying to brute force one of our Windows machines. I'm becoming less and less impressed with Digital Ocean as time goes on.
- notsound 4y ago
  This is not the best solution (fail2ban and 2fa would be better) but https://github.com/skeeto/endlessh https://github.com/skeeto/endlessh is a neat tool if you want to annoy someone with unsophisticated scripts. It’s worth noting that annoying a script kiddie might get you ddosed.
  - floor_ 4y ago
    Because were are so small we were able to easily set up allow lists.
- fasterthanlime 4y ago
  I snitch-tagged Digital Ocean and other VPS providers involved in the attack, but didn't expect much. They're much bigger than this: what's a big deal for my toy server is barely a blip on their radar. And realistically, there's only so much they can do about someone running Tor exit nodes / an open proxy on their infra. Everyone in the cloud space has been fighting that off (and miners) for years, it's one arms race among many.
- jrockway 4y ago
  My impression from running a service that was attacked is that Digital Ocean is pretty much average here. Free CI providers, cheap VPSes, and compromised boutique hosts all provide a significant amount of traffic. The variance is such that there is no one ASN to block that mitigates any sort of coordinated attack in a meaningful way. I think what is happening here is that there are lots of free hosts that let you send traffic to websites (in my case, volume didn't matter, just people signing up for free trials to get a little bit of free compute), and there is really no way for cloud providers to reduce the volume in a meaningful way. They are not necessarily serving malicious customers, rather their legitimate customers have gotten hacked and are now the attack vector. Or, their business is hosting, and people using THEIR free trials are using the free trials for abuse. (Consider if you just want a new IP address with which to sign up for some web service how easy it is to use something like the CircleCI "free for open source" plan to do that.) If I ever started my own cloud provider, one thing I'd want to get under control is a good view of traffic leaving the cloud provider. Probably more than ports + bits per second; actually proxy the HTTPS or whatever. That way, if someone starts abusing other people's stuff, there is at least a point where I can rate limit it ("kill all video downloads to notable Rust personality's website because they asked me to") while hacked customers get their stuff cleaned up. This is a hard problem, balancing security and good Internet citizenship, but something I'd want to spend some time on. Anyway, TL;DR, DigitalOcean shows up on your radar because they are pretty popular. Lots of Linux VPSes equals lots of insecure Linux VPSes, which is the perfect point for launching another attack. There is only so much the cloud provider can do, but doing more would certainly be nice.
oxff 4y ago
Why was the site breaking under such a little stress? For requests per second, that is really not that much.
- Aissen 4y ago
  It's... explained in the article?
- fasterthanlime 4y ago
  The article covers this in great detail. Do a search for "fair enough" to jump to the relevant section.
  - oxff 4y ago
    (embarrassing, thanks!)
CraigJPerry 4y ago
I had never considered service back-pressure or circuit breakers until it became a buzz in the java world a few years ago. The concepts are universal though and genuinely interesting building blocks for architecture. Site Reliability Engineering is a fascinating problem space.
tristor 4y ago
This inspired me to actually look at my CloudFlare stats, and I realized it's not caching some of my HTML pages even though I had page rules setup to cache everything. I use a static site generator, and it's never been an issue at this point (nobody has any reason to DDoS me). But this is a good motivator to fix that up. I've been procrastinating on fixing a few things on my site, so I'm adding this to the list when I update it. I might also setup the CloudFlare Tunnel at the same time.
- adamch 4y ago
  Hello, friendly Cloudflare employee here... there's a much simpler alternative to managing cache policy if your whole website is static. We built a tool for deploying static websites to Cloudflare [1]. It watches your Git repo for changes, then rebuilds your site and serves _all_ the assets from our edge servers. No need to configure page rules or cache policy, because there's no more origin. The free tier covers 500 builds per month (at most, you'll do 1 build per commit, so you should be covered just fine). I helped develop Tunnel over the last ~3 years, and I love it, but it's definitely overkill if you just want to serve some static files on the edge. (Please delete if this goes against HN policy, I'm trying not to be a shill here, just help tristor avoid spending an hour configuring page rules and cache policies) [1] https://pages.cloudflare.com/ https://pages.cloudflare.com/
  - tristor 4y ago
    This is great info. I haven't tried out CloudFlare Pages yet. I may consider it for my next site, but I mostly host my personal site on a server because I have it for other reasons, and the site takes up minimal resources. Plus learning how to manage the caching rules correctly seems like a good skill to pick up :)
justinsaccount 4y ago
Probably doesn't matter much with only a few networks, but this is using the wrong data structure: if let Some(net) = ip_nets.load() .iter() .find(|net| net.contains(&addr.ip())) ip_nets is a 'HashSet<IpNet>' but it should be a radix/patricia tree. Something like https://lib.rs/crates/iprange https://lib.rs/crates/iprange
- mleonhard 4y ago
  The `iprange` library contains two bits of `unsafe` code [0]. [0] https://github.com/sticnarf/iprange-rs/search?q=unsafe https://github.com/sticnarf/iprange-rs/search?q=unsafe
  - LegionMammal978 4y ago
    The first one seems to be a necessary workaround for the lack of GATs; its safety comment is pretty trivially correct. The second one is entirely unnecessary, and I've just filed a PR to get rid of it.
- asadawadia 4y ago
  why
  - justinsaccount 4y ago
    Because a radix tree has O(1) lookups even if you have 500,000 ip ranges. It's how routers route.
- fasterthanlime 4y ago
  I looked for that kind of data structure for 30 seconds in the ipnet crate itself, didn't find it, noticed there were only 23 IP ranges and decided it was fine. (Keep in mind this happened during the attack, so compromises)

461 points by 0xedb 4y ago

I'm interested in what the fly.io would cost if you didn't work there. I got a fly app and can't proxy it through cloudflare cause that doesn't work.

julianbuse 4y ago
I have proxied a fly.io app through cloudflare. The last time I tried, it worked.
- langsoul-com 4y ago
  Not sure if it's because I'm using a subdomain or not. But I created a cert, added the a and aaaa records to CF, then after it was verified, turned back on the CF proxy. However, my app stopped working until I turned off the proxy

Slaren 4y ago

[dead]

mleonhard 4y ago

I'm working on two projects that could let a stand-alone Rust web server weather a moderate DDoS attack like the author had: Beatrice [0] - A web server with built-in connection limits and thread limits. It's async but supports non-async request handlers. fair-rate-limiter [1] - In theory, one could use this to shed most of the load from DDoS attacking nodes. [0] https://crates.io/crates/beatrice https://crates.io/crates/beatrice [1] https://crates.io/crates/fair-rate-limiter https://crates.io/crates/fair-rate-limiter

2143 4y ago

> Minutes after I posted this article, the attack resumed. Same shit, different AS. Noob question: was that caused by the article getting posted to HN? Or was it really an attack?

mou_sesp_az 4y ago

This guy has some great energy. And humor: >> Because it doesn't return an AddrStream but instead a Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>... >> Gesundheit.

d4a 4y ago

> That lets me answer questions like "what RSS readers (that aren't browsers) is my audience using?" Very interesting... I also use NNW and FreshRSS

fasterthanlime 4y ago

Mirror in case the attack successfully picks up again: https://web.archive.org/web/20220502013024/https://fasterthanli.me/articles/i-won-free-load-testing# https://web.archive.org/web/20220502013024/https://fastertha... As a treat, this is a testament to a logic error I made in the caching code (inserted uncachable versions of pages into the cache for a little while). Enjoy!

terhechte 4y ago
I'm spending a lot of time in Rust but I always learn so much new when reading your articles. Particularly, how do you manage find all these interesting crates that you're using. Like arc-swap, or Moka, or color-eyre. Do you actively search for these things or do you have a habit of reading the what's new crates.io feed in the morning with a cup of coffee?
- nicoburns 4y ago
  Not the OP, but the r/rust subreddit is my other lurking spot besides HN. (And I’m also currently working on a site to document some of these useful crates)
  - fasterthanlime 4y ago
    /r/rust is a great place for that, I also occasionally click through most of the https://lib.rs/ https://lib.rs/ categories, just to see what the names I don't recognize are about. My twitter timeline also has tons of neat little crates pop up every month, and I have to thank my colleagues at Netlify (back when they were doing Rust) and now at fly, for showing me some of those I showcase in the article.
    - no_wizard 4y ago
      Is Netlify not doing rust now? What’s the story there?
      - 4y ago

the_alchemist 4y ago

What a great post-mortem write-up! Thans for sharing

daenz 4y ago

Out of curiosity, I googled how much a DDoS attack goes for these days. Apparently they can cost as low as $10/hour. I don't know if the shady people will deliver, but that's what the internet says. So apparently it's pretty easy to DDoS anyone and make it difficult to trace back to you.

alexforster 4y ago
This is only if you have _no_ idea how to use very basic open-source tools to wreak havoc via some open proxies. The real cost of launching small-scale attacks like this is $5/month on your favorite VPS provider.
- deleted 4y ago
  [deleted]
- jorvi 4y ago
  I imagine your VPS provider will gladly give your info to law enforcement whereas a DDoS company might not.
  - alexforster 4y ago
    Unfortunately, this type of abuse is essentially only acted on if you either (a) cause some kind of problem for ops at the VPS company, or (b) the victim tells on you. Even then, the absolute worst thing that will happen to you is your account will be closed. There is just no way on the modern internet to investigate/prosecute these kinds of things. That's why one of our primary goals with the free Cloudflare plan is to invert the problem: make DDoS go away by making it it free to mitigate.
- fasterthanlime 4y ago
  For anyone reading this: Alex knows about the attack and has my permission to talk about it publicly.
- FooBarWidget 4y ago
  Open proxies? I heard about this more than a decade ago, but why would anybody in 2022 run an open proxy? Or are these open proxies unintentional, i.e. misconfigured?

minroot 4y ago

Not to criticize, but how the hell do you write this long posts? Why don't others do it too?

fasterthanlime 4y ago
Grew up in a very religious, very "literal interpretation of the texts" home. They had everyone hit the books and even write reports. I guess the habit stuck with me after I left the fold.
- _AzMoo 4y ago
  I assume you're keeping extensive notes as well, right?
kretaceous 4y ago
+1. I love Amos' extensive writings.

ReactiveJelly 4y ago

It's more fun in my head if "CloudFlare Attack Mode" allows you to wield CloudFlare as a Black-ICE weapon https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Electronics#blackice https://en.wikipedia.org/wiki/Intrusion_Countermeasures_Elec...

mastax 4y ago

That was fun! Showed some caveats to "I have cloudflare so I'm fine."

ameliaquining 4y ago

Regarding the issue of people sending requests directly to your server, bypassing Cloudflare: While you can use IP allowlisting for this, Cloudflare can also be configured to send requests to your server exclusively over mTLS, with a client certificate chaining up to a root CA which they publish. If I were worried about this kind of attack, I'd probably turn that feature on (and then reject requests that don't present a valid client cert); I don't fully trust IP allowlisting. (On the other hand, I can't say for sure that the performance overhead wouldn't be a problem in a DDoS scenario. But it doesn't seem super likely?)

dx034 4y ago
Or you use Cloudflare Tunnel and don't open Ports 80/443 at all.
kungfufrog 4y ago
This is a good alternative when security is the priority. Wouldn't verifying an mTLS certs validity be significantly more expensive than whitelisting IP blocks though? Especially when a server is potentially under duress? Can you cache the verification so it wouldn't have to be done each time? All legitimate questions! I don't have the answers but am curious.
- fasterthanlime 4y ago
  mTLS certainly seems like the the most expensive option here (not that expensive outside attacks, though). S-tier implementations include: firewall rules or a BPF program, or a VPN-based approach (like Cloudflare Tunnel). The way I did it is fine for small-scale attacks like that one, but a large enough attack will have you spend too much time on syscalls and waste valuable kernel resources. I'd love to read a write-up about how these different approaches perform in practice, because this is largely gut feeling / the popular wisdom that "the sooner you block, the better".
- kevin_nisbet 4y ago
  > Wouldn't verifying an mTLS certs validity be significantly more expensive than whitelisting IP blocks though? I wouldn't expect this to be all that significant if already using TLS. So in the context of only allowing cloudflare to establish connections to the service, and be 100% fronted by cloudflare, mTLS isn't much more expensive than already using TLS, which we all do every day. And my read of the blog post, is most of the expense / optimizations are in the time it takes the server to generate and serve a page. > Especially when a server is potentially under duress? Well, it's a tradeoff. The desired property is only cloudflare can make requests to the actual web server, so compared to IP whitelisting there might be some tradeoffs. I think the biggest one is maintenance, IP whitelisting requires staying ontop of any changes from cloudflare. I'm sure cloudflare is good at pre-announcing new IPs, PoPs, etc, but there is a small risk to missing this, especially if it's a set it and forget approach. Although to be fair, mTLS also has this depending on when the root is set to expire. > Can you cache the verification so it wouldn't have to be done each time? Sort of, but it depends on some support on both the client and server, so it depends on whether both sides have opted into it. The terms you're looking for to do more research are TLS session resumption, which does involve it's own security concerns. More effective at this level, is clients will re-use connections. So as long as the client/server support Connection: keep-alive, you do the mTLS once and run hundreds or thousands of queries in the single TLS verification. My 2 cents is the biggest downside of the mTLS approach is you may still leak the location of the webserver. If the web server is on the same domain or a domain known to be used by the target, some certificate scans may turn it up. IIRC server cert is exchanged before client certificate, so it's possible to leak this without presenting a client certificate. I'd have to double check the spec to be sure though. Although to be fair, this problem also exists in the specific implementation of whitelisting from the OP, as it appears to be code embedded in the web server to drop the connection if not from an approved source. So someone scanning for certs / servers could still learn about the server from the TLS connection, and do a simple flood the server out of existence type attack, or some more limited resource type attacks on number of active connections, etc. Ideally, with this type of whitelisting you would want to do it in iptables or the host firewall, so that you just blackhole the unapproved IPs and reveal nothing about the existence of the server.

jrockway 4y ago

Fun! I tend to be paranoid about exposing things to the Internet, so just put my raw servers behind Envoy. I have tuned that to do rate limiting, circuit breaking (stop sending requests to an upstream when it returns too many errors), idle connection termination, and to shed load when a certain amount of memory is in use, so without any additional configuration for a new service behind the proxy it's somewhat difficult to get the proxy and other services to not respond at all. I'm guessing that in a real attack, the rate limiting service is a weak link. I use a custom rate limit service to aggregate rate limits across a /24 (and hacked that together in an evening), and that is likely the first thing to blow up and erroneously deny service to legitimate users. (I'm sure I have it set up to fail closed, which will be annoying.) I had a hard time ever generating enough load to test any of this for the static serving path. I just set up a mirror of my production environment on my workstation, limited the critical services (Envoy + nginx + rate limit + Redis) to some low amount of CPUs, and then had 31 workers generate synthetic load. I was able to get circuit breakers to open to at least prove that that code works, but I somehow think that I'll run out of network bandwidth before I run out of memory to keep track of open streams. Difficult to load test when the upstream can respond to most requests out of memory. Would be interesting to dig into it more. But for those of you reading this and thinking "I'm going to launch an attack right now", I will just turn off the site if I go over my bandwidth quota. Clone the config repo, host everything locally, run your tests, and send me the results ;)

andrelaszlo 4y ago

I was very surprised to see that your article about golang got flagged here on HN. Never realized that go was such a sensitive topic. I assume this is related? Can anyone in the golang community give some context?

komuW 4y ago
I don't think the flagging has anything to do with the Golang community. From my understanding, that came from the HN moderators themselves. @dang made some explanations on one of those threads
- andrelaszlo 4y ago
  Oh, I missed that! Didn't realize the article was considered so inflammatory. I guess I'm desensitized :) Reading what dang wrote makes it seem really sensible. Links to dang's comments: https://news.ycombinator.com/item?id=31207191 https://news.ycombinator.com/item?id=31207191 https://news.ycombinator.com/item?id=31207126 https://news.ycombinator.com/item?id=31207126
- dang 4y ago
  The flags came from users. That's almost always the case (see https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html).
  - andrelaszlo 4y ago
    Not sure what happened to my comment but I guess you hid it dang? Thanks! It wasn't my intention to reignite the whole thing.
    - dang 4y ago
      A user marked it offtopic, which downweighted it. A few longstanding HN users have the ability to do this, as part of an experiment we've been running. I've written about it in a few places: https://news.ycombinator.com/item?id=31228950 https://news.ycombinator.com/item?id=31228950 https://news.ycombinator.com/item?id=31227642 https://news.ycombinator.com/item?id=31227642 https://news.ycombinator.com/item?id=31227584 https://news.ycombinator.com/item?id=31227584 https://news.ycombinator.com/item?id=31117569 https://news.ycombinator.com/item?id=31117569 The downweighting is intended for when overly generic or offtopic subthreads are stuck at the top of a page, choking out more interesting or on-topic conversation. (Unfortunately it caused some trouble a while back because one such downweighted subthread was critical of a YC startup, and a much more important rule is that we moderate HN less when YC or YC startups are the topic. But it was a straightforward mistake—the user who marked the thread generic just didn't realize that the topic was YC related.) Since it's an experiment, we want to keep an eye on it. Currently I have the software send an email each time a downweight is applied, so we can review them as part of going through the regular HN inbox. When I saw this one, it was such a perfect example that I came to the thread and auto-collapsed it as well. So that's its current status: downweighted and auto-collapsed. Btw, basically none of these situations get created intentionally. They're a tragedy-of-the-commons thing where each person does what they do innocently but it all adds up to something suboptimal. That's pretty much what moderation exists for: to jiggle the system when it gets stuck in one of its failure modes.

londons_explore 4y ago

34 million requests really isn't many. Bigger sites might handle that number of requests every second. Hand coded and highly optimized services can handle that number of cached small requests every minute on one machine. After all, that's only an egress rate of a few GBits. And your homepage certainly ought to be both cached and small.

dx034 4y ago
Depends on the request. 34 million calls to a static page isn't much, even for the smallest VPS. 34 million requests to a data-heavy page with dynamically generated and hard to cache content can be a lot.
- deleted 4y ago
  [deleted]
lambda 4y ago
I take it you didn't read the article? Because that was the author's conclusion, and caching content for logged out users was the main solution.
efficax 4y ago
> Bigger sites might handle that number of requests every second. I can think of maybe 10 sites that get anywhere near that amount of traffic and they have millions and millions of dollars of infrastructure behind them. 34 million requests per second is an absurd amount! Nobody even knew how to handle that level of traffic 10 years ago.

komuW 4y ago

The code in this article is great advertisement for Golang. Pin<Box<TimeoutWriter<TimeoutReader<TcpStream>>>>? Gesundheit indeed. Great writing as always.

viraptor 4y ago
Not that great. The first outer layer here is pining. It's been discussed for some time now but not yet available in go. It may hit 1.19 maybe... (https://github.com/golang/go/issues/46787 https://github.com/golang/go/issues/46787)

dym_sh 4y ago

does it make you a freeloader xD

tothrowaway 4y ago

OpenResty (Nginx + LuaJIT) can help you limit the damage of unsophisticated DDoS attacks like these. I keep a count of the requests-per-second I'm getting in each nginx worker. I also set a special cookie for every response from the upstream (it could literally be foo=bar). When the RPS goes approve a certain threshold, if the special cookie is not present, I serve a static HTML page (bypassing the upstream) that sets the cookie and reloads the page (Nginx can do 20K+ RPS without breaking a sweat). In my experience, these fly by DDoS attacks never use cookies, so legitimate users can get through, but the bots are blocked. Of course, if you get hit with something slightly more targeted, this defense is worthless.

krono 4y ago
Either it's the cost/benefit ratio that been keeping them from handling this, or it simply hadn't crossed their minds until now in which case you might soon need to come up with a new mitigation strategy ;)
- pythux 4y ago
  Definitely a matter of cost/benefit ratio. DDOS using headless browsers are pretty common.

weird-eye-issue 4y ago

For sites that are mostly read only from non-logged in users Cloudflare can be great. I've got a WordPress content site configured with Cloudflare's APO. Almost 100% of actual human requests for HTML pages or static resources are cached. And there are also a few caching layers at the server level too (setup automatically by Cloudways). The site generates around $15k per month and growing fast on an overprovisioned $50/mo server. I don't think I'll need to spend much more on hosting at even 10x the current traffic due to the caching by Cloudflare. Maybe a bit more for disk space since we store large images but DigitalOcean's Block Storage also makes that incredibly cheap

stavros 4y ago
I switched my site to static and it's become hassle-free to deploy ever since. Just copy the files anywhere and you're done. I assume it can't be DDoSed either, though I don't think nobody has ever tried (or they did and I didn't notice).
- weird-eye-issue 4y ago
  By using Cloudflare APO my site is essentially static. The origin can go down and 99% of pages and all the associated resources can still be served. For our business use case WordPress is a necessity and so switching to a true static solution simple isn't feasible (we acquire and merge other content sites. 99% of content sites being sold are built on WordPress so being in the ecosystem is critical for this reason and many others)

deleted 4y ago

[deleted]

mwcampbell 4y ago

> if you want to skip it, search for "After the storm". > Yes, yes, I know, I should add anchor links for headers. This is a rare case where blind people using screen readers have it (a little) easier. Every serious screen reader I know of has a command to skip to the next heading. It's too bad most sighted web users don't have a similar feature handy.

bmn__ 4y ago
An attempt was made: https://news.ycombinator.com/item?id=30463632 https://news.ycombinator.com/item?id=30463632
- stavros 4y ago
  Oh how I miss this... If Vivaldi has this feature, I might have to switch.

sammy811 4y ago

8organicbits 4y ago
What are you using to generate the traffic? It would be cool to see what's needed on the other end of the pipe.
- sammy811 4y ago
  - fasterthanlime 4y ago
    Why were some only some requests Range requests? (the status codes included 206 and 200)
    - sammy811 4y ago
      I think whichever PoP india gets served by was very overloaded, resulting in a lot of timeouts, so when wget retries (I set infinite retry attempts), it will do a range request from wherever it left off
      - deleted 4y ago
        [deleted]
  - dinedal 4y ago
    Just so I understand, you decided to protest this pricing not by boycotting the service, or making a public statement about it, but instead by targeting an employee's use of the platform with an attack? I'm sorry, I really don't get why you did this or why you thought it would be effective.
  - 8organicbits 4y ago
    How are you connecting from so many different IP addresses?

oefrha 4y ago

> As for fly.io, well, I work there, so, they pay me. Well, that’s nice and all, but if a fly.io customer were attacked with 3.1GB/s throughput, according to the lowest outbound bandwidth price of $0.02/GB [1] they’d be burning at least $3.72/min. 6 times that if attacked from India. That would be a lot less fun. [1] https://fly.io/docs/about/pricing/ https://fly.io/docs/about/pricing/ Edit: They mentioned they waive charges as a result of attacks: https://community.fly.io/t/about-rate-limiting/156/4 https://community.fly.io/t/about-rate-limiting/156/4

sammy811 4y ago
- kuschku 4y ago
  This sounds like you're claiming responsibility for the attack?
  - steveklabnik 4y ago
    They do so more explicitly downthread.
    - LegionMammal978 4y ago
      Do you recall what this user said downthread? They seem to have deleted their other comments.
      - bronson 4y ago
        The posts were flagged, you can see them if you turn on showdead. Here's sammy811's threadstarter: > I was the one "attacking" the video platform! I saw fly io had insane bandwidth pricing for India, so I spawned a couple of VMs in India to constantly pull the 4k video. Sad the whole ASN got blocked!
        4y ago

junon 4y ago

> The traffic doesn't look like something like headless Chrome was used Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence, what could be the vector for a botnet infection if this were true? Extensions?

teraku 4y ago
Could be anything. Could even be just a user with a few beefy machines and a lot of proxies.
marginalia_nu 4y ago
Most botnets I've encountered seen seem to consist of enterprise routers and the like, random IoT-stuff. Most of it probably too old and low-powered to even run headless chrome.
mlyle 4y ago
> Setting aside the fact that headless chrome or other browser testbeds do a good job at hiding their presence It doesn't look like headless chrome, or headed chrome, or any kind of chrome... because there's just requests for one file and no other resources. Chrome would do a lot of other stuff. This is a wget loop or some other minimal agent with a changed user agent string.

floor_ 4y ago

A security audit of my small 6 person office found a Digital Ocean machine trying to brute force one of our Windows machines. I'm becoming less and less impressed with Digital Ocean as time goes on.

notsound 4y ago
This is not the best solution (fail2ban and 2fa would be better) but https://github.com/skeeto/endlessh https://github.com/skeeto/endlessh is a neat tool if you want to annoy someone with unsophisticated scripts. It’s worth noting that annoying a script kiddie might get you ddosed.
- floor_ 4y ago
  Because were are so small we were able to easily set up allow lists.
fasterthanlime 4y ago
I snitch-tagged Digital Ocean and other VPS providers involved in the attack, but didn't expect much. They're much bigger than this: what's a big deal for my toy server is barely a blip on their radar. And realistically, there's only so much they can do about someone running Tor exit nodes / an open proxy on their infra. Everyone in the cloud space has been fighting that off (and miners) for years, it's one arms race among many.
jrockway 4y ago
My impression from running a service that was attacked is that Digital Ocean is pretty much average here. Free CI providers, cheap VPSes, and compromised boutique hosts all provide a significant amount of traffic. The variance is such that there is no one ASN to block that mitigates any sort of coordinated attack in a meaningful way. I think what is happening here is that there are lots of free hosts that let you send traffic to websites (in my case, volume didn't matter, just people signing up for free trials to get a little bit of free compute), and there is really no way for cloud providers to reduce the volume in a meaningful way. They are not necessarily serving malicious customers, rather their legitimate customers have gotten hacked and are now the attack vector. Or, their business is hosting, and people using THEIR free trials are using the free trials for abuse. (Consider if you just want a new IP address with which to sign up for some web service how easy it is to use something like the CircleCI "free for open source" plan to do that.) If I ever started my own cloud provider, one thing I'd want to get under control is a good view of traffic leaving the cloud provider. Probably more than ports + bits per second; actually proxy the HTTPS or whatever. That way, if someone starts abusing other people's stuff, there is at least a point where I can rate limit it ("kill all video downloads to notable Rust personality's website because they asked me to") while hacked customers get their stuff cleaned up. This is a hard problem, balancing security and good Internet citizenship, but something I'd want to spend some time on. Anyway, TL;DR, DigitalOcean shows up on your radar because they are pretty popular. Lots of Linux VPSes equals lots of insecure Linux VPSes, which is the perfect point for launching another attack. There is only so much the cloud provider can do, but doing more would certainly be nice.

oxff 4y ago

Why was the site breaking under such a little stress? For requests per second, that is really not that much.

Aissen 4y ago
It's... explained in the article?
fasterthanlime 4y ago
The article covers this in great detail. Do a search for "fair enough" to jump to the relevant section.
- oxff 4y ago
  (embarrassing, thanks!)

CraigJPerry 4y ago

I had never considered service back-pressure or circuit breakers until it became a buzz in the java world a few years ago. The concepts are universal though and genuinely interesting building blocks for architecture. Site Reliability Engineering is a fascinating problem space.

tristor 4y ago

This inspired me to actually look at my CloudFlare stats, and I realized it's not caching some of my HTML pages even though I had page rules setup to cache everything. I use a static site generator, and it's never been an issue at this point (nobody has any reason to DDoS me). But this is a good motivator to fix that up. I've been procrastinating on fixing a few things on my site, so I'm adding this to the list when I update it. I might also setup the CloudFlare Tunnel at the same time.

adamch 4y ago
Hello, friendly Cloudflare employee here... there's a much simpler alternative to managing cache policy if your whole website is static. We built a tool for deploying static websites to Cloudflare [1]. It watches your Git repo for changes, then rebuilds your site and serves _all_ the assets from our edge servers. No need to configure page rules or cache policy, because there's no more origin. The free tier covers 500 builds per month (at most, you'll do 1 build per commit, so you should be covered just fine). I helped develop Tunnel over the last ~3 years, and I love it, but it's definitely overkill if you just want to serve some static files on the edge. (Please delete if this goes against HN policy, I'm trying not to be a shill here, just help tristor avoid spending an hour configuring page rules and cache policies) [1] https://pages.cloudflare.com/ https://pages.cloudflare.com/
- tristor 4y ago
  This is great info. I haven't tried out CloudFlare Pages yet. I may consider it for my next site, but I mostly host my personal site on a server because I have it for other reasons, and the site takes up minimal resources. Plus learning how to manage the caching rules correctly seems like a good skill to pick up :)

justinsaccount 4y ago

Probably doesn't matter much with only a few networks, but this is using the wrong data structure: if let Some(net) = ip_nets.load() .iter() .find(|net| net.contains(&addr.ip())) ip_nets is a 'HashSet<IpNet>' but it should be a radix/patricia tree. Something like https://lib.rs/crates/iprange https://lib.rs/crates/iprange

mleonhard 4y ago
The `iprange` library contains two bits of `unsafe` code [0]. [0] https://github.com/sticnarf/iprange-rs/search?q=unsafe https://github.com/sticnarf/iprange-rs/search?q=unsafe
- LegionMammal978 4y ago
  The first one seems to be a necessary workaround for the lack of GATs; its safety comment is pretty trivially correct. The second one is entirely unnecessary, and I've just filed a PR to get rid of it.
asadawadia 4y ago
why
- justinsaccount 4y ago
  Because a radix tree has O(1) lookups even if you have 500,000 ip ranges. It's how routers route.
fasterthanlime 4y ago
I looked for that kind of data structure for 30 seconds in the ipnet crate itself, didn't find it, noticed there were only 23 IP ranges and decided it was fine. (Keep in mind this happened during the attack, so compromises)