4 ms·

Some fun numbers: 25M requests to fasterthanli.me (just the frontpage, the most expensive page to do queries / render templates for), not cached by CloudFlare (

Some fun numbers: 25M requests to fasterthanli.me (just the frontpage, the most expensive page to do queries / render templates for), not cached by CloudFlare (they don't cache HTML unless you set up a rule specifically for it), and a ~7Gb/s peak to tube.fasterthanli.me (hitting a single video asset) Like any other DDoS, the goals are 1) take the site down temporarily, 2) cost me some money, 3) try to get whichever provider I'm using to boot me off their network. They've achieved 1) for a few hours off today, the rest has been fairly entertaining honestly.

Matthias247 4y ago
You should definitely try to enable HTML caching - that's what CDNs are far, and it will the workload much cheaper to serve and an origin server less likely to fall over. Besides that, looking at the Rust code for the origin you link on twitter, you should make some changes to make it reliable at scale. First of all, I recommend to add timeouts. Otherwise the amount of open sockets will just creep up if RSTs got lost/dropped and there is no data to send - which ultimately makes things prone to resource exhaustion. Also be aware that the tower ConcurrencyLimitLayer alone is not a great solution for this problem - it will build a queue of requests and if clients don't give up the queue gets longer and longer until also no current requests are served anymore and clients will again time out (=> website becomes unreachable). It's better to reject requests fast once a limit is reached than to build infinite queues. Regarding observability, one can log the amount of processed requests, connections, active requests and connections (to determine which things are stuck), maybe status codes. All these things should require emitting one datapoint per minute or so, which is cheap.
- fasterthanlime 4y ago
  This is good advice! All this stuff is very close to my day job, I just never bothered doing it for my personal website because it withstood the front page several times over the past two years. I've done most of what you mentioned (minus load shedding & connection metrics) and have posted about on Twitter, if you want to check out the thread again!

Matthias247 4y ago

You should definitely try to enable HTML caching - that's what CDNs are far, and it will the workload much cheaper to serve and an origin server less likely to fall over. Besides that, looking at the Rust code for the origin you link on twitter, you should make some changes to make it reliable at scale. First of all, I recommend to add timeouts. Otherwise the amount of open sockets will just creep up if RSTs got lost/dropped and there is no data to send - which ultimately makes things prone to resource exhaustion. Also be aware that the tower ConcurrencyLimitLayer alone is not a great solution for this problem - it will build a queue of requests and if clients don't give up the queue gets longer and longer until also no current requests are served anymore and clients will again time out (=> website becomes unreachable). It's better to reject requests fast once a limit is reached than to build infinite queues. Regarding observability, one can log the amount of processed requests, connections, active requests and connections (to determine which things are stuck), maybe status codes. All these things should require emitting one datapoint per minute or so, which is cheap.

fasterthanlime 4y ago
This is good advice! All this stuff is very close to my day job, I just never bothered doing it for my personal website because it withstood the front page several times over the past two years. I've done most of what you mentioned (minus load shedding & connection metrics) and have posted about on Twitter, if you want to check out the thread again!