11 ms·
Adaptive rate limiting is a game-changer
- TekMol 3y agoI don't believe that Twitter has to limit requests to avoid bots. They would just have to limit requests by IP and thats it. Say 10 tweets per IP per day. For requests which do not come from logged in users. I can't see how scrapers would be able to come up with enough IPs to make a dent in Twitters costs.
- malfist 3y agoIPv6 makes it trivial to come up with a mind boggling number of ip addresses. You don't even have to leave your network
- jsnell 3y agoNobody does rate limiting or other abuse work on a full /128. Typically a /64 is treated as the equivalent of an IPv4 address.
- charcircuit 3y agoThis will have many false positives. Especially if the address is from a mobile network.
- zhfliz 3y agopeople don't care about false positives on CGNAT either, so not much difference to the IPv4 situation if you target /64s
- costco 3y agoLast I read this is how Instagram rate limits IPV6. See https://news.ycombinator.com/item?id=32687694 https://news.ycombinator.com/item?id=32687694 So apparently not enough false positives for them to care.
- fulafel 3y agoIs it common for mobile networks to hand out short (64 or smaller) prefixes to phones? Phones are routers after all (hotspot) so this would seem to break stuff.
- deleted 3y ago[deleted]
- TekMol 3y agoFirst of all, Twitter does not support IPv6. Secondly, if they would, they could simply rate limit by the first x chars of the address.
- berkle4455 3y agoSo your solution is to just make it a walled garden. What about the scraping bots who simply log-in?
- TekMol 3y agoMy solution is to make it less of a walled garden. At the moment, they often refuse even reading a single tweet when not logged in. And for the "simply log-in": It is even more expensive (way more expensive) to get good email adresses as it is to get good IPs. Email providers have very tough checks these days before they give you one.
- garciasn 3y agoIt has to be behavioral, not IP based for the most part because of the ways these networks operate—primarily on personal computers/routers on US-based residential connections. I deal with these constantly for our clients. It’s amazing how smart they function to get around limits (thousands of IPs and valid recent user agents) but how they miss easy things like using the same cookie to submit 1000s of different entries with obvious purchased lists of names/emails, many of whom are deceased.
- TekMol 3y ago"Thousands of IPs" can not hurt Twitters bottom line in any way. With a "read 10 tweets per IP per day" limit, that is 10k tweets. Delivering 10k tweets does not even cost a penny.
- semicolon_storm 3y ago10 per day is a ridiculously low limit. Also, NAT and shared networks (think college campus) mean that hundreds of people often share the same IP, making that limit even more ridiculous. Reality would be that limit would have to be much higher.
- costco 3y agohttps://brightdata.com/proxy-types/residential-proxies https://brightdata.com/proxy-types/residential-proxies gives you access to 70 million IPs. You'd have to be a little insane to scrape millions of tweets like this though because the bandwidth costs would be very high as even with massive volume you're probably not getting any price lower than $5-7/GB. Fortunately Twitter doesn't rate limit by IP (at least as long as you're not posting), they allow guest access and they have an endpoint that lets you get 100 tweets at a time so it's not very difficult to scrape as I described in https://news.ycombinator.com/item?id=36667225 https://news.ycombinator.com/item?id=36667225.
- 38 3y agoI dont know what the hell Twitter is doing, but I simply dont use it anymore. I cannot access this page without logging in: https://twitter.com/SarahKSilverman https://twitter.com/SarahKSilverman I check it ONCE per day, thats it. and for weeks every time it just prompts me to login. no thanks, done with Twitter.
- deleted 3y ago[deleted]
- pabs3 3y agoYou can use nitter to access Twitter content anonymously at the moment: https://nitter.net/SarahKSilverman https://nitter.net/SarahKSilverman
- jokoon 3y agoNo surprise that it's not already implemented because we just throw more cheap hosting and bandwidth at this problem until it's fixed. The "premature optimization is the root of all evil" misquote really caused a lot of damage to favor growth and profit making instead of better software. It seems programmers have a general habit of just implementing things in the most naive and cheap way and just solve any problem with expensive hardware. We really need a new trend of fast and minimal software because climate change and a finite planet will not allow us to follow Wirth's law for very long.
- andrewstuart2 3y agoThe misquote, in my experience, is abusing the "premature" aspect. It's often thrown around as time-based but my understanding of the original quote is that it's all about assumptions rather than data driving complexity in the name of optimization. Optimization should have measurable improvement and the need for it should be demonstrable. Neither of these is harder than writing a benchmark and/or performing some simple math on data measured for that purpose before going about the optimization.
- mcpackieh 3y agoThe quotes from wikiquote: > "The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming." and another instance of the same: > "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%." https://en.wikiquote.org/wiki/Donald_Knuth#Computer_Programming_as_an_Art_(1974) https://en.wikiquote.org/wiki/Donald_Knuth#Computer_Programm... I'm not really seeing much of a "misquote" to be honest. I have always understood this quote to mean that spending too much time micro-optimizing things unnecessarily will waste your time and become a burden to you later. Instead, optimizations should only be performed when the need for optimization becomes manifest.
- ComputerGuru 3y agoIt was really quite the let down of an article. All that build up, then basically nothing about how adaptive rate limiting would be implemented in practice, how expensive it would be, what it requires, the challenges it involves, etc.
- Sudhanshu456 3y agoThanks for the feedback. I agree I could've gone into more technical depth and implementation details. So, taking your feedback in account, I've modified and added more details in the blog. But, I already have a next blog in planning covering the topic in-depth, along the similar lines of your feedback, because of that I didn't intend to make a long one, decided to split the blog :)
- deleted 3y ago[deleted]
- nisa 3y agoin theory (and also probably in practice) it's a control theory problem and there are already algorithms and ideas out there - here is an article not exactly about rate limiting but implementing something based on this might work really well: https://www.eecs.umich.edu/courses/eecs571/reading/control-to-computer-zaher.pdf https://www.eecs.umich.edu/courses/eecs571/reading/control-t...
- gillh 3y agoAperture project's policy language is indeed inspired by Control Systems, especially PID Control. See: https://docs.fluxninja.com/concepts/advanced/circuit https://docs.fluxninja.com/concepts/advanced/circuit
- ryantgtg 3y agoThis article was crossposted to hell and back last week on reddit. And it has an AI vibe…
- gillh 3y ago
- Dwedit 3y agoArticle is broken if you have JS from cloudfront denied.
- dadie 3y agoI do not know why, but by default the body element has an overflow of hidden set via css. There is probably a good reason to do this that I do not know about. Would love to know the reason behind that from a WebDev as it strikes me as unnecessary having seen the page with JS and without JS and not being able to see much of a difference.
- jazzyjackson 3y agoMight be to avoid a visual reflow - otherwise the content will render, then the webfont will load and the whole page will reflow and jump around. Google amp used to have a ridiculous 4 second timeout waiting for the font
- tracer4201 3y ago[dead]
- thih9 3y agoI feel this has potential to bring new levels of painful ux. > They might prioritize serving the VIPs and slow down the pace when there is an influx of newcomers. Or they might prioritize people who are about to convert and deprioritize them once they pay.
- mgaunard 3y agoAs a user, I paid for a bandwidth and I want that bandwidth. The rest is the service provider's problem.
- michaelcampbell 3y agoNot sure it "changed my game" but I have a process that I run on many files at once. In a *nix I would run the processes like so: for i in list_of_files; do run_intensive_process & sleep $(awk '{print $3}' < /proc/loadavg) done As more processes start upping my load average, it sleeps longer before running each one in the background. I guess GNU parallel can do this sort of thing with a load-sensitive command line arg, but I find this pattern easy to remember, and parallel's decidedly not. Works for me.
- caro11ne 3y agoStop starting more jobs if load > number of cpus: cat list_of_file | parallel --load 100% run_intensive_process
- twic 3y agoYou could also hand all the invocations of run_intensive_process to at, and let it take care of running them when the load average permits: https://linux.die.net/man/1/at https://linux.die.net/man/1/at I don't think at is very heavily used on modern systems, so it's quite possible that it doesn't actually have very useful behaviour here.
- dredmorbius 3y agoMore specifically, batch(1). at(1) runs jobs after a specified time. (A common mistake is to presume that "at" means "will run when specified" when in fact it means "will run no earlier than specified", and depending on system characteristics, could run substantially later.) batch runs jobs ... as system load permits, without any time specification, in a FIFO queue. There's also cron, the third Unix scheduler, which executes a set of specified tasks on a regular schedule. All three are related.
- twic 3y agoI have the idea that at is more widespread than batch, and since you can say 'at now' (or 'at $(date)'), at is a better bet. As far as I know, cron doesn't pay attention to load average, but I would love to be corrected. All of this feels like a very crufty and highly neglected part of Unix, which is a shame. There's an alternate history of Unix where it's all about batch jobs and datagram Unix domain sockets.
- mjb 3y agoThe biggest problem with adaptive rate limiting is that, in many contexts, clients value predictability more than they do peak performance. Adaptive rate limits change client behavior in ways that clients can't predict or control. Even in networking, the classic application domain of adaptive rate limits (like TCP's algorithms), core networks are built to offer predictable performance. Edge networks, like public WiFi, that offers best-effort performance with adaptive rate limits, are often met with frustration and even anger the moment those limits kick in.
- twic 3y agoEasy enough to implement on top of a token bucket filter - just adjust the rate at which tokens are added.