5 ms·
Ask HN: How can I tell if a site is being rate limited?
I live in a country with less than ideal Internet freedom standards and I have a suspicion that a popular publishing site is being rate limited. How can I know for sure? Is it possible to show a smoking gun?
- chefkoch 6y agoYou could test from vps in another country?
- Raed667 6y agoMy country used to heavily censor internet pages, do finishing attacks on Facebook, Gmail etc.. The easiest way to prove it is by using a VPN or a proxy. Check with different ISPs, also try mobile vs home.
- caseyslaught 6y agoI can definitely notice an improvement in performance when I use a VPN, however it would be cool to see where things are being delayed. Is traceroute or curl something that could help here?
- Retric 6y agoYea, if it’s faster via VPN that’s a strong hint. If you want proof you need to examine the internet between you and the site. If a VPN connection that uses a specific router to to the site is fast, your connection through that same router is slow, and you can connect to some other site through that same router and get a fast connection, then that’s a smoking gun that your connection is deliberately slowed. That said, it’s not easy to prove it’s your country/ISP doing the slow down vs that website slowing down connections to your country.
- rdbell 6y agoMy company has a residential proxy network that lets you test endpoints with requests from residential broadband ISPs all over the world. Some of our customers are using our network for exactly this use case: https://packetstream.io https://packetstream.io
- sairamkunala 6y agoits possible that your ISP has a bad peering connection. Checking with your ISP usually helps understand what is going on. It could be the bandwidth limitation of your ISP itself. Sites like Github are slow when accessed from few countries. It's just that their servers are located mainly in the US(afaik).
- caseyslaught 6y agoI ran a curl test and noticed that the TCP connect time is where things are getting hung up, taking over 80 seconds! Does anyone know if this is indicative of deliberate rate limiting or just a bad peering connection?
- toast0 6y agoIf it takes 80 seconds to TCP connect, something is pretty wrong. tcpdump / wireshark would likely show lots of retransmits. Traceroute/mtr may give you an idea of where the loss starts. If you can find servers on similar hosting, you can get some idea of if it's targetted or not.
- bbulkow 6y agoPossibly neither, that is, there are many possible reasons. for example, stateful firewalls can only track a certain number of connections. If you don't have a public IP, or if they are trying to protect you, it may be necessary to wait for a connection resource to come available. Connections are generally expensive, different from data transfer, for these and other reasons, such as port starvation, syn flood defenses inappropriately targeting you, so many others.
- mtlynch 6y agoI don't have personal experience using it, but I believe Ooni Probe is designed for your exact use-case: https://ooni.org/about/ https://ooni.org/about/
- caseyslaught 6y agoThanks for the recommendation!
- leowoo91 6y agoIf implemented with respect to the http code, it should say 429 in the developer console.
- sfkdjf9j3j 6y agoState internet restrictions aren't implemented in the application layer.
- bmm6o 6y agoThat's not the kind of rate limiting he's asking about. He is talking about the data stream being throttled.
- crazygringo 6y agoJust curious -- if a country didn't want a site to be seen, why would they rate-limit it instead of just blocking connections completely? Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?
- Liru 6y agoPeople's attention spans are ridiculously short. If the person is actively looking for the content, sure, that obviously won't work. However, if it's a more casual browser, then going to a site that takes a long time to load between different pages is definitely a way to get them to not visit it any more. This even works on tech-savvy people, as opposed to just blocking a site which will just get them to use a proxy. > Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability? China and Russia do this with a lot of western sites, from what I've heard.
- jedberg 6y agoIt would be too obvious. If you can't get to the site you know it's blocked. If it's just super slow all the time, you won't know why and may just get frustrated and leave.
- opportune 6y agoYes: https://news.ycombinator.com/item?id=22541960 https://news.ycombinator.com/item?id=22541960 I have seen this actually happen IRL
- crazygringo 6y agoThanks so much for the link. That was really educational, answered my question completely.
- 7ewis 6y agoWould mtr[0] help diagnose this? [0] - https://github.com/traviscross/mtr https://github.com/traviscross/mtr
- deleted 6y ago[deleted]
- willcipriano 6y agoOne possible method: Write a script to request the site continuously and see if it fails to respond with any regularity. From there you can start dialing it in and determine what the rate limit actually is. A rate limit should be consistently reproducible unless it's implemented in a clever way.
- W4phle_Stomp 6y agoHave some fun at the same time: 1) get a vpn account (proton, other) 2) hack your worst enemy's wifi with your favourite tools 3) while outside your enemy's home, boot a linux live distro 4) change your computer's name to your enemy's name 5) connect to the enemy's wifi 6) try to visit all the sites banned by your country 7) insult your country's leader in a public forum using your enemy's name 8) test the speed of the suspected rate-limited site 9) set-up your vpn account settings 10) using your vpn, re-test the rate-limited site 11) subtracting a fair percentage slow down because of the vpn, gauge if there's a significant difference between the two tests 12) turn off computer, go home, enjoy tasty beverage 13) grab some popcorn, go watch your enemy's house from a reasonable distance 14) post video on youtube of your enemy getting busted, using your favourite heading If that's all too much fun for you, then skip most of it and only do #s 8 to 11 from your own network or favourite hot-spot.
- DeathArrow 6y agoHe should also modify the said enemy's browser cache for this to work well.
- caseyslaught 6y agoThis is very insightful. Tip of the hat to you!
- meesterdude 6y agoThe thing about rate limits is, there are often hard limits. Try querying the site a bunch and see what the data transfer rate is. if it's consistently incapable of going above say, 128K/sec, you've got your rate limit. Other times, they'll allow for bursts but sustained get rate limited - these can be detected just the same depending on their approach; if it's per connection (and not IP) you may have to find a big file to download. It also helps to establish that traffic to other sites is not rate limited, and to use 3rd party checkers that check for connection rate across the globe. Or DIY and spin up a bunch of EC2 machines that are geographically diverse.
- rozab 6y agoI don't know about this, I've heard that limiting is often intentionally inconsistent because this makes people less likely to visit. I think Harlow did some similar stuff with his monkeys.
- bitcrazy 6y agoIf the government entity really is throttling that site, assuming they are competent, you're unlikely to find conclusive proof that they're actually throttling. Compare this with a full on blackout (i.e. Great Firewall of China, where that's obvious proof of government intervention); the fact that they're allowing some requests to go thru means that the meddling is likely engineered to not be detectable. Best you can find is maybe the speeds are slower when accessed from within the country, but they can always plausibly deny it.