6 ms·

One starts to wonder, at what point might it be actually feasible to do it the other way around, by whitelisting IP ranges. I could see this happening as a comm

One starts to wonder, at what point might it be actually feasible to do it the other way around, by whitelisting IP ranges. I could see this happening as a community effort, similar to adblocker list curation etc.

worthless-trash 1y ago
I admin a few local business sites.. I whitelist all the countries isps and the strangeness in the logs and attack counts have gone down. Google indexes in country, as does a few other search engines.. Would recommend.
- coffee_am 1y ago
  Is there a public curated list of "good ips" to whitelist ?
  - worthless-trash 1y ago
    So, its relatively easy because there is limited ISP's in my country. I imagine its a much harder option for the US. I looked at all the IP ranges delegated by APNIC, along with every local ISP that I could find, unioned this with https://lite.ip2location.com/australia-ip-address-ranges https://lite.ip2location.com/australia-ip-address-ranges And so far i've not had any complaints. and I think that I have most of them. At some time in the future, i'll start including https://github.com/ebrasha/cidr-ip-ranges-by-country https://github.com/ebrasha/cidr-ip-ranges-by-country
  - partyguy 1y ago
    > Is there a public curated list of "good ips" to whitelist ? https://github.com/AnTheMaker/GoodBots https://github.com/AnTheMaker/GoodBots
plaguna 1y ago
[flagged]
- valvix 1y ago
  [flagged]
  - plaguna 1y ago
    [flagged]
    - ThrowMeAway1618 1y ago
      >There is no need to disagree on such strongly worded statements. What's the bigoted history of those terms? from here[0]: "The English dramatist Philip Massinger used the phrase "black list" in his 1639 tragedy The Unnatural Combat.[2] "After the restoration of the English monarchy brought Charles II of England to the throne in 1660, a list of regicides named those to be punished for the execution of his father.[3] The state papers of Charles II say "If any innocent soul be found in this black list, let him not be offended at me, but consider whether some mistaken principle or interest may not have misled him to vote".[4] In a 1676 history of the events leading up to the Restoration, James Heath (a supporter of Charles II) alleged that Parliament had passed an Act requiring the sale of estates, "And into this black list the Earl of Derby was now put, and other unfortunate Royalists".[5]" Are you an enemy of Charles II? Is that what the problem is? [0] https://en.wikipedia.org/wiki/Blacklisting#Origins_of_the_term https://en.wikipedia.org/wiki/Blacklisting#Origins_of_the_te...
      - defrost 1y ago
        [flagged]
        ThrowMeAway1618 1y ago
        The origin of the term 'black list' had absolutely nothing to do with the melanin content of anyone. In fact, when that term was coined, it had nothing to do with the melanin content of anyone. It was a list of the enemies of Charles II. That's why I posted that. I'd also point out that in my lifetime, folks with darker skin called themselves black and proudly so. As Mr. Brown[0][1] will unambiguously tell you. Regardless, claiming that a term for the property of absorbing visible light is bigoted, to every use of such a term is ridiculous on its face. By your logic, if I wear black socks, I'm a bigot? Or am only a bigot if I actually refer to those socks as "black." Should I use "socks of color" so as not to be a bigot? If I like that little black dress, I'm a bigot as well? Or only if I say "I like that little black dress?" Look. I get it. Melanin content is worthless as a determinant of the value of a human. And anyone who thinks otherwise is sorely and sadly mistaken. It's important to let folks know that there's only one race of sentient primates on this planet -- Homo Sapiens. What's more, we are all, no matter where we come from, incredibly closely related from a genetic standpoint. The history of bigotry, murder and enslavement by and to our fellow humans is long, brutal and disgusting. But nitpicking terms (like black list) that never had anything to do with that bigotry seems performative at best. As I mentioned above, do you also make such complaints about black socks or shoes? Black dresses? Black foregrounds/backgrounds? If not, why not? That's not a rhetorical question. [0] https://www.youtube.com/watch?v=oM1_tJ6a2Kw https://www.youtube.com/watch?v=oM1_tJ6a2Kw [1] https://www.azlyrics.com/lyrics/jamesbrown/sayitloudimblackandimproud.html https://www.azlyrics.com/lyrics/jamesbrown/sayitloudimblacka...
ygritte 1y ago
Came here to say something similar. The sheer amount of IP addresses one has to block to keep malware and bots at bay is becoming unmanageable.
- immibis 1y ago
  Can you explain more about blocking malware as opposed to bots?
  - ygritte 1y ago
    No opposition. Just block the IP address.
- leviathant 1y ago
  Knowing my audience, I've blocked entire countries to stop the pain. Even that was a bit of whack-a-mole. Blocking China cooled off the traffic for a few days, then it came roaring back via Singapore. Blocked Singapore, had a reprieve for a while, and then it was India, with a vengeance. Cloudflare has been a godsend for protecting my crusty old forum from this malicious, wasteful behavior.
bobbiechen 1y ago
Unfortunately, well-behaved bots often have more stable IPs, while bad actors are happy to use residential proxies. If you ban a residential proxy IP you're likely to impact real users while the bad actor simply switches. Personally I don't think IP level network information will ever be effective without combining with other factors. Source: stopping attacks that involve thousands of IPs at my work.
- throwawayffffas 1y ago
  > If you ban a residential proxy IP you're likely to impact real users while the bad actor simply switches. Are you really? How likely do you think is a legit customer/user to be on the same IP as a residential proxy? Sure residential IPS get reused, but you can handle that by making the block last 6-8 hours, or a day or two.
  - micahdeath 1y ago
    We blocked AT&T Mobile once... You get lots of complaints that way and we only blocked them for an hour.
  - richardwhiuk 1y ago
    In these days of CGNAT, a residential IP is shared by multiple customers.
  - immibis 1y ago
    Very likely. You can voluntarily run one to make ~$10/month in cryptocurrency. Many others are botnets. They aren't signing up for new internet connections solely to run proxies on.
- BLKNSLVR 1y ago
  Blocking a residential proxy doesn't sound like a bad idea to me. My single-layer thought process: If they're knowingly running a residential proxy then they'll likely know "the cost of doing business". If they're unknowingly running a residential proxy then blocking them might be a good way for them to find out they're unknowingly running a residential proxy and get their systems deloused.
delusional 1y ago
At that point it almost sounds like we're doing "peering" agreements at the IP level. Would it make sense to have a class of ISPs that didn't peer with these "bad" network participants?
- JimDabell 1y ago
  If this didn’t happen for spam, it’s not going to happen for crawlers.
- shortrounddev2 1y ago
  Why not just ban all IP blocks assigned to cloud providers? Won't halt botnets but the IP range owned by AWS, GCP, etc is well known
  - hnlmorg 1y ago
    Because crawlers would then just use a different IP which isn’t owned by cloud vendors.
  - jjayj 1y ago
    But my work's VPN is in AWS, and HN and Reddit are sometimes helpful... Not sure what my point is here tbh. The internet sucks and I don't have a solution
  - aorth 1y ago
    Tricky to get a list of all cloud providers, all their networks, and then there are cases like CATO Networks Ltd and ZScaler, which are apparently enterprise security products that route clients traffic through their clouds "for security".
lxgr 1y ago
Many US companies do it already. It should be illegal, at least for companies that still charge me while I’m abroad and don’t offer me any other way of canceling service or getting support.
- withinboredom 1y ago
  I'm pretty sure I still owe t-mobile money. When I moved to the EU, we kept our old phone plans for awhile. Then, for whatever reason, the USD didn't make it to the USD account in time and we missed a payment. Then t-mobile cut off the service and you need to receive a text message to login to the account. Obviously, that wasn't possible. So, we lost the ability to even pay, even while using a VPN. We just decided to let it die, but I'm sure in t-mobile's eyes, I still owe them.
- thenthenthen 1y ago
  This! Dealing with European services from China is also terrible. As is the other way around. Welcome to the intranet!
  - thenthenthen 1y ago
    In addition, my tencent and alicloud instances are also hammered to death by their own bots. Just to add a bit of perspective.
partyguy 1y ago
That's what I'm trying to do here, PRs welcome: https://github.com/AnTheMaker/GoodBots https://github.com/AnTheMaker/GoodBots
- aorth 1y ago
  Noble effort. I might make some pull requests, though I kinda feel it's futile. I have my own list of "known good" networks.
friendzis 1y ago
It's never either/or: you don't have to choose between white and black lists exclusively and most of the traffic is going to come from grey areas anyway. Say you whitelist an address/range and some systems detect "bad things". Now what? You remove that address/range from whitelist? Doo you distribute the removal to your peers? Do you communicate removal to the owner of unwhitelisted address/range? How does owner communicate dealing with the issue back? What if the owner of the range is hosting provider where they don't proactively control the content hosted, yet have robust anti-abuse mechanisms in place? And so on. Whitelist-only is a huge can of worms and whitelists works best with trusted partner you can maintain out-of-band communication with. Similarly blacklists work best with trusted partners, however to determine addresses/ranges that are more trouble than they are worth. And somewhere in the middle are grey zone addresses, e.g. ranges assigned to ISPs with CGNATs: you just cannot reliably label an individual address or even a range of addresses as strictly troublesome or strictly trustworthy by default. Implement blacklists on known bad actors, e.g. the whole of China and Russia, maybe even cloud providers. Implement whitelists for ranges you explicitly trust to have robust anti-abuse mechanisms, e.g. corporations with strictly internal hosts.
jampa 1y ago
The Pokémon Go company tried that shortly after launch to block scraping. I remember they had three categories of IPs: - Blacklisted IP (Google Cloud, AWS, etc), those were always blocked - Untrusted IPs (residential IPs) were given some leeway, but quickly got to 429 if they started querying too much - Whitelisted IPs (IPV4 addresses are used legitimately by many people), for example, my current data plan tells me my IP is from 5 states over, so anything behind a CGNAT. You can probably guess what happens next. Most scrapers were thrown out, but the largest ones just got a modem device farm and ate the cost. They successfully prevented most users from scraping locally, but were quickly beaten by companies profiting from scraping. I think this was one of many bad decisions Pokémon Go made. Some casual players dropped because they didn't want to play without a map, while the hardcore players started paying for scraping, which hammered their servers even more.
- aorth 1y ago
  I have an ad hoc system that is similar, comprised of three lists of networks: known good, known bad, and data center networks. These are rate limited using a geo map in nginx for various expensive routes in my application. The known good list is IPs and ranges I know are good. The known bad list is specific bad actors. The data center networks list is updated periodically based on a list of ASNs belonging to data centers. There are a lot of problems with using ASNs, even for well-known data center operators. First, they update so often. Second, they often include massive subnets like /13(!), which can apparently overlap with routes announced by other networks, causing false positives. Third, I had been merging networks (to avoid overlaps causing problems in nginx) with something like https://github.com/projectdiscovery/mapcidr https://github.com/projectdiscovery/mapcidr but found that it also caused larger overlaps that introduced false positives from adjacent networks where apparently some legitimate users are. Lastly, I had seen suspicious traffic from data center operators like CATO Networks Ltd and ZScaler that are some kind of enterprise security products that route clients through their clouds. Blocking those resulted in some angry users in places I didn't expect... And none of the accounts for the residential ISPs that bots use to appear like legitimate users https://www.trendmicro.com/vinfo/us/security/news/vulnerabilities-and-exploits/a-closer-exploration-of-residential-proxies-and-captcha-breaking-services https://www.trendmicro.com/vinfo/us/security/news/vulnerabil....

worthless-trash 1y ago

I admin a few local business sites.. I whitelist all the countries isps and the strangeness in the logs and attack counts have gone down. Google indexes in country, as does a few other search engines.. Would recommend.

coffee_am 1y ago
Is there a public curated list of "good ips" to whitelist ?
- worthless-trash 1y ago
  So, its relatively easy because there is limited ISP's in my country. I imagine its a much harder option for the US. I looked at all the IP ranges delegated by APNIC, along with every local ISP that I could find, unioned this with https://lite.ip2location.com/australia-ip-address-ranges https://lite.ip2location.com/australia-ip-address-ranges And so far i've not had any complaints. and I think that I have most of them. At some time in the future, i'll start including https://github.com/ebrasha/cidr-ip-ranges-by-country https://github.com/ebrasha/cidr-ip-ranges-by-country
- partyguy 1y ago
  > Is there a public curated list of "good ips" to whitelist ? https://github.com/AnTheMaker/GoodBots https://github.com/AnTheMaker/GoodBots

plaguna 1y ago

[flagged]

valvix 1y ago
[flagged]
- plaguna 1y ago
  [flagged]
  - ThrowMeAway1618 1y ago
    >There is no need to disagree on such strongly worded statements. What's the bigoted history of those terms? from here[0]: "The English dramatist Philip Massinger used the phrase "black list" in his 1639 tragedy The Unnatural Combat.[2] "After the restoration of the English monarchy brought Charles II of England to the throne in 1660, a list of regicides named those to be punished for the execution of his father.[3] The state papers of Charles II say "If any innocent soul be found in this black list, let him not be offended at me, but consider whether some mistaken principle or interest may not have misled him to vote".[4] In a 1676 history of the events leading up to the Restoration, James Heath (a supporter of Charles II) alleged that Parliament had passed an Act requiring the sale of estates, "And into this black list the Earl of Derby was now put, and other unfortunate Royalists".[5]" Are you an enemy of Charles II? Is that what the problem is? [0] https://en.wikipedia.org/wiki/Blacklisting#Origins_of_the_term https://en.wikipedia.org/wiki/Blacklisting#Origins_of_the_te...
    - defrost 1y ago
      [flagged]
      - ThrowMeAway1618 1y ago
        The origin of the term 'black list' had absolutely nothing to do with the melanin content of anyone. In fact, when that term was coined, it had nothing to do with the melanin content of anyone. It was a list of the enemies of Charles II. That's why I posted that. I'd also point out that in my lifetime, folks with darker skin called themselves black and proudly so. As Mr. Brown[0][1] will unambiguously tell you. Regardless, claiming that a term for the property of absorbing visible light is bigoted, to every use of such a term is ridiculous on its face. By your logic, if I wear black socks, I'm a bigot? Or am only a bigot if I actually refer to those socks as "black." Should I use "socks of color" so as not to be a bigot? If I like that little black dress, I'm a bigot as well? Or only if I say "I like that little black dress?" Look. I get it. Melanin content is worthless as a determinant of the value of a human. And anyone who thinks otherwise is sorely and sadly mistaken. It's important to let folks know that there's only one race of sentient primates on this planet -- Homo Sapiens. What's more, we are all, no matter where we come from, incredibly closely related from a genetic standpoint. The history of bigotry, murder and enslavement by and to our fellow humans is long, brutal and disgusting. But nitpicking terms (like black list) that never had anything to do with that bigotry seems performative at best. As I mentioned above, do you also make such complaints about black socks or shoes? Black dresses? Black foregrounds/backgrounds? If not, why not? That's not a rhetorical question. [0] https://www.youtube.com/watch?v=oM1_tJ6a2Kw https://www.youtube.com/watch?v=oM1_tJ6a2Kw [1] https://www.azlyrics.com/lyrics/jamesbrown/sayitloudimblackandimproud.html https://www.azlyrics.com/lyrics/jamesbrown/sayitloudimblacka...

ygritte 1y ago

Came here to say something similar. The sheer amount of IP addresses one has to block to keep malware and bots at bay is becoming unmanageable.

immibis 1y ago
Can you explain more about blocking malware as opposed to bots?
- ygritte 1y ago
  No opposition. Just block the IP address.
leviathant 1y ago
Knowing my audience, I've blocked entire countries to stop the pain. Even that was a bit of whack-a-mole. Blocking China cooled off the traffic for a few days, then it came roaring back via Singapore. Blocked Singapore, had a reprieve for a while, and then it was India, with a vengeance. Cloudflare has been a godsend for protecting my crusty old forum from this malicious, wasteful behavior.

bobbiechen 1y ago

Unfortunately, well-behaved bots often have more stable IPs, while bad actors are happy to use residential proxies. If you ban a residential proxy IP you're likely to impact real users while the bad actor simply switches. Personally I don't think IP level network information will ever be effective without combining with other factors. Source: stopping attacks that involve thousands of IPs at my work.

throwawayffffas 1y ago
> If you ban a residential proxy IP you're likely to impact real users while the bad actor simply switches. Are you really? How likely do you think is a legit customer/user to be on the same IP as a residential proxy? Sure residential IPS get reused, but you can handle that by making the block last 6-8 hours, or a day or two.
- micahdeath 1y ago
  We blocked AT&T Mobile once... You get lots of complaints that way and we only blocked them for an hour.
- richardwhiuk 1y ago
  In these days of CGNAT, a residential IP is shared by multiple customers.
- immibis 1y ago
  Very likely. You can voluntarily run one to make ~$10/month in cryptocurrency. Many others are botnets. They aren't signing up for new internet connections solely to run proxies on.
BLKNSLVR 1y ago
Blocking a residential proxy doesn't sound like a bad idea to me. My single-layer thought process: If they're knowingly running a residential proxy then they'll likely know "the cost of doing business". If they're unknowingly running a residential proxy then blocking them might be a good way for them to find out they're unknowingly running a residential proxy and get their systems deloused.

delusional 1y ago

At that point it almost sounds like we're doing "peering" agreements at the IP level. Would it make sense to have a class of ISPs that didn't peer with these "bad" network participants?

JimDabell 1y ago
If this didn’t happen for spam, it’s not going to happen for crawlers.
shortrounddev2 1y ago
Why not just ban all IP blocks assigned to cloud providers? Won't halt botnets but the IP range owned by AWS, GCP, etc is well known
- hnlmorg 1y ago
  Because crawlers would then just use a different IP which isn’t owned by cloud vendors.
- jjayj 1y ago
  But my work's VPN is in AWS, and HN and Reddit are sometimes helpful... Not sure what my point is here tbh. The internet sucks and I don't have a solution
- aorth 1y ago
  Tricky to get a list of all cloud providers, all their networks, and then there are cases like CATO Networks Ltd and ZScaler, which are apparently enterprise security products that route clients traffic through their clouds "for security".

lxgr 1y ago

Many US companies do it already. It should be illegal, at least for companies that still charge me while I’m abroad and don’t offer me any other way of canceling service or getting support.

withinboredom 1y ago
I'm pretty sure I still owe t-mobile money. When I moved to the EU, we kept our old phone plans for awhile. Then, for whatever reason, the USD didn't make it to the USD account in time and we missed a payment. Then t-mobile cut off the service and you need to receive a text message to login to the account. Obviously, that wasn't possible. So, we lost the ability to even pay, even while using a VPN. We just decided to let it die, but I'm sure in t-mobile's eyes, I still owe them.
thenthenthen 1y ago
This! Dealing with European services from China is also terrible. As is the other way around. Welcome to the intranet!
- thenthenthen 1y ago
  In addition, my tencent and alicloud instances are also hammered to death by their own bots. Just to add a bit of perspective.

partyguy 1y ago

That's what I'm trying to do here, PRs welcome: https://github.com/AnTheMaker/GoodBots https://github.com/AnTheMaker/GoodBots

aorth 1y ago
Noble effort. I might make some pull requests, though I kinda feel it's futile. I have my own list of "known good" networks.

friendzis 1y ago

It's never either/or: you don't have to choose between white and black lists exclusively and most of the traffic is going to come from grey areas anyway. Say you whitelist an address/range and some systems detect "bad things". Now what? You remove that address/range from whitelist? Doo you distribute the removal to your peers? Do you communicate removal to the owner of unwhitelisted address/range? How does owner communicate dealing with the issue back? What if the owner of the range is hosting provider where they don't proactively control the content hosted, yet have robust anti-abuse mechanisms in place? And so on. Whitelist-only is a huge can of worms and whitelists works best with trusted partner you can maintain out-of-band communication with. Similarly blacklists work best with trusted partners, however to determine addresses/ranges that are more trouble than they are worth. And somewhere in the middle are grey zone addresses, e.g. ranges assigned to ISPs with CGNATs: you just cannot reliably label an individual address or even a range of addresses as strictly troublesome or strictly trustworthy by default. Implement blacklists on known bad actors, e.g. the whole of China and Russia, maybe even cloud providers. Implement whitelists for ranges you explicitly trust to have robust anti-abuse mechanisms, e.g. corporations with strictly internal hosts.

jampa 1y ago

The Pokémon Go company tried that shortly after launch to block scraping. I remember they had three categories of IPs: - Blacklisted IP (Google Cloud, AWS, etc), those were always blocked - Untrusted IPs (residential IPs) were given some leeway, but quickly got to 429 if they started querying too much - Whitelisted IPs (IPV4 addresses are used legitimately by many people), for example, my current data plan tells me my IP is from 5 states over, so anything behind a CGNAT. You can probably guess what happens next. Most scrapers were thrown out, but the largest ones just got a modem device farm and ate the cost. They successfully prevented most users from scraping locally, but were quickly beaten by companies profiting from scraping. I think this was one of many bad decisions Pokémon Go made. Some casual players dropped because they didn't want to play without a map, while the hardcore players started paying for scraping, which hammered their servers even more.

aorth 1y ago
I have an ad hoc system that is similar, comprised of three lists of networks: known good, known bad, and data center networks. These are rate limited using a geo map in nginx for various expensive routes in my application. The known good list is IPs and ranges I know are good. The known bad list is specific bad actors. The data center networks list is updated periodically based on a list of ASNs belonging to data centers. There are a lot of problems with using ASNs, even for well-known data center operators. First, they update so often. Second, they often include massive subnets like /13(!), which can apparently overlap with routes announced by other networks, causing false positives. Third, I had been merging networks (to avoid overlaps causing problems in nginx) with something like https://github.com/projectdiscovery/mapcidr https://github.com/projectdiscovery/mapcidr but found that it also caused larger overlaps that introduced false positives from adjacent networks where apparently some legitimate users are. Lastly, I had seen suspicious traffic from data center operators like CATO Networks Ltd and ZScaler that are some kind of enterprise security products that route clients through their clouds. Blocking those resulted in some angry users in places I didn't expect... And none of the accounts for the residential ISPs that bots use to appear like legitimate users https://www.trendmicro.com/vinfo/us/security/news/vulnerabilities-and-exploits/a-closer-exploration-of-residential-proxies-and-captcha-breaking-services https://www.trendmicro.com/vinfo/us/security/news/vulnerabil....