5 ms·

> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also d

> I can't help but feel like these are the dying breaths of the open Internet though I agree with the over zealous tracking by the megacorps but this is also due to bad actors, I work for a financial company and the amount of API abuse, ATO, DDoS, nefarious bot traffic, etc. we see on a daily basis is absolutely insane

berkes 2y ago
But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to application firewalls and private subnets and whatnot. E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. So, I think the amount of attacks indeed can be insane. But the amount that need over zealous tracking is to be countered, is, AFAICS, rather small.
- throwaway99210 2y ago
  > E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. the arms race is always escalating
  - JohnMakin 2y ago
    we see the same thing, also with a financial company, the most successful strategies we’ve seen is making stuff like this extremely expensive for whoever it is if we see it, and they stop or slow down to a point it becomes not worth it and they move on. sometimes that’s really all you can do without harming legit traffic.
  - josephcsible 2y ago
    Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]
    - RiverCrochet 2y ago
      That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header. While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests. As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attempts to access `.env`, for example.
      - josephcsible 2y ago
        When legitimate users viewed that forum post, their browsers would, in the course of loading the image, attempt to HTTP GET /phpmyadmin/whatever.png, with that being the URL in the actual request in the header.
      - berkes 2y ago
        A lot of modern standards rely on .well-known urls to convey abilities, endpoints, related services and so on. In my case, I never run anything PHP so I'll just plain block out anything PHP (same for python, lua, activedirectory etc). And, indeed, .htaccess, .env etc. A rather large list of hardcoded stuff that gets an instant-ban. It drops the bot-traffic with 90% or more. These obviously aren't targeted attacks. Protecting against those is another issue alltogether.
- tialaramex 2y ago
  A big problem is that where we have a good solution you'll lose if you insist on that solution but other people get away with doing something that's crap but customers like better. We often have to mandate a poor solution that will be tolerated because if we mandate the better solution it will be rejected, and if we don't mandate anything the outcomes are far worse. Today for example I changed energy company†. I made a telephone call, from a number the company has never seen before. I told them my name (truthfully but I could have lied) and address (likewise). I agreed to about five minutes of parameters, conditions, etc. and I made one actual meaningful choice (a specific tariff, they offer two). I then provided 12 digits identifying a bank account (they will eventually check this account exists and ask it to pay them money, which by default will just work) and I'm done. Notice that anybody could call from a burner and that would work too. They could move Aunt Sarah's energy to some random outfit, assign payments to Jim's bank account, and cause maybe an hour of stress and confusion for both Sarah and Jim when months or years later they realise the problem. We know how to do this properly, but it would be high friction and that's not in the interests of either the "energy companies" or the politicians who created this needlessly complicated "Free Market" for energy. We could abolish that Free Market, but again that's not in their interests. So, we're stuck with this waste of our time and money, indefinitely. There have been simpler versions of this system, which had even worse outcomes. They're clumsier to use, they cause more people to get scammed AND they result in higher cost to consumers, so that's not great. And there are better systems we can't deploy because in practice too few consumers will use them, so you'd have 0% failure but lower total engagement and that's what matters. † They don't actually supply either gas or electricity, that's a last mile problem solved by a regulated monopoly, nor do they make electricity or drill for gas - but they do bill me for the gas and electricity I use - they're an artefact of Capitalism.
- Szpadel 2y ago
  I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to ridiculous number of large CIDRs, blocking any IP caused it instantly replace it with another. blocking by AS number was also mixed bag, as this list growed up really quickly, most of that were registered to suspicious looking Gmail addresses. (I feel that such activity might own significant percentage of total ipv4 space) This was basically cat and mouse game of finding some specific characteristic in requests that matches all that traffic and filtering it, but the other side would adapt next day or on Sunday. aggregated amount of traffic was in range of 2-20k r/s to basically heaviest endpoint in the shop, with was the main reason we needed to block that traffic (it generated 20-40x load of organic traffic) cloudflare was also not really successful with default configuration, we had to basically challenge everyone by default with whitelist of most common regions from where we expected customers. So best solution is to track everyone and calculate long term reputation.
  - stareatgoats 2y ago
    Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the internet announce their full name and residential address in order to satisfy the requirements of a social score ...
    - Szpadel 2y ago
      The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. As stated before the main reason we needed to block it was volume of the traffic, you migh imagine identical scenario for dealing with DDoS attack.
- mattpallissard 2y ago
  That's not the same type of bot net. Fail 2 ban simply is not going to work when you have a popular unauthenticated endpoint. You have hundreds of thousands of rps spread across thousands of legitimate networks that. The requests are always modified to look legitimate in a never ending game of whack-a-mole. You wind up having to use things like tls fingerprinting with other heuristics to identify what to traffic to reject. These all take engineering hours and require infrastructure. It is SO MUCH SIMPLER to require auth and reject everything else outright. I know that the BigCo's want to track us and you originally mentioned tracking not auth. But my point is yeah, they have malicious reasons for locking things down, but there are legitimate reasons too.
  - sangnoir 2y ago
    > You wind up having to use things like tls fingerprinting ...and we've circled back to the post's subject - a version of curl that impersonates browsers TLS handshake behavior to bypass such fingerprinting.
  - fijiaarone 2y ago
    Easy solution to rate limit. Require initial request to get 1 time token with a 1 second delay And then require valid requests to include the token. The token returned has a salt with something like timestamp and ip. That way they can only bombard the token generator. get /token Returns token with timestamp in salted hash get /resource?token=abc123xyz Check for valid token and drop or deny.
    - int0x29 2y ago
      As at least one person working on this has pointed out in this thread: their adversaries have IP blocks and ASNs.
- jsnell 2y ago
  The question is a bit of a non sequitur, since this is not tracking. The TLS fingerprint is not a useful tracking vector, by itself nor as part of some composite fingerprint.
  - fijiaarone 2y ago
    The point is that you have to use an approved client (eg browser, os) with an approved cert authority that goes through approved gatekeepers (eg Cloudflare, Akamai)
    - jsnell 2y ago
      That seems pretty unlikely to be the original point of https://news.ycombinator.com/item?id=42549415 https://news.ycombinator.com/item?id=42549415, which mentions none of that, and doesn't even have directionally the same concerns. But also, what you wrote is basically nonsense. Clients don't need "an approved cert authority". Nor are there any "approved gatekeepers", all major browsers are equally happy connecting to your Raspberry Pi as they are connecting to Cloudflare.
- miki123211 2y ago
  This depends on what you're fighting. If you're fighting adversaries that go for scale, AKA trying to hack as many targets as possible, mostly low-sophistication, using techniques requiring 0 human work and seeing what sticks, yes, blocking those simple techniques works. Those attackers don't ever expect to hack Facebook or your bank, that's just not the business they're in. They're fine with posting unsavory ads on your local church's website, blackmailing a school principal with the explicit pictures he stores on the school server, or encrypting all the data on that server and demanding a ransom. If your company does something that is specifically valuable to someone, and there are people whose literal job it is to attack your company's specific systems, no, those simple techniques won't be enough. If you're protecting a Church with 150 members, the simple techniques are probably fine, if you're working for a major bank or a retailer that sells gaming consoles or concert tickets, they're laughably inadequate.
cle 2y ago
Yep totally agree these are problems. I don't have a good alternative proposal either, I'm just disappointed with what we're converging on.
code51 2y ago
Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value that's not envisioned by the original company. I'm guessing investors actually like a healthy dose of open access and a healthy dose of defence. We see them (YC, as an example) betting on multiple teams addressing the same problem. The difference is their execution, the angle they attack. If, say, the financial company you work for is capable in both product and technical aspect, I assume it leaves no gap. It's the main place to access the service and all the side benefits.
- miki123211 2y ago
  > Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value Sometimes the customer you have isn't the customer you want. As a bank, you don't want the customers that will try to log in to 1000 accounts, and then immediately transfer any money they find to the Seychelles. As a ticketing platform, you don't want the customers that buy tickets and then immediately sell them on for 4x the price. As a messaging app, you don't want the customers who have 2000 bot accounts and use AI to send hundreds of thousands of spam messages a day. As a social network, you don't want the customers who want to use your platform to spread pro-russian misinformation. In a sense, those are "customer needs left changing", but neither you nor otherr customers want those needs to be automatible.
deleted 2y ago
[deleted]

berkes 2y ago

But how much of this "bad actor" interaction is countered with tracking? And how many of these attempts are even close to successfull with even the simplest out of the box security practices set up? And when it does get more dangerous, is over zealous tracking the best counter for this? I've dealt with a lot of these threats as well, and a lot are countered with rather common tools, from simple fail2ban rules to application firewalls and private subnets and whatnot. E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. So, I think the amount of attacks indeed can be insane. But the amount that need over zealous tracking is to be countered, is, AFAICS, rather small.

throwaway99210 2y ago
> E.g. a large fai2ban rule to just ban anything that attempts to HTTP GET /admin.php or /phpmyadmin etc, even just once, gets rid of almost all nefarious bot traffic. unfortunately fail2ban wouldn't even make a dent in the attack traffic hitting the endpoints in my day-to-day work, these are attackers utilizing residential proxy infrastructure that are increasingly capable of solving JS/client-puzzle challenges.. the arms race is always escalating
- JohnMakin 2y ago
  we see the same thing, also with a financial company, the most successful strategies we’ve seen is making stuff like this extremely expensive for whoever it is if we see it, and they stop or slow down to a point it becomes not worth it and they move on. sometimes that’s really all you can do without harming legit traffic.
- josephcsible 2y ago
  Such a rule is a great way to let malicious users lock out a bunch of your legitimate customers. Imagine if someone makes a forum post and includes this in it: [img]https://example.com/phpmyadmin/whatever.png[/img]
  - RiverCrochet 2y ago
    That would be in the body of the request. OP is talking about URLs in the actual request, which is part of the header. While I don't have experience with a great number of WAFs I'm sure sophisticated ones let you be quite specific on where you are matching text to identify bad requests. As an aside, another "easy win" is assuming any incoming HTTP request for a dotfile is malicious. I see constant unsolicitied attempts to access `.env`, for example.
    - josephcsible 2y ago
      When legitimate users viewed that forum post, their browsers would, in the course of loading the image, attempt to HTTP GET /phpmyadmin/whatever.png, with that being the URL in the actual request in the header.
    - berkes 2y ago
      A lot of modern standards rely on .well-known urls to convey abilities, endpoints, related services and so on. In my case, I never run anything PHP so I'll just plain block out anything PHP (same for python, lua, activedirectory etc). And, indeed, .htaccess, .env etc. A rather large list of hardcoded stuff that gets an instant-ban. It drops the bot-traffic with 90% or more. These obviously aren't targeted attacks. Protecting against those is another issue alltogether.
tialaramex 2y ago
A big problem is that where we have a good solution you'll lose if you insist on that solution but other people get away with doing something that's crap but customers like better. We often have to mandate a poor solution that will be tolerated because if we mandate the better solution it will be rejected, and if we don't mandate anything the outcomes are far worse. Today for example I changed energy company†. I made a telephone call, from a number the company has never seen before. I told them my name (truthfully but I could have lied) and address (likewise). I agreed to about five minutes of parameters, conditions, etc. and I made one actual meaningful choice (a specific tariff, they offer two). I then provided 12 digits identifying a bank account (they will eventually check this account exists and ask it to pay them money, which by default will just work) and I'm done. Notice that anybody could call from a burner and that would work too. They could move Aunt Sarah's energy to some random outfit, assign payments to Jim's bank account, and cause maybe an hour of stress and confusion for both Sarah and Jim when months or years later they realise the problem. We know how to do this properly, but it would be high friction and that's not in the interests of either the "energy companies" or the politicians who created this needlessly complicated "Free Market" for energy. We could abolish that Free Market, but again that's not in their interests. So, we're stuck with this waste of our time and money, indefinitely. There have been simpler versions of this system, which had even worse outcomes. They're clumsier to use, they cause more people to get scammed AND they result in higher cost to consumers, so that's not great. And there are better systems we can't deploy because in practice too few consumers will use them, so you'd have 0% failure but lower total engagement and that's what matters. † They don't actually supply either gas or electricity, that's a last mile problem solved by a regulated monopoly, nor do they make electricity or drill for gas - but they do bill me for the gas and electricity I use - they're an artefact of Capitalism.
Szpadel 2y ago
I can tell you about my experience with blocking traffic from scalpers bots that were very active during pandemic. All requests produced by those bots were valid ones, nothing that could be flagged by tools like fail2ban etc (my assumption is that it would be the same for financial systems). Any blocking or rate limiting by IP is useless, we saw about 2-3 requests per minute per IP, and those actors had access to ridiculous number of large CIDRs, blocking any IP caused it instantly replace it with another. blocking by AS number was also mixed bag, as this list growed up really quickly, most of that were registered to suspicious looking Gmail addresses. (I feel that such activity might own significant percentage of total ipv4 space) This was basically cat and mouse game of finding some specific characteristic in requests that matches all that traffic and filtering it, but the other side would adapt next day or on Sunday. aggregated amount of traffic was in range of 2-20k r/s to basically heaviest endpoint in the shop, with was the main reason we needed to block that traffic (it generated 20-40x load of organic traffic) cloudflare was also not really successful with default configuration, we had to basically challenge everyone by default with whitelist of most common regions from where we expected customers. So best solution is to track everyone and calculate long term reputation.
- stareatgoats 2y ago
  Blocking scalper bot traffic by any means, be it by source or certified identification seems a lost cause, i.e. not possible because it can always be circumvented. Why did you not have that filter at point of sale instead? I'm sure there are reasons, but to have a battery of captchas and a limit on purchases per credit card seems on the surface much more sturdy. And it doesn't require that everyone browsing the internet announce their full name and residential address in order to satisfy the requirements of a social score ...
  - Szpadel 2y ago
    The product they tried to buy what not in stock anyways, but their strategy was to constantly try anyways, so in case it would become in stock they would be the first to get it. It was all for guest checkout, so no address yet to validate nor credit card. Because they used API endpoints used by the frontend we could not use any captcha at this place because of technical requirements. As stated before the main reason we needed to block it was volume of the traffic, you migh imagine identical scenario for dealing with DDoS attack.
mattpallissard 2y ago
That's not the same type of bot net. Fail 2 ban simply is not going to work when you have a popular unauthenticated endpoint. You have hundreds of thousands of rps spread across thousands of legitimate networks that. The requests are always modified to look legitimate in a never ending game of whack-a-mole. You wind up having to use things like tls fingerprinting with other heuristics to identify what to traffic to reject. These all take engineering hours and require infrastructure. It is SO MUCH SIMPLER to require auth and reject everything else outright. I know that the BigCo's want to track us and you originally mentioned tracking not auth. But my point is yeah, they have malicious reasons for locking things down, but there are legitimate reasons too.
- sangnoir 2y ago
  > You wind up having to use things like tls fingerprinting ...and we've circled back to the post's subject - a version of curl that impersonates browsers TLS handshake behavior to bypass such fingerprinting.
- fijiaarone 2y ago
  Easy solution to rate limit. Require initial request to get 1 time token with a 1 second delay And then require valid requests to include the token. The token returned has a salt with something like timestamp and ip. That way they can only bombard the token generator. get /token Returns token with timestamp in salted hash get /resource?token=abc123xyz Check for valid token and drop or deny.
  - int0x29 2y ago
    As at least one person working on this has pointed out in this thread: their adversaries have IP blocks and ASNs.
jsnell 2y ago
The question is a bit of a non sequitur, since this is not tracking. The TLS fingerprint is not a useful tracking vector, by itself nor as part of some composite fingerprint.
- fijiaarone 2y ago
  The point is that you have to use an approved client (eg browser, os) with an approved cert authority that goes through approved gatekeepers (eg Cloudflare, Akamai)
  - jsnell 2y ago
    That seems pretty unlikely to be the original point of https://news.ycombinator.com/item?id=42549415 https://news.ycombinator.com/item?id=42549415, which mentions none of that, and doesn't even have directionally the same concerns. But also, what you wrote is basically nonsense. Clients don't need "an approved cert authority". Nor are there any "approved gatekeepers", all major browsers are equally happy connecting to your Raspberry Pi as they are connecting to Cloudflare.
miki123211 2y ago
This depends on what you're fighting. If you're fighting adversaries that go for scale, AKA trying to hack as many targets as possible, mostly low-sophistication, using techniques requiring 0 human work and seeing what sticks, yes, blocking those simple techniques works. Those attackers don't ever expect to hack Facebook or your bank, that's just not the business they're in. They're fine with posting unsavory ads on your local church's website, blackmailing a school principal with the explicit pictures he stores on the school server, or encrypting all the data on that server and demanding a ransom. If your company does something that is specifically valuable to someone, and there are people whose literal job it is to attack your company's specific systems, no, those simple techniques won't be enough. If you're protecting a Church with 150 members, the simple techniques are probably fine, if you're working for a major bank or a retailer that sells gaming consoles or concert tickets, they're laughably inadequate.

cle 2y ago

Yep totally agree these are problems. I don't have a good alternative proposal either, I'm just disappointed with what we're converging on.

code51 2y ago

Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value that's not envisioned by the original company. I'm guessing investors actually like a healthy dose of open access and a healthy dose of defence. We see them (YC, as an example) betting on multiple teams addressing the same problem. The difference is their execution, the angle they attack. If, say, the financial company you work for is capable in both product and technical aspect, I assume it leaves no gap. It's the main place to access the service and all the side benefits.

miki123211 2y ago
> Much of this "bad actor" activity is actually customer needs left hanging - for either the customer to automate herself or other companies to fill the gap to create value Sometimes the customer you have isn't the customer you want. As a bank, you don't want the customers that will try to log in to 1000 accounts, and then immediately transfer any money they find to the Seychelles. As a ticketing platform, you don't want the customers that buy tickets and then immediately sell them on for 4x the price. As a messaging app, you don't want the customers who have 2000 bot accounts and use AI to send hundreds of thousands of spam messages a day. As a social network, you don't want the customers who want to use your platform to spread pro-russian misinformation. In a sense, those are "customer needs left changing", but neither you nor otherr customers want those needs to be automatible.

deleted 2y ago

[deleted]