10 ms·

Browser automation is an arms race between fraud detection and criminals, with users and some legitimate use cases (cough science) caught in between. I find it

Browser automation is an arms race between fraud detection and criminals, with users and some legitimate use cases (cough science) caught in between. I find it's very useful to think of the problem not as governed by technical possibility, but rather by costs. Good spoofing detection does not prevent fraud, it increases its costs to fraudsters (hoping to render fraud uneconomical). Bad spoofing detection harms legitimate users and is inefficient against capable actors. All of this gets much more interesting and complex when we consider privacy (and privacy compliance) to be part of the puzzle - now fraud prevention incurs a cost to the defender as well. Finally, it's important to consider asymmetric risk/benefit counts. Are you defending against a single major heist, or against billions of tiny events (click fraud)? The trade-off will look different.

gavanm 5y ago
Some other genuine legitimate use cases are: Monitoring (Synthetic Users) Performance Testing (though usually in a separate dedicated environment)
- uniqueuid 5y ago
  Right, I guess it's important to have some automated way of testing the billions of configuration possibilities and network/device conditions that you could expect, because they are impossible to systematically test in human-driven QA. How else would you test your SPA against millions of different content-modifying browser plugins, for example.
  - oauea 5y ago
    > How else would you test your SPA against millions of different content-modifying browser plugins, for example. You don't.
- evgen 5y ago
  If it is your site then you will be able to bypass the fraud detection, and if it is not your site then the site owner probably does not consider these legitimate use cases to be concerned about.
Nextgrid 5y ago
Another legitimate usage use-case is to wade through bullshit, dark patterns, ads & co in an automated way. See Invidious, Nitter, Teddit, etc.
- uniqueuid 5y ago
  While I love those attempts (kind of like ublock origin on a server), you can easily see how they are stuck between a rock and a hard place. They have all the vulnerability of fraudsters, and none of the benefits of authentic users. That's why most of them seem to be perpetually banned. [edit] Now, if someone would build an onion router that ran on people's end devices and served as a proxy network for these front ends, that would be interesting ... but would probably end up getting lots of people blocked, so please don't take it seriously.
  - Nextgrid 5y ago
    > but would probably end up getting lots of people blocked These scum companies rely on "engagement", so they can only block people if it's a minority. If _everyone_ participates, they have no choice but to accept it (or maybe change their product so there's no incentive to run these unofficial front-ends in the first place).
kfichter 5y ago
> Bad spoofing detection harms legitimate users and is inefficient against capable actors. On a somewhat related note: I used to write sneaker buying bots and captcha was one of the best things to happen to the bot industry. Captcha was easily "bypassed" by services that had humans sitting at computers generating tokens. Recaptcha tokens are valid for two minutes by default, so you'd be able to generate tokens up to two minutes in advance and have dozens of tokens available when the product in question became available for purchase. Real buyers had to wait for drop and then spend 15s+ filling out the captcha. Bots would take most of the stock in less than a second. I always felt like this was a fantastic example of a bot prevention mechanism that actually actively harmed "legitimate" users.
- paulpauper 5y ago
  If the landing page generates unique captachaa that are random how does this help unless the hashes are valid globally
  - rapsey 5y ago
    Recaptcha sends the same image to multiple browsers at the same time and it gets solved by consensus I think.
    - colejohnson66 5y ago
      You’re thinking of reCaptcha “v2”. That was when it was about digitizing books. One distorted photo of a word had a known spelling, and the other didn’t. If you spelled the known one right, it let you through. If enough people spelled the unknown one the same way, it would “learn” how to OCR that word (of sorts). If you had time to spare/waste, you could spell the first one right, and put garbage for the second one. About half the time, it would let you through. Not sure how reCaptcha “v3” (the photos of streets one) works. It could work the same way, but just with more known and unknown (unclassified) pictures? It’s also a lot more harsh if you get it wrong: artificial delays, switching to the “click until no more remain” mode (with more delays), and more. Sometimes, it just refuses to let you through no matter what: https://youtu.be/zGW7TRtcDeQ https://youtu.be/zGW7TRtcDeQ It’s extremely frustrating.
Siira 5y ago
Scraping user-generated data should be made a legal right. It’s not like the companies paid for the data they collected from users, and the whole thing is anti-competitive as well, and hurts the economy.
- judge2020 5y ago
  UGC can range from a Facebook post about tacos to intimate details about one’s private life or medical history. Blanket statements like yours do nothing to help the conversation.
  - luffapi 5y ago
    You can get only scrape data you are allowed to access. If your eyes can see it, you should be able to scrape it. The analog hole is always there.
    - EMM_386 5y ago
      This gets very tricky when you think about JavaScript. What if you only pop up important information via an onBlur event in a dynamically created div element that is overlayed on the page, only after asynchronously querying a REST endpoint to get information from a database? How would a crawler even know that this element is shown when you leave a specific input field? How would it know to wait a second while the database is getting the data? Like it or not, we're well past the days of complete HTML documents being served for a request.
luffapi 5y ago
I don’t understand why programmatic access is being equated with fraud. Device identification is never a benefit for end users. Lack of privacy, waste of resources, hostile ux… we’re basically talking about DRM.
- tovej 5y ago
  Yeah, HTTP should be a protocol that users can use freely (as long as they don't use it for Denial of Service). I don't see why I shouldn't be able to fetch a website, scrape it for information and reassemble that information in the way I want it to be presented.
  - chii 5y ago
    > reassemble that information in the way I want it to be presented site owners want the advertising that was embedded to be seen and thus make revenue from it.
    - megous 5y ago
      Not all of them, some just want to sell stuff, but make it positively terrible to search for things. Some site owners provide you service, like banks, or chat services like Slack, or Jira, but the user interface is again a turd. Some just publish info on a blog, but you can expect it to go away, so why not save it for later. Slack limits the amount of searchable messages to 10k, so if you save them regularly to local DB, you can search without limits. All kinds of owners don't sell/show ads, or care about scraping.
- mike_hearn 5y ago
  It's because programmatic access on websites with anything social, or with user generated content, is very strongly correlated with spam, and spam is quite often linked with fraud of various kinds. Like 99%+ of the time if something is posted by a bot outside of any officially provided APIs, it will be unwanted by other users.
  - 5y ago
emodendroket 5y ago
It seems to me that one way through this challenge is just automating the browser itself. Internet Explorer was particularly friendly to this but since it's out of support maybe browser extensions would be the way to go.
captainmuon 5y ago
I think we are trying to solve a social problem with a technical solution. Can we make an unbeatable spoof detector? I think we probably could, but we shouldn't. Locking down things takes the fun out of them. Ideally, you could get rid of every captcha and just have an open API for everything. Somebody wants to create lots of email addresses to send spam? Remove the incentive to send spam. Somebody posts thousands of manipulative tweets? Maybe the "feed" model of social media is wrong. Don't assume every random account is equally important. Better just have a simple social network where you can connect with your friends. Or even better, design your society such that it's decisionmaking is more resilient and can't be gamed. The really hard but worthwile problem is not in tech, it is how we evolve our society along with the tech.
- jpttsn 5y ago
  People hurt each other? Just remove their reasons. Not enough food? Just stop being hungry. But how?
  - asdadsdad 5y ago
    lol
- KirillPanov 5y ago
  This was the original attitude of the Old School Internet. Unfortunately a new generation has come along that believes advertising-supported business models are their birthright.
  - zahrc 5y ago
    I’d honestly rather pay a (higher) fee to a service I like and is transparent. A service where my money is paying for the uptime and development and I give consent, and not where some “evil” AI is tracking my every step and then has the audacity to recommend me ads of things totally irrelevant to me. I respect that services have running costs and I’m happy to cover my usage, but I have to understand that the average consumer doesn’t give a crap, or is simply ignorant and doesn’t know.
navierstokes 5y ago
The recent Microsoft signed rootkit has specific requests for collecting CPU-ID. That says something about the state of device fingerprinting, but I'm not sure exactly what that is. https://www.bleepingcomputer.com/news/security/microsoft-admits-to-signing-rootkit-malware-in-supply-chain-fiasco/ https://www.bleepingcomputer.com/news/security/microsoft-adm...
- shakkhar 5y ago
  CPUID is just a bunch of flags used to discover the capabilities of your CPU: https://en.wikipedia.org/wiki/CPUID https://en.wikipedia.org/wiki/CPUID

gavanm 5y ago

Some other genuine legitimate use cases are: Monitoring (Synthetic Users) Performance Testing (though usually in a separate dedicated environment)

uniqueuid 5y ago
Right, I guess it's important to have some automated way of testing the billions of configuration possibilities and network/device conditions that you could expect, because they are impossible to systematically test in human-driven QA. How else would you test your SPA against millions of different content-modifying browser plugins, for example.
- oauea 5y ago
  > How else would you test your SPA against millions of different content-modifying browser plugins, for example. You don't.
evgen 5y ago
If it is your site then you will be able to bypass the fraud detection, and if it is not your site then the site owner probably does not consider these legitimate use cases to be concerned about.

Nextgrid 5y ago

Another legitimate usage use-case is to wade through bullshit, dark patterns, ads & co in an automated way. See Invidious, Nitter, Teddit, etc.

uniqueuid 5y ago
While I love those attempts (kind of like ublock origin on a server), you can easily see how they are stuck between a rock and a hard place. They have all the vulnerability of fraudsters, and none of the benefits of authentic users. That's why most of them seem to be perpetually banned. [edit] Now, if someone would build an onion router that ran on people's end devices and served as a proxy network for these front ends, that would be interesting ... but would probably end up getting lots of people blocked, so please don't take it seriously.
- Nextgrid 5y ago
  > but would probably end up getting lots of people blocked These scum companies rely on "engagement", so they can only block people if it's a minority. If _everyone_ participates, they have no choice but to accept it (or maybe change their product so there's no incentive to run these unofficial front-ends in the first place).

kfichter 5y ago

> Bad spoofing detection harms legitimate users and is inefficient against capable actors. On a somewhat related note: I used to write sneaker buying bots and captcha was one of the best things to happen to the bot industry. Captcha was easily "bypassed" by services that had humans sitting at computers generating tokens. Recaptcha tokens are valid for two minutes by default, so you'd be able to generate tokens up to two minutes in advance and have dozens of tokens available when the product in question became available for purchase. Real buyers had to wait for drop and then spend 15s+ filling out the captcha. Bots would take most of the stock in less than a second. I always felt like this was a fantastic example of a bot prevention mechanism that actually actively harmed "legitimate" users.

paulpauper 5y ago
If the landing page generates unique captachaa that are random how does this help unless the hashes are valid globally
- rapsey 5y ago
  Recaptcha sends the same image to multiple browsers at the same time and it gets solved by consensus I think.
  - colejohnson66 5y ago
    You’re thinking of reCaptcha “v2”. That was when it was about digitizing books. One distorted photo of a word had a known spelling, and the other didn’t. If you spelled the known one right, it let you through. If enough people spelled the unknown one the same way, it would “learn” how to OCR that word (of sorts). If you had time to spare/waste, you could spell the first one right, and put garbage for the second one. About half the time, it would let you through. Not sure how reCaptcha “v3” (the photos of streets one) works. It could work the same way, but just with more known and unknown (unclassified) pictures? It’s also a lot more harsh if you get it wrong: artificial delays, switching to the “click until no more remain” mode (with more delays), and more. Sometimes, it just refuses to let you through no matter what: https://youtu.be/zGW7TRtcDeQ https://youtu.be/zGW7TRtcDeQ It’s extremely frustrating.

Siira 5y ago

Scraping user-generated data should be made a legal right. It’s not like the companies paid for the data they collected from users, and the whole thing is anti-competitive as well, and hurts the economy.

judge2020 5y ago
UGC can range from a Facebook post about tacos to intimate details about one’s private life or medical history. Blanket statements like yours do nothing to help the conversation.
- luffapi 5y ago
  You can get only scrape data you are allowed to access. If your eyes can see it, you should be able to scrape it. The analog hole is always there.
  - EMM_386 5y ago
    This gets very tricky when you think about JavaScript. What if you only pop up important information via an onBlur event in a dynamically created div element that is overlayed on the page, only after asynchronously querying a REST endpoint to get information from a database? How would a crawler even know that this element is shown when you leave a specific input field? How would it know to wait a second while the database is getting the data? Like it or not, we're well past the days of complete HTML documents being served for a request.

luffapi 5y ago

I don’t understand why programmatic access is being equated with fraud. Device identification is never a benefit for end users. Lack of privacy, waste of resources, hostile ux… we’re basically talking about DRM.

tovej 5y ago
Yeah, HTTP should be a protocol that users can use freely (as long as they don't use it for Denial of Service). I don't see why I shouldn't be able to fetch a website, scrape it for information and reassemble that information in the way I want it to be presented.
- chii 5y ago
  > reassemble that information in the way I want it to be presented site owners want the advertising that was embedded to be seen and thus make revenue from it.
  - megous 5y ago
    Not all of them, some just want to sell stuff, but make it positively terrible to search for things. Some site owners provide you service, like banks, or chat services like Slack, or Jira, but the user interface is again a turd. Some just publish info on a blog, but you can expect it to go away, so why not save it for later. Slack limits the amount of searchable messages to 10k, so if you save them regularly to local DB, you can search without limits. All kinds of owners don't sell/show ads, or care about scraping.
mike_hearn 5y ago
It's because programmatic access on websites with anything social, or with user generated content, is very strongly correlated with spam, and spam is quite often linked with fraud of various kinds. Like 99%+ of the time if something is posted by a bot outside of any officially provided APIs, it will be unwanted by other users.
- 5y ago

emodendroket 5y ago

It seems to me that one way through this challenge is just automating the browser itself. Internet Explorer was particularly friendly to this but since it's out of support maybe browser extensions would be the way to go.

captainmuon 5y ago

I think we are trying to solve a social problem with a technical solution. Can we make an unbeatable spoof detector? I think we probably could, but we shouldn't. Locking down things takes the fun out of them. Ideally, you could get rid of every captcha and just have an open API for everything. Somebody wants to create lots of email addresses to send spam? Remove the incentive to send spam. Somebody posts thousands of manipulative tweets? Maybe the "feed" model of social media is wrong. Don't assume every random account is equally important. Better just have a simple social network where you can connect with your friends. Or even better, design your society such that it's decisionmaking is more resilient and can't be gamed. The really hard but worthwile problem is not in tech, it is how we evolve our society along with the tech.

jpttsn 5y ago
People hurt each other? Just remove their reasons. Not enough food? Just stop being hungry. But how?
- asdadsdad 5y ago
  lol
KirillPanov 5y ago
This was the original attitude of the Old School Internet. Unfortunately a new generation has come along that believes advertising-supported business models are their birthright.
- zahrc 5y ago
  I’d honestly rather pay a (higher) fee to a service I like and is transparent. A service where my money is paying for the uptime and development and I give consent, and not where some “evil” AI is tracking my every step and then has the audacity to recommend me ads of things totally irrelevant to me. I respect that services have running costs and I’m happy to cover my usage, but I have to understand that the average consumer doesn’t give a crap, or is simply ignorant and doesn’t know.

navierstokes 5y ago

The recent Microsoft signed rootkit has specific requests for collecting CPU-ID. That says something about the state of device fingerprinting, but I'm not sure exactly what that is. https://www.bleepingcomputer.com/news/security/microsoft-admits-to-signing-rootkit-malware-in-supply-chain-fiasco/ https://www.bleepingcomputer.com/news/security/microsoft-adm...

shakkhar 5y ago
CPUID is just a bunch of flags used to discover the capabilities of your CPU: https://en.wikipedia.org/wiki/CPUID https://en.wikipedia.org/wiki/CPUID