5 ms·

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system

According to the NDA with my company I can't reveal anything about the architecture beyond the fact that it is hosted locally on a homebuilt distributed system that randomly chooses from a pool of 120 residential IPs. We do have human emulation routines that helped avoid most detection, and that library is decoupled in such a way that we can edit behavior down to the individual site. Some sites are just so damn good and detecting us and I just don't get it.

nutjob2 7y ago
They can characterise the (browsing) behaviour of all their visitors, and then further characterise those who fall outside their "normal" thresholds. The outsiders that exhibit some sort of correlation (ie their characteristics are not independent of each other) are banned. Any quirks or patterns your systems have would be identifiable as "artificial", and even those that are randomised or seek to emulate humans will have features that are identifiable. An NDA is ineffective against machine learning. The countermeasure would be to have a bunch of humans use the websites in any way they want, totally undirected, then use the totality of that browsing to facilitate your scraping probabilistically. It would be less efficient, but very difficult to catch.
- Pinbenterjamin 7y ago
  That's the general direction I'd like to take. When we capture the inputs for the scrapers, I'd like to persist everything. Mouse jiggles, delays, idle time. I think it would definitely help advance the software.
  - UweSchmidt 7y ago
    In the grand scheme of things all of this is a wasteful process. Maybe you could direct your worklife towards other challenges that are more rewarding for society and equally profitable?
    - pault 7y ago
      I think that's unjustified and a little rude. OP is providing an automated service for publicly accessible data that isn't accessible for automation. If the sources are notified and they are operating within the confines of the law, this is no different than writing a search engine crawler.
    - formercoder 7y ago
      OP is being reasonably compensated for something that is perfectly legal.
    - dang 7y ago
      That crosses into personal attack. Please don't do that on Hacker News. We've had to ask you this before. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
underwater 7y ago
This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.
- abcpassword 7y ago
  He’s definitely personally liable. I was tasked with a similar thing at work and refused. Prison time isn’t worth the paycheck.
- Pinbenterjamin 7y ago
  We have an enormous legal team that communicates constantly with end points to ensure they are aware of our scraping. And as I said in another comment, we store no results other than what is already available to anyone else using the web. We've had this division for many many years, and before my time we paid another company to do this. There's no legal issues.
  - underwater 7y ago
    Your legal teamn is in contact with them, but their security is actively trying to block you? That doesn't make sense. Computer security laws are very broad. It doesn't matter if it's just a website that the public can access. If you're accessing it in a matter that they don't want AND you're aware of that, then I struggle to see how your lawyers can justify it. > Computer hacking is broadly defined as intentionally accesses a computer without authorization or exceeds authorized access. https://definitions.uslegal.com/c/computer-hacking/ https://definitions.uslegal.com/c/computer-hacking/ Hiding your user agent because you know they don't want automated retrieval of information is "without authorisation".
- Havoc 7y ago
  >Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Don't think connecting a computer to a private network to suck up subscriber data is comparable to scraping publicly accessible internet content.
arpa 7y ago
A pool of 120 residential ips is way too small - patterns are more emergent. Go for thousands, even better, hundreds of thousands. Outsource the residential proxy system to luminati or oxylabs.

nutjob2 7y ago

They can characterise the (browsing) behaviour of all their visitors, and then further characterise those who fall outside their "normal" thresholds. The outsiders that exhibit some sort of correlation (ie their characteristics are not independent of each other) are banned. Any quirks or patterns your systems have would be identifiable as "artificial", and even those that are randomised or seek to emulate humans will have features that are identifiable. An NDA is ineffective against machine learning. The countermeasure would be to have a bunch of humans use the websites in any way they want, totally undirected, then use the totality of that browsing to facilitate your scraping probabilistically. It would be less efficient, but very difficult to catch.

Pinbenterjamin 7y ago
That's the general direction I'd like to take. When we capture the inputs for the scrapers, I'd like to persist everything. Mouse jiggles, delays, idle time. I think it would definitely help advance the software.
- UweSchmidt 7y ago
  In the grand scheme of things all of this is a wasteful process. Maybe you could direct your worklife towards other challenges that are more rewarding for society and equally profitable?
  - pault 7y ago
    I think that's unjustified and a little rude. OP is providing an automated service for publicly accessible data that isn't accessible for automation. If the sources are notified and they are operating within the confines of the law, this is no different than writing a search engine crawler.
  - formercoder 7y ago
    OP is being reasonably compensated for something that is perfectly legal.
  - dang 7y ago
    That crosses into personal attack. Please don't do that on Hacker News. We've had to ask you this before. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

underwater 7y ago

This sounds, at best, ethically dubious and at worst illegal. Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Given that your run this division there is a good chance you are personally liable.

abcpassword 7y ago
He’s definitely personally liable. I was tasked with a similar thing at work and refused. Prison time isn’t worth the paycheck.
Pinbenterjamin 7y ago
We have an enormous legal team that communicates constantly with end points to ensure they are aware of our scraping. And as I said in another comment, we store no results other than what is already available to anyone else using the web. We've had this division for many many years, and before my time we paid another company to do this. There's no legal issues.
- underwater 7y ago
  Your legal teamn is in contact with them, but their security is actively trying to block you? That doesn't make sense. Computer security laws are very broad. It doesn't matter if it's just a website that the public can access. If you're accessing it in a matter that they don't want AND you're aware of that, then I struggle to see how your lawyers can justify it. > Computer hacking is broadly defined as intentionally accesses a computer without authorization or exceeds authorized access. https://definitions.uslegal.com/c/computer-hacking/ https://definitions.uslegal.com/c/computer-hacking/ Hiding your user agent because you know they don't want automated retrieval of information is "without authorisation".
Havoc 7y ago
>Aaron Swartz was arrested and charged under hacking laws for doing exactly what you're describing. Don't think connecting a computer to a private network to suck up subscriber data is comparable to scraping publicly accessible internet content.

arpa 7y ago

A pool of 120 residential ips is way too small - patterns are more emergent. Go for thousands, even better, hundreds of thousands. Outsource the residential proxy system to luminati or oxylabs.