5 ms·

I used Tor for surveillance. But an appropriate kind, IMHO. I used Tor as a small part of one of the capabilities of a supply chain integrity startup. I buil

by neilv 1y ago

I used Tor for surveillance. But an appropriate kind, IMHO. I used Tor as a small part of one of the capabilities of a supply chain integrity startup. I built a fancy scraper/crawler to discreetly monitor a major international marketplace (mainstream, not darknet), including selecting appropriate Tor exit nodes for each regional site, to try to ensure that we were seeing the same site content that people from those regions were seeing. Tor somehow worked perfectly for those needs. So my only big concern was making sure everyone in the startup knew not to go bragging about this unusually good data we had. Since we were one C&D letter away from not being able to get the data at all. (Unfortunately, this had to be a little adversarial with the marketplace, not done as a data-sharing partnership, since the marketplace benefited from a cut of all the counterfeit and graymarket sales that we were trying to fight. But I made sure the scraper was gentle yet effective, both to not be a jerk, and also to not attract attention.) (I can talk about it now, since the startup ran out of runway during Covid investor skittishness.)

RGamma 1y ago
> selecting appropriate Tor exit nodes for each regional site So, a proxy? Onion routing doesn't really play a role for this use case.
- neilv 1y ago
  > So, a proxy? Onion routing doesn't really play a role for this use case. The onion routing obscured our identity from the "proxy" exit nodes. Separately, Tor was also a convenient way to get a lot of arbitrary country-specific "proxies", without dealing with the sometimes sketchy businesses that are behind residential IP proxies. (Counterfeiting/graymarket operations can be organized crime. I'd rather just fire up Tor, and trust math a little, than to try to vet the legitimacy and intentions of a residential IP broker.)
  - sidewndr46 1y ago
    Why would you need to obscure your identity from the exit nodes?
    - qualeed 1y ago
      So that the exit node can't go to the site they were scraping and say "this is the person scraping your site".
      - radicaldreamer 1y ago
        But you'd have relays in between, there's no way an exit node would know who is scraping...
        qualeed 1y ago
        Right, but the question was "why would you need to obscure your identity from the exit nodes", in the context of why the person chose Tor vs. a simple proxy.
        radicaldreamer 1y ago
        Ohh I see!
        chatmasta 1y ago
        Yes, but many scrapers configure their Torrc into single-relay mode (following tutorials and open source “rotating tor proxy” repos), in which case the exit relay can easily identify who is scraping which site.
    - chasil 1y ago
      The exit nodes are free to observe and exploit the traffic that they move. https://www.vice.com/en/article/badonion-honeypot-malicious-tor-exit-nodes/ https://www.vice.com/en/article/badonion-honeypot-malicious-...
  - wslh 1y ago
    The Tor exit nodes are public.
    - qualeed 1y ago
      They were concerned about the exit node identifying them, not the site identifying that a Tor exit node is connecting.
- trod1234 1y ago
  Honestly what he describes sounds like Raptor (Princeton Report, 2015)
  - neilv 1y ago
    How is this related to Princeton's Raptor, other than having the keywords "Tor" and "surveillance"? https://www.princeton.edu/~pmittal/publications/raptor-USENIX15.pdf https://www.princeton.edu/~pmittal/publications/raptor-USENI... (Strange coincidence: We also had different key tech with the codename of Raptor, but it had nothing to do with Tor nor Web scraping. It was for discreet smartphone-based field auditing of physical product, in global physical retail and other locations. The codename was the result of a great morale-boosting impromptu brainstorming session between engineering and marketing people ("can you help think of a cool codename for this..."), and the resulting name highly apt, at least for the movie velociraptors. I built it, and, until Covid disrupted our F500 customers and investors, I was looking forward to hiring engineers to do further work on something cool-sounding like "Raptor", rather than "internal-app" or whatever first came to mind when creating the Git repo. :)
    - trod1234 1y ago
      The major attack of concern described in the paper is the transparent early terminated encryption attack, and root trust signing that fall under effectively the same centralized hands at the AS level. Where an AS level entity MITMs all outbound connections from a region in automated fashion for collection, before that traffic ever makes it to TOR or its destination. It works for TOR, TLS, pretty much any protocol out there where key exchange or trust occurs; so long as the protocol is known and has distinct classifiable characteristics allowing computation to automatically do this. There have been instances where public certs issued by a CA with the same domain names, but are issued from a root CA that is other than the legitimate site's root CA which are used for attacks. CT logs don't stop this either. There is a lot of ephemeral content, and private information that can be both collected, and injected on a targeted basis if one has access to such junctions which the industry (Telecom) has proven time and again that they can't secure following basic practice; largely because mandates to backwards compatibility at the regulatory level. Social credit, where invisible factors people don't control force those same people into poverty through targeted denial of service (communications for job hunting/social contacts), zersetzung, etc; that all would be a breeze to set up without any external indicator, or remedy using that attack. What the target sees vs what everyone else sees would be quite different, and of course there would be people that gaslight and torture on top of it all (as a natural psychological defense mechanism of denial). Compromised communications under such type of attacks are madness inducing.
cedws 1y ago
What was the scraper gathering specifically?
- woadwarrior01 1y ago
  If I could wager a guess, it sounds like the startup was in the business of scraping Amazon.
  - neilv 1y ago
    No. And when people share info on HN, I don't like to see speculation in the comments about things they obviously intentionally didn't say (assuming that they seem to be speaking in good faith). That person, and other people who see the dynamic, presumably are less likely to share in the future.
    - keysdev 1y ago
      Thank you for pointing that out. Really appreciate you sharing. To the parent, please do not try to lure info out of people it is just not cool online or in real life when people obviously are being generic for a reason.
    - ribosometronome 1y ago
      I feel there is a level of irony in you being bothered about people interacting with content you've shared in a way you don't like when said content is a story about you interacting with other's content in a way they've explicitly put up barriers to try and stop you from doing that.
      - neilv 1y ago
        Who said the site put up barriers? I think you have a valid general question (and you'll note I said "appropriate kind, IMHO" at the top of the original comment, acknowledging others might disagree that it was appropriate), but I'd like to contrast two distinct situations: * A collegial forum, where people might go to share information, sometimes with discretion about what can and can't be said (or just comfort levels). * A large corporation that was profiting off of illegal businesses (e.g., contract-violating, IP-violating, defrauding buyers, possibly fencing), and we wanted to gather evidence of that on behalf of some of the harmed parties, to try to stop it. And we did that in a technologically gentle, non-disruptive way. And (as I mentioned in the original comment) we had a conscious policy to immediately cease if we were ever told to.
RobRivera 1y ago
HEH I'm letting my imagination fill in the color on the specifics here and I'm working up a little grin. A hat tip to you
cakealert 1y ago
This is not a good way to do this. Tor exit nodes are public and may be marked for special behavior by the marketplace you are surveying. There is no reason to believe you are getting good information this way. The right way to do this would be through a VPN/tor + Residential proxy to hide your intentions from everyone involved.
- neilv 1y ago
  > There is no reason to believe you are getting good information this way. Spot checks checked out. And it was a perfectly fine way to do it. You are correct that Tor exit nodes often get special handling (at the moment, including by Cloudflare, and by Google Recaptcha). And the idea of poisoning of data is starting to propagate, due do anti-AI-scraper sentiment.
  - tomschlick 1y ago
    Poisoning the data has been around for at least a decade now. I used to work for a very large pricing analytics company that would track product pricing for Fortune 500 manufacturers. We found on several occasions that some shady retailers would find the CIDR of the manufacturers corp networks and comply with the MAP policies on pricing if traffic came from them. Then when our bot went through with obviously generic AWS / Proxy ips we would see a much lower price that broke their agreement. That one was a fun realization for the manufacturers as to the level of shadiness some retailers would go through.
electroly 1y ago
Next time you find yourself in this situation, a $5 VPN subscription (Mullvad, etc.) gets you the same result without the IP being an obvious Tor exit node. Faster, too, in latency, bandwidth, and the time it takes to change locations. You only care about the VPN part for this, not the onion part.
- apaprocki 1y ago
  Worth mentioning a $5 cloud instance and installing Algo VPN on it gets you the same thing without having to trust a 3rd party VPN provider (only a generic VM provider such as AWS). It’s always worth minimizing companies you deal with if you already use AWS, GCP, etc.
  - rrgok 1y ago
    It is not. VPN Providers have server all over the world. How many cloud instances in the different part of the world you can buy with $5?
    - bamboozled 1y ago
      A lot of, a t3.micro is free, I guess Amazon know who you are and which IP you were using at a certain time though.
      - ciberado 1y ago
        Also, public cloud machines come with IPs of well known ranges. It may be easy to spot them if required. On the other side, this is the approach that I'm currently using, without any problem for my particular case.
    - apaprocki 1y ago
      AWS offers all the regions I’d care about, but I do concede many-exit scenarios are not what I was referring to. I’ve been more than satisfied with my personal use, but have never required that feature.
  - 1y ago
deleted 1y ago
[deleted]
globular-toast 1y ago
Is it that difficult for a business to colo in various locations? Not rhetorical, I've never thought about it.
- sulandor 1y ago
  exit node is not really a colo. though there are commercial residential-proxy services available

by neilv 1y ago

RGamma 1y ago

> selecting appropriate Tor exit nodes for each regional site So, a proxy? Onion routing doesn't really play a role for this use case.

neilv 1y ago
> So, a proxy? Onion routing doesn't really play a role for this use case. The onion routing obscured our identity from the "proxy" exit nodes. Separately, Tor was also a convenient way to get a lot of arbitrary country-specific "proxies", without dealing with the sometimes sketchy businesses that are behind residential IP proxies. (Counterfeiting/graymarket operations can be organized crime. I'd rather just fire up Tor, and trust math a little, than to try to vet the legitimacy and intentions of a residential IP broker.)
- sidewndr46 1y ago
  Why would you need to obscure your identity from the exit nodes?
  - qualeed 1y ago
    So that the exit node can't go to the site they were scraping and say "this is the person scraping your site".
    - radicaldreamer 1y ago
      But you'd have relays in between, there's no way an exit node would know who is scraping...
      - qualeed 1y ago
        Right, but the question was "why would you need to obscure your identity from the exit nodes", in the context of why the person chose Tor vs. a simple proxy.
        radicaldreamer 1y ago
        Ohh I see!
      - chatmasta 1y ago
        Yes, but many scrapers configure their Torrc into single-relay mode (following tutorials and open source “rotating tor proxy” repos), in which case the exit relay can easily identify who is scraping which site.
  - chasil 1y ago
    The exit nodes are free to observe and exploit the traffic that they move. https://www.vice.com/en/article/badonion-honeypot-malicious-tor-exit-nodes/ https://www.vice.com/en/article/badonion-honeypot-malicious-...
- wslh 1y ago
  The Tor exit nodes are public.
  - qualeed 1y ago
    They were concerned about the exit node identifying them, not the site identifying that a Tor exit node is connecting.
trod1234 1y ago
Honestly what he describes sounds like Raptor (Princeton Report, 2015)
- neilv 1y ago
  How is this related to Princeton's Raptor, other than having the keywords "Tor" and "surveillance"? https://www.princeton.edu/~pmittal/publications/raptor-USENIX15.pdf https://www.princeton.edu/~pmittal/publications/raptor-USENI... (Strange coincidence: We also had different key tech with the codename of Raptor, but it had nothing to do with Tor nor Web scraping. It was for discreet smartphone-based field auditing of physical product, in global physical retail and other locations. The codename was the result of a great morale-boosting impromptu brainstorming session between engineering and marketing people ("can you help think of a cool codename for this..."), and the resulting name highly apt, at least for the movie velociraptors. I built it, and, until Covid disrupted our F500 customers and investors, I was looking forward to hiring engineers to do further work on something cool-sounding like "Raptor", rather than "internal-app" or whatever first came to mind when creating the Git repo. :)
  - trod1234 1y ago
    The major attack of concern described in the paper is the transparent early terminated encryption attack, and root trust signing that fall under effectively the same centralized hands at the AS level. Where an AS level entity MITMs all outbound connections from a region in automated fashion for collection, before that traffic ever makes it to TOR or its destination. It works for TOR, TLS, pretty much any protocol out there where key exchange or trust occurs; so long as the protocol is known and has distinct classifiable characteristics allowing computation to automatically do this. There have been instances where public certs issued by a CA with the same domain names, but are issued from a root CA that is other than the legitimate site's root CA which are used for attacks. CT logs don't stop this either. There is a lot of ephemeral content, and private information that can be both collected, and injected on a targeted basis if one has access to such junctions which the industry (Telecom) has proven time and again that they can't secure following basic practice; largely because mandates to backwards compatibility at the regulatory level. Social credit, where invisible factors people don't control force those same people into poverty through targeted denial of service (communications for job hunting/social contacts), zersetzung, etc; that all would be a breeze to set up without any external indicator, or remedy using that attack. What the target sees vs what everyone else sees would be quite different, and of course there would be people that gaslight and torture on top of it all (as a natural psychological defense mechanism of denial). Compromised communications under such type of attacks are madness inducing.

cedws 1y ago

What was the scraper gathering specifically?

woadwarrior01 1y ago
If I could wager a guess, it sounds like the startup was in the business of scraping Amazon.
- neilv 1y ago
  No. And when people share info on HN, I don't like to see speculation in the comments about things they obviously intentionally didn't say (assuming that they seem to be speaking in good faith). That person, and other people who see the dynamic, presumably are less likely to share in the future.
  - keysdev 1y ago
    Thank you for pointing that out. Really appreciate you sharing. To the parent, please do not try to lure info out of people it is just not cool online or in real life when people obviously are being generic for a reason.
  - ribosometronome 1y ago
    I feel there is a level of irony in you being bothered about people interacting with content you've shared in a way you don't like when said content is a story about you interacting with other's content in a way they've explicitly put up barriers to try and stop you from doing that.
    - neilv 1y ago
      Who said the site put up barriers? I think you have a valid general question (and you'll note I said "appropriate kind, IMHO" at the top of the original comment, acknowledging others might disagree that it was appropriate), but I'd like to contrast two distinct situations: * A collegial forum, where people might go to share information, sometimes with discretion about what can and can't be said (or just comfort levels). * A large corporation that was profiting off of illegal businesses (e.g., contract-violating, IP-violating, defrauding buyers, possibly fencing), and we wanted to gather evidence of that on behalf of some of the harmed parties, to try to stop it. And we did that in a technologically gentle, non-disruptive way. And (as I mentioned in the original comment) we had a conscious policy to immediately cease if we were ever told to.

RobRivera 1y ago

HEH I'm letting my imagination fill in the color on the specifics here and I'm working up a little grin. A hat tip to you

cakealert 1y ago

This is not a good way to do this. Tor exit nodes are public and may be marked for special behavior by the marketplace you are surveying. There is no reason to believe you are getting good information this way. The right way to do this would be through a VPN/tor + Residential proxy to hide your intentions from everyone involved.

neilv 1y ago
> There is no reason to believe you are getting good information this way. Spot checks checked out. And it was a perfectly fine way to do it. You are correct that Tor exit nodes often get special handling (at the moment, including by Cloudflare, and by Google Recaptcha). And the idea of poisoning of data is starting to propagate, due do anti-AI-scraper sentiment.
- tomschlick 1y ago
  Poisoning the data has been around for at least a decade now. I used to work for a very large pricing analytics company that would track product pricing for Fortune 500 manufacturers. We found on several occasions that some shady retailers would find the CIDR of the manufacturers corp networks and comply with the MAP policies on pricing if traffic came from them. Then when our bot went through with obviously generic AWS / Proxy ips we would see a much lower price that broke their agreement. That one was a fun realization for the manufacturers as to the level of shadiness some retailers would go through.

electroly 1y ago

Next time you find yourself in this situation, a $5 VPN subscription (Mullvad, etc.) gets you the same result without the IP being an obvious Tor exit node. Faster, too, in latency, bandwidth, and the time it takes to change locations. You only care about the VPN part for this, not the onion part.

apaprocki 1y ago
Worth mentioning a $5 cloud instance and installing Algo VPN on it gets you the same thing without having to trust a 3rd party VPN provider (only a generic VM provider such as AWS). It’s always worth minimizing companies you deal with if you already use AWS, GCP, etc.
- rrgok 1y ago
  It is not. VPN Providers have server all over the world. How many cloud instances in the different part of the world you can buy with $5?
  - bamboozled 1y ago
    A lot of, a t3.micro is free, I guess Amazon know who you are and which IP you were using at a certain time though.
    - ciberado 1y ago
      Also, public cloud machines come with IPs of well known ranges. It may be easy to spot them if required. On the other side, this is the approach that I'm currently using, without any problem for my particular case.
  - apaprocki 1y ago
    AWS offers all the regions I’d care about, but I do concede many-exit scenarios are not what I was referring to. I’ve been more than satisfied with my personal use, but have never required that feature.
- 1y ago

deleted 1y ago

[deleted]

globular-toast 1y ago

Is it that difficult for a business to colo in various locations? Not rhetorical, I've never thought about it.

sulandor 1y ago
exit node is not really a colo. though there are commercial residential-proxy services available