6 ms·

Log them in. Give them the data they want but make it false. They think it's working and mark those passwords as good but they don't know otherwise. Now they h

Log them in. Give them the data they want but make it false. They think it's working and mark those passwords as good but they don't know otherwise. Now they have no idea if they're really good or bad and they have to verify with multiple other attempts somewhere else.

bmpafa 8y ago
As someone often engaged in nonmalicious (but unwelcomed) scraping, this is one of the anti scraping measures I fear most
- Triesault 8y ago
  Out of curiosity, what kind of "nonmalicious but unwelcomed" scraping?
  - scrollaway 8y ago
    Not GP but scraping data for personal projects off companies overprotective of their data is probably the most common example.
    - Figs 8y ago
      Trying to get your entire reddit history, for example, is an obnoxiously difficult thing to do. It's impossible to get more than 1000 comments with the officially supported mechanism, leading people to do all kinds of strange workarounds to try to find older comments. As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more.
      - mathgeek 8y ago
        > As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more. I'd much rather they let you get your own history out and then make it impossible to get said history for anyone else (unless that person chooses to allow it).
      - toomuchtodo 8y ago
        Curious if Reddit will have to provide an easy mechanism to retrieve all of your comments now as part of GDPR.
        maxyme 8y ago
        Check the announcement post. In a comment one of the admins said they will eventually but for now it's email only for residents of the EU. https://www.reddit.com/r/announcements/comments/8m2yr4/were_updating_our_user_agreement_and_privacy/dzkdmbs/ https://www.reddit.com/r/announcements/comments/8m2yr4/were_...
        duxup 8y ago
        If everyone in the EU could email them to speed this up that would be great.
  - geezerjay 8y ago
    Not GP but scraping price data off some stores (particularly travel sites) tends to be unwelcomed by operators.
    - TeMPOraL 8y ago
      Myself, I did scrapping of a classified ads site once, to have notifications when a particular product I was looking for showed up for sale.
  - lostcolony 8y ago
    I've scraped content from sites with (and without) logins, for offline perusal when I was on a cruise.
  - bmpafa 8y ago
    (GP here) re: the second part, I guess I consider most scraping 'unwelcome' To the first part: I would characterize certain scraping, usually behind an authentication wall, as malicious--though admittedly that's not the right word. An example would be scraping Facebook profiles to build a marketing list. So, by 'non malicious,' i mostly mean 'publicly available data'
- probably_wrong 8y ago
  Google did this to me. I wrote a script to find the cheapest flight in matrix.itasoftware.com (now Google Flights), and ran it once per hour. Pretty soon they started giving me only the most expensive flights, as in, 7K+ for a 1K flight.
  - heyoni 8y ago
    That sounds like someone personally turned this on for you. Did you try to mask yourself and repeat?
    - probably_wrong 8y ago
      As funny as that would be, I doubt it was done personally against me - who would even take the time to block one person from getting one single cheap flight slightly cheaper? But I didn't try anything fancy. In retrospective, I probably should have turned on my VPN and see what happened.
tehlike 8y ago
This. Make it probabilistic, though, do not log them in every time.
- jakobegger 8y ago
  Better to make it deterministic, eg. display a fake success page whenever sha1('salt'+login+password) has two leading zeros. That way you can easily control the fake success rate, and you make sure that if the attacker realises they are being tricked, they can't just retry successful logins to double check, since they get the same result every time.
  - tehlike 8y ago
    Good idea. Doesn't have to be complex - even java hash code with modulus would work.
    - daniel_iversen 8y ago
      These suggestions sound good but the real challenge will be to reliably detect fake logins to not suddently and accidentally mess with your real users..
      - hlecuanda 8y ago
        Easy. Set it up just like a honeypot. Except that instead of being a sticky honeypot, it's poisoned honey they get. OP mentioned the attack is easily identified so legitimate traffic gets served correctly bad traffic gets "logged in" to the poisoned honeypot. 301 after login perhaps
deleted 8y ago
[deleted]
danols 8y ago
Giving scrapers fake but realistic data is my favorite way of dealing with them (as long as it does not consume too much resources). It will hopefully waste their time and discourage them from trying to find other ways.

bmpafa 8y ago

As someone often engaged in nonmalicious (but unwelcomed) scraping, this is one of the anti scraping measures I fear most

Triesault 8y ago
Out of curiosity, what kind of "nonmalicious but unwelcomed" scraping?
- scrollaway 8y ago
  Not GP but scraping data for personal projects off companies overprotective of their data is probably the most common example.
  - Figs 8y ago
    Trying to get your entire reddit history, for example, is an obnoxiously difficult thing to do. It's impossible to get more than 1000 comments with the officially supported mechanism, leading people to do all kinds of strange workarounds to try to find older comments. As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more.
    - mathgeek 8y ago
      > As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more. I'd much rather they let you get your own history out and then make it impossible to get said history for anyone else (unless that person chooses to allow it).
    - toomuchtodo 8y ago
      Curious if Reddit will have to provide an easy mechanism to retrieve all of your comments now as part of GDPR.
      - maxyme 8y ago
        Check the announcement post. In a comment one of the admins said they will eventually but for now it's email only for residents of the EU. https://www.reddit.com/r/announcements/comments/8m2yr4/were_updating_our_user_agreement_and_privacy/dzkdmbs/ https://www.reddit.com/r/announcements/comments/8m2yr4/were_...
        duxup 8y ago
        If everyone in the EU could email them to speed this up that would be great.
- geezerjay 8y ago
  Not GP but scraping price data off some stores (particularly travel sites) tends to be unwelcomed by operators.
  - TeMPOraL 8y ago
    Myself, I did scrapping of a classified ads site once, to have notifications when a particular product I was looking for showed up for sale.
- lostcolony 8y ago
  I've scraped content from sites with (and without) logins, for offline perusal when I was on a cruise.
- bmpafa 8y ago
  (GP here) re: the second part, I guess I consider most scraping 'unwelcome' To the first part: I would characterize certain scraping, usually behind an authentication wall, as malicious--though admittedly that's not the right word. An example would be scraping Facebook profiles to build a marketing list. So, by 'non malicious,' i mostly mean 'publicly available data'
probably_wrong 8y ago
Google did this to me. I wrote a script to find the cheapest flight in matrix.itasoftware.com (now Google Flights), and ran it once per hour. Pretty soon they started giving me only the most expensive flights, as in, 7K+ for a 1K flight.
- heyoni 8y ago
  That sounds like someone personally turned this on for you. Did you try to mask yourself and repeat?
  - probably_wrong 8y ago
    As funny as that would be, I doubt it was done personally against me - who would even take the time to block one person from getting one single cheap flight slightly cheaper? But I didn't try anything fancy. In retrospective, I probably should have turned on my VPN and see what happened.

tehlike 8y ago

This. Make it probabilistic, though, do not log them in every time.

jakobegger 8y ago
Better to make it deterministic, eg. display a fake success page whenever sha1('salt'+login+password) has two leading zeros. That way you can easily control the fake success rate, and you make sure that if the attacker realises they are being tricked, they can't just retry successful logins to double check, since they get the same result every time.
- tehlike 8y ago
  Good idea. Doesn't have to be complex - even java hash code with modulus would work.
  - daniel_iversen 8y ago
    These suggestions sound good but the real challenge will be to reliably detect fake logins to not suddently and accidentally mess with your real users..
    - hlecuanda 8y ago
      Easy. Set it up just like a honeypot. Except that instead of being a sticky honeypot, it's poisoned honey they get. OP mentioned the attack is easily identified so legitimate traffic gets served correctly bad traffic gets "logged in" to the poisoned honeypot. 301 after login perhaps

deleted 8y ago

[deleted]

danols 8y ago

Giving scrapers fake but realistic data is my favorite way of dealing with them (as long as it does not consume too much resources). It will hopefully waste their time and discourage them from trying to find other ways.