4 ms·

Why can’t we?

• gentleman11 5y ago
  What percent of visits are bots and how do you detect them serverside? That’s the only part I’m unsure of
  • yjftsjthsd-h 5y ago
    Can you detect bots client side? I sorta assumed they used headless Chrome and looked like real users nowadays, although I guess with effort you can probably fingerprint?
    • thayne 5y ago
      You can if you track all of the user's online activity and use machine learning to determine if the behavior seems genuine. That's how invisible captcha works, I'd be surprised if google analytics didn't do something similar. Of course, that does mean you have to track all the user's online activity....
      • NmAmDa 5y ago
        What about GDPR potential violation?
        • adrianN 5y ago
          GDPR has exceptions for fraud detection.
          • Nextgrid 5y ago
            A bot does not immediately imply fraud. If you're processing personal data to detect bots to prevent let's say card fraud, fair enough. If you're processing personal data to detect bots for analytics purposes, this might not fly. Of course, this assumes someone actually gives a shit about enforcing the GDPR, which is currently not the case.
            • adrianN 5y ago
              I have literally worked on bot detection and the high powered company lawyers told me that it's okay for my code to collect and analyze PII for these purposes because GDPR has these exceptions. I'm not a lawyer myself though, so that might have been wrong.
    • jerven 5y ago
      For one of the sites I work with. Headless chrome defenitly inflates the GA user numbers. We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping.
      • yjftsjthsd-h 5y ago
        > We can spot this because they tend to come from big cloud data centre locations when doing ip to geo mapping. Of course, you can do that with server side logs, so in this case client side is less accurate:)
    • NavinF 5y ago
      The default build of chromedriver is pretty easy to detect client side because it injects js variables with known names. Most bots don't really care if they're detected and excluded from analytics. Unfortunately a few sites unfairly throttled me to 1 request/hour despite the cost of my requests being fuck-all. So a couple of years ago I had to randomize those variable names, distribute requests over 64 IPs, and screw up all their analytics numbers in the process. I hope it was worth saving $0.01 for them.
    • adrianN 5y ago
      Most bots are pretty stupid, so it's easy to detect them. Bot detection need not be 100% accurate. It's an arms race with the goal to make writing bots financially unattractive.
  • digitalengineer 5y ago
    This can help: https://www.darwin.so/features/bots https://www.darwin.so/features/bots
  • inetknght 5y ago
    > What percent of visits are bots It doesn't matter. > how do you detect them serverside? Instead of trying to detect whether something isn't human you should be trying to detect whether something is malicious.
• darig 5y ago
  because developers don't know how to build anything anymore. they paint themselves into smaller and smaller focuses relying on someone else to provide them with everything they need. no more web developers... now we have front-end and back-end all fighting over who needs to do the architecture and build the necessary tooling.