4 ms·

When I got down to Wizz Air's statement about "bugs in adblockers" making browsers "act unexpectedly" thereby triggering the robot detection code, I was reminde

When I got down to Wizz Air's statement about "bugs in adblockers" making browsers "act unexpectedly" thereby triggering the robot detection code, I was reminded of the robot detection functionality in a very common enterprise WAF middle-box that injects background JavaScript on the page to detect bots. The code supposedly produced no user-visible change but would participate in some SOAP challenge/response fluff. We ended up never deploying it because the false positive rate was absurdly high- on the order of 38% or so- with no tuning options available (short of falling back to a captcha). Having said that, I'd expect that this is a very common practice. I also suspect that blaming the ad blockers for lazy middle-box usage (if indeed that's what this particular case proves to be) is _not_ going to age well.

jacquesm 4y ago
That's pretty close to an admission of incompetence.
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]
unethical_ban 4y ago
You might say the WAF was Imperva-ious. I remember an instance where Lowe's website was broken on the corporate network. Our proxy re-arranged the order of HTTP content headers from the site, and Akamai took it as malicious behavior.
- ninefathom 4y ago
  Not quite, but I've used that one before as well. I suspect many of them offer similar(ly naïve) functionality.
- capitainenemo 4y ago
  Lowes website has been 100% broken for me ever since I enabled Resist Fingerprinting in Firefox. I can load exactly one page, but on any navigation or refresh I get: ===== Access Denied You don't have permission to access "http://www.lowes.com/ http://www.lowes.com/" on this server. Reference #18.cc69dc17.1661724957.fe4ef4 ==== Result, unless I use the profile with fingerprinting enabled, I just have to buy elsewhere. Drupal.org triggers "prove you're not a robot" every few page navigations with Resist Fingerprinting enabled. Walmart.com too. Fedex package tracking errors (seemingly due to the API server refusing the connection) if resist fingerprinting is enabled. Amusingly if you use the website help bot and say "track XXXXX" that does work to get some basic information.
  - gruez 4y ago
    I have RFP enabled and it works fine for me. I did get the "Access Denied" error you mentioned on my first try, but after switching VPN servers it worked fine.
    - capitainenemo 4y ago
      On Lowes.com? Retest in a clean profile. Seems that once they trust you you are ok for a while, at least from a friend's test, who was able to reproduce in a clean profile. But maybe it is IP linked and takes a little bit to accumulate. Did you just enable privacy.resistFingerprinting recently? Also. Doublecheck that it is enabled. Also, I'm using Nightly firefox. It may be the resist fingerprinting is more robust there. BTW, this isn't using a VPN or anything that might seem suspicious. Just my bog standard US broadband.
      - gruez 4y ago
        >On Lowes.com? Retest in a clean profile. I tested on a fresh container so for all intents and purposes it's a "clean profile". > Did you just enable privacy.resistFingerprinting recently? No, but it shouldn't matter given that I was using a fresh container and VPN. >Also. Doublecheck that it is enabled. Also, I'm using Nightly firefox. It may be the resist fingerprinting is more robust there. It's definitely enabled. I'm not using nightly though.
squeaky-clean 4y ago
I used to work for an airfare marketing company. We would get our 3rd party scripts onto an airlines booking engine to be able to run our own analytics and gather data for ads. We'd mainly collect things like prices, number of available seats, etc, because it turns out airlines can't really give you those answers through an API without it costing too much, so we piggybacked on real customer searches. Almost no one in the office ran adblockers, which was weird to me. When our analytics traffic dropped by like 50% one day, I was the only one to notice that our domain made it onto EasyList. I created a GH issue about it, had a productive chat with a maintainer about what data we collected, and which data they thought was PII. If we wanted our domain unblocked we could remove the PII data from the requests, or create a secondary domain that only received the non-PII data. We were gathering data that was not legally considered PII by something like GDPR, but I understand why an adblocker would be even more strict than the legal minimums. I brought this up with the executives and instead they tried to threaten the maintainers of the block list and tried to educate them on how "technically this isn't personally identifiable data according to this legal spec". The maintainers stopped responding (rightfully so) and our data collection was forever halved.

jacquesm 4y ago

That's pretty close to an admission of incompetence.

deleted 4y ago
[deleted]

deleted 4y ago

[deleted]

unethical_ban 4y ago

You might say the WAF was Imperva-ious. I remember an instance where Lowe's website was broken on the corporate network. Our proxy re-arranged the order of HTTP content headers from the site, and Akamai took it as malicious behavior.

ninefathom 4y ago
Not quite, but I've used that one before as well. I suspect many of them offer similar(ly naïve) functionality.
capitainenemo 4y ago
Lowes website has been 100% broken for me ever since I enabled Resist Fingerprinting in Firefox. I can load exactly one page, but on any navigation or refresh I get: ===== Access Denied You don't have permission to access "http://www.lowes.com/ http://www.lowes.com/" on this server. Reference #18.cc69dc17.1661724957.fe4ef4 ==== Result, unless I use the profile with fingerprinting enabled, I just have to buy elsewhere. Drupal.org triggers "prove you're not a robot" every few page navigations with Resist Fingerprinting enabled. Walmart.com too. Fedex package tracking errors (seemingly due to the API server refusing the connection) if resist fingerprinting is enabled. Amusingly if you use the website help bot and say "track XXXXX" that does work to get some basic information.
- gruez 4y ago
  I have RFP enabled and it works fine for me. I did get the "Access Denied" error you mentioned on my first try, but after switching VPN servers it worked fine.
  - capitainenemo 4y ago
    On Lowes.com? Retest in a clean profile. Seems that once they trust you you are ok for a while, at least from a friend's test, who was able to reproduce in a clean profile. But maybe it is IP linked and takes a little bit to accumulate. Did you just enable privacy.resistFingerprinting recently? Also. Doublecheck that it is enabled. Also, I'm using Nightly firefox. It may be the resist fingerprinting is more robust there. BTW, this isn't using a VPN or anything that might seem suspicious. Just my bog standard US broadband.
    - gruez 4y ago
      >On Lowes.com? Retest in a clean profile. I tested on a fresh container so for all intents and purposes it's a "clean profile". > Did you just enable privacy.resistFingerprinting recently? No, but it shouldn't matter given that I was using a fresh container and VPN. >Also. Doublecheck that it is enabled. Also, I'm using Nightly firefox. It may be the resist fingerprinting is more robust there. It's definitely enabled. I'm not using nightly though.

squeaky-clean 4y ago

I used to work for an airfare marketing company. We would get our 3rd party scripts onto an airlines booking engine to be able to run our own analytics and gather data for ads. We'd mainly collect things like prices, number of available seats, etc, because it turns out airlines can't really give you those answers through an API without it costing too much, so we piggybacked on real customer searches. Almost no one in the office ran adblockers, which was weird to me. When our analytics traffic dropped by like 50% one day, I was the only one to notice that our domain made it onto EasyList. I created a GH issue about it, had a productive chat with a maintainer about what data we collected, and which data they thought was PII. If we wanted our domain unblocked we could remove the PII data from the requests, or create a secondary domain that only received the non-PII data. We were gathering data that was not legally considered PII by something like GDPR, but I understand why an adblocker would be even more strict than the legal minimums. I brought this up with the executives and instead they tried to threaten the maintainers of the block list and tried to educate them on how "technically this isn't personally identifiable data according to this legal spec". The maintainers stopped responding (rightfully so) and our data collection was forever halved.