4 ms·

How would you differentiate the traffic coming form headless browsers from normal users ? with a proper user-agent and javascript enabled, isn't it almost impos

by level09 13y ago

How would you differentiate the traffic coming form headless browsers from normal users ? with a proper user-agent and javascript enabled, isn't it almost impossible ?

zzzcpan 13y ago
No, it's not impossible. Basically, you detect anomalies, i.e. gather statistical information about your users and their behavior in advance and apply that as a filter during attacks. For example, if you have like 10 new refererless users per hour, i.e. without a cookie and a referrer, but suddenly there are 1000 of them and backend is already slow - you should probably drop their requests. Same with specific referrers and any other information you have about your users. But this is kind of preventative only approach, it won't work if your site is already under attack.
- STRML 13y ago
  The Referer header is just one more thing that a sophisticated attacker will spoof. I think the GP is correct - with enough preparation, a concentrated PhantomJS attack will look exactly like users accessing the website en masse, and it can be very, very difficult to tell who is real and who is not. That is why they use a tool as heavy as PhantomJS, which (from experience) is not very quick compared to more traditional methods. It is because it spoofs a real browser quite completely that it is so good for these methods. For example, some pages might do a trick where they set a cookie, load another page, and redirect / drop based on whether or not that cookie is present in exactly the form it should be present. Maybe they set two, one expires immediately and the other contains some complex data. PhantomJS can handle that easily (as it is just Webkit + QT) but traditional tools do not.
  - zzzcpan 13y ago
    > The Referer header is just one more thing > that a sophisticated attacker will spoof. It doesn't really matter, as attacker wouldn't have your data. For example, you know that on average you have 10 users per hour from some specific referrer and that most of them are coming from a single country. And you know this for every referrer. But attacker doesn't, even if he tries to fake it, his requests are either not going to satisfy the profile (he won't get something right, like country, user-agent, etc) or he will reach the limit (by trying to send more than N requests per hour for specific referrer, as he doesn't know the limit either). Of course you could make it very precise by using a lot more data points for each user: where users usually go after which URL, how long do they stay on each URL, how many requests do they usually make, do they have your cookie, how long, etc. And I'm talking about cookies as a sign of a user, who already visited your website in the past.

by level09 13y ago

How would you differentiate the traffic coming form headless browsers from normal users ? with a proper user-agent and javascript enabled, isn't it almost impossible ?

zzzcpan 13y ago

No, it's not impossible. Basically, you detect anomalies, i.e. gather statistical information about your users and their behavior in advance and apply that as a filter during attacks. For example, if you have like 10 new refererless users per hour, i.e. without a cookie and a referrer, but suddenly there are 1000 of them and backend is already slow - you should probably drop their requests. Same with specific referrers and any other information you have about your users. But this is kind of preventative only approach, it won't work if your site is already under attack.

STRML 13y ago
The Referer header is just one more thing that a sophisticated attacker will spoof. I think the GP is correct - with enough preparation, a concentrated PhantomJS attack will look exactly like users accessing the website en masse, and it can be very, very difficult to tell who is real and who is not. That is why they use a tool as heavy as PhantomJS, which (from experience) is not very quick compared to more traditional methods. It is because it spoofs a real browser quite completely that it is so good for these methods. For example, some pages might do a trick where they set a cookie, load another page, and redirect / drop based on whether or not that cookie is present in exactly the form it should be present. Maybe they set two, one expires immediately and the other contains some complex data. PhantomJS can handle that easily (as it is just Webkit + QT) but traditional tools do not.
- zzzcpan 13y ago
  > The Referer header is just one more thing > that a sophisticated attacker will spoof. It doesn't really matter, as attacker wouldn't have your data. For example, you know that on average you have 10 users per hour from some specific referrer and that most of them are coming from a single country. And you know this for every referrer. But attacker doesn't, even if he tries to fake it, his requests are either not going to satisfy the profile (he won't get something right, like country, user-agent, etc) or he will reach the limit (by trying to send more than N requests per hour for specific referrer, as he doesn't know the limit either). Of course you could make it very precise by using a lot more data points for each user: where users usually go after which URL, how long do they stay on each URL, how many requests do they usually make, do they have your cookie, how long, etc. And I'm talking about cookies as a sign of a user, who already visited your website in the past.