3 ms·

One as-yet-unmentioned technique is to adjust a bot score by taking an OS fingerprint, and comparing that to the listed user agent. It's not perfect for a vari

by rscale 14y ago

One as-yet-unmentioned technique is to adjust a bot score by taking an OS fingerprint, and comparing that to the listed user agent. It's not perfect for a variety of reasons, but I found it to be a useful input for a similar bot detection problem. That said, this was a long time ago so I'd need to re-run some experiments to see if the hypothesis remained valid. For those not familiar with OS fingerprinting, it's a method of looking at the details of a connection and determining which operating system is most likely to have created packets with those options and flags. In BSD, it's built into pf.

dredmorbius 14y ago
Interesting. Any Linux equivalents or Java classes which offer similar capabilities (we're using a Java-based application server / webserver).
- rscale 14y ago
  The linux equivalent is p0f (http://freecode.com/projects/p0f http://freecode.com/projects/p0f) I can't speak to Java classes, but they'd need to be dealing with raw packets which isn't an option in a typical stack.
- biot 14y ago
  It works off of checking the SYN packet [0], which happens at the transport layer [1]. Once it gets to the application layer the information is not available unless the transport layer stores the information and provides a method to query a given connection. [0] http://www.openbsd.org/faq/pf/filter.html#osfp http://www.openbsd.org/faq/pf/filter.html#osfp [1] http://en.wikipedia.org/wiki/OSI_model http://en.wikipedia.org/wiki/OSI_model
  - dredmorbius 14y ago
    I suspected something like this, but was hoping enough of the signature might survive for a daemon (or Java class acting as one) to take a peek.
tripzilch 14y ago
On the packet level ... hm that's pretty clever, definitely not trivial to fake, either. Just that if your bot runs from, say, Windows 7 and it spoofs an IE8 user-agent header, or even runs by directly automating IE itself, how do you detect it then? Both of those scenarios are not unlikely at all.
- rscale 14y ago
  If the bot spoofs a user-agent that is viable for the OS, then you need to rely on other signals. It only works as a component of a defense in depth strategy.

by rscale 14y ago

dredmorbius 14y ago

Interesting. Any Linux equivalents or Java classes which offer similar capabilities (we're using a Java-based application server / webserver).

rscale 14y ago
The linux equivalent is p0f (http://freecode.com/projects/p0f http://freecode.com/projects/p0f) I can't speak to Java classes, but they'd need to be dealing with raw packets which isn't an option in a typical stack.
biot 14y ago
It works off of checking the SYN packet [0], which happens at the transport layer [1]. Once it gets to the application layer the information is not available unless the transport layer stores the information and provides a method to query a given connection. [0] http://www.openbsd.org/faq/pf/filter.html#osfp http://www.openbsd.org/faq/pf/filter.html#osfp [1] http://en.wikipedia.org/wiki/OSI_model http://en.wikipedia.org/wiki/OSI_model
- dredmorbius 14y ago
  I suspected something like this, but was hoping enough of the signature might survive for a daemon (or Java class acting as one) to take a peek.

tripzilch 14y ago

On the packet level ... hm that's pretty clever, definitely not trivial to fake, either. Just that if your bot runs from, say, Windows 7 and it spoofs an IE8 user-agent header, or even runs by directly automating IE itself, how do you detect it then? Both of those scenarios are not unlikely at all.

rscale 14y ago
If the bot spoofs a user-agent that is viable for the OS, then you need to rely on other signals. It only works as a component of a defense in depth strategy.