4 ms·

It's worth noting that many journals don't control the platform their scholarly content. It looks like ACS uses this [Atypon](http://www.atypon.com http://www.a

It's worth noting that many journals don't control the platform their scholarly content. It looks like ACS uses this [Atypon](http://www.atypon.com http://www.atypon.com). That's the likely source of this spider trap, not ACS. Atypon has [a relatively small client list](http://www.atypon.com/our-clients/featured-clients.php http://www.atypon.com/our-clients/featured-clients.php). Compare it to [Highwire](http://highwire.stanford.edu/lists/allsites.dtl http://highwire.stanford.edu/lists/allsites.dtl). I'd be willing to bet that all journals hosted with Atypon share this spider trap—even journals that are supposed to be open access where spidering should be OK. Scientific publishing is weird. Source: I work in scientific publishing.

Blahah 13y ago
All the Atypon clients seem to have now disabled this trap (I tried the 5 biggest ones).
greglindahl 13y ago
In Highwire's case, they typically have a robots.txt blocking everyone but Google ... and the reason is not malice, it's inefficient software. Fetching a page once every few seconds is enough to overload their system.
- sgt101 13y ago
  That might have been true in 1997, but in 2014 a page every few seconds is so far back in the noise that it's irrelevant. If you are getting problems from an IP it's normally that they are trying 1000's requests per second, and this isn't any halfway sensible spider, it's a dos attack. Robots.txt only stops you if you read and abide by it, and if you are getting a page every few seconds it's almost certain that the domain will not notice.
- AYBABTME 13y ago
  A problem with robots is that they will scan all of a website. Normal user traffic is somewhat focused and is easy to cache. When one robot comes and scans everything at once, it brings a bunch of unpopular pages in the cache, possibly evicting more popular ones in the process. The more popular one will then need to be re-cached as requests come back for it. If you avoid caching requests by robots, then instead you end up having to go through all the layers of your app, possibly going in the database. In most situation, I don't think the above matters much. But I can see why it could be a worst case for some stacks.
  - batbomb 13y ago
    That's easily detectable and mitigable via a MRU cache, or no caching at all.
ak217 13y ago
Scientific publishing is not just weird, it's sinister. That said, ACS is pretty sinister, too. They opposed PubChem (http://en.wikipedia.org/wiki/PubChem#ACS.27s_concerns http://en.wikipedia.org/wiki/PubChem#ACS.27s_concerns) and in general don't behave like the nonprofit scientist trade organization that they present themselves as.
lazyjeff 13y ago
Those client lists are a bit unfair to compare: Atypon lists large publishers (Elsevier, IEEE, Oxford University Press, Taylor & Francis, ACS) while HighWire's list has a lot of individual journals (Journal of Early Childhood Research, Monthly Notices of the Royal Astronomical Society: Letters, etc.)

Blahah 13y ago

All the Atypon clients seem to have now disabled this trap (I tried the 5 biggest ones).

greglindahl 13y ago

In Highwire's case, they typically have a robots.txt blocking everyone but Google ... and the reason is not malice, it's inefficient software. Fetching a page once every few seconds is enough to overload their system.

sgt101 13y ago
That might have been true in 1997, but in 2014 a page every few seconds is so far back in the noise that it's irrelevant. If you are getting problems from an IP it's normally that they are trying 1000's requests per second, and this isn't any halfway sensible spider, it's a dos attack. Robots.txt only stops you if you read and abide by it, and if you are getting a page every few seconds it's almost certain that the domain will not notice.
AYBABTME 13y ago
A problem with robots is that they will scan all of a website. Normal user traffic is somewhat focused and is easy to cache. When one robot comes and scans everything at once, it brings a bunch of unpopular pages in the cache, possibly evicting more popular ones in the process. The more popular one will then need to be re-cached as requests come back for it. If you avoid caching requests by robots, then instead you end up having to go through all the layers of your app, possibly going in the database. In most situation, I don't think the above matters much. But I can see why it could be a worst case for some stacks.
- batbomb 13y ago
  That's easily detectable and mitigable via a MRU cache, or no caching at all.

ak217 13y ago

Scientific publishing is not just weird, it's sinister. That said, ACS is pretty sinister, too. They opposed PubChem (http://en.wikipedia.org/wiki/PubChem#ACS.27s_concerns http://en.wikipedia.org/wiki/PubChem#ACS.27s_concerns) and in general don't behave like the nonprofit scientist trade organization that they present themselves as.

lazyjeff 13y ago

Those client lists are a bit unfair to compare: Atypon lists large publishers (Elsevier, IEEE, Oxford University Press, Taylor & Francis, ACS) while HighWire's list has a lot of individual journals (Journal of Early Childhood Research, Monthly Notices of the Royal Astronomical Society: Letters, etc.)