2 ms·
Since Coral considers itself a distributed caching proxy rather than a crawler, it doesn't respect robots.txt. It does respect the relevant cache-control header
by _delirium 16y ago
Since Coral considers itself a distributed caching proxy rather than a crawler, it doesn't respect robots.txt. It does respect the relevant cache-control headers, like "no-cache", but this site doesn't appear to set them.
However, the main problem here is that Coral doesn't cache any files >50mb, so almost none of these are cached. The few <50mb do seem to be, though, e.g.: http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20Essential%20Mix%20-%20Pete%20Tong%20(Part%201).mp3 http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20E...
- deleted 16y ago[deleted]