3 ms·

Using Coral CDN: wget -rl 0 -np -A .mp3 http://thenine.ca.nyud.net/essential/

• aidenn0 16y ago
  Nogood, checkout the robots.txt
  • forkqueue 16y ago
    Yup, thenine.ca is blocking coral cache for .mp3 :(
  • _delirium 16y ago
    Since Coral considers itself a distributed caching proxy rather than a crawler, it doesn't respect robots.txt. It does respect the relevant cache-control headers, like "no-cache", but this site doesn't appear to set them. However, the main problem here is that Coral doesn't cache any files >50mb, so almost none of these are cached. The few <50mb do seem to be, though, e.g.: http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20Essential%20Mix%20-%20Pete%20Tong%20(Part%201).mp3 http://thenine.ca.nyud.net/essential/1998/1998.01.01%20-%20E...
    • deleted 16y ago
      [deleted]
• jc-denton 16y ago
  -e robots=off However aren't they copyrighted?!