5 ms·

Stream of crawled sites from the Internet Archive (somewhat NSFW)

36 points by Mithrandir 14y ago

ck2 14y ago
Either there is a security loophole and you can force it to crawl sites or wayback has an open thumbnail service. http://crawl432.us.archive.org/crawling/domain/thumb.py/news.ycombinator.com http://crawl432.us.archive.org/crawling/domain/thumb.py/news... http://crawl432.us.archive.org/crawling/domain/thumb.py/www.google.com http://crawl432.us.archive.org/crawling/domain/thumb.py/www.... Judging by the delay, basically changing the end url forces it to crawl and then the thumbnail is cached and served. They might want to fix that.
- cleverjake 14y ago
  nah - its a lookup. http://crawl432.us.archive.org/crawling/domain/thumb.py/www.urbanoutfitters.com http://crawl432.us.archive.org/crawling/domain/thumb.py/www.... vs www.urbanoutfitters.com the slide that is shown is from august.
ronyeh 14y ago
Pretty cool. It's funny how many thumbnails include the parked domain girl (with backpack). http://www.urlesque.com/2010/11/03/parked-domain-girl/ http://www.urlesque.com/2010/11/03/parked-domain-girl/
frozenport 14y ago
It only works for few seconds when I switch the browser focus (by changing my tab), then the tiles don't show anything. Is this a feature?
- nodata 14y ago
  Are you using Adblock?
  - frozenport 14y ago
    yes
DigitalSea 14y ago
I see images for a few seconds and then the images disappear and I just see moving black squares in place of imagery like it caches only a set amount and doesn't fetch any more unless you reload the page again. Cool, but pretty pointless.
aj 14y ago
The speed at which the crawling is taking place, the tile source (the site being crawled) changes before my internet connection can even download the tile. So after a few seconds, all I see are black blinking tiles
Nursie 14y ago
Somewhat NSFW? As at work not going to visit, but it does now strike me that archive.org probably has probably accumulated the world's largest porn collection.
- brazzy 14y ago
  Actually when I just looked, there was not a single obvious porn site.
Mithrandir 14y ago
For future reference, it looks like this: http://i.imgur.com/QHPWW.jpg http://i.imgur.com/QHPWW.jpg

36 points by Mithrandir 14y ago

ck2 14y ago

Either there is a security loophole and you can force it to crawl sites or wayback has an open thumbnail service. http://crawl432.us.archive.org/crawling/domain/thumb.py/news.ycombinator.com http://crawl432.us.archive.org/crawling/domain/thumb.py/news... http://crawl432.us.archive.org/crawling/domain/thumb.py/www.google.com http://crawl432.us.archive.org/crawling/domain/thumb.py/www.... Judging by the delay, basically changing the end url forces it to crawl and then the thumbnail is cached and served. They might want to fix that.

cleverjake 14y ago
nah - its a lookup. http://crawl432.us.archive.org/crawling/domain/thumb.py/www.urbanoutfitters.com http://crawl432.us.archive.org/crawling/domain/thumb.py/www.... vs www.urbanoutfitters.com the slide that is shown is from august.

ronyeh 14y ago

Pretty cool. It's funny how many thumbnails include the parked domain girl (with backpack). http://www.urlesque.com/2010/11/03/parked-domain-girl/ http://www.urlesque.com/2010/11/03/parked-domain-girl/

frozenport 14y ago

It only works for few seconds when I switch the browser focus (by changing my tab), then the tiles don't show anything. Is this a feature?

nodata 14y ago
Are you using Adblock?
- frozenport 14y ago
  yes

DigitalSea 14y ago

I see images for a few seconds and then the images disappear and I just see moving black squares in place of imagery like it caches only a set amount and doesn't fetch any more unless you reload the page again. Cool, but pretty pointless.

aj 14y ago

The speed at which the crawling is taking place, the tile source (the site being crawled) changes before my internet connection can even download the tile. So after a few seconds, all I see are black blinking tiles

Nursie 14y ago

Somewhat NSFW? As at work not going to visit, but it does now strike me that archive.org probably has probably accumulated the world's largest porn collection.

brazzy 14y ago
Actually when I just looked, there was not a single obvious porn site.

Mithrandir 14y ago

For future reference, it looks like this: http://i.imgur.com/QHPWW.jpg http://i.imgur.com/QHPWW.jpg