2 ms·
It crawls 300 sites and scans the images from just the front page, and it does this daily. 30k images includes a bit of fluff, which is trimmed down, so the act
by chrischen 12y ago
It crawls 300 sites and scans the images from just the front page, and it does this daily. 30k images includes a bit of fluff, which is trimmed down, so the actual real images from articles is a bit smaller. This was also from the initial scan, so a daily yield of only the newly discovered images is probably only around 5k.
The crossover would be a lot larger if I ran this from say a week of submitted articles (whereas right now it's a crossover of basically front page articles from 1-2 days).
If you compare this to hypem.com, the collisions are actually as frequent if not more, considering hypem.com gets updated about once a week.