3 ms·
Sharing for anyone who didn't know there is a very good dataset you can use now. If you don't have a nvme ssd in your computer, I highly recommend getting one f
by utefan001 7y ago
Sharing for anyone who didn't know there is a very good dataset you can use now. If you don't have a nvme ssd in your computer, I highly recommend getting one for fast i/o.
http://commoncrawl.org/ http://commoncrawl.org/
http://commoncrawl.org/the-data/ http://commoncrawl.org/the-data/
http://index.commoncrawl.org/ http://index.commoncrawl.org/
related..
Mark's blog is amazing and worth more than any data science degree imho.
https://tech.marksblogg.com/petabytes-of-website-data-spark-emr.html https://tech.marksblogg.com/petabytes-of-website-data-spark-...
https://tech.marksblogg.com https://tech.marksblogg.com
- kingludite 7y agowow, thanks. [edit] in my experience yacy works really well. You have it crawl the sites you frequently visit and their external links and it quickly accumulates to something more accurate than google.
- jaytaylor 7y agohttps://yacy.net/en/index.html https://yacy.net/en/index.html