3 ms·

> Process the whole Wikidata in 7 minutes with your laptop Wikidata is several magnitudes smaller than Freebase (closed by Google in May) and it won't fit in y

• thibaut_barrere 11y ago
  What are your favorite large, publicly available datasets?
  • nextos 11y ago
    The Cancer Genome Atlas, Ensembl, 1000Genomes.
  • Smerity 11y ago
    Biased reply (I'm a data scientist there): Common Crawl[1]. We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone completely free. [1]: http://commoncrawl.org/ http://commoncrawl.org/
  • rcpt 11y ago
    This thread is pretty good http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public http://www.quora.com/Where-can-I-find-large-datasets-open-to...
• xomateix 11y ago
  As you comment, Freebase is bigger than Wikidata. It is 22GB compressed (250GB uncompressed) while Wikidata is 5GB compressed (49GB uncompressed) [1]. Said that, I believe the process described in the blog post is not loading the whole Wikidata dump into memory and it would work the same to process Freebase or even larger data dumps with your laptop. From the post: How Akka Streams can be used to process the Wikidata dump in parallel and using constant memory with just your laptop. [1] https://developers.google.com/freebase/data https://developers.google.com/freebase/data http://dumps.wikimedia.org/other/wikidata/ http://dumps.wikimedia.org/other/wikidata/