3 ms·

It won’t help you learn to write a scraper, but using the common crawl dataset will get you access to a crazy amount of data without paying to acquire it yourse

It won’t help you learn to write a scraper, but using the common crawl dataset will get you access to a crazy amount of data without paying to acquire it yourself. https://commoncrawl.org/the-data/ https://commoncrawl.org/the-data/

aspyct 7y ago
Cool, didn't know about this. Thanks!
- Reelin 7y ago
  > As part of my learning in data science, I need/want to gather data. Also not web scraping, but a few other public data set sources to check. https://registry.opendata.aws https://registry.opendata.aws https://github.com/awesomedata/awesome-public-datasets https://github.com/awesomedata/awesome-public-datasets
  - aspyct 7y ago
    Thanks!
    - smcnally 7y ago
      also https://www.reddit.com/r/datasets https://www.reddit.com/r/datasets
      - aspyct 7y ago
        The comments that keep on giving :D
        analyticascent 7y ago
        I appreciate you starting this thread, these are great resources people are posting. Common Crawl is the data set to master if someone wants to use the fruits of web scraping without actually doing the web scraping.
my_green_book 7y ago
This is quite useful. Thanks a lot