Depending on what you're trying to do you have some options, but you should probably outsource. Option 1: If you're working on image recognition or anything si

Depending on what you're trying to do you have some options, but you should probably outsource. Option 1: If you're working on image recognition or anything similar, it's easy to get a ton of data. There are many corpora of image data available, probably on the order of hundreds of terabytes, with tags or at least some structured data. If you're going to "spider the web" you should use Common Crawl instead, and can access all of Blekko's data for the cost of data transfer via AWS. Same for text data, use the Google N-gram corpus. Knowing that all of that exists, I would still recommend Option 2: outsourcing your AI needs unless you're a researcher or have a decent budget for AI development. Go search "ai api" and pick one that matches what you want to do. Match your skill and risk tolerance; nobody got fired for using IBM, but you may get a better experience from a startup with the possibility of flaming out in a year. You'll get the leverage of whatever company is spending those dollars on your behalf, and you'll be able to concentrate on the user experience instead of the science-ey part. Option 3 is "if you can't beat 'em, join 'em". Go work for MS / GOOG / FB / IBM building something, and get access to those resources for yourself. Then at some point in the future you'll know their API interface, and you can go back to option #2 with much better data.