4 ms·
>> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns. Well, if I
by YeGoblynQueenne 8y ago
>> The headline of this article is "More data is not better", which is a stronger claim than diminishing returns - it's neutral or negative returns.
Well, if I was paying $10,000 for 10,000 examples (to collect, cleanup, process, train with, etc), getting 90% accuracy and making $90,000 from the training model, and now I'm paying $10,000,000 for 10,000,000 examples, getting 91% accuracy and making $91,000 from the trained model, I'm losing money where before I was making some. That's "not better".
- nostrademons 8y agoFew big tech companies pay for training data - it usually arises organically out of usage data from their products. You need to collect this anyway for product function / business metrics / abuse prevention, and build the cleaning & processing pipelines. So the only marginal cost of feeding more training data into your machine learning pipeline is the computational cost of training it, which is usually tiny fractions of a penny per sample.
- YeGoblynQueenne 8y agoIf it was so simple to collect, process and train with (very) large amounts of data, everyone woudl be doing it. Instead, it's just a few very large companies that can do that, Google, Facebook et al. Anyway, the cost per example doesn't have to be astronomical. If you need a few millinos of those, you can pay a fraction of a penny and still have a big black hole in your budget, unless you can significantly improve performance.
- nostrademons 8y agoOnly a few big companies can do it because there's a bootstrapping problem. To get large amounts of virtually free data, you need lots of users who have signed up for giving you their data in exchange for a useful service. This was much easier to achieve for companies started between 1995-2005, when the web was young, because the Internet was such a huge leap forwards over what came before it. Existing startups now have to compete with the products of these giants, many of which have been enhanced by years of machine learning. That's challenging. To give you a sense of how cheap computing power is, my startup regularly processes roughly 2B webpages with some complicated algorithms that need to go node-by-node over the whole DOM tree. That's roughly 77TB of (gzipped) data, and around 100 trillion nodes. It costs me a few hundred bucks of AWS time. That's a rounding error for a big corp; a single data scientist's salary for one day will run you around that much.