3 ms·

> ‘5 OOMs off’ I think Google, Microsoft and facebook could easily have 5 OOM data than the entire public web combined if we just count text. Majority of peopl

> ‘5 OOMs off’ I think Google, Microsoft and facebook could easily have 5 OOM data than the entire public web combined if we just count text. Majority of people don't have any content on public web except for personal photos. A minority has few public social media posts and it is rare for people to write blog or research paper etc. And almost everyone has some content written in mail or docs or messaging.

nmca 3y ago
From the article, and relevant here: I’m worried that when people hear ‘5 OOMs off’, how they register it is, “Oh we have 5x less data than we need - we just need a couple of 2x improvements in data efficiency, and we’re golden”. After all, what’s a couple OOMs between friends? No, 5 OOMs off means we have 100,000x less data than we need.
- YetAnotherNick 3y ago
  I meant 100,000x. At least for everyone I know, they have 100,000x data in mail/messaging/docs/notes/meeting etc. than their blog or any public site they own. Hell I would even say that if you just have all the meetings of zoom, it will be few order of magnitude higher than the entire public web.
  - saulpw 3y ago
    If I have 1MB on my blog, 100,000x would be 100GB. Just, no. OOMs are not to be trifled with.
    - YetAnotherNick 3y ago
      How many people have blogs? How many people sent any message or created a google docs? The answer could easily be 10,000x times of people having blog. Also I was just counting text content as I mentioned. For reference, there are 175,000 authors in medium compared to billions using whatsapp or gmail or difference of around 50,000.
- jodrellblank 3y ago
  > "we just need a couple of 2x improvements in data efficiency, and we’re golden”" 100,000x is only seventeen 2x improvements. This[1] says world volume of data doubles every two years, citing a 2016 McKinsey study as the source. That puts it 34 years away. 2056. This[2] says the estimated compound growth rate of data creation is around 61%, that's a doubling time of 1.32 years and puts it 22 years away. 2045. It's not tomorrow, but it may not be all that far away. [1] https://rivery.io/blog/big-data-statistics-how-much-data-is-there-in-the-world/ https://rivery.io/blog/big-data-statistics-how-much-data-is-... [2] https://theconversation.com/the-worlds-data-explained-how-much-were-producing-and-where-its-all-stored-159964 https://theconversation.com/the-worlds-data-explained-how-mu...
HarHarVeryFunny 3y ago
Maybe, and certainly with the current trend of synthetic data they can also create it, but I don't think quantity of data beyond what something like GPT-4 has been trained on will in of itself change much other than reducing brittleness by providing coverage of remaining knowledge gaps. Quality of data (which I believe is at least part of why synthetic data is being used) can perhaps make more of a difference and perhaps at least partly compensate in a crude way for these models lack of outlier rejection and any generalization prediction-feedback loop. Just feed them consistent correct data in the first place.
simonebrunozzi 3y ago
Try the NSA. We speak a lot on the phone, and we did so for the past 30 years.

nmca 3y ago

From the article, and relevant here: I’m worried that when people hear ‘5 OOMs off’, how they register it is, “Oh we have 5x less data than we need - we just need a couple of 2x improvements in data efficiency, and we’re golden”. After all, what’s a couple OOMs between friends? No, 5 OOMs off means we have 100,000x less data than we need.

YetAnotherNick 3y ago
I meant 100,000x. At least for everyone I know, they have 100,000x data in mail/messaging/docs/notes/meeting etc. than their blog or any public site they own. Hell I would even say that if you just have all the meetings of zoom, it will be few order of magnitude higher than the entire public web.
- saulpw 3y ago
  If I have 1MB on my blog, 100,000x would be 100GB. Just, no. OOMs are not to be trifled with.
  - YetAnotherNick 3y ago
    How many people have blogs? How many people sent any message or created a google docs? The answer could easily be 10,000x times of people having blog. Also I was just counting text content as I mentioned. For reference, there are 175,000 authors in medium compared to billions using whatsapp or gmail or difference of around 50,000.
jodrellblank 3y ago
> "we just need a couple of 2x improvements in data efficiency, and we’re golden”" 100,000x is only seventeen 2x improvements. This[1] says world volume of data doubles every two years, citing a 2016 McKinsey study as the source. That puts it 34 years away. 2056. This[2] says the estimated compound growth rate of data creation is around 61%, that's a doubling time of 1.32 years and puts it 22 years away. 2045. It's not tomorrow, but it may not be all that far away. [1] https://rivery.io/blog/big-data-statistics-how-much-data-is-there-in-the-world/ https://rivery.io/blog/big-data-statistics-how-much-data-is-... [2] https://theconversation.com/the-worlds-data-explained-how-much-were-producing-and-where-its-all-stored-159964 https://theconversation.com/the-worlds-data-explained-how-mu...

HarHarVeryFunny 3y ago

Maybe, and certainly with the current trend of synthetic data they can also create it, but I don't think quantity of data beyond what something like GPT-4 has been trained on will in of itself change much other than reducing brittleness by providing coverage of remaining knowledge gaps. Quality of data (which I believe is at least part of why synthetic data is being used) can perhaps make more of a difference and perhaps at least partly compensate in a crude way for these models lack of outlier rejection and any generalization prediction-feedback loop. Just feed them consistent correct data in the first place.

simonebrunozzi 3y ago

Try the NSA. We speak a lot on the phone, and we did so for the past 30 years.