5 ms·

Yeah I posted this in another thread but your claim is simply untrue. There is little to no need for anyone to scrape the web themselves, CommonCrawl does it f

by Kranar 4y ago

Yeah I posted this in another thread but your claim is simply untrue. There is little to no need for anyone to scrape the web themselves, CommonCrawl does it for free and provides access to all that data for free, they are a charity similar to The Internet Archive: https://commoncrawl.org/ https://commoncrawl.org/ As for using that data to train a model, I've done it at my own company and it's not that expensive, certainly not the astronomically expensive figures people seem to think it is. Maybe a hobbyist can't do it for themselves, but my small company managed to get pretty good results training on our existing hardware. It's hard to give concrete figures because we manage our own hardware instead of using AWS, but all in we're talking on the order of 100k, as opposed to what I've heard people on HN think it costs, as if it's a $10 million+ enterprise.

anon291 4y ago
Yeah and the model will be useless when the courts determine it's illegal to use it commercially without having properly licensed the data. Remember... Bits have color and AI models are a rainbow.
- deleted 4y ago
  [deleted]
- joenot443 4y ago
  “The courts” don’t have rules which apply globally for this sort of thing. My understanding is that even if the models were made illegal to host in America, there’s no shortage of other countries much looser copyright laws where one could hypothetically host such a thing. There are clearnet sites which host entire movies and games and there have been since the dawn of the internet, wouldn’t you expect those would be bigger targets to go after if the powers were there? All that to say, I’m not bullish on the future global leverage of American copyright law, especially when it comes to AI models.
  - piva00 4y ago
    But courts have jurisdiction over their consumer markets, if a court in the US says it's illegal to use non-licenced data for training models and any product doing so can't be used/sold in the US you have lost access to a massive consumer market. Case in point: GDPR, you can vacuum as much private data as you wish from non-EU citizens, the moment you touch EU-citizen data you must abide by the GDPR or you don't have access to a market of 400+ million of some of the wealthiest countries in the world. You don't need global jurisdiction when your economical power is large enough.
    - soco 4y ago
      Which leaves the citizens in countries without large enough economical power at the mercy of those corporations. Nothing new, you will say.
leloctai 4y ago
Big corp will always have more data. Even just the web, search engine have a big head start. Then there are "private" data that big provider use for "improving our product": emails, word documents, chat messages, non public social media posts,...
Animats 4y ago
That's a useful number. Just yesterday we had an announcement of an open source system you can train with one GPU on a desktop for simple models. That could usefully use 16 or 24 GPUs, but it's not like you need a whole data center. It doesn't look like this is an industry with excessive monopoly potential. The real issue for machine learning is that it lets you build systems that optimize for some metric. That's what corporations do. I've been pointing out lately that the ethical problems for AI in the near term look very much like those for corporations.
alchemist1e9 4y ago
I’ve been thinking along the same lines as what you seem to have done. I looked into the CommonCrawl data and also the Pile and I’m not sure they are as large as people think. We also manage some of our own hardware including GPUs. It looked to me that with maybe .5 PB decent storage and 8 high end gpus it could be possible to train a decent LLM in a few months, how does that estimate compare to your experience?

by Kranar 4y ago

anon291 4y ago

Yeah and the model will be useless when the courts determine it's illegal to use it commercially without having properly licensed the data. Remember... Bits have color and AI models are a rainbow.

deleted 4y ago
[deleted]
joenot443 4y ago
“The courts” don’t have rules which apply globally for this sort of thing. My understanding is that even if the models were made illegal to host in America, there’s no shortage of other countries much looser copyright laws where one could hypothetically host such a thing. There are clearnet sites which host entire movies and games and there have been since the dawn of the internet, wouldn’t you expect those would be bigger targets to go after if the powers were there? All that to say, I’m not bullish on the future global leverage of American copyright law, especially when it comes to AI models.
- piva00 4y ago
  But courts have jurisdiction over their consumer markets, if a court in the US says it's illegal to use non-licenced data for training models and any product doing so can't be used/sold in the US you have lost access to a massive consumer market. Case in point: GDPR, you can vacuum as much private data as you wish from non-EU citizens, the moment you touch EU-citizen data you must abide by the GDPR or you don't have access to a market of 400+ million of some of the wealthiest countries in the world. You don't need global jurisdiction when your economical power is large enough.
  - soco 4y ago
    Which leaves the citizens in countries without large enough economical power at the mercy of those corporations. Nothing new, you will say.

leloctai 4y ago

Big corp will always have more data. Even just the web, search engine have a big head start. Then there are "private" data that big provider use for "improving our product": emails, word documents, chat messages, non public social media posts,...

Animats 4y ago

That's a useful number. Just yesterday we had an announcement of an open source system you can train with one GPU on a desktop for simple models. That could usefully use 16 or 24 GPUs, but it's not like you need a whole data center. It doesn't look like this is an industry with excessive monopoly potential. The real issue for machine learning is that it lets you build systems that optimize for some metric. That's what corporations do. I've been pointing out lately that the ethical problems for AI in the near term look very much like those for corporations.

alchemist1e9 4y ago

I’ve been thinking along the same lines as what you seem to have done. I looked into the CommonCrawl data and also the Pile and I’m not sure they are as large as people think. We also manage some of our own hardware including GPUs. It looked to me that with maybe .5 PB decent storage and 8 high end gpus it could be possible to train a decent LLM in a few months, how does that estimate compare to your experience?