4 ms·

Olmo from AllenAI has been releasing their full pipelines including data [1]. A lot of it is just repackaged and resampled dumps from copyrighted data that has

Olmo from AllenAI has been releasing their full pipelines including data [1]. A lot of it is just repackaged and resampled dumps from copyrighted data that has long been publicly available as dumps: Common Crawl, arxiv, Wikipedia, StackExchange, reddit --- all of which are presumably copyrighted with different licenses. Go in Huggingface and you can find massive multi TB data dumps used for pre training. It is just as legal as when Uber and AirBNB were running illegal taxis and hotels during their growth phase. I'm just waiting for some corporate IP law firm to learn about Huggingface. [1] https://huggingface.co/datasets/allenai/dolma3_pool https://huggingface.co/datasets/allenai/dolma3_pool

__float 4mo ago
It's rather off-topic at this point, but I've never understood how HF can afford to be a CDN for such huge files. It seems like enterprise customers must be subsidizing a lot, but...at that point, is there not a cheaper alternative that doesn't subsidize every hobbyist and startup around?
- tw1984 4mo ago
  > how HF can afford to be a CDN for such huge files bandwidth and storage are literally free when compared to the cost of GPU clusters. HF gets rewarded heavily on capital market for being in AI without actually doing much AI stuff, that is a huge win when compared to costs they are paying for bandwidth and storage.
- re-thc 4mo ago
  > how HF can afford to be a CDN for such huge files To be precise, Amazon Cloudfront is the CDN. Maybe they got some startup deal? Amazon does now also have flat rate plans that are a lot cheaper.
hnfong 4mo ago
> I'm just waiting for some corporate IP law firm to learn about Huggingface. Presumably they already know. The issue is that IP law firms are tiny compared to the trillions of capital pouring into "AI". And if you believe the USA is a capitalist country where the side with deeper pockets win, you know you're not going to win against the trillionaires.
alchemist1e9 4mo ago
Why is the text field in dataset preview table populated with pornographic labels?
- yencabulator 4mo ago
  Because it's a random sample of the Internet?
  - alchemist1e9 4mo ago
    Could be but then it means like 98% is pornography I guess, because it’s every row, so if random a bad sign!

__float 4mo ago

It's rather off-topic at this point, but I've never understood how HF can afford to be a CDN for such huge files. It seems like enterprise customers must be subsidizing a lot, but...at that point, is there not a cheaper alternative that doesn't subsidize every hobbyist and startup around?

tw1984 4mo ago
> how HF can afford to be a CDN for such huge files bandwidth and storage are literally free when compared to the cost of GPU clusters. HF gets rewarded heavily on capital market for being in AI without actually doing much AI stuff, that is a huge win when compared to costs they are paying for bandwidth and storage.
re-thc 4mo ago
> how HF can afford to be a CDN for such huge files To be precise, Amazon Cloudfront is the CDN. Maybe they got some startup deal? Amazon does now also have flat rate plans that are a lot cheaper.

hnfong 4mo ago

> I'm just waiting for some corporate IP law firm to learn about Huggingface. Presumably they already know. The issue is that IP law firms are tiny compared to the trillions of capital pouring into "AI". And if you believe the USA is a capitalist country where the side with deeper pockets win, you know you're not going to win against the trillionaires.

alchemist1e9 4mo ago

Why is the text field in dataset preview table populated with pornographic labels?

yencabulator 4mo ago
Because it's a random sample of the Internet?
- alchemist1e9 4mo ago
  Could be but then it means like 98% is pornography I guess, because it’s every row, so if random a bad sign!