4 ms·

This collection is 40 times larger than books3. No one seems to care at all about copyright. It’s tempting to leave the English speaking world behind and focus

This collection is 40 times larger than books3. No one seems to care at all about copyright. It’s tempting to leave the English speaking world behind and focus on someplace that supports academic work. Will people call this theft? No one has so far. In the meantime, simply abandoning European and American knowledge is a realistic option. LLMs will learn in Chinese. On the other hand, it would suck much more to be thrown in a Chinese prison. So probably don’t do this if you plan on visiting China. (If China reacts by fully endorsing this, that would be the uno reverse of the century.)

squigz 3y ago
Sharing of information is more important to me personally than business interests
deleted 3y ago
[deleted]
alex_duf 3y ago
China may find it attractive to have a model trained on their languages and cultures. It would send a message of power, and ensure that local tech companies will use a model with their influence rather than western corporate influence. Not saying they should or would endorse, but it wouldn't be _that_ surprising.
sillysaurusx 3y ago
Replying to a deleted comment, because I think this is worth underscoring: By explicitly supporting this, China could signal to AI researchers worldwide that China is a safe haven. It’s the materialization of what everyone said would happen: when copyright enthusiasts shut down AI work in America, companies will simply go elsewhere. Imagine if the epicenter of AI research shifted away from the US. This may seem unlikely, but lack of data is the biggest problem that open source LLMs face. Mistral had to completely refuse to disclose their training sources just to release a decent model. Personally, I’m tempted to focus on this full time. It’s what I’ve always wanted to do. I never gave a thought to legal consequences, and with this collection there appears to be none.
- TerrifiedMouse 3y ago
  > China is a safe haven Heh. It’s a safe haven until your LLM says something the CCP doesn’t like.
  - FartyMcFarter 3y ago
    "Safe"and "haven" are indeed bold worlds when applied to the current Chinese regime...
  - Vetch 3y ago
    This issue is overstated. Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Furthermore, there have been a series of open-weights Chinese models that give reasonable answers to sensitive (in China) questions. Unless you plan to release something customer facing in China, it's not something to stress about. The idea that overly burdensome regulations around open source models in the US shifts the global center of mass of LLMs to China is not implausible. One reason resonant among government officials against opensource LLMs is to keep them from China and Russia. But among the very best embeddings, ~13Bs and perhaps 34B open-weights models are the Chinese ones. The recent DeepSeekCoder also tops EvalPlus: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html For those concerned about over-training, ChatGPT and GPT4 are almost certainly also contaminated and there have been independent confirmations of DeepSeekCoder's strength.
    - rfoo 3y ago
      > Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Especially due to no freedom of the press in China (you need a permit for each published book) they are all already censored. I still don't think China could be a safe haven though, they haven't shot down open source models not because they are nice, it's just that they are incompetent.
      - TerrifiedMouse 3y ago
yorwba 3y ago
Anna's Archive is already blocked by the Great Firewall https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fannas-archive.org https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fann... , chance of endorsement is nil.
- sillysaurusx 3y ago
  Is there a way to find out why? It could be anything from "most new websites get blocked by default, or automatically" to "a censor didn’t like a specific thing." I’m not a fan of bowing to censorship (to say the least), but at this point I am even less a fan of being verbally spit on, called a thief, and being included in lawsuits. I’d take the lesser of two evils, since there’s no way to even do any work in the first place in America. (I’ve been depressed for months that the inevitable outcome of all the LLM lawsuits is that book authors will demand licensing fees, which completely kills open source AI models for all but the most funded institutions. I.e. my original work would’ve been completely impossible.)
  - yorwba 3y ago
    Most websites do not get blocked. When it happens, the censors won't tell you, but if there's a search bar, it's usually easy to find content that explains the block: https://annas-archive.org/search?q=Prisoner+of+the+State+Zhao+Ziyang https://annas-archive.org/search?q=Prisoner+of+the+State+Zha...
    - sillysaurusx 3y ago
      Thank you. I have no way of knowing anything about China, so this kind of thing is super helpful. I wouldn’t have even known what to search for to trigger the censors.
      - rfoo 3y ago
        In principal it's easy to explain - as long as you have uncensored content on your website, it's going to be blocked. For example, HN is blocked. If it didn't happen, then it's because you are not big enough. A few cases: - FB, Instagram, Reddit, Twitter, Youtube are all blocked: user generated content and without outrageous censorship effort (Douyin has 100k employees dedicated to censor and is top cost center, Weibo 50k IIRC) it is very predictable, just assume such things get blocked. - Google is blocked. They announced in 2010 they didn't want to censor any more. - A censored version of Bing is not blocked. - Dropbox is blocked: the same reason. - Notably, GitHub is NOT blocked. GitHub pages and Gist are blocked. This is because GitHub does censor repos upon government request, plus that when it was once blocked scientist spent effort and managed to reverse the decision. - WhatsApp and Signal are blocked. SMS from Signal is also blocked for all +86 numbers so people simply can't register. - iMessage is not blocked. This is, IMO, an outlier. My guess is it's because Apple + no group chat + incompetence.
BiteCode_dev 3y ago
If the AI is accessible to the rest of the world, it's not a problem: the model can query its knowledge and answer in any language since it already is very good at translating things. The problem is rather than, given the economic war between the US and China, the chip ban, the taiwan issues and the GFC, we may not have access to this.
morelisp 3y ago
> simply abandoning European and American knowledge is a realistic option. Haha, OK, yeah.

squigz 3y ago

Sharing of information is more important to me personally than business interests

deleted 3y ago

[deleted]

alex_duf 3y ago

China may find it attractive to have a model trained on their languages and cultures. It would send a message of power, and ensure that local tech companies will use a model with their influence rather than western corporate influence. Not saying they should or would endorse, but it wouldn't be _that_ surprising.

sillysaurusx 3y ago

Replying to a deleted comment, because I think this is worth underscoring: By explicitly supporting this, China could signal to AI researchers worldwide that China is a safe haven. It’s the materialization of what everyone said would happen: when copyright enthusiasts shut down AI work in America, companies will simply go elsewhere. Imagine if the epicenter of AI research shifted away from the US. This may seem unlikely, but lack of data is the biggest problem that open source LLMs face. Mistral had to completely refuse to disclose their training sources just to release a decent model. Personally, I’m tempted to focus on this full time. It’s what I’ve always wanted to do. I never gave a thought to legal consequences, and with this collection there appears to be none.

TerrifiedMouse 3y ago
> China is a safe haven Heh. It’s a safe haven until your LLM says something the CCP doesn’t like.
- FartyMcFarter 3y ago
  "Safe"and "haven" are indeed bold worlds when applied to the current Chinese regime...
- Vetch 3y ago
  This issue is overstated. Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Furthermore, there have been a series of open-weights Chinese models that give reasonable answers to sensitive (in China) questions. Unless you plan to release something customer facing in China, it's not something to stress about. The idea that overly burdensome regulations around open source models in the US shifts the global center of mass of LLMs to China is not implausible. One reason resonant among government officials against opensource LLMs is to keep them from China and Russia. But among the very best embeddings, ~13Bs and perhaps 34B open-weights models are the Chinese ones. The recent DeepSeekCoder also tops EvalPlus: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html For those concerned about over-training, ChatGPT and GPT4 are almost certainly also contaminated and there have been independent confirmations of DeepSeekCoder's strength.
  - rfoo 3y ago
    > Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Especially due to no freedom of the press in China (you need a permit for each published book) they are all already censored. I still don't think China could be a safe haven though, they haven't shot down open source models not because they are nice, it's just that they are incompetent.
    - TerrifiedMouse 3y ago

yorwba 3y ago

Anna's Archive is already blocked by the Great Firewall https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fannas-archive.org https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fann... , chance of endorsement is nil.

sillysaurusx 3y ago
Is there a way to find out why? It could be anything from "most new websites get blocked by default, or automatically" to "a censor didn’t like a specific thing." I’m not a fan of bowing to censorship (to say the least), but at this point I am even less a fan of being verbally spit on, called a thief, and being included in lawsuits. I’d take the lesser of two evils, since there’s no way to even do any work in the first place in America. (I’ve been depressed for months that the inevitable outcome of all the LLM lawsuits is that book authors will demand licensing fees, which completely kills open source AI models for all but the most funded institutions. I.e. my original work would’ve been completely impossible.)
- yorwba 3y ago
  Most websites do not get blocked. When it happens, the censors won't tell you, but if there's a search bar, it's usually easy to find content that explains the block: https://annas-archive.org/search?q=Prisoner+of+the+State+Zhao+Ziyang https://annas-archive.org/search?q=Prisoner+of+the+State+Zha...
  - sillysaurusx 3y ago
    Thank you. I have no way of knowing anything about China, so this kind of thing is super helpful. I wouldn’t have even known what to search for to trigger the censors.
    - rfoo 3y ago
      In principal it's easy to explain - as long as you have uncensored content on your website, it's going to be blocked. For example, HN is blocked. If it didn't happen, then it's because you are not big enough. A few cases: - FB, Instagram, Reddit, Twitter, Youtube are all blocked: user generated content and without outrageous censorship effort (Douyin has 100k employees dedicated to censor and is top cost center, Weibo 50k IIRC) it is very predictable, just assume such things get blocked. - Google is blocked. They announced in 2010 they didn't want to censor any more. - A censored version of Bing is not blocked. - Dropbox is blocked: the same reason. - Notably, GitHub is NOT blocked. GitHub pages and Gist are blocked. This is because GitHub does censor repos upon government request, plus that when it was once blocked scientist spent effort and managed to reverse the decision. - WhatsApp and Signal are blocked. SMS from Signal is also blocked for all +86 numbers so people simply can't register. - iMessage is not blocked. This is, IMO, an outlier. My guess is it's because Apple + no group chat + incompetence.

BiteCode_dev 3y ago

If the AI is accessible to the rest of the world, it's not a problem: the model can query its knowledge and answer in any language since it already is very good at translating things. The problem is rather than, given the economic war between the US and China, the chip ban, the taiwan issues and the GFC, we may not have access to this.

morelisp 3y ago

> simply abandoning European and American knowledge is a realistic option. Haha, OK, yeah.