10 ms·

Exclusive access for LLM companies to largest Chinese nonfiction book collection

235 points by sillysaurusx 3y ago

sillysaurusx 3y ago
This collection is 40 times larger than books3. No one seems to care at all about copyright. It’s tempting to leave the English speaking world behind and focus on someplace that supports academic work. Will people call this theft? No one has so far. In the meantime, simply abandoning European and American knowledge is a realistic option. LLMs will learn in Chinese. On the other hand, it would suck much more to be thrown in a Chinese prison. So probably don’t do this if you plan on visiting China. (If China reacts by fully endorsing this, that would be the uno reverse of the century.)
- squigz 3y ago
  Sharing of information is more important to me personally than business interests
- deleted 3y ago
  [deleted]
- alex_duf 3y ago
  China may find it attractive to have a model trained on their languages and cultures. It would send a message of power, and ensure that local tech companies will use a model with their influence rather than western corporate influence. Not saying they should or would endorse, but it wouldn't be _that_ surprising.
- sillysaurusx 3y ago
  Replying to a deleted comment, because I think this is worth underscoring: By explicitly supporting this, China could signal to AI researchers worldwide that China is a safe haven. It’s the materialization of what everyone said would happen: when copyright enthusiasts shut down AI work in America, companies will simply go elsewhere. Imagine if the epicenter of AI research shifted away from the US. This may seem unlikely, but lack of data is the biggest problem that open source LLMs face. Mistral had to completely refuse to disclose their training sources just to release a decent model. Personally, I’m tempted to focus on this full time. It’s what I’ve always wanted to do. I never gave a thought to legal consequences, and with this collection there appears to be none.
  - TerrifiedMouse 3y ago
    > China is a safe haven Heh. It’s a safe haven until your LLM says something the CCP doesn’t like.
    - FartyMcFarter 3y ago
      "Safe"and "haven" are indeed bold worlds when applied to the current Chinese regime...
    - Vetch 3y ago
      This issue is overstated. Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Furthermore, there have been a series of open-weights Chinese models that give reasonable answers to sensitive (in China) questions. Unless you plan to release something customer facing in China, it's not something to stress about. The idea that overly burdensome regulations around open source models in the US shifts the global center of mass of LLMs to China is not implausible. One reason resonant among government officials against opensource LLMs is to keep them from China and Russia. But among the very best embeddings, ~13Bs and perhaps 34B open-weights models are the Chinese ones. The recent DeepSeekCoder also tops EvalPlus: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html For those concerned about over-training, ChatGPT and GPT4 are almost certainly also contaminated and there have been independent confirmations of DeepSeekCoder's strength.
      - rfoo 3y ago
        > Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Especially due to no freedom of the press in China (you need a permit for each published book) they are all already censored. I still don't think China could be a safe haven though, they haven't shot down open source models not because they are nice, it's just that they are incompetent.
        TerrifiedMouse 3y ago
- yorwba 3y ago
  Anna's Archive is already blocked by the Great Firewall https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fannas-archive.org https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fann... , chance of endorsement is nil.
  - sillysaurusx 3y ago
    Is there a way to find out why? It could be anything from "most new websites get blocked by default, or automatically" to "a censor didn’t like a specific thing." I’m not a fan of bowing to censorship (to say the least), but at this point I am even less a fan of being verbally spit on, called a thief, and being included in lawsuits. I’d take the lesser of two evils, since there’s no way to even do any work in the first place in America. (I’ve been depressed for months that the inevitable outcome of all the LLM lawsuits is that book authors will demand licensing fees, which completely kills open source AI models for all but the most funded institutions. I.e. my original work would’ve been completely impossible.)
    - yorwba 3y ago
      Most websites do not get blocked. When it happens, the censors won't tell you, but if there's a search bar, it's usually easy to find content that explains the block: https://annas-archive.org/search?q=Prisoner+of+the+State+Zhao+Ziyang https://annas-archive.org/search?q=Prisoner+of+the+State+Zha...
      - sillysaurusx 3y ago
        Thank you. I have no way of knowing anything about China, so this kind of thing is super helpful. I wouldn’t have even known what to search for to trigger the censors.
        rfoo 3y ago
        In principal it's easy to explain - as long as you have uncensored content on your website, it's going to be blocked. For example, HN is blocked. If it didn't happen, then it's because you are not big enough. A few cases: - FB, Instagram, Reddit, Twitter, Youtube are all blocked: user generated content and without outrageous censorship effort (Douyin has 100k employees dedicated to censor and is top cost center, Weibo 50k IIRC) it is very predictable, just assume such things get blocked. - Google is blocked. They announced in 2010 they didn't want to censor any more. - A censored version of Bing is not blocked. - Dropbox is blocked: the same reason. - Notably, GitHub is NOT blocked. GitHub pages and Gist are blocked. This is because GitHub does censor repos upon government request, plus that when it was once blocked scientist spent effort and managed to reverse the decision. - WhatsApp and Signal are blocked. SMS from Signal is also blocked for all +86 numbers so people simply can't register. - iMessage is not blocked. This is, IMO, an outlier. My guess is it's because Apple + no group chat + incompetence.
- BiteCode_dev 3y ago
  If the AI is accessible to the rest of the world, it's not a problem: the model can query its knowledge and answer in any language since it already is very good at translating things. The problem is rather than, given the economic war between the US and China, the chip ban, the taiwan issues and the GFC, we may not have access to this.
- morelisp 3y ago
  > simply abandoning European and American knowledge is a realistic option. Haha, OK, yeah.
fithisux 3y ago
The biggest theft in history with governments as complices
- sillysaurusx 3y ago
  With the most valuable haul of all: not gold or coin, but knowledge. How terrible. But no, I don’t think governments are accomplices. The smartest move would be for China to wait and see how this plays out. It could significantly shift the epicenter of AI work to China. The single biggest limiting factor for open source LLMs is data, and no one in America or Europe wants to be sued.
  - fragmede 3y ago
    So fund it with crypto and release it on 4chan.
  - pilimi_anna 3y ago
    For what it's worth, we also have millions of previously unreleased (in bulk) books in English, mostly non-fiction, that are available for torrent on our website. High-speed access available for anyone who can do at-scale text extraction, or who can supply us with new collections.
    - sillysaurusx 3y ago
      Anna! I just want to say, I love you. Everything about what you’re doing is heroic. Whoever and wherever you are, thank you. Please focus on your opsec. The more visible you become, the progressively angrier people will get. Don’t do anything silly like edit your Wikipedia page from your house. With that out of the way, someone I know happens to have the original books3 epub files. I think they can be convinced to send them to you. It’s only 200,000 books, but that could theoretically grow your collection by 10% or so. I don’t know whether that would be helpful to you (you’ve far surpassed books3 at this point), but if so, let me know. Given the legal risks, the best course of action for AI companies is probably to ignore English and European books entirely. There is plenty of Chinese data, and the models would learn all the same concepts without exposing anyone to lawsuits.
bomewish 3y ago
Oh my gosh. Duxiu is an absolutely incredible resource. The prospect that it will be fully searchable in a year is incredible!!
- pilimi_anna 3y ago
  It really is
  - Zacharias030 3y ago
    Please elaborate on Duxiu!
bberrry 3y ago
Do LLMs really encode knowledge regardless of language, and tap that knowledge irrespective of which language you query in? My mental model of LLMs being next-word predictors with a long context window suggests they don't. Are there any papers on this?
- LudwigNagasena 3y ago
  What is a better way of predicting the next word if not knowledge encoding?
- gabipurcaru 3y ago
  this is fairly easy to test for yourself. What did you try?
  - bberrry 3y ago
    I don't think it's easy at all without access to the training data. I could ask about some information I find obscure in my native language but I can't be confident someone didn't write about it in English on e.g. reddit.
  - sebzim4500 3y ago
    How exactly are you proposing he tests this without access to hundreds of thousands of dollars worth of compute? Toy models don't work for this kind of thing, small language models behave qualitatively differently from large ones.
- stevenhuang 3y ago
  you may disabuse yourself this mental model by learning about the concept of universal function approximators and the possibility that intelligence itself can be approximated.
- golol 3y ago
  SGD in high dimensions is good at finding efficient, generalizing models. Naturally, it is more efficient to learn a representation of the abstract concept of a table rather than learn representations for the hundreds of different words for tables. I don't have reference but for me it feels very inthuitive that the former is the case.
JCharante 3y ago
Chinese languages have a higher information density which makes me hypothesize that tokenizing works better than in Romance languages which can lead to more efficient LLMs.
- teruakohatu 3y ago
  Right now English language LLMs favour more tokens over fewer tokens and more unique tokens. For example in GPT4, “ Altogether now” is four tokens. I don’t think visual information density is much help.
  - JCharante 3y ago
    I'm just an observer to the space, but I imagine each chinese character (or each word) being a token would be pretty effective, especially as inputting/outputting more tokens requires more vram. Additionally Chinese doesn't have conjugations which I imagine makes tokenization easier. I think it's easier to connect semantic meaning between these tokens/charcaters, because you can relate a character to other characters by their presence in words (most words are 2 characters long (compound words), although there are sometimes words like AB and BC which mean the same thing, but in conversations people will just say A or C and the meaning (AB or BC) is inferred by the context of the conversation, so in a semantic meaning database you would want a connection between A and AB, and to BC & C). You could also have a dimension for each radical that isn't a phonetic component. Radicals do link vaguely related words together, and I think this is a powerful language feature missing in English that could be exploited in the NLP field. I'm not a fluent or proficient speaker, so maybe I'm not the best person to explain things, but before OpenAI's Whisper, speech-to-text in WeChat in Mandarin was way better than any English speech-to-text I've ever used. The structure of the Chinese language and its large userbase would make me bet long on Chinese NLP progress over the west.
    - vasco 3y ago
      Imagine losing WW3 because your language is less efficient to store and query than the enemy, so your war-llms need more resources and are less effective.
Havoc 3y ago
I’m guessing one of the big Chinese firms will outbid everyone. They’re obviously trying to catch up with OpenAI and throwing whatever money necessary at this makes sense as a result
- sillysaurusx 3y ago
  That’s ok. It’s the expected result in a market economy. America is throwing out their chances of making open source LLMs. Copyright holders are demanding license fees for training. That’s analogous to someone demanding licensing fees before you make a YouTube video. No one would be able to do it for free. Whereas it was completely possible to train a high grade LLM for free (clusters are surprisingly accessible for researchers) and it wasn’t till recently that you had to worry about being sued for it. Net result: open source LLMs die, except for companies that can open source their smaller (lamer) models as an upsell for the real ones that anybody cares about. That’s not a world where open source makes a big impact. That’s a world where (metaphorically) GPL software is subservient to business interests for the rest of eternity. Say what you will about whether that’s true, but no business has influence over Emacs, and it’s fantastic, powerful software. No one will be able to make the equivalent open source fantastic LLM in America at this rate.
  - visarga 3y ago
    Doesn't have to be made in America, you can download a LLaMA or Mistral in minutes anywhere. A Chinese base model can be fine-tuned anywhere.
  - colinsane 3y ago
    i'm not sure i follow. you're saying that no company (except the small ones) will be able to open source a LLM. but then you cite software created outside of the traditional company structure as an example of what we'll never have in the LLM space. doesn't your example negate the premise?
    - sillysaurusx 3y ago
      Not at all. Maybe it’s lost to time, but most of the important models were created by academics, not companies, up till recently. GPT-J for example was trained by one person acting alone (Ben Wang). I fine tuned GPT 1.5B on chess games. AI Dungeon fine tuned on fantasy novels. All of this type of work will become impossible with the specter of lawsuits hovering over their heads. EDIT: also most impactful older models were by one person (e.g. Yolo). What I like about ML is that lone wolves can have a big impact.
codedokode 3y ago
AI is a technology that is going to bring ridiculous profits, billions of dollars. Why don't AI researchers want to pay for the books and for the work to write them? They want to train their models for free, and then sell access for money like OpenAI.
- sillysaurusx 3y ago
  Imagine saying that someone should pay for the right to make open source software, before you’ve made anything, or proven any kind of market value. In a world with power law distributions, the vast majority of people will not come close to even millions in profit. At one point I was hoping just to run my own GPT API to self fund my measly $7k/mo burn rate. Then I could live life happy as a clam doing research all day in my little shell. Such a dream is completely impossible when rightsholders demand everyone pay for training data, rather than sharing in the profits of the result. It’s backwards. The actual market profit is what matters. But authors are frothing at the mouth, upset that their work is being used at all without their explicit consent. And some of those authors, like Sarah Silverman and more recently former governor Mike Huckabee, are going after LLM companies and demanding injunctions preventing them from using books at all, under any circumstance, unless they go through authors. Good luck going through 100,000 different authors. The net result is that we’re headed for a world where the YouTube of LLMs will dominate the space, and no one else will be able to run their own platforms. I’ve done everything humanly possible to try to steer us away from that outcome — you like using stable diffusion yourself? Imagine if no one released stable diffusion in the first place. But things are not looking good. Lawsuit against Eleuther for books3, courtesy of Huckabee: https://news.ycombinator.com/item?id=37962244 https://news.ycombinator.com/item?id=37962244 Example of authors being furious, though thankfully with an exception for academic curiosity: https://news.ycombinator.com/item?id=37949012 https://news.ycombinator.com/item?id=37949012 Twitter is completely merciless. Digging up some examples to illustrate the point… Here’s one: look through the replies of someone who released a 3D dataset. https://x.com/mattdeitke/status/1678855859089326080?s=61&t=jQbmCk1JqL7depzFWJNuPA https://x.com/mattdeitke/status/1678855859089326080?s=61&t=j... "Eat shit Matt" (5O likes) "Hope you enjoy the storm" "you make me sick" and on and on. I can tell you firsthand that this isn’t unusual; it’s the default reaction now.
Alifatisk 3y ago
I wonder if LLMs will enable us to access information in chinese much easier now by acting as an interface.
popcalc 3y ago
The guy running this site is undoubtedly intelligent but also foolhardy enough to where it’s going to be hard to feel sorry when his door gets broken down. He started this project after seeing how dangerous of a situation zlib’s spotlight put them in, yet he tried to replicate their success. He succeeded but I recommend he doesn’t take being a free man for granted.
- sillysaurusx 3y ago
  Others will carry the torch. Silk Road was proof of that. Of course, it’s little consolation to Ulbricht, but he deserved to be imprisoned for ordering hits. (Incidentally, I was skeptical of that claim till https://youtu.be/GpMP6Nh3FvU?si=q0KXzP2FNJDin2tW https://youtu.be/GpMP6Nh3FvU?si=q0KXzP2FNJDin2tW.) The benefit of Anna’s Archive is that they have goodwill on their side. They are trying to contribute to academic research directly, whereas libgen seems to prioritize profit (and has to, to survive). Anna’s Archive does that too. The "donations" are actually paying to access features. But by proactively reaching out to the academic community like this, they’ll make a lot of powerful allies. EDIT: on second thought, how does libgen monetize? I thought they ran ads, but I just realized I didn’t verify that. Whoever runs a library like this for altruistic reasons stands to win. The desire to get rich is most of the problem with doing shady things scalably. One can argue that it’s the risk/reward tradeoff, but I suspect as technology makes everything more accessible, an operation like this only costs a few thousand a month. That’s achievable while flying under the radar long enough to hopefully avoid getting in a lot of trouble. Of course, as with death note, everything comes down to: don’t get caught.
  - OfSanguineFire 3y ago
    > how does libgen monetize The original LibGen, under anonymous and altruistic founder bookwarrior, did not aim to monetize but was built purely on volunteer effort. Some of the LibGen domains, however, are forks that put up advertisements.
    - sillysaurusx 3y ago
      That’s so cool.
RecycledEle 3y ago
The Chinese system where all books are in a central database gives them huge advantages in training AI. I just realized that western companies are screwed because of the people who wanted to make a buck by creacting artificial scarcity. I have been predicting for decades that IP laws would bring about the end of our civilization. I did not know how it would happen. Now I know.
- dragonelite 3y ago
  That way to dooming the west will be fine the rest of the world will just do better then they have been doing the last 3 centuries. It will create a level playing field and the current tier like system. We are going back to the time where GDP was just how many farmers did your country have. But in 2000s it will be how many STEM people does your country have and how much IP one can create and license. tech IP is like a global rent from everybody that consumes tech.
mensetmanusman 3y ago
The Chinese have a very different cultural response to copying, and often see it as a form of honoring. They would be more than happy if everyone on earth had every Chinese work in existence.
- deleted 3y ago
  [deleted]
- faitswulff 3y ago
  That seems essentialist. Chinese incentives are aligned with a lack of IP protection right now. For instance, IIRC, bunny’s explanation of gongkai (https://www.bunniestudios.com/blog/?p=4297 https://www.bunniestudios.com/blog/?p=4297) explains that Chinese companies freely trade chip designs because they make money off of manufacturing, regardless of who designed it.
  - NotSuspicious 3y ago
    This seems to me a much more moral way of doing things. Actually making the stuff is what's important.
    - error_logic 3y ago
      Patents and copyright are intended to encourage publishing of information (and profitability of individuals so they can establish themselves with economic mobility and unlock more potential investments for the world) rather than keeping it as trade secret. That doesn't mean they always do their job, but neither will there always be people running a company or a nation who can maintain such sharing in the face of competitive pressures during different times.
  - yorwba 3y ago
    > Chinese incentives are aligned with a lack of IP protection right now. Even that is essentialist. In some areas, sharing IP is beneficial to the originators and incentives are aligned. In other areas, not so much. Duxiu makes money selling access to their collection of scanned books. Their incentives are not aligned with having others use those scans for free. Currently, such cases seem to be mostly handled under the most general provisions of the Anti-Unfair Competition Law (反不正当竞争法), but new amendments are likely to make it more explicitly illegal https://www.whitecase.com/insight-alert/china-releases-draft-amendments-anti-unfair-competition-law https://www.whitecase.com/insight-alert/china-releases-draft... ("Improperly obtaining or using another business operator's commercial data")
richwater 3y ago
[flagged]
xeckr 3y ago
Does "Chinese nonfiction" include Mao's bibliography and "Xi Jinping thought"? Do you realize how all-consuming China's censorship apparatus is, and that this censorship introduces all sorts of biases in their literary corpus which you now want to be used as training data for LLMs?
- staticman2 3y ago
  That's why you fine tune models, right? To make sure the undesirable things in their data set don't lead to undesirable output.
  - xeckr 3y ago
    Fine-tuning will never get rid of all biases introduced by the training data, some of which are subtle enough that you won't even know that you should apply fine-tuning techniques to try to mitigate them. For example, did you know that there is a trend for physics textbooks to be dedicated to Xi Jinping? If you didn't know this, because who has time to sift through millions of books to find undesirable biases, isn't it clear that your LLM will develop subtle but favourable biases towards the CCP?
tway223 3y ago
This collection has been on the internet for quite a while, likely started around 2015-ish. It is highly duplicated and I suspect the total number is around 4 million books. Still a lot. The source was from a company named DuXiu, or previously SuperStar. They collaborated with the libraries around China and scanned their collections since early 2000-ish. Before that I think they just bought some junk books from recycling stations based on the quality of early samples. Many of the books are translated versions of the textbooks from the west (most likely the US) and many are pure political propaganda junk. Some literature and history stuff which were published when censorship wasn't so extreme. Many of the Chinese tech companies should have access to this collection (especially Baidu for sure) but the books were not censored based on today's standards so I doubt any of them would openly use them not only due to the copyright issue but also the political risks.
gomijacogeo 3y ago
Searle's finally going to get his Chinese room.

235 points by sillysaurusx 3y ago

This collection is 40 times larger than books3. No one seems to care at all about copyright. It’s tempting to leave the English speaking world behind and focus on someplace that supports academic work. Will people call this theft? No one has so far. In the meantime, simply abandoning European and American knowledge is a realistic option. LLMs will learn in Chinese. On the other hand, it would suck much more to be thrown in a Chinese prison. So probably don’t do this if you plan on visiting China. (If China reacts by fully endorsing this, that would be the uno reverse of the century.)

squigz 3y ago
Sharing of information is more important to me personally than business interests
deleted 3y ago
[deleted]
alex_duf 3y ago
China may find it attractive to have a model trained on their languages and cultures. It would send a message of power, and ensure that local tech companies will use a model with their influence rather than western corporate influence. Not saying they should or would endorse, but it wouldn't be _that_ surprising.
sillysaurusx 3y ago
Replying to a deleted comment, because I think this is worth underscoring: By explicitly supporting this, China could signal to AI researchers worldwide that China is a safe haven. It’s the materialization of what everyone said would happen: when copyright enthusiasts shut down AI work in America, companies will simply go elsewhere. Imagine if the epicenter of AI research shifted away from the US. This may seem unlikely, but lack of data is the biggest problem that open source LLMs face. Mistral had to completely refuse to disclose their training sources just to release a decent model. Personally, I’m tempted to focus on this full time. It’s what I’ve always wanted to do. I never gave a thought to legal consequences, and with this collection there appears to be none.
- TerrifiedMouse 3y ago
  > China is a safe haven Heh. It’s a safe haven until your LLM says something the CCP doesn’t like.
  - FartyMcFarter 3y ago
    "Safe"and "haven" are indeed bold worlds when applied to the current Chinese regime...
  - Vetch 3y ago
    This issue is overstated. Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Furthermore, there have been a series of open-weights Chinese models that give reasonable answers to sensitive (in China) questions. Unless you plan to release something customer facing in China, it's not something to stress about. The idea that overly burdensome regulations around open source models in the US shifts the global center of mass of LLMs to China is not implausible. One reason resonant among government officials against opensource LLMs is to keep them from China and Russia. But among the very best embeddings, ~13Bs and perhaps 34B open-weights models are the Chinese ones. The recent DeepSeekCoder also tops EvalPlus: https://evalplus.github.io/leaderboard.html https://evalplus.github.io/leaderboard.html For those concerned about over-training, ChatGPT and GPT4 are almost certainly also contaminated and there have been independent confirmations of DeepSeekCoder's strength.
    - rfoo 3y ago
      > Given the material is in Chinese, it stands to reason the bulk (but of course not all) of it will not be in violation of whatever policies. Especially due to no freedom of the press in China (you need a permit for each published book) they are all already censored. I still don't think China could be a safe haven though, they haven't shot down open source models not because they are nice, it's just that they are incompetent.
      - TerrifiedMouse 3y ago
yorwba 3y ago
Anna's Archive is already blocked by the Great Firewall https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fannas-archive.org https://blocky.greatfire.org/detail/535678/https%3A%2F%2Fann... , chance of endorsement is nil.
- sillysaurusx 3y ago
  Is there a way to find out why? It could be anything from "most new websites get blocked by default, or automatically" to "a censor didn’t like a specific thing." I’m not a fan of bowing to censorship (to say the least), but at this point I am even less a fan of being verbally spit on, called a thief, and being included in lawsuits. I’d take the lesser of two evils, since there’s no way to even do any work in the first place in America. (I’ve been depressed for months that the inevitable outcome of all the LLM lawsuits is that book authors will demand licensing fees, which completely kills open source AI models for all but the most funded institutions. I.e. my original work would’ve been completely impossible.)
  - yorwba 3y ago
    Most websites do not get blocked. When it happens, the censors won't tell you, but if there's a search bar, it's usually easy to find content that explains the block: https://annas-archive.org/search?q=Prisoner+of+the+State+Zhao+Ziyang https://annas-archive.org/search?q=Prisoner+of+the+State+Zha...
    - sillysaurusx 3y ago
      Thank you. I have no way of knowing anything about China, so this kind of thing is super helpful. I wouldn’t have even known what to search for to trigger the censors.
      - rfoo 3y ago
        In principal it's easy to explain - as long as you have uncensored content on your website, it's going to be blocked. For example, HN is blocked. If it didn't happen, then it's because you are not big enough. A few cases: - FB, Instagram, Reddit, Twitter, Youtube are all blocked: user generated content and without outrageous censorship effort (Douyin has 100k employees dedicated to censor and is top cost center, Weibo 50k IIRC) it is very predictable, just assume such things get blocked. - Google is blocked. They announced in 2010 they didn't want to censor any more. - A censored version of Bing is not blocked. - Dropbox is blocked: the same reason. - Notably, GitHub is NOT blocked. GitHub pages and Gist are blocked. This is because GitHub does censor repos upon government request, plus that when it was once blocked scientist spent effort and managed to reverse the decision. - WhatsApp and Signal are blocked. SMS from Signal is also blocked for all +86 numbers so people simply can't register. - iMessage is not blocked. This is, IMO, an outlier. My guess is it's because Apple + no group chat + incompetence.
BiteCode_dev 3y ago
If the AI is accessible to the rest of the world, it's not a problem: the model can query its knowledge and answer in any language since it already is very good at translating things. The problem is rather than, given the economic war between the US and China, the chip ban, the taiwan issues and the GFC, we may not have access to this.
morelisp 3y ago
> simply abandoning European and American knowledge is a realistic option. Haha, OK, yeah.

fithisux 3y ago

The biggest theft in history with governments as complices

sillysaurusx 3y ago
With the most valuable haul of all: not gold or coin, but knowledge. How terrible. But no, I don’t think governments are accomplices. The smartest move would be for China to wait and see how this plays out. It could significantly shift the epicenter of AI work to China. The single biggest limiting factor for open source LLMs is data, and no one in America or Europe wants to be sued.
- fragmede 3y ago
  So fund it with crypto and release it on 4chan.
- pilimi_anna 3y ago
  For what it's worth, we also have millions of previously unreleased (in bulk) books in English, mostly non-fiction, that are available for torrent on our website. High-speed access available for anyone who can do at-scale text extraction, or who can supply us with new collections.
  - sillysaurusx 3y ago
    Anna! I just want to say, I love you. Everything about what you’re doing is heroic. Whoever and wherever you are, thank you. Please focus on your opsec. The more visible you become, the progressively angrier people will get. Don’t do anything silly like edit your Wikipedia page from your house. With that out of the way, someone I know happens to have the original books3 epub files. I think they can be convinced to send them to you. It’s only 200,000 books, but that could theoretically grow your collection by 10% or so. I don’t know whether that would be helpful to you (you’ve far surpassed books3 at this point), but if so, let me know. Given the legal risks, the best course of action for AI companies is probably to ignore English and European books entirely. There is plenty of Chinese data, and the models would learn all the same concepts without exposing anyone to lawsuits.

bomewish 3y ago

Oh my gosh. Duxiu is an absolutely incredible resource. The prospect that it will be fully searchable in a year is incredible!!

pilimi_anna 3y ago
It really is
- Zacharias030 3y ago
  Please elaborate on Duxiu!

bberrry 3y ago

Do LLMs really encode knowledge regardless of language, and tap that knowledge irrespective of which language you query in? My mental model of LLMs being next-word predictors with a long context window suggests they don't. Are there any papers on this?

LudwigNagasena 3y ago
What is a better way of predicting the next word if not knowledge encoding?
gabipurcaru 3y ago
this is fairly easy to test for yourself. What did you try?
- bberrry 3y ago
  I don't think it's easy at all without access to the training data. I could ask about some information I find obscure in my native language but I can't be confident someone didn't write about it in English on e.g. reddit.
- sebzim4500 3y ago
  How exactly are you proposing he tests this without access to hundreds of thousands of dollars worth of compute? Toy models don't work for this kind of thing, small language models behave qualitatively differently from large ones.
stevenhuang 3y ago
you may disabuse yourself this mental model by learning about the concept of universal function approximators and the possibility that intelligence itself can be approximated.
golol 3y ago
SGD in high dimensions is good at finding efficient, generalizing models. Naturally, it is more efficient to learn a representation of the abstract concept of a table rather than learn representations for the hundreds of different words for tables. I don't have reference but for me it feels very inthuitive that the former is the case.

JCharante 3y ago

Chinese languages have a higher information density which makes me hypothesize that tokenizing works better than in Romance languages which can lead to more efficient LLMs.

teruakohatu 3y ago
Right now English language LLMs favour more tokens over fewer tokens and more unique tokens. For example in GPT4, “ Altogether now” is four tokens. I don’t think visual information density is much help.
- JCharante 3y ago
  I'm just an observer to the space, but I imagine each chinese character (or each word) being a token would be pretty effective, especially as inputting/outputting more tokens requires more vram. Additionally Chinese doesn't have conjugations which I imagine makes tokenization easier. I think it's easier to connect semantic meaning between these tokens/charcaters, because you can relate a character to other characters by their presence in words (most words are 2 characters long (compound words), although there are sometimes words like AB and BC which mean the same thing, but in conversations people will just say A or C and the meaning (AB or BC) is inferred by the context of the conversation, so in a semantic meaning database you would want a connection between A and AB, and to BC & C). You could also have a dimension for each radical that isn't a phonetic component. Radicals do link vaguely related words together, and I think this is a powerful language feature missing in English that could be exploited in the NLP field. I'm not a fluent or proficient speaker, so maybe I'm not the best person to explain things, but before OpenAI's Whisper, speech-to-text in WeChat in Mandarin was way better than any English speech-to-text I've ever used. The structure of the Chinese language and its large userbase would make me bet long on Chinese NLP progress over the west.
  - vasco 3y ago
    Imagine losing WW3 because your language is less efficient to store and query than the enemy, so your war-llms need more resources and are less effective.

Havoc 3y ago

I’m guessing one of the big Chinese firms will outbid everyone. They’re obviously trying to catch up with OpenAI and throwing whatever money necessary at this makes sense as a result

sillysaurusx 3y ago
That’s ok. It’s the expected result in a market economy. America is throwing out their chances of making open source LLMs. Copyright holders are demanding license fees for training. That’s analogous to someone demanding licensing fees before you make a YouTube video. No one would be able to do it for free. Whereas it was completely possible to train a high grade LLM for free (clusters are surprisingly accessible for researchers) and it wasn’t till recently that you had to worry about being sued for it. Net result: open source LLMs die, except for companies that can open source their smaller (lamer) models as an upsell for the real ones that anybody cares about. That’s not a world where open source makes a big impact. That’s a world where (metaphorically) GPL software is subservient to business interests for the rest of eternity. Say what you will about whether that’s true, but no business has influence over Emacs, and it’s fantastic, powerful software. No one will be able to make the equivalent open source fantastic LLM in America at this rate.
- visarga 3y ago
  Doesn't have to be made in America, you can download a LLaMA or Mistral in minutes anywhere. A Chinese base model can be fine-tuned anywhere.
- colinsane 3y ago
  i'm not sure i follow. you're saying that no company (except the small ones) will be able to open source a LLM. but then you cite software created outside of the traditional company structure as an example of what we'll never have in the LLM space. doesn't your example negate the premise?
  - sillysaurusx 3y ago
    Not at all. Maybe it’s lost to time, but most of the important models were created by academics, not companies, up till recently. GPT-J for example was trained by one person acting alone (Ben Wang). I fine tuned GPT 1.5B on chess games. AI Dungeon fine tuned on fantasy novels. All of this type of work will become impossible with the specter of lawsuits hovering over their heads. EDIT: also most impactful older models were by one person (e.g. Yolo). What I like about ML is that lone wolves can have a big impact.

codedokode 3y ago

AI is a technology that is going to bring ridiculous profits, billions of dollars. Why don't AI researchers want to pay for the books and for the work to write them? They want to train their models for free, and then sell access for money like OpenAI.

sillysaurusx 3y ago
Imagine saying that someone should pay for the right to make open source software, before you’ve made anything, or proven any kind of market value. In a world with power law distributions, the vast majority of people will not come close to even millions in profit. At one point I was hoping just to run my own GPT API to self fund my measly $7k/mo burn rate. Then I could live life happy as a clam doing research all day in my little shell. Such a dream is completely impossible when rightsholders demand everyone pay for training data, rather than sharing in the profits of the result. It’s backwards. The actual market profit is what matters. But authors are frothing at the mouth, upset that their work is being used at all without their explicit consent. And some of those authors, like Sarah Silverman and more recently former governor Mike Huckabee, are going after LLM companies and demanding injunctions preventing them from using books at all, under any circumstance, unless they go through authors. Good luck going through 100,000 different authors. The net result is that we’re headed for a world where the YouTube of LLMs will dominate the space, and no one else will be able to run their own platforms. I’ve done everything humanly possible to try to steer us away from that outcome — you like using stable diffusion yourself? Imagine if no one released stable diffusion in the first place. But things are not looking good. Lawsuit against Eleuther for books3, courtesy of Huckabee: https://news.ycombinator.com/item?id=37962244 https://news.ycombinator.com/item?id=37962244 Example of authors being furious, though thankfully with an exception for academic curiosity: https://news.ycombinator.com/item?id=37949012 https://news.ycombinator.com/item?id=37949012 Twitter is completely merciless. Digging up some examples to illustrate the point… Here’s one: look through the replies of someone who released a 3D dataset. https://x.com/mattdeitke/status/1678855859089326080?s=61&t=jQbmCk1JqL7depzFWJNuPA https://x.com/mattdeitke/status/1678855859089326080?s=61&t=j... "Eat shit Matt" (5O likes) "Hope you enjoy the storm" "you make me sick" and on and on. I can tell you firsthand that this isn’t unusual; it’s the default reaction now.

Alifatisk 3y ago

I wonder if LLMs will enable us to access information in chinese much easier now by acting as an interface.

popcalc 3y ago

The guy running this site is undoubtedly intelligent but also foolhardy enough to where it’s going to be hard to feel sorry when his door gets broken down. He started this project after seeing how dangerous of a situation zlib’s spotlight put them in, yet he tried to replicate their success. He succeeded but I recommend he doesn’t take being a free man for granted.

sillysaurusx 3y ago
Others will carry the torch. Silk Road was proof of that. Of course, it’s little consolation to Ulbricht, but he deserved to be imprisoned for ordering hits. (Incidentally, I was skeptical of that claim till https://youtu.be/GpMP6Nh3FvU?si=q0KXzP2FNJDin2tW https://youtu.be/GpMP6Nh3FvU?si=q0KXzP2FNJDin2tW.) The benefit of Anna’s Archive is that they have goodwill on their side. They are trying to contribute to academic research directly, whereas libgen seems to prioritize profit (and has to, to survive). Anna’s Archive does that too. The "donations" are actually paying to access features. But by proactively reaching out to the academic community like this, they’ll make a lot of powerful allies. EDIT: on second thought, how does libgen monetize? I thought they ran ads, but I just realized I didn’t verify that. Whoever runs a library like this for altruistic reasons stands to win. The desire to get rich is most of the problem with doing shady things scalably. One can argue that it’s the risk/reward tradeoff, but I suspect as technology makes everything more accessible, an operation like this only costs a few thousand a month. That’s achievable while flying under the radar long enough to hopefully avoid getting in a lot of trouble. Of course, as with death note, everything comes down to: don’t get caught.
- OfSanguineFire 3y ago
  > how does libgen monetize The original LibGen, under anonymous and altruistic founder bookwarrior, did not aim to monetize but was built purely on volunteer effort. Some of the LibGen domains, however, are forks that put up advertisements.
  - sillysaurusx 3y ago
    That’s so cool.

RecycledEle 3y ago

The Chinese system where all books are in a central database gives them huge advantages in training AI. I just realized that western companies are screwed because of the people who wanted to make a buck by creacting artificial scarcity. I have been predicting for decades that IP laws would bring about the end of our civilization. I did not know how it would happen. Now I know.

dragonelite 3y ago
That way to dooming the west will be fine the rest of the world will just do better then they have been doing the last 3 centuries. It will create a level playing field and the current tier like system. We are going back to the time where GDP was just how many farmers did your country have. But in 2000s it will be how many STEM people does your country have and how much IP one can create and license. tech IP is like a global rent from everybody that consumes tech.

mensetmanusman 3y ago

The Chinese have a very different cultural response to copying, and often see it as a form of honoring. They would be more than happy if everyone on earth had every Chinese work in existence.

deleted 3y ago
[deleted]
faitswulff 3y ago
That seems essentialist. Chinese incentives are aligned with a lack of IP protection right now. For instance, IIRC, bunny’s explanation of gongkai (https://www.bunniestudios.com/blog/?p=4297 https://www.bunniestudios.com/blog/?p=4297) explains that Chinese companies freely trade chip designs because they make money off of manufacturing, regardless of who designed it.
- NotSuspicious 3y ago
  This seems to me a much more moral way of doing things. Actually making the stuff is what's important.
  - error_logic 3y ago
    Patents and copyright are intended to encourage publishing of information (and profitability of individuals so they can establish themselves with economic mobility and unlock more potential investments for the world) rather than keeping it as trade secret. That doesn't mean they always do their job, but neither will there always be people running a company or a nation who can maintain such sharing in the face of competitive pressures during different times.
- yorwba 3y ago
  > Chinese incentives are aligned with a lack of IP protection right now. Even that is essentialist. In some areas, sharing IP is beneficial to the originators and incentives are aligned. In other areas, not so much. Duxiu makes money selling access to their collection of scanned books. Their incentives are not aligned with having others use those scans for free. Currently, such cases seem to be mostly handled under the most general provisions of the Anti-Unfair Competition Law (反不正当竞争法), but new amendments are likely to make it more explicitly illegal https://www.whitecase.com/insight-alert/china-releases-draft-amendments-anti-unfair-competition-law https://www.whitecase.com/insight-alert/china-releases-draft... ("Improperly obtaining or using another business operator's commercial data")

richwater 3y ago

[flagged]

xeckr 3y ago

Does "Chinese nonfiction" include Mao's bibliography and "Xi Jinping thought"? Do you realize how all-consuming China's censorship apparatus is, and that this censorship introduces all sorts of biases in their literary corpus which you now want to be used as training data for LLMs?

staticman2 3y ago
That's why you fine tune models, right? To make sure the undesirable things in their data set don't lead to undesirable output.
- xeckr 3y ago
  Fine-tuning will never get rid of all biases introduced by the training data, some of which are subtle enough that you won't even know that you should apply fine-tuning techniques to try to mitigate them. For example, did you know that there is a trend for physics textbooks to be dedicated to Xi Jinping? If you didn't know this, because who has time to sift through millions of books to find undesirable biases, isn't it clear that your LLM will develop subtle but favourable biases towards the CCP?

tway223 3y ago

This collection has been on the internet for quite a while, likely started around 2015-ish. It is highly duplicated and I suspect the total number is around 4 million books. Still a lot. The source was from a company named DuXiu, or previously SuperStar. They collaborated with the libraries around China and scanned their collections since early 2000-ish. Before that I think they just bought some junk books from recycling stations based on the quality of early samples. Many of the books are translated versions of the textbooks from the west (most likely the US) and many are pure political propaganda junk. Some literature and history stuff which were published when censorship wasn't so extreme. Many of the Chinese tech companies should have access to this collection (especially Baidu for sure) but the books were not censored based on today's standards so I doubt any of them would openly use them not only due to the copyright issue but also the political risks.

gomijacogeo 3y ago

Searle's finally going to get his Chinese room.