4 ms·

> I wonder if there's even unused data out there? I think you're massively underestimating the amount of data out there. The challenge is how to access and cat

by nolok 3y ago

> I wonder if there's even unused data out there? I think you're massively underestimating the amount of data out there. The challenge is how to access and categorize that data. Every usenet message, forum post from old school bbs to php forums to modern javascript abomination and closed discord boards, every email, every text message, every irc message, ... Those are probably a pain point to access due to rules and regulation and yada yada, but that alone dwarfs the 15 trillions, and you've not even started on actual quality content. (not saying these would specifically be good for llm, just answering to your "unused data" assessment)

MyFirstSass 3y ago
That's a good point, though i already thought the OpenAI team had been very agressive in sweeping both reddit, usenet, + various illegal megatorrents of books, forum dumps etc. I remember there were some controversy around it on twitter a few months ago. One thing though is books/content/media from other language spheres though that could probably at least 10x the size of the data, and as far as i know translation starts to work rather well in these larger models so it would probably just plug right into the knowledgegraph for all languages?
- vidarh 3y ago
  There are still vast amounts of data locked up behind login screens etc., though. E.g. to the foreign language data, a lot of national libraries around the world are either not even fully digitized yet or have lots of locked-down content. The Norwegian one is pretty open, but there's still huge amounts (like most newspapers newer than a century or so) that is either only available based on geolocation (I have my VPN for genealogy because of that - I'm Norwegian but live in the UK, and it's a nuisance), or only in a physical library in Norway. Similarly I was looking for something from the British Library at one point and it was behind a paywall (a copying fee). I have no idea how to even start to estimate how much data is locked down like that, and it's harder yet to try to figure out which parts of that it'd be possible to negotiate access to for various players, and what they can circumvent (e.g. say by buying book collections and the like - OpenAI is large enough by market cap it could afford to buy some of the largest extant publishers, for example, if they thought it gave them sufficient benefits).
  - MyFirstSass 3y ago
    That is incredibly interesting to me because i've heard both historians, linguists and people "just not from the anglosphere" complain about just how isolated and limited our language, cultural perspectives are. In other words if LLM's could somehow bridge that gap through both regions and time i'm pretty sure something magical could happen, different than the already a bit tired and conformist "echo chamber" like quality to LLM's mostly trained on reddit, corporate speak, and anglo pop culture, or even just western thought in general.
    - vidarh 3y ago
      It's been pretty fascinating. ChatGPT clearly has a very small Norwegian corpus, but it can not only translate to and from both Norwegian written languages (they're more like dialects - they're mutually intelligible) but can make a passable attempt at translating into at least some highly localized dialects, and can explain key differences between certain sociolets. And it took only a slight explanation to get it to give a plausible translation into this weird sociolect constructed by the circle around a fringe Maoist group from the 70's that wanted to sound more working class and adopted a bunch of affections that does not match any "natural" Norwegian dialect (several members ended up as prominent authors, and so it spread wider than the size of the group otherwise would have allowed for). That said, English and Norwegian are pretty close. How well it will handle languages with more significant differences without larger amounts of tokens is another matter. Even for pretty small language groups there ought to be enough, though.

by nolok 3y ago

MyFirstSass 3y ago

That's a good point, though i already thought the OpenAI team had been very agressive in sweeping both reddit, usenet, + various illegal megatorrents of books, forum dumps etc. I remember there were some controversy around it on twitter a few months ago. One thing though is books/content/media from other language spheres though that could probably at least 10x the size of the data, and as far as i know translation starts to work rather well in these larger models so it would probably just plug right into the knowledgegraph for all languages?

vidarh 3y ago
There are still vast amounts of data locked up behind login screens etc., though. E.g. to the foreign language data, a lot of national libraries around the world are either not even fully digitized yet or have lots of locked-down content. The Norwegian one is pretty open, but there's still huge amounts (like most newspapers newer than a century or so) that is either only available based on geolocation (I have my VPN for genealogy because of that - I'm Norwegian but live in the UK, and it's a nuisance), or only in a physical library in Norway. Similarly I was looking for something from the British Library at one point and it was behind a paywall (a copying fee). I have no idea how to even start to estimate how much data is locked down like that, and it's harder yet to try to figure out which parts of that it'd be possible to negotiate access to for various players, and what they can circumvent (e.g. say by buying book collections and the like - OpenAI is large enough by market cap it could afford to buy some of the largest extant publishers, for example, if they thought it gave them sufficient benefits).
- MyFirstSass 3y ago
  That is incredibly interesting to me because i've heard both historians, linguists and people "just not from the anglosphere" complain about just how isolated and limited our language, cultural perspectives are. In other words if LLM's could somehow bridge that gap through both regions and time i'm pretty sure something magical could happen, different than the already a bit tired and conformist "echo chamber" like quality to LLM's mostly trained on reddit, corporate speak, and anglo pop culture, or even just western thought in general.
  - vidarh 3y ago
    It's been pretty fascinating. ChatGPT clearly has a very small Norwegian corpus, but it can not only translate to and from both Norwegian written languages (they're more like dialects - they're mutually intelligible) but can make a passable attempt at translating into at least some highly localized dialects, and can explain key differences between certain sociolets. And it took only a slight explanation to get it to give a plausible translation into this weird sociolect constructed by the circle around a fringe Maoist group from the 70's that wanted to sound more working class and adopted a bunch of affections that does not match any "natural" Norwegian dialect (several members ended up as prominent authors, and so it spread wider than the size of the group otherwise would have allowed for). That said, English and Norwegian are pretty close. How well it will handle languages with more significant differences without larger amounts of tokens is another matter. Even for pretty small language groups there ought to be enough, though.