9 ms·

I know everyone's (as usual) "amazed" by this, but here are a few stark observations: (1) a linguist could do this (come up with a toy language, conjugations, a

by dvt 4y ago

I know everyone's (as usual) "amazed" by this, but here are a few stark observations: (1) a linguist could do this (come up with a toy language, conjugations, and sentence structure, probably the Python code too) in like 30 minutes, (2) it's disappointing that the AI constantly messes up very simple grammar rules (rules a grade-schooler could understand)—I'm actually very surprised it doesn't have hardcoded grammar rules it can merely apply to different tokens—, (3) "Here’s the final code ChatGPT generated"—yeah, after like 10 rounds of debugging. Come on, give me a break. Lastly, this is actually not really "inventing" a language, as it's literally translated 1:1 from English with pretty common grammar cases. I was recently in Donostia/San Sebastian, where Basque is spoken (interestingly, Basque is a language isolate). Grammar rules there are completely alien[1] (though still human). Expecting it to come up with something like that would be interesting. As far as I can see, though, it's still a far cry even from other "invented" languages like Tolkien's Elvish or even Star Trek's Klingon. [1] https://en.wikipedia.org/wiki/Basque_grammar https://en.wikipedia.org/wiki/Basque_grammar

pfortuny 4y ago
All the text in the post reads like an easy exercise in linguistics. I would not be surprised if there are hundreds of these in the training texts.
Scarblac 4y ago
ChatGPT doesn't have much trouble writing in different languages (eg asking for "the same but In Dutch" seems to work well most of the time) which would be harder with hardcoded language rules, I think.
stavros 4y ago
This reads like a "my dog can sing!" "yeah, but it's too flat in parts" comment.
- ogogmad 4y ago
  I think people don't really understand what's happening, and therefore why it's impressive that it's happening.
  - stavros 4y ago
    Probably? I can't fathom how people are unimpressed by an AI that can output coherent sentences on a topic, let alone invent a language!
  - dwohnitmok 4y ago
    I dunno. The people who I talk to who do seem to understand what's happening seem even more impressed than the average layperson (mainly that such a fundamentally simple technique is paying off with such crazy dividends).
    - dvt 4y ago
      > mainly that such a fundamentally simple technique is paying off with such crazy dividends I will concede that I am amazed by this. Even more generally, I'm even more amazed by Markov chains: an even simpler technique that can generate crazy dividends as well.
      - ogogmad 4y ago
        [edit: Was snark. Sorry. I also may have misunderstood the parent comment.]
        dvt 4y ago
        What a weirdly snarky comment (it's like your third one in this thread, maybe you need to take a breather). I wasn't comparing them, though there's an obvious stochastic argument to be made; I was just commenting on the former's lack of complexity and still fascinatingly good results, I'm not sure what your point is.
        krackers 4y ago
        Markov chains are actually not a bad comparison from what i can see as an utter layman. At a high level seems like the difference between transformer-models and markov chains is that instead of the next token being determined purely based on n-gram frequencies on the training corpus, you have the attention mechanism so in some sense it has access to the entire previous context as well as some latent space of learned embeddings. But at the end of the day it still is just a fancy way to predict the next token, so that's why it's surprising you get all this neat behavior.
- deleted 4y ago
  [deleted]
fnordpiglet 4y ago
I find it amazing from the point of view that this is not a linguist or a grade schooler, but a general purpose computer program. Have you seen a computer program that can do these things, especially one that wasn’t written to do these things but can also turn itself into a plausible linux box, translate python to rust, and all sorts of other wacky party tricks? That’s the amazing thing - not that the feats themselves are novel or unique, but they’re novel and unique for a a general purpose computer program to be so adaptive and flexible in both what it takes as input but also how it interprets that input into remarkably coherent and salient outputs. Taking into account it’s a beta and that the underlying tech is really new, you can extrapolate that some of these gaps can be tackled - then will you be amazed?
- dvt 4y ago
  > this is not a linguist or a grade schooler, but a general purpose computer program This is sleight of hand. The training data almost certainly includes thousands of grade schoolers and linguists. Your argument seems to be that this is all generated ad nihilum from first principles (it's "just" a computer program bro), but that's not how it actually came to be. Rather, it was trained on literally terrabytes and terrabytes of human-generated data. So, in a sense, it is actually a linguist. Just a really bad one.
  - fnordpiglet 4y ago
    It’s still a computer program with some state attached. That’s literally all it is. There are no linguists in the program. That it is able to distill the knowledge and ability of a linguist by parsing ascii text into byte buffers and doing some fancy math on the bytes should be breathtaking. The sleight of hand would be if it had a mechanical Turk and there was a linguist. The fact that it collected knowledge to have knowledge and can apply that knowledge is no different than a linguist for sure, but that it’s a computer program distilling binary data into abstract concepts at the level of a grade schooler or a linguist (not sure who is more sophisticated) is mind bending. If you can’t see it, you’ve lost sight of things my friend. I hope you find it again. The world is a beautiful place.
wetpaws 4y ago
Haters gonna hate
asveikau 4y ago
Yeah, I was reading this and thinking it's mostly substituting English for another vocabulary. It's even referencing English when it comes up with something new, eg. referencing use of the "sl" consonant cluster in English.
fpgaminer 4y ago
I'm not a linguist, but wanted to invent a basic language for use in an escape room I'm building. Tools like this are a great help to people like me; hence the amazement. (I had to use GPT-3 and NovelAI, since this was two weeks ago)
pmontra 4y ago
This is about the same argument we can do about Google Translate or text to speech. Any professional translator can take a web page and give me a better translation than GT. However I get the GT translation in a few seconds and not in maybe hours, after I signed a contract with the translator. And text to speech would allow me to post an audio version of this reply in a good English accent instead of my inconsistent foreign one. Or paying a professional speaker, after signing a contract. Inventing a language is maybe useless but using ChatGPT as a tool to increase our speed is definitely useful.
CamperBob2 4y ago
"This talking dog makes embarrassing grammatical errors. Besides that, it learned everything it knows from humans, not from other dogs. I don't see the big deal." Anyone familiar with Chomsky's school of thought here? I'm not sure that "it's just a language model" is the effective dismissal that a lot of people think it is.

by dvt 4y ago

pfortuny 4y ago

All the text in the post reads like an easy exercise in linguistics. I would not be surprised if there are hundreds of these in the training texts.

Scarblac 4y ago

ChatGPT doesn't have much trouble writing in different languages (eg asking for "the same but In Dutch" seems to work well most of the time) which would be harder with hardcoded language rules, I think.

stavros 4y ago

This reads like a "my dog can sing!" "yeah, but it's too flat in parts" comment.

ogogmad 4y ago
I think people don't really understand what's happening, and therefore why it's impressive that it's happening.
- stavros 4y ago
  Probably? I can't fathom how people are unimpressed by an AI that can output coherent sentences on a topic, let alone invent a language!
- dwohnitmok 4y ago
  I dunno. The people who I talk to who do seem to understand what's happening seem even more impressed than the average layperson (mainly that such a fundamentally simple technique is paying off with such crazy dividends).
  - dvt 4y ago
    > mainly that such a fundamentally simple technique is paying off with such crazy dividends I will concede that I am amazed by this. Even more generally, I'm even more amazed by Markov chains: an even simpler technique that can generate crazy dividends as well.
    - ogogmad 4y ago
      [edit: Was snark. Sorry. I also may have misunderstood the parent comment.]
      - dvt 4y ago
        What a weirdly snarky comment (it's like your third one in this thread, maybe you need to take a breather). I wasn't comparing them, though there's an obvious stochastic argument to be made; I was just commenting on the former's lack of complexity and still fascinatingly good results, I'm not sure what your point is.
        krackers 4y ago
        Markov chains are actually not a bad comparison from what i can see as an utter layman. At a high level seems like the difference between transformer-models and markov chains is that instead of the next token being determined purely based on n-gram frequencies on the training corpus, you have the attention mechanism so in some sense it has access to the entire previous context as well as some latent space of learned embeddings. But at the end of the day it still is just a fancy way to predict the next token, so that's why it's surprising you get all this neat behavior.
deleted 4y ago
[deleted]

fnordpiglet 4y ago

I find it amazing from the point of view that this is not a linguist or a grade schooler, but a general purpose computer program. Have you seen a computer program that can do these things, especially one that wasn’t written to do these things but can also turn itself into a plausible linux box, translate python to rust, and all sorts of other wacky party tricks? That’s the amazing thing - not that the feats themselves are novel or unique, but they’re novel and unique for a a general purpose computer program to be so adaptive and flexible in both what it takes as input but also how it interprets that input into remarkably coherent and salient outputs. Taking into account it’s a beta and that the underlying tech is really new, you can extrapolate that some of these gaps can be tackled - then will you be amazed?

dvt 4y ago
> this is not a linguist or a grade schooler, but a general purpose computer program This is sleight of hand. The training data almost certainly includes thousands of grade schoolers and linguists. Your argument seems to be that this is all generated ad nihilum from first principles (it's "just" a computer program bro), but that's not how it actually came to be. Rather, it was trained on literally terrabytes and terrabytes of human-generated data. So, in a sense, it is actually a linguist. Just a really bad one.
- fnordpiglet 4y ago
  It’s still a computer program with some state attached. That’s literally all it is. There are no linguists in the program. That it is able to distill the knowledge and ability of a linguist by parsing ascii text into byte buffers and doing some fancy math on the bytes should be breathtaking. The sleight of hand would be if it had a mechanical Turk and there was a linguist. The fact that it collected knowledge to have knowledge and can apply that knowledge is no different than a linguist for sure, but that it’s a computer program distilling binary data into abstract concepts at the level of a grade schooler or a linguist (not sure who is more sophisticated) is mind bending. If you can’t see it, you’ve lost sight of things my friend. I hope you find it again. The world is a beautiful place.

wetpaws 4y ago

Haters gonna hate

asveikau 4y ago

Yeah, I was reading this and thinking it's mostly substituting English for another vocabulary. It's even referencing English when it comes up with something new, eg. referencing use of the "sl" consonant cluster in English.

fpgaminer 4y ago

I'm not a linguist, but wanted to invent a basic language for use in an escape room I'm building. Tools like this are a great help to people like me; hence the amazement. (I had to use GPT-3 and NovelAI, since this was two weeks ago)

pmontra 4y ago

This is about the same argument we can do about Google Translate or text to speech. Any professional translator can take a web page and give me a better translation than GT. However I get the GT translation in a few seconds and not in maybe hours, after I signed a contract with the translator. And text to speech would allow me to post an audio version of this reply in a good English accent instead of my inconsistent foreign one. Or paying a professional speaker, after signing a contract. Inventing a language is maybe useless but using ChatGPT as a tool to increase our speed is definitely useful.

CamperBob2 4y ago

"This talking dog makes embarrassing grammatical errors. Besides that, it learned everything it knows from humans, not from other dogs. I don't see the big deal." Anyone familiar with Chomsky's school of thought here? I'm not sure that "it's just a language model" is the effective dismissal that a lot of people think it is.