4 ms·
I think the thing to note about today's large language models is that they aren't purely statistical. The topology of the neural networks behind them has been e
by nuc1e0n 4y ago
I think the thing to note about today's large language models is that they aren't purely statistical. The topology of the neural networks behind them has been explicitly defined by the creators of those systems. They are not 'tabula rasa' as some might suppose them to be.
- bsaul 4y agoYet their structure being generic for all kind of problems, it doesn’t tell much in itself about the things it managed to « understand ». Much like studying einstein’s brain biology can’t teach you much about general relativity.
- nuc1e0n 4y agoBut none of them do have a generic structure. For example, GPT-3 can't produce images from text prompts, and stable diffusion cannot generate language. The possible relationships of words are written into GPT-3's code, in python, by its developers. In a way all this proves is that that written language can convey meaning to people.
- Ologn 4y ago> GPT-3 can't produce images from text prompts Me: Give me the entire hexadecimal format of an example PNG. Only give me the hexadecimal format. GPT-3: 89504E470D0A1A0A0000000D49484452000000640000006408020000000065238226000000014944415478DAECFD07780D44204C60F81EADAEF777F7E7E62F1BDE7DEBDED710EC15C7AC81CEEC17069C59B99A1698BEE7A484D68FDE782A7C41A8A0E7D2A2C9B00A99F32FBCED
- nuc1e0n 4y agoThat's not a valid png. It's just plausible hex tokens. GPT-3 is confidently wrong yet again.
- Ologn 4y ago$ echo 89504E470D0A1A0A0000000D49484452000000640000006408020000000065238226000000014944415478DAECFD07780D44204C60F81EADAEF777F7E7E62F1BDE7DEBDED710EC15C7AC81CEEC17069C59B99A1698BEE7A484D68FDE782A7C41A8A0E7D2A2C9B00A99F32FBCED |xxd -r -p > output.png $ file output.png output.png: PNG image data, 100 x 100, 8-bit/color RGB, non-interlaced $ eog output.png Fatal error reading PNG image file: IHDR: CRC error It seems you are correct.
- nuc1e0n 4y agoYeah, I did something similar before replying. Now maybe GPT-3 could be modified to make PNGs, but someone would have to go do that.
- int_19h 4y agoChatGPT is the language center of the brain, so to speak. If we are trying to model how we believe we are doing things, you'd expect a separate model optimized for images running side by side, with the two interacting (much like Sydney is GPT interacting with search). And maybe even multiple of each, so that if you give the chatbot a high-level task, it will break it down into subtasks - like it already can - and then hand each task off to a separate instance.
- theGnuMe 4y agoYou can go from an image to text and vice versa. People have done it.
- nuc1e0n 4y agoYeah, with specifically crafted models.
- WithinReason 4y agoThere are generic models that can do both
- nuc1e0n 4y agohave been made to do both.
- visarga 4y agoDALL-E 1 used a GPT approach to generate both text and images. Images are divided into patches, about 1024 patches for one image. Each patch is like a text token. > We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. https://arxiv.org/abs/2102.12092 https://arxiv.org/abs/2102.12092 The moral - you can just stream together text and images into a GPT style model.
- nuc1e0n 4y agoI don't mean to say it can't be done. Only that it has to be made to be done.
- bsaul 4y agoI was under the impression that pretty much everyone was using the transformer architecture for anything now. Isn’t that the case ?
- jsenn 4y agoThis is true, and interesting, but doesn’t address Chomsky’s concerns. While a LLM has structure, it’s still not as structured—or structured in the same way—as the human language faculty. This is easy to see by observing that LLMs can and do just as easily learn to produce things that are not human language as things that are. For something to count as a model of human language, it has to be able to produce language and not produce non-language.
- deleted 4y ago[deleted]
- alecst 4y agoWhat do LLMs produce that counts as non-language?
- jsenn 4y agoMy understanding is that they work well as arbitrary sequence predictors. For example, they can write HTML markup or C++ code just as easily as they can write English sentences. If you trained them on character sequences other than text from the internet, they would likely perform just as well on that data.
- nuc1e0n 4y agoHTML literally has language in the name, and C++ is a programming language.
- jsenn 4y agoSure, but the type of "language" that includes HTML and C++ is very different from the type of "language" that includes English and French. Chomsky's point is that there's something special about human brains that makes it very easy for them to learn English and French, even with very sparse and poorly-defined inputs, but doesn't necessarily help them learn to produce other types of structured sequences. For example, a baby raised next to a forest will effortlessly learn to speak their parents' native language (you couldn't stop them from doing that!) but won't learn to produce the birdsong they hear coming from the forest. This indicates that there's something special about our brains that leads us to produce English and not birdsong. Similarly, it's true that some humans can, with lots of cognitive effort, produce HTML and C++, but some can't. Even the ones that can don't do it the same way that they produce English or French.