13 ms·

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 mon

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool to do the same narrow task) - when it's a little wrong, you (the expert) can fix the issue and move on without friction - when it's any amount of wrong and you are less than an expert, or specifically you are completely unfamiliar with the topic, you can waste an immense amount of time researching the output and/or iterating with the system to refine the result Initially I thought it was a 10:1 ratio of performance to effort. But finally (as a developer) I settled on (1 to 1.5):1. It basically just changed the game for me from doing the actual hard work to working out how to tease the system into producing a reasonable result. And in the process (same as with co-pilot) I started to recognize how it was leading me to change my habits to avoid thinking and effort and instead rely on an external brain. If you could have a reliable external brain always available, that would be a fair trade. But when the external "brain" is unreliable and only available via certain interfaces, it's better to train yourself to be proactive, voracious (with respect to documentation), and tolerant of learning/producing cycles. I once thought it would be a game-changer. Now I realize it is a game-changer, but in a similar way that offshoring was... it didn't improve or solve any problems, but it merely changed the work.

letitgo12345 3y ago
The next iteration will be trained on your own data where "when it's a little wrong, you (the expert) can fix the issue and move on without friction" so that case will become "when it's right" and some amount of "when it's any amount of wrong" cases will become "when it's a little wrong". A few more cycles of this and we could be looking at GPT-10 which is a complete replacement for most tasks.
- ChatGTP 3y ago
  Hmm, wouldn’t this only be useful if you were solving the same problems over and over again, which does happen but say in the case of programming, I’d put the reusable pieces into a library, not a language model? Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?
  - letitgo12345 3y ago
    the idea is it wld learn how to do reasoning better based on this
- croes 3y ago
  Better result from less data? I doubt that.
  - nah93 3y ago
    [dead]
  - ftxbro 3y ago
    I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs. I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help. A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.
    - hooande 3y ago
      Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality. Yes, the correct thing to do is get more data. Much more.
      - ftxbro 3y ago
        > Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.
tbrownaw 3y ago
> don't actually understand anything (as greater concepts), but rather operate as complex probability machines? But, things are defined by how they interact with the world around them. A concept is its relations to other concepts. Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.
- andsoitis 3y ago
  > A concept is its relations to other concepts. where does a concept begin? where does a concept end? what is a concept?
  - Legend2440 3y ago
    A concept is an abstraction. Abstractions are a way to characterize the behavior of complex systems. It's not feasible to directly compute their behavior, but they still have predictable properties. Concepts let you handle emergence; you can manipulate an object as its own thing rather than a collection of atoms. (And yes, I did just read Stephen Wolfram's book and this idea is largely based on it. I think he has some delusions of grandeur but is also onto something.)
- Retric 3y ago
  It’s using tokens not concepts. ‘Apple’ the token could apply to a company or fruit, but tokens aren’t concepts. It’s a fairly fundamental limitation on the approach that’s most noticeable while feeding it data it can’t basically copy from it’s vast training data. Thus the sharp drop off where it can for example guess the correct answer to some math problems yet get others that are conceptually identical completely wrong.
  - regularfry 3y ago
    They turn tokens into concepts internally. Try asking ChatGPT "can you eat Apple?" and "who is the CEO of Apple?".
    - 3y ago
YetAnotherNick 3y ago
I have very similar observation, and while it is amazing at times at helping me with complex tasks, it just isn't like any human. If we rank the tasks by difficulty for human and AI, it just has very small correlation. Also GPT 4 is idiot in conversation, compared to its problem solving skill. I would have expected completely opposite trajectory for AI. eg it's hard for it to get it to ask good clarifying questions. There are lot of cheap tricks in prompting like asking it to act as expert, or prompt for chain of though that works.
RossBencina 3y ago
I like your takeaways and reflection, especially the "changes the game" idea. There is an analogy with pocket calculators and mental arithmetic. Personally I'm more comfortable reaching for the pocket calculator than offloading all thinking to an LLM. On the other hand, it's not so long ago that manual calculation was a specialized occupation. I could maybe see software coding becoming automated just as calculation was -- except for the fact that calculations are much easier to specify than software. I don't quite get where you're coming from with "LLM's don't actually understand anything (as greater concepts)". I have heard the view coming from researchers that the larger models do form representations of higher-order information structures ("concepts"). Perhaps what you're getting at is that current models don't encode enough higher-order structure to deal accurately with your domain? Whether the models can be made to do so seems like an open question to me. The boosters say it will be here by next year.
- sumanthvepa 3y ago
  This observation is spot on. I was initially euphoric about LLMs like ChatGPT, but it is increasingly becoming obvious that unless you are yourself an expert in the subject you are using the LLM for and can therefore easily verify its accuracy, the output is not reliable enough to use without extensive manual verification. More importantly it is difficult to incorporate its output into larger automated workflows.
- richardw 3y ago
  > I don't quite get where you're coming from with "LLM's don't actually understand anything Well, Wikipedia doesn’t understand anything, despite having a lot of knowledge encoded in it. This is similar in that it approximates the output of someone who can generate the world’s written word, but there’s a big gap between the monkeys that wrote the original text and the machine that now regurgitates it. What we know is that our written word has enough encoded in it that we can generate new combinations. But one day into a truly novel problem, LLM’s would have no idea. I kept trying to get ChatGPT 4 to generate code with an AWS API that was released in ~2020. Couldn’t do it. Kept predicting the earlier text because that was the bulk of what it saw. A year of text (to the 2021 cutoff) and it was still unable to let go of the old, now wrong, “knowledge”. Zero understanding, just parroting. No coder with a year of training on a new API would be that wrong. It would say things like “you’re right I must not do X” and then do X again, in the same output.
syntheweave 3y ago
I do think there's an element to it of "uncanny valleyness". If you know absolutely nothing about a topic the authoritative tone and factualness of what it says is very appealing and even helpful in the same way that consulting an encyclopedia is helpful: it tells you of things you could investigate further that would never appear through a keyword search. But if you stop there, your knowledge is "roughly encyclopedic" which means it contains the hidden bias of some anonymous author, and not the harder-earned relationships of facts and logic. If you use it to translate things between formal encodings("turn this into hexadecimal bytes, now role-play a lawyer arguing about why that is meaningful") it can produce occasionally useful aesthetic results and speed along tasks that would be challenging to model formally and don't need a lot of rigor. But once you start pushing it to be technically accurate in a narrow, measurable direction it flounders and the probabilistic element is revealed. Once, I asked it to translate a short string of Japanese characters and it confidently said that it was Kenshiro's catch phrase from Fist of the North Star, "Omae wa mou shinderu" (you are already dead) which I could clearly see it wasn't - not a single character matched. It's just the thing if you need to learn some anime Japanese, though.
- corey_moncure 3y ago
  This is very surprising to me. I found ChatGPT/4 to be extremely adept at translation between well asserted languages including English/Japanese in which I am an expert in both. I'm curious how you managed to make it blow up.
  - int_19h 3y ago
    One thing that really floored me wrt GPT-4 translation capabilities is using it to make sense of the Veritable Records of the Joseon Dynasty (https://sillok.history.go.kr/main/main.do https://sillok.history.go.kr/main/main.do). Take something from 1400s and see what it can do with that after you tell it what it is translating. Then try Google/Bing Translate for comparison. However, it does have a thing about pretending to know languages that it really doesn't (because of how little of them there was in the training data and/or in general).
bemmu 3y ago
I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.
- thereare5lights 3y ago
  There's no evidence for nondeterminism in the human brain
  - anikan_vader 3y ago
    There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.
    - Legend2440 3y ago
      Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see. Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.
      - kedean 3y ago
        If it's not actually possible to simulate a system within the confines of physics, does it being deterministic actually matter outside of thought experiments? I feel like a random system and a deterministic system which cannot be simulated are effectively the same thing.
        zappchance 3y ago
        It only matters in matters of free will and ethics. One actual scenario where it's relavant would be the discussion around the criminal justice system. If the universe is deterministic, how can punitive justice be justified?
01100011 3y ago
I still think LLMs are a game changer. They are seemingly amazing at deducing meaning and intent from natural language. That's enough to be a game changer. Is it a game changer like the internet is/will-be? No. Is it on the level of the television or the telephone? Probably not. But you can still leverage it in many ways to add value or reduce human work. If we're talking about LLMs as general purpose AI or a tool to replace programmers... mostly a fail so far. LLMs seem like they could eventually be a component of something bigger, but I don't see a line between what they are and general purpose AI.
Legend2440 3y ago
It's a game-changer in that it shows computers being able to climb levels of abstraction on their own. In the past, if you wanted programs to work with a high-level idea, you had to explicitly hand it to them. Humans had to do the work of turning data into concepts. These generative AI systems are different - they can learn abstractions from data, and manipulate them in complex ways. Is ChatGPT a good and accurate chatbot? Maybe not. But it's a fundamental change in what computers are capable of.
visarga 3y ago
One field where it works rather well is in semantic data extraction. Do you remember the dream of semantic web? Having access to structured information from unstructured sources, auto-matching API functions and arguments to make interoperation easier. We can do that now. Here is a sample where I used my own post (just copy pasted the raw text from the browser) and got this schema comment: meta: points: author: time: action: parent: next: edit: delete: post: content: action: reply: It also guessed it was a Hacker News comment by the formatting of the meta section.
csomar 3y ago
I'm guessing that's how people figured out it was getting "nerfed". For Copilot, I kind of developed a relationship with the "AI". I kind of expect what it is going to generate, and I use it as a shortcut to (find function names, write variable names, complete unit tests, etc...). On its own, it's really bad to come up with the whole picture but as something that auto-completes you on steroid, it's really perfect. When it got nerfed, its perspective got completely out of whack and it's making very different and inconsistent generation.
mtsolitary 3y ago
Very well said, this has been precisely my experience too
tempodox 3y ago
Only you, a human developer can be truly creative. An LLM can only ever reproduce what it has seen before.
- throwaway2037 3y ago
  An LLM can only ever reproduce what it has seen before. I don't believe it. How do you explain Midjourney? The art that it produces is incredible by any measurement.
- CamperBob2 3y ago
  Where, exactly, did the LLM see this epilogue to The Great Gatsby before? https://twitter.com/tsimonite/status/1653065940463157248 https://twitter.com/tsimonite/status/1653065940463157248
  - Applejinx 3y ago
    I responded 'way to not getsby the point'. How is that in ANY WAY an epilogue to The Great Gatsby? This is exactly the problem. That original story builds with a series of revelations into the conclusion 'so we beat on, boats against the current, borne back ceaselessly into our past': establishing a PURPOSE, perhaps a bleak and unwelcome one. Fitzgerald's revealing an insight into the delusions of humanity. He's picturing even the greatest of us as surfers on the river of nihilism and reality. Our aspirations sparkle prettily… and are gone, like froth in the rapids. And this is just a little bit beautiful. We can imagine beyond our reach. That's a human thing. The fact that our minds can cling so longingly to something that is simply not real, is kind of wonderful. Everyone's their own little world of unreality, and we aspire so earnestly (much like these AI folks do). And then the AI, having drunk up all of Fitzgerald and everybody else, 'continues' past the point. With what? Hardly matters. It has no point to make. I'd be impressed if it refused, said 'nope, that was where it ended. Can't add anything worth adding, try reading it again'. But no, because the LLM has no intention and doesn't successfully get one from what it's 'read'. It's constructed an epilogue out of nebulous religious feelgoodism in the rough style of Fitzgerald's sentence construction, and undermined the whole conclusion of the story… and not maliciously, for that would require intent. Nope… it sort of ambled on, going 'what would feel nice here? ok, now what seems like it would go with this sort of thing? ok, something else, let's have more, what kind of concepts go here? what do people normally say when they talk in this way?' In so doing, it's less than Gatsby and way less than Fitzgerald. There is nothing here in this 'epilogue'. GPT4 criticised the 'epilogue', in a rather fascinating way! https://pastebin.com/B0zxbvNv https://pastebin.com/B0zxbvNv It successfully works out some of the problems with the first AI's writing, and yet it too fails to get the idea expressed by Fitzgerald, and rather than pivot to religious feelgoodism, it pressures the first AI to instead emphasize how its narrator and Gatsby shared a special bond, the very special specialness of being "the only one who understood what it meant to be young and restless in this restless world." A lot of people can write that idea, and in fact a lot of people did and that's why GPT4 found it a probable argument to make. Fitzgerald gave us a moment of viscerally grokking that it doesn't mean s*t… and yet, we will still paddle against the current of time and decay and collapse, because what else can we do? Tomorrow we'll get it. Tomorrow we'll really understand and it'll all make sense. And so…
thomasahle 3y ago
> LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? What does it mean to "operate as a probability machine"? And what does it mean to understand anything? One recent example of understanding is that llms/transformers learn to parse context free grammars via dynamic programming (https://arxiv.org/abs/2305.02386 https://arxiv.org/abs/2305.02386). Basically they've understood what's going on well enough to mold their neurons I to the optimal algorthm for parsing this kind of text. I think they understand lots of things like this. Of course there's other things they don't understand or just pretend to understand.
davidguetta 3y ago
Have you been to a bar lately and overheard people talking about politics ? They 100% are probability machine, of lower quality than ChatGPT
- michaelteter 3y ago
  Indeed. But I don't expect any of those bar patrons to help me get my work done... and I would be very skeptical of their advice.
  - davidguetta 3y ago
    What's even the point of your message ? ChatGPT is a tool, you can use it or not nobody cares, but a lot of people think if you use it the right way it IS helpful
rpozarickij 3y ago
> avoid thinking and effort and instead rely on an external brain For me personally this was probably the biggest game changer because I'm now able to offload a lot of thinking to GPT-based tools and use my brain cycles for the less automatable activities. Now instead of searching for something on Google and going through multiple pages before finding an answer, I can ask a precise question and get the answer right away. Especially when I know that the answer _is_ there somewhere, and all I need is to find it. If I'm not happy with the answer, I can continue the conversation until I get what I'm looking for. I made Bing Chat, ChatGPT, and Warp AI (a feature of the Warp terminal that allows you to access GPT from within the terminal) a part of my daily life and I feel like I'm achieving much more with the time I have.
quickthrower2 3y ago
This probability thing might be a red herring. I reckon it is just that softmax/cross entropy loss is a handy tool to get words out of the NN (and now transformers)
sfpotter 3y ago
This is a great set of observations. The only thing missing is an analysis of how much power it takes to accomplish each of these tasks. If ChatGPT-4 is at about 1:1 in terms of “effectiveness”, all that remains is to divide by the amount of power required to reach the answer using ChatGPT-4 vs by conventional means. If requires significantly more energy, then it’s a waste, and because of climate change we should really not pursue it further, IMO.
- michaelteter 3y ago
  This is a good point. There's a hidden cost (energy consumption) which we will eventually pay for. However, even in the 1:1 case it means I am training myself to become a "prompt engineer" rather than to be an actual thinker and problem solver. As long as there will always be another system for me to depend on, maybe that's ok. But as with people who never learned to read maps and navigate without GPS tend to be very confused and lost when their phone dies, I would like to be able to be a useful human even when the power is out.
  - sfpotter 3y ago
    I totally agree, although I think it's even worse than that. In my opinion, a precondition for creativity and inventiveness is understanding. If you rely on a surrogate to give you answers, you will never reach the level of proficiency required to come up with something new. If we train a generation of thinkers to rely on an external brain to get anything done, they will only understand things superficially, and our ability to innovate at the society level will suffer.
nicetryguy 3y ago
> Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? I'm genuinely unsure if my own brain is any different.
School-Cotton 3y ago
I still don’t understand what it means when people say stuff like “ChatGPT just predicts the most likely next word with the highest probability” or “ChatGPT is just a probability machine”. Concretely, given the N most recent words, what algorithm are you proposing/claiming it uses to assign probabilities to word N+1? Just saying “it chooses the next word with highest probability” doesn’t explain how the probabilities are estimated.
- nulltype 3y ago
  [0] [0] http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/

letitgo12345 3y ago

The next iteration will be trained on your own data where "when it's a little wrong, you (the expert) can fix the issue and move on without friction" so that case will become "when it's right" and some amount of "when it's any amount of wrong" cases will become "when it's a little wrong". A few more cycles of this and we could be looking at GPT-10 which is a complete replacement for most tasks.

ChatGTP 3y ago
Hmm, wouldn’t this only be useful if you were solving the same problems over and over again, which does happen but say in the case of programming, I’d put the reusable pieces into a library, not a language model? Like you’d train the model to give you the most accurate response based on your current problem space, but when they changes, you’d have to retrain on what you’re working on but by that stage it’s already out of date ?
- letitgo12345 3y ago
  the idea is it wld learn how to do reasoning better based on this
croes 3y ago
Better result from less data? I doubt that.
- nah93 3y ago
  [dead]
- ftxbro 3y ago
  I mean 'better result from less data' is at least a little bit possible. For example you can just clean out obviously bad data from the trillions of tokens data sets. It's things like the subreddit where they are counting to a million or just like long lists of hash values in random cryptocurrency logs. I agree that in the bigger picture this doesn't matter, but it's technically true that cleaning the data in some way would help. A related project is TinyStories where they try to use good data for unlocking the LLM cognitive capabilities without requiring as many parameters or exaflops. Again, there is obviously a limit to this, and maybe the effort is better spent on just getting even more gigantic dataset instead of nitpicking the useless or redundant data in the dataset.
  - hooande 3y ago
    Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. Any individual reddit thread is an atom in a drop in a bucket. All of reddit is < 2% of the total training data, regardless of quality. Yes, the correct thing to do is get more data. Much more.
    - ftxbro 3y ago
      > Data quality like you're describing just doesn't matter. GPT is trained on PEBIBYTES of data. It matters a little bit, in a quantitative but not qualitative way. Probably with good data cleaning you could get as high quality result with only one pebibyte of data if it normally needs two pebibytes. If training time is proportional to dataset size then maybe it takes three months instead of six months to train. Maybe it would save hundreds of millions or a billion dollars which I guess would matter to someone. It probably wouldn't matter qualitatively though.

tbrownaw 3y ago

> don't actually understand anything (as greater concepts), but rather operate as complex probability machines? But, things are defined by how they interact with the world around them. A concept is its relations to other concepts. Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.

andsoitis 3y ago
> A concept is its relations to other concepts. where does a concept begin? where does a concept end? what is a concept?
- Legend2440 3y ago
  A concept is an abstraction. Abstractions are a way to characterize the behavior of complex systems. It's not feasible to directly compute their behavior, but they still have predictable properties. Concepts let you handle emergence; you can manipulate an object as its own thing rather than a collection of atoms. (And yes, I did just read Stephen Wolfram's book and this idea is largely based on it. I think he has some delusions of grandeur but is also onto something.)
Retric 3y ago
It’s using tokens not concepts. ‘Apple’ the token could apply to a company or fruit, but tokens aren’t concepts. It’s a fairly fundamental limitation on the approach that’s most noticeable while feeding it data it can’t basically copy from it’s vast training data. Thus the sharp drop off where it can for example guess the correct answer to some math problems yet get others that are conceptually identical completely wrong.
- regularfry 3y ago
  They turn tokens into concepts internally. Try asking ChatGPT "can you eat Apple?" and "who is the CEO of Apple?".
  - 3y ago

YetAnotherNick 3y ago

I have very similar observation, and while it is amazing at times at helping me with complex tasks, it just isn't like any human. If we rank the tasks by difficulty for human and AI, it just has very small correlation. Also GPT 4 is idiot in conversation, compared to its problem solving skill. I would have expected completely opposite trajectory for AI. eg it's hard for it to get it to ask good clarifying questions. There are lot of cheap tricks in prompting like asking it to act as expert, or prompt for chain of though that works.

RossBencina 3y ago

I like your takeaways and reflection, especially the "changes the game" idea. There is an analogy with pocket calculators and mental arithmetic. Personally I'm more comfortable reaching for the pocket calculator than offloading all thinking to an LLM. On the other hand, it's not so long ago that manual calculation was a specialized occupation. I could maybe see software coding becoming automated just as calculation was -- except for the fact that calculations are much easier to specify than software. I don't quite get where you're coming from with "LLM's don't actually understand anything (as greater concepts)". I have heard the view coming from researchers that the larger models do form representations of higher-order information structures ("concepts"). Perhaps what you're getting at is that current models don't encode enough higher-order structure to deal accurately with your domain? Whether the models can be made to do so seems like an open question to me. The boosters say it will be here by next year.

sumanthvepa 3y ago
This observation is spot on. I was initially euphoric about LLMs like ChatGPT, but it is increasingly becoming obvious that unless you are yourself an expert in the subject you are using the LLM for and can therefore easily verify its accuracy, the output is not reliable enough to use without extensive manual verification. More importantly it is difficult to incorporate its output into larger automated workflows.
richardw 3y ago
> I don't quite get where you're coming from with "LLM's don't actually understand anything Well, Wikipedia doesn’t understand anything, despite having a lot of knowledge encoded in it. This is similar in that it approximates the output of someone who can generate the world’s written word, but there’s a big gap between the monkeys that wrote the original text and the machine that now regurgitates it. What we know is that our written word has enough encoded in it that we can generate new combinations. But one day into a truly novel problem, LLM’s would have no idea. I kept trying to get ChatGPT 4 to generate code with an AWS API that was released in ~2020. Couldn’t do it. Kept predicting the earlier text because that was the bulk of what it saw. A year of text (to the 2021 cutoff) and it was still unable to let go of the old, now wrong, “knowledge”. Zero understanding, just parroting. No coder with a year of training on a new API would be that wrong. It would say things like “you’re right I must not do X” and then do X again, in the same output.

syntheweave 3y ago

I do think there's an element to it of "uncanny valleyness". If you know absolutely nothing about a topic the authoritative tone and factualness of what it says is very appealing and even helpful in the same way that consulting an encyclopedia is helpful: it tells you of things you could investigate further that would never appear through a keyword search. But if you stop there, your knowledge is "roughly encyclopedic" which means it contains the hidden bias of some anonymous author, and not the harder-earned relationships of facts and logic. If you use it to translate things between formal encodings("turn this into hexadecimal bytes, now role-play a lawyer arguing about why that is meaningful") it can produce occasionally useful aesthetic results and speed along tasks that would be challenging to model formally and don't need a lot of rigor. But once you start pushing it to be technically accurate in a narrow, measurable direction it flounders and the probabilistic element is revealed. Once, I asked it to translate a short string of Japanese characters and it confidently said that it was Kenshiro's catch phrase from Fist of the North Star, "Omae wa mou shinderu" (you are already dead) which I could clearly see it wasn't - not a single character matched. It's just the thing if you need to learn some anime Japanese, though.

corey_moncure 3y ago
This is very surprising to me. I found ChatGPT/4 to be extremely adept at translation between well asserted languages including English/Japanese in which I am an expert in both. I'm curious how you managed to make it blow up.
- int_19h 3y ago
  One thing that really floored me wrt GPT-4 translation capabilities is using it to make sense of the Veritable Records of the Joseon Dynasty (https://sillok.history.go.kr/main/main.do https://sillok.history.go.kr/main/main.do). Take something from 1400s and see what it can do with that after you tell it what it is translating. Then try Google/Bing Translate for comparison. However, it does have a thing about pretending to know languages that it really doesn't (because of how little of them there was in the training data and/or in general).

bemmu 3y ago

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

thereare5lights 3y ago
There's no evidence for nondeterminism in the human brain
- anikan_vader 3y ago
  There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.
  - Legend2440 3y ago
    Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see. Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.
    - kedean 3y ago
      If it's not actually possible to simulate a system within the confines of physics, does it being deterministic actually matter outside of thought experiments? I feel like a random system and a deterministic system which cannot be simulated are effectively the same thing.
      - zappchance 3y ago
        It only matters in matters of free will and ethics. One actual scenario where it's relavant would be the discussion around the criminal justice system. If the universe is deterministic, how can punitive justice be justified?

01100011 3y ago

I still think LLMs are a game changer. They are seemingly amazing at deducing meaning and intent from natural language. That's enough to be a game changer. Is it a game changer like the internet is/will-be? No. Is it on the level of the television or the telephone? Probably not. But you can still leverage it in many ways to add value or reduce human work. If we're talking about LLMs as general purpose AI or a tool to replace programmers... mostly a fail so far. LLMs seem like they could eventually be a component of something bigger, but I don't see a line between what they are and general purpose AI.

Legend2440 3y ago

It's a game-changer in that it shows computers being able to climb levels of abstraction on their own. In the past, if you wanted programs to work with a high-level idea, you had to explicitly hand it to them. Humans had to do the work of turning data into concepts. These generative AI systems are different - they can learn abstractions from data, and manipulate them in complex ways. Is ChatGPT a good and accurate chatbot? Maybe not. But it's a fundamental change in what computers are capable of.

visarga 3y ago

One field where it works rather well is in semantic data extraction. Do you remember the dream of semantic web? Having access to structured information from unstructured sources, auto-matching API functions and arguments to make interoperation easier. We can do that now. Here is a sample where I used my own post (just copy pasted the raw text from the browser) and got this schema comment: meta: points: author: time: action: parent: next: edit: delete: post: content: action: reply: It also guessed it was a Hacker News comment by the formatting of the meta section.

csomar 3y ago

I'm guessing that's how people figured out it was getting "nerfed". For Copilot, I kind of developed a relationship with the "AI". I kind of expect what it is going to generate, and I use it as a shortcut to (find function names, write variable names, complete unit tests, etc...). On its own, it's really bad to come up with the whole picture but as something that auto-completes you on steroid, it's really perfect. When it got nerfed, its perspective got completely out of whack and it's making very different and inconsistent generation.

mtsolitary 3y ago

Very well said, this has been precisely my experience too

tempodox 3y ago

Only you, a human developer can be truly creative. An LLM can only ever reproduce what it has seen before.

throwaway2037 3y ago
An LLM can only ever reproduce what it has seen before. I don't believe it. How do you explain Midjourney? The art that it produces is incredible by any measurement.
CamperBob2 3y ago
Where, exactly, did the LLM see this epilogue to The Great Gatsby before? https://twitter.com/tsimonite/status/1653065940463157248 https://twitter.com/tsimonite/status/1653065940463157248
- Applejinx 3y ago
  I responded 'way to not getsby the point'. How is that in ANY WAY an epilogue to The Great Gatsby? This is exactly the problem. That original story builds with a series of revelations into the conclusion 'so we beat on, boats against the current, borne back ceaselessly into our past': establishing a PURPOSE, perhaps a bleak and unwelcome one. Fitzgerald's revealing an insight into the delusions of humanity. He's picturing even the greatest of us as surfers on the river of nihilism and reality. Our aspirations sparkle prettily… and are gone, like froth in the rapids. And this is just a little bit beautiful. We can imagine beyond our reach. That's a human thing. The fact that our minds can cling so longingly to something that is simply not real, is kind of wonderful. Everyone's their own little world of unreality, and we aspire so earnestly (much like these AI folks do). And then the AI, having drunk up all of Fitzgerald and everybody else, 'continues' past the point. With what? Hardly matters. It has no point to make. I'd be impressed if it refused, said 'nope, that was where it ended. Can't add anything worth adding, try reading it again'. But no, because the LLM has no intention and doesn't successfully get one from what it's 'read'. It's constructed an epilogue out of nebulous religious feelgoodism in the rough style of Fitzgerald's sentence construction, and undermined the whole conclusion of the story… and not maliciously, for that would require intent. Nope… it sort of ambled on, going 'what would feel nice here? ok, now what seems like it would go with this sort of thing? ok, something else, let's have more, what kind of concepts go here? what do people normally say when they talk in this way?' In so doing, it's less than Gatsby and way less than Fitzgerald. There is nothing here in this 'epilogue'. GPT4 criticised the 'epilogue', in a rather fascinating way! https://pastebin.com/B0zxbvNv https://pastebin.com/B0zxbvNv It successfully works out some of the problems with the first AI's writing, and yet it too fails to get the idea expressed by Fitzgerald, and rather than pivot to religious feelgoodism, it pressures the first AI to instead emphasize how its narrator and Gatsby shared a special bond, the very special specialness of being "the only one who understood what it meant to be young and restless in this restless world." A lot of people can write that idea, and in fact a lot of people did and that's why GPT4 found it a probable argument to make. Fitzgerald gave us a moment of viscerally grokking that it doesn't mean s*t… and yet, we will still paddle against the current of time and decay and collapse, because what else can we do? Tomorrow we'll get it. Tomorrow we'll really understand and it'll all make sense. And so…

thomasahle 3y ago

> LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? What does it mean to "operate as a probability machine"? And what does it mean to understand anything? One recent example of understanding is that llms/transformers learn to parse context free grammars via dynamic programming (https://arxiv.org/abs/2305.02386 https://arxiv.org/abs/2305.02386). Basically they've understood what's going on well enough to mold their neurons I to the optimal algorthm for parsing this kind of text. I think they understand lots of things like this. Of course there's other things they don't understand or just pretend to understand.

davidguetta 3y ago

Have you been to a bar lately and overheard people talking about politics ? They 100% are probability machine, of lower quality than ChatGPT

michaelteter 3y ago
Indeed. But I don't expect any of those bar patrons to help me get my work done... and I would be very skeptical of their advice.
- davidguetta 3y ago
  What's even the point of your message ? ChatGPT is a tool, you can use it or not nobody cares, but a lot of people think if you use it the right way it IS helpful

rpozarickij 3y ago

> avoid thinking and effort and instead rely on an external brain For me personally this was probably the biggest game changer because I'm now able to offload a lot of thinking to GPT-based tools and use my brain cycles for the less automatable activities. Now instead of searching for something on Google and going through multiple pages before finding an answer, I can ask a precise question and get the answer right away. Especially when I know that the answer _is_ there somewhere, and all I need is to find it. If I'm not happy with the answer, I can continue the conversation until I get what I'm looking for. I made Bing Chat, ChatGPT, and Warp AI (a feature of the Warp terminal that allows you to access GPT from within the terminal) a part of my daily life and I feel like I'm achieving much more with the time I have.

quickthrower2 3y ago

This probability thing might be a red herring. I reckon it is just that softmax/cross entropy loss is a handy tool to get words out of the NN (and now transformers)

sfpotter 3y ago

This is a great set of observations. The only thing missing is an analysis of how much power it takes to accomplish each of these tasks. If ChatGPT-4 is at about 1:1 in terms of “effectiveness”, all that remains is to divide by the amount of power required to reach the answer using ChatGPT-4 vs by conventional means. If requires significantly more energy, then it’s a waste, and because of climate change we should really not pursue it further, IMO.

michaelteter 3y ago
This is a good point. There's a hidden cost (energy consumption) which we will eventually pay for. However, even in the 1:1 case it means I am training myself to become a "prompt engineer" rather than to be an actual thinker and problem solver. As long as there will always be another system for me to depend on, maybe that's ok. But as with people who never learned to read maps and navigate without GPS tend to be very confused and lost when their phone dies, I would like to be able to be a useful human even when the power is out.
- sfpotter 3y ago
  I totally agree, although I think it's even worse than that. In my opinion, a precondition for creativity and inventiveness is understanding. If you rely on a surrogate to give you answers, you will never reach the level of proficiency required to come up with something new. If we train a generation of thinkers to rely on an external brain to get anything done, they will only understand things superficially, and our ability to innovate at the society level will suffer.

nicetryguy 3y ago

> Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? I'm genuinely unsure if my own brain is any different.

School-Cotton 3y ago

I still don’t understand what it means when people say stuff like “ChatGPT just predicts the most likely next word with the highest probability” or “ChatGPT is just a probability machine”. Concretely, given the N most recent words, what algorithm are you proposing/claiming it uses to assign probabilities to word N+1? Just saying “it chooses the next word with highest probability” doesn’t explain how the probabilities are estimated.

nulltype 3y ago
[0] [0] http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/