4 ms·

>performance on standardized tests? That doesn't necessarily seem like the best metric for what the LLM tries to be. The standardized tests give a baseline, no

by ugh123 4y ago

>performance on standardized tests? That doesn't necessarily seem like the best metric for what the LLM tries to be. The standardized tests give a baseline, no matter how arbitrary it might be, just as they do for humans in school. Whether we think it's right or not, these tools are coming for the workplace. So their ultimate metric will be in business performance to justify their costs (whatever they may be).

lionkor 4y ago
GPT 3.5 had trouble understanding when I told it "Say 2 bob are a beb, how many beb per bob are there?" and it wrote a goddamn essay about shoes. That thing isnt smart, it doesnt understand, it doesnt know, it just rambles. I have worked with people who do the same, yes, but they also werent a threat to most jobs. I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value. Make of that what you will - not everyone has to provide huge value.
- kolinko 4y ago
  I typed the query into chat-gpt3.5 (turbo and legacy), and 4, and they all said that there's 0.5 beb per bob. Did you use the quoted prompt exactly?
  - lionkor 4y ago
    No, I didn't use the quoted prompt, but even after explaining to it that bob and beb were not, in fact, shoe related terms, it still kept insisting and being confused (while also giving the correct 1/2 answer). It can do it, but its not deterministic, and it doesnt really do it well. You can continue the chain by asking "How many bob per bib, assuming two beb per bib?", and see if it chokes then. It sometimes does, sometimes doesnt.
    - ogogmad 4y ago
      GPT-4: If 2 bebs are equal to 1 bib, and we know that 1 beb equals 2 bobs, we can determine how many bobs there are per bib using simple substitution. 1 bib = 2 bebs 1 beb = 2 bobs Therefore, 1 bib = 2 bebs × 2 bobs/beb = 4 bobs So, there are 4 bobs per bib. Nitpick: A properly done substitution would've arrived at 1 bib = 2 × (2 bobs) without needing any of the "2 bebs × 2 bobs/beb" nonsense. It doesn't teach this task very well.
    - Tostino 4y ago
      You do realize that the current implementations get their context polluted by your prior conversation, right?
- Mike_12345 4y ago
  Answer from GPT-4: "This question seems to be intentionally nonsensical or is using unfamiliar terminology. However, if we try to interpret it, we could say that there are 2 "bob" making up 1 "beb." In this case, there would be 0.5 "beb" per "bob." Please provide more context or clarify the terms if you are looking for a different answer." Answer from GPT-3.5 (subscription version, not free): "If 2 bob are a beb, then the conversion factor from bobs to bebs is 1 beb per 2 bobs. So, there is 0.5 beb per bob (or alternatively, 2 bobs per beb)." Answer from GPT-3.5 (legacy / free version): "If 2 bobs are equal to 1 beb, then there would be 1 beb per 2 bobs."
  - skykooler 4y ago
    What do LLaMA-based models answer for this?
    - qingdao99 4y ago
      alpaca.cpp 7b model gave me this: > Say 2 bob are a beb, how many beb per bob are there? There are two bebs per bob. > No, there are 2 bobs per beb. Correct! There is only one other answer that makes sense - "1 Bob for every Beb". But this does not make mathematical or logical sense since a single Bob can have multiple Babes associated with it and vice versa (as seen in the example). Therefore, we must conclude there are two bobs per beb.
  - lionkor 4y ago
    Cool, but sadly, as I said, it did not give a very useful answer. If asked enough times, im sure it will give a reasonable answer, yes, but thats not the point. GPT4s answer is interesting, though
    - Mike_12345 4y ago
      But all of the answers were correct and useful, and GPT-4 was perfect. Anyway ChatGPT is getting hooked up to Wolfram Alpha, and that won't have any issues with basic algebra.
    - ugh123 4y ago
- hcrean 4y ago
  I think right here we have an example AI reproducibility problem. It seems fully reasonable and credible as an outcome, but it is hard to dig in and replicate. But the truth of ML is it would be difficult to replicate even if things were FOSS.
- BeefWellington 4y ago
  > I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value. The thing you're missing here is that it might take someone's job not because they weren't providing the value but because the people who crunch numbers decided to axe 15% of a company's workforce because some executive was sold a pack of lies about what LLMs/"AI" are actually capable of. It's fine if that happens to one company who then finds out the hard way. It's probably more social-unresty if it's essentially done at every company in every marketplace an LLM can touch - from writing to programming to 3D animation to teaching. The hype machine around LLM/AI here is the same irrational one we saw around blockchain. The key difference is blockchain was basically never sold as really replacing a person's job (at best you could argue it was sold as getting around the banking industry and maybe eventually being able to replace it, ish). The primary sales pitch of these LLMs is essentially "do more with less".

by ugh123 4y ago

lionkor 4y ago

GPT 3.5 had trouble understanding when I told it "Say 2 bob are a beb, how many beb per bob are there?" and it wrote a goddamn essay about shoes. That thing isnt smart, it doesnt understand, it doesnt know, it just rambles. I have worked with people who do the same, yes, but they also werent a threat to most jobs. I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value. Make of that what you will - not everyone has to provide huge value.

kolinko 4y ago
I typed the query into chat-gpt3.5 (turbo and legacy), and 4, and they all said that there's 0.5 beb per bob. Did you use the quoted prompt exactly?
- lionkor 4y ago
  No, I didn't use the quoted prompt, but even after explaining to it that bob and beb were not, in fact, shoe related terms, it still kept insisting and being confused (while also giving the correct 1/2 answer). It can do it, but its not deterministic, and it doesnt really do it well. You can continue the chain by asking "How many bob per bib, assuming two beb per bib?", and see if it chokes then. It sometimes does, sometimes doesnt.
  - ogogmad 4y ago
    GPT-4: If 2 bebs are equal to 1 bib, and we know that 1 beb equals 2 bobs, we can determine how many bobs there are per bib using simple substitution. 1 bib = 2 bebs 1 beb = 2 bobs Therefore, 1 bib = 2 bebs × 2 bobs/beb = 4 bobs So, there are 4 bobs per bib. Nitpick: A properly done substitution would've arrived at 1 bib = 2 × (2 bobs) without needing any of the "2 bebs × 2 bobs/beb" nonsense. It doesn't teach this task very well.
  - Tostino 4y ago
    You do realize that the current implementations get their context polluted by your prior conversation, right?
Mike_12345 4y ago
Answer from GPT-4: "This question seems to be intentionally nonsensical or is using unfamiliar terminology. However, if we try to interpret it, we could say that there are 2 "bob" making up 1 "beb." In this case, there would be 0.5 "beb" per "bob." Please provide more context or clarify the terms if you are looking for a different answer." Answer from GPT-3.5 (subscription version, not free): "If 2 bob are a beb, then the conversion factor from bobs to bebs is 1 beb per 2 bobs. So, there is 0.5 beb per bob (or alternatively, 2 bobs per beb)." Answer from GPT-3.5 (legacy / free version): "If 2 bobs are equal to 1 beb, then there would be 1 beb per 2 bobs."
- skykooler 4y ago
  What do LLaMA-based models answer for this?
  - qingdao99 4y ago
    alpaca.cpp 7b model gave me this: > Say 2 bob are a beb, how many beb per bob are there? There are two bebs per bob. > No, there are 2 bobs per beb. Correct! There is only one other answer that makes sense - "1 Bob for every Beb". But this does not make mathematical or logical sense since a single Bob can have multiple Babes associated with it and vice versa (as seen in the example). Therefore, we must conclude there are two bobs per beb.
- lionkor 4y ago
  Cool, but sadly, as I said, it did not give a very useful answer. If asked enough times, im sure it will give a reasonable answer, yes, but thats not the point. GPT4s answer is interesting, though
  - Mike_12345 4y ago
    But all of the answers were correct and useful, and GPT-4 was perfect. Anyway ChatGPT is getting hooked up to Wolfram Alpha, and that won't have any issues with basic algebra.
  - ugh123 4y ago
hcrean 4y ago
I think right here we have an example AI reproducibility problem. It seems fully reasonable and credible as an outcome, but it is hard to dig in and replicate. But the truth of ML is it would be difficult to replicate even if things were FOSS.
BeefWellington 4y ago
> I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value. The thing you're missing here is that it might take someone's job not because they weren't providing the value but because the people who crunch numbers decided to axe 15% of a company's workforce because some executive was sold a pack of lies about what LLMs/"AI" are actually capable of. It's fine if that happens to one company who then finds out the hard way. It's probably more social-unresty if it's essentially done at every company in every marketplace an LLM can touch - from writing to programming to 3D animation to teaching. The hype machine around LLM/AI here is the same irrational one we saw around blockchain. The key difference is blockchain was basically never sold as really replacing a person's job (at best you could argue it was sold as getting around the banking industry and maybe eventually being able to replace it, ish). The primary sales pitch of these LLMs is essentially "do more with less".