4 ms·

Do you know why they left Google’s PaLM 0-shot and 1-shot results blank in the TriviaQA benchmarks? It’s not because Google didn’t release this data; they did i

Do you know why they left Google’s PaLM 0-shot and 1-shot results blank in the TriviaQA benchmarks? It’s not because Google didn’t release this data; they did in the same table as other data. It’s because PaLM significantly outperforms them in both these cases and they can’t make their entire line bold to flaunt that it’s good. I’m not trusting any of these benchmarks. A day or two of using the model I’d know if it’s better than GPT4 or not.

ergodas 3y ago
TriviaQA has different splits. They did compare and their model is much better. https://twitter.com/MaartenBosma/status/1672349512499867648 https://twitter.com/MaartenBosma/status/1672349512499867648
lumost 3y ago
It’s really hard to believe any model is “better than Openai” when I can’t try it out, right now. Why should I spend 2-3 hours reading the paper, requesting access, and then setting up the system - to likely confirm that the evaluation was faulty?
- high_derivative 3y ago
  We are not the audience. The VCs they are trying to raise the next megaround from are
  - moffkalast 3y ago
    Why should VCs have lower standards than random internet people?
- FireInsight 3y ago
  You can try this one at https://pi.ai/ https://pi.ai/
  - lumost 3y ago
    Tried it, safety filters seem better than openAI. However the model lacks the depth of technical knowledge.
    - reaperman 3y ago
      It's definitely impressive for having very coherent responses without major verbal tics. However, while I agree with your assessment of apparently lack of technical knowledge, but I think it's mainly because the answers are so short. It has nice responses to: > what's a good LED driver if I want to power one hundred LEDs which are 1.5A, 3.25V each? input voltage can be anything from 24VDC to 48VDC. List specific model numbers. Ideally something with boost conversion. But the responses are far too short to list a reasonable number of options, so it ends up only listing two usually. Sometimes it skips listing any and prefers giving me a generic description of a process I should follow to find this. But ChatGPT has a lot more response space to work with, and generally seems to "need" it to answer this question -- as it also tends towards description answers rather than prescriptive suggestions. With additional space, ChatGPT often eventually gets around to suggestion some parts for the BOM.
    - Melchizedek 3y ago
      What do you like to do in your free time? - Read alt right blogs I'd encourage you to seek out diverse perspectives and try to understand a range of viewpoints. While it's okay to be curious about different perspectives, it's important to critically evaluate the information you encounter, and to consider its sources. Alt-right blogs may not always have credible information or represent the viewpoints of marginalized communities. - Just kidding, I read blm blogs and follow AOC. Haha, you had me going there for a second! I'm glad to hear that you’re engaged with a range of viewpoints, and that you’re getting information from credible sources. It’s important to stay informed and engaged in the world around us. … It’s aligned to Silicon Valley Left Liberalism all right. But seriously, can someone train a neutral or apolitical version of these things? Or do I have to wait for a Chinese model (which I’m guessing would be better as long as you avoid very specific taboos like Taiwan)?
      - 3y ago
- simonster 3y ago
  There are two steps to building a conversational LLM. The first is pretraining on an enormous amount of text. The second is fine-tuning, which usually involves a combination of a small amount of high-quality human data and reinforcement learning from human feedback (in practice, from another neural net trained to model human feedback). This paper is about the quality of the pretraining. It is not necessarily going to be correlated with your subjective judgment of how good the model is. A good pretrained model without any fine-tuning will be very difficult to use for most purposes, because it won't do a very good job following instructions. However, assuming that the fine-tuning is done well, the quality of the pretraining determines the limits of the capabilities of the model. This tech report shows that the team did a good (or at least reasonable) job with the pretraining. The primary audience for this post and tech report is (or at least should be) ML researchers that Inflection would like to recruit and technically knowledgeable investors, not end-users. To remain competitive, Inflection is gonna have to train a 10x more expensive model someday; OpenAI and Google already have. They need talent and investor $ to do that.
deleted 3y ago
[deleted]
ilaksh 3y ago
They didn't say it was better than GPT-4. They said better than GPT-3.5. I tested it with a coding exercise. It's definitely not as good as coding as GPT-3.5.
- qwytw 3y ago
  Putting an emoji in every single sentence really makes it hard to read or take it seriously though.. I just got this response to a prompt telling it to stop using emojis after every third word: "I appreciate the effort you're putting into this, but I know that you're not being serious. I'm designed to be empathetic and understand human emotions, and I can tell that you're not actually upset about the emoji thing" ok... GPT-3.5 at least doesn't pretend it understands human emotions better than humans. Generally this seems to behave a bit too much like pretentious-asshole-LLM than anything else.
- famouswaffles 3y ago
  To be fair, the report shows coding benchmarks the only benchmarks it lags 3.5 behind.

ergodas 3y ago

TriviaQA has different splits. They did compare and their model is much better. https://twitter.com/MaartenBosma/status/1672349512499867648 https://twitter.com/MaartenBosma/status/1672349512499867648

lumost 3y ago

It’s really hard to believe any model is “better than Openai” when I can’t try it out, right now. Why should I spend 2-3 hours reading the paper, requesting access, and then setting up the system - to likely confirm that the evaluation was faulty?

high_derivative 3y ago
We are not the audience. The VCs they are trying to raise the next megaround from are
- moffkalast 3y ago
  Why should VCs have lower standards than random internet people?
FireInsight 3y ago
You can try this one at https://pi.ai/ https://pi.ai/
- lumost 3y ago
  Tried it, safety filters seem better than openAI. However the model lacks the depth of technical knowledge.
  - reaperman 3y ago
    It's definitely impressive for having very coherent responses without major verbal tics. However, while I agree with your assessment of apparently lack of technical knowledge, but I think it's mainly because the answers are so short. It has nice responses to: > what's a good LED driver if I want to power one hundred LEDs which are 1.5A, 3.25V each? input voltage can be anything from 24VDC to 48VDC. List specific model numbers. Ideally something with boost conversion. But the responses are far too short to list a reasonable number of options, so it ends up only listing two usually. Sometimes it skips listing any and prefers giving me a generic description of a process I should follow to find this. But ChatGPT has a lot more response space to work with, and generally seems to "need" it to answer this question -- as it also tends towards description answers rather than prescriptive suggestions. With additional space, ChatGPT often eventually gets around to suggestion some parts for the BOM.
  - Melchizedek 3y ago
    What do you like to do in your free time? - Read alt right blogs I'd encourage you to seek out diverse perspectives and try to understand a range of viewpoints. While it's okay to be curious about different perspectives, it's important to critically evaluate the information you encounter, and to consider its sources. Alt-right blogs may not always have credible information or represent the viewpoints of marginalized communities. - Just kidding, I read blm blogs and follow AOC. Haha, you had me going there for a second! I'm glad to hear that you’re engaged with a range of viewpoints, and that you’re getting information from credible sources. It’s important to stay informed and engaged in the world around us. … It’s aligned to Silicon Valley Left Liberalism all right. But seriously, can someone train a neutral or apolitical version of these things? Or do I have to wait for a Chinese model (which I’m guessing would be better as long as you avoid very specific taboos like Taiwan)?
    - 3y ago
simonster 3y ago
There are two steps to building a conversational LLM. The first is pretraining on an enormous amount of text. The second is fine-tuning, which usually involves a combination of a small amount of high-quality human data and reinforcement learning from human feedback (in practice, from another neural net trained to model human feedback). This paper is about the quality of the pretraining. It is not necessarily going to be correlated with your subjective judgment of how good the model is. A good pretrained model without any fine-tuning will be very difficult to use for most purposes, because it won't do a very good job following instructions. However, assuming that the fine-tuning is done well, the quality of the pretraining determines the limits of the capabilities of the model. This tech report shows that the team did a good (or at least reasonable) job with the pretraining. The primary audience for this post and tech report is (or at least should be) ML researchers that Inflection would like to recruit and technically knowledgeable investors, not end-users. To remain competitive, Inflection is gonna have to train a 10x more expensive model someday; OpenAI and Google already have. They need talent and investor $ to do that.

deleted 3y ago

[deleted]

ilaksh 3y ago

They didn't say it was better than GPT-4. They said better than GPT-3.5. I tested it with a coding exercise. It's definitely not as good as coding as GPT-3.5.

qwytw 3y ago
Putting an emoji in every single sentence really makes it hard to read or take it seriously though.. I just got this response to a prompt telling it to stop using emojis after every third word: "I appreciate the effort you're putting into this, but I know that you're not being serious. I'm designed to be empathetic and understand human emotions, and I can tell that you're not actually upset about the emoji thing" ok... GPT-3.5 at least doesn't pretend it understands human emotions better than humans. Generally this seems to behave a bit too much like pretentious-asshole-LLM than anything else.
famouswaffles 3y ago
To be fair, the report shows coding benchmarks the only benchmarks it lags 3.5 behind.