6 ms·

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.

1024core 2y ago
> they couldn't replicate it because they're idiots If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?
- twothreeone 2y ago
  It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.
  - skybrian 2y ago
    I don't know how you rebut a flawed paper without making its authors look bad? That would be a general-purpose argument against criticizing papers. Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right. The only way to avoid that is to get critical feedback before publishing the paper, and it's not always possible, so then the scientific debate happens in public.
    - twothreeone 2y ago
      The situation here is different though.. If I'm making an existence claim by demonstrating a constructive argument and then being criticized for it, the most effective response to that critique would be a second, alternative construction, not attacking the critic's argument. After all, I'm the one claiming existence.. the burden of proof is on me, not my critics.
      - skybrian 2y ago
        I don't know which argument is more constructive, though? Both teams reported what they did. They got different results. Figuring out why is the next step, and pointing out that they did different things seems useful. Though, the broader question is how useful the results of the original paper are to other people who might do the same thing.
  - xpe 2y ago
    > But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. > It's not a value judgement, just doesn't help his case at all. Calling it "bullying" looks like a value judgment to me. Am I missing something? To me, Dean's response is quite sensible, particularly given his claims the other papers made serious mistakes and have potential conflicts of interest.
    - twothreeone 2y ago
      I'm not saying "Bullying is bad and bullies are bad people", that would be a value judgement. I'm saying bullying is the strictly worse strategy for strengthening his paper's claims in this scenario. The better strategy would be to foster an environment in which people can easily replicate your claims.
      - xpe 2y ago
        I think for most people the word “bullying” has a value judgment built-in.
      - xpe 2y ago
        In a perfect world, making a paper easier to replicate has advantages, sure. (But it also has costs.) Second, even a healthy environment can be undermined by lack of skills or resources, intellectual dishonesty, or conflicts of interest.
      - xpe 2y ago
        Are you suggesting Dean take a different approach in his response? Are you saying it was already too late given the environment? (I’m also not sure I know what you mean by environment here.)
  - danielmarkbruce 2y ago
    they have open source code.
danpalmer 2y ago
> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.
- bayarearefugee 2y ago
  At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the sort of question the market will quickly decide over the next couple of years and not worth arguing over. Why do a non-zero amount of people have seemingly religious beliefs about this topic on one side or the other?
  - refulgentis 2y ago
    > why does anyone care n.b. you're on a social news site > pull way ahead of all these other "idiots" Pulling way ahead sounds sufficient, not necessary. Can we prove it's not the case? Let's say someone says that's why Gemini inference is so cheap. Can we show that's wrong? > "idiots" ?
  - pclmulqdq 2y ago
    The reason Jeff Dean cares is that his team's improvement compared to standard EDA tools was marginal at best and may have overfitted to a certain class of chips. Thus, he is defending his research because it is not widely accepted. Open source code has been out for years and in that time the EDA companies have largely done their own ML-based approaches that do not match his. He attributes this not to failings in his own research but to the detractors at these companies not giving it a fair chance. The guys at EDA companies care because Google's result makes them look like idiots when you take the paper at face value, and does advance the state of the art a bit. They have been working hard for marginal improvements, and that some team of ML people can come in and make a big splash with something like this is offensive to them. Furthermore, the result is not that impressive and does not generalize enough to be useful to them (and competent teams at these companies absolutely have checked). The fact that the result is so minor is the reason that this is so contentious.
jsnell 2y ago
It's a good thing that he didn't say that, then. The tweet just says that the reproduction attempt didn't didn't actually follow the original methodology. There is no claim that the authors of the replication attempt were "idiots" or anything similar, you just made that up. The obviously fallacious logic in "they couldn't replicate it ..., therefore it's replicable" is also a total fabrication on your part.
- twothreeone 2y ago
  A Google Nature Paper has not been replicated for over 3 years, but I'm the one fabricating stuff :D Making a novel claim implies its *_claimed_ replicability. "You did not follow the steps" is calling them idiots. The only inference I made is that he's pressed to comment. He could have said nothing.. instead he's lashing out publicly, because other people were unable to replicate it. If there's no problem replicating the work, why hasn't that happend? Any other author would be worried if a publication about their work were saying "it's not replicable" and trying their best to help replicate it.. but somehow that doesn't apply to him.
  - griomnib 2y ago
    “You can only validate my results if you have an entire Google data center worth of compute available. Since you don’t, you can’t question us.”
    - jeffbee 2y ago
      We're actually talking about the difference between Cheng using 8 GPUs and 2 CPUs while Google used 16 GPUs and 40 CPUs. These are under-your-desk levels of resources. Cheng et al authors are all affiliated with UCSD which owns the Expanse supercomputer which is orders of magnitude larger than what you would need to reproduce the original work. Cheng et al does not explain why they used fewer resources.
      - griomnib 2y ago
        That’s a fair complaint then.
make3 2y ago
"they couldn't replicate it because they're idiots, therefore it's replicable" That's literally not what he says though. He says, "they didn't replicate it so their conclusions are invalid", which is a completely different thing than what you're accusing him of, and is valid.
griomnib 2y ago
I love how he’s claiming bias due to his critic’s employer. As though working for Google has no conflicts? A company that is desperately hyping each and every “me too” AI development to juice the stock price? Jeff drank so much kool aid he forget what water is.
- jeffbee 2y ago
  He's criticizing Markov for not disclosing the conflict, not for the conflict itself. Hiding your affiliation in a scientific publication is far outside the norms of science, and they should be criticized for that. The publication we are discussing — "That Chip Has Sailed" — dismisses Markov in a few paragraph and spends the bulk its arguments on Cheng.
  - griomnib 2y ago
    I know the norms of science, I also know norms of present day Google. Nobody from Google should have the gall to accuse others of anything.
  - dogleg77 2y ago
    Markov's article has been on arxiv since 2023 before the perceived conflict. https://arxiv.org/abs/2306.09633 https://arxiv.org/abs/2306.09633 All his affiliations have been disclosed and likely known to Jeff Dean. Jeff is simply unable to respond to Markov's article in the technical dimension and is resorting to dirty tricks instead.
johnfn 2y ago
> "they couldn't replicate it because they're idiots, therefore it's replicable" Where does it say that? Dean outlines explicit steps that the authors missed in the tweet.
iamnotafraid 2y ago
Interesting point, I give u this
MicolashKyoka 2y ago
if they are idiots and couldn't replicate it, it's worth saying it. better that than sugarcoating idiocy until it harms future research.

1024core 2y ago

> they couldn't replicate it because they're idiots If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?

twothreeone 2y ago
It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.
- skybrian 2y ago
  I don't know how you rebut a flawed paper without making its authors look bad? That would be a general-purpose argument against criticizing papers. Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right. The only way to avoid that is to get critical feedback before publishing the paper, and it's not always possible, so then the scientific debate happens in public.
  - twothreeone 2y ago
    The situation here is different though.. If I'm making an existence claim by demonstrating a constructive argument and then being criticized for it, the most effective response to that critique would be a second, alternative construction, not attacking the critic's argument. After all, I'm the one claiming existence.. the burden of proof is on me, not my critics.
    - skybrian 2y ago
      I don't know which argument is more constructive, though? Both teams reported what they did. They got different results. Figuring out why is the next step, and pointing out that they did different things seems useful. Though, the broader question is how useful the results of the original paper are to other people who might do the same thing.
- xpe 2y ago
  > But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. > It's not a value judgement, just doesn't help his case at all. Calling it "bullying" looks like a value judgment to me. Am I missing something? To me, Dean's response is quite sensible, particularly given his claims the other papers made serious mistakes and have potential conflicts of interest.
  - twothreeone 2y ago
    I'm not saying "Bullying is bad and bullies are bad people", that would be a value judgement. I'm saying bullying is the strictly worse strategy for strengthening his paper's claims in this scenario. The better strategy would be to foster an environment in which people can easily replicate your claims.
    - xpe 2y ago
      I think for most people the word “bullying” has a value judgment built-in.
    - xpe 2y ago
      In a perfect world, making a paper easier to replicate has advantages, sure. (But it also has costs.) Second, even a healthy environment can be undermined by lack of skills or resources, intellectual dishonesty, or conflicts of interest.
    - xpe 2y ago
      Are you suggesting Dean take a different approach in his response? Are you saying it was already too late given the environment? (I’m also not sure I know what you mean by environment here.)
- danielmarkbruce 2y ago
  they have open source code.

danpalmer 2y ago

> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.

bayarearefugee 2y ago
At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the sort of question the market will quickly decide over the next couple of years and not worth arguing over. Why do a non-zero amount of people have seemingly religious beliefs about this topic on one side or the other?
- refulgentis 2y ago
  > why does anyone care n.b. you're on a social news site > pull way ahead of all these other "idiots" Pulling way ahead sounds sufficient, not necessary. Can we prove it's not the case? Let's say someone says that's why Gemini inference is so cheap. Can we show that's wrong? > "idiots" ?
- pclmulqdq 2y ago
  The reason Jeff Dean cares is that his team's improvement compared to standard EDA tools was marginal at best and may have overfitted to a certain class of chips. Thus, he is defending his research because it is not widely accepted. Open source code has been out for years and in that time the EDA companies have largely done their own ML-based approaches that do not match his. He attributes this not to failings in his own research but to the detractors at these companies not giving it a fair chance. The guys at EDA companies care because Google's result makes them look like idiots when you take the paper at face value, and does advance the state of the art a bit. They have been working hard for marginal improvements, and that some team of ML people can come in and make a big splash with something like this is offensive to them. Furthermore, the result is not that impressive and does not generalize enough to be useful to them (and competent teams at these companies absolutely have checked). The fact that the result is so minor is the reason that this is so contentious.

jsnell 2y ago

It's a good thing that he didn't say that, then. The tweet just says that the reproduction attempt didn't didn't actually follow the original methodology. There is no claim that the authors of the replication attempt were "idiots" or anything similar, you just made that up. The obviously fallacious logic in "they couldn't replicate it ..., therefore it's replicable" is also a total fabrication on your part.

twothreeone 2y ago
A Google Nature Paper has not been replicated for over 3 years, but I'm the one fabricating stuff :D Making a novel claim implies its *_claimed_ replicability. "You did not follow the steps" is calling them idiots. The only inference I made is that he's pressed to comment. He could have said nothing.. instead he's lashing out publicly, because other people were unable to replicate it. If there's no problem replicating the work, why hasn't that happend? Any other author would be worried if a publication about their work were saying "it's not replicable" and trying their best to help replicate it.. but somehow that doesn't apply to him.
- griomnib 2y ago
  “You can only validate my results if you have an entire Google data center worth of compute available. Since you don’t, you can’t question us.”
  - jeffbee 2y ago
    We're actually talking about the difference between Cheng using 8 GPUs and 2 CPUs while Google used 16 GPUs and 40 CPUs. These are under-your-desk levels of resources. Cheng et al authors are all affiliated with UCSD which owns the Expanse supercomputer which is orders of magnitude larger than what you would need to reproduce the original work. Cheng et al does not explain why they used fewer resources.
    - griomnib 2y ago
      That’s a fair complaint then.

make3 2y ago

"they couldn't replicate it because they're idiots, therefore it's replicable" That's literally not what he says though. He says, "they didn't replicate it so their conclusions are invalid", which is a completely different thing than what you're accusing him of, and is valid.

griomnib 2y ago

I love how he’s claiming bias due to his critic’s employer. As though working for Google has no conflicts? A company that is desperately hyping each and every “me too” AI development to juice the stock price? Jeff drank so much kool aid he forget what water is.

jeffbee 2y ago
He's criticizing Markov for not disclosing the conflict, not for the conflict itself. Hiding your affiliation in a scientific publication is far outside the norms of science, and they should be criticized for that. The publication we are discussing — "That Chip Has Sailed" — dismisses Markov in a few paragraph and spends the bulk its arguments on Cheng.
- griomnib 2y ago
  I know the norms of science, I also know norms of present day Google. Nobody from Google should have the gall to accuse others of anything.
- dogleg77 2y ago
  Markov's article has been on arxiv since 2023 before the perceived conflict. https://arxiv.org/abs/2306.09633 https://arxiv.org/abs/2306.09633 All his affiliations have been disclosed and likely known to Jeff Dean. Jeff is simply unable to respond to Markov's article in the technical dimension and is resorting to dirty tricks instead.

johnfn 2y ago

> "they couldn't replicate it because they're idiots, therefore it's replicable" Where does it say that? Dean outlines explicit steps that the authors missed in the tweet.

iamnotafraid 2y ago

Interesting point, I give u this

MicolashKyoka 2y ago

if they are idiots and couldn't replicate it, it's worth saying it. better that than sugarcoating idiocy until it harms future research.