11 ms·

Jeff Dean responds to EDA industry about AlphaChip

248 points by nsoonhui 2y ago

twothreeone 2y ago
I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.
- 1024core 2y ago
  > they couldn't replicate it because they're idiots If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?
  - twothreeone 2y ago
    It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.
    - skybrian 2y ago
      I don't know how you rebut a flawed paper without making its authors look bad? That would be a general-purpose argument against criticizing papers. Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right. The only way to avoid that is to get critical feedback before publishing the paper, and it's not always possible, so then the scientific debate happens in public.
      - twothreeone 2y ago
        The situation here is different though.. If I'm making an existence claim by demonstrating a constructive argument and then being criticized for it, the most effective response to that critique would be a second, alternative construction, not attacking the critic's argument. After all, I'm the one claiming existence.. the burden of proof is on me, not my critics.
        skybrian 2y ago
        I don't know which argument is more constructive, though? Both teams reported what they did. They got different results. Figuring out why is the next step, and pointing out that they did different things seems useful. Though, the broader question is how useful the results of the original paper are to other people who might do the same thing.
    - xpe 2y ago
      > But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. > It's not a value judgement, just doesn't help his case at all. Calling it "bullying" looks like a value judgment to me. Am I missing something? To me, Dean's response is quite sensible, particularly given his claims the other papers made serious mistakes and have potential conflicts of interest.
      - twothreeone 2y ago
        I'm not saying "Bullying is bad and bullies are bad people", that would be a value judgement. I'm saying bullying is the strictly worse strategy for strengthening his paper's claims in this scenario. The better strategy would be to foster an environment in which people can easily replicate your claims.
        xpe 2y ago
        I think for most people the word “bullying” has a value judgment built-in.
        xpe 2y ago
        In a perfect world, making a paper easier to replicate has advantages, sure. (But it also has costs.) Second, even a healthy environment can be undermined by lack of skills or resources, intellectual dishonesty, or conflicts of interest.
        xpe 2y ago
        Are you suggesting Dean take a different approach in his response? Are you saying it was already too late given the environment? (I’m also not sure I know what you mean by environment here.)
    - danielmarkbruce 2y ago
      they have open source code.
- danpalmer 2y ago
  > But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.
  - bayarearefugee 2y ago
    At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the sort of question the market will quickly decide over the next couple of years and not worth arguing over. Why do a non-zero amount of people have seemingly religious beliefs about this topic on one side or the other?
    - refulgentis 2y ago
      > why does anyone care n.b. you're on a social news site > pull way ahead of all these other "idiots" Pulling way ahead sounds sufficient, not necessary. Can we prove it's not the case? Let's say someone says that's why Gemini inference is so cheap. Can we show that's wrong? > "idiots" ?
    - pclmulqdq 2y ago
      The reason Jeff Dean cares is that his team's improvement compared to standard EDA tools was marginal at best and may have overfitted to a certain class of chips. Thus, he is defending his research because it is not widely accepted. Open source code has been out for years and in that time the EDA companies have largely done their own ML-based approaches that do not match his. He attributes this not to failings in his own research but to the detractors at these companies not giving it a fair chance. The guys at EDA companies care because Google's result makes them look like idiots when you take the paper at face value, and does advance the state of the art a bit. They have been working hard for marginal improvements, and that some team of ML people can come in and make a big splash with something like this is offensive to them. Furthermore, the result is not that impressive and does not generalize enough to be useful to them (and competent teams at these companies absolutely have checked). The fact that the result is so minor is the reason that this is so contentious.
- jsnell 2y ago
  It's a good thing that he didn't say that, then. The tweet just says that the reproduction attempt didn't didn't actually follow the original methodology. There is no claim that the authors of the replication attempt were "idiots" or anything similar, you just made that up. The obviously fallacious logic in "they couldn't replicate it ..., therefore it's replicable" is also a total fabrication on your part.
  - twothreeone 2y ago
    A Google Nature Paper has not been replicated for over 3 years, but I'm the one fabricating stuff :D Making a novel claim implies its *_claimed_ replicability. "You did not follow the steps" is calling them idiots. The only inference I made is that he's pressed to comment. He could have said nothing.. instead he's lashing out publicly, because other people were unable to replicate it. If there's no problem replicating the work, why hasn't that happend? Any other author would be worried if a publication about their work were saying "it's not replicable" and trying their best to help replicate it.. but somehow that doesn't apply to him.
    - griomnib 2y ago
      “You can only validate my results if you have an entire Google data center worth of compute available. Since you don’t, you can’t question us.”
      - jeffbee 2y ago
        We're actually talking about the difference between Cheng using 8 GPUs and 2 CPUs while Google used 16 GPUs and 40 CPUs. These are under-your-desk levels of resources. Cheng et al authors are all affiliated with UCSD which owns the Expanse supercomputer which is orders of magnitude larger than what you would need to reproduce the original work. Cheng et al does not explain why they used fewer resources.
        griomnib 2y ago
        That’s a fair complaint then.
- make3 2y ago
  "they couldn't replicate it because they're idiots, therefore it's replicable" That's literally not what he says though. He says, "they didn't replicate it so their conclusions are invalid", which is a completely different thing than what you're accusing him of, and is valid.
- griomnib 2y ago
  I love how he’s claiming bias due to his critic’s employer. As though working for Google has no conflicts? A company that is desperately hyping each and every “me too” AI development to juice the stock price? Jeff drank so much kool aid he forget what water is.
  - jeffbee 2y ago
    He's criticizing Markov for not disclosing the conflict, not for the conflict itself. Hiding your affiliation in a scientific publication is far outside the norms of science, and they should be criticized for that. The publication we are discussing — "That Chip Has Sailed" — dismisses Markov in a few paragraph and spends the bulk its arguments on Cheng.
    - griomnib 2y ago
      I know the norms of science, I also know norms of present day Google. Nobody from Google should have the gall to accuse others of anything.
    - dogleg77 2y ago
      Markov's article has been on arxiv since 2023 before the perceived conflict. https://arxiv.org/abs/2306.09633 https://arxiv.org/abs/2306.09633 All his affiliations have been disclosed and likely known to Jeff Dean. Jeff is simply unable to respond to Markov's article in the technical dimension and is resorting to dirty tricks instead.
- johnfn 2y ago
  > "they couldn't replicate it because they're idiots, therefore it's replicable" Where does it say that? Dean outlines explicit steps that the authors missed in the tweet.
- iamnotafraid 2y ago
  Interesting point, I give u this
- MicolashKyoka 2y ago
  if they are idiots and couldn't replicate it, it's worth saying it. better that than sugarcoating idiocy until it harms future research.
nsoonhui 2y ago
The context of Jeff Dean's response: https://news.ycombinator.com/item?id=41673769 https://news.ycombinator.com/item?id=41673769 https://news.ycombinator.com/item?id=41673808 https://news.ycombinator.com/item?id=41673808
bsder 2y ago
The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is array logic", "This is random logic". This is the BIG win. And it won't just be a couple of percentage points in area or a couple of days saved when it works--it will be 25%+ in area and months in time. 2) Saving a couple of percentage points in random logic isn't impressive. If I have the compute power to run EDA tools with a couple of different random seeds, at least one run will likely be a couple percentage points better. 3) I really don't understand why they don't do stuff on analog/RF. The patterns are smaller and much better matches to the kind of reinforcement learning that current "AI" is suited for. I put this snake oil in the same category as "financial advice"--if it worked, they wouldn't be sharing it and would simply be printing money by taking advantage of it.
- throwaway2037 2y ago
  > EDA companies are garbage I don't understand this comment. Can you please explain? Are they unethical? Or do they write poor software?
  - bsder 2y ago
    Yes and yes. EDA companies are gatekeeping monopolies. They absolutely abuse their monopoly position to extract huge chunks of money out of companies, and are pretty much single-handedly responsible for the fact that the hardware startup ecosystem is moribund compared to that of the software startup ecosystem. They have been horrible liars about performance and benchmarketing for decades. They dragged their feet miserably over releasing Linux versions of their software because they were extracting money based upon number of CPU licenses (everything was on Sparc which was vastly inferior). Their software hasn't really improved all that much over decades--mostly they benefited from Moore's Law. They have made a point of stifling attempts at interoperability and open data exchange. They have bought lots of competitors mostly to just shut them down. I can go on and on. The EDA companies aren't quite Oracle--but they're not far off. This is one of the reasons why Google is getting pounded over this--maybe even unfairly. People in the field are super sensitive about bullshit claims from EDA vendors--we've heard them all and been on the receiving end of the stick far too many times.
vighneshiyer 2y ago
I have published an addendum to an article I wrote about AlphaChip (https://vighneshiyer.com/misc/ml-for-placement/ https://vighneshiyer.com/misc/ml-for-placement/) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general. In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still makes it non-competitive with commercial autofloorplanners and AutoDMP. Regardless, I think the ISPD authors owe the Nature authors an even more rigorous study that addresses all their criticisms. Even if they just try to evaluate the pre-trained checkpoint that Google published, that would be a useful piece of data to add to the debate.
- wholehog 2y ago
  We're talking 16 GPUs for ~6 hrs for inference, and 48 hrs for pre-training. This is not an exorbitant amount of compute. A GPU costs $1-2/hr on the cloud market. So, ~$100-200 for inference, and ~$800-1600 for pre-training, which amortizes across chips. Cloud prices are an upper bound -- most CS labs will have way more than this available on premises. In an industry context, these costs are completely dwarfed by the rest of the chip design process. (For context, the licensing costs alone for most commercial EDA software are in the millions of dollars.)
  - bushbaba 2y ago
    h100 GPU instances are multiple orders of magnitude more expensive.
    - radq 2y ago
      Not true, H100s cost $2-3/GPU/hr on the open market.
      - menaerus 2y ago
        Yes, they even do at $1/GPU/hr. However, 8xH100 cluster at full utilization is ~8kWh of electricity and costs almost ~0.5M$. 16xH100 cluster is probably 2x of that. How many years before you break-even at ~24$/GPU/day income?
AtlasBarfed 2y ago
How the hell would you verify an AI-generated silicon design? Like, for a CPU, you want to be sure it behaves properly for the given inputs. Anyone remember that floating point error in, was it Pentium IIs or Pentium IIIs? I mean, I guess if the chip is designed for AI, and AIs are inherently nonguaranteed output/responses, then the AI chip design being nonguaranteed isn't any difference in nonguarantees. Unless it is...
- asveikau 2y ago
  The famous FPU issue that I can think of was the original Pentium.
- lisper 2y ago
  > How the hell would you verify an AI-generated silicon design? The same way you verify a human-generated one. > Anyone remember that floating point error in, was it Pentium IIs or Pentium IIIs? That was 1994. The industry has come a long way in the intervening 30 years.
  - mnau 2y ago
    > The industry has come a long way in the intervening 30 years. Didn't Intel screw up design recently? Oxidation and degradation of ring bus IIRC.
- gwervc 2y ago
  A well working CPU is probably beside the point. What's important now is for researchers to publish papers using or speaking about AI. Then executives and managers to deploy AI in their companies. Then selling AI PC (somehow, we are already at this step). Whatever the results are. Customers issues will be solved by using more AI (think chatbots) until morale improves.
- quadrature 2y ago
  > How the hell would you verify an AI-generated silicon design? I think you're asking a different question, but in the context of the OP researchers are exploring AI for solving deterministic but intractable problems in the field of chip design and not generating designs end to end. Here's an excerpt from the paper. "The objective is to place a netlist graph of macros (e.g., SRAMs) and standard cells (logic gates, such as NAND, NOR, and XOR) onto a chip canvas, such that power, performance, and area (PPA) are optimized, while adhering to constraints on placement density and routing congestion (described in Sections 3.3.6 and 3.3.5). Despite decades of research on this problem, it is still necessary for human experts to iterate for weeks with the existing placement tools, in order to produce solutions that meet multi-faceted design criteria." The hope is that Reinforcement Learning can find solutions to such complex optimization problems.
wholehog 2y ago
The paper: https://arxiv.org/abs/2411.10053 https://arxiv.org/abs/2411.10053
- deleted 2y ago
  [deleted]
_cs2017_ 2y ago
Curious why there's so much emotion and unpleasantness in this dispute? How did it evolve from the boring academic argument about benchmarks, significance, etc to a battle of personal attacks?
- boredatoms 2y ago
  A lot of people work on non-AI implementations
  - benreesman 2y ago
    This is a big part of the reason. But it behooves us to ask why a key innovation in a field (and I trust Jeff Dean that this is one, I’ve never seen any reason to doubt either his integrity or ability) should produce such a reaction. What could make people act not just chagrined that their approach wasn’t the end state, but as though it was existential to discredit such an innovation? Surely all of the people who did the work that the innovation rests on should be confident they will be relevant, involved, comfortable, and safe in the post-innovation world? And yet it’s not clear they should feel this way. Luddism seems an unfounded ideology over the scope of history since the origin of the term. But over the period since “AI” entered the public discussion at the current level? Almost two years exactly? Making the Luddite agenda credible has seemed a ubiquitous talking point. Over that time frame technical people have been laid off in staggering numbers, a steadily-shrinking number of employers have been slashing headcount and posting EPS beats, and “AI” has been mentioned in every breath. It’s so extreme that even sophisticated knowledge of the kinds of subject matter that goes into AlphaChip is (allegedly) useless without access to the Hopper FLOPs. If the AI Cartel was a little less rapacious, people might be a little more open to embracing the AI revolution.
  - wholehog 2y ago
    Bitter Lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
- jart 2y ago
  If you think this is unpleasant, you should see the environmentalists who try to take a poke at Jeff Dean on Twitter.
rowanG077 2y ago
I don't get. Why isn't the model open if it works? If it isn't this is just a fart in the wind. If it is the findings should be straightforward to replicate.
- amelius 2y ago
  Yes, the community should force Nature to up its standards or ditch it. Software replication should be trivial in this day and age.
  - woodson 2y ago
    All these papers doing "research" on how to better prompt ChatGPT would be unpublishable then, given that API access to older models gets retired, so the findings of these papers can no longer be reproduced. (I agree with you in principle; my example above is meant to show that standards for things such as reproducibility aren't easily defined. There are so many factors to consider.)
    - amelius 2y ago
      Well since you put "research" in quotes, I think you also agree that this type of work does not really belong in a quality journal with a high impact factor ;)
      - rowanG077 2y ago
        This, their training data doesn't even seem to be open either. So it's literally impossible to replicate their model. This makes me highly skeptical.
- wholehog 2y ago
  It is open: https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training
  - LittleTimothy 2y ago
gabegobblegoldi 2y ago
Additional context: Jeff Dean has been accused of fraud and misconduct in AlphaChip. https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf
- jiveturkey 2y ago
  the link is for a wrongful termination lawsuit, related to the fraud but not a case for the fraud itself. settled may 2024
  - Analemma_ 2y ago
    "Settled" does not mean "Dean did nothing wrong". It means "Google paid the plaintiffs a lot of money so they'd stop saying publicly that Dean did something wrong", which is very different.
    - deleted 2y ago
      [deleted]
oesa 2y ago
In the tweet Jeff Dean says that Cheng at al. failed to follow the steps required to replicate the work of the Google researchers. Specifically: > In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), robbing our learning-based method of its ability to learn from other chip designs But in the Circuit Training Google repo[1] they specifically say: > Our results training from scratch are comparable or better than the reported results in the paper (on page 22) which used fine-tuning from a pre-trained model. I may be misunderstanding something here, but which one is it? Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction? Also, the UCSD group had to reverse-engineer several steps to reproduce the results so it seems like the paper's results weren't reproducible by themselves. [1]: https://github.com/google-research/circuit_training/blob/main/docs/ARIANE.md#results https://github.com/google-research/circuit_training/blob/mai...
- gabegobblegoldi 2y ago
  Markov’s paper also has links to Google papers from two different sets of authors that shows minimal advantage of pretraining. And given the small number of benchmarks using a pretrained model from Google whose provenance is not known would be counterproductive. Google likely trained it on all available benchmarks to regurgitate the best solutions of commercial tools.
- cma 2y ago
  Training from scratch could presumably mean including the new design attempts and old designs mixed in. So no contradiction: pretrain on old designs then finetune on new design, vs train on everything mixed together throughout. Finetuning can cause catastrophic forgetting. Both could have better performance than not including old designs.
- wholehog 2y ago
  > Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction? The Circuit Training repo was just going through an example. It is common for an open-source repo to describe simple examples for testing / validating your setup --- that does not mean this is how you should get optimal results in general. The confusion may stem from their statement that, in this example, they produced results that were comparable with the pre-trained results in the paper. This is clearly not a general repudiation of pre-training. If Cheng et al. genuinely felt this was ambiguous, they should have reached out to the corresponding authors. If they ran into some part of the repo they felt they had to "reverse-engineer", they should have asked about that, too.
puff_pastry 2y ago
The biggest disappointment is that these discussions are still happening on Twitter/X. Leave that platform already
- xpe 2y ago
  Sure, we want individuals to act in a way to mitigate collective action problems. But the collective action problem exists (by definition) because individuals are trapped in some variation of a prisoner's dilemma. So, collective action problems are nearly a statistical certainty across a wide variety of situations. And yet we still "blame" individuals? We should know better.
  - pas 2y ago
    So you're saying Head of AI of Google of Jeff can't choose a better venue? He's not the first Jeffery with a lot of power who doesn't care.
    - xpe 2y ago
      > So you're saying Head of AI of Google of Jeff can't choose a better venue? Phrasing it this way isn't useful. Talking about choice in the abstract doesn't help with a game-theoretic analysis. You need costs and benefits too. There are many people who face something like a prisoner's dilemma (on Twitter, for example). We could assess the cost-benefit of a particular person leaving Twitter. We could even judge them according to some standards (ethical, rational, and so on). But why bother?... ...Think about major collective action failures. How often are they the result of just one person's decisions? How does "blaming" or "judging" an individual help make a situation better? This effort on blaming could be better spent elsewhere; such as understanding the system and finding leverage points. There are cases where blaming/guilt can help, but only in the prospective sense: if a person knows they will be blamed and face consequences for an action, it will make that action more costly. This might be enough to deter than decision. But do you think this applies in the context of the "do I leave Twitter?" decision? I'd say very little, if at all.
      - 2y ago
segmondy 2y ago
It's ridiculous how expensive the wrong hire can be https://www.wired.com/story/google-brain-ai-researcher-fired-tension/ https://www.wired.com/story/google-brain-ai-researcher-fired...
- benreesman 2y ago
  It is indeed a big deal to hire people who will commit or contrive at fraud: academic, financial, or otherwise. But the best (probably only) way to put downward pressure on that is via internal incentives, controls, and culture. You push hard enough for such percent per cadence with no upper bound and graduate the folks who reliably deliver it without checking if the win was there to begin with? This is scale-invariant: it could be in a pod, a department, a company, a hedge fund that owns much of those companies, a fund of those funds, the federal government. Sooner or later your leadership is substantially penetrated by the unscrupulous. We see this in academia with the spate of scandals around publications. We see this in finance with, who can even count that high anymore. You see Holmes and SBF in prison but the folks they funded still at the apex of relevance and everyone from that clique? Everyone who didn’t just fall of a turnip truck knows has carried that ideology with them and has better lawyers now. There’s an old saw that a “fish rots from the head”. We can’t look at every manner of shadiness and constant scandal from the iconic leaders of our STEM industry and say “good for them, they outsmarted the system” and expect any result other than a broad-spectrum attack on any honest, fair, equitable status quo. We all voted with our feet (and I did my share of that too before I quit in disgust) for a “might makes right” quasi-religious system of ideals, known variously as Objectivism, Effective Altruism, and Capitalism (of which it is no kind). We shouldn’t be surprised that everything is kind of tarnished sticky now. The answer today? I don’t know. Work for the less bad as opposed to more bad companies, speak out at least anonymously about abuses, listen to the leaders speak in interviews and scrutinize it. I’m open to suggestions.
- fxtentacle 2y ago
  It seems that Chatterjee - the bad guy in your linked article - is now suing Google because he thinks he got canned for pointing out that his boss - Jeff Dean mentioned in the article discussed here - was knowingly publishing fraudulent claims. "To be clear, we do NOT have evidence to believe that RL outperforms academic state-of—art and strongest commercial macro placers. The comparisons for the latter were done so poorly that in many cases the commercial tool failed to run due to installation issues." and that's supposedly a screenshot from an internal presentation done by Jeff Dean. https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf As an outsider, I find it very difficult to judge if Chatterjee was a bad and expensive hire (because he suppressed good results by coworkers) or if he was a very valuable employee (because he tried to prevent publishing false statements).
dang 2y ago
Related. Others? AI Alone Isn't Ready for Chip Design - https://news.ycombinator.com/item?id=42207373 https://news.ycombinator.com/item?id=42207373 - Nov 2024 (2 comments) That Chip Has Sailed: Critique of Unfounded Skepticism Around AI for Chip Design - https://news.ycombinator.com/item?id=42172967 https://news.ycombinator.com/item?id=42172967 - Nov 2024 (9 comments) Reevaluating Google's Reinforcement Learning for IC Macro Placement (AlphaChip) - https://news.ycombinator.com/item?id=42042046 https://news.ycombinator.com/item?id=42042046 - Nov 2024 (1 comment) How AlphaChip transformed computer chip design - https://news.ycombinator.com/item?id=41672110 https://news.ycombinator.com/item?id=41672110 - Sept 2024 (194 comments) Tension Inside Google over a Fired AI Researcher’s Conduct - https://news.ycombinator.com/item?id=31576301 https://news.ycombinator.com/item?id=31576301 - May 2022 (23 comments) Google is using AI to design chips that will accelerate AI - https://news.ycombinator.com/item?id=22717983 https://news.ycombinator.com/item?id=22717983 - March 2020 (1 comment)
lumb63 2y ago
I’ve not followed this story at all, and have no idea what is true or not, but generally when people use a boatload of adjectives which serve no purpose but to skew opinion, I assume they are not being honest. Using certain words to describe a situation does not make the situation what the author is saying, and if it is as they say, then the actual content should speak for itself. For instance: > Much of this unfounded skepticism is driven by a deeply flawed non-peer-reviewed publication by Cheng et al. that claimed to replicate our approach but failed to follow our methodology in major ways. In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), This could easily be written more succinctly, and with less bias, as: > Much of this skepticism is driven by a publication by Cheng et al. that claimed to replicate our approach but failed to follow our methodology in major ways. In particular the authors did no pre-training, Calling the skepticism unfounded or deeply flawed does not make it so, and pointing out that a particular publication is not peer reviewed does not make its contents false. The authors would be better served by maintaining a more neutral tone rather than coming off accusatory and heavily biased.
- wholehog 2y ago
  If someone tries to run your method but messes it up, and then accuses you of fraud when the results don't match their expectations, I'm not sure they're entitled to a neutral tone in response. Maybe you're right, and a more neutral tone would have been effective! I think it's just that Jeff is just really done with this.
- dogleg77 2y ago
  At least they are acknowledging skepticism now. That's a step in the right direction.
Keyframe 2y ago
At this point in time, why wouldn't we give at least benefit of the doubt to Jeff Dean immediately? His track record is second to none, and he's still going strong. Has something happened that cast a shadow on him? Sometimes it is the messenger that brings in the weight.
- gabegobblegoldi 2y ago
  Looks like he aligned himself with the wrong folks here. He is a system builder at heart but not an expert in chip design or EDA. And also not really an ML researcher. Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out. His focus on this project didn’t help with his case at Google. They moved all the important stuff away from him and gave it to Demis last year and left him with an honorary title. Quite sad really for someone of his accomplishments.
  - alsodumb 2y ago
    I mean Jeff Dean is probably more ML researcher than probably 90% of the ML researchers out there. Sure, he may not be working on state of the art stuff himself; but he's too up the chain to do that.
    - dogleg77 2y ago
      That's an appeal to authority, and not an effective one. Jeff Dean doesn't have a good track record in chip design.
      - wholehog 2y ago
        What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.
        sijnapq 2y ago
7e 2y ago
Google's ultimate proof is to build better chips than the competition. Which, at this point, they cannot.
LittleTimothy 2y ago
"You didn't use enough compute in your reproduction of our methods" is kind of a funny criticism today. Well yeah, you're Google. Sorry guys, no reviewing of our methodology unless you own a cloud. It wouldn't surprise me if it's true that more compute, more pre-training etc. provides a lot of utility, but that does make it difficult to verify the work. One interesting aspect of this though is vice-versa, whilst Google has oodles of compute, Synopsys has oodles of data to train on (if, and this is a massive if, they can get away with training on customer IP).
- dogleg77 2y ago
  Chip designers run EDAtools on premises. How would EDA companies have access to customer data? Maybe for debugging purposes under NDA, but that won't allow training on customer IP.

248 points by nsoonhui 2y ago

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.

1024core 2y ago
> they couldn't replicate it because they're idiots If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?
- twothreeone 2y ago
  It's not a value judgement, just doesn't help his case at all. He'd need to counter the replication problem, but apparently that's not an option. Instead, he's making people who were unable to replicate it look bad, which actually strengthens their criticism.
  - skybrian 2y ago
    I don't know how you rebut a flawed paper without making its authors look bad? That would be a general-purpose argument against criticizing papers. Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right. The only way to avoid that is to get critical feedback before publishing the paper, and it's not always possible, so then the scientific debate happens in public.
    - twothreeone 2y ago
      The situation here is different though.. If I'm making an existence claim by demonstrating a constructive argument and then being criticized for it, the most effective response to that critique would be a second, alternative construction, not attacking the critic's argument. After all, I'm the one claiming existence.. the burden of proof is on me, not my critics.
      - skybrian 2y ago
        I don't know which argument is more constructive, though? Both teams reported what they did. They got different results. Figuring out why is the next step, and pointing out that they did different things seems useful. Though, the broader question is how useful the results of the original paper are to other people who might do the same thing.
  - xpe 2y ago
    > But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. > It's not a value judgement, just doesn't help his case at all. Calling it "bullying" looks like a value judgment to me. Am I missing something? To me, Dean's response is quite sensible, particularly given his claims the other papers made serious mistakes and have potential conflicts of interest.
    - twothreeone 2y ago
      I'm not saying "Bullying is bad and bullies are bad people", that would be a value judgement. I'm saying bullying is the strictly worse strategy for strengthening his paper's claims in this scenario. The better strategy would be to foster an environment in which people can easily replicate your claims.
      - xpe 2y ago
        I think for most people the word “bullying” has a value judgment built-in.
      - xpe 2y ago
        In a perfect world, making a paper easier to replicate has advantages, sure. (But it also has costs.) Second, even a healthy environment can be undermined by lack of skills or resources, intellectual dishonesty, or conflicts of interest.
      - xpe 2y ago
        Are you suggesting Dean take a different approach in his response? Are you saying it was already too late given the environment? (I’m also not sure I know what you mean by environment here.)
  - danielmarkbruce 2y ago
    they have open source code.
danpalmer 2y ago
> But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.
- bayarearefugee 2y ago
  At the end of the day my question is simply why does anyone care about the drama over this one way or another? Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the sort of question the market will quickly decide over the next couple of years and not worth arguing over. Why do a non-zero amount of people have seemingly religious beliefs about this topic on one side or the other?
  - refulgentis 2y ago
    > why does anyone care n.b. you're on a social news site > pull way ahead of all these other "idiots" Pulling way ahead sounds sufficient, not necessary. Can we prove it's not the case? Let's say someone says that's why Gemini inference is so cheap. Can we show that's wrong? > "idiots" ?
  - pclmulqdq 2y ago
    The reason Jeff Dean cares is that his team's improvement compared to standard EDA tools was marginal at best and may have overfitted to a certain class of chips. Thus, he is defending his research because it is not widely accepted. Open source code has been out for years and in that time the EDA companies have largely done their own ML-based approaches that do not match his. He attributes this not to failings in his own research but to the detractors at these companies not giving it a fair chance. The guys at EDA companies care because Google's result makes them look like idiots when you take the paper at face value, and does advance the state of the art a bit. They have been working hard for marginal improvements, and that some team of ML people can come in and make a big splash with something like this is offensive to them. Furthermore, the result is not that impressive and does not generalize enough to be useful to them (and competent teams at these companies absolutely have checked). The fact that the result is so minor is the reason that this is so contentious.
jsnell 2y ago
It's a good thing that he didn't say that, then. The tweet just says that the reproduction attempt didn't didn't actually follow the original methodology. There is no claim that the authors of the replication attempt were "idiots" or anything similar, you just made that up. The obviously fallacious logic in "they couldn't replicate it ..., therefore it's replicable" is also a total fabrication on your part.
- twothreeone 2y ago
  A Google Nature Paper has not been replicated for over 3 years, but I'm the one fabricating stuff :D Making a novel claim implies its *_claimed_ replicability. "You did not follow the steps" is calling them idiots. The only inference I made is that he's pressed to comment. He could have said nothing.. instead he's lashing out publicly, because other people were unable to replicate it. If there's no problem replicating the work, why hasn't that happend? Any other author would be worried if a publication about their work were saying "it's not replicable" and trying their best to help replicate it.. but somehow that doesn't apply to him.
  - griomnib 2y ago
    “You can only validate my results if you have an entire Google data center worth of compute available. Since you don’t, you can’t question us.”
    - jeffbee 2y ago
      We're actually talking about the difference between Cheng using 8 GPUs and 2 CPUs while Google used 16 GPUs and 40 CPUs. These are under-your-desk levels of resources. Cheng et al authors are all affiliated with UCSD which owns the Expanse supercomputer which is orders of magnitude larger than what you would need to reproduce the original work. Cheng et al does not explain why they used fewer resources.
      - griomnib 2y ago
        That’s a fair complaint then.
make3 2y ago
"they couldn't replicate it because they're idiots, therefore it's replicable" That's literally not what he says though. He says, "they didn't replicate it so their conclusions are invalid", which is a completely different thing than what you're accusing him of, and is valid.
griomnib 2y ago
I love how he’s claiming bias due to his critic’s employer. As though working for Google has no conflicts? A company that is desperately hyping each and every “me too” AI development to juice the stock price? Jeff drank so much kool aid he forget what water is.
- jeffbee 2y ago
  He's criticizing Markov for not disclosing the conflict, not for the conflict itself. Hiding your affiliation in a scientific publication is far outside the norms of science, and they should be criticized for that. The publication we are discussing — "That Chip Has Sailed" — dismisses Markov in a few paragraph and spends the bulk its arguments on Cheng.
  - griomnib 2y ago
    I know the norms of science, I also know norms of present day Google. Nobody from Google should have the gall to accuse others of anything.
  - dogleg77 2y ago
    Markov's article has been on arxiv since 2023 before the perceived conflict. https://arxiv.org/abs/2306.09633 https://arxiv.org/abs/2306.09633 All his affiliations have been disclosed and likely known to Jeff Dean. Jeff is simply unable to respond to Markov's article in the technical dimension and is resorting to dirty tricks instead.
johnfn 2y ago
> "they couldn't replicate it because they're idiots, therefore it's replicable" Where does it say that? Dean outlines explicit steps that the authors missed in the tweet.
iamnotafraid 2y ago
Interesting point, I give u this
MicolashKyoka 2y ago
if they are idiots and couldn't replicate it, it's worth saying it. better that than sugarcoating idiocy until it harms future research.

nsoonhui 2y ago

The context of Jeff Dean's response: https://news.ycombinator.com/item?id=41673769 https://news.ycombinator.com/item?id=41673769 https://news.ycombinator.com/item?id=41673808 https://news.ycombinator.com/item?id=41673808

bsder 2y ago

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil. This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple. That having been said: 1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is array logic", "This is random logic". This is the BIG win. And it won't just be a couple of percentage points in area or a couple of days saved when it works--it will be 25%+ in area and months in time. 2) Saving a couple of percentage points in random logic isn't impressive. If I have the compute power to run EDA tools with a couple of different random seeds, at least one run will likely be a couple percentage points better. 3) I really don't understand why they don't do stuff on analog/RF. The patterns are smaller and much better matches to the kind of reinforcement learning that current "AI" is suited for. I put this snake oil in the same category as "financial advice"--if it worked, they wouldn't be sharing it and would simply be printing money by taking advantage of it.

throwaway2037 2y ago
> EDA companies are garbage I don't understand this comment. Can you please explain? Are they unethical? Or do they write poor software?
- bsder 2y ago
  Yes and yes. EDA companies are gatekeeping monopolies. They absolutely abuse their monopoly position to extract huge chunks of money out of companies, and are pretty much single-handedly responsible for the fact that the hardware startup ecosystem is moribund compared to that of the software startup ecosystem. They have been horrible liars about performance and benchmarketing for decades. They dragged their feet miserably over releasing Linux versions of their software because they were extracting money based upon number of CPU licenses (everything was on Sparc which was vastly inferior). Their software hasn't really improved all that much over decades--mostly they benefited from Moore's Law. They have made a point of stifling attempts at interoperability and open data exchange. They have bought lots of competitors mostly to just shut them down. I can go on and on. The EDA companies aren't quite Oracle--but they're not far off. This is one of the reasons why Google is getting pounded over this--maybe even unfairly. People in the field are super sensitive about bullshit claims from EDA vendors--we've heard them all and been on the receiving end of the stick far too many times.

vighneshiyer 2y ago

I have published an addendum to an article I wrote about AlphaChip (https://vighneshiyer.com/misc/ml-for-placement/ https://vighneshiyer.com/misc/ml-for-placement/) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general. In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still makes it non-competitive with commercial autofloorplanners and AutoDMP. Regardless, I think the ISPD authors owe the Nature authors an even more rigorous study that addresses all their criticisms. Even if they just try to evaluate the pre-trained checkpoint that Google published, that would be a useful piece of data to add to the debate.

wholehog 2y ago
We're talking 16 GPUs for ~6 hrs for inference, and 48 hrs for pre-training. This is not an exorbitant amount of compute. A GPU costs $1-2/hr on the cloud market. So, ~$100-200 for inference, and ~$800-1600 for pre-training, which amortizes across chips. Cloud prices are an upper bound -- most CS labs will have way more than this available on premises. In an industry context, these costs are completely dwarfed by the rest of the chip design process. (For context, the licensing costs alone for most commercial EDA software are in the millions of dollars.)
- bushbaba 2y ago
  h100 GPU instances are multiple orders of magnitude more expensive.
  - radq 2y ago
    Not true, H100s cost $2-3/GPU/hr on the open market.
    - menaerus 2y ago
      Yes, they even do at $1/GPU/hr. However, 8xH100 cluster at full utilization is ~8kWh of electricity and costs almost ~0.5M$. 16xH100 cluster is probably 2x of that. How many years before you break-even at ~24$/GPU/day income?

AtlasBarfed 2y ago

How the hell would you verify an AI-generated silicon design? Like, for a CPU, you want to be sure it behaves properly for the given inputs. Anyone remember that floating point error in, was it Pentium IIs or Pentium IIIs? I mean, I guess if the chip is designed for AI, and AIs are inherently nonguaranteed output/responses, then the AI chip design being nonguaranteed isn't any difference in nonguarantees. Unless it is...

asveikau 2y ago
The famous FPU issue that I can think of was the original Pentium.
lisper 2y ago
> How the hell would you verify an AI-generated silicon design? The same way you verify a human-generated one. > Anyone remember that floating point error in, was it Pentium IIs or Pentium IIIs? That was 1994. The industry has come a long way in the intervening 30 years.
- mnau 2y ago
  > The industry has come a long way in the intervening 30 years. Didn't Intel screw up design recently? Oxidation and degradation of ring bus IIRC.
gwervc 2y ago
A well working CPU is probably beside the point. What's important now is for researchers to publish papers using or speaking about AI. Then executives and managers to deploy AI in their companies. Then selling AI PC (somehow, we are already at this step). Whatever the results are. Customers issues will be solved by using more AI (think chatbots) until morale improves.
quadrature 2y ago
> How the hell would you verify an AI-generated silicon design? I think you're asking a different question, but in the context of the OP researchers are exploring AI for solving deterministic but intractable problems in the field of chip design and not generating designs end to end. Here's an excerpt from the paper. "The objective is to place a netlist graph of macros (e.g., SRAMs) and standard cells (logic gates, such as NAND, NOR, and XOR) onto a chip canvas, such that power, performance, and area (PPA) are optimized, while adhering to constraints on placement density and routing congestion (described in Sections 3.3.6 and 3.3.5). Despite decades of research on this problem, it is still necessary for human experts to iterate for weeks with the existing placement tools, in order to produce solutions that meet multi-faceted design criteria." The hope is that Reinforcement Learning can find solutions to such complex optimization problems.

wholehog 2y ago

The paper: https://arxiv.org/abs/2411.10053 https://arxiv.org/abs/2411.10053

deleted 2y ago
[deleted]

_cs2017_ 2y ago

Curious why there's so much emotion and unpleasantness in this dispute? How did it evolve from the boring academic argument about benchmarks, significance, etc to a battle of personal attacks?

boredatoms 2y ago
A lot of people work on non-AI implementations
- benreesman 2y ago
  This is a big part of the reason. But it behooves us to ask why a key innovation in a field (and I trust Jeff Dean that this is one, I’ve never seen any reason to doubt either his integrity or ability) should produce such a reaction. What could make people act not just chagrined that their approach wasn’t the end state, but as though it was existential to discredit such an innovation? Surely all of the people who did the work that the innovation rests on should be confident they will be relevant, involved, comfortable, and safe in the post-innovation world? And yet it’s not clear they should feel this way. Luddism seems an unfounded ideology over the scope of history since the origin of the term. But over the period since “AI” entered the public discussion at the current level? Almost two years exactly? Making the Luddite agenda credible has seemed a ubiquitous talking point. Over that time frame technical people have been laid off in staggering numbers, a steadily-shrinking number of employers have been slashing headcount and posting EPS beats, and “AI” has been mentioned in every breath. It’s so extreme that even sophisticated knowledge of the kinds of subject matter that goes into AlphaChip is (allegedly) useless without access to the Hopper FLOPs. If the AI Cartel was a little less rapacious, people might be a little more open to embracing the AI revolution.
- wholehog 2y ago
  Bitter Lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
jart 2y ago
If you think this is unpleasant, you should see the environmentalists who try to take a poke at Jeff Dean on Twitter.

rowanG077 2y ago

I don't get. Why isn't the model open if it works? If it isn't this is just a fart in the wind. If it is the findings should be straightforward to replicate.

amelius 2y ago
Yes, the community should force Nature to up its standards or ditch it. Software replication should be trivial in this day and age.
- woodson 2y ago
  All these papers doing "research" on how to better prompt ChatGPT would be unpublishable then, given that API access to older models gets retired, so the findings of these papers can no longer be reproduced. (I agree with you in principle; my example above is meant to show that standards for things such as reproducibility aren't easily defined. There are so many factors to consider.)
  - amelius 2y ago
    Well since you put "research" in quotes, I think you also agree that this type of work does not really belong in a quality journal with a high impact factor ;)
    - rowanG077 2y ago
      This, their training data doesn't even seem to be open either. So it's literally impossible to replicate their model. This makes me highly skeptical.
wholehog 2y ago
It is open: https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training
- LittleTimothy 2y ago

gabegobblegoldi 2y ago

Additional context: Jeff Dean has been accused of fraud and misconduct in AlphaChip. https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf

jiveturkey 2y ago
the link is for a wrongful termination lawsuit, related to the fraud but not a case for the fraud itself. settled may 2024
- Analemma_ 2y ago
  "Settled" does not mean "Dean did nothing wrong". It means "Google paid the plaintiffs a lot of money so they'd stop saying publicly that Dean did something wrong", which is very different.
  - deleted 2y ago
    [deleted]

oesa 2y ago

In the tweet Jeff Dean says that Cheng at al. failed to follow the steps required to replicate the work of the Google researchers. Specifically: > In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), robbing our learning-based method of its ability to learn from other chip designs But in the Circuit Training Google repo[1] they specifically say: > Our results training from scratch are comparable or better than the reported results in the paper (on page 22) which used fine-tuning from a pre-trained model. I may be misunderstanding something here, but which one is it? Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction? Also, the UCSD group had to reverse-engineer several steps to reproduce the results so it seems like the paper's results weren't reproducible by themselves. [1]: https://github.com/google-research/circuit_training/blob/main/docs/ARIANE.md#results https://github.com/google-research/circuit_training/blob/mai...

gabegobblegoldi 2y ago
Markov’s paper also has links to Google papers from two different sets of authors that shows minimal advantage of pretraining. And given the small number of benchmarks using a pretrained model from Google whose provenance is not known would be counterproductive. Google likely trained it on all available benchmarks to regurgitate the best solutions of commercial tools.
cma 2y ago
Training from scratch could presumably mean including the new design attempts and old designs mixed in. So no contradiction: pretrain on old designs then finetune on new design, vs train on everything mixed together throughout. Finetuning can cause catastrophic forgetting. Both could have better performance than not including old designs.
wholehog 2y ago
> Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction? The Circuit Training repo was just going through an example. It is common for an open-source repo to describe simple examples for testing / validating your setup --- that does not mean this is how you should get optimal results in general. The confusion may stem from their statement that, in this example, they produced results that were comparable with the pre-trained results in the paper. This is clearly not a general repudiation of pre-training. If Cheng et al. genuinely felt this was ambiguous, they should have reached out to the corresponding authors. If they ran into some part of the repo they felt they had to "reverse-engineer", they should have asked about that, too.

puff_pastry 2y ago

The biggest disappointment is that these discussions are still happening on Twitter/X. Leave that platform already

xpe 2y ago
Sure, we want individuals to act in a way to mitigate collective action problems. But the collective action problem exists (by definition) because individuals are trapped in some variation of a prisoner's dilemma. So, collective action problems are nearly a statistical certainty across a wide variety of situations. And yet we still "blame" individuals? We should know better.
- pas 2y ago
  So you're saying Head of AI of Google of Jeff can't choose a better venue? He's not the first Jeffery with a lot of power who doesn't care.
  - xpe 2y ago
    > So you're saying Head of AI of Google of Jeff can't choose a better venue? Phrasing it this way isn't useful. Talking about choice in the abstract doesn't help with a game-theoretic analysis. You need costs and benefits too. There are many people who face something like a prisoner's dilemma (on Twitter, for example). We could assess the cost-benefit of a particular person leaving Twitter. We could even judge them according to some standards (ethical, rational, and so on). But why bother?... ...Think about major collective action failures. How often are they the result of just one person's decisions? How does "blaming" or "judging" an individual help make a situation better? This effort on blaming could be better spent elsewhere; such as understanding the system and finding leverage points. There are cases where blaming/guilt can help, but only in the prospective sense: if a person knows they will be blamed and face consequences for an action, it will make that action more costly. This might be enough to deter than decision. But do you think this applies in the context of the "do I leave Twitter?" decision? I'd say very little, if at all.
    - 2y ago

segmondy 2y ago

It's ridiculous how expensive the wrong hire can be https://www.wired.com/story/google-brain-ai-researcher-fired-tension/ https://www.wired.com/story/google-brain-ai-researcher-fired...

benreesman 2y ago
It is indeed a big deal to hire people who will commit or contrive at fraud: academic, financial, or otherwise. But the best (probably only) way to put downward pressure on that is via internal incentives, controls, and culture. You push hard enough for such percent per cadence with no upper bound and graduate the folks who reliably deliver it without checking if the win was there to begin with? This is scale-invariant: it could be in a pod, a department, a company, a hedge fund that owns much of those companies, a fund of those funds, the federal government. Sooner or later your leadership is substantially penetrated by the unscrupulous. We see this in academia with the spate of scandals around publications. We see this in finance with, who can even count that high anymore. You see Holmes and SBF in prison but the folks they funded still at the apex of relevance and everyone from that clique? Everyone who didn’t just fall of a turnip truck knows has carried that ideology with them and has better lawyers now. There’s an old saw that a “fish rots from the head”. We can’t look at every manner of shadiness and constant scandal from the iconic leaders of our STEM industry and say “good for them, they outsmarted the system” and expect any result other than a broad-spectrum attack on any honest, fair, equitable status quo. We all voted with our feet (and I did my share of that too before I quit in disgust) for a “might makes right” quasi-religious system of ideals, known variously as Objectivism, Effective Altruism, and Capitalism (of which it is no kind). We shouldn’t be surprised that everything is kind of tarnished sticky now. The answer today? I don’t know. Work for the less bad as opposed to more bad companies, speak out at least anonymously about abuses, listen to the leaders speak in interviews and scrutinize it. I’m open to suggestions.
fxtentacle 2y ago
It seems that Chatterjee - the bad guy in your linked article - is now suing Google because he thinks he got canned for pointing out that his boss - Jeff Dean mentioned in the article discussed here - was knowingly publishing fraudulent claims. "To be clear, we do NOT have evidence to believe that RL outperforms academic state-of—art and strongest commercial macro placers. The comparisons for the latter were done so poorly that in many cases the commercial tool failed to run due to installation issues." and that's supposedly a screenshot from an internal presentation done by Jeff Dean. https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf As an outsider, I find it very difficult to judge if Chatterjee was a bad and expensive hire (because he suppressed good results by coworkers) or if he was a very valuable employee (because he tried to prevent publishing false statements).

dang 2y ago

Related. Others? AI Alone Isn't Ready for Chip Design - https://news.ycombinator.com/item?id=42207373 https://news.ycombinator.com/item?id=42207373 - Nov 2024 (2 comments) That Chip Has Sailed: Critique of Unfounded Skepticism Around AI for Chip Design - https://news.ycombinator.com/item?id=42172967 https://news.ycombinator.com/item?id=42172967 - Nov 2024 (9 comments) Reevaluating Google's Reinforcement Learning for IC Macro Placement (AlphaChip) - https://news.ycombinator.com/item?id=42042046 https://news.ycombinator.com/item?id=42042046 - Nov 2024 (1 comment) How AlphaChip transformed computer chip design - https://news.ycombinator.com/item?id=41672110 https://news.ycombinator.com/item?id=41672110 - Sept 2024 (194 comments) Tension Inside Google over a Fired AI Researcher’s Conduct - https://news.ycombinator.com/item?id=31576301 https://news.ycombinator.com/item?id=31576301 - May 2022 (23 comments) Google is using AI to design chips that will accelerate AI - https://news.ycombinator.com/item?id=22717983 https://news.ycombinator.com/item?id=22717983 - March 2020 (1 comment)

lumb63 2y ago

I’ve not followed this story at all, and have no idea what is true or not, but generally when people use a boatload of adjectives which serve no purpose but to skew opinion, I assume they are not being honest. Using certain words to describe a situation does not make the situation what the author is saying, and if it is as they say, then the actual content should speak for itself. For instance: > Much of this unfounded skepticism is driven by a deeply flawed non-peer-reviewed publication by Cheng et al. that claimed to replicate our approach but failed to follow our methodology in major ways. In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), This could easily be written more succinctly, and with less bias, as: > Much of this skepticism is driven by a publication by Cheng et al. that claimed to replicate our approach but failed to follow our methodology in major ways. In particular the authors did no pre-training, Calling the skepticism unfounded or deeply flawed does not make it so, and pointing out that a particular publication is not peer reviewed does not make its contents false. The authors would be better served by maintaining a more neutral tone rather than coming off accusatory and heavily biased.

wholehog 2y ago
If someone tries to run your method but messes it up, and then accuses you of fraud when the results don't match their expectations, I'm not sure they're entitled to a neutral tone in response. Maybe you're right, and a more neutral tone would have been effective! I think it's just that Jeff is just really done with this.
dogleg77 2y ago
At least they are acknowledging skepticism now. That's a step in the right direction.

Keyframe 2y ago

At this point in time, why wouldn't we give at least benefit of the doubt to Jeff Dean immediately? His track record is second to none, and he's still going strong. Has something happened that cast a shadow on him? Sometimes it is the messenger that brings in the weight.

gabegobblegoldi 2y ago
Looks like he aligned himself with the wrong folks here. He is a system builder at heart but not an expert in chip design or EDA. And also not really an ML researcher. Some would say he got taken for a ride by a young charismatic grifter and is now in too deep to back out. His focus on this project didn’t help with his case at Google. They moved all the important stuff away from him and gave it to Demis last year and left him with an honorary title. Quite sad really for someone of his accomplishments.
- alsodumb 2y ago
  I mean Jeff Dean is probably more ML researcher than probably 90% of the ML researchers out there. Sure, he may not be working on state of the art stuff himself; but he's too up the chain to do that.
  - dogleg77 2y ago
    That's an appeal to authority, and not an effective one. Jeff Dean doesn't have a good track record in chip design.
    - wholehog 2y ago
      What are you even talking about? Jeff had a hand in TPU, which is so successful that all other AI companies are trying to clone this project and spin up their own efforts to make custom AI chips.
      - sijnapq 2y ago

7e 2y ago

Google's ultimate proof is to build better chips than the competition. Which, at this point, they cannot.

LittleTimothy 2y ago

"You didn't use enough compute in your reproduction of our methods" is kind of a funny criticism today. Well yeah, you're Google. Sorry guys, no reviewing of our methodology unless you own a cloud. It wouldn't surprise me if it's true that more compute, more pre-training etc. provides a lot of utility, but that does make it difficult to verify the work. One interesting aspect of this though is vice-versa, whilst Google has oodles of compute, Synopsys has oodles of data to train on (if, and this is a massive if, they can get away with training on customer IP).

dogleg77 2y ago
Chip designers run EDAtools on premises. How would EDA companies have access to customer data? Maybe for debugging purposes under NDA, but that won't allow training on customer IP.