4 ms·
So, I've been reading Google research papers for decades now and also worked there for a decade and wrote a few papers of my own. When google publishes papers,
by dekhn 2y ago
So, I've been reading Google research papers for decades now and also worked there for a decade and wrote a few papers of my own.
When google publishes papers, they tend to juice the results significance (google is not the only group that does this, but they are pretty egregious). You need to be skilled in the field of the paper to be able to pare away the exceptional claims. A really good example is https://spectrum.ieee.org/chip-design-controversy https://spectrum.ieee.org/chip-design-controversy while I think Google did some interesting work there and it's true they included some of the results in their chip designs, their comparison claims are definitely over-hyped and they did not react well when they got called out on it.
- tsumnia 2y agoRemember Google is a publicly traded company, so everything must be reviewed to "ensure shareholder value". Like dekhn said, its impressive, but marketing wants more than "impressive".
- dekhn 2y agoThis is true for public universities and private universities; you see the same thing happening in academic papers (and especially the university PR around the paper)
- hall0ween 2y agoI would say anecdotal. This hasn't been my case across four universities and ten years.
- BeetleB 2y agoThe actual papers don't overhype. But the university PR's regarding those papers? They can really overhype the results. And of course, the media then takes it up an extra order of magnitude.
- dekhn 2y agoI've definitely seen many examples of papers where the conclusions went far beyond what the actual results warranted. Scientists are incentivized to claim their discovery generalizes as much as possible. But yes, it's normally: "science paper says an experiment in mice shows promising results in cancer treatment" then "University PR says a new treatment for cancer is around the corner" and "Media says cure for all cancer"
- hall0ween 2y agoFair point!
- nyrikki 2y agoDepends on what you call "overhype". Wishful mnemonics in the field was called out by Drew McDermott in the mid 1970's and it is still a problem today. https://www.inf.ed.ac.uk/teaching/courses/irm/mcdermott.pdf https://www.inf.ed.ac.uk/teaching/courses/irm/mcdermott.pdf And: > As a field, I believe that we tend to suffer from what might be called serial silver bulletism, defined as follows: the tendency to believe in a silver bullet for AI, coupled with the belief that previous beliefs about silver bullets were hopelessly naive. (H. J. Levesque. On our best behaviour. Artificial Intelligence, 212:27–35, 2014.)
- ein0p 2y agoThat applies to absolutely everyone. Convenient results are highlighted, inconvenient are either not mentioned or de-emphasized. You do have to be well read in the field to see what the authors _aren't_ saying, that's one of the purposes of being well-read in the first place. That is also why 100% of science reporting is basically disinformation - journalists are not equipped with this level of nuanced understanding.
- dekhn 2y agoyes, but google has a long history of being egregious, with the additional detail that their work is often irreproducible for technical reasons (rather than being irreproducible for missing methods). For example, we published an excellent paper but nobody could reproduce it because at the time, nobody else had a million spare cores to run MD simulations of proteins.
- ein0p 2y agoIt's hardly Google's problem that nobody else has a million cores, wouldn't you agree? Should they not publish the result at all if it's using more than a handful of cores so that anyone in academia can reproduce it? That'd be rather limiting.
- dekhn 2y agoWell, a goal of most science is to be reproducible, and it couldn't be reproduced, merely for technical reasons (and so we shared as much data from the runs as possible so people could verify our results). This sort of thing comes up when CERN is the only place that can run an experiment and nobody can verify it.
- ein0p 2y agoIt is probably reproducible, if you have the requisite million cores. That isn't even difficult today - a million cores is about 100 GPUs.
- 2y ago
- warbaker 2y agoThe article you linked is not an example of this happening. Google open-sourced the chip design method, and uses it in production for TPU and other chips. https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training https://deepmind.google/discover/blog/how-alphachip-transformed-computer-chip-design/ https://deepmind.google/discover/blog/how-alphachip-transfor...
- dekhn 2y agoIt's an ongoing debacle with multiple people making extremely good arguments that Google overstated the results. Yes, I know it's in TPUs and I said exactly that. You simply can't take Google press at face value.
- warbaker 2y agoGoogle has responded to this "controversy" already: https://x.com/JeffDean/status/1858540085794451906 https://x.com/JeffDean/status/1858540085794451906 https://arxiv.org/abs/2411.10053 https://arxiv.org/abs/2411.10053
- dekhn 2y agoYes, I am aware. I didn't find Jeff's argument particularly convincing. Please note: I've worked personally with Jeff before and shared many a coffee with him. He's done great work and messed up a lot of things, too.
- confused_boner 2y agoFrom your perspective, which arguments were not convincing, are you able to share why not?
- Der_Einzige 2y agoUnironically "just trust me bro" is actually fine here. They're objectively right and you'll find they are when you do your painstaking analysis to figure it out.
- Bloating 2y agohttps://www.youtube.com/watch?v=shFUDPqVmTg https://www.youtube.com/watch?v=shFUDPqVmTg
- killjoywashere 2y agoI have worked with Google teams as well, and they taught me a fair bit about how to be rigorously skeptical. It takes domain knowledge, statistical knowledge, data, time and the computational resources to challenge them. I've done it, but it took real resources. That said, it's a useful exercise to figure out the plan of attack. My experience is the "juice" was mainly in "easy true negative" subclasses. They weren't oversampled, but the human brain wouldn't even consider most of that data. Once you ablate those subclasses from the dataset, (which takes a lot of additional labelling effort), you can start challenging their assertions. But it's hard. And that said I also review a number of articles in that domain, and I haven't seen a group with stronger datasets overall.