21 ms·
Why did Google Brain exist?
- dekhn 3y agobecause once Jeff Dean had solved Google's maslow problems (scaling web search, making ads profitable, developing high performance machine learning systems) he wanted to return to doing academic-style research, but with the benefit of Google's technical and monetary resources, and not part of X, which never produces anything of long-term value. I know for sure he wanted to make an impact in medical AI and felt that being part of a research org would make that easier/more possible than if he was on a product team.
- marricks 3y ago[flagged]
- mupuff1234 3y agoOr maybe it's just not perceived as controversial? Her boss told her to do something, she refused and got sacked.
- qumpis 3y agoWiki doesn't seem to give detail into the situation, nor the paper in question
- dekhn 3y agoThat's a very simplified version of the story, but I would say that Dean greatly reduced his stature when he defended Megan Kacholia for her abrupt termination of Timnit. Note that Timnit was verbally abusive to Jeff's reports, anybody who worked there could see what she was posting to internal group discussions, so her time at Google was limited, but most managers would say that she should had at least been put on a pip and given 6 months. Dean has since cited the paper in a subsequent paper (which tears apart the Stochastic Parrots paper).
- marricks 3y agoGoogle has since fired other folks on her team and was in crisis mode to protect Dean. Like, I’m not really going to give them the benefit of the doubt on this. When people brought Dean up Timnit came up as something to consider, it’s interesting to see how all anyone has to say in these threads is reverence towards him. People should try to see the whole picture.
- opportune 3y agoBeing somewhat involved in one bad thing doesn’t justify cancelling someone. To my knowledge Dean was essentially doing post-hoc damage control for what one of the middle managers in his org did. Even if they did want Timnit gone (as others mention, you are getting only one side of the story in media) they did it in a bad way, for sure. At the same time I don’t think one botched firing diminishes decades of achievements from a legitimately kind person.
- jeffbee 3y agoTimnit and the other ex-ML ethics crowd who got fired from Google seem like some of the most ignorant people around. I don't defend Dean reflexively, it just seems like he is on the right side of the issue. For example, here is Emma Strubell accusing Dean of creating a "toxic workplace culture" after he and David Patterson had to refute her paper in print. https://twitter.com/strubell/status/1634164078691098625?lang=en https://twitter.com/strubell/status/1634164078691098625?lang... The thing is if David Patterson and Jeff Dean think your numbers for the energy cost of machine learning might be wrong, then you are probably wrong. These ML meta-researchers are not practitioners and appear to have no idea what they are talking about. Keeping a person like Timnit or Strubell on staff seems like it costs more than its worth.
- dragonwriter 3y ago> Timnit and the other ex-ML ethics crowd Timnit is ex-Google, but very much not ex-ML ethics (fouded Distributed AI Research Institute focussed on the field in late 2021). Very much also true of Margaret Mitchell, who has been at Hugging Face since 2021.
- Traubenfuchs 3y agoA lot of people, especially on hacker news, feel disdain for researchers of ethics, bias and fairness, as they are perceived as both holding technology back and profiting from advances in it (that they can then analyse and criticize).
- calrissian 3y agoI don't think you're necessarily wrong in your assesment about HN and AI enthusiasts, but also in this case I think it's more accurate to talk about a Twitter agitator and race-baiter [1], rather than a "researcher of ethics, bias and fairness". [1] https://twitter.com/timnitGebru/status/1651055495271137280?s=20 https://twitter.com/timnitGebru/status/1651055495271137280?s...
- renewiltord 3y agoThese safety people guarantee a useless product that never does unsafe things. ChatGPT proved that you can have a product do unsafe things and still be useful if you put a disclaimer on it. Overall, as a user, I couldn't give a damn if things are unsafe by the definition of this style of ethicist. They were a ZIRP and my life is better for their absence.
- Silverback_VII 3y agoShe appears to be a symbol for everything that went wrong at Google. These are the kind of problems that arise when life is too easy, just before the downfall. In other words, decadence. How else can one explain that Google's AI research was dethroned by OpenAI?
- hintymad 3y agoYeah, Dean's fault is hiring such people in the first place. If you hire an activist, you get activism. And if you hire someone whose livelihood depends on finding more problems, well, they will scream more problems, one way or another. Otherwise, why would state U of Mich got one DEI officer per three staff members?
- calrissian 3y ago> Also not surprised at the immediate down votes for questioning Googles new AI lead! That's because you are wrong to pretend he did anything wrong by firing T.G. And also, because you added this weird lie/mudslinging/whatever on top of it: > while she was on vacation
- hintymad 3y agoControvery of what? Did you read Gebru's paper? For instance, her calculation of carbon footprint of training BERT assumes that companies will train BERT 24x7. Gebru is a disgrace to the community because she always, I mean literally always, attacks her critics by motives. You think bias is a data problem? You're a bigot (See her dispute with LeCun). You disagree with my assessment on an ML model? You are white male oppressor (her attacking a Google's SVP). Gebru is not a researcher. She is a modern-age Trofim Lysenko, who politicizes everything and weaponizes political correctness. And yeah, she deserves to be fired. Many times.
- marricks 3y agoCrazy that her boss and people she mentored all loved her. It’s almost like the system screwed her and she’s rightly angry at it. I get it, easier to type cast her as an angry black woman though!
- erenyeager 3y agoOk but the lack of underrepresented minorities in the field and the important role people like Gebru played in extending the political and status of minorities is ok to extinguish? We need more than just white male / Chinese male / Indian male monoculture “STEM lords”. This is already recognized in fields like medicine, where minorities treating minorities results in better outcomes and the greater push to open positions of status to minorities.
- qumpis 3y agoYes, uplifting minorities is great, but anyone should be accountable equally when it comes to workplace conduct
- logicchains 3y ago>Ok but the lack of underrepresented minorities in the field and the important role people like Gebru played in extending the political and status of minorities is ok to extinguish? Yes it's okay to extinguish it if hiring underrepresented minorities means hiring bad actors like her who contribute nothing of value. Scientific truth is scientific truth; if you hire people for the color of their skin or their sexuality instead of their ability to produce truth, you slow the progress of science and make the world worse for everyone.
- choppaface 3y agoI agree that the OP makes a bunch of interesting points, but I think historically Brain really grew out of what Dean wanted to do and the fact that he wanted it to be full-stack, e.g. including the TPU. Also, crucially, Brain would use Google data and contribute back to Ads/Search directly versus Google X which was supposed to be more of an incubator. But it's also notable how the perspective of an ex-Brain employee might differ from what sparked the Brain founders in the first place.
- leoh 3y agoWaymo?
- dekhn 3y agowaymo hasn't produced anything of long-term value yet. And everything about it that worked well wasn't due to Google X
- ra7 3y agoCan you expand why being part of Google X hinders a team? I believe Waymo "graduated" from Google X to its own entity.
- dekhn 3y agoX exists as a press-release generation system, not as a real technology creation system. They onboard many impractical projects that are either copies of something being done already in industry ("but with more Google and ML!") or doesn't have a market (space elevators).
- alphabetting 3y agoWaymo has developed the modern autonomous vehicle from the ground up. It's basically a matter of scale now. It's a mindblowing tech stack. The first time riding in one is much more otherwordly than using GPT for the first time. The value of the technology is far greater whatever PR they have generated (not many people know about it)
- dekhn 3y agoI have infinite respect for the process that Waymo followed to get to where they are. And I'm impressed that Google continued to fund the project and move it forward even when it represents such a long-term bet. but it's not a product that has any real revenue. and most car companies keep their distance from google self-driving tech because they're afraid. afraid google wants to put them out of business. It's unclear if google could ever sell (as a product, as an IP package, etc) what they've created because it depends so deeply on a collection of technology google makes available to waymo.
- dgacmu 3y agoI generally agree with this though with some tweaks. I think Jeff wanted to do something that he thought was both awesome (he's liked neural networks for a long time - his undergrad thesis was on them) and likely to have long-term major impact for Google, and he was able to justify the Awesome Thing by identifying a way for it to have significant potential revenue impact for Google via improvements to ad revenue, as well as significant potential "unknown huge transformative possibilities" benefits. But I do suspect that you're right that the heart of it was "Jeff was really passionate about this thing". Of course, this starts to get at different versions of the question: Why did Google Brain exist in 2012 as a scrappy team of builders, and why did Brain exist in 2019 as a mega-powerhouse of AI research? I think you and I are talking about the former question, and TFA may be focusing more on the second part of the question. [I was randomly affiliated with Brain from 2015-2019 but wasn't there in the early days]
- dekhn 3y agoIt grew from the scrappy group to the mega-powerhouse by combining a number of things: being the right place at the right time with the right resources and people. They had a great cachet- I was working hard to join Brain in 2012 because it seemed like they were one of the few groups who had access to the necessary CPU and data sets and mental approaches that would transform machine learning. And at that time, they used that cachet to hire a bunch of up and coming researchers (many of them Hinton's students or in his sphere) and wrote up some pretty great papers. Many people outside of Brain were researchers working on boring other projects who transferred in, bringing their internal experience in software development and deployment, which helped a lot on the infra side.
- light_hue_1 3y ago> The next obvious reason for Google to invest in pure research is for the breakthrough discoveries it has yielded and can continue to yield. As a rudimentary brag sheet, Brain gave Google TensorFlow, TPUs, significantly improved Translate, JAX, and Transformers. Except that these advances have made other companies an existential threat for Google. 2 years ago it was hard to imagine what could topple Google. Now a lot of people can see a clear path: large language models. From a business perspective it's astounding what a massive failure Google Brain has been. Basically nothing has spun out of it to benefit Google. And yet at the same time, so much has leaked out, and so many people have left with that knowledge Google paid for, that Google might go the way of Yahoo in 10 years. This is the simpler explanation of the Brain-Deep Mind merger: both Brain and Deep Mind have fundamentally failed as businesses.
- deleted 3y ago[deleted]
- cma 3y ago> And yet at the same time, so much has leaked out, and so many people have left with that knowledge Google paid for, that Google might go the way of Yahoo in 10 years. Google couldn't have hired the talent they did without allowing them to publish.
- dekhn 3y agoGoogle never talked much about it externally, but Google Research (the predecessor to Brain) had a single project which almost entirely funded the entire division- a growth-oriented machine learnign system called Sibyl. What was sibyl used for? Growing youtube and google play and other products by making the more addictive. Sibyl wasn't a very good system (I've never seen a product that had more technical debt) but it did basically "pay for" all of the research for a while.
- temac 3y agoSeems to be quite evil though.
- vl 3y ago>PyTorch/Nvidia GPUs easily overtaking TensorFlow/Google TPUs. TF lost to PyTorch, and this is Google’s fault - TF APIs are both insane and badly documented. But nothing comes close to performance of Google’s TPU exaflop mega-clusters. Nvidia is not even in the same ballpark.
- jdlyga 3y agoI can speak from experience on this. Getting started with TensorFlow was very complicated with sparse documentation, so we dropped the idea of using it.
- vl 3y agoI had to use TF when I worked at G, when I left I immediately started to use PyTorch and never looked back.
- belval 3y agoThere is a first mover handicap there though. TF1.0 included a bunch of things that were harder to understand like tf.Session(). PyTorch was inspired from the good parts and "we will eager-everything". Internally I'm sure there was a lot of debate in the TF team that culminated with TF2.0, but by that time the damage was done and people saw PyTorch as easier.
- dr_dshiv 3y agoLots of great insight. Here’s one: “Given the long timelines of a PhD program, the vast majority of early ML researchers were self-taught crossovers from other fields. This created the conditions for excellent interdisciplinary work to happen. This transitional anomaly is unfortunately mistaken by most people to be an inherent property of machine learning to upturn existing fields. It is not. Today, the vast majority of new ML researcher hires are freshly minted PhDs, who have only ever studied problems from the ML point of view. I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML.”
- michaelrpeskin 3y agoobligatory XKCD: https://xkcd.com/793/ https://xkcd.com/793/
- fknorangesite 3y agoSimilarly: https://www.smbc-comics.com/comic/2012-03-21 https://www.smbc-comics.com/comic/2012-03-21
- javajosh 3y agoThat's a great xkcd, but there are 2 upsides to this arrogant approach. First, arrogance is a nerd-snipe maximizer. Second, there is a small chance you're absolutely right, and you've just obviated a whole field from first principles. It doesn't happen often, but when it does happen and there is no clout like "emporer's new clothes" clout. EDIT: The downside, of course, is that you appear arrogant, and people won't like you. This can hurt your reputation because it is apparently anti-social behavior on several levels. I think its fair to call it a little bit of an intellectual punk rock move that is probably better left to the young. It's an interesting emotional anchor to mapping a new field, though.
- jojosbuddy 3y agoNot laughing! /s (physicist here) Actually most applied physicists like myself go down that path cause we're pretty efficient, lazy folk & skip through as fast as possible--I call it the principle of maximum laziness.
- zgin4679 3y agoIt thinks, therefore it did.
- rossdavidh 3y agoSo if it doesn't exist now, that means it didn't think?
- deleted 3y ago[deleted]
- DavidSJ 3y ago> … the publication of key research like LSTMs in 2014 … Minor nitpick, but LSTMs date to 1997 and were not invented by Google. [1] [1] Hochreiter and Schmidhuber (1997). Long short-term memory. https://ieeexplore.ieee.org/abstract/document/6795963 https://ieeexplore.ieee.org/abstract/document/6795963
- Scea91 3y agoSeems more than a nitpick to me. I find the essay interesting but this line raised some distrust in me. How can someone have these deep insights into Google's ML strategy and the evolution of the field and simultaneously think LSTMs were invented by Google in 2014?
- brilee 3y agosorry, I had a mind fart. I was thinking of seq2seq https://research.google/pubs/pub43155/ https://research.google/pubs/pub43155/ Pushing the fix now...
- logicchains 3y ago>How can someone have these deep insights into Google's ML strategy and the evolution of the field and simultaneously think LSTMs were invented by Google in 2014? It may not have been accidental; there's a deliberate movement among some people in the ML community to deny Jürgen Schmidhuber credit for inventing LSTMs and GANs.
- 3y ago
- amoss 3y ago> Today, thought leaders casually opine on how and where ML will be useful, and MBAs feel like this is an acceptable substitute for expert opinion. Sounds like standard operating procedure.
- kps 3y agoSounds like something LLMs would actually be good for. They're not getting us fusion power or cancer cures.
- asdfman123 3y agoMy theory is that broadly, tech learned not to act like Microsoft in the 90s -- closed off, anti-competitive, unpopular -- but swung too far in the opposite direction. Google has been basically giving away technology for free, which was easy because of all the easy money. It's good for reputation and attracting the best talent. That is, until a competitor starts to threaten to overtake you with the technology you gave them (ChatGPT based on LLM research, Edge based on Chromium, etc.).
- potatolicious 3y agoEhh, I mildly disagree. I'm not entirely bought-in on the notion that giving one's technical innovations for free is obviously the right move, but I don't think it's why the company is in trouble. Chrome is experiencing unprecedented competition because it faltered on the product. Chrome went from synonymous with fast-and-snappy to synonymous with slow-and-bloated. Likewise Google invented transformers - but the sin isn't giving it away, it's failing to exercise the technology itself in a compelling way. At any moment in time Google could have released ChatGPT (or some variation thereof), but they didn't. I've made this point before - but Google's problems have little to do with how it's pursuing fundamental research, but everything to do with how it pursues its products. The failure to apply fundamental innovations that happened within its own halls is organizational.
- asdfman123 3y agoWell, Google could have easily not have shared its technology. However, the bloat problem you’ve described are difficult problems to solve, and are to some degree endemic to large businesses with established products.
- potatolicious 3y ago> "Well, Google could have easily not have shared its technology." Sure, but the idea is that if they didn't share their technology, they'd still be in the same spot: they would have invented transformers and still not shipped major products around it. Sure maybe OpenAI won't exist, but competitors will find other ways to compete. They always do. So at best they are very very slightly better off than the alternative, but being secretive IMO wouldn't have been a major change to their market position. Meanwhile, if Google was better at productizing its research, it matters relatively little what they give away. They would be first to market with best-in-class products, the fact that there would be a litany of clones would be a minor annoyance at best.
- vrglvrglvrgl 3y ago[dead]
- rib3ye 3y agoOrganized, concise, and not wordy. Props to the writer, he shows a deep degree of written communication skills on a topic frequently cluttered with jargon.
- khazhoux 3y ago> I sat on it because I wasn’t sure of the optics of posting such an essay while employed by Google Brain. But then Google made my decision easier by laying me off in January. My severance check cleared... I'm really baffled by how people think it's OK to write public accounts of their previous (and sometime current!) employers' inner workings. This guy got paid a shitload of money to do work and to keep all internal details private, even after he leaves. They could not be more clear about this when you join the company. Why do people think it's OK to share like this? This isn't a whistleblowing situation -- he's just going for internet brownie points. It's just an attempt to squeeze a bit more personal benefit out of your (now-ended) employment. Contractual/legal issues aside, I think this kind of post shows a lack of personal integrity (because he did sign a paper agreeing not to disclose info), and even a betrayal of former teammates who now have to deal with the fallout.
- q845712 3y agoI read the article and thought he did a fine job of not spilling too many secrets - I'm curious what you thought he said that crossed the line? I'm not personally aware of signing something that says "I'll keep all internal details private" though I agree I'd be highly unlikely to refer to anyone below the SVP level by name -- but I think that's exactly what OP did?
- khazhoux 3y agoTrue, this wasn't the most egregious. But the principle still applies. He said himself he held this back until his final check cleared.
- rurp 3y agoA large company acting out spite towards a former employee has happened more than a couple times. Minimizing that risk seems entirely reasonable. As others have said, I really don't see anything that's especially private in the article. The author wrote in pretty general terms.
- TaylorAlexander 3y ago
- htrp 3y agoTo prevent talented people from developing tech elsewhere.
- dbish 3y agoMSR seemed like it had a similar underlying purpose (or at least nice side effect).
- choppaface 3y ago> Neither side “won” this merger. I think both Brain and DeepMind lose. I expect to see many project cancellations, project mergers, and reallocations of headcount over the next few months, as well as attrition. This merger will be a big test for Sundar, who has openly admitted years ago to there being major trust issues [1]. Can Sundar maintain the perspective of being the alpha company while bleeding a ton of talent that doesn't actively contribute to tech dominance? Or will he piss off the wrong people internally? It's OK to have a Google Plus / Stadia failure if the team really wanted to do the project. If the team does _not_ want to work together though, and they fail, then Sundar's request that the orgs work together to save the company is going to get totally ignored in the finger-pointing. [1] https://www.axios.com/2019/10/26/google-trust-employee-immigration https://www.axios.com/2019/10/26/google-trust-employee-immig... .
- ergocoder 3y agoThe merger will fail. If 5000 people are not enough to do things, 10000 people will unlikely change that.
- jayzalowitz 3y agonitpick Cockroach was built by the team that built spanner. So your phrasing is off there.
- deleted 3y ago[deleted]
- nologic01 3y ago> it is becoming increasingly apparent to Google that it does not know how to capture that value To paraphrase, its the business model, stupid. Inventing algorithms, building powerful tools and infrastructure etc is actually a tractable problem: you can throw money and brains at it (and the latter typically follows the former). While the richness of research fields is not predictable, you can bet that the general project of employing silicon to work with information will keep bearing fruits for a long time. So creating that value is not the problem. The problem with capitalizing (literally) on that intellectual output is that it can only be done 1) within a given business model that can channel effectively it or 2) through the invention of totally new business models. 1) is a challenge: These billions of users on which AI goodies can surface are not customers, they are product. They don't pay for anything and they don't create any virtuous circle of requirements and solutions. Alas, option 2) inventing major new business models is highly non-trivial. The track record is poor: the only major alternative business model to adtech (cloud unit) was not invented there anyway and in any case selling sophisticated IT services whether to consumers or enterprise is a can of worms that others have much more experience in. For a industrial research unit to thrive, its output must be congruent with what the organization is doing. Not necessarily in the detail, but definitely in the big picture.
- simonster 3y agoI work for Google Brain. I remember meeting Brian at a conference and I have nothing but good things to say about him. That said, I think Brian is underestimating the extent to which the Brain/DeepMind merger is happening because it's what researchers want. Many of us have a strong sense that the future of ML involves models built by large teams in industry environments. My impression is that the goal of the merger is to create a better, more coordinated environment for that kind of research.
- gowld 3y agoThe goal of the merger is for execs to look like they are doing something to drive progress. Actual progress comes from the researchers and developers.
- anonylizard 3y agoWell, where exactly is this progress? Where is Google's answer to GPT-4? Why weren't the 'researchers and developers' making a GPT-4 equivalent? Turns out you sometimes you need a top down, centralised vision to execute on projects. When the goal is undefined, you can allow researchers to run free and explore, now its full on wartime, with clear goals (make GPT-5,6,7....).
- oofbey 3y agoGoogle is fundamentally allergic to top-down management. Most googlers will reject any attempt to be told what to do as wrong, because lots of IC's voting with their feet are smarter than any (google) exec at figuring out what to do. Last time Google got spooked by a competitor was Facebook, and they built Google Plus in response. We all know that was an utter failure. Googlers could escape that one with their egos in tact because winning in "social" is just some UX junk, not hard-core engineering like ML. It's gonna be super hard for them to come to grips with the fact that they are way behind on something that they should be good at. Plan for lots of cognitive dissonance ahead.
- 3y ago
- liuyipei 3y agoGoogle has good engineers and a long history of high throughput computing. This, combined with a lack of understanding what ML research is like (versus deployment), led to the original TF1 API. Also, the fact that google has good engineers working in a big bureaucracy probably hid a lot of the design problems as well. TF2 was a total failure, in that TF1 can do a few things really well when you get the hang of it, but TF2 was just a strictly inferior version of pytorch, further plagued by confusion due to TF1. In alternate history, if Google pivoted in to JAX much earlier and more aggressively, they could still be in the game. I speak as someone who has at some point knew all the intricacies and differences between TF1 and TF2.
- activitypea 3y ago> google has good engineers working in a big bureaucracy probably hid a lot of the design problems as well. I feel like this is true of every google product in the last decade, maybe more. Their customer products, but especially their dev tools like Angular and k8s screams "We were so preoccupied with whether we could, we didn’t stop to think if we should."
- antipaul 3y agoWhy does Google X exist?
- ironman1478 3y ago"I’ve seen repeatedly that it’s much harder for a ML PhD to learn chemistry than for a chemist to learn ML. (This may be survivorship bias; the only chemists I encounter are those that have successfully learned ML, whereas I see ML researchers attempt and fail to learn chemistry all the time.)" This is something that rings really true to me. I work in imaging and it's just very clear that there are groups of people in ML that don't want to learn how things actually work and just want to throw a model at it (this is a generalization obviously, but it's more often than not the case). It only gets you 80% there, which is fine usually, but not fine when the details are make or break for a company. Unfortunately that last 20% requires understanding of the domain and people just don't like digging into a topic to actually understanding things.
- drakenot 3y agoThis seems to kind of be the opposite opinion of The Bitter Lesson[0]. [0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- jpace121 3y agoI don’t think it’s really the opposite opinion. To pick on the game of Go as an example, the insight from the Bitter Lesson is the best method to use for Go is probably going to be the most broad method that abuses the magic of brute force the most, compared to a method that really encodes the best existing strategies of Go. I think what OP is referring to and what I’ve observed is that in some fields you have to have a certain amount of expertise to just understand the rules of the game, and there are a lot of applications where someone approaching a problem with the goal of applying ML can’t quite get over the hump of understanding all the rules.
- sidibe 3y agoSomeone still needs to have enough domain knowledge to figure out what needs solving and how to validate that it was solved. Being able to frame problems, and being able to cobble together the data and infra for the AI to solve them are the most valuable things in the medium term.
- jaimex2 3y agoThis is great. While Google is busy imploding the next generation of startups can flourish. I'm being hopeful that they decimate a lot of big tech and they don't just all get bought out. Diversity might return to the Internet. Wishful thinking, I know.
- Nevermark 3y agoHmm. Anything that slows Google down and maintains a diversity of leaders in the field is ok with me. Imagine a host of "helpful" Google AI's, Facebook AI's, Amazon AI's, etc., that know their very existence depends them monetizing you more effectively than competitive AI's. Of course, the first versions will be very helpful. But continuous efforts to remain "the most helpful" will cost a lot, and eventually need to pay for themselves.
- pbj1968 3y agoIt was an interesting read, but he also links to that misinformed article about indirect costs.