9 ms·
The Google employees who created transformers
- Solvency 3y ago[flagged]
- elromulous 3y agoPuff piece or not, give credit where credit is due. Google fostered the environment that enabled that paper.
- ctrw 3y agoTransformers is modern AI the same way that a cherry is an ice-cream.
- deleted 3y ago[deleted]
- rvnx 3y agoOr the other way around, perhaps if it stayed within Google it would have stayed in the dark and take xx years to discover again
- Filligree 3y agoOld Google did. I miss that place.
- braza 3y agoRegardless of the publication intentions, I like to see contemporary folks who have made an impact on the world through technology being recognized and elevated. In economics, where I come from, people are not afraid to give accolades to relevant people, whether treasury secretaries, banking economists, or professors. I feel that in tech, due to the lack of publications that have technical capabilities to understand that impact plus the lack of goodwill from the media in general, there are a lot of folks doing relevant work out there that is ostracised due to this "cult of tech CEO/entrepreneur" or intentionally under the radar to not bring any negative light in their lives.
- tpm 3y agoBut they don't work at Google anymore, so what's in it for Google?
- slumpt_ 3y agoI don’t think it reads that way either. Inclined to think they didn’t read it.
- seneca 3y agoWhat's in it is positive association between AI and Google. Google is sort of a seen as a joke in the latest wave of consumer facing AI tools, and they desperately want to change that.
- CharlesW 3y agoI'm not sure why you're being downvoted, but you're right — a story suggesting that 'modern' AI originated at Google is precisely the kind of narrative that Google's PR firm would shop around.
- alephxyz 3y agoThe article starts off by saying they all left Google. There's also an entire section narrating how google execs passed on transformers and didn't pick up until OpenAI released GPT. >The picture internally is more complicated. “It was pretty evident to us that transformers could do really magical things,” says Uszkoreit. “Now, you may ask the question, why wasn’t there ChatGPT by Google back in 2018? Realistically, we could have had GPT-3 or even 3.5 probably in 2019, maybe 2020. The big question isn’t, did they see it? The question is, why didn’t we do anything with the fact that we had seen it? The answer is tricky.” >Many tech critics point to Google’s transition from an innovation-centered playground to a bottom-line-focused bureaucracy. As Gomez told the Financial Times, “They weren’t modernizing. They weren’t adopting this tech.”
- HarHarVeryFunny 3y ago> There's also an entire section narrating how google execs passed on transformers and didn't pick up until OpenAI released GPT. Google certainly seem to have fumbled on rapidly exploiting the tech, but I'm not sure that's totally fair. Google built BERT in 2018, around the same time as OpenAI built GPT-1, but neither were released. OpenAI's GPT-2 was built in february 2019 but OpenAI were hesitant to release it due to potential danger, only doing so later in november 2019. GPT-3 followed in july 2020, but ChatGPT which the first accessible to public view of the tech didn't appear until november 2022 (how time flies!). Google's first transformer-based chatbot, LaMDA, was released in may 2021, well over a year before ChatGPT, but was then rapidly withdrawn due to negative feedback and behavior. The difference between ChatGPT's acceptable behavior, and success, compared to LaMBDA, was OpenAI's invention of RLHF to better control it. One could argue that Google should have invented something similar themselves earlier, but surely this might have been expected to come from these same people who had been working on language models at Google ...
- CharlesW 3y agohttps://archive.is/gT5Ot https://archive.is/gT5Ot
- DueDilligence 3y ago[dead]
- janpmz 3y ago[flagged]
- pugio 3y agoAs someone who periodically listens to papers via text-to-speech, I was intrigued when I saw this link. I'm a little uncertain about the "AI simplification" - If I'm reading a paper I don't want to have the second guess everything I hear as to whether it completely misinterpreted, elided, or hallucinated a point. Do you have any personal experience with the results?
- dkural 3y ago“I took a speed-reading course and read War and Peace in twenty minutes. It involves Russia.” ― Woody Allen
- reidjs 3y agoIf the text fits within the context window most models are pretty reliable at summarizing texts effectively.
- nicklecompte 3y agoThey’re good at summarizing text convincingly if you didn’t read the original document, but GPT-4 still makes a ton of basic factual mistakes: https://arxiv.org/abs/2402.13249 https://arxiv.org/abs/2402.13249 I think it will be several decades - maybe longer - before you can throw an arbitrary 20-page document into a computer program and get an accurate one-page summary. People need to take seriously that transformer LLMs don’t understand human language.
- whatever1 3y agoI am all in for allowing tax free bloated RnD departments just with the hope that once in a decade idea will propell the overall economy. The marvel of modern computing is a result of RnD bloat that was done without immediate impact to the bottomlines of their own companies.
- deleted 3y ago[deleted]
- utopcell 3y agoTax-free ? This was a regular Google team.
- bozhark 3y agoMaybe they meant revenue free?
- orev 3y agoIn the US, R&D expenses can often be deducted when filing taxes. I’m sure large companies like Google would be taking advantage of those rules.
- ant6n 3y agoCan't all expenses be deducted when filing taxes? I mean, taxes are paid on profit, which is revenue minus expenses.
- mikeyouse 3y agoYeah the person you are replying to was imprecise - there's actually tax credits for R&D expense in the US rather than just tax deductions. https://www.plantemoran.com/explore-our-thinking/insight/2020/08/is-your-business-getting-the-rd-tax-credit-it-deserves https://www.plantemoran.com/explore-our-thinking/insight/202...
- 3y ago
- saadatq 3y agoOf note: “ Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively.”
- deleted 3y ago[deleted]
- IncreasePosts 3y agoWhy would that be of note, especially in America? I think it would be an interesting observation in China or Japan, or some other country which is generally less welcoming to immigrants than the US
- svnt 3y agoFirst generation immigrants are still a tiny minority of the population. The fact that the entire team consists effectively of first generation immigrants says something, probably both about higher education and American culture.
- bugglebeetle 3y ago> First generation immigrants are still a tiny minority of the population Not in California. Last I remember, something like a quarter of the state’s population is foreign born.
- svnt 3y agoIt looks like you are roughly right, but still, a sampling of 8 students from this population is not likely to come up that way (by my calculation 1.4x10^-7) > All Students in Higher Education in California > 2,737,000 > First-Generation Immigrant Students > 387,000 14% are first-generation immigrants from: https://www.higheredimmigrationportal.org/state/california/ https://www.higheredimmigrationportal.org/state/california/
- 3y ago
- whywhywhywhy 3y agoLooking forward to the inevitable movie about how they invented it and the company failed to productize it because of it's culture and management
- CharlesW 3y agoXerox 2: Electric Boogoogle
- deleted 3y ago[deleted]
- galoisscobi 3y agoSame thing is happening with the Figure One robot. There isn't much new in the demo Figure did recently, all the work has been published in the palm saycan paper and other deepmind papers. Google is good at laying golden eggs and not realizing that they've laid them. Then employees get fed up, leave and start their own companies so they can productize their work without management getting in the way.
- UberFly 3y agoThis is a story that could be written over and over again. Companies with big R&D budgets but not nimble enough any more to take advantage.
- photochemsyn 3y ago> "Realistically, we could have had GPT-3 or even 3.5 probably in 2019, maybe 2020. The big question isn’t, did they see it? The question is, why didn’t we do anything with the fact that we had seen it? The answer is tricky.” The answer is that monopolies stifle technological innovation because one well-established part of their business (advertising-centric search) would be negatively impacted by an upstart branch (chatbots) that would cut into search ad revenue. This is comparable to a investor-owned consortium of electric utilities, gas-fired power plants, and fracked natural gas producers. Would they want the electric utility component to install thousands of solar panels and cut off the revenue from natural gas sales to the utility? Of course not. It's a good argument for giving Alphabet the Ma Bell anti-trust treatment, certainly.
- tyree731 3y agoOn the other hand, Alphabet's inability to deploy GPT-3 or GPT-3.5 has led to the possibility of its disruption, so anti-trust treatment may not be necessary.
- IX-103 3y agoDisrupted by a whom? Microsoft? Facebook? The company formerly known as Twitter? Even if one of these takes over we'd just be trading masters. And that's ignoring how Alphabet's core business, Search, has little to fear from GPT-3 or GPT-3.5. These models are decent for a chatbot, but for anything where you want reliably correct answers they are lacking.
- lelag 3y agoA better example of that behaviour would be Kodak which invented the first digital camera as soon as 1975 then killed the project, because it was a threat to their chemical business.
- jl6 3y agoDigital photography works not just because of the camera but because of the surrounding digital ecosystem. What would people do with digital photos in 1975?
- bigcat12345678 3y agoTransformer == Modern AI? Wow, I dont know where should I start My mind literally stuck at the absurdity
- ozymandias1337 3y agoAI is in the eye of the beholder.
- deleted 3y ago[deleted]
- supportengineer 3y agoAnd in this case, the transformers are more than meets the eye.
- selimthegrim 3y agoI’m sure there’s a beam search joke in here somewhere but it’s too late in the day for me to think about it.
- itsoktocry 3y agoIt's 2024; do we really have to point out the fact that publications take liberties with headlines? It's been that way forever. Besides that, are you denying that transformers are the fundamental piece behind all the AI hype today? That's the point of the article. I think the fact that a mainstream publication is writing about the Transformers paper is awesome.
- YeGoblynQueenne 3y agoOK, so the title should be changed to "8 Google Employees Invented Modern AI Hype"?
- bigcat12345678 3y ago
- mkl95 3y agoI thought level 4+ self-driving was going to be the face of "modern AI"... instead we got this thing
- lelag 3y agoWhy do you think transformers won't be key in making a level 4+ self-driving AI. It seems to me that Vision-capable multi-modal transformers could be the missing part: they can understand what is happening in the world in a deductive way. The Vision transformer is capable of predicting that a running child is likely to follow that rolling soccer ball. It is capable of deducting that a particular situation looks dangerous or unusual and it should slow down, or stay away from danger in ways that previous crop of AI could not. Imo, the only thing currently preventing transformers to change everything is the large amount of compute power required to run them. It's not currently possible to imagine GPT4-V running on a embedded computer inside a car. Maybe AI asic type of chip will solve that issue, maybe Edge computing and 5G will find it's use-case... Let's wait and see, but I would bet that transformer will find it's way in many places and change the world in many more ways than bringing us chatbots.
- KerrAvon 3y agoBandwidth alone isn’t what prevents 5G from this sort of application, at least in the USA. Coverage maps tell the story: coverage is generally spotty away from major roads. Cell coverage isn’t a fixable problem in the near term, because every solution for doing that intersects with negative political externalities (antivax, NIMBYism, etc); if you can get people vaccinated for measles consistently again, then we can talk. It all needs to be onboard. That’s where money should be going.
- koala_man 3y agoIn Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"
- frame_ranger 3y ago[flagged]
- elevatedastalt 3y agoNice story, but Google's heyday was probably 10 years before that. By 2014 the decline had already started.
- nostrademons 3y agoA little after that. I'd put the heyday between 2003-2010, starting with the GMail launch and ending with the Chrome & Android launches. That period includes GMail, Maps, Scholar, Orkut, Reader, the acquisitions of Blogger/YouTube/Docs/Sheets/Slides/Analytics/Android, Summer of Code, OAuth, Translate, Voice Search, search suggestions, universal search, Picasa, etc. Basically I can look at my phone or computer and basically everything I routinely use dates from that period.
- mistrial9 3y agoGMail, the service that tells every user that they will be indexed and profiled personally for ad leads? that is an achievement ? GMail was one of the red flags for many that "don't be evil" was not going to be what it appeared. History says that this kind of mass profiling never ends well.
- pests 3y agoGmail doesn't show personalized ads for some years FWIW.
- OldMatey 3y ago
- ralph84 3y agoAnd none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.
- hotstickyballs 3y agoJust like Xerox and Bell. Innovation isn’t necessarily their most important job, at least when innovations are being made.
- nemo44x 3y agoHe was too distracted with all the other things going on under his watch. He spent too much time trying to convince rank and file that it's OK to do business with customers they don't like, trying to ensure arbitrary hiring goals were made, and appeasing the internal activists that led protests and pressured to exit people that didn't fall in line. This is why the CEO of Coinbase sent the memo a few years back stating that these types of people (not mission focused, self absorbed, and distracting) should leave the company. The CEO of Google should be fired.
- tdb7893 3y agoHow has he been appeasing internal activists? He's been pretty clear about firing them Edit: I'm also unclear on what makes you think he's spending much time on "arbitrary hiring goals". I remember an initiative for hiring in a wider variety, also conveniently much cheaper, locations but there wasn't indication he was personally spending time on it
- hrmyeah 3y ago[dead]
- calf 3y agoIt is intellectually dishonest to call the military-industrial complex merely "customers that some Google employees didn't like". Humans are more than their companies' mission, and your freedom to exercise your political views on how companies should be run inherently relies on this principle, too. So your argument is fundamentally a hypocrites projection.
- Thaxll 3y agoIt's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.
- dpflan 3y agoWhat is OpenAI today? Can you elaborate? Google is a varied $trillion company. OpenAI sells access to large generative models.
- bitcharmer 3y agoSure, Google is worth more but they essentially reduced themselves to an ad business. They aren't focused on innovation as much as before. Shareholder value is the new god.
- dpflan 3y agoAgreed that Friedman cursed the market with "shareholder theory", and it is a totally subversive and corrupting concept because whatever will increase value is accepted because it's the doctrine.
- glial 3y agoEconomic "theories" have an annoying way of being self-fulfilling.
- ripe 3y agoAn ad business is all Google ever was. But they made 20 billion in profits last quarter. Why mess with a good thing? Innovator’s Dilemma. Also, to be fair, OpenAI’s huge valuation is only a promise right now. Hopefully they will eventually turn profitable…
- codethatwerks 3y agoShareholder value is fine. Share seller value is what is targeted.
- 3y ago
- freddealmeida 3y agoAttention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in compute with simple algorithms were just better. To argue one group discovered and changed AI and ignore all the other groups is really annoying. I'm glad for these researchers. They deserve the accolade but they didn't invent modern AI. They advanced it. In an interesting way. But even now we want to return to a more deterministic approach. World models. Memory. Graph. Energy minimization. Generative is fun and it taught us something but I'm not sure we can just keep adding more and more chips to solve AGI/SGI through compute. Or maybe we can. But that paper is not written yet.
- calepayson 3y agoI’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?
- voiceblue 3y agoFor phrases, Google's TF glossary [0] is a good resource, but it does not cover certain subsets of AI (and more specifically, is mostly focused on TensorFlow). [0] https://developers.google.com/machine-learning/glossary https://developers.google.com/machine-learning/glossary
- peppertree 3y agoIf you are in neuroscience I would recommend looking into neural radiance fields rendering as well. I find it fascinating since it's essentially an over-fitted neural network.
- svachalek 3y agoSomeone put this link up on another discussion the other day and I found it really fascinating: https://bbycroft.net/llm https://bbycroft.net/llm I believe energy minimization is literal, just look at the size of that thing and imagine the power bill.
- xnx 3y ago> Not only were the authors all Google employees, they also worked out of the same offices. Subtle plug for return-to-office. In-person face-to-face collaboration (with periods of solo uninterrupted deep focus) probably is the best technology we have for innovation.
- tarsinge 3y ago> with periods of solo uninterrupted deep focus Which is usually impossible in the office. So more like a mix, which is what all reasonable people are saying.
- kick_in_the_dor 3y ago+1. I worked remotely the past 2 years. I just RTO'd (new team) and it's life-changing when you actually want to get stuff done quickly.
- bevekspldnw 3y ago“Office” does not have to mean open office. Academics all have offices with doors for a reason. I can’t stand open office, but private office in a building with other people is great.
- steven 3y agoNot so subtle!
- throw0101d 3y agoThis 'conversation' with Geoffrey Hinton and Fei-Fei Li goes over a lot of the history of things (1h50m): * https://www.youtube.com/watch?v=QWWgr2rN45o https://www.youtube.com/watch?v=QWWgr2rN45o * https://www.youtube.com/watch?v=E14IsFbAbpI https://www.youtube.com/watch?v=E14IsFbAbpI ('mirror') Goes over Hinton's history and why he went the direction he did with his research, as well as Li's efforts with ImageNet.
- lordswork 3y agoIt's pretty cool that Google employees can see the first transformer implementation (and the reviewer comments) in cs/. So many monumental moments in AI history are archived in Google's intranet.
- tinyhouse 3y agoIt's amazing how much industry changed CS in the last decade. Significantly more than Academia. Resources is part of it of course, but I also think that the majority of best people in CS choose research labs in industry over academia. No surprise here of course - if openai offers you more than $500K a year, why would you go to academia, make peanuts in comparison and worry about politics for your tenure? Then when they get bored at big tech they raise money and start a company.
- nojvek 3y agoGoogle is an Ads company ad the end of the day. They are making 10% revenue from GCloud but still bulk of their profits are from Ads. If they were threatened, Google could have easily owned the cloud. They were the first to a million servers and exabyte storage scale. Their internal infra is top notch. Under no circumstances can they damage their cash cow. They have already shown to be UX hostile by making Ads look like results, and Youtube forcing more ads onto users. The biggest breakthrough an AI researcher can have at Google is to get more ad clicks across their empire. Google management knows this. Without ad revenue, they are fucked. That is my biggest fear. That with AI we are able to exploit the human mind for dopamine addiction through screens. Meta has shown that their algorithm can make the feeds more addictive (higher engagement). This results in more ads. With Insta Reels, Youtube, TikTok e.t.c "Digital Heroin" keeps on getting better year after year.
- poorman 3y agoI'm getting tired of the "Google employee who created X". There is so much prior art it makes me roll my eyes when I see these posts.
- victor106 3y ago> The group is also culturally diverse. Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively. As much as I think the America has lot of things it needs to fix, there is no other country on earth this would be possible. That's just a fact.
- yeshellohi 3y ago> there is no other country on earth this would be possible. That's just a fact I don’t think this the case. If anything the US makes life very hard for even high-skilled work-based immigrants. Many countries have a higher % of foreign born residents than the US (Singapore, Australia, Germany, Canada) I myself used to work at Google UK and my own team was 100% foreign born engineers from every continent.
- malshe 3y agoI agree. As someone who has lived or worked in Asia, Europe, and the US, I always had the most diverse group of colleagues in the US.