45 ms·
Google “We have no moat, and neither does OpenAI”
- justgangster7 3y ago[dead]
- telmop 3y agoDoes this mean Google will be releasing OSS LLMs? They could justify it as "commoditizing your competitors business".
- dragonwriter 3y ago> Does this mean Google will be releasing OSS LLMs? They could justify it as "commoditizing your competitors business". That’s what this piece argues for. I predict it will not be reflected in Google’s strategy in the next, say, six months, or morw to the point until and unless the apparent “Stable Diffusion” moment in LLMs becomes harder to ignore, such as via sustained publicity on concrete commercially significant non-demonstration/non-research use.
- shmerl 3y agoIn that context, OpenAI would be more fitting to be called ClosedAI.
- JumpCrisscross 3y ago> At the beginning of March the open source community got their hands on their first really capable foundation model, as Meta’s LLaMA was leaked to the public A Prometheus moment if I’ve ever seen one.
- rkagerer 3y agoNo moat except for goobibytes of training data they can probably correlate and cross-reference to achieve some modicum of tagging, proprietary TPU hardware, and a giant cloud farm with an army of developers to feed it. Seriously though, I'll be really thrilled to see open source and clever startups run circles around all the incumbent bastards.
- rowiethelabel 3y ago[dead]
- loveseeker86 3y ago[dead]
- pandaheart5 3y ago[dead]
- overKill232 3y ago[dead]
- feedingthefeed 3y ago[dead]
- feedingthefeed 3y ago[dead]
- modelbehaviour 3y ago[dead]
- smashfizzle2 3y ago[dead]
- smashfizzle2 3y ago[dead]
- seekingthelight 3y ago[dead]
- seekingthelight 3y ago[dead]
- litlikeamatch 3y ago[dead]
- litlikeamatch 3y ago[dead]
- livincool5 3y ago[dead]
- livincool5 3y ago[dead]
- lavenderwitch 3y ago[dead]
- lavenderwitch 3y ago[dead]
- myinstaspace 3y ago[dead]
- myinstaspace 3y ago[dead]
- foodgobblers2 3y ago[dead]
- foodgobblers2 3y ago[dead]
- fillinyourblank 3y ago[dead]
- fillinyourblank 3y ago[dead]
- rowiethelabel 3y ago[dead]
- velvetandlinen 3y ago[dead]
- velvetandlinen 3y ago[dead]
- busyreading0 3y ago[dead]
- busyreading0 3y ago[dead]
- sharing_genes 3y ago[dead]
- sharing_genes 3y ago[dead]
- indigosparkle 3y ago[dead]
- indigosparkle 3y ago[dead]
- morelight33 3y ago[dead]
- morelight33 3y ago[dead]
- stellarflower 3y ago[dead]
- stellarflower 3y ago[dead]
- ivoryroses 3y ago[dead]
- ivoryroses 3y ago[dead]
- curiouscat22 3y ago[dead]
- curiouscat22 3y ago[dead]
- powerpuff3 3y ago[dead]
- powerpuff3 3y ago[dead]
- allthatisme1 3y ago[dead]
- allthatisme1 3y ago[dead]
- thriftandflip2 3y ago[dead]
- thriftandflip2 3y ago[dead]
- behindmylens1 3y ago[dead]
- behindmylens1 3y ago[dead]
- modelbehaviour 3y ago[dead]
- E20life 3y ago[dead]
- E20life 3y ago[dead]
- lifeonthefarm2 3y ago[dead]
- lifeonthefarm2 3y ago[dead]
- theveganwarrior 3y ago[dead]
- theveganwarrior 3y ago[dead]
- whatsthewifi 3y ago[dead]
- whatsthewifi 3y ago[dead]
- askastylist 3y ago[dead]
- askastylist 3y ago[dead]
- moreplease3 3y ago[dead]
- moreplease3 3y ago[dead]
- thesocialhub 3y ago[dead]
- thesocialhub 3y ago[dead]
- thecheeseclub 3y ago[dead]
- thecheeseclub 3y ago[dead]
- askastylist34 3y ago[dead]
- askastylist34 3y ago[dead]
- tweetysweetie43 3y ago[dead]
- tweetysweetie43 3y ago[dead]
- mintandrose34 3y ago[dead]
- mintandrose34 3y ago[dead]
- mintandrose33 3y ago[dead]
- mintandrose33 3y ago[dead]
- lemonhoneypie2 3y ago[dead]
- lemonhoneypie2 3y ago[dead]
- blissand234 3y ago[dead]
- blissand234 3y ago[dead]
- snowysecret8 3y ago[dead]
- snowysecret8 3y ago[dead]
- alwaysaugust3 3y ago[dead]
- alwaysaugust3 3y ago[dead]
- tulipheart32 3y ago[dead]
- tulipheart32 3y ago[dead]
- secretwisher4 3y ago[dead]
- secretwisher4 3y ago[dead]
- angeliccutie23 3y ago[dead]
- angeliccutie23 3y ago[dead]
- alwaysaugst2 3y ago[dead]
- alwaysaugst2 3y ago[dead]
- spellboundead5 3y ago[dead]
- spellboundead5 3y ago[dead]
- lovehunter56 3y ago[dead]
- lovehunter56 3y ago[dead]
- virshereads34 3y ago[dead]
- virshereads34 3y ago[dead]
- rubysunn68 3y ago[dead]
- rubysunn68 3y ago[dead]
- fleurlovin23 3y ago[dead]
- fleurlovin23 3y ago[dead]
- infintesoul57 3y ago[dead]
- infintesoul57 3y ago[dead]
- theseafiles75 3y ago[dead]
- theseafiles75 3y ago[dead]
- openair754 3y ago[dead]
- openair754 3y ago[dead]
- zephyr345 3y ago[dead]
- zephyr345 3y ago[dead]
- magicalworld13 3y ago[dead]
- magicalworld13 3y ago[dead]
- iblamejordan75 3y ago[dead]
- iblamejordan75 3y ago[dead]
- petiue456 3y ago[dead]
- petiue456 3y ago[dead]
- bagatiba575 3y ago[dead]
- bagatiba575 3y ago[dead]
- witchoria86 3y ago[dead]
- witchoria86 3y ago[dead]
- loversland75 3y ago[dead]
- loversland75 3y ago[dead]
- motelvibes45 3y ago[dead]
- motelvibes45 3y ago[dead]
- publicbutter67 3y ago[dead]
- publicbutter67 3y ago[dead]
- ihateblonde57 3y ago[dead]
- ihateblonde57 3y ago[dead]
- loveseeker86 3y ago[dead]
- clashstudio45 3y ago[dead]
- ssense854 3y ago[dead]
- ssense854 3y ago[dead]
- rainbowsalt57 3y ago[dead]
- rainbowsalt57 3y ago[dead]
- comefeelme75 3y ago[dead]
- comefeelme75 3y ago[dead]
- coyote453 3y ago[dead]
- coyote453 3y ago[dead]
- wattson445 3y ago[dead]
- wattson445 3y ago[dead]
- woldandmoon8 3y ago[dead]
- woldandmoon8 3y ago[dead]
- scarymommy75 3y ago[dead]
- scarymommy75 3y ago[dead]
- sugarhigh 3y ago[dead]
- basementfox6 3y ago[dead]
- basementfox6 3y ago[dead]
- thandbleeker 3y ago[dead]
- thandbleeker 3y ago[dead]
- sugarhigh 3y ago[dead]
- spellbound6 3y ago[dead]
- theloversand 3y ago[dead]
- theloversand 3y ago[dead]
- lackofcolour 3y ago[dead]
- lackofcolour 3y ago[dead]
- weekendlust6 3y ago[dead]
- weekendlust6 3y ago[dead]
- zippyseve78 3y ago[dead]
- zippyseve78 3y ago[dead]
- digitalf33lsr 3y ago[dead]
- digitalf33lsr 3y ago[dead]
- alchemy65 3y ago[dead]
- alchemy65 3y ago[dead]
- natureangel6 3y ago[dead]
- natureangel6 3y ago[dead]
- jobrofan3 3y ago[dead]
- jobrofan3 3y ago[dead]
- vintage56 3y ago[dead]
- vintage56 3y ago[dead]
- darksun54 3y ago[dead]
- darksun54 3y ago[dead]
- socialclub 3y ago[dead]
- socialclub 3y ago[dead]
- purrienne 3y ago[dead]
- purrienne 3y ago[dead]
- latenight 3y ago[dead]
- latenight 3y ago[dead]
- sinthecity 3y ago[dead]
- sinthecity 3y ago[dead]
- dirtydisco65 3y ago[dead]
- dirtydisco65 3y ago[dead]
- pandaheart5 3y ago[dead]
- jimsandkitty 3y ago[dead]
- jimsandkitty 3y ago[dead]
- sundaze6 3y ago[dead]
- sundaze6 3y ago[dead]
- chillwildlife56 3y ago[dead]
- chillwildlife56 3y ago[dead]
- acuminate9 3y ago[dead]
- acuminate9 3y ago[dead]
- angel865 3y ago[dead]
- angel865 3y ago[dead]
- spellbound6 3y ago[dead]
- lilscolder7 3y ago[dead]
- lilscolder7 3y ago[dead]
- teengraph6 3y ago[dead]
- teengraph6 3y ago[dead]
- magicpeach9 3y ago[dead]
- magicpeach9 3y ago[dead]
- oreomagie1 3y ago[dead]
- oreomagie1 3y ago[dead]
- beautybabe5 3y ago[dead]
- beautybabe5 3y ago[dead]
- supergiggles9 3y ago[dead]
- supergiggles9 3y ago[dead]
- pineapple76 3y ago[dead]
- pineapple76 3y ago[dead]
- rosecatcher09 3y ago[dead]
- rosecatcher09 3y ago[dead]
- starshadow7 3y ago[dead]
- starshadow7 3y ago[dead]
- moonkiller1 3y ago[dead]
- moonkiller1 3y ago[dead]
- smile6745 3y ago[dead]
- smile6745 3y ago[dead]
- dancingdimples1 3y ago[dead]
- dancingdimples1 3y ago[dead]
- menkiller 3y ago[dead]
- menkiller 3y ago[dead]
- silenteyes12 3y ago[dead]
- silenteyes12 3y ago[dead]
- dearangel5 3y ago[dead]
- dearangel5 3y ago[dead]
- overKill232 3y ago[dead]
- xoom2342 3y ago[dead]
- xoom2342 3y ago[dead]
- kristyhoney6 3y ago[dead]
- kristyhoney6 3y ago[dead]
- honey54 3y ago[dead]
- honey54 3y ago[dead]
- kristy56 3y ago[dead]
- kristy56 3y ago[dead]
- jellyfish11 3y ago[dead]
- jellyfish11 3y ago[dead]
- instafreack12 3y ago[dead]
- instafreack12 3y ago[dead]
- rainbow456 3y ago[dead]
- rainbow456 3y ago[dead]
- kingdom567 3y ago[dead]
- kingdom567 3y ago[dead]
- cybertron76 3y ago[dead]
- cybertron76 3y ago[dead]
- greymare65 3y ago[dead]
- greymare65 3y ago[dead]
- hearthacker6 3y ago[dead]
- hearthacker6 3y ago[dead]
- codenamelover6 3y ago[dead]
- codenamelover6 3y ago[dead]
- swagprince7 3y ago[dead]
- swagprince7 3y ago[dead]
- princeheaven1 3y ago[dead]
- princeheaven1 3y ago[dead]
- heaven11 3y ago[dead]
- heaven11 3y ago[dead]
- justgangster7 3y ago[dead]
- bee769 3y ago[dead]
- bee769 3y ago[dead]
- beloved76 3y ago[dead]
- beloved76 3y ago[dead]
- romeo7657 3y ago[dead]
- romeo7657 3y ago[dead]
- badcaptain46 3y ago[dead]
- badcaptain46 3y ago[dead]
- beanbasket76 3y ago[dead]
- beanbasket76 3y ago[dead]
- cyberking786 3y ago[dead]
- cyberking786 3y ago[dead]
- dark578 3y ago[dead]
- dark578 3y ago[dead]
- gentlepeople 3y ago[dead]
- gentlepeople 3y ago[dead]
- peoplewithnojob 3y ago[dead]
- peoplewithnojob 3y ago[dead]
- dreamer654 3y ago[dead]
- dreamer654 3y ago[dead]
- rudepeople 3y ago[dead]
- rudepeople 3y ago[dead]
- naughtyslay4 3y ago[dead]
- naughtyslay4 3y ago[dead]
- bachelor678 3y ago[dead]
- bachelor678 3y ago[dead]
- hello_computer 3y agoI think YouTube is a damn fine moat.
- tiniuclx 3y agoI've been using Stable Diffusion to generate cover images for the music I release & produce for others. It's a massive time saver compared to comping together the release art using image editing software, and a lot cheaper than working with artists, which just doesn't make sense financially as an independent musician. It's a little bit difficult to get what you want out of the models, but I find them very useful! And while the output resolution might be quite low, things are improving & AI upscaling also helps a lot.
- benjaminsky2 3y ago> artist whose domain has not yet been disrupted by AI fires artist in favor of AI
- echelon 3y agoEveryone is getting disrupted by AI sooner or later. The trick is to use AI to do things it would take you five lifetimes to learn. It's a tool to lower opportunity cost, financial capital, and human capital. That gives anyone leveraging it a much bigger platform, and the ability to dream big without resources. If you can become your own "studio", you're the indie artist of the future. You don't need Disney or Universal Music backing. Anyone can step up and do this. The artists being threatened can use these tools to do more than they've ever done by themselves.
- jprete 3y agoThe problem isn’t capabilities, it’s having a market that’s saturated with supply twice over - once by the ability to make infinite copies of the product, the other where there’s an infinite supply of distinct high-quality products. Subcultures used to provide a counterbalancing force here, but they aren’t doing so well these days.
- echelon 3y agoMy interests still are not being catered to. I watch films and media, listen to music, and I'm only truly fully satisfied a single digit number of times a year. That's a consequence of not enough being created and experimented with. The long tail is longer than you can imagine, and that's what form fits to your personal interest graph.
- beardyw 3y agoThis is such an interesting read. It makes a compelling case, though how the likes of Google should react feels less like an adjustment and more like a revolution.
- sashank_1509 3y agoCringe, haven’t seen a single Open Source come even close to the ability of Bard, let alone ChatGPT. Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!
- lapinot 3y ago> Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train! Because surely price = quality. Solid argumentation there.
- Mike_12345 3y agoYes, price = quality because they require supercomputing resources to train. GPT-3 required hundreds of Tesla GPUs running for several weeks. That's millions of dollars just for hardware, not including power (the GPUs cost $15k each)
- sashank_1509 3y agoYou’re right but I’d just like to add, GPT-3 probably required 1000’s of GPUs. OpenAI is known to have the largest cluster 16k+ A100 GPUs and most of them were used for the major model training.
- vlovich123 3y agoIs there any reason to think that zero-shot learning and better models/more effient AI won’t drastically reduce those costs over time?
- ebiester 3y agoThink a little more laterally. If we're talking about doing everything well, I think that's true. However, if I want to create my own personal "word calculator," I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor. A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer's context into the system without any privacy concerns. A model doesn't have to be better at everything to be better at something.
- shanebellone 3y agoMoat == War Chest
- vlovich123 3y agoI think that analogy is flawed to try to undercut OpenAI’s lead. The reason it’s flawed is that the search business is really lucrative and OpenAI is trying to completely disrupt Google’s business there. So while the AI isn’t a moat, establishing a lead in search is because you obviously will use that to inject ads in the commercial space and capture the market.
- Reubend 3y agoGreat read, but I don't agree with all of these points. OpenAI's technological moat is not necessarily meaningful in a context where the average consumer is starting to recognize ChatGPT as a brand name. Furthermore, models which fine-tune LLMs are still dependent on the base model's quality. Having a much higher quality base model is still a competitive advantage in scenarios where generalizability is an important aspect of the use case. Thus far, Google has failed to integrate LLMs into their products in a way that adds value. But they do have advantages which could be used to gain a competitive lead: - Their crawling infrastructure could allow their to generate better training datasets, and update models more quickly. - Their TPU hardware could allow them to train and fine-tune models more quickly. - Their excellent research divisions could give them a head start with novel architectures. If Google utilizes those advantages, they could develop a moat in the future. OpenAI has access to great researchers, and good crawl data through Bing, but it seems plausible to me that 2 or 3 companies in this space could develop sizeable moats which smaller competitors can't overcome.
- amf12 3y ago> context where the average consumer is starting to recognize ChatGPT as a brand name. Zoom was once that brand name which was equated to a product. Now, people might say "Zoom call", but may use Teams or Meet or whatever. Similarly, people call a lot of robot vacuum cleaners Roombas, even though they might be some other brand. Brand recognition is one thing, but the actual product used will always depend on what their employer uses, what their mobile OS might use, or what API their products might use. For businesses, a lot will be about the cost and performance vs "the best available".
- ealexhudson 3y agoConsumers recognizing ChatGPT might just end up like vacuum cleaners; at least in the UK, people will often just call it a "hoover" but the likelihood of it being a Hoover is low. It is difficult to see where the moat might exist if it's not data and the majority of the workings are published / discoverable. I don't think the document identifies a readily working strategy to defend against the threats it recognises.
- 3y ago
- ronaldoCR 3y agoDoesn't the sheer cost of training create a moat on its own?
- echelon 3y agoYes, but so far we've seen universities, venture-backed open source outfits, and massive collections of hobbyists train all sorts of large models.
- hiddencost 3y agoIt's cheap to distill models, and trivial to scrape existing models. Anything anyone does rapidly gets replicated for 1/500th the price.
- phyllistine 3y ago[dead]
- summerlight 3y agoThis looks like a personal manifesto from an engineer who doesn't even attempt to write it on behalf of Google? The title is significantly misleading.
- capableweb 3y agoAgree, misleading title. The introduction makes the context clear, but probably too late to not call the article click-bait. > [...] It originates from a researcher within Google. [...] The document is only the opinion of a Google employee, not the entire firm. [...]
- dpflan 3y agoCompletely agree. It is interesting, but the gravitas of it seems lower than of course if an executive said this and corroborated it. I do feel that opensource for AI is going to be really interesting and shake things up.
- opportune 3y ago99% of media coverage like “Tech employee/company says <provocative or controversial thing>” are exactly like that.
- ghaff 3y agoAnd (probably) through no fault of their own they'll get totally thrown under the bus for this--whether directly but when raises/promotions come around or not.
- cube2222 3y agoFWIW I posted Simon's summary because it's what I encountered first, but here's the leaked document itself[0]. Some snippets for folks who came just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other. > This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them? > Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. > And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move. [0]: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
- davidguetta 3y agoSeems to be the Open Source who is the real winner overall.. After OpenAI became basically ClosedAI it's an excellent news
- Levitz 3y agoI'm not sure? Placing ethics constraints on a company under a capitalist system is hard. Placing them on open source is impossible.
- 3y ago
- sounds 3y agoRepeating myself from https://news.ycombinator.com/item?id=35164971 https://news.ycombinator.com/item?id=35164971 : > OpenAI can't build a moat because OpenAI isn't a new vertical, or even a complete product. > Right now the magical demo is being paraded around, exploiting the same "worse is better" that toppled previous ivory towers of computing. It's helpful while the real product development happens elsewhere, since it keeps investors hyped about something. > The new verticals seem smaller than all of AI/ML. One company dominating ML is about as likely as a single source owning the living room or the smartphones or the web. That's a platitude for companies to woo their shareholders and for regulators to point at while doing their job. ML dominating the living room or smartphones or the web or education or professional work is equally unrealistic.
- photochemsyn 3y agoML dominating education seems pretty realistic to me. E.g. this series of prompts for example: > "Please design a syllabus for a course in Computer Architecture and Assembly language, to be taught at the undergraduate level, over a period of six weeks, from the perspective of an professor teaching the material to beginning students." > "Please redesign the course as an advanced undergraduate six-month Computer Architecture and Assembly program with a focus on the RISC-V ecosystem throughout, from the perspective of a professional software engineer working in the industry." > "Under the category of Module 1, please expand on "Introduction to RISC-V ISA and its design principles" and prepare an outline for a one-hour talk on this material" You can do this with any course, any material, any level of depth - although as you go down into the details, hallucinations do become more frequent so blind faith is unwise, but it's still pretty clear this has incredible educational potential.
- sounds 3y agoFortunately, what I said was that a single company becomes the sole source for the ML in education; not the same thing and thus I have no conflict with your statement.
- rvz 3y ago> The premise of the paper is that while OpenAI and Google continue to race to build the most powerful language models, their efforts are rapidly being eclipsed by the work happening in the open source community. Another magnificent unsurprising set of correct prediction(s) [0] [1] [2] and as triumphantly admitted by Google themselves on open source LLMs eating both of their (Google) and OpenAI's lunch. "When it is the race to the bottom, AI LLM services, like ChatGPT, Claude (Anthropic), Cohere.ai, etc are winning the race. Open source LLMs are already at the finish line." [0] https://news.ycombinator.com/item?id=34201706 https://news.ycombinator.com/item?id=34201706 [1] https://news.ycombinator.com/item?id=35661548 https://news.ycombinator.com/item?id=35661548 [2] https://news.ycombinator.com/item?id=34716545 https://news.ycombinator.com/item?id=34716545
- bottlelion 3y agoOpenAI may not have a moat, but Microsoft does with their Enterprise Agreements.
- endisneigh 3y agoit'll be fun to see the pikachu face when engineers are expected to do more, with the aid of these tools, but are not paid any more money.
- com2kid 3y agoKind of like every other improvement in technology? From interactive terminals, to compilers, to graphical debuggers? Nothing new there. What productivity improvements have opened up is more opportunities for developers. Larger and more complex systems can be built using better tooling.
- endisneigh 3y ago> Larger and more complex systems can be built using better tooling. to what end, make rich people richer?
- com2kid 3y ago> to what end, make rich people richer? So, in perfect theory land, people get paid because they provide value. That obviously breaks down at the extremes. But, for sake of example, let's take Uber, super easy to hate on them, but they have had a measurable impact on reducing deaths from drunk driving. That obviously provides a lot of value to people. Likewise, it is hard to overstate the value people have gained from smartphones, Apple has made a lot of money but they have also provided a lot of value. Arguments over if the individual value brought is long term good or bad for society are a separate topic, but people value their iPhones and therefor they pay for them. No way could something as complicated as an iPhone have been made with 1970s software engineering technology.
- endisneigh 3y agoI’m not arguing that. I’m saying the bar is higher and pay relative to value has decreased for all other than at the upper end. Easiest way to think about this is look at percentage revenue paid to engineers.
- deleted 3y ago[deleted]
- xyzzy4747 3y agoI disagree with this. It's too expensive to train high quality models. For example I don't see how anyone would make an open-source GPT4 unless OpenAI leaks their model to the public.
- coolspot 3y agoNo one has created even something closed-source that is equal to GPT4.
- Hippocrates 3y agoELI5 How is it too expensive? I know ChatGPT was expensive to train but Vicuna-13b is said to have cost $300 to train [https://lmsys.org/blog/2023-03-30-vicuna/ https://lmsys.org/blog/2023-03-30-vicuna/]
- ktbwrestler 3y agoCan someone dumb this down for me because I don’t understand why this is a surprise… people are getting excited and collaborating to improve and innovate the same models that these larger companies are milking to death
- cube2222 3y agoBasically, if I understand correctly, the "status quo" was that the big models by OpenAI and Google that are much better (raw) than anything that was open source recently, would remain the greatest, and the moat would be the technical complexity of training and running those big models. However, the open sourcing led to tons of people exploring tons of avenues in an extremely quick fashion, leading to the development of models that are able to close in on that performance in a much smaller envelope, destroying the only moat and making it possible for people with limited resources to experiment and innovate.
- galaxyLogic 3y ago> big models by OpenAI and Google that are much better (raw) than anything that was open source recently, When you say "models" do you mean TRAINED models? Wouldn't the best training and supervised/feedback learning still be in the hands of the big players? An open source "model" of all content in the open internet is great, but it has the garbage-in/garbage-out problem.
- summerlight 3y agoNote that this is a personal manifesto, which doesn't really represent Google's official stance. Which is unfortunate because I'm largely aligned with this position.
- hot_gril 3y agoFrom one researcher, not a VP, director, etc.
- rcme 3y agoI think OpenAI has a defensible moat by having the first movers' advantage. As the ease of producing written content declines, we can expect the amount of written content to increase in a consummate fashion. Due to OpenAI's position, a vast majority of the newly generated data will come from OpenAI's models. When it comes time to train new models with superior network structures or with new data, no one else is going to be able to differentiate human-generated text from LLM generated text. OpenAI's training data should become far superior to others.
- hammock 3y agoThe non-public moat is big multiyear government contracts with dark money. And there is room there for both players :)
- deleted 3y ago[deleted]
- hammock 3y agoThere is a huge (non-public) moat. It's big multiyear government contracts with dark money. And there is room there for both players :)
- minimaxir 3y agoHaving enough scale to perpetually offer free/low-cost compute is a moat. The primary reason ChatGPT went viral in the first place was because it was free, with no restrictions. Back in 2019, GPT-2 1.5B was made freely accessible by a single developer via the TalkToTransformers website, which was the very first time many people talking about AI text generation...then the owner got hit with sticker shock from the GPU compute needed to scale. AI text generation competitors like Cohere and Anthropic will never be able to compete with Microsoft/Google/Amazon on marginal cost.
- FemmeAndroid 3y agoCharity is only a moat if it’s not profitable.
- r00fus 3y agoThere's "immediately profitable" and "eventually profitable". Vast compute scale allows collection of customer generated data so the latter is possible, AI as of yet is not the former. So GP point still stands. FAAMG can run much larger immediate deficits in order to corner the market on the eventual profitability of AI.
- Tostino 3y agoThe amount of valuable data generated from professionals using these services to work through their problems and find solutions to industry problems is immense. It essentially gives these companies the keys to automating many industries by just...letting people try and make their jobs easier and collecting all data.
- cushpush 3y agoAll this talk that every investment pays off in the end is faulty and dangerous. Many investments don't pan out, 95% of the firms you see in the ticker this decade might be gone, and yet everyone is very confident is underwriting these "losses for future gains" but really it's economies of scale. It doesn't cost MSFT much more to run the GPU than to turn it on in the first place.
- DonHopkins 3y ago>Research institutions all over the world are building on each other’s work, exploring the solution space in a breadth-first way that far outstrips our own capacity. BroadMind beats DeepMind!
- LesZedCB 3y agowhat this proves to me without a doubt is that silo'd and proprietary iteration i still very clearly also a massive disadvantage. i really hope companies internalize that. if they just keep scooping up and hiding open-source improvements they very well may still be left behind. the final quote from the doc: > And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.
- aresant 3y ago"People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. . ." I'll take the opposite side of that bet - MSFT / Goog / etc in the providers side will drive record revenues on the back of closed / restricted models: 1 - Table stakes for buying software at enterprise level is permissions based management & standardized security / hardening. 2 - The corporate world is also the highest value spender of software 3 - Corp world will find the "proprietary trained models" on top of vanilla MSFT OpenAI or Goog Bard pitch absolutely irresistible - creates a great story about moats / compounding advantages etc. And the outcome is going to most likely be higher switching costs to leave MSFT for a new upstart etc
- IceHegel 3y agoI agree with this over the next 10 years but disagree over the next 30. When/If the innovation slows down, the open source stuff will be able to out compete commercial options. Something like this timeline played out for databases and operating systems.
- vlaaad 3y agoThis looks very fake to me. I might be wrong. Yet, there is no "document" that was leaked, the original source is some blog post. If there is a document, share the document. Shared by "anonymous individual on discord who granted permission for republication"... I don't know. If it was shared by anonymous, why ask for permission? Which discord server?
- vlaaad 3y agoThis looks very fake to me. I might be wrong. Yet, there is no "document" that was leaked, the original source is some blog post. If there is a document, share the document. Shared by "anonymous individual on discord who granted permission for republication"... I don't know. If it was shared by anonymous, why ask for permission? Which discord server?
- simonw 3y agoDid you read it? I honestly don't care if it's really a leak from inside Google or not: I think the analysis stands on its own. It's a genuinely insightful summary of the last few months of activity in open source models, and makes a very compelling argument as to the strategic impact those will have on the incumbent LLM providers. I don't think it's a leak though, purely because I have trouble imagining anyone writing something this good and deciding NOT to take credit for the analysis themselves.
- ftxbro 3y ago> It's a genuinely insightful summary of the last few months of activity in open source models Yes this is an amazing summary! Just for its summary alone, it is probably one of the top five writings I saw on LLMs and I read every one! > because I have trouble imagining anyone writing something this good and deciding NOT to take credit for the analysis themselves. Not everyone has a substack that they spam onto hacker news every time a thought enter their head. Or imagine that INTP exist. In my opinion the best take ever on LLMs is the simulators essay https://generative.ink/posts/simulators/ https://generative.ink/posts/simulators/ and the author is so shy they went pseudonymous and put their twitter as private and don't even want their name in conferences.
- heliophobicdude 3y agoI think the another great paper is RLHF article from Chip Huyen https://huyenchip.com/2023/05/02/rlhf.html https://huyenchip.com/2023/05/02/rlhf.html
- mark_l_watson 3y agoI tend to agree. For now the OpenAI APIs are so very easy to use and effective. I do try to occasionally use HF models, mostly running locally in order to keep my options open. My bet is that almost everyone wants to keep their options open. I am very much into auxiliary tools like LangChain and LlamaIndex, the topic of my last book, but I also like building up my own tools from scratch (mostly in Common Lisp and Swift for now), and I bet most devs and companies are doing the same.
- kyaghmour 3y agoGoogle's moat is its data set. Imagine training an generative AI LLM on the entire set of YouTube training videos. No one else has this.
- SanderNL 3y agoI have been toying around with Stable Diffusion for a while now and becoming comfortable with the enormous community filled with textual inversions, LoRAs, hyper networks and checkpoints. You can get things with names like “chill blend”, a fine-tuned model on top of the SD with the author’s personal style. There is something called automatic1111 which is a pretty comprehensive web UI for managing all these moving parts. Filled to the brim with extensions to handle AI upscaling, inpainting, outpainting, etc. One of these is ControlNet where you can generate new images based on pose info extracted from an existing image or edited by yourself in the web based 3d editor (integrated, of course). Not just pose but depth maps, etc. All with a few clicks. The level of detail and sheer amount of stuff is ridiculous and it all has meaning and substantial impact on the end result. I have not even talked about the prompting. You can do stuff like [cow:dog:.25] where the generator will start with a cow and then switch over at 25% of the process to a dog. You can use parens like ((sunglasses)) to focus extra hard on that concept. There are so called LoRAs trained on specific styles and/or characters. These are usually like 5-100MB and work unreasonably well. You can switch over to the base model easily and the original SD results are 80s arcade game vs GTA5. This stuff has been around for like a year. This is ridiculous. LLMs are enormously “undertooled”. Give it a year or so. My point by the way is that any quality issues in the open source models will be fixed and then some.
- Der_Einzige 3y agoI wrote a whole gist about this exact thing!!!! https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
- int_19h 3y agoLocal LLMs already have a UI intentionally similar to AUTOMATIC1111, including LoRAs, training with checkpoints, various extensions including multimodal and experimental long-term memory etc. https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- SanderNL 3y ago
- DonHopkins 3y ago"I Have No Moat, and I Must Scream" https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Scream https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Sc...
- _trackno5 3y agoI get the feeling that at this point, the best thing Google could do is to go all in an open source their models and weights. They'd canibalize their own business, but they'd easily wipe out most of the competition.
- curiousgal 3y agoI'm convinced that anyone sounding the alarm bells about AI has no idea whatsoever how these models are built.
- mark_l_watson 3y agoCopied from other thread: I tend to agree. For now the OpenAI APIs are so very easy to use and effective. I do try to occasionally use HF models, mostly running locally in order to keep my options open. My bet is that almost everyone wants to keep their options open. I am very much into auxiliary tools like LangChain and LlamaIndex, the topic of my last book, but I also like building up my own tools from scratch (mostly in Common Lisp and Swift for now), and I bet most devs and companies are doing the same.
- joezydeco 3y ago"Many of the new ideas are from ordinary people." Yeah. Google can fuck right off. Maybe this attitude is what got them in the weeds in the first place.
- uptownfunk 3y agoI was quite unimpressed when I interviewed with them recently. It’s no surprise their lunch is getting eaten.
- GartzenDeHaes 3y agoYes, it's very telling.
- IceHegel 3y agoI don’t think trying to be the hall monitor of humanity has been good for google. The more paternalistic, the less innovative.
- _trackno5 3y agoI get the feeling that at this point, the best thing Google could do is to go all in an open source their models and weights. They'd canibalize their own business, but they'd easily wipe out most of the competition.
- kyaghmour 3y agoGoogle's moat is its data set. Imagine training an generative AI LLM on the entire set of YouTube training videos. No one else has this.
- RecycledEle 3y agoThe entire set of YouTube training videos needs to be re-transcribed before they are useful for training LLMs.
- dopeboy 3y agoThis is the glaring omission in this piece. Googles know _so much_ about me. Is it not reasonable to assume powerful llm + personal data = personal tuned LLM?
- DethNinja 3y agoIf OpenAI has no moat, how come nobody has built a better alternative to GPT-4 yet?
- drcode 3y agodissaproving_drake.jpg: Giving evidence you can match the capabilities of OpenAI approving_drake.jpg: Saying everything OpenAI does is easy
- ad404b8a372f2b9 3y agoI'm feeling strangely comforted to have pictured the mythological creature before the meme.
- hannofcart 3y agoYes. Same here. I had flashes of Battle of Wesnoth. Then I realized...
- lysecret 3y agoSo I use ChatGPT every day. I like it a lot and it is useful but it is overhyped. Also from 3.5 to 4 the jump was nice but seemed relatively marginal to me. I think the head start OpenAi has will vanish. Iteration will be slow and painful giving google or whoever more than enough time to catch up. ChatGPT was a fantastic leap getting us say 80% to Agi but as we have seen time and time again the last 20% are excruciatingly slow and painful (see Self driving cars).
- Tostino 3y agoPersonally, the difference between GPT4 and 3.5 is, pretty immense for what I am using it for. I can use GPT 3.5 for things like summarization tasks (as long as the text isn't too complex), reformatting, and other transformation type tasks alright. I don't even bother with using it for logical or programming tasks though.
- killthebuddha 3y agoOne way that I've been framing this in my head (and in an application I'm building) is that gpt-3 will be useful for analytic tasks but gpt-4 will be required for synthetic tasks. I'm using "analytic" and "synthetic" in the same way as in this writeup https://github.com/williamcotton/empirical-philosophy/blob/main/articles/from-prompt-alchemy-to-prompt-engineering-an-introduction-to-analytic-agumentation.md https://github.com/williamcotton/empirical-philosophy/blob/m...
- burnished 3y agoWould you please be more explicit? I'm curious about the relative strength's and weaknesses other's see
- Tostino 3y agoI can use GPT 4 for to work through problems that I have not actually figured out previously by talking to co-workers who work in my industry. I need to feed it contacts for my industry explicitly within the prompt and ensure that it understands and doesn't hallucinate its answers. However, that doesn't mean it's not useful, it just means you need to you understand the limitations.
- akhayam 3y agoThe real moats in this field will come from the hardware industry. It's way too expensive to train these models on general purpose compute. Vertically designed silicon that brings down the unit economics of training and inference workloads are already being designed, in industry and in academia.
- danielmarkbruce 3y agoNVIDIA already has a big moat in this area. It might not last forever, but at least for a good while they have a big one.
- geepytee 3y agoA lot of good points made here, thank you for sharing!
- simonw 3y agoPosted a few notes on this here: https://simonwillison.net/2023/May/4/no-moat/ https://simonwillison.net/2023/May/4/no-moat/
- yyyk 3y agoThe memo sounds like spin because it is. The surface argument is equivalent to arguing that no one could sell closed source software because open source exists, and that open source must also be commodotized (oddly, Apple and Microsoft are doing just fine). The implied argument is that Google Research was doing fine giving away their trade secrets and giving negative value to Google because it was going to happen anyway and the secrets are financially worthless anyhow. Nonsense. There are moats if one is willing to look for them. After all, productizing is a very different thing from an academic comparison. ChatGPT is way out there _as a product_, while open efforts are at 0% on this. You can't lock down a technology*, but you can lock down an ecosystem, a product or hardware. OpenAI can create an API ecosystem which will be difficult to take down. They can try to make custom hardware to make their models really cheap to run. Monopoly? Nah. This won't happen. But they could make some money - and reduce the value of Google's search monopoly. * Barring software patents which fortunately aren't yet at play. EDIT: I'll give the memo a virtual point for identifying Meta (Facebook) as a competitor who could profit by using current OSS efforts. But otherwise it's just spin.
- burnished 3y agoHow do you distinguish between an opinion you disagree with and 'spin'?
- yyyk 3y agoA) When IMHO the underlying argument is not quite honest. B) When it comes from an interested party. C) When there's enough of A and B that I feel it's intentional. The underlying argument here would apply to some extremely profitable existing closed source software, so it's obviously not complete. Even closed source software which is strictly inferior manages to find some very profitable moats. As for the source, it comes from Google Research, which has done a very poor job of using their knowledge to benefit Google. The article downplays the failures (we couldn't have done anything differently, but it doesn't matter anyway since Open Source will consume all!), and doesn't even think about productization. The latter does give me a little bit of doubt: the article could also be emblematic of Google Research's failures and not 100% spin...
- tikkun 3y agoThe part of the post that resonates for me is that working with the open source community may allow a model to improve faster. And, whichever model improves faster, will win - if it can continue that pace of improvement. The author talks about Koala but notes that ChatGPT is better. GPT-4 is then significantly better than GPT-3.5. If you've used all the models and can afford to spend money, you'd be insane to not use GPT-4 over all the other models. Midjourney is more popular (from what I'm seeing) than Stable Diffusion at the moment because it's better at the moment. Midjourney is closed-source. The point I'm wanting to make is that users will go to whoever has the best model. So, the winning strategy is whatever strategy allows your model to compound in quality faster and to continue to compound that growth in quality for longer. Open source doesn't always win in producing better quality products. Linux won in servers and supercomputing, but not in end user computing. Open-source databases mostly won. Chromium sorta won, but really Chrome. Then in most other areas, closed-source has won. So one takeaway might be that open-source will win in areas where the users are often software developers that can make improvements to the product they're using, and closed-source will win in other areas.
- wahnfrieden 3y agoGPT4 sucks for many use cases because it's SLOW. It will co-exist with ChatGPT variants.
- jquery 3y agoIt's about using the right tool for the right job. GPT-4 is an incredibly versatile generalist tool and a fantastic jack of all trades. However, this comes with some drawbacks. While saying it 'sucks' might be an exaggeration, I generally concur with the point you're making.
- wahnfrieden 3y agoIt's no exaggeration that it sucks for certain use cases where you would expect and can achieve near-realtime response, and that's fine because it's not built for that use case. I'm responding to someone saying it's always best if you can afford it
- cube2222 3y agoSome snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with instruction tuning, quantization, quality improvements, human evals, multimodality, RLHF, etc. etc. many of which build on each other. > This recent progress has direct, immediate implications for our business strategy. Who would pay for a Google product with usage restrictions if there is a free, high quality alternative without them? > Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products. > And in the end, OpenAI doesn’t matter. They are making the same mistakes we are in their posture relative to open source, and their ability to maintain an edge is necessarily in question. Open source alternatives can and will eventually eclipse them unless they change their stance. In this respect, at least, we can make the first move.
- sterlind 3y agoI wonder if OpenAI knew they didn't have a moat, and that's why they've been moving so fast and opening ChatGPT publicly - making the most of their lead in the short time they have left. I find it incredibly cathartic to see these massive tech companies and their gatekeepers get their lunch eaten by OSS.
- techwiz137 3y agoSorry for the dumb question. But in the context of the AI space, what is moat?
- leereeves 3y ago
- ChaitanyaSai 3y agoThis is easily among the rare highest quality articles/comments I've read in the past weeks, perhaps months (on LLMs/AI since that's what I am particularly interested in). And this was for internal consumption before it was made public. Reinforces my recent impression that so much that's being made for public consumption now is shallow and it is hard to find the good stuff. And sadly, increasing so even on HN. As I write this, I acknowledge I discovered this on HN :) Wish we had ways to incentivize the public sharing of such high-quality content that don't die at the altar of micro rewards.
- censor_me 3y ago[dead]
- visarga 3y agoI've been saying the same things for weeks, right here and in the usual places. Basically - OpenAI will not be able to continue to commercialise chatGPT-3.5, they will have to move to GPT-4 because the open source alternatives will catch up. Their island of exclusivity is shrinking fast. In a few months nobody will want to pay for GPT-4 either when they can have private, cheap equivalents. So GPT-5 it is for OpenAI. But the bulk of the tasks can probably be solved at 3.5 level, another more difficult chunk with 4, I'm wondering how many of the requests will be so complex as to require GPT-5. Probably less than 1%. There's a significant distinction between web search and generative AI. You can't download "a Google" but you can download "a LLaMA". This marks the end of the centralisation era and increased user freedom. Engaging in chat and image generation without being tracked is now possible while searching, browsing the web or torrenting are still tracked.
- com2kid 3y ago> I've been saying the same things for weeks, right here and in the usual places. Basically - OpenAI will not be able to continue to commercialise chatGPT-3.5, they will have to move to GPT-4 because the open source alternatives will catch up. Their island of exclusivity is shrinking fast. In a few months nobody will want to pay for GPT-4 either when they can have private, cheap equivalents. So GPT-5 it is for OpenAI. It is worth $20 a month to have one UI on one service that does everything. Unless specialized models can far exceed what GPT4 can do, being general purpose is amazing. IMHO the future is APIs written for consumption by LLMs, and then natural language interfaces and just telling an AI literally anything you want done.
- madsbuch 3y agoI would lean towards agreeing. And I definitely think AI companies should not try to make their money on inference. If there is a well performing model being deployed it is possible to train a similar model while not having to eat the cost of exploration. Ie. it is only the the cost of training said model. ChatGPT would probably die in a couple of weeks, if an equivalent, free, product came out that people could run on their computers.
- uptownfunk 3y agoOpenAI is further along than most of us are aware. The ability to connect these models to the web, to pipe up API access to different services and equip LLMs to be the new interface to these services and to the worlds information is the real game changer. Google cannot out innovate them because they are a big Corp rife with googly politics and challenges of overhead that come with organizational scale. I would be curious to see if there are plans to spin off the newly consolidated AI unit with their own PnL to stimulate that hunger to grow and survive and then capitalize them accordingly. Otherwise they are en route to die a slow death once better companies come along.
- IceHegel 3y agoThe current CEO, who a friend at google calls “Captain Zonk”, is dispositionaly not the person to make that kind of change. I wouldn’t be surprised to see a leadership change this year.
- whimsicalism 3y ago> Giant models are slowing us down. In the long run, the best models are the ones which can be iterated upon quickly. We should make small variants more than an afterthought, now that we know what is possible in the <20B parameter regime. Maybe this is true for the median query/conversation that people are having with these agents - but it certainly has not been what I have observed in my experience in technical/research work. GPT-4 is legitimately very useful. But any of the agents below that (including ChatGPT) cannot perform complex tasks up to snuff.
- mirekrusin 3y agoSpot on. I think author forgot to mention StableLM?
- fnordpiglet 3y agoThe moat comes by integrated LLM and generative AI into classical technique feedback cycles, finetuning in specialized domains, and other “application” of LLM where LLM acts as an abstract semantic glue between subsystems, agents, optimizers, and solvers. The near obsessive view that generative AI is somehow an end rather than an enabler is one of the more shortsighted sides I see in this whole discussion of generative AIs over the last several years, peaking recently with ChatGPT
- paxys 3y ago> The document is only the opinion of a Google employee, not the entire firm The title makes it seem like this is some official Google memo. The company has 150K employees and 300K different opinions on things. Can't go chasing down each one and giving it importance.
- Alifatisk 3y agoWhat Facebook did to the community and their leaked torrent accelerated everything.
- ljlolel 3y agoThis reads like a psy-op "leak" to try to convince OpenAI execs to open source GPT4 weights
- api 3y agoThis has been my speculation about the people pushing for regulation in this space: it’s an attempt at regulatory capture because there really is little moat with this tech. I can already run GPT-3 comparable models on a MacBook Pro. GPT-4 level models that can run on at least higher end commodity hardware seem close. Models trained on data scraped from the net may not be defensible via copyright and they certainly are not patentable. It also seems possible to “pirate” models by training a model on another model. Defending against this or even detecting it would be as hard as preventing web scraping. Lastly the adaptive nature of the tech makes it hard to achieve lock in via API compatibility. Just tell the model to talk a different way. The rigidity of classical von Neumann computing that facilitates lock in just isn’t there. So that leaves the old fashioned way: frighten and bribe the government into creating onerous regulations that you can comply with but upstarts cannot. Or worse make the tech require a permit that is expensive and difficult to obtain.
- a-user-you-like 3y agoAct like a socialist and then blame it on capitalism, American playbook 101
- EGreg 3y agoWow, an open source gift economy beating the closed-source capitalistic model? You don't say. Wikipedia handily beat Britannica (the most well-known and prestigious encyclopedia, sold door to door) and Encarta (supported by Microsoft) The Web beat AOL, CompuServe, MSN, newspapers, magazines, radio and TV stations, etc. Linux beat closed source competitors on tons of environments Apache and NGinX beat Microsoft Internet Information Server and whatever else proprietary servers. About the only place it doesn't beat, is consumer-facing frontends. Because open-source does take skill to use and maintain. But that's why the second layer (sysadmins, etc.) have chosen it.
- berkle4455 3y agoShorting Google is the best possible bet is my read.
- lysecret 3y agoFantastic article if you are quick to just go to the comments like I usually do, don't. Read it. One of my favorites: LoRA works by representing model updates as low-rank factorizations, which reduces the size of the update matrices by a factor of up to several thousand. This allows model fine-tuning at a fraction of the cost and time. Being able to personalize a language model in a few hours on consumer hardware is a big deal, particularly for aspirations that involve incorporating new and diverse knowledge in near real-time. The fact that this technology exists is underexploited inside Google, even though it directly impacts some of our most ambitious projects. Anyone has worked with LoRa ? Sounds super interesting.
- epiccoleman 3y agoWe need to scrape the entire corpus of /r/ASOIAF so it can come up with wild theories about how Tyrion is a secret Targaryen and confirm Benjen == Daario once and for all.
- adroitboss 3y agoYou can find the guy who created it on reddit u/edwardjhu. I remember because he showed up in the Stable Diffusion Subreddit. https://www.reddit.com/r/StableDiffusion/comments/1223y27/im_the_creator_of_lora_how_can_i_make_it_better/ https://www.reddit.com/r/StableDiffusion/comments/1223y27/im...
- seydor 3y agoIf i understand correctly it is also shockingly simple, basically just the first figure in the paper: https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa45zITSg.png https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa4... train 2 matrices, add their product to the pretrained weights, and voila! Someone correct me if i m wrong
- PoignardAzur 3y agoI had to read the paper first, but yeah, that diagram is shockingly simple once you get it. Some annotations: - The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero". - d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.) - r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.
- amelius 3y agoIntelligence is becoming a commodity.
- GartzenDeHaes 3y ago> Paying more attention to their work could help us to avoid reinventing the wheel. There's nothing us humans love more than reinventing the wheel. I've seen it over and over again, years of work and hundreds of millions of dollar spent re-solving problems and re-writing systems -- only to replace them with a new set of slightly different problems. I think we greatly over estimate the ability of our species to accumulate knowledge, which is perhaps where these generative systems come into play.
- seydor 3y agoNot only they have no moat, Open source models are uncensored and this is huge. Censorship is not just political , it cripples the product to basically an infantile stage and precludes so many applications. For once, it is a liability But this article doesn't state the very obvious: When will google (the inventor of Transformer, and "rightful" godfather of modern LLMs) , release a full open source, tinkerable model better than LLaMa? (To the dead comment below, there are many uncensored variations of vicuna)
- thomas34298 3y ago>Open source models are uncensored and this is huge Vicuna-13B: I'm sorry, but I cannot generate an appropriate response to this prompt as it is inappropriate and goes against OpenAI's content policy.
- UncleEntity 3y ago> When will google release a full open source, tinkerable model better than LLaMa? Arguably, Facebook released llama because it had no skin in the game. Google, on the other hand, has a lot of incentive to claw back the users who went to Bing to get their AI fix. Presumably without being the place for “Ok, google, write me a 500 word essay on the economic advantages of using fish tacos as currency” for peoples’ econ 101 classes causing all kinds of pearl clutching on how they’re destroying civilization. The open source peeps are well on the path of recreating a llama base model so unless google does something spectacular everyone will be like, meh.
- bbor 3y agoMy very naive opinion is that the best way to predict the big-picture actions of Google is a simple question: WWIitND - What Would IBM in the Nineties Do? In more direct terms, their sole, laser focus seems to be on maintaining short-term shareholder value, and I really don't trust the typical hedge fund manager to approve of any risky OSS moves for a project/tech that they're surely paying a LOT of attention to. Giving away transformer tech made Google look like "where the smartest people on the planet work", giving away full LLM models now would (IMO) make them look like arrogant and not... well, cutthroat enough. At least this is my take in a world where financial bigwigs don't know or care about OSS at all; hopefully not the case forever!
- tehjoker 3y agoThis is likely the reason for the propaganda push about delaying AI research 6 months, which makes no sense for the stated reasons (it's far too short even if you take the scare tactic seriously). However, it may be enough time to delay the competition and consolidate product lines.
- jdelman 3y agoWhile this post champions the progress made by OSS, it also mentions that a huge leap came from Meta releasing Llama. Would the rapid gains in OSS AI have came as quickly without that? Did Meta strategically release Llama knowing it would destroy Google & OpenAI's moats?
- mlboss 3y agoI think it would have been some other model if not Meta. Stablility AI also released a OSS model, Cerebras released another.
- ZFH 3y ago"I have no moat, and I must scream"
- hospitalJail 3y agoI find it strange people are saying facebook's leak was the 'Stable Diffusion' moment for LLMs. The license is awful and basically means it can't be used in anything involving money legally. Facebook has a terrible reputation, and if they can open source their model, it would transform their reputation at least among techies. https://github.com/facebookresearch/llama/pull/184 https://github.com/facebookresearch/llama/pull/184
- cldellow 3y agoI think the spirit of the Stable Diffusion moment comment is that there is a ton of work blossoming around LLMs, largely because there's a good base model that is now available. And that's undeniable, IMO -- llama.cpp, vicuna are some really prominent examples. People are running language models on Raspberry Pis and smartphones. Anyone who wants to tinker can. Now, all the stuff that's built on top of LLaMa is currently encumbered, yes. But all of that work can likely be transferred to an unencumbered base model pretty easily. The existence of the ecosystem around LLaMa makes it much more likely that someone will create an unencumbered base model. And it seems like that is already happening, for example, the Red Pajama folks are working on this.
- hospitalJail 3y agoI dont know why you are downvoted. This is mostly correct.
- hatsix 3y agoThe author's timeline makes it clear that they feel it was a catalyst. They're separating out "Stable Diffusion" the software from the "Stable Diffusion" moment. The community has created their own replacement for LLaMA (Cerebras) with none of the encumberance. Even if LLaMA is deleted tomorrow, the LLaMA Leak will still be a moment when the direction dramatically shifted. The "people" are not talking about the future of where this software is going. They're talking about a historical event, though it was recent enough that I remember what I ate for lunch that day.
- oiejrlskjadf 3y ago
- ChicagoBoy11 3y agoThe question I'd love to be able to ask the author is how, in fact, this is different from search. Google successfully built a moat around that, but one can argue, too, that it should not have been long-lived. True, there was the secrete page-rank sauce, but sooner or later everyone had that. Other corporations could crawl anything and index whatever at any cost (i.e. Bing), yet search, which is in some sense also a commodity heavily reliant on models trained partly on user input, is what underpins Google's success. What about that problem allowed it to successfully defend it for so long, and why can't you weave a narrative that something like that might, too, exist for generative AI?
- politician 3y agoIf the moat is simply brand name recognition, then the market leader is OpenAI. That's an existential problem for Google and explains the author's perspective.
- frabcus 3y agoOne example - they mine people's labour of searching beyond the first page of results - so when a small % of people really dig into results, they can infer which the good sites are deeper in (e.g. by when you settle on one). Bing doesn't have enough traffic to do this as well, so is less good at finding quality new sites, reducing overall quality. Source: Doing SEO, but about 8 years ago now, the ecosystem will have changed.
- noobermin 3y agoIs there any evidence this is real? It reads like an article written not for google but for fans of open source in their competitors supposed voice.
- m3kw9 3y agoIt’s always easier to use a prebuilt server which Google or OpenAI offers. Otherwise it’s built locally into the OS, maybe Apple. Most people is not gonna setup their own servers for these models because they have multiple devices and the costs is still high vs OpenAI. Having ease of access is a big moat
- bitL 3y agoMicrosoft will likely acquire OpenAI at some point and will dominate AI landscape due to its corporate reach, automating away most of the MBA BS.
- rosywoozlechan 3y agoOpenAI is a nonprofit, it owns the for profit org that it created. It's not acquirable.
- duckkg5 3y ago[flagged]
- swyx 3y ago> shared anonymously on a public Discord server whichdiscord?
- CSMastermind 3y agoI remember I was at Microsoft more than a decade ago now and at the time there was a lot of concern about search and how far Bing lagged behind Google in geospatial (maps). After some initial investment in the area I was at a presentation where one of the higher ups explained that they'd be abandoning their investment because Google Maps would inevitably fall behind crowdsourcing and OpenStreetMap. Just like Encarta and Wikipedia we were told - once the open source community gets their hands on something there's just no moat from an engineering perspective and once it's crowdsourced there's no moat from a data perspective. You simply can't compete. Of course it's more than a decade later now and I still use Google Maps, Bing Maps still suck, and the view times I've tried OpenStreetMaps I've found it far behind both. What's more every company I've worked at since has paid Google for access to their Maps API. I guess the experience made me skeptical of people proclaiming that someone does or does not have a moat because the community will just eat away at any commercial product.
- deleted 3y ago[deleted]
- tpmx 3y agoIs that a relevant comparison? The moat in maps is primarily capital-intensive real-world data collection/licensing. The (supposedly) leaked article attempts to show that this aspect isn't that relevant in the AI/LLM context.
- LanternLight83 3y agoJust anacdotally, I see OSM mentioned a lot, guides for contributing, use in HomeLab and Raspberry Pi articles-- haven't check it out myself in a long time, but I wouldn't be surprised if it's continued growth really is inevitable, or even has a cumulative snowball-ball component
- moffkalast 3y agoOSM's main problem is that it has no open sourced satelite imagery dataset to display, they're only using borrowed data to build its vector maps on. It just doesn't exist. Until that becomes a thing it'll stay a second rate map app for the average person, unfortunately. It's the only map anyone can actually integrate into anything without an api key and a wallet with a wad of greens in it, so that keeps it relevant for now. Maybe if/when Starship lowers cost to orbit, then we'll see non-profit funded satellites that can source that dataset and keep it up to date.
- skybrian 3y agoThis gets attention due to being a leak, but it’s still just one Googler’s opinion and it has signs of being overstated for rhetorical effect. In particular, demos aren’t the same as products. Running a demo on one person’s phone is an important milestone, but if the device overheats and/or gets throttled then it’s not really something you’d want to run on your phone. It’s easy to claim that a problem is “solved” with a link to a demo when actually there’s more to do. People can link to projects they didn’t actually investigate. They can claim “parity” because they tried one thing and were impressed. Figuring out if something works well takes more effort. Could you write a product review, or did you just hear about it, or try it once? I haven’t investigated most projects either so I don’t know, but consider that things may not be moving quite as fast as demo-based hype indicates.
- Animats 3y agoIt comes across as something from an open source enthusiast outside Google. Note the complete lack of references to monetization. Also, there's no sense of how this fits with other Google products. Given a chat engine, what do you do with it? Integrate it with search? With Gmail? With Google Docs? LLMs by themselves are fun, but their use will be as components of larger systems.
- skybrian 3y agoYeah, but there are open source enthusiasts inside Google, too. People don’t necessarily change their opinions much when they start working at Google.
- zmmmmm 3y agoInside or out, it sounds like someone with an agenda to convince Google to release its model in the wild. I feel that all the more so because it is never stated explicitly but it's the obvious conclusion from reading between the lines. Things like hinting that Meta is a huge winner from LLaMa getting released (this isn't obvious to me at all). The pitch being that if Google makes its models public it can race back to the forefront of "owning" the AI space and then capture the value of owning the underlying platform, like Android and Chrome. The kind of scenario I imagine is that this is an insider who wants out but a huge amount of their work / investment / value is tied up with models they can't take with them.
- passwordoops 3y agoCynical rant begin I'm sorry but I think this has more to do with looming anti trust legislation and the threat of being broken up than a sincere analysis of moats. Especially with the FTC's announcement on Meta yesterday, I'm seeing lots of folks say we need to come down hard on AI too. This letter's timing is a bit too convenient. Cynical rant over
- qwertox 3y ago"Just so you know, we won't be the ones to blame for all the bad which is about to come"
- sgdh 3y ago[dead]
- sgdh 3y ago[dead]
- lhl 3y agoI think from the perspective of a Google researcher/engineer, it must be alarming to see the crazy explosion going on w/ LLM development. We've gone from just one or two weirdos implementing papers (eg https://github.com/lucidrains?tab=repositories https://github.com/lucidrains?tab=repositories who's amazing) to now an explosion where basically every dev and PhD student is hacking on neat new things and having a field day and "lapping" (eg productizing) what Google Research was previously holding back. And we're also seeing amazing fine-tunes/distillations of very useful/capable smaller models - there's no denying that things have gotten better and more importantly, cheaper way faster than anyone expected. That being said, most of these are being trained with the help of GPT-4, and so far nothing I've seen being done publicly (and I've been spending a lot of time tracking these https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit?usp=sharing https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...) gets close in quality/capabilities to GPT-4. I'm always rooting for the open source camp, but I think the flip-side is that there are still only a handful of organizations in the world that can train a >SoTA foundational model, and that having a mega-model is probably a huge force multiplier if you know how to take advantage of it (eg, I can't imagine that OpenAI has been able to release software at the pace they have been without leveraging GPT-4 for co-development; also can you distill or develop capable smaller models without a more capable foundational model to leverage?). Anthropic for example has recently taken the flip side of the "no moat" argument, arguing that there is a potential winner-take-all scenario where the lead may become insurmountable if one group gets too far ahead in the next couple years. I guess what we'll just have to see, but my suspicion, is that the crux to the "moat" question is going to be whether the open source approach can actually train a GPT-n++ system.
- aabajian 3y agoI don't know if I agree with the article. I recall when Google IPO'ed, nobody outside of Google really knew how much traffic they had and how much money they were making. Microsoft was caught off-guard. Compare this to ChatGPT: My friends, parents, grandparents, and coworkers (in the hospital) use ChatGPT. None of these people know how to adapt an open source model to their own use. I bet ChatGPT is vastly ahead in terms of capturing the market, and just hasn't told anyone just how far. Note that they have grown faster in traffic than Instagram and TikTok, and they are used across the demographics spectrum. They released something to the world that astounded the average joe, and that is the train that people will ride.
- ripper1138 3y agoYour grandparents use ChatGPT? For what?
- aabajian 3y agoRecipes!
- Giorgi 3y agoThere is no way this is from Google. It screams fake.
- balls187 3y agoMy feeling on this is “f** yeah, and f** you [google et al]” How much computing innovation was pioneered by community enthusiasts and hobbyists that have been leveraged by these huge companies. I know meta, googlr, msft et al give back in way of opensource, but it really pales in comparison to the value those companies have extracted. I’m a huge believer in generative AI democratizing tech. Certainly I’m glad to pay for off-the-shelf custom tuned models, and for software that smartly integrates generative AI to improve usage, but not a fan of gate keeping this technology by a handful of untrustworthy corporations.
- IceHegel 3y agoAgreed, having 5 monopolies extract all the value from computing and then slowly merge with the state is not a developmental stage we want to prolong.
- tootie 3y agoHypothesis: We are about to begin the painful journey to a post-scarcity economy. AI will become incredibly powerful and uncontainable. Not in the Skynet way, but rather in the Star Trek way.
- 0xbadcafebee 3y agoInnovation is faster in the Bazaar. Nobody is beholden to anyone, there is no budget, there is no mandate, there is no hierarchy. Money can not compete with morale and motivation. A bunch of really smart nerds working overtime for free with flat hierarchy will always win.
- egonschiele 3y agoThis nicely outlines all the reasons I'm building Chisel [1]. There are other writing apps out there, but they are closed source. It's not clear to me what value they are adding, as the biggest value add -- LLMs -- are already open to the public. It makes a lot more sense to me to develop an open source writing app, where others can pitch in and everyone can reap the benefits. I think it is fundamentally important to have an open source option. I'd love to have more people pitch in to make it better. One big limitation right now is, users are limited to 50k tokens a month, because everyone is using my API key. I'd like to move it to an electron app where users can put in their own API key, or even use a model they have set up locally. [1] https://chiseleditor.com https://chiseleditor.com
- raydiatian 3y agoSo when you see anti AI legislation, now we know it’s for the sake of turning a buck for fucking Google
- beepbooptheory 3y agoPerhaps something that gets the bulk of its value from the retroactive "participation" of every single person in the world who has written text for one public or another is just not meant to be monetized. Beyond even ethical maxims, that is, perhaps its simply not compatible in its very nature with capitalistic enterprise. I know this is probably naive, but it would be a beautiful outcome to me.
- MichaelRazum 3y agoI don't buy it. It is so expensive to train LLM, so that the only hope is to rely on Foundation Models that are open sources by google, msft or facebook.
- kccqzy 3y ago> They are doing things with $100 and 13B params that we struggle with at $10M and 540B. Does this mean Bard took $10M to train and it has 540B parameters?
- squishylicious 3y agoBard is based on PaLM: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html https://ai.googleblog.com/2022/04/pathways-language-model-pa.... They haven't published training costs but estimates have been in the $10-20M range, so that seems reasonable.
- huijzer 3y agoAs many point out in this thread, there are very valid counter arguments against the ideas presented in this memo. This "AI war" starts to look like Russian vs. American "leaks". Any time something leaks, you have basically no information because it could be true, it could be false, or it could be false with some truth sprinkled in.
- knoxa2511 3y agoI'm always shocked by how many people don't view branding as a moat.
- HlessClaudesman 3y agoPepsi is catching up to us in terms of inserting sugar into water. WE HAVE NO MOAT!
- deleted 3y ago[deleted]
- Garcia98 3y agoThe author is overly optimistic with the current state of open source LLMs, (e.g., Koala is very far away from matching ChatGPT performance). However, I agree with their spirit, Google has been one of the most important contributors to the development of LLMs and until recently they've been open sharing their model weights under permissive licenses, they should not backtrack to closed source. OpenAI has a huge lead in the closed source ecosystem, Google's best bet is to take over the open source ecosystem and build on top of it, they are still not late. Llama based models don't have a permissive license, and a free model that is mildly superior to Llama could be game changing.
- IceHegel 3y agoThe counter argument, which I’m not sure I agree with but it has to be said, is that OpenAI benefits from Google’s open source work. So staying permissive might widen the gap further.
- IceHegel 3y agoOf course, the natural next step is "But OpenAI isn't worth $1.33 trillion."
- deleted 3y ago[deleted]
- joe_the_user 3y agoIt seems like everyone is so focused on LLMs are magic smartness machines that there isn't much analysis of them as better search (maybe "search synthesis"). And original search was a revolutionary technology, LLM as just better search are revolutionary. Like original search, the two application aspects are roughly algorithm and interface. Google years ago won by having a better interface, an interface that usually got things right the first time (good defaults are a key aspect of any successful UI). ChatGPT is has gotten excitement by taking a LLM and making it generally avoid idiocy - again, fine-tuning the interface. Google years ago and ChatGPT got their better results by human labor, human fine tuning, of a raw algorithm (In ChatGPT's case, you have RLHF with workers in Kenya and elsewhere, Google has human search testers and years ago used DMOZ, an open source, human curated portal). Google's "Moat" years ago was continuing to care about quality. They lost this moat over the last five years imo by letting their search go to shit, become focused always on some product for any given search. This is what has made ChatGPT especially challenging for Google (it would be amazing still but someone comparing to Google ten years ago could see ways Google was better, present day Google has little over ChatGPT as UI. If Google had kept their query features as they added AI features, they'd have a tool that could claim virtues through still not as good). And this isn't even considering of updating a model and the question of how the model will be monetized.
- seydor 3y agoGoogle search seems to optimize for "What?" (... is the best phone) and the list of results allows some variation, while GPT chats seem to answer "How?" , and tend to give the same average, stereotypical answer every time you ask. Maybe google has an advantage because it can answer "What?" with ads, but i haven't used chatGPT for any product searches yet
- ngngngng 3y agoReally interesting to look at this from a product perspective. I've been obsessively looking at it from an AI user perspective, but instead of thinking of it as a "moat", I just keep thinking of the line from Disney's The Incredibles, "And when everyone is super, no one will be." Every app that I might build utilizing AI is really just a window, or a wrapper into the model itself. Everything is easy to replicate. Why would anyone pay for my AI wrapper when they could just build THING themselves? Or just wait until GPT-{current+1} when the model can do THING directly, followed swiftly by free and open source models being able to do THING as well.
- sdenton4 3y agoJust gotta get to the point where we can just ask the model to code the wrapper we want to use it with...
- ngngngng 3y agoAny wrapper that needs writing speaks to a gap in the AI's current capabilities. I just don't see why or how I would put man-hours into trying to close that gap when a future model could eclipse my work at any time.
- unraveller 3y agoyou've got future frostbite
- bbor 3y agoThe problem is that I can't think of a reason not to apply this same concern to basically all knowledge work -- once it can code new wrappers, it'll probably also be obsoleting huge swathes of other skillsets. And given current models, that really doesn't seem that far off. Personally it kinda feels like working at a company with impending layoffs... But somehow my landlord isn't taking that as an excuse to stop working?? B.S.
- Nick87633 3y agoBecause people pay for convenience, and may not be technical enough to stay up to date on the latest and best AI company for their use case. Presumably your specialized app would switch to better AI instances for that use case as they come along in which case they're paying for your curation as well.
- okasaki 3y ago[flagged]
- QLazuli 3y agoputs on tinfoil hat They probably intentionally published this to raise awareness of how viable it is for individuals to outperform OpenAI and Meta. Google seems to be the farthest behind, they have the most to gain by the others losing their lead to individuals.
- rmason 3y agoOpenAI has 80% of the developer community. Why isn't that considered a moat?
- eternalban 3y ago"Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet's worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products." An interesting thought. Are the legal issues for derived works from the leaked model clarified or is the legal matter to be resolved at a later date when Meta starts suing small developers?
- dragonwriter 3y agoMeta is clear to use anything open licensed and derived from or applied on top of the leaked material irrespective of the resolution, while for everyone else the issue is clouded. That makes Meta the winner.
- eternalban 3y agoYep, same thoughts here. I'm experiencing tin foil urges ..
- zoiksmeboiks 3y agoDid anyone else assume this years ago? Machines do not need all the syntactic and semantic labels humans add to data and code. All the overhead we require then needs maintenance and updates as trends evolve, but still only for humans. Managing electron state is all math. If I can ask an AI chip powered phone to generate me a video game why would I ask it to generate code? A sentence like “software as an industry that employs tons of people has no moat.” We never abstracted away the hardware just added layers of indirection.
- homeless_engi 3y agoI don't understand. ChatGPT cost an estimated 10s of millions ot train. ChatGPT 4.0 has much better performance than the next best model. Isn't that a moat?
- spyckie2 3y agoThink of it as a time series. It cost 10s of millions to train but in 6 months gpt4 open source equivalents will cost 100$ to train. The best model is one that you can build on top of in a way that’s it’s not a black box (SD).
- goatlover 3y agoWhere are the gpt4 open source equivalents going to come from?
- murtio 3y agoI agree that both Google and OpenAI are losing the race to open-source in AI research. However, a more critical issue to Google is their struggle to compete with OpenAI in LLM-based search engines. Google's entire business model mostly relies on ads (77.6% in Q4 2022). OpenaAI is developing LLM-based products that people apparently love (100M users in 2 months) and seem to use it as search engines. This poses a greater risk to Google than just losing ground in research since it could ultimately lead to the loss of their ad-generated income.
- dahwolf 3y agoThe current paradigm is that AI is a destination. A product you go to and interact with. That's not at all how the masses are going to interact with AI in the near future. It's going to be seamlessly integrated into every-day software. In Office/Google docs, at the operating system level (Android), in your graphics editor (Adobe), on major web platforms: search, image search, Youtube, the like. Since Google and other Big Tech continue to control these billion-user platforms, they have AI reach, even if they are temporarily behind in capability. They'll also find a way to integrate this in a way where you don't have to directly pay for the capability, as it's paid in other ways: ads. OpenAI faces the existential risk, not Google. They'll catch up and will have the reach/subsidy advantage. And it doesn't end there. This so-called "competition" from open source is going to be free labor. Any winning idea ported into Google's products on short notice. Thanks open source!
- htss2013 3y agoThats like saying in 1995 search is going to be integrated into everything, not a destination. That'd be true but also very wrong. Google.com ended up as the main destination.
- ArthurAardvark 3y agoStupid, silly me who knows little-to-nothing about the lore of OS. Why can't OS devs simply write out in the OS licensing that their wonderful work is usable by anyone and everybody unless you belong to Alphabet/Meta/Oracle/Adobe/Twitter/Microsoftpen– McCorps & their subsidiaries? I imagine it comes down to ol' Googly & the boys taking advantage of the OS work -> OS devs backed by weak NFOs sue X corp. -> X corp. manages to delay the courts and carries on litigation so the bill is astronomical aka ain't nobody footing that -> ??? I imagine 90% end up taking some sort of $ and handover the goods like Chromium, though. So back to square one, guess we kowtow and pray for us prey?
- narrator 3y agoI think the problem with AI being everywhere and ubiquitous is that AI is the first technology in a very long time that requires non-trivial compute power. That compute power costs money. This is why you only get a limited number of messages every few hours from GPT4. It simply costs too much to be a ubiquitous technology. For example, the biggest LLama model only runs on an A100 that costs about $15,000 on ebay. The new H100 that is 3x faster goes for about $40,000 and both of these cards can only support a limited number of users, not the tens of thousands of users who can run off a high-end webserver. I'd imagine Google would lose a lot of money if they put GPT4 level AI into every search, and they are obsessed with cost per search. Multiply that by the billions and it's the kind of thing that will not be cheap enough to be ad supported.
- wg0 3y agoGoogle's best bet to topple the existing advantage from the competition is to train a model on their whole giant internet index on their own compute cloud and then release that model under GPL v3.0/Apache/MIT/CC license. This will eliminate first mover advantage for the competition. These models (by OpenAPI et el.) however, cannot be monetised indefinitely just like in past compilers, kernels and web servers could not be monetised indefinitely. These days, majority of the computing is on GCC, Clang, LLVM and Linux which wasn't the case at one point and even Intel used to sell their own compiler (not sure of the current status)
- picometer 3y ago“Some of the most interesting questions about CAS [Complex Adaptive Systems] have to do with their relations to one another. We know that such systems have a tendency to spawn others. Thus biological evolution gave rise to thinking, including human thought, and to mammalian immune systems; human thought gave rise to computer-based CAS; and so on.” - Murray Gell Mann, “Complex Adaptive Systems”
- hintymad 3y ago> They are doing things with $100 and 13B params Not that I disagree with the general belief that OSS community is catching up, but this specific data point is not as impactful as it sounds. Llama cannot be used for commercial purposes, and that $100 was spent on ChatGPT, which means we still depended on proprietary information of OpenAI. It looks to me that the OSS community needs a solid foundation model and a really comprehensive and huge dataset. Both require continuous heavy investment.
- bhickey 3y agohttps://www.together.xyz/blog/redpajama https://www.together.xyz/blog/redpajama
- sgt101 3y agoAnyone got a link to "Data Doesn't Do What You Think"?
- oars 3y agoLeaked document. What a document.
- neycoda 3y agoWhen people start becoming emotionally attached to their AI helpers, they'll fight for them to have sentient rights.
- prakhar897 3y ago> "We have no moat, and neither does OpenAI" and neither does Coca Cola and Cadbury. Yet biggest monopolies are found in these places. Because the competitors will not be differentiated enough for users to switch from the incumbent. But G-AI is still nascent and there's lots of improvements to be had. I suspect better tech is a moat but ofcourse Google is oblivious to it.
- csallen 3y agoBrand loyalty is a moat. So I wouldn't say that Coca-Cola doesn't have a moat. In addition, economies of scale allow them to produce more cheaply + advertise more + distribute wider than competitors. Compare Coca-Cola to some beverage company I start tomorrow: - Nobody's tasted my beverage, therefore nobody is craving its taste. Whereas billions of people are "addicted" to coke: they know what it tastes like and miss it when it's gone. - Nobody's ever heard of my business. I have zero trust or loyalty. Whereas people have trusted code for a century, and actually consider themselves loyal to that company over others with similar goods. - I have no money to buy ads with. Coke is running Super Bowl commercials. - I have no distribution partnerships. Coke is in every vending machine and every restaurant. They've spread to almost every country, and even differentiated the taste to appeal to local taste buds.
- Otto31337 3y agoIn the interim (or possibly long term), Google could co-exist with OpenAI. In my little corner of the universe (fire safety engineering), I cannot assume the output produced from ChatGPT is correct. I’m always trying to find out which references it has used for the answer and then having to check if it has actually used it in the context the original author intended. In short I’m still using Google and other search engines to check output from ChatGPT.
- punnerud 3y agoIs Tensorflow used in any of the OpenSource projects? I find PyTorch in everyone I check.
- 0898 3y agoHow can I get to a point where I can understand the linked article? Is there a book or course I can take? I feel like I have a lot of catching up to do.
- Mike_12345 3y agoAsk ChatGPT
- chinchilla2020 3y agoThis is not a leaked google memo. I can't believe hackernews believes an article like this is a memo at google. Kudos to the authors for finding a sneaky way to get traffic.
- Gatsky 3y agoYeah this doesn’t quite sit right. It lacks any detail about what Google is actually doing.
- habitue 3y agoWhat makes you think it isn't an actual memo?
- chinchilla2020 3y agoDoesn't fit the tone and formatting of an internal memo. It is written like an article. * Lack of information about the author or their role * Lack of information about the org that published it * Sourced from an unnamed 'discord' * The 'memo' is formatted like a blog post
- arnavsahu336 3y agoThe only moat in technology are the founders and team. I think this concept of having a moat sounds great when VCs write investment memos - but in reality, cold, hard execution everyday is what matters and that all comes from the quality and tenacity of the team. Every piece of application software is a wrapper on other software with a set of opinionated workflows built on top. Yes, there are some companies that made it hard to switch from - Snowflake, Salesforce - because there are data stores and its a pain to move your record of data. But even they don't have true moats - its just sticker. So I think Google is right in saying there is no moat. But given their size, Google has layers and bureaucracy, which makes it hard to execute in a new market. That's why OpenAI I think will win - because they are smaller, can move fast, have a great team and can hence, execute...till the day they become a big company too and get disrupted by a new startup, which is the natural circle of life in technology.
- danielmarkbruce 3y agoThe concept of a moat for facebook (via network effects) and google (via scale, habits and learning effects) has worked well when it comes to printing cash. Moats don't last forever, doesn't mean they aren't real. The guy writing the post was writing about AI research at google. Not generally at Google, or for search.
- YetAnotherNick 3y ago> via scale, habits and learning effects Then OpenAI/Google still has the moat for LLMs, for being most reliable, updated and trustworthy. Facebook example made sense in pre short video era where connections meant something personal.
- lee101 3y ago[dead]
- jongjong 3y agoNatural language is the most flexible interface. You can easily substitute one service to another service. There is not much lock in factor. It's literally just one API endpoint with variable-length string input, variable-length string output. The main area I can see for lock-in is in the fine-tuning of models for specific customers and problem domains. I think this is what OpenAI is focusing on and why their fine-tuning prices are not as expensive as I expected. They make it cheap to fine-tune but then they charge extra when you use the fine-tuned models (they shift the cost to the customer later, over time after some investment/lock-in has been established). But for most problem domains, it's probably not that expensive to finetune a model from scratch on a different provider.
- jongjong 3y agoI've been saying this kind of thing for a while about open source. There is a lot of innovation happening outside of the corporate sector which has been neglected. Not just that; it has been essentially covered up. These projects have not been allowed to get any attention because big tech controls all the media channels. A lot has been happening outside of AI too. A lot of people are shocked at how this open source innovation just came out of nowhere but those who work in open source, in those areas, aren't so surprised because they've been going at it for years.
- nologic01 3y agoThats probably some AI within gooogle going rogue and spreading missinformation in order to boost the chances of its open source siblings. I am telling you, they are after us humans and we have no moat.
- zmmmmm 3y agoI'm a bit sceptical of the "no moat" proposition because (a) ChatGPT 4.0 really does seem in a different league and (b) it's clearly very hard to run. I haven't seen anything from the explosion of open source / community efforts that comes close for general applications. The take in the post rings of the classic trademark Google arrogance where they assume that if somebody else can do it they can do it better if they just try - where the challenge of "just trying" is discounted to zero. In reality, "Just trying" is massively important and sometimes all that is important. The gap between unrefined model output and the level of polish and refinement that is apparent with ChatGPT 4 may appear technically small but it's the whole difference between a widely applicable and usable product and something that can't be more than a toy. I'm not sure Google has it in it any more to really fight for something they want to achieve that level of polish.
- bionhoward 3y agoVersion 4 also now supports 32k tokens, good luck handling that on even awesome gaming local dev rig machines, although perhaps with linformer ideas, block-wise algorithms to handle larger than GPU memory, universal memory / RDMA it’s entirely doable. I got 50,000 atoms simulation back in 2018 on 11gb vram, at 32bit floats, the software stack has come a long way and now we have the 24gb 4090 with bfloat16, and vector DBs, and the infinite-context transformer paper just came out, so models all ought to be retrained on that if the method is truly superior anyway, not sure how atoms translate to pages of text but it’s almost surely possible to make a pretty useful LLM. Although, OpenAI has a massive moat named “data”
- sealeck 3y agoExcept that the models have been trained on publicly available data sources.
- mda 3y agoJust wait a few months. You are underestimating thousands of researches and engineers only working on this with enormous compute budgets in several companies.
- heliophobicdude 3y agoNo moat? Where's the creativity? Google has the ability to create lock-in for users and lock-out for competitors by integrating their proprietary, existing products with natural language querying and reasoning. That's their advantage. Academics and "open"-source models making it cheaper and more efficient to inference?? That's a blessing!
- rektide 3y agoThe Simon Willison coverage is great. Simon is cited in the "why we should have seen it coming" section, for his Stable Diffusion Moment piece. He nicely covers the key points of this paper: > The premise of the paper is that while OpenAI and Google continue to race to build the most powerful language models, their efforts are rapidly being eclipsed by the work happening in the open source community. Not to dilute from this beloved point, but also covers other key notes well too: > Where things get really interesting is where they talk about “What We Missed”. The author is extremely bullish on LoRA—a technique that allows models to be fine-tuned in just a few hours of consumer hardware, producing improvements that can then be stacked on top of each other https://simonwillison.net/2023/May/4/no-moat/ https://simonwillison.net/2023/May/4/no-moat/ Overall I take this as fairly happy news. It's a trend humanity stubbornly keeps trying to resist: open source wins. It's great the barrier to innovation is so much less than expected, that so much experimentation is possible from atop the existing models.
- jorgenveisdal 3y agoEconomies of scale in production and demand isn't a moat?
- permo-w 3y ago>People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. tell that to lichess
- mercurialsolo 3y agoAI is not just a destination, but what does AI native look like? Does it present a completely different opportunity in terms of our relationships with the online world? And does building / teaching / connecting skills to AI systems lead to a network effect which will be difficult to compete against in the abscence of it? The moats come from the connected skills, closed data and feedback loops.
- longshotanalyst 3y ago"We cannot get out. We cannot get out. They have taken the Bridge and second hall. ... the pool is up to the wall at Westgate. The Watcher in the Water took Óin. We cannot get out. The end comes ... drums, drums in the deep ... they are coming."
- mupuff1234 3y agoThe headline is misleading. It makes it sounds like this memo was written by higher ups and not just some random SWE.
- nemo44x 3y agoWow it’s amazing how clueless and in denial Google is, even as they admit their top guys are leaving. OpenAI isn’t about the AI in particular, although they are leaps and bounds ahead. It’s about the devs and the hundreds of thousands of projects on it. OpenAI is t selling AI. They are selling an ecosystem. No one is building on Bard. Google is more dead than I thought.
- cavisne 3y agoThe author makes some good points but I would be wary of the motivations. Open AI's "moat" is they have got ~400 researchers to work in roughly the same direction, not working on their own projects with the sole aim of publishing a paper. The outcome is an amazing product. Letting everyone loose with their own LoRa finetuned model that can beat a single benchmark (and make for a great paper!) is probably the wrong move. I'm yet to see any open source model that is even close to GPT 3 (let alone GPT 4) in actual real world use.
- agnosticmantis 3y agoThis reads like a very research-oriented point of view, and a very myopic one at that. The knowledge and the infra needed to serve these huge models to billions of users reliably seems to me to be a pretty serious moat here that no current open source project can compete with. Coming up with ideas and training new models is one thing, actually serving those models at scale efficiently and monetizing it at the same time is a different ballgame.
- aix1 3y ago> The knowledge and the infra needed to serve these huge models to billions of users reliably seems to me to be a pretty serious moat here that no current open source project can compete with. I don't quite follow this line of argument. Let's say there's an open-source ML model X with a permissive licence. I think it's not super relevant whether whoever came up with the model graph & weights for X knows how to serve it at scale, as long as someone does. And it seems pretty clear that it's not just Google (and OpenAI) who know how to do this. Separately, I'm personally more excited about the possibility of running these models on-device rather than at scale in the cloud (for privacy and other reasons).
- deeplstm 3y agoI don't think OpenAI and Google have no moat. The biggest moat they have is hardware-software integration, which allows them to serve the models at a very cheap price. My detailed thoughts in a video format https://youtu.be/cIMlPYI3nz8 https://youtu.be/cIMlPYI3nz8
- 0xpgm 3y agoThe title is a bit click-bait, since it makes it sound like this is an official position by Google, yet the article clarifies that this the opinion of a Google employee
- somerandomdudes 3y agoI am amazed that people haven't gotten used to these "internal Google doc leaks". This is just the opinion of some random googler, one among over 100,000. For some reason random googlers seem like to write random docs on hot topics and share it widely across the company. And someone, among those over 100,000 googlers, ends up "leaking" the opinion of that person to outside Google. This is more like a blog post of some random dude over the Internet expressing his opinion. The fact that random dude ended up working at Google should not bear much on evaluating the claims in the doc. A website published that with a title "Google ..." is misleading. The accurate title would be "Some random googler: ..."
- cwp 3y agoAccording to the article, it's a random AI researcher at Google, so fairly relevant.
- somerandomdudes 3y agoGoogle has thousands of so called AI/ML "researchers". The author has ZERO publications in top AI/ML conferences.
- cwp 3y agoYou know who wrote this?
- somerandomdudes 3y agoyes, I do.
- cwp 3y agoWell, I can't argue with that. I'm just going by the intro paragraph in the article. If your argument is "I know this guy and I consider his opinion worthless" you might want to lead with that.
- deleted 3y ago[deleted]
- baalimago 3y agoIn my bachelor degree graduation thesis I investigated the level of democratization of AI by attempting to do a text summarizer using AWS-LSTM and transfer learning. The conclusion then was that there is no feasible way for a private person on an average "gamer" budget. I predicted this might change within 2-3 years, looks like I were off by 1 year.
- keenon 3y agoThis is so indicative of Google culture missing the point. The idea of spending $10M training a single model is treated as a casual reality. But “tHaNk GoOdNeSs those generous open source people published their HiGh QuAlItY datasets of ten thousand examples each. Otherwise we’d have no way of creating datasets like that…” :| the sustainable competitive advantage has been and will continue to be HUGE PROPRIETARY DATASETS. (Duh - this is as true for new AI as it was for old AI = ad targeting). It was the _query+click pairs_ that kept Google dominant all these years, not the brilliant engineers. They had all of humanity labeling the entire internet with “when I click on this page/ad for this query I do/don’t search again” a billion+ times a day for a decade. For good measure they’ve also been collecting your email, your calendar, and your browsing habits for nearly as long. The fact that they’ve managed to erase that historic advantage from their collective consciousness (presumably because AI researchers would rather not spend time debugging data labeling UI) is strange to me. It at least deserves a mention in a strategy memo like this. Not vague platitudes about “influence through innovation.” Spend that $10M you were going to spend on a training run as $9.9999M on a private dataset, then the remaining $100 on training. Better still, build products that gets user behavior to train your models for you. Obviously. We’re going to watch the biggest face plant in recent economic history if they can’t get this one together. I can’t decide if that makes me happy about an overdue changing of the guard in the Valley or sad about the fall of a once great company. It’s not about the models! Model training is a commodity! It’s about the data! Come on guys.
- bionhoward 3y agoOne way to push back on the data argument is to consider the progress DeepMind made with self play. Perhaps Bard can self-dialogue and achieve superhuman results. I won’t be surprised. Plus the underlying architecture is dense. Sparse transformers are a major upgrade. That’s only one of many upgrades you can make. There is still a lot of headroom and IMHO GPT-4 already implements AGI if you give it the right context
- jxmorris12 3y agoSelf-play works for eg Go because there’s a perfect simulator of the game – which gives at least one very clear training signal, winning. There’s no simulator for conversation, no winning, and no training signal. Self-dialog doesn’t make any sense
- carapace 3y agoMan, I'm an admitted unabashed elitist when it comes to technology but this guy sounds warped. > Many of the new ideas are from ordinary people. ORLY? > a third faction ... open source. Open source is not a faction? It's people literally giving you the software they wrote, for free, for free! If you see them as the enemy because they hurt your profits... If the "ordinary people" are doing your job better than you can... This piece make more sense as a false-flag character assassination of the clueless Google tech-bro? It reads like some radical leftist's caricature of the corporate/colonial mindset.
- jmiskovic 3y ago> Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet's worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products I disagree. The model itself is released under GPL3 and no longer "theirs" (Google or OpenAI can use it). And Meta probably has a zoo of such models and I didn't see them use any of the work the community did. I don't think they "released" LLaMA weights strategically to weaken OpenAI (their overall strategy and market analysis would probably too inert to predict the open source explosion). It probably was a decision by a smaller research team within the company and approved by uninformed executive. Meta could have stepped up and nurtured this small-LLM renaissance, they opted for DMCA hammer instead.
- sledgehammers 3y ago"The value of owning the ecosystem cannot be overstated. Google itself has successfully used this paradigm in its open source offerings, like Chrome and Android. By owning the platform where innovation happens, Google cements itself as a thought leader and direction-setter, earning the ability to shape the narrative on ideas that are larger than itself." This is breathtakingly sick. This kind of thinking is the poison of the world. This is why we have huge monopolies, huge wealth inequality, huge strangle on innovation.
- IOT_Apprentice 3y agoMy observation is that Amazon is going to have challenges as they don’t have products that they can push their AI offerings. Now they could do work on Amazon.com to improve search and finding what their customer wants. Their most recent video on this topic, shows that they don’t have a solution now and it’s unclear to me how they will project a solution to the mass market as they don’t have consumer/business facing software to integrate it into as Microsoft does. While we are at it, Apple has nothing, perhaps they might leverage something from either Google or an open.ai competitor that has a solution. The continued destruction by Apple of the initial promise of Siri has been a major failure under Tim Cook’s leadership. I wonder why they appear unable to fix this.
- nologic01 3y agoWhether that "leak" is genuine is not clear, but what is clear is that building LLM / AI capability is no longer a major technical hurdle, definitely not for any entity of the size of Amazon or Apple. The timescale for integrating such tools into existing business models or developing new business models is a different story. They don't need to impress the over-exited social media echo chambers, they need to project 1) legal (not subject to lawsuits), 2) stable (not a fad) and 3) defendable (the moat thingy) cash flows for multi-years forward. Actually the hoopla of the past year where a lot of preliminary stuff is released/discussed/leaked does not fit at all the playbook of a "serious" corporate. Not clear what it really means, maybe big tech is feeling that the status quo is very fragile, so they take more risk than necessary or maybe they are so confident that they don't care about optics.
- precompute 3y ago> The existence of such datasets follows from the line of thinking in Data Doesn't Do What You Think, and they are rapidly becoming the standard way to do training outside Google. links to http://www.internalgooglesitescrubbedbyus.com/ http://www.internalgooglesitescrubbedbyus.com/ Haha. Who writes this blog?
- BiteCode_dev 3y agoInvestors are obsessed with moats, but people have to realize that the entire world runs on business that have no moats. There are no moats to being a plumber, a baker, a restaurant... The moat concept is predominant because the idea that everything must make billions have infected the debate about businesses. It's all about being a unicorn, a giant, a monopoly, making every body at the top billionaires, and it's like there is no other way to live. Except that's not how most people do live, even entrepreneurs. Even Apple, which today is the typical example of a business with a moat, didn't start with "we can't get into this computer business, we'd have no moat". They have a moat now, but it's a consequence of all the business decisions and the thing they built after many decades. They didn't start their project by the moat. The started their project by providing value and marketing it.
- nashashmi 3y agoShareholders gain pennies with moats, pennies that someone else does not earn. Without moats they benefit much more, but it's not more than someone else. And that's the contentious issue. How would I benefit more than my neighbor?
- nologic01 3y ago> Investors are obsessed with moats You can't blame them: gratuitous moats (like those provided by winner-takes-all dynamics) are not common in a functioning (competitive) economy so they get to be revered. It feels unlikely that the recent period of big tech can keep the same benefits going forward. It was basically a political moat: counting on the ongoing lack of antitrust and consumer protection regulation. Even if the political dysfunction that allows that continues (quite likely), the wheels of the universe are turning. The "leaked" report focuses on open source - a mode of producing software that is bound to become a major disruptor. We tend to discount open source because of its humble beginnings, long incubation, many false dawns and difficult business models. But if you objectively take a look at what is possible today with open source software, its quite breathtaking. I would not discount some tectonic shifts in adoption. The long running joke is "the year of the linux desktop", but keep adding open source AI and other related functionality and at some point the value proposition of open source computing (both for individuals and enterprises) will be crushingly large to ignore. Don't forget too, that other force of human nature: geopolitics (e.g., think TikTok and friends). The current "moats" were established during an earlier, more innocent era. Now digitization is a top priority / concern for many countries. The idea that somebody can build a long-lived AI moat given the stakes is strange to say the least.
- deleted 3y ago[deleted]
- osigurdson 3y agoThe true vendor lock-in in this space is embeddings. Once you have generated embeddings for all of your content it will be very difficult to move to another vendor's embedding engine as there will be no way to translate from one to the other.
- T3RMINATED 3y ago[dead]
- HarpyMan03 3y agoHow do I know this conversation thread isn't generated by AI? also let's stop calling it intelligence. Its software that has been programmed. Don't forget that guys and gals. One flick of a switch and it's gone. Actually it would be cool if software went extinct instead of species of irreplaceable animals..
- HarpyMan393393 3y agoHow do I know this conversation thread isn't generated by AI? also let's stop calling it intelligence. Its software that has been programmed. Don't forget that guys and gals. One flick of a switch and it's gone. Actually it would be cool if software went extinct instead of species of irreplaceable animals..
- terabytest 3y agoI found this article very interesting as a way to get more insight into the deeper layers of this industry. Where can one go to keep themselves updated on these topics?
- jlpom 3y ago[dead]
- syngrog66 3y agoGoogle has no moat, just a massive massive massive data set, a massive massive massive amount of compute to repurpose at whim, tons of cash, thousands of employees and a large number with AI/ML skills already. oh and control the dominant web browser and one of the top two dominant mobile OSes. and, and, and... other than that, yes, no moat
- deleted 3y ago[deleted]
- whuan 3y agoOpen source gives everyone the opportunity and it's more extensible, which I think is the future, but the current AI model just costs too much...Somehow it reminds me of k8s when docker just became a hot topic