10 ms·
Meta AI: "The Future of AI Is Open Source and Decentralized"
- jsheard 2y agoDecentralized inferencing perhaps, but the training is very much centralized around Metas continued willingness to burn obscene amounts of money. The open source community simply can't afford to pick up the torch if Meta stops releasing free models.
- deleted 2y ago[deleted]
- leetharris 2y agoThere's plenty of open source AI out there that isn't Meta. It's just not as good. The #1 problem is not compute, but data and the manpower required to clean that data up. The main thing you can do is support companies and groups who are releasing open source models. They are usually using their own data.
- jsheard 2y ago> There's plenty of open source AI out there that isn't Meta. It's just not as good. To my knowledge all of the notable open source models are subsidised by corporations in one way or another, whether by being the side project of a mega-corp which can absorb the loss (Meta) or coasting on investor hype (Mistral, Stability). Neither of those give me much confidence that they will continue forever, especially the latter category which will just run out of money eventually. For open source AI to actually be sustainable it needs to stand on its own, which will likely require orders of magnitude more efficient training, and even then the data cleaning and RLHF are a huge money sink.
- exe34 2y agoif you can do 100x more efficient training with open source, closeAI can simply take that and train a model that's 100x bigger/longer/more tokens.
- bugglebeetle 2y agoAKA why Unsloth is now YC backed for their even better (but closed source) fine-tuning.
- moffkalast 2y agohttps://huggingface.co/datasets/HuggingFaceFW/fineweb https://huggingface.co/datasets/HuggingFaceFW/fineweb The #1 problem is absolutely compute. People barely get funding for fine tunes, and even if you physically buy the GPUs it'll cost you in power consumption. That said, good data is definitely the #2 problem. But nowadays you can just get good synthetic datasets from calling closed model APIs or just using existing local LLMs to sift through trash. That'll cost you too.
- citboin 2y ago>The main thing you can do is support companies and groups who are releasing open source models. They are usually using their own data. Alternatively we could create standardized open source training data like wikipedia, wikimedia as well as public domain literature and open courseware. I'm sure that there are many other such free and legal sources of data.
- KaiserPro 2y agobut the training data is one of the key bits that makes or breaks your model's performance. There is a reason why datasets are private and the model weights aren't.
- Der_Einzige 2y agoCompute is for sure the number one problem. Look at how long it’s taking for anything better than Pony Diffusion to come out for NSFW image gen despite the insane amount of demand for it. Look at how much computer purple AI actually has. It’s basically nothing.
- cynicalpeace 2y agoOne area that's interesting, but easy to dismiss because it's the ultimate cross-section of hype (AI and crypto) is bittensor. AFAICT it decentralizes the training of these models by giving you an incentive to train models which will mine the crypto if you're improving it. I learned about it years ago, mined some crypto, lost the keys and now kicking myself cuz I would've made a pretty penny lol
- jsheard 2y agoDoes it actually work? AIUI the current consensus is that you need massive interconnect bandwidth to train big models efficiently, and the internet is nowhere near that. I'm sure the Nvidia DGX boxes have 10x400Gb NICs for a reason.
- cynicalpeace 2y agoI have no idea. The idea is certainly interesting but I've never actually understood how to run inference on these models... the people that run it seem to be unable to just talk simply.
- CaptainFever 2y agoI've seen bittensor before. I think it makes sense, as a way to incentivise people to rent their GPUs, without relying on a central platform. But I've always felt it was kind of a scam because it was so hard to find any guides on how to use it. Also, this doesn't seem to actually solve the issue of fine tuners needing funding to rent those GPUs? One alternative is something like AI Horde, which pays GPU providers with "labour vouchers" that allow them to get priority next time they want GPU. Requires a central platform to track vouchers and ban those who exchange them. Basically a sort of real-life comparison of mutualism (AI Horde) vs capitalism (bittensor).
- bloatedGoat 2y agoThere are methods that make it feasible to train models over the internet. DiLoCo is one [1] and NousResearch has found a way to improve on that using a method they call DisTro [2]. 1. https://arxiv.org/abs/2311.08105 https://arxiv.org/abs/2311.08105 2. https://github.com/NousResearch/DisTrO?tab=readme-ov-file https://github.com/NousResearch/DisTrO?tab=readme-ov-file
- numpad0 2y agoCentralized production, decentralized consumption.
- pjkundert 2y agoThe future of everything you depend on is open source and decentralized. Because all indications are that the powers over you cannot abide your freedoms of association, communication and commerce. So, if it’s something your family needs to survive - it has better be distributed and cryptographically secured against interference. This includes interference in the training dataset of whatever AIs you use; this has become a potent influence on the formation of beliefs, and thus extremely valuable.
- caeril 2y agoIt's not the training dataset. All of these models, including the "open" ones, have been RLHF'ed by teams of politically-motivated people to be "safe" after initial foundation training.
- pjkundert 2y agoAnd I’m not even remotely interested in the “corrections” supplied by some group of right-thinking meddlers! This corruption must be disclosed as assiduously as the base dataset, if not more so.
- _yid9 2y agoOr, at least package them up as "personnas" and give them an appropriate name, eg. "Church Lady", "Jr. Marxist Barista", "Undergrad Philosophy Major", ... Actually, those seem like an apt composite description of the PoV of the typical mass-market AI... 8/
- Der_Einzige 2y agoNot mistrals. Mistral large is willing to tell me how to genocide minorities or NSFW without any kind of orthogonalization or fine tuning. Please actually try models instead of pontificating without evidence. Try it for yourself: https://huggingface.co/mistralai/Mistral-Large-Instruct-2407 https://huggingface.co/mistralai/Mistral-Large-Instruct-2407
- 2y ago
- Qshdg 2y agoGreat, who gives me $500,000,000, Nvidia connections to actually get graphics cards and a legal team to protect against copyright lawsuits from the entities whose IP was stolen for training? Then I can go ahead and train my open source model.
- riku_iki 2y agoyou can pick existing pretrained foundational model from corp (google, MS, Meta) and then finetune it(much cheaper) with your innovative ideas.
- deleted 2y ago[deleted]
- monkeydust 2y agoCurious but is there a path where llm training or inference could be distributed across the BOINC network: https://en.m.wikipedia.org/wiki/Berkeley_Open_Infrastructure_for_Network_Computing https://en.m.wikipedia.org/wiki/Berkeley_Open_Infrastructure...
- WithinReason 2y agoNot yet, the bandwidth requirement is too high. But if someone figures this out that's when we will have true open source models. A crowdsourced supercomputer can outcompete any corporation's server farm.
- monkeydust 2y agoYea seems like an inflection moment when it happens. Curious who's working on this problem.
- WithinReason 2y agoIt's not in the interest of the big players for sure!
- alecco 2y ago* pre-trained models * does not apply to training data
- exabrial 2y agoonly when it's financially convenient for them...
- dkga 2y agoWell, yes. They are a company, with shareholders and all. So while not breaching any law, they should indeed pursue strategies that they think would be profitable. And for all the negativity seen in many of the comments here I think it’s actually quite remarkable that they make model checkpoints available freely. It’s an externality, but a positive one. Not quite there yet in terms of the ideal - which is definitely open source - and surely with an abuse of language, which I also note. But overall, the best that is achievable now I think. The true question we should be tackling is, is there an incentive-compatible way to develop foundation models in a truly open source way? How to promote these conditions, if they do exist?
- nis0s 2y agoI like the idea of this! But is there any reason to be concerned about walled gardens in this case, like how Apple does with its iOS ecosystem? For example, what if access to model weights could be revoked. There is a lot of interest in regulating open source AI, but many sources of criticism miss the point that open source AI helps democratize access to technologies. It worries me that Meta is proposing an open source and decentralized future because how does that serve their company? Or is there some hope of creating a captive audience? I hate to be a pessimist or cynic, but just wondering out loud, haha. I am happy to be proven wrong.
- candiddevmike 2y agoStop releasing your models under a non FOSS license.
- hyuuu 2y agothe view of the comments here seems to be quite negative for what meta is doing. Honest question, should they go to the route of openai and closed source + paid access instead? OpenAI or Claude seem to garner more positive views than llama open sourced.
- naming_the_user 2y agoThe models are not open source, you're getting the equivalent of a precompiled binary. They are free to use.
- RealStickman_ 2y agoFree to use with restrictions, so you maybe get 1.5/4 FOSS freedoms.
- Palmik 2y agoThat's a bad analogy. The weights are much closer to source code, because you can directly modify them (fine tune, merge or otherwise) using open source software that Meta released (torchtune, but there are tons of other libraries and frameworks).
- progval 2y agoYou can also modify a precompiled binary with the right tools.
- Palmik 2y agoExcept doing continued pre-training or fine tuning of the released model weights is the same process through which the original weights were created in the first place. There's no reverse engineering required. Meta engineers working on various products that need custom versions of the Llama model will use the same processes / tools.
- meiraleal 2y agoNot much would change if they did. Meta intentions and OpenAI intentions are the same: reach monopoly and take all the investment back with a 100x return. Anyone that achieves it will be as evil as the other one. > OpenAI or Claude seem to garner more positive views than llama open sourced. that's more about Meta than the others. Although OpenAI isn't that far from Meta already.
- rkou 2y agoAnd what about the future of social media? This is such devious, but increasingly obvious, narrative crafting by a commercial entity that has proven itself adversarial to an open and decentralized internet / ideas and knowledge economy. The argument goes as follows: - The future of AI is open source and decentralized - We want to win the future of AI instead, become a central leader and player in the collective open-source community (a corporate entity with personhood for which Mark is the human mask/spokesperson) - So let's call our open-weight models open-source, and benefit from its imago, require all Llama developers to transfer any goodwill to us, and decentralize responsibility and liability, for when our 20 million dollar plus "AI jet engine" Waifu emulator causes harm. Read the terms of use / contract for Meta AI products. If you deploy it, some producer finds the model spits out copyrighted content, knocks on Meta's door, Meta will point to you for the rest of the court case. If that's the future for AI, then it doesn't really matter whether China wins.
- foobar_______ 2y agoIt has been clear from the beginning that Meta's supposed desire for an open source AI, is just a coping mechanism for the fact that got beat out of the gate. This is an attempt to commoditize AI and reduce OpenAI/Google/Whoever's advantage. It is effective, not doubt, but all this wankery about how noble they are for creating an open-source AI future is just bullshit.
- CaptainFever 2y agoI feel the same way. I'm grateful to Meta for releasing libre models, but I also understand that this is simply because they're second in the AI race. The winner always plays dirty, the underdog always plays nice.
- KaiserPro 2y agobut they've _always_ released their stuff. Thats part of the reason why the industry uses pytorch, that and because its better than tensorflow. In the same way that detectron and Segment anything is an industry standard. Sure, for LLMs openAI released a product first. but its not unusual for meta to release useful models.
- abetusk 2y agoThis is the modern form of embrace, extend and extinguish. "Embrace" open source, "extend" the definition to make it non open/libre and finally extinguish the competition by shoring up the drawbridge to the moat they've just built.
- troupo 2y agoI've had as a comment to a comment, but I'll repost it at the top level: They use "open source" to whitewash their image. Now ask yourself a question: where does Meta's data come from? Perhaps from their users' data? And they opted everyone in by default. And made the opt-out process as cumbersome as possible: https://threadreaderapp.com/thread/1794863603964891567.html https://threadreaderapp.com/thread/1794863603964891567.html And now complain that the EU is preventing them from "collecting rich cultural context" or something https://x.com/nickclegg/status/1834594456689066225 https://x.com/nickclegg/status/1834594456689066225
- rkou 2y agoAlso known as https://en.wikipedia.org/wiki/Openwashing https://en.wikipedia.org/wiki/Openwashing > In 2012, Red Hat Inc. accused VMWare Inc. and Microsoft Corp. of openwashing in relation to their cloud products.[6] Red Hat claimed that VMWare and Microsoft were marketing their cloud products as open source, despite charging fees per machine using the cloud products. Other companies are way more careful using "open source" in relation to their AI models. Meta now practically owns the term "Open Source AI" for whatever they take it to mean, might as well call it Meta AI and be done with it: https://opensource.org/blog/metas-llama-2-license-is-not-open-source https://opensource.org/blog/metas-llama-2-license-is-not-ope...
- dzonga 2y agothe reason - i'm a little bearish on AI is due to its cost. small companies won't innovate on models if they don't have billions to burn to train the models. yet when you look back at history, things that were revolutionary, it was due to low cost of production. web, bicycles, cars, steam engine cars etc.
- rafaelmn 2y ago> yet when you look back at history, things that were revolutionary, it was due to low cost of production. Nuclear everything, rockets/satellites, tons of revolutionary things that are very expensive to produce and develop. Also software scales differently.
- zwijnsberg 2y agoyet if the weights are made public, smaller companies can leverage these pretrained models can't they?
- miguelaeh 2y agoThe first cars, networks, and many other things were not unexpensive. They became so with time and growing adoption. Cost of compute will continue decreasing and we will reach that point where it is feasible to have AI everywhere. I think with this particular technology we have already reached a no return point
- farco12 2y agoI could see the cost of licensing data to train models increasing significantly, but the cost of compute for training models is only going to drop on a $/PFLOP basis.
- Manuel_D 2y agoI suspect that models will become smaller, getting pruned to focus on relevant tasks. Someone using an LLM to power tech support chat doesn't want, nor need, the ability to generate random short stories. In this sense, AI is akin to cars prior to assembly line manufacturing: expensive and bespoke machines, with their full potential tapped when they're later made in a more efficient manner.
- CatWChainsaw 2y agoFacebook promised to connect the world in a happy circle of friendship and instead causes election integrity controversies, bizarre conspiracy theories about pandemics and immigrants to go viral, and massive increases in teen suicide. Not sure why anyone would trust them with their promises of decentralized-AI and roses.
- atq2119 2y agoGood. Now compare to OpenAI. Clearly what Meta is doing is better than OpenAI from the perspective of freedom and decentralization.
- CatWChainsaw 2y agoCool. What Meta is doing is better than cigarettes from the perspective of addiction. If Meta is the best we have, then we'd better create something better, or prepare for the inevitable enshittification.
- jazzyjackson 2y agoI love it. TOBACCO PRODUCTION MUST BE DECENTRLIZED.
- CatWChainsaw 2y agoGood old tobacco farms.
- menacingly 2y agoDecentralized on centralized hardware?
- latchkey 2y agoEvidence is showing that AMD MI300x are proving to be a strong contender.
- YetAnotherNick 2y ago> AMD MI300x Which costs significantly more than H100 at least when renting[1]. Also the price of hardware isn't significantly lower. Also, both AMD and Nvidia have been deliberately stopping progress in cheaper consumer graphics card by not increasing VRAM and removing things like fast interconnect. [1]: https://getdeploying.com/runpod https://getdeploying.com/runpod
- latchkey 2y agoYour source is a paid advertisement[0] that only lists providers who pay the tax. I'm not really interested in that sort of game. We've done the research and our MI300x pricing is competitive with H100's and even more so if you consider the amount of vram in a MI300x. I am also not sure if it is deliberate or just realizing that running this stuff is error prone and requires a lot of capex, power and infrastructure. It is difficult to support that at the consumer level, so why bother when enterprises are now offering super computers for rent. It is not their wheelhouse, so I can see why they do not want to take on the extra risk. [0] https://getdeploying.com/plans https://getdeploying.com/plans
- YetAnotherNick 2y ago> We've done the research and our MI300x pricing is competitive More info would be appreciated. Because I tried finding the pricing for all the providers and they aren't similar. In my research, in almost all the cases, 2*A100 is superior than both H100 and MI300x in VRAM, performance and pricing if the usecase supports multi GPU.
- deleted 2y ago[deleted]
- mrkramer 2y agoYea, I believe you Zuck, it's not like Facebook is closed centralized privacy breaking walled garden.
- pie420 2y agoIBM Social Media Head: "The Future of Social Media is OPEN SOURCE and DECENTRALIZED" This must be a sign that Meta is not confident in their AI offerings.
- jmyeet 2y agoTwo things spring to mind: 1. Open source is for losers. I'm not calling anyone involved in open source a loser, to be clear. I have deep respect for anyone who volunteers their time for this. I'm saying that when companies push for open source it's because they're losing in the marketplace. Always. No companiy that is winning ever open sources more than a token amount for PR; and 2. Joel Spolsky's now 20+ year old letter [1]: > Smart companies try to commoditize their products’ complements. Meta is clearly behind the curve on AI here so they're trying to commoditize it. There is no moral high ground these companies are operating from. They're not using their vast wisdom to predict the future. They're trying to bring about the future the most helps them. Not just Meta. Every company does this. It's why you'll never see Meta saying the future of social media is federation, open source and democratization. [1]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
- uptownfunk 2y agoIt’s marketing to get the best researchers. The researchers want the meta pay and they want to hedge their careers to continue to publish. That’s the real game, it’s a war for talent. Everything else is just secondary effects.
- deleted 2y ago[deleted]
- croes 2y agoAlso Meta: The future is VR
- stonethrowaway 2y agoI’ll link to my comment here from approx. 52 days ago: https://news.ycombinator.com/item?id=41090142 https://news.ycombinator.com/item?id=41090142 This is chess pieces being moved around the board at the moment.
- Refusing23 2y ago'AI is so expensive, we'd rather have it handled with communism!' If that means it'll be free/cheaper... sure
- lccerina 2y ago"The Future" in the meantime we will keep doing our stuff, building walled gardens of AI generated spam and slop, and claiming our AI models are open source when they are not. The faster Meta dies, the better it would be.
- deleted 2y ago[deleted]
- Marcus_Ford 2y ago[dead]