7 ms·
Even a linear growth rate of average RAM capacity would obviate the need to run current SOTA LLMs remotely in short order. Historically average RAM has grown f
by adam_arthur 3y ago
Even a linear growth rate of average RAM capacity would obviate the need to run current SOTA LLMs remotely in short order.
Historically average RAM has grown far faster than linear, and there really hasn't been anything pressing manufacturers to push the envelope here in the past few years... until now.
It could be that LLM model sizes keep increasing such that we continue to require cloud consumption, but I suspect the sizes will not increase as quickly as hardware for inference.
Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases.
I think people will be surprised that consumers ultimately end up benefitting far more from LLMs than the providers. There's not going to be much moat or differentiation to defend margins... more of a race to the bottom on pricing
- tomohelix 3y agoRAM is easy. The hard part is making the unified memory SOC like Apple's. From what I know, Apple performance is almost magic. And whatever Apple is making, they are at peak capacity already and they can't make more even if they want to. Nobody else has a comparable technology. Apple is in its own league.
- AnthonyMouse 3y agoApple is just using a wide memory bus, the same as GPUs and server-class x86 CPUs do. It's not even hard, it's just not something desktop CPUs previously had any use for so the current sockets don't support it. And you could do the same thing without even changing the socket by including RAM on the CPU package as an L4 cache. Some of the Intel server CPUs are already doing this.
- ls612 3y agoFor me the test is; when will a Siri-LLM be able to run locally on my iPhone at at least GPT-4 levels? 2030? Farther out? Never because of governments forbidding it? To what extent will improvements be driven by the last gasps of Moore’s Law vs by improving model architectures to be more efficient?
- adam_arthur 3y agoGiven that phones are a few years behind PCs on RAM, likely whenever the average PC can do it, plus a few years. There are phones out there with 24GB of RAM already, it looks like. Of course battery life would be a concern there, so I think LLM usage on phones will remain in the cloud. Haven't studied phone RAM capacity growth rates in detail though
- nico 3y agoThat’s for LLMs, but at the same time, there are other types of models coming out Wouldn’t be surprised if we get small models that can run locally on a phone and just retrieve data from the network as needed (without sending your data out), within the next couple of years
- baq 3y agoWonder if someone is thinking of LLM specific RAM, slower but much denser. Bonus points for not having to reload the model after power cycling. Maybe call this fantastic technology something idiotic like 3d XPoint?
- ronsor 3y agoThe problem with that is LLM speed is mostly bottlenecked by memory bandwidth. Slower RAM means worse performance.
- AnthonyMouse 3y ago> slower but much denser. Bonus points for not having to reload the model after power cycling. This is called a solid state drive.
- baq 3y agoGoes to show how badly Intel executed that one.
- 3y ago
- ramesh31 3y ago>I think people will be surprised that consumers ultimately end up benefitting far more from LLMs than the providers. There's not going to be much moat or differentiation to defend margins... more of a race to the bottom on pricing Should be pointed out that this didn't just happen out of thin air. These open models still cost millions of dollars to create. Meta let the genie out of the bottle, but it won't be free forever.
- logicchains 3y ago>These open models still cost millions of dollars to create. Meta let the genie out of the bottle, but it won't be free forever. This particular model was funded by the UAE government. If they could do it, it should be similarly possible for a western government to create and release one as a public good.
- MuffinFlavored 3y ago> Given how useful GPT-4 is already. Maybe one more iteration would unlock the vast majority of practical use cases. Unless I'm misunderstanding, doesn't OpenAI have a very vested interest to keep making their products so good/so complex/so large that consumer hobbyists can't just `git clone` an alternative that's 95% as good running locally?
- Frannyies 3y agoThey have a huge cost incentive to optimize it for runtime. The magic of openai is their training data and architecture. There is a real risk that a model gets leaked.
- slt2021 3y agoit is not really a moat if one engineer can leave openai with all the secret sauce in his head and replicate it elsewhere (anthropic?)
- foobiekr 3y agoName one software-based tech company where this isn’t true.
- slt2021 3y agomicrosoft? gogel? FB?
- Frannyies 3y agoI only meant the trained model. You would need to steal it all over again as soon as the next model is trained.
- slt2021 3y agono need to steal the model if training process can be reliably replicated/adopted in clean room implementation with additional optimisations. startup as legal entity has close to 0 value, most value is in intellectual property which is stored and transmitted by meatbags.
- cs702 3y agoI agree: No one has any technological advantage when it comes to LLMs anymore. Some companies, like OpenAI, may have other advantages, like an ecosystem of developers. But most of the gobs of money that so many companies have burned to train giant proprietary models is unlikely to see any payback. What I think will happen is that more companies will come to the realization it's in their best interest to open their giant models. The cost of training all those giant models is already a sunk cost. If there's no profit to be made by keeping a model proprietary, why not open it to gain or avoid losing mind-share, and to mess with competitors' plans? First, it was LLaMA, with up to 65B params, opened against Meta's wishes. Then, it was LLaMA 2, with up to 70B params, opened by Meta on purpose, to mess with Google's and Microsoft/OpenAI's plans. Now, it's Falcon 180B. Like you, I'm wondering, what comes next?
- bugglebeetle 3y agoI think it’s the opposite. Models will become more commoditized and closed/invisible as the basis of other service offerings. Apple isn’t going to start offering general API access to the model they’re training, but will bake it into a bunch of stuff and maybe give platform developers limited access. Meta will probably continue to drive the commoditization train because they have a killer ML/AI team, but the same thing will likely happen there once it’s the basis for a service that generates money.
- foobiekr 3y agoThis. We haven’t even entered the get-serious monetization era. Now that the infinite free money pump has been turned down a bunch, we’re going to see what reality looks like.
- 6510 3y agoOkay ill tell you. You need to start a startup that sets up a good number of cameras at manual labor jobs. Most of the footage will be completely useless but every day you hit a once in a day event, every week you get a once in a week event, every month, every year, every decade etc! Then the guy working there for 40 years wacks pipe 224 with a hammer 50 cm from the outlet and production resumes. The footage can be aggressively pruned to fit on the disk. When the robot is delivered in 2033 it can easily figure out, from the footage, all these weird and rare edge cases. The difference will be like that between a competent but new employee and someone with 10 years of experience. I can see the Tesla bots disassembling the production line already. Or do you think it wont happen?
- visarga 3y ago> I think people will be surprised that consumers ultimately end up benefitting far more from LLMs than the providers. LLMs make possible the great skill sharing, they are learning from some people through web and books, and then assist other people in their particular problems. This level of sharing and customisation is even greater and more accessible than open source.
- passion__desire 3y agoAll the great points Salman Khan made about Khan Academy in his famous ted talk apply here. The only difference is LLMs can go from Eli5 to EliPhD in just few back and forth. Then to put cherry on the top, you can ask it summarize the conversation in a poem written in style of Walt Whitman.
- gorbypark 3y agoI can't wait for my phone to have something like 512Gb-1TB of RAM to run some really interesting models locally :D
- AnthonyMouse 3y agoYou can buy 768GB of DDR3 and an Ivy Bridge Xeon E5 to put it in for a total of around $500, most of which is the memory. (The CPUs wouldn't be fast for a model that size though.)
- astrange 3y agoI'd be impressed if you fit that into a phone.
- AnthonyMouse 3y agoIt'll make phone calls. Just put a VoIP app on it. Obviously what you can do in practice is put the interface on your phone. It doesn't have to run on battery to run locally.
- noiv 3y agoRAM may be growing, but free and acceptable content to train models isn't. Question is which is the last model one might install to satisfy all needs.