5 ms·
GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consu
by somenameforme 4y ago
GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consumer level hardware. A system with 4 RTX 3090s (< $4k) gets results in about 16 seconds per query.
The proverbial 'some guy on Twitter'[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university).
If these technologies end up as anything more than a 'novelty of the year' type event, then I expect to see them able to be run locally on phones within a decade. There will be a convergence between hardware improving and the software getting more efficient.
[1] - https://github.com/THUDM/GLM-130B https://github.com/THUDM/GLM-130B
[2] - https://twitter.com/alexjc/status/1617152800571416577 https://twitter.com/alexjc/status/1617152800571416577
- brookst 4y agoAgree, but much less than 10 years. Now that the transformer is establishing itself as the model, we’ll see dedicated HW acceleration for transformers, encoders, decoders, etc. I will eat my hat* if we don’t see local inference for 200B+ parameters within 5 years. * I don’t own a hat
- senttoschool 4y agoWhich chip company do you think will benefit from the move to transformers?
- flakeoil 4y agoNo-one in particular. It will be integrated in the processors (the intels/AMDs on PC, the Mx on Macs and the Qualcomms, Mediateks and Samsungs on phones). Not much different from video and graphics accelerators being integrated today, or DSP focused instructions in the instruction set. They just have to do it to stay relevant.
- chii 4y agoi would imagine these models would be considered trade secrets, and that the models (esp. good ones that take a lot of resources to have trained) would not leave the data center. Your access to such models would be dictated by an api instead of locally run.
- jononor 4y agoNot sure about 200B+ models on phone hardware in 10 years. But I think we'll be able to deliver the same predictive performance with models 1/10 the size and those will fit. That is what happened with CNN vision models over the last 8 years.
- firefeathered 4y agoStable diffusion can already run on an iphone. Hopefully that trend will come to LLMs too.
- waynesonfire 4y agoOh it will. We'll be asking our iphone questions and it'll be returning seo spam and ads for viagra in a chatbox. Meanwhile, the big boys get the firehose feed.
- cromwellian 4y agoGoogle currently handles 100,000 queries per second. The costs to run GLM-130B or GPT-3 at this rate would be astonishingly high.
- somenameforme 4y agoWould they? An array of a million of these machines would cost $4 billion at consumer retail prices. That's 1.4% of Google's annual revenue for one-off bulk cost. The operational cost, at consumer retail electric using current inflated levels, was a small fraction of a cent per query. This is ignoring economy of scale, better electric pricing, caching, etc.
- cromwellian 4y agoWhere are you getting a CPU + RAM + RTX 3090 for $1k? To even install a million of these machines, you'd have to build a new datacenter, the capital costs are going to be beyond just the wholesale price of GPU boards, and you'll have to hire a ton of datacenter technicians. But leaving that aside, look at OpenAI's pricing. $.02/1K tokens. Let's say the average query would be 20 tokens, so you'd get 50 queries/$.02 = 2500 queries/1$, or for 100k, $40/sec * 86400 * 365 = $1.2b. My guess is OpenAI's costs right now are not scaled to handle 100k QPS, so they're way underpriced for that load. This might be a cost Google could stomach. I just think blindly shoe-horning these 100B+ param models into this use case is probably the wrong strategy, DeepMind's Chinchilla has shown it's possible to significantly reduce parameter size/cost while staying competitive in accuracy. I think Google's going to eventually get there, but they're going to do it more efficiently that brute forcing a GPT-3 style model. These very large parameter models are tech demos IMHO at this point.
- hansvm 4y agoWhy would cost per query go up measurably for a highly parallelizeable workload?
- somenameforme 4y agoYou can get an RTX 3090 for < $1k. I was largely handwaving away the rest of the costs since all the processing is done on those cards and basic hardware is really cheap now a days. But in hindsight that might not be entirely reasonable because you would need a motherboard that could support a 4x setup, as well as a reasonable power supply. But the cost there is still going to be in the same ballpark, so I don't think it changes much. That said I do agree with your final conclusion. Bigger is not necessarily better in neural networks, and I also expect to see requirements rapidly decline. I also don't really see this as being something that's going to gets ultra-monopolized and centralized. One big difference between natural language interfaces and something like search is user expectations. With natural language the user has an expectation of a result, and if a service can't meet that expectation - then they'll go elsewhere. And I think it is literally impossible for any single service to meet the expectations of everybody.