11 ms·
I think the problem with AI being everywhere and ubiquitous is that AI is the first technology in a very long time that requires non-trivial compute power. Tha
by narrator 3y ago
I think the problem with AI being everywhere and ubiquitous is that AI is the first technology in a very long time that requires non-trivial compute power. That compute power costs money. This is why you only get a limited number of messages every few hours from GPT4. It simply costs too much to be a ubiquitous technology.
For example, the biggest LLama model only runs on an A100 that costs about $15,000 on ebay. The new H100 that is 3x faster goes for about $40,000 and both of these cards can only support a limited number of users, not the tens of thousands of users who can run off a high-end webserver.
I'd imagine Google would lose a lot of money if they put GPT4 level AI into every search, and they are obsessed with cost per search. Multiply that by the billions and it's the kind of thing that will not be cheap enough to be ad supported.
- colordrops 3y agoIs the assumption that GPU power and advancements in AI will not get to a reasonable price point in the near future? Because it seems to me that advances in computation have not slowed down at all since it started.
- tomjen3 3y agoWe used to get only a limited number of Internet hours. By the time December 2003 rolled around my family had always on internet. Besides what Google does here is besides the point, because Bing has already unleashed ai search. Google will either follow along or stop being relevant.
- airstrike 3y agoTime for a dedicated "AI box" at home with hotswapping compute boards? Maybe put it inside a humanoid or animal-like robot with TTS capabilities? Sign me up for that kickstarter! EDIT: based on some quick googling (should I have asked ChatGPT instead?), Nvidia sells the Jetson Xavier Nx dev kit for ~$610 https://www.electromaker.io/shop/product/nvidia-jetson-xavier-nx-developer-kitpre-order https://www.electromaker.io/shop/product/nvidia-jetson-xavie... Just need the robot toy dog enclosure (See https://www.electromaker.io/blog/article/best-sbc-for-ai-single-board-computer-for-artificial-intelligence https://www.electromaker.io/blog/article/best-sbc-for-ai-sin... for a list of alternatives if that one is too expensive)
- yayr 3y agoeach billion parameters using 16 bit floats requires around 2 GB of GPU or TPU RAM. ChatGPT is expected to have around 1000 billion. Good open source LLMs have around 7-20 billion currently. Consumer GPUs currently max out at 24 GB. You can now quantize the model to e.g. 4 bits instead of 32 per parameter and do other compressions, but still there is quite a limit what you can do with 24 GB of RAM. The Apple unified memory approach may be a path forward to increase that... so one box gives you access to the small models, for a GPT4 like model you'd need (for inference and if you had the model and tools) probably 100 of those 4090s or 25 of H100 with 96 GBs I guess to fit in 2 TB of model data.
- deleted 3y ago[deleted]
- niemandhier 3y agoCurrently we do not explore sparsity. The next iteration of models will be much more compact by focusing on reducing effective tensor size.
- bee_rider 3y agoIt seems like a ton of engineering effort has been put into these neural network frameworks. How didn’t they explore sparsity yet? With numerical linear algebra that’s, like, the most obvious thing to do (which is to say, you probably know beforehand if your problem can be mapped to sparse matrices). (Edit: just to be clear here, I’m not saying I expect the whole field is full of dummies who missed something obvious or something like that, I don’t know much at all about machine learning so I’m sure I’m missing something).
- taneq 3y agoFrom what I’ve read recently, most sparse methods just haven’t given that much improvement yet, and we’re only recently pushing up against the limits of the “just buy more RAM” approach. It sounds like there is a lot of work happening on sparse networks now, so it’ll be interesting to see how this changes in the near future.
- Certhas 3y agoThe article talks about this explicitly though. Reasonably good models are running on raspberry Pis now.
- ChatGTP 3y agoIs a reasonably good model what people get value out of though? Maybe this is why Sam Altman talked about "the end of the large LLMs is here"? He understands anything bigger than ChatGPT-4 isn't viable to run at scale and be profitable?
- RonnieOwnsLexus 3y agoDoes this mean models larger then ChatGpt would still be better for the same data size as long as someone is ready to pay? At what limit does it stop getting better?
- neltnerb 3y agoI thought he was fairly explicit that he thought larger models would provide incremental gains for exponentially greater cost, so yeah, I guess not profitable is a way to put it...
- Animats 3y agoIt's a win for Google that LLMs are getting cheaper to run. OpenAI's service is too expensive to be ad-funded. Google needs a technology that's cheaper to provide to maintain their ad-supported business model.
- patrickk 3y agoGoogle could make a bet like they did with YouTube. At the time, operating YouTube was eye wateringly expensive and lost billions. But google could see where things were going: a triple trend of falling storage costs, falling bandwidth and transmission costs (I’m trying to dig up a link I read years ago about this but google search has gotten so shit that I can’t find it). It was similar for Asic miners for Bitcoin. Given enough demand, specialised, lower cost hardware specially for LLMs will emerge.
- mycall 3y agoOn the flip side, I found only one person (I'm sure there are more) that are attacking the software efficiency side of things. You would be quite surprised how inefficient the current LLM software stack is, as I learned on a CPP podcast [0]. Ashot Vardanian has a great github repo [1] that demonstrates many ways compute can come way down in complexity and thus cost. [0] https://cppcast.com/ai_infrastructure/ https://cppcast.com/ai_infrastructure/ [1] https://github.com/orgs/unum-cloud/repositories?type=all https://github.com/orgs/unum-cloud/repositories?type=all
- scarface74 3y agoYou realize that most reports are that YouTube is barely profitable.
- unicornmama 3y agoThis cost argument is being overblown. While it's a limitation for today's product, enginners are very good at optimization. Therefore the costs will drop in the medium to long term from efforts on both the software and hardware side.
- ok123456 3y agoWithin a decade mid-level consumer cards will be just as powerful as those $40k cards.
- deleted 3y ago[deleted]
- bcrosby95 3y agoConsidering how long it took mid level consumer cards to beat my $600 1080, you're way more optimistic than I am.
- Taek 3y agoThe main moat here is VRAM, not raw compute power.
- reissbaker 3y agoSo... Four years? The 1080 launched in 2016, and the 3070 launched in 2020, for $100 cheaper — the launch price of the 1080 was $699, and the 3070 was $599. The 3070 easily trounced the 1080 in benchmarks. The 3060 effectively matched a 1080 at $329 in 2021 (and has 50% more VRAM at 12GB instead of the 1080's 8GB), so call it five years if the 3070 isn't mid-range enough. The 3060 Ti launched in 2022 at $399 and handily beat the 1080 on benchmarks, so call it six years if you want the midrange card to beat (not just match) the previous top-of-the-line card, and if a *70 card doesn't count as midrange enough. Less than a decade still seems like a reasonable claim for a midrange card to beat a top-of-the-line card.
- MichaelZuo 3y agoThe 3060 was only readily available quite recently, so it's about 6 years from ready availability of the 1080 at $600 to 3060. Taking 6 years to double the perf/$ implies that it would take ~42 years for a $40000 H100 to reach mid-range levels. Assuming scaling, particularly VRAM, holds. And plus it would be getting really close to the Landauer limit by that point.
- 3y ago
- rileyphone 3y agoYou can run it (quantified at least) on a $4000 Mac thanks to Apple's unified memory. Surely other manufacturers are looking for how to expand VRAM, hopefully Intel or AMD.
- LoganDark 3y agoNot to mention Apple chips have a bunch of very nice accelerators and also (!!!) macOS contains system frameworks that actually use them.
- james-revisoai 3y agoYou're right and this is why they didn't heavily use BERT(in the full sense), arguably the game-changing NLP model of the 10s. They couldn't justify bringing the cost per search up.
- yieldcrv 3y agonah, Lora quantized LLM’s are going to be at the OS level in 2 years and consumer architecture refreshes are just going to extend more RAM to already existing dedicated chips like Neural Engine client side tokens per second will be through the roof and the models will be smaller
- computerex 3y agoLoRa is not a quantization method, it's a fine tuning method.
- yieldcrv 3y agoyou read that whole paragraph and assumed this prediction didn't involve consumers using fine tuned models despite lora being explicitly mentioned? -EQ moment edit: its about the combination of those methods making models accessible on consumer hardware
- robbintt 3y agoIt's two adjectives on the noun.
- Taek 3y agoThe biggest llama model has near 100% fidelity (its like 99.3%) at 4 bit quantization, which allows it to fit on any 40GB or 48GB GPU, which you can get for $3500. Or at about a 10x speed reduction you can run it on 128 GB of RAM for only around $250. The story is not anywhere near as bleak as you paint.
- brimwats 3y agoI haven't seen any repos or guides to using llama on that level of RAM, which is something I do have. any pointers?
- spiffytech 3y agoSomething I haven't figured out: should I think about these memory requirements as comparable to the baseline memory an app uses, or like per-request overhead? If I needed to process 10 prompts at once, do I need 10x those memory figures?
- kiratp 3y agoYou need roughly (model size + (n * (prompt + generated text)) where n. Is the number of parallel users/ request.
- atq2119 3y agoIt should be noted that that last part has a pretty large factor to it that also scales with model size, because to run transformers efficiently you cache some of the intermediate activations from the attention block. The factor is basically 2 * number of layers * number of embeddings values (e.g. fp16) that are stored per token.
- deegles 3y agoI was in Japan recently and they sell these pocket size translator devices with a microphone, camera and screen. You can speak to it or take pictures and it will translate on the fly. Maybe $100 usd range for a nice one. It's only a matter of time before someone makes a similar device with a decent LLM on it, and premium ones will have more memory/cpu power.
- tyree731 3y agoI think we as humans have a tendency to extrapolate from our present position to a position we can imagine that we’d like, even if there isn’t a foreseeable path from here to there. I believe this may end up being one of those cases.
- Zuiii 3y agoWhy? What GP describes seems both feasible and inevitable to me.
- nmfisher 3y agoI mean...isn't that just a smartphone? I know exactly what you're talking about because my father-in-law had the same thing. I'm just very skeptical that specialist hardware will overtake general commoditized computing devices for mass-market usage. The economics alone make it unlikely.
- rvnx 3y agoYes, usually they are locked-down Android devices. One of them has an unlimited "free forever" internet subscription so it can fetch network translations online if the local dictionary doesn't have the word.
- ben_w 3y agoEven though Word Lens was first released over 12 ago, I keep surprising people[0] by showing them the same technology built into Google Translate. [0] even other tech developers who, like me, migrated somewhere where a language barrier came up
- sroussey 3y agoOr you can quantize the model and run it on your laptop.
- cush 3y agoYeah, today.
- pavelstoev 3y agoOr you can apply GPU optimizations for such ML workloads. By optimizing the way these models run on GPUs, significantly improve efficiency and slash costs by a factor of 10 or even more. These techniques include kernel fusion, memory access optimization, and efficient use of GPU resources, which can lead to substantial improvements in both training and inference speed. This allows AI models to run on more affordable hardware and still deliver exceptional performance. For example, LLMs running on A100 can also run on 3090s with no change in accuracy and comparable inference latency.
- hsbauauvhabzb 3y agoWould google even care about integrating LLMs into search? They don’t even prune all the spam entries, presumably because they increase advertising revenue and analytics profit.
- greenfield1 3y agoThe thing you have in your pocket would have meant an enormous investment for equivalent compute power just decades ago and filled a whole basement with server racks.
- isp 3y agoThe legendary Cray-2 was the fastest supercomputer in the world in 1985, with peak 1.9 GFLOPS. Less than four decades ago. By comparison, the Cray is outperformed by my smartphone. Actually, it is outperformed by my previous smartphone, which I purchased in 2016 and replaced in 2018. Actually, it is outperformed by a single core on my previous smartphone, of which it has eight cores.
- jlpom 3y ago[dead]
- lerchmo 3y agocaching + simpler models for classification / triage should reduce the load on the big model.