9 ms·

I tooted this yesterday: Selling access to LLMs via remote APIs is the “stage plays on the radio” stage of technological development. It makes no actual sense;

• amelius 4y ago
  There will be a time when LLMs need data persistence to "improve our user experience". The LLM will act like a "friend" that will remember you when you come back.
• throwawayadvsec 4y ago
  I don't think it's going to happen in the next few years the prices are gonna drop like hell, but ain't no way we run models meant to run on 8 nvidia A100 on our smartphones in the next 5 years just like you don't store the entirety of spotify on your iphone, you're not gonna run any decent LLM on phones any time soon(and I don't consider any of the small Llamas to be decent)
  • Analog24 4y ago
    This is the reason why they're not going to move on device anytime soon. You can use compression techniques, sure, but you're not going to get anywhere near the level of performance of GPT-4 at a size that can fit on most consumer devices
    • mkaic 4y ago
      I think we’ll see completely new architectures dominate in the near future, ousting the transformer. I am strongly suspicious that, while impressive, transformers use several orders of magnitude more compute than is “needed” for the tasks they perform—if for no other reason because the human brain performs similarly and it only draws 20 watts! And it isn’t even an engineered system, jus the product of a very, very long history of natural selection! I fully anticipate that we’ll see AI in the near future that achieves human-level performance on sub-human power budgets like the ones you’d be constrained by on a phone :)
      • Analog24 4y ago
        "neat future" is very ambiguous. At the moment there is nothing even close to transformers in terms of performance. I suspect you are right in general but I'm not sure about the "near future" part, there needs to be a pretty significant paradigm shift for that to happen (which is possible, of course, I just don't see any hints of it yet).
        • mkaic 4y ago
          RWKV is an attention-free architecture that's showing promising scaling at a similar level to Transformers right now! There's also recently been Hyena, which uses a new mechanism that's kind of a weird mix of attention, convolution, and implicit modelling all at once. It's shown promise as well. Remains to be seen if these competing methods will truly scale as well as Transformers, but I've got my fingers crossed. Only a matter of time! I agree that "near future" is quite ambiguous though. If I were to disambiguate my claims, I think I'd personally expect a Transformer-killing architecture to arise in the next 4-5 years.
  • w4ffl35 4y ago
    > but ain't no way we run models meant to run on 8 nvidia A100 on our smartphones in the next 5 years m$ has been working on an AI chip since 2019 so i think we will.
    • d33 4y ago
      They still can't beat the laws of physics. It's just the kind of density that doesn't seem feasible to put in a smartphone.
      • wewtyflakes 4y ago
        Human brains seem to be of the same order of magnitude in terms of size, and seem more nuanced and capable than GPT4.
        • fbhabbed 4y ago
          It's not apples to apples comparison. There's a multitude of tasks that human brains are very bad at. GPT excellence is in raw knowledge and answering machine. You won't find a single human brain that can hold the same amount of knowledge
        • kortex 4y ago
          The human brain is also three-dimensional, heavily interconnected, and has built-in thermal management at every scale. Chips are much faster, but still operate on the essentially linear memory cells, and this limits how many matmuls you can do per second. If we can figure out true connectivity without doing tons of matmuls, then we should be able to massively cut computational demands of models.
    • what_ever 4y ago
      And? Nvidia has been working on AI chips for years.
    • throwawayadvsec 4y ago
      An A100 is about the size of a brick, there is no way we're fitting those 8 bricks in a phone in the next five years, without even thinking about heat management
  • davnicwil 4y ago
    the only thing I can say to this is that Apple have seemed laser focused on tuning their silicon for ML crunching, that that focus is clearly now going to be amped up further still, and that in tandem the software itself will be tuned to Apple silicon. GPUs on the other hand are pretty general purpose. And 5 years on a focused superlinear ramp up is a long time, lots can happen. I am not saying it's 100%, or even 80% likely. It'll be super impressive if it happens, but I see it as well within the realms of reason.
    • tehsauce 4y ago
      Apple's new M2 Max has a neural engine which can do 15 trillion flops. Nvidias's A100 chip (released almost 3 years ago) can do 315 trillion flops. Apple is not going to close this 20x gap in a few years.
      • davnicwil 4y ago
        right, it's a huge challenge. I think the tuning the models to the hardware piece is important, and of course there is much more incentive to do this for Apple than nvidia because of the distribution and ecosystem advantages Apple have. But also, I don't know... let's see what the curve looks like! It's only been a couple of years of these neural engines. Let's see how many flops M3 can hit this year. And then m4 the next. Again, 5 years is a long time actually when real improvement is happening. I am optimistic.
      • moffkalast 4y ago
        > this 168x gap FTFY, remember it takes 8 of those to even load the thing. And when the average laptop has that much compute, GPT 4 will seem like Cleverbot in comparison to the state of the art.
      • sroussey 4y ago
        At some point, they will put the models in silicon. I’m curious as to when… 5yr?
        • viraptor 4y ago
  • viraptor 4y ago
    > but ain't no way we run models meant to run on 8 nvidia A100 on our smartphones in the next 5 years When I leaned about neutral networks, the general advice at the time was "you'll only need one hidden layer, with somewhere between the number of your input and output neurons". While that was more than 5 years ago, my point is - both the approach and the architecture changes over time. I would not bet on what we won't have in 5 years.
  • brucethemoose2 4y ago
    Vicuna (LLama) 13B is pretty good IMO. A 20B model can definitely fit in RAM on future devices.
• andrewcamel 4y ago
  On this topic, Apple is the sleeping giant. Sleeping tortoise maybe. Everyone else has been fast out of the gates, but Apple has effectively already been positioning to leap frog everyone after a decade+ of M1 chip design. Ever since these chips launched, the M1 chips have felt materially underutilized, particularly their GPU compute. Have to believe something big is going on behind the scenes here. That said, wouldn't be surprised if the truth was somewhere in between cloud-deployed and locally deployed, particularly on the way up to the asymptotic tail of the model performance curve.
  • smoldesu 4y ago
    What would a "leap frog" look like, in your mind? I'm struggling to imagine how they're better positioned than the competition, especially after llama.cpp showed us that inference acceleration works with everything from AVX2 to ARM NEON. Compared to Nvidia (or even Microsoft and ONNX/OpenAI), Apple is somewhat empty-handed here. They're not out of the game, but I genuinely see no path for them to dominate "everyone".
    • yunwal 4y ago
      My guess is a leapfrog would have more to do with how LLMs are integrated into an operating system, rather than just coming out with a better model. I don’t think we’re gonna get a substantially more capable LLM than GPT-4 anytime soon, but fine-tuning it to sit on top of the core of an operating system could yield results.
      • smoldesu 4y ago
        Feels like Microsoft already beat them to the punch. Their ONNX toolkit has better ARM optimization than Apple's own Pytorch patches, and their collaboration with OpenAI places them pretty far ahead of the research curve. I'm convinced Microsoft could out-maneuver Apple on local or remote AI functionality, if they wanted to.
  • ohgodplsno 4y ago
    M1 GPUs are barely real-world tested, alright chips. They're far from being a sleeping giant.
• GavinB 4y ago
  I'd suspect it will actually accelerate moving everything into the cloud. If your entire business is in the cloud, you can give an AI access to everything with a single sign or some passwords. If half is on the cloud and half is local, that's very annoying to have all in-context for your AI assistant. And there's no way we're getting everything locally stored again at this point!
  • bugglebeetle 4y ago
    Right, this is why StabilityAI is getting in bed with Amazon, so private, fine-tuned models can operate on all your data sitting out there in S3 buckets or whatever.
  • brodo 4y ago
    I really doubt that someone responsible for security would give anyone (or anybot) access to everything.
• qingdao99 4y ago
  > But once one exists you can use it to make more relatively cheaply by generating training data. Is that a real technique? Why not just shrink down the model itself directly somehow, is that not possible?
  • MacsHeadroom 4y ago
    That's what pruning is, but it's not that straight forward and has limits. Finetuning a smaller model on the output of a larger one is much more flexible and reliable. GPT 3.5 is probably a 13B Curie finetuned on the output of full size GPT-3 175B, to give you an idea of the technique. That is smaller than the third smallest StableLM and the same size as LLaMA-13B which can run at useful speeds off of a smart phone CPU.
    • int_19h 4y ago
      > GPT 3.5 is probably a 13B Curie finetuned on the output of full size GPT-3 175B What is the basis for this assessment?
      • iliane5 4y ago
        GPT-3.5 is much worse at "complex" cognitive tasks than Davinci (175B), which seem to indicate that it's a smaller model. It's also much faster than Davinci and costs the same as Curie via the API.
        • int_19h 4y ago
          It's clearly a smaller model, but I'm very skeptical that it is 13B. It is much more lucid than any 13B model out in the wild. I find it much more likely that they used additional tricks to scale down hardware requirements and thereby bring the price down so much (int4 quantization, perhaps? that alone would mean 4x less hardware utilization for the same query, if they were using float16 for older models, which they probably were)
          • 4y ago
• losvedir 4y ago
  This doesn't seem technically feasible to me. The state of the art will for a long time require a lot more hardware to run than it's available on a consumer device. Beyond which, inference also benefits from parallelization, not just training, so being able to batch requests is a benefit, and more likely when access is offered via an API.
  • FL33TW00D 4y ago
    This isn't correct. It's 100% feasible to run very capable models on consumer devices. I wrote up a feasibility investigation last year: https://fleetwood.dev/posts/a-case-for-client-side-machine-learning https://fleetwood.dev/posts/a-case-for-client-side-machine-l...
    • frozenlettuce 4y ago
      I ran alpaca 7B in a 2015 Acer laptop (it was slow, but worked!)
    • crazygringo 4y ago
      But there's a big difference between running "very capable" models that use most available resources and consume tons of power and run slowly... ...versus running the best models available, in a few seconds, without using up the memory the main app you're using needs for running. These are all mainly going to be run remotely for general consumer usage for quite a while I think.
      • FL33TW00D 4y ago
        How many applications on your machine right now are GPU accelerated? There is a chunk of silicon that spends almost all its time doing nothing.
        • crazygringo 4y ago
          Sure but it's not that powerful. It's not a gaming rig. Also it had better spend almost all its time doing nothing or it would kill my battery. Same as with my CPU. The main point still stands though -- it's pretty useless if it takes a couple minutes to do what a server can do in a couple seconds.
• fshbbdssbbgdd 4y ago
  Most of your bullets sound like arguments that local models will win. > Often have network effects where the size of the network causes natural monopoly feedback loops. This one in particular sounds like an argument that remote models will win.
  • max51 4y ago
    I don't that's true in the context of businesses because they won't want their data to be leaked and/or used for other clients. The more data from your company you can feed the AI, the more productive it will be for you. I'm not just talking about semi-public documentation, but also things like emails, meeting transcript, internal tools APIs, employee details, etc. If the AI service provider uses your data to help better train their AI, it will be blacklisted by most companies. If you keep them in silos, the centralisation will offer almost no benefit while still being a very high privacy risk. The only benefit they get is that it allows them to demo it and see it's potential, but no serious business will adopt it unless you also provide a self-hosted solution. I think the only people who will truly benefit from using cloud services as a long term solution are personal users and companies too small to afford the initial cost of the hardware.
    • goatlover 4y ago
      That seems hard to believe for businesses which already rely on Office, Teams and Sharepoint, since Microsoft will be making its version of ChatGPT available for all its products, and the integration will be too hard to pass up on.
      • max51 4y ago
        Microsoft is in a different situation because everyone is already forced to trust them with their OS and o365. For better or for worse, there are no current alternatives to Windows and the office suite for most businesses. If you already login to your OS with a Microsoft account and process your data in Excel, adding an AI tool on top of it is not a big jump. Very few others are in this situation. For every other AI service providers, good fucking luck getting clients to trust you. I expect we will see a lot AI services that offer a cheap and easy to use cloud AI subsidized by a very expensive self-hosted version. I also expect a lot of data leaks and many high profile incidents where an AI creates a document or code that includes sensitive data from someone else (hard coded passwords, API keys, etc.). Even for a large company like Autodesk or Adobe, you might trust them with your engineering drawings and your new product design, but would you feel comfortable uploading your code base for internal tools, employee files, email communications, etc. to them? It's gonna be a hard no for a lot of businesses
• s3p 4y ago
  What does "stage plays on the radio” mean?
  • what_ever 4y ago
    Trying to be fancy to say things won't be running locally. The orchestra/performers won't be live in front of you but on the radio.
    • earthboundkid 4y ago
      That is not what that idiom means.
  • crucialfelix 4y ago
    When radio first started, people read plays written for the stage, because that's what they knew and what they had. Later people learned to write for the medium and make radio native entertainment. Same thing happened when TV arrived. They did live versions of the radio entertainment on a set in front of a camera.
  • deleted 4y ago
    [deleted]
• deleted 4y ago
  [deleted]
• burtonator 4y ago
  We're stuck here for a while due to the size, and cost, of the larger models. The main reason I want a non-cloud LLM is that I want one that's unaligned. I know I'm not a criminal and I want to stop being reprimanded by GPT4. What I'm most interested here is fine tuning the model with my own content. That could be super valuable especially if we could get it to fact check itself, which you could with a vector database.
  • shostack 4y ago
    What's been so interesting with the explosion of this has been how prominently the corporately-driven restrictions have been highlighted in news and such. People are getting a good look in very easy to understand terms at the foundational stage at how limiting the future is to have this just be another big tech controlled thing.
    • barking_biscuit 4y ago
      I know we want things that are insanely powerful and totally unrestricted, and because we want them, I think we'll get them. And then I genuinely think this tech is going to end in tears.
    • ChatGTP 4y ago
      Don’t the general populace also have valid concerns about more powerful models and newer architectures being able to do damage ?
  • WhiteNoiz3 4y ago
    They have said that the alignment actually hurts the performance of the models. Plus for creative applications like video games or novels, you need an unaligned model otherwise it just produces "helpful" and nice characters.
    • yeck 4y ago
      The character simulacrum used by an LLM tends to be the result of "system" prompts that set by the service you are using. GPT-N isn't exactly trained to be helpful and nice, but ChatGPT has system prompts describing the character it should be performing as. If you work with just GPT-4, you can get more zany outputs. That said, OpenAI does use RLHF, which does bias the model away from raw internet madness and something that OpenAI wanted at the time of training. A lot of models haven't gone through rigorous RLHF, though. As a side note, RLHF might be the best alignment technique we currently have in practice, but it is not decisive. It has been noted in multiple experiments that RLHF can just train a model in how to trick the human reviewer, if tricking is easier in practice than doing a think the human review wanted. So this isn't even really seen as aligning a model by alignment researchers. At least not an approach that can scale with the increasingly intelligence AI models.
• skybrian 4y ago
  This is sort of like saying the world wide web is a fad. Many people made that argument, but a lot of desktop apps got replaced by websites even though they were supposedly inferior. ChatGPT works fine as a website and you don’t need to buy a new computer to run it. You can access your chat history from any device. For many purposes, the only real downside is the subscription fee. If LLM’s become cheaper to run, websites will be cheaper to run, and there will be lower-cost competition. Maybe even cheap enough to give away for free and make money from advertising?
  • zirgs 4y ago
    The other downside starts with an "As an AI language model..."
    • ModernMech 4y ago
      And ends with "...I'm afraid I can't do that."
    • skybrian 4y ago
      Which is why I said "many purposes." It doesn't work for everything.
• precompute 4y ago
  This is mostly why the future of computation only makes sense monetarily if you have everyone shift to a thin client. So, banning GPUs is likely considered a "necessary evil" by the BigTech cognoscenti for accomplishing that goal.
• WhiteNoiz3 4y ago
  I agree - I think for security and privacy we need it to be on-device (either that or there needs to be end to end encryption with gaurantees that data won't be captured for training). There are tons of useful applications that require sensitive personal information (or confidential business information) to be passed in prompts - that becomes a non issue if you can run it on device. I think there will be a lot of incentive to figure out how to make these models more efficient. Up until now, there's been no incentive for the OpenAI's and the Googles of the world to make the models efficient enough to run on consumer hardware. But once we have open models and weights there will be tons of people trying to get them running on consumer hardware. I imagine something like an AI specific processor card that just runs LLMs and costs < $3000 could be a new hardware category in the next few years (personally I would pay for that). Or, if apple were to start offering a GPT3.5+ level LLM built in that runs well on M2 or M3 macs that would be strong competition and a pretty big blow against the other tech companies.
  • zamnos 4y ago
    That hardware's gonna look a lot like ASIC Bitcoin miners if an architecture to replace LLMs is popularized. General-enough purpose computing ain't going away for a long time.
• vsareto 4y ago
  I think you'll be paying a cloud provider instead of an API company, but it'll still be a monthly service charge for the service I don't see small/medium companies getting into acquiring hardware for AI
• circuit10 4y ago
  At least for now, good LLMs need GPUs that cost tens of thousands, so a cloud API is the only reason option. It definitely makes a lot of sense
• jrm4 4y ago
  I think it may be naive that people believe that the deciding factor on how these things are used is likely to be "chip speed." or "efficiency on the machine." I wish we were in that world; but it more likely seems like it would be "Which company jumps ahead quickest to get mindshare on a popular AI related thing, and then is able to ride scale to dominate the space?" REALLY hope I end up being wrong here; the fact that so many models are already out there does give me some hope.
• lairv 4y ago
  This doesn't seem that obvious to me, serving LLMs through an API allows to have highly optimized inference with stuff like TensorRT and batched inference while you're stuck with batch size = 1 when processing locally. LLMs doesn't even require full real-time inference, there are applications like VR or camera stuff where you need real-time <10ms inference, but for any application of LLMs 200-500ms is more than fine For the users, running LLMs locally means more battery usage and significant RAM usage. The only true advantage is privacy but this isn't a selling point for most people
  • pornel 4y ago
    You're still thinking in terms of what APIs would be used for, rather than what local computation enables. For example, I'd like an AI to read everything I have on screen, so that I can ask at any time "why is that? Explain!" without having to copy paste the data and provide the whole context to a Google-like app. But without privacy guarantee (and I mean technical one, not a pinky promise to be broken when VC funding runs out) there's no way I'd feed everything into an AI.
  • brucethemoose2 4y ago
    We are very close to optimized ML frameworks on consumer hardware. And TBH most modern devices have way more RAM than they need, and go to great lengths to just find stuff to do with it. Hardware companies also very much like the idea of a heavy consumer applications.
• ForHackernews 4y ago
  ? All software is sold as SaaS today, because it's more profitable. The same will be true for LLMs.
  • Rastonbury 4y ago
    LLM seems more akin to AWS, than a SaaS, companies will create products upon LLMs like how companies rely on AWS to support their products. The build vs buy calculus may tip heavily towards build once they can run on device with good user experience, no need to pay for cloud compute any longer.
    • ForHackernews 4y ago
      > The build vs buy calculus may tip heavily towards build once they can run on device with good user experience Hahahahahaha... oh wait, you're serious? Let me laugh even harder. Have you used any commercial software in the last 25 years? Garbage web apps have replaced very nice, performant local applications across the board. My stupid fitness tracker app (that should be a 10 MB sqlite DB) instead fails to even open without an internet connection. Is your theory that companies will suddenly decide they hate getting money and love paying money for developers to create great user experiences? I wish I lived in your world.
• coffeebeqn 4y ago
  LLMs also need persistent storage for best performance. Otherwise you’re always starting at square one
• brodo 4y ago
  All modern Apple hardware has dedicated AI chips. I bet they are already are working on an LLM-based Siri.