5 ms·

The point about the economics of running these models is an important one that slides under the radar a lot of times. The training costs for large language mode

The point about the economics of running these models is an important one that slides under the radar a lot of times. The training costs for large language models like GPT are enormous, and the inference costs are substantial too. Right now things like ChatGPT are very cool parlor tricks, but there's absolutely no way to justify them in terms of the economics of running the service today. Obviously this is all going to change in the near to mid future: innovation will drive down costs of both training and inference, and the models will be monetized in ways that bring in more revenue. But I don't think the long term economics are obvious to anyone, including Google or OpenAI. It's really hard to predict how much more efficient we'll get at training/serving these models as most of the gains there are going to come from improved model architectures, and it's very difficult to predict how much room for improvement there is there. Google (and Microsoft, Yandex, Baidu, etc.) know how to index the web and serve search queries to users at an extremely low cost per query that can be compensated by ads that make fractions of a cent per impression. It's not obvious at all if that's possible with LLMs, or if it possible, what the timescale is to get to a place where the economics make sense and the service actually makes money.

dan-robertson 4y ago
1. How much did it cost to train ChatGPT/GPT3? The only estimate I’ve seen was not enormous in the grand scheme of things (eg more money than I have but less than Google have stuck down the back of the sofa). I think that number didn’t count training precursor models or paying for people to come up with the models/training data/infra. 2. Don’t Google have specialised hardware for training neural networks? If the costs of training/inference are very significant won’t Google (with their ASIC and hardware design team) have a significant advantage? It seems to me that their AI hardware was developed because they saw this problem coming a long way off.
- vineyardmike 4y ago
  1. We don't know the exact cost, but its well into the millions. When Microsoft invested in OpenAI, it did so in the form of ~$500M in azure credits, so they're expecting at least that much in compute spend. Another company estimated that GPT-3 alone cost the equivalent of $4.5M in cloud costs (ignoring OpenAI's other models). 2. Yes. Yes they will/are developing custom silicon that will likely be a significant advantage here. GPU costs were always crazy, and many companies are designing AI chips now. Even the iPhone chips have custom AI Cores. We'll see if Azure releases AI co-processors to aid them...
  - somsak2 4y ago
    to clarify -- Microsoft invested $1bn in 2019, half of which was Azure credits. The other half is cash. Since then, they invested a further $2bn.
    - galaxyLogic 4y ago
      And META reportedly has spent billions on the meta-verse. It's kind of interesting that language models are now making meta-tech look outdated. Then again maybe language models will create the metaverse
      - ericd 4y ago
        Did metaverse ever not look like a VR rehash of second life? I’m genuinely curious, I‘ve had VR headsets since Oculus DK1 I’ve never seen anything very compelling on the VR persistent alternate reality front.
        fennecfoxy 4y ago
        Metaverse wanted to do what VRChat had already done tbh. VRChat just needs polish & moderation/to create a proper platform. Realistically VR clients needs to pose as browsers to load vr:// links which can be connected to objects/actions in VR ie <portal color="green" size="200,200" pos="122.469,79.420,1337.25" href="vr://some-online-shop-showroom.domain.tld" /> That way it's done in an open, browsable way compatible with the expectations that we've gained from regular web experiences. Ie you have your VR home, there's a "bookmark" door you can walk through to go to Amazon's showroom, you search for a particular product and it lets you walk around and pick-up and examine the various options, you can then jump to a particular brand's individual showroom, etc. Some people might feel that's a little dystopian I suppose but I think it's cool.
      - hansvm 4y ago
        Maybe we need to progress tech a bit before we try to move up a level to meta tech.
  - 4y ago
james-revisoai 4y ago
Inference costs are substantial, but not insurmountable, especially in the endgame. A decent chunk of the tech community can already run smaller T5 Flan models or the 6B EleutherAI LLM GPT-J (and the likely similarly sized upcoming Open Assistant) on their own machines, at decent inference speed (< 0.2 s per token, which is ok enough most of the time). By 2027 or so the majority of consumers will likely exceed that point. What happens when models are updated every day automatically and you can run all your search and answer tasks on your local machine? With GPT-J - which is unreliable and aging now - I can already learn core intro to psychology topics more accurately/faster than with google or wikipedia, and I can do that offline. That's a cherry picked use case, but imagine the future. Why would you use something that has ads when you can run it locally, and perhaps even augment it with your documents/files? In the end game this is where Google is in the same place as Kodak in my opinion right now. Sure it's $0.01 or more a search for OpenAI now, but it won't stay that way (they reduced prices by 66% half a year ago), and at that rate you can already make the unit economics work anyhow as a startup.
- pavelstoev 4y ago
  It is a loosing uphill battle to reduce the operational costs as deep learning models get larger and more complex. Nvidia CEO says hardware alone will not be able to address the growing compute demand. The solution is computational optimization which is what we do.
  - brookst 4y ago
    Yes, and also HW optimization further up the stack.
somenameforme 4y ago
GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consumer level hardware. A system with 4 RTX 3090s (< $4k) gets results in about 16 seconds per query. The proverbial 'some guy on Twitter'[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university). If these technologies end up as anything more than a 'novelty of the year' type event, then I expect to see them able to be run locally on phones within a decade. There will be a convergence between hardware improving and the software getting more efficient. [1] - https://github.com/THUDM/GLM-130B https://github.com/THUDM/GLM-130B [2] - https://twitter.com/alexjc/status/1617152800571416577 https://twitter.com/alexjc/status/1617152800571416577
- brookst 4y ago
  Agree, but much less than 10 years. Now that the transformer is establishing itself as the model, we’ll see dedicated HW acceleration for transformers, encoders, decoders, etc. I will eat my hat* if we don’t see local inference for 200B+ parameters within 5 years. * I don’t own a hat
  - senttoschool 4y ago
    Which chip company do you think will benefit from the move to transformers?
    - flakeoil 4y ago
      No-one in particular. It will be integrated in the processors (the intels/AMDs on PC, the Mx on Macs and the Qualcomms, Mediateks and Samsungs on phones). Not much different from video and graphics accelerators being integrated today, or DSP focused instructions in the instruction set. They just have to do it to stay relevant.
  - 4y ago
alfor 4y ago
1- the chips are not efficient currently (graphic card reused as neural net) 10X-100X gain 2- Moore’s Laws 3- Algorithm/architecture improvement
- DannyBee 4y ago
  1. Im not sure who you believe will produce these chips, or who will use them. You are correct that specialized inference chips will get you a 10x gain. So what? If you want several million of them in a datacenter, that's a tall order. That's on top of A. the hundreds of millions it will take to get a design to production. B. The complete and total lack of allocation at anybody who could make you chips, except at very very very high cost, if at all - have you forgotten that automakers still can't get cheap chips made on older processes? Most allocation of newer processes is bought out for years. While there is some ability to get things made at newer process, building a chip for 7nm is 10-100x as expensive as say 45nm. C. The fact that someone has to be willing to build, plan, and execute putting them in datacenters. This will all happen, but like, everyone just assumes the hard part is the chip inefficiency. We already can make designs that are ~10x more efficient at inference (though it depends on if you mean power or speed or what). The fact that there are not millions in datacenters should tell you something about the difficulty and economics of accomplishing this. People aren't sitting around twiddling their thumbs. If Microsoft or Google or anyone could make themselves a "100x better cloud for AI", they would do it. 2. Dead. Dennard scaling went out the window years ago. Other scaling has mostly followed. The move to specialization you see and push to higher frequencies is because its all dead. 3. Will take a while. The economics of this kind of thing sucks. It will not change instantly, there isn't the capability to make it happen.
  - ddalex 4y ago
    I think you missed the TPU, which is a Google chip that gets you the 10x in inference, and there are millions of them ALREADY in the datacenters, designed, fabricated and installed. You can use one, for free, with Colab
    - DannyBee 4y ago
DannyBee 4y ago
+1 For a group of people on a site frequented by startup people, did nobody read the terms of MS's investment into OpenAI? "Microsoft would reportedly put down $10 billion for a 75% share of the profits that OpenAI earns until the money on the investment is paid back. Then when Microsoft breaks even on the $10 billion investment, they would get a 49% stake in OpenAI. These are not the terms you would take if, tomorrow, or even two years from now, you were about to be wildly profitable because everything was about to be so easy. These are the terms you would take if Microsoft was the only hope you had of getting the resources you need, or if getting somewhere was going to be very expensive and you needed to defray costs. Honestly, with the level of optimism in the rest of this thread about how easy this will all be, they would probably be profitable enough to just buy MS in like 3 years , and wouldn't have needed investment at all!
- svnt 4y ago
  Not having read the terms in any detail, I will say this: it can be very easy to not report profits for a very long time.
  - aardvarkr 4y ago
    The point was not that the profits are long tailed but rather if openai thought there were massive future profits and the potential to become a google killer then they wouldn’t give away half of their company today for $10B.
    - chii 4y ago
      But how do you know that they didn't predict that it is only by having a $10 billion that they _can_ become the next google killer, because of the initial cost outlay, and the inevitable fight back from google (which would then require a warchest, of which i'm sure this 10 billion is a part of)?
      - UncleMeat 4y ago
        Plenty of people have already said "Google is dead" in certain terms. If somebody needs ten billion dollars in additional work and investment to make that a reality, how certain can they be? OpenAI has 375 employees (according to Google). At 300,000 a head thats 110M per year in compensation. Let's say that their compute costs are enormous and go to 200M in expenses per year. 10B is fifty years of expenses. So if they need 10B in investment it becomes obvious that they believe that they have to change something about their business in a fundamental way. Maybe that is going to be enough, but if it is so certain it becomes hard to believe that they'd need this kind of investment.

dan-robertson 4y ago

1. How much did it cost to train ChatGPT/GPT3? The only estimate I’ve seen was not enormous in the grand scheme of things (eg more money than I have but less than Google have stuck down the back of the sofa). I think that number didn’t count training precursor models or paying for people to come up with the models/training data/infra. 2. Don’t Google have specialised hardware for training neural networks? If the costs of training/inference are very significant won’t Google (with their ASIC and hardware design team) have a significant advantage? It seems to me that their AI hardware was developed because they saw this problem coming a long way off.

vineyardmike 4y ago
1. We don't know the exact cost, but its well into the millions. When Microsoft invested in OpenAI, it did so in the form of ~$500M in azure credits, so they're expecting at least that much in compute spend. Another company estimated that GPT-3 alone cost the equivalent of $4.5M in cloud costs (ignoring OpenAI's other models). 2. Yes. Yes they will/are developing custom silicon that will likely be a significant advantage here. GPU costs were always crazy, and many companies are designing AI chips now. Even the iPhone chips have custom AI Cores. We'll see if Azure releases AI co-processors to aid them...
- somsak2 4y ago
  to clarify -- Microsoft invested $1bn in 2019, half of which was Azure credits. The other half is cash. Since then, they invested a further $2bn.
  - galaxyLogic 4y ago
    And META reportedly has spent billions on the meta-verse. It's kind of interesting that language models are now making meta-tech look outdated. Then again maybe language models will create the metaverse
    - ericd 4y ago
      Did metaverse ever not look like a VR rehash of second life? I’m genuinely curious, I‘ve had VR headsets since Oculus DK1 I’ve never seen anything very compelling on the VR persistent alternate reality front.
      - fennecfoxy 4y ago
        Metaverse wanted to do what VRChat had already done tbh. VRChat just needs polish & moderation/to create a proper platform. Realistically VR clients needs to pose as browsers to load vr:// links which can be connected to objects/actions in VR ie <portal color="green" size="200,200" pos="122.469,79.420,1337.25" href="vr://some-online-shop-showroom.domain.tld" /> That way it's done in an open, browsable way compatible with the expectations that we've gained from regular web experiences. Ie you have your VR home, there's a "bookmark" door you can walk through to go to Amazon's showroom, you search for a particular product and it lets you walk around and pick-up and examine the various options, you can then jump to a particular brand's individual showroom, etc. Some people might feel that's a little dystopian I suppose but I think it's cool.
    - hansvm 4y ago
      Maybe we need to progress tech a bit before we try to move up a level to meta tech.
- 4y ago

james-revisoai 4y ago

Inference costs are substantial, but not insurmountable, especially in the endgame. A decent chunk of the tech community can already run smaller T5 Flan models or the 6B EleutherAI LLM GPT-J (and the likely similarly sized upcoming Open Assistant) on their own machines, at decent inference speed (< 0.2 s per token, which is ok enough most of the time). By 2027 or so the majority of consumers will likely exceed that point. What happens when models are updated every day automatically and you can run all your search and answer tasks on your local machine? With GPT-J - which is unreliable and aging now - I can already learn core intro to psychology topics more accurately/faster than with google or wikipedia, and I can do that offline. That's a cherry picked use case, but imagine the future. Why would you use something that has ads when you can run it locally, and perhaps even augment it with your documents/files? In the end game this is where Google is in the same place as Kodak in my opinion right now. Sure it's $0.01 or more a search for OpenAI now, but it won't stay that way (they reduced prices by 66% half a year ago), and at that rate you can already make the unit economics work anyhow as a startup.

pavelstoev 4y ago
It is a loosing uphill battle to reduce the operational costs as deep learning models get larger and more complex. Nvidia CEO says hardware alone will not be able to address the growing compute demand. The solution is computational optimization which is what we do.
- brookst 4y ago
  Yes, and also HW optimization further up the stack.

somenameforme 4y ago

GLM-130B[1] is something comparable to GPT-3. It's a 130billion parameter model vs GPT-3's 175 billion, and it can comfortably run on current-gen high end consumer level hardware. A system with 4 RTX 3090s (< $4k) gets results in about 16 seconds per query. The proverbial 'some guy on Twitter'[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university). If these technologies end up as anything more than a 'novelty of the year' type event, then I expect to see them able to be run locally on phones within a decade. There will be a convergence between hardware improving and the software getting more efficient. [1] - https://github.com/THUDM/GLM-130B https://github.com/THUDM/GLM-130B [2] - https://twitter.com/alexjc/status/1617152800571416577 https://twitter.com/alexjc/status/1617152800571416577

brookst 4y ago
Agree, but much less than 10 years. Now that the transformer is establishing itself as the model, we’ll see dedicated HW acceleration for transformers, encoders, decoders, etc. I will eat my hat* if we don’t see local inference for 200B+ parameters within 5 years. * I don’t own a hat
- senttoschool 4y ago
  Which chip company do you think will benefit from the move to transformers?
  - flakeoil 4y ago
    No-one in particular. It will be integrated in the processors (the intels/AMDs on PC, the Mx on Macs and the Qualcomms, Mediateks and Samsungs on phones). Not much different from video and graphics accelerators being integrated today, or DSP focused instructions in the instruction set. They just have to do it to stay relevant.
- 4y ago

alfor 4y ago

1- the chips are not efficient currently (graphic card reused as neural net) 10X-100X gain 2- Moore’s Laws 3- Algorithm/architecture improvement

DannyBee 4y ago
1. Im not sure who you believe will produce these chips, or who will use them. You are correct that specialized inference chips will get you a 10x gain. So what? If you want several million of them in a datacenter, that's a tall order. That's on top of A. the hundreds of millions it will take to get a design to production. B. The complete and total lack of allocation at anybody who could make you chips, except at very very very high cost, if at all - have you forgotten that automakers still can't get cheap chips made on older processes? Most allocation of newer processes is bought out for years. While there is some ability to get things made at newer process, building a chip for 7nm is 10-100x as expensive as say 45nm. C. The fact that someone has to be willing to build, plan, and execute putting them in datacenters. This will all happen, but like, everyone just assumes the hard part is the chip inefficiency. We already can make designs that are ~10x more efficient at inference (though it depends on if you mean power or speed or what). The fact that there are not millions in datacenters should tell you something about the difficulty and economics of accomplishing this. People aren't sitting around twiddling their thumbs. If Microsoft or Google or anyone could make themselves a "100x better cloud for AI", they would do it. 2. Dead. Dennard scaling went out the window years ago. Other scaling has mostly followed. The move to specialization you see and push to higher frequencies is because its all dead. 3. Will take a while. The economics of this kind of thing sucks. It will not change instantly, there isn't the capability to make it happen.
- ddalex 4y ago
  I think you missed the TPU, which is a Google chip that gets you the 10x in inference, and there are millions of them ALREADY in the datacenters, designed, fabricated and installed. You can use one, for free, with Colab
  - DannyBee 4y ago

DannyBee 4y ago

+1 For a group of people on a site frequented by startup people, did nobody read the terms of MS's investment into OpenAI? "Microsoft would reportedly put down $10 billion for a 75% share of the profits that OpenAI earns until the money on the investment is paid back. Then when Microsoft breaks even on the $10 billion investment, they would get a 49% stake in OpenAI. These are not the terms you would take if, tomorrow, or even two years from now, you were about to be wildly profitable because everything was about to be so easy. These are the terms you would take if Microsoft was the only hope you had of getting the resources you need, or if getting somewhere was going to be very expensive and you needed to defray costs. Honestly, with the level of optimism in the rest of this thread about how easy this will all be, they would probably be profitable enough to just buy MS in like 3 years , and wouldn't have needed investment at all!

svnt 4y ago
Not having read the terms in any detail, I will say this: it can be very easy to not report profits for a very long time.
- aardvarkr 4y ago
  The point was not that the profits are long tailed but rather if openai thought there were massive future profits and the potential to become a google killer then they wouldn’t give away half of their company today for $10B.
  - chii 4y ago
    But how do you know that they didn't predict that it is only by having a $10 billion that they _can_ become the next google killer, because of the initial cost outlay, and the inevitable fight back from google (which would then require a warchest, of which i'm sure this 10 billion is a part of)?
    - UncleMeat 4y ago
      Plenty of people have already said "Google is dead" in certain terms. If somebody needs ten billion dollars in additional work and investment to make that a reality, how certain can they be? OpenAI has 375 employees (according to Google). At 300,000 a head thats 110M per year in compensation. Let's say that their compute costs are enormous and go to 200M in expenses per year. 10B is fifty years of expenses. So if they need 10B in investment it becomes obvious that they believe that they have to change something about their business in a fundamental way. Maybe that is going to be enough, but if it is so certain it becomes hard to believe that they'd need this kind of investment.