5 ms·

1. How much did it cost to train ChatGPT/GPT3? The only estimate I’ve seen was not enormous in the grand scheme of things (eg more money than I have but less th

1. How much did it cost to train ChatGPT/GPT3? The only estimate I’ve seen was not enormous in the grand scheme of things (eg more money than I have but less than Google have stuck down the back of the sofa). I think that number didn’t count training precursor models or paying for people to come up with the models/training data/infra. 2. Don’t Google have specialised hardware for training neural networks? If the costs of training/inference are very significant won’t Google (with their ASIC and hardware design team) have a significant advantage? It seems to me that their AI hardware was developed because they saw this problem coming a long way off.

vineyardmike 4y ago
1. We don't know the exact cost, but its well into the millions. When Microsoft invested in OpenAI, it did so in the form of ~$500M in azure credits, so they're expecting at least that much in compute spend. Another company estimated that GPT-3 alone cost the equivalent of $4.5M in cloud costs (ignoring OpenAI's other models). 2. Yes. Yes they will/are developing custom silicon that will likely be a significant advantage here. GPU costs were always crazy, and many companies are designing AI chips now. Even the iPhone chips have custom AI Cores. We'll see if Azure releases AI co-processors to aid them...
- somsak2 4y ago
  to clarify -- Microsoft invested $1bn in 2019, half of which was Azure credits. The other half is cash. Since then, they invested a further $2bn.
  - galaxyLogic 4y ago
    And META reportedly has spent billions on the meta-verse. It's kind of interesting that language models are now making meta-tech look outdated. Then again maybe language models will create the metaverse
    - ericd 4y ago
      Did metaverse ever not look like a VR rehash of second life? I’m genuinely curious, I‘ve had VR headsets since Oculus DK1 I’ve never seen anything very compelling on the VR persistent alternate reality front.
      - fennecfoxy 4y ago
        Metaverse wanted to do what VRChat had already done tbh. VRChat just needs polish & moderation/to create a proper platform. Realistically VR clients needs to pose as browsers to load vr:// links which can be connected to objects/actions in VR ie <portal color="green" size="200,200" pos="122.469,79.420,1337.25" href="vr://some-online-shop-showroom.domain.tld" /> That way it's done in an open, browsable way compatible with the expectations that we've gained from regular web experiences. Ie you have your VR home, there's a "bookmark" door you can walk through to go to Amazon's showroom, you search for a particular product and it lets you walk around and pick-up and examine the various options, you can then jump to a particular brand's individual showroom, etc. Some people might feel that's a little dystopian I suppose but I think it's cool.
    - hansvm 4y ago
      Maybe we need to progress tech a bit before we try to move up a level to meta tech.
- dan-robertson 4y ago
  Yeah I figured that most of the costs would be from iterating and training many models over time. $4.5m is surely not the kind of spending that will make google nervous or will give OpenAI much of a moat.

vineyardmike 4y ago

1. We don't know the exact cost, but its well into the millions. When Microsoft invested in OpenAI, it did so in the form of ~$500M in azure credits, so they're expecting at least that much in compute spend. Another company estimated that GPT-3 alone cost the equivalent of $4.5M in cloud costs (ignoring OpenAI's other models). 2. Yes. Yes they will/are developing custom silicon that will likely be a significant advantage here. GPU costs were always crazy, and many companies are designing AI chips now. Even the iPhone chips have custom AI Cores. We'll see if Azure releases AI co-processors to aid them...

somsak2 4y ago
to clarify -- Microsoft invested $1bn in 2019, half of which was Azure credits. The other half is cash. Since then, they invested a further $2bn.
- galaxyLogic 4y ago
  And META reportedly has spent billions on the meta-verse. It's kind of interesting that language models are now making meta-tech look outdated. Then again maybe language models will create the metaverse
  - ericd 4y ago
    Did metaverse ever not look like a VR rehash of second life? I’m genuinely curious, I‘ve had VR headsets since Oculus DK1 I’ve never seen anything very compelling on the VR persistent alternate reality front.
    - fennecfoxy 4y ago
      Metaverse wanted to do what VRChat had already done tbh. VRChat just needs polish & moderation/to create a proper platform. Realistically VR clients needs to pose as browsers to load vr:// links which can be connected to objects/actions in VR ie <portal color="green" size="200,200" pos="122.469,79.420,1337.25" href="vr://some-online-shop-showroom.domain.tld" /> That way it's done in an open, browsable way compatible with the expectations that we've gained from regular web experiences. Ie you have your VR home, there's a "bookmark" door you can walk through to go to Amazon's showroom, you search for a particular product and it lets you walk around and pick-up and examine the various options, you can then jump to a particular brand's individual showroom, etc. Some people might feel that's a little dystopian I suppose but I think it's cool.
  - hansvm 4y ago
    Maybe we need to progress tech a bit before we try to move up a level to meta tech.
dan-robertson 4y ago
Yeah I figured that most of the costs would be from iterating and training many models over time. $4.5m is surely not the kind of spending that will make google nervous or will give OpenAI much of a moat.