3 ms·

You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L59 https://github.com/fauxpilot/fauxpi

You details of what is required can be found per project. eg. https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L59 https://github.com/fauxpilot/fauxpilot/blob/main/setup.sh#L5... /shrug If you don't want to run it locally, don't. If the hardware you have doesn't support it, don't. ...but to say it's not possible is, again, bluntly, wrong.

throwifasd 4y ago
yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao
- adam_arthur 4y ago
  Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.
  - throwifasd 4y ago
    This post [0] is a good primer and they are talking about a 20b model gpt3 has 175b. Their other posts go more into details but yes, it really is a massive operation. Some hard facts from here [1] talking about BLOOM with 175b parameters.. >> Installing the full 175B version is a challenge though as it requires around 350GB of GPU VRAM, which is not something one can easily afford. But hey why deploy a model like chatGPT at all when you guys can be confidently incorrect all by yourselves. [0] https://nlpcloud.com/deploying-gpt-neox-20-production-focus-deepspeed.html https://nlpcloud.com/deploying-gpt-neox-20-production-focus-... [1] https://nlpcloud.com/chatgpt-open-source-alternatives.html https://nlpcloud.com/chatgpt-open-source-alternatives.html
    - adam_arthur 4y ago
      So it sounds like this is a question of loading the model into VRAM, and not a question of the cost of a single query. I assume once a model is loaded, many queries can be serviced by that model quickly. There's nothing incorrect about my assertion. If it were to actually take many GPUs to service one query, then there is no mass scale cost viable consumer product. That's just a clear economic fact. Regardless if a model could be theoretically spun up in a cost inefficient manner. And even 100s of GB of VRAM is not far off from consumer hardware. Look at how quickly graphics ram has expanded over time. About ~10x in ~10 years for high end cards, at a cursory glance at various Nvidia cards. At the same trajectory we could see a 400GB vram card within the next decade (though lots of assumptions)
      - throwifasd 4y ago
        You really are like a chatbot... look at the last three node sizes and the density of ram in them. It's not gonna happen as fast as you dream about it especially not with the discounts of the last Gens. The hope is to go to fp4 if you want to run it on consumer hardware and we are still not talking to about 2-3 cards. Why not at least try to Google before hammering down on stupid and uninformed hot takes?
      - visarga 4y ago
        > I assume once a model is loaded, many queries can be serviced by that model quickly. Depends. If you have room to load the whole model, yes. If you need to swap in and out parts of the model, then it matters if you have enough RAM.
  - jacobr1 4y ago
    One thing to consider, there is considerable asymmetry between the high training costs and the lower (relatively) operating costs. > Perhaps ChatGPT is run with massive operating losses and its all VC subsidized. I'm not in the know either, but the current access is providing both marketing awareness, and significant human training/feedback to be used for improvements on future commercial projects. It could certainly be the case that that benefits outweigh costs compared to the classic, dumber "get eyeballs now, monetize later" strategy.
  - visarga 4y ago
    $3M/day is what I heard
space_fountain 4y ago
The largest model is an order of magnitude smaller than gpt3 right? Does it perform similarly to copilot?
visarga 4y ago
You need something like an NVIDIA DGX station which tops $300K and uses over 5kW of power. https://www.deltacomputer.com/nvidia-dgx-h100-640gb.html https://www.deltacomputer.com/nvidia-dgx-h100-640gb.html
- blagie 4y ago
  No, you can get away with under $20k these days. The models can now distribute adequately well, so you can get away with a swarm of thirty two 12GB 3060 Ti cards. 32*$400 = $12,800. Add in computers to stick them in and interlink, and that's sort of where you land. When you needed this in one machine, the cost to build a box capable of running OPT was around $50-$150k. An 80GB GPU runs around $10k-$20k, and you need five of them. So you can get away with a good bit less than an NVidia DGX Station. Unless you're a FAANG SWE of a high level of some kind, it's well outside of hobbyist use, but it's not an unreasonable expense for most business use. The way I think of this expense is it's 10% of the annual cost of a SWE. If it improves productivity by more than 10%, it's worth it for a one-person team. With a 10-person team, it's worth it at a 1% improvement.

throwifasd 4y ago

yeah, that's not what op asked about. That's one of those smaller models that "suck" as you said so confidently in your first comment. Not even close to gpt3/chatgpt. Next time when I talk bullshit I just post a random py file to make my argument. lmao

adam_arthur 4y ago
Though if true, it does beg the question, how can these services possibly be offered profitably if a single query requires many high end GPUs to run? Doesn't really track logically either. Though I'm not in the know. Perhaps ChatGPT is run with massive operating losses and its all VC subsidized.
- throwifasd 4y ago
  This post [0] is a good primer and they are talking about a 20b model gpt3 has 175b. Their other posts go more into details but yes, it really is a massive operation. Some hard facts from here [1] talking about BLOOM with 175b parameters.. >> Installing the full 175B version is a challenge though as it requires around 350GB of GPU VRAM, which is not something one can easily afford. But hey why deploy a model like chatGPT at all when you guys can be confidently incorrect all by yourselves. [0] https://nlpcloud.com/deploying-gpt-neox-20-production-focus-deepspeed.html https://nlpcloud.com/deploying-gpt-neox-20-production-focus-... [1] https://nlpcloud.com/chatgpt-open-source-alternatives.html https://nlpcloud.com/chatgpt-open-source-alternatives.html
  - adam_arthur 4y ago
    So it sounds like this is a question of loading the model into VRAM, and not a question of the cost of a single query. I assume once a model is loaded, many queries can be serviced by that model quickly. There's nothing incorrect about my assertion. If it were to actually take many GPUs to service one query, then there is no mass scale cost viable consumer product. That's just a clear economic fact. Regardless if a model could be theoretically spun up in a cost inefficient manner. And even 100s of GB of VRAM is not far off from consumer hardware. Look at how quickly graphics ram has expanded over time. About ~10x in ~10 years for high end cards, at a cursory glance at various Nvidia cards. At the same trajectory we could see a 400GB vram card within the next decade (though lots of assumptions)
    - throwifasd 4y ago
      You really are like a chatbot... look at the last three node sizes and the density of ram in them. It's not gonna happen as fast as you dream about it especially not with the discounts of the last Gens. The hope is to go to fp4 if you want to run it on consumer hardware and we are still not talking to about 2-3 cards. Why not at least try to Google before hammering down on stupid and uninformed hot takes?
    - visarga 4y ago
      > I assume once a model is loaded, many queries can be serviced by that model quickly. Depends. If you have room to load the whole model, yes. If you need to swap in and out parts of the model, then it matters if you have enough RAM.
- jacobr1 4y ago
  One thing to consider, there is considerable asymmetry between the high training costs and the lower (relatively) operating costs. > Perhaps ChatGPT is run with massive operating losses and its all VC subsidized. I'm not in the know either, but the current access is providing both marketing awareness, and significant human training/feedback to be used for improvements on future commercial projects. It could certainly be the case that that benefits outweigh costs compared to the classic, dumber "get eyeballs now, monetize later" strategy.
- visarga 4y ago
  $3M/day is what I heard

space_fountain 4y ago

The largest model is an order of magnitude smaller than gpt3 right? Does it perform similarly to copilot?

visarga 4y ago

You need something like an NVIDIA DGX station which tops $300K and uses over 5kW of power. https://www.deltacomputer.com/nvidia-dgx-h100-640gb.html https://www.deltacomputer.com/nvidia-dgx-h100-640gb.html

blagie 4y ago
No, you can get away with under $20k these days. The models can now distribute adequately well, so you can get away with a swarm of thirty two 12GB 3060 Ti cards. 32*$400 = $12,800. Add in computers to stick them in and interlink, and that's sort of where you land. When you needed this in one machine, the cost to build a box capable of running OPT was around $50-$150k. An 80GB GPU runs around $10k-$20k, and you need five of them. So you can get away with a good bit less than an NVidia DGX Station. Unless you're a FAANG SWE of a high level of some kind, it's well outside of hobbyist use, but it's not an unreasonable expense for most business use. The way I think of this expense is it's 10% of the annual cost of a SWE. If it improves productivity by more than 10%, it's worth it for a one-person team. With a 10-person team, it's worth it at a 1% improvement.