6 ms·

> I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Isn’t that already the case? Sure, it costs

> I have to wonder if 10 years down the line, everyone will be able to run models like this on their own computers. Isn’t that already the case? Sure, it costs $60K, but that is accessible to a surprisingly large minority, considering the potency of this software.

alexb_ 4y ago
...what? 60 thousand dollars for a dedicated computer that you can't use is not everyone, not on their own computers, and is also a crazy large amount of money for nearly everyone. Sure there are some that could, but that's not what I said.
- H8crilA 4y ago
  Indeed. What "everyone" can use is a ~$200 smartphone, so there's a ~300x gap to be bridged.
  - wjnc 4y ago
    log(300) / log(2) = only 8.2 doublings away. That's near future material.
    - thfuran 4y ago
      Maybe at 90s hardware growth rates, but not now.
      - rictic 4y ago
        The dream of the 90s is alive in the GPU market: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... Moore's law didn't stop, just Dennard scaling. Expect graphics and AI to continue to improve radically in performance/price, while more ordinary workloads see only modest improvements.
        thfuran 4y ago
        GPU TDP seems on the verge of going exponential, cost per transistor isn't really decreasing so much at the very latest nodes, and even that article seems to suggest it'd likely be decades before 300x flops/$
  - ben_w 4y ago
    Most of the cost of a phone isn’t the processor, so probably closer to x1000. Hardware may get that much cheaper, but it was never guaranteed, and we’re not making progress as fast as we used to.
- paganel 4y ago
  Plus, that's the energy costs involved when running a computer now worth 60k, I'm pretty sure that in the current socio-economic climate those power costs will surpass the initial acquisition cost (those 60k, that is) pretty easily.
  - colechristensen 4y ago
    An 80GB nvidia A100 goes for $20k and uses 300 watts, the energy costs of using one (or three) isn’t going to surpass the hardware costs for… a while.
    - paganel 4y ago
      I wanted to add that I was writing it metaphorically in a way, as in, seeing as how high those energy bills will be they might as well all add up to 60k. Not sure about most of the people in here, but I would get really nervous at the thought of running something that eats up 3x300 watts per hour, for 24/7, just as part of a personal/hobby project. The incoming power bills would be too high, you have to be in the wage-percentile for which dropping 60k on a machine just to carry out some hobby project is ok, i.e. you’d have to be “high-ish” middle-class at least. The recent increases in consumer power prices are a heavy blow for most of the middle-class around Europe (not sure about how things are in the States), so a project like this one is just a no-go for most of middle-class European programmers/computer people.
      - colechristensen 4y ago
        At full power 3 of those would cost me ~$3.50 per day ($0.15 per kWh is what I paid for last month's electricity, though I could pay less if I made some difference choices), I occasionally have a more expensive coffee order, or have a cocktail worth three times as much. Things are getting more expensive here but nothing like the situation in Europe (essentially none of our energy was imported from Russia, historically ~10% of oil imports but that was mostly to refine and re-export, we have all the natural gas locally that we need) The US crossed the line into being a net hyrdocarbon energy exporter a while ago (unsure what the case is recently but it is at worst about at parity)
- px43 4y ago
  Eh, 60k is just a bit more expensive than your average car, and lots of people have cars, and that's just how things are today. I imagine capabilities will be skyrocketing and prices will fall drastically at the same time.
  - fuzzer37 4y ago
    > 60k is just a bit more expensive than your average car If by "A bit" you mean about 30-40k
    - CRConrad 4y ago
      > > 60k is just a bit more expensive than your average car > If by "A bit" you mean about 30-40k 30k more expensive: Than your very-low-end-"average" car. 40k more expensive: Than your average used car. AFAICS. All in what one sees as an "average" car, I suppose.
kamray23 4y ago
You're grossly overestimating. People who make 60k annually are getting a bit rarer nowadays, it's not like everyone can afford it. For the majority of people it'd be a multi-decade project, for a few it might only take 7 years, very few people could buy it all at once.
uniqueuid 4y ago
Nitpick: This uses 8x A100 which are at least $10k a piece to my knowledge. Add in the computer and you're closer to $100k.
- sascha_sl 4y ago
  And also, NVIDIA does not sell them to the consumer market whatsoever. Linus Tech Tips could only show one because someone in the audience sent theirs over for review.
- taink 4y ago
  I believe you're confusing the amount of A100 graphics cards used to train the model (the cluster was actually made up of 800 A100s), and the amount you need to run the model : > The model [...] is supposed to run on multiple GPUs with tensor parallelism. > It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, [but should work] with ≈200GB of GPU memory. I don't know what the price of a V100 is, but given $10k a piece for A100s we would be closer to the $60k estimate.
  - uniqueuid 4y ago
    The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough. Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at least 1.6kw PSUs etc etc.
    - taink 4y ago
      Do you happen to know the cost of the 80GB variant?
      - colechristensen 4y ago
        Dell sells them for $20k
joshvm 4y ago
You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable at home. It's going to cost 1-2k on the DIMMs alone, less if you use 8x32GB, but that'll come down. You could definitely do it for less than $5k all in. Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work. But for really transient loads you're better off just renting seconds-minutes on a VM.
- sascha_sl 4y ago
  From the readme, it looks like you need that RAM on your GPU.
  - joshvm 4y ago
    There isn't any reason you can't run a neural net on a CPU. It's still just a bunch of big matrix operations. The advantage of the GPU is it's a lot faster, but "a lot" might be 1 second versus 10 seconds, and for some applications 10 seconds of inference latency is just fine (I have no idea how long this model would take). All the major ML libraries will operate in CPU-only mode if you request it.
    - visarga 4y ago
      They are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.
wellthisisgreat 4y ago
What kind of computer would they be? Can you spec it out roughly?

alexb_ 4y ago

...what? 60 thousand dollars for a dedicated computer that you can't use is not everyone, not on their own computers, and is also a crazy large amount of money for nearly everyone. Sure there are some that could, but that's not what I said.

H8crilA 4y ago
Indeed. What "everyone" can use is a ~$200 smartphone, so there's a ~300x gap to be bridged.
- wjnc 4y ago
  log(300) / log(2) = only 8.2 doublings away. That's near future material.
  - thfuran 4y ago
    Maybe at 90s hardware growth rates, but not now.
    - rictic 4y ago
      The dream of the 90s is alive in the GPU market: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... Moore's law didn't stop, just Dennard scaling. Expect graphics and AI to continue to improve radically in performance/price, while more ordinary workloads see only modest improvements.
      - thfuran 4y ago
        GPU TDP seems on the verge of going exponential, cost per transistor isn't really decreasing so much at the very latest nodes, and even that article seems to suggest it'd likely be decades before 300x flops/$
- ben_w 4y ago
  Most of the cost of a phone isn’t the processor, so probably closer to x1000. Hardware may get that much cheaper, but it was never guaranteed, and we’re not making progress as fast as we used to.
paganel 4y ago
Plus, that's the energy costs involved when running a computer now worth 60k, I'm pretty sure that in the current socio-economic climate those power costs will surpass the initial acquisition cost (those 60k, that is) pretty easily.
- colechristensen 4y ago
  An 80GB nvidia A100 goes for $20k and uses 300 watts, the energy costs of using one (or three) isn’t going to surpass the hardware costs for… a while.
  - paganel 4y ago
    I wanted to add that I was writing it metaphorically in a way, as in, seeing as how high those energy bills will be they might as well all add up to 60k. Not sure about most of the people in here, but I would get really nervous at the thought of running something that eats up 3x300 watts per hour, for 24/7, just as part of a personal/hobby project. The incoming power bills would be too high, you have to be in the wage-percentile for which dropping 60k on a machine just to carry out some hobby project is ok, i.e. you’d have to be “high-ish” middle-class at least. The recent increases in consumer power prices are a heavy blow for most of the middle-class around Europe (not sure about how things are in the States), so a project like this one is just a no-go for most of middle-class European programmers/computer people.
    - colechristensen 4y ago
      At full power 3 of those would cost me ~$3.50 per day ($0.15 per kWh is what I paid for last month's electricity, though I could pay less if I made some difference choices), I occasionally have a more expensive coffee order, or have a cocktail worth three times as much. Things are getting more expensive here but nothing like the situation in Europe (essentially none of our energy was imported from Russia, historically ~10% of oil imports but that was mostly to refine and re-export, we have all the natural gas locally that we need) The US crossed the line into being a net hyrdocarbon energy exporter a while ago (unsure what the case is recently but it is at worst about at parity)
px43 4y ago
Eh, 60k is just a bit more expensive than your average car, and lots of people have cars, and that's just how things are today. I imagine capabilities will be skyrocketing and prices will fall drastically at the same time.
- fuzzer37 4y ago
  > 60k is just a bit more expensive than your average car If by "A bit" you mean about 30-40k
  - CRConrad 4y ago
    > > 60k is just a bit more expensive than your average car > If by "A bit" you mean about 30-40k 30k more expensive: Than your very-low-end-"average" car. 40k more expensive: Than your average used car. AFAICS. All in what one sees as an "average" car, I suppose.

kamray23 4y ago

You're grossly overestimating. People who make 60k annually are getting a bit rarer nowadays, it's not like everyone can afford it. For the majority of people it'd be a multi-decade project, for a few it might only take 7 years, very few people could buy it all at once.

uniqueuid 4y ago

Nitpick: This uses 8x A100 which are at least $10k a piece to my knowledge. Add in the computer and you're closer to $100k.

sascha_sl 4y ago
And also, NVIDIA does not sell them to the consumer market whatsoever. Linus Tech Tips could only show one because someone in the audience sent theirs over for review.
taink 4y ago
I believe you're confusing the amount of A100 graphics cards used to train the model (the cluster was actually made up of 800 A100s), and the amount you need to run the model : > The model [...] is supposed to run on multiple GPUs with tensor parallelism. > It was tested on 4 (A100 80g) and 8 (V100 32g) GPUs, [but should work] with ≈200GB of GPU memory. I don't know what the price of a V100 is, but given $10k a piece for A100s we would be closer to the $60k estimate.
- uniqueuid 4y ago
  The $10k price is for an A100 with 40GB ram, so you need 8 of those. If you can get your hands on the 80GB variant, 4 are enough. Also, if you want to have a machine with eight of these cards, it will need to be a pretty high-spec rack-mounted or large tower. To feed these GPUs, you will want to have a decent amount of PCIe-4 lanes, meaning EPYC are the logical choice. So that's $20k for an AMD EPYC server with at least 1.6kw PSUs etc etc.
  - taink 4y ago
    Do you happen to know the cost of the 80GB variant?
    - colechristensen 4y ago
      Dell sells them for $20k

joshvm 4y ago

You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable at home. It's going to cost 1-2k on the DIMMs alone, less if you use 8x32GB, but that'll come down. You could definitely do it for less than $5k all in. Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work. But for really transient loads you're better off just renting seconds-minutes on a VM.

sascha_sl 4y ago
From the readme, it looks like you need that RAM on your GPU.
- joshvm 4y ago
  There isn't any reason you can't run a neural net on a CPU. It's still just a bunch of big matrix operations. The advantage of the GPU is it's a lot faster, but "a lot" might be 1 second versus 10 seconds, and for some applications 10 seconds of inference latency is just fine (I have no idea how long this model would take). All the major ML libraries will operate in CPU-only mode if you request it.
  - visarga 4y ago
    They are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.

wellthisisgreat 4y ago

What kind of computer would they be? Can you spec it out roughly?