5 ms·

A $3500 GPU requirement is far from democratization of AI.

glitchc 3y ago
For reference, a basic office computer in the 1980s cost upwards of $8000. If you factor in inflation, a $3500 GPU for cutting tech is a steal.
- DeepYogurt 3y ago
  Yes but moore's law ain't what it used to be
  - junon 3y ago
    You're right, it's been debunked and misquoted for decades.
  - zxexz 3y ago
    This is very true, however there is a long way to go in terms of chip design specific to DL architectures. I’m sure we’ll see lots of players release chips that are an order of magnitude more efficient for certain model types, but still fabricated on the same process node.
  - SanderNL 3y ago
    Moore is not helping here. Software and algorithms will fix this up, which is already happening at a frightening rate. Not too long ago, like months, we were still debating if it was ever even possible to run LLMs locally.
    - regularfry 3y ago
      There is going to be a computational complexity floor on where this can go, just from a Kolmogorov complexity argument. Very hard to tell how far away the floor is exactly but things are going so fast now I suspect we'll see diminishing returns in a few months as we asymptote towards some sort of efficiency boundary and the easy wins all get hoovered up.
      - SanderNL 3y ago
        Yes indeed and it’ll be interesting to see where that line is. I still think there is a lot to be gained from just properly and efficiently composing the parts we already have (like how the community handled stable diffusion) and exposing them in an accessible manner. I think that’ll take years even if the low hanging algorithm fruits start thinning out.
  - amoss 3y ago
    We've reached an inflection point, the new version would be: Nvidia can sell twice as many transitors for twice the price every 18 months.
  - reaperman 3y ago
    Moore's law isn't dead. Only Dennard's law. See slide 13 here[0] (2021). Moore's law stated that the number of transistors per area will double every n months. That's still happening. Besides, neither Moore's law nor Dennard scaling are even the most critical scaling law to be concerned about... ...that's probably Koomey's law[1][3], which looks well on track to hold for the rest of our careers. But eventually as computing approaches the Landauer limit[2] it must asymptotically level off as well. Probably starting around year 2050. Then we'll need to actually start "doing more with less" and minimizing the number of computations done for specific tasks. That will begin a very very productive time for custom silicon that is very task-specialized and low-level algorithmic optimization. [0] Shows that Moore's law (green line) is expected to start leveling off soon, but it has not yet slowed down. It also shows Koomey's law (orange line) holding indefinitely. Fun fact, if Koomey's law holds, we'll have exaflop power in <20W in about 20 years. That's equivalent to a whole OpenAI/DeepMind-worth of power in every smartphone. The neural engine in the A16 bionic on the latest iPhones can perform 17 TOPS. The A100 is about 1250 TOPS. Both these performance metrics are very subject to how you measure them, and I'm absolutely not sure I'm comparing apples to bananas properly. However, we'd expect the iPhone has reached its maximum thermal load. So without increasing power use, it should match the A100 in about 6 to 7 doublings, which would be about 11 years. In 20 years the iPhone would be expected to reach the performance of approximately 1000 A100's. At which point anyone will be able to train a GPT-4 in their pocket in a matter of days. There's some argument to be made that Koomey himself declared in 2016 that his law was dead[4], but that was during a particularly "slump-y" era of semiconductor manufacturing. IMHO, the 2016 analysis misses the A11 Bionic through A16 Bionic and M1 and M2 processors -- which instantly blew way past their competitors, breaking the temporary slump around 2016 and reverting us back to the mean slope. Mainly note that now they're analyzing only "supercomputers" and honestly that arena has changed, where quite a bit of the HPC work has moved to the cloud [e.g. Graviton] (not all of it, but a lot), and I don't think they're analyzing TPU pods, which also probably have far better TOPS/watt than traditional supercomputers like the ones on top500.org. 0: (Slide 13) https://www.sec.gov/Archives/edgar/data/937966/000119312521287742/d133133dex993.htm https://www.sec.gov/Archives/edgar/data/937966/0001193125212... 1: "The constant rate of doubling of the number of computations per joule of energy dissipated" https://en.wikipedia.org/wiki/Koomey%27s_law https://en.wikipedia.org/wiki/Koomey%27s_law 2: "The thermodynamic limit for the minimum amount of energy theoretically necessary to perform an irreversible single-bit operation." https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle 3: https://www.koomey.com/post/14466436072 https://www.koomey.com/post/14466436072 4: https://www.koomey.com/post/153838038643 https://www.koomey.com/post/153838038643
- kortilla 3y ago
  And hardly anyone had them in the 1980s.
  - deleted 3y ago
    [deleted]
- peoplefromibiza 3y ago
  virtually no office had them in 1980 by mid 1980s personal computers costed less than $500
  - foo-bar-bat 3y ago
    This is false. Created account just to rebut. While it may have been true that it was technically possible to assemble a PC for $500... good luck. In the real world people were spending $1500-$2500+ for PCs, and that price point held remarkably constant. By the time you were done buying a monitor, external drives, printer etc $3000+ was likely. https://en.m.wikipedia.org/wiki/IBM_Personal_Computer#:~:text=The%20IBM%20PC%20debuted%20on,comparable%20machines%20in%20the%20market https://en.m.wikipedia.org/wiki/IBM_Personal_Computer#:~:tex.... Or see Apple Mac 512 introduced at approx $2800? One reason this was interesting (if you could afford it) was the physical integration and elimination of "PC" cable spaghetti. https://en.m.wikipedia.org/wiki/Macintosh_512K https://en.m.wikipedia.org/wiki/Macintosh_512K But again having worked my way thru college with an 8 MB external hard drive... which was a huge improvement over having to swap and preload floppies in twin external disk drives just to play stupid (awesome) early video games, all of this stuff cost a lot more than youre saying. And continued to well into the 90s. Of course there are examples of computers that cost less. I got a TI-99/4a for Christmas which cost my uncle about $500-600. But then you needed a TV to hook it up to, and a bunch of tapes too. And unless you were a nerd and wanted to program, it didn't really DO anything. I spent months manually recreating arcade video games for myself on that. Good times. Conversely, if you bought an IBM or Apple computer, odds were you were also going to spend another $1000 or more buying shrinkwrap software to run on it. Rather than writing your own. Source: I remember.
    - peoplefromibiza 3y ago
      > This is false. The CPC 464 is the first personal home computer built by Amstrad in 1984. It was one of the bestselling and best produced microcomputers, with more than 2 million units sold in Europe. Price £199 (with green monitor), £299 (with colour monitor) > But again having worked my way thru college with an 8 MB external hard drive that was a minicomputer at the time, not a PC (personal computer) > Source: I remember. Source: I owned, used and programmed PCs in the 80s
- webmaven 3y ago
  I think a more relevant comparison may be a peripheral: the $7,000 LaserWriter which kicked off the desktop publishing revolution in 1985.
jlarocco 3y ago
$3500 is less than one week of a developer salary for most companies. It woudn't pay a month's rent for most commercial office space. It's a lower cost of entry than almost any other industry I can think of. A cargo van with a logo on it (for a delivery business or painting business, for example) would easily cost 10-20x as much.
- fomine3 3y ago
  * in the US
- Naracion 3y ago
  1. I don't know what kind of world you live in to think that USD 3500 is "less than one week of a developer salary for most companies." I think you really just mean FAANG (or whatever the current acronym is) or potentially SV / offices in cities with very high COL. 2. The problem is scaling. To support billions of search queries you would have to invest in a lot more than a single GPU. You also wouldn't only need a single van, but once you take scaling into account even at $3500 the GPUs will be much more expensive. That said, costs will come down eventually. The question in my mind is whether OpenAI (who already has the hardware resources and backed by Microsoft funding to boot) will be able to dominate the market to the extent that Google can't make a comeback by the time they're able to scale.
  - hanselot 3y ago
    1 - Yes, I agree on this, but even so, most developers already are investing in SOTA GPU's for other reason (so not as much of a barrier as purported) 2 - Scaling is not a problem in other industries? If you want to scale your food truck, you will need more food trucks, this doesn't seem to really do anything for your point. GGML and GPTQ have already revolutionised the situation, and now there are tiny models with insane quality as well, that can run on a conventional CPU. I don't think you have any idea what is happening around you, and this is not me being nasty, just go and take a look at how exponential this development is and you will realise that you need to get in on it before its too late.
    - 3y ago
sumtechguy 3y ago
and that 3500 worth of kit will be a couple of hundred bucks on ebay in 5 years.
qup 3y ago
Yeah, I bet people won't get cars, either, they're a lot more expensive than that.

For reference, a basic office computer in the 1980s cost upwards of $8000. If you factor in inflation, a $3500 GPU for cutting tech is a steal.

DeepYogurt 3y ago
Yes but moore's law ain't what it used to be
- junon 3y ago
  You're right, it's been debunked and misquoted for decades.
- zxexz 3y ago
  This is very true, however there is a long way to go in terms of chip design specific to DL architectures. I’m sure we’ll see lots of players release chips that are an order of magnitude more efficient for certain model types, but still fabricated on the same process node.
- SanderNL 3y ago
  Moore is not helping here. Software and algorithms will fix this up, which is already happening at a frightening rate. Not too long ago, like months, we were still debating if it was ever even possible to run LLMs locally.
  - regularfry 3y ago
    There is going to be a computational complexity floor on where this can go, just from a Kolmogorov complexity argument. Very hard to tell how far away the floor is exactly but things are going so fast now I suspect we'll see diminishing returns in a few months as we asymptote towards some sort of efficiency boundary and the easy wins all get hoovered up.
    - SanderNL 3y ago
      Yes indeed and it’ll be interesting to see where that line is. I still think there is a lot to be gained from just properly and efficiently composing the parts we already have (like how the community handled stable diffusion) and exposing them in an accessible manner. I think that’ll take years even if the low hanging algorithm fruits start thinning out.
- amoss 3y ago
  We've reached an inflection point, the new version would be: Nvidia can sell twice as many transitors for twice the price every 18 months.
- reaperman 3y ago
  Moore's law isn't dead. Only Dennard's law. See slide 13 here[0] (2021). Moore's law stated that the number of transistors per area will double every n months. That's still happening. Besides, neither Moore's law nor Dennard scaling are even the most critical scaling law to be concerned about... ...that's probably Koomey's law[1][3], which looks well on track to hold for the rest of our careers. But eventually as computing approaches the Landauer limit[2] it must asymptotically level off as well. Probably starting around year 2050. Then we'll need to actually start "doing more with less" and minimizing the number of computations done for specific tasks. That will begin a very very productive time for custom silicon that is very task-specialized and low-level algorithmic optimization. [0] Shows that Moore's law (green line) is expected to start leveling off soon, but it has not yet slowed down. It also shows Koomey's law (orange line) holding indefinitely. Fun fact, if Koomey's law holds, we'll have exaflop power in <20W in about 20 years. That's equivalent to a whole OpenAI/DeepMind-worth of power in every smartphone. The neural engine in the A16 bionic on the latest iPhones can perform 17 TOPS. The A100 is about 1250 TOPS. Both these performance metrics are very subject to how you measure them, and I'm absolutely not sure I'm comparing apples to bananas properly. However, we'd expect the iPhone has reached its maximum thermal load. So without increasing power use, it should match the A100 in about 6 to 7 doublings, which would be about 11 years. In 20 years the iPhone would be expected to reach the performance of approximately 1000 A100's. At which point anyone will be able to train a GPT-4 in their pocket in a matter of days. There's some argument to be made that Koomey himself declared in 2016 that his law was dead[4], but that was during a particularly "slump-y" era of semiconductor manufacturing. IMHO, the 2016 analysis misses the A11 Bionic through A16 Bionic and M1 and M2 processors -- which instantly blew way past their competitors, breaking the temporary slump around 2016 and reverting us back to the mean slope. Mainly note that now they're analyzing only "supercomputers" and honestly that arena has changed, where quite a bit of the HPC work has moved to the cloud [e.g. Graviton] (not all of it, but a lot), and I don't think they're analyzing TPU pods, which also probably have far better TOPS/watt than traditional supercomputers like the ones on top500.org. 0: (Slide 13) https://www.sec.gov/Archives/edgar/data/937966/000119312521287742/d133133dex993.htm https://www.sec.gov/Archives/edgar/data/937966/0001193125212... 1: "The constant rate of doubling of the number of computations per joule of energy dissipated" https://en.wikipedia.org/wiki/Koomey%27s_law https://en.wikipedia.org/wiki/Koomey%27s_law 2: "The thermodynamic limit for the minimum amount of energy theoretically necessary to perform an irreversible single-bit operation." https://en.wikipedia.org/wiki/Landauer%27s_principle https://en.wikipedia.org/wiki/Landauer%27s_principle 3: https://www.koomey.com/post/14466436072 https://www.koomey.com/post/14466436072 4: https://www.koomey.com/post/153838038643 https://www.koomey.com/post/153838038643
kortilla 3y ago
And hardly anyone had them in the 1980s.
- deleted 3y ago
  [deleted]
peoplefromibiza 3y ago
virtually no office had them in 1980 by mid 1980s personal computers costed less than $500
- foo-bar-bat 3y ago
  This is false. Created account just to rebut. While it may have been true that it was technically possible to assemble a PC for $500... good luck. In the real world people were spending $1500-$2500+ for PCs, and that price point held remarkably constant. By the time you were done buying a monitor, external drives, printer etc $3000+ was likely. https://en.m.wikipedia.org/wiki/IBM_Personal_Computer#:~:text=The%20IBM%20PC%20debuted%20on,comparable%20machines%20in%20the%20market https://en.m.wikipedia.org/wiki/IBM_Personal_Computer#:~:tex.... Or see Apple Mac 512 introduced at approx $2800? One reason this was interesting (if you could afford it) was the physical integration and elimination of "PC" cable spaghetti. https://en.m.wikipedia.org/wiki/Macintosh_512K https://en.m.wikipedia.org/wiki/Macintosh_512K But again having worked my way thru college with an 8 MB external hard drive... which was a huge improvement over having to swap and preload floppies in twin external disk drives just to play stupid (awesome) early video games, all of this stuff cost a lot more than youre saying. And continued to well into the 90s. Of course there are examples of computers that cost less. I got a TI-99/4a for Christmas which cost my uncle about $500-600. But then you needed a TV to hook it up to, and a bunch of tapes too. And unless you were a nerd and wanted to program, it didn't really DO anything. I spent months manually recreating arcade video games for myself on that. Good times. Conversely, if you bought an IBM or Apple computer, odds were you were also going to spend another $1000 or more buying shrinkwrap software to run on it. Rather than writing your own. Source: I remember.
  - peoplefromibiza 3y ago
    > This is false. The CPC 464 is the first personal home computer built by Amstrad in 1984. It was one of the bestselling and best produced microcomputers, with more than 2 million units sold in Europe. Price £199 (with green monitor), £299 (with colour monitor) > But again having worked my way thru college with an 8 MB external hard drive that was a minicomputer at the time, not a PC (personal computer) > Source: I remember. Source: I owned, used and programmed PCs in the 80s
webmaven 3y ago
I think a more relevant comparison may be a peripheral: the $7,000 LaserWriter which kicked off the desktop publishing revolution in 1985.

jlarocco 3y ago

$3500 is less than one week of a developer salary for most companies. It woudn't pay a month's rent for most commercial office space. It's a lower cost of entry than almost any other industry I can think of. A cargo van with a logo on it (for a delivery business or painting business, for example) would easily cost 10-20x as much.

fomine3 3y ago
* in the US
Naracion 3y ago
1. I don't know what kind of world you live in to think that USD 3500 is "less than one week of a developer salary for most companies." I think you really just mean FAANG (or whatever the current acronym is) or potentially SV / offices in cities with very high COL. 2. The problem is scaling. To support billions of search queries you would have to invest in a lot more than a single GPU. You also wouldn't only need a single van, but once you take scaling into account even at $3500 the GPUs will be much more expensive. That said, costs will come down eventually. The question in my mind is whether OpenAI (who already has the hardware resources and backed by Microsoft funding to boot) will be able to dominate the market to the extent that Google can't make a comeback by the time they're able to scale.
- hanselot 3y ago
  1 - Yes, I agree on this, but even so, most developers already are investing in SOTA GPU's for other reason (so not as much of a barrier as purported) 2 - Scaling is not a problem in other industries? If you want to scale your food truck, you will need more food trucks, this doesn't seem to really do anything for your point. GGML and GPTQ have already revolutionised the situation, and now there are tiny models with insane quality as well, that can run on a conventional CPU. I don't think you have any idea what is happening around you, and this is not me being nasty, just go and take a look at how exponential this development is and you will realise that you need to get in on it before its too late.
  - 3y ago

sumtechguy 3y ago

and that 3500 worth of kit will be a couple of hundred bucks on ebay in 5 years.

qup 3y ago

Yeah, I bet people won't get cars, either, they're a lot more expensive than that.