5 ms·

If you're FB, GOOG, AAPL, AMZN, BIDU, etc, this strategy makes sense because much like they have siloed data, they also have siloed computation graphs for which

If you're FB, GOOG, AAPL, AMZN, BIDU, etc, this strategy makes sense because much like they have siloed data, they also have siloed computation graphs for which they can lovingly design artisan transistors to make the perfect craft ASIC. There's big money in this. Or you can be like BIDU, buy 100K consumer GPUs, and put them in your datacenter. In response, Jensen altered the CUDA 9.1 licensing agreement and the EULA for Titan V such that going forward, you cannot deploy Titan V in a datacenter for anything but mining cryptocurrency, and his company reserves the right going forward to audit your use of their SW and HW at any time to force compliance with whatever rules Jensen pulled out of his butt that day after his morning weed. And that's a shame. Because there's no way any of these companies can beat the !/$ of consumer GPUs and NVDA is lying out of its a$$ to say you can't do HPC on them. But beyond NVDA shenanigans, I think it's incredibly risky to second guess those siloed computation graphs from the outside in the hopes of anything but an acqui-hire for an internal effort. Things ended well for Nervana even if their HW didn't ship in time, but when I see a 2018 company (http://nearist.ai/k-nn-benchmarks-part-wikipedia http://nearist.ai/k-nn-benchmarks-part-wikipedia) comparing their unavailable powerpoint processor to GPUs from 2013, and then doubling down on doing so when someone rightly points out how stupid that is, I see a beached fail whale in the making, not a threat to NVDA's Deepopoly.

lostgame 8y ago
FYI I think your comment is informative and I understood a lot of it but that's a shitton of acronymns for the uninitiated.
- deleted 8y ago
  [deleted]
- bytematic 8y ago
  Most are stock ticker names from the NYSE
- hueving 8y ago
  FB: Facebook GOOG: Google AAPL: Apple AMZN: Amazon BIDU: Baidu ASIC: Application specific integrated circuit GPU: Graphics processing unit CUDA: Compute-unified device architecture EULA: End-user license agreement weed: marijuana HPC: High-performance computing NVDA: Nvidia HW: hardware
mattnewton 8y ago
s /Jensen / nvidia ?
- guipsp 8y ago
  Jensen refers to NVDA's President & CEO, Jensen Huang
joshuamorton 8y ago
I'm a little confused here, are you saying that ML ASICs can't beat compute per $ of GPUs? That seems, on its face, to be a ridiculous assertion, so I'm confused where I'm misunderstanding you.
- oneshot908 8y ago
  No, they can't when you can get tens of TFLOPS per GPU for <$1000 that comes with a solid software ecosystem for all the major AI frameworks out of the box. That's the power of the gaming/industrial complex: NVDA can fund the development of AI chips and software from the nearly bottomless pockets of millions of gamers and cryptocurrency miners. ASIC startups figuratively have one bullet in their gun and they have to hit the bullseye to live. Now when a Tesla GPU costs $9-10K instead of <$1000, that's a somewhat different story, but even then, NVDA tapes out a brand new architecture almost annually. Good luck keeping up with that pace ASIC startups. And that's exactly what happened to Nervana. Their ASIC was better than a Pascal GP100, but it's clobbered by Volta V100. So at best you get a 6-12 month lead on them. In contrast, if you can right-size the transistor count for expensive and very specific workloads across 100K+ machines like companies with big datacenters and big data sets can do, I see an opportunity for all of the BigCos to build custom ASICs for their siloed data sets and graphs. That's what GOOG is doing and it seems to be working for them so far. FB is now hiring to do the same thing I suspect.
  - joshuamorton 8y ago
    Yes but over the lifetime of a GPU, you'll spend more on power draw than the physical hardware. That's where the savings come from, or at least that's what I've been told. A V100 costs ~300/yr in electricity. If you are buying at the scale of 100k units, but can price per operation, even by just 10% (for example, by dropping features you don't care about), that's a million dollars of electricity over the lifetime of your hardware.
- emcq 8y ago
  At a high level there is a design tradeoff where you put your transistors for a given chip. For a dense linear algebra/tensor processor, it basically comes down to using your transistors for memory or compute. GPUs (and DSPs) historically are way on the compute side. You get kilobytes of on chip memory, and really fat parallel buses to off chip RAM. On the other end, you have some chips that put more memory near the compute. It means you have less compute, but way better power efficiency. Each hop from a register to cache to off chip to between boards to between nodes is roughly 2-10x power hit, so you get orders of magnitudes here. In terms of training, it's really hard to fit your training data on chip. In which case you end up with an architecture very similar to a GPU or DSP/TPU. NVIDIA is no slouch here. A couple years ago the big trick was reducing precision - you dont need single or double floats during training so an architecture more specialized at fp16 or int8 would get some big savings for power and throughput. NVIDIA is playing this game too. At Google/FB scale tweaks to improve cost may make sense but it doesnt seem like architecturally speaking there is some major design decision being left on the table anymore (I'd love to hear a specific counterpoint though!). In terms of inference, there can be some big power savings putting the weights next to compute. This isn't rocket science and folks have been doing this for a while. In a strange way it can sometimes be more efficient to use a CPU, with fat caches.
zantana 8y ago
I just had an image of pranksters "swatting" datacenters by feeding false information to Nvidia that they are using consumer cards.

lostgame 8y ago

FYI I think your comment is informative and I understood a lot of it but that's a shitton of acronymns for the uninitiated.

deleted 8y ago
[deleted]
bytematic 8y ago
Most are stock ticker names from the NYSE
hueving 8y ago
FB: Facebook GOOG: Google AAPL: Apple AMZN: Amazon BIDU: Baidu ASIC: Application specific integrated circuit GPU: Graphics processing unit CUDA: Compute-unified device architecture EULA: End-user license agreement weed: marijuana HPC: High-performance computing NVDA: Nvidia HW: hardware

mattnewton 8y ago

s /Jensen / nvidia ?

guipsp 8y ago
Jensen refers to NVDA's President & CEO, Jensen Huang

joshuamorton 8y ago

I'm a little confused here, are you saying that ML ASICs can't beat compute per $ of GPUs? That seems, on its face, to be a ridiculous assertion, so I'm confused where I'm misunderstanding you.

oneshot908 8y ago
No, they can't when you can get tens of TFLOPS per GPU for <$1000 that comes with a solid software ecosystem for all the major AI frameworks out of the box. That's the power of the gaming/industrial complex: NVDA can fund the development of AI chips and software from the nearly bottomless pockets of millions of gamers and cryptocurrency miners. ASIC startups figuratively have one bullet in their gun and they have to hit the bullseye to live. Now when a Tesla GPU costs $9-10K instead of <$1000, that's a somewhat different story, but even then, NVDA tapes out a brand new architecture almost annually. Good luck keeping up with that pace ASIC startups. And that's exactly what happened to Nervana. Their ASIC was better than a Pascal GP100, but it's clobbered by Volta V100. So at best you get a 6-12 month lead on them. In contrast, if you can right-size the transistor count for expensive and very specific workloads across 100K+ machines like companies with big datacenters and big data sets can do, I see an opportunity for all of the BigCos to build custom ASICs for their siloed data sets and graphs. That's what GOOG is doing and it seems to be working for them so far. FB is now hiring to do the same thing I suspect.
- joshuamorton 8y ago
  Yes but over the lifetime of a GPU, you'll spend more on power draw than the physical hardware. That's where the savings come from, or at least that's what I've been told. A V100 costs ~300/yr in electricity. If you are buying at the scale of 100k units, but can price per operation, even by just 10% (for example, by dropping features you don't care about), that's a million dollars of electricity over the lifetime of your hardware.
emcq 8y ago
At a high level there is a design tradeoff where you put your transistors for a given chip. For a dense linear algebra/tensor processor, it basically comes down to using your transistors for memory or compute. GPUs (and DSPs) historically are way on the compute side. You get kilobytes of on chip memory, and really fat parallel buses to off chip RAM. On the other end, you have some chips that put more memory near the compute. It means you have less compute, but way better power efficiency. Each hop from a register to cache to off chip to between boards to between nodes is roughly 2-10x power hit, so you get orders of magnitudes here. In terms of training, it's really hard to fit your training data on chip. In which case you end up with an architecture very similar to a GPU or DSP/TPU. NVIDIA is no slouch here. A couple years ago the big trick was reducing precision - you dont need single or double floats during training so an architecture more specialized at fp16 or int8 would get some big savings for power and throughput. NVIDIA is playing this game too. At Google/FB scale tweaks to improve cost may make sense but it doesnt seem like architecturally speaking there is some major design decision being left on the table anymore (I'd love to hear a specific counterpoint though!). In terms of inference, there can be some big power savings putting the weights next to compute. This isn't rocket science and folks have been doing this for a while. In a strange way it can sometimes be more efficient to use a CPU, with fat caches.

zantana 8y ago

I just had an image of pranksters "swatting" datacenters by feeding false information to Nvidia that they are using consumer cards.