5 ms·

Incredible company. It’s absolutely insane how far ahead they are with the investments they made over a decade ago. So nice to see a “hard” engineering (from s

Incredible company. It’s absolutely insane how far ahead they are with the investments they made over a decade ago. So nice to see a “hard” engineering (from silicon to software) SV-founded company getting all this recognition. Especially after what has felt like a decade of SV hype software companies dominating the mainstream financial markets pre-pandemic with a spate of overpriced IPOs or large ad-revenue generating mega corporations.

epolanski 3y ago
Are they so far ahead? AMD GPUs get comparable results as of late on Stable Diffusion. Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science.
- johnvanommen 3y ago
  > Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science. I made the mistake of buying an A770 from Intel, based on the spec sheet. Hardware is comparable to what Nvidia is selling, for 70% of the price. It's basically a useless paperweight. The AI software crashes constantly, and when it's not crashing, it performs at half the level of Nvidia's cards. Turns out that drivers and software compatibility are a big deal, and Intel is way way behind in that arena.
  - epolanski 3y ago
    Sure, but there's lots of room for improvement and plenty of financial benefits to do so.
    - andromeduck 3y ago
      The problem is HW companies like Intel are dominated by EEs and bean counters. Both see software as cost center.
  - geodel 3y ago
    Its similar story with iPhone/ iOS vs Android. Endless talk about how Android is about to get so much better in performance compared to iPhone didn't yield much. I guess lately people have accepted perf will never match up. At least Android massive market share in rest of the world so with perf/pricing/compatibily/ localization it will remain competitive in some sense. I hope but do really foresee that AI/ML systems will have similar competitive stack like Nvidia.
    - callalex 3y ago
      I haven’t crossed over to the other side in a while, what do you find lacking in Android performance? I was under the impression that these days it’s solidly in “good enough not to notice” territory.
  - ant6n 3y ago
    Has it been improving?
- david-gpu 3y ago
  > Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science. I used to think like that, until I got a job there and... Oh, boy! I left five years later still amazed at all the ever more mind bending ways you can multiply two damn matrices. It was the most tedious yet also most intellectually challenging work I've ever done. My coworkers there were also the brightest group of engineers I've ever met.
  - kortilla 3y ago
    > was the most tedious yet also most intellectually challenging work I've ever done What does this mean? Tedious is pretty much the opposite of “intellectually challenging” when I think of careers.
    - em500 3y ago
      Trying to beat a good chess engine at 2500 Elo is probably both tedious and intellectually challenging.
- smoldesu 3y ago
  Nvidia has a small lead on the industry in a few places, adding up to super attractive backend hardware options. They aren't invincible, but they profit off the hostility between their competitors. Until those companies gang up to fund an open alternative, it's open season for Nvidia and HPC customers. The recent Stable Diffusion results are great news, but also don't include comparisons to an Nvidia card using the same optimizations. Nvidia claims that Microsoft Olive doubles performance on their cards too, so it might be a bit of a wash: https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvidia-ai-windows-rtx/ https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvi... Plus, none of those optimizations were any more open than CUDA (since it used DirectML). > crunching 4/8/16 bit width numbers is no rocket science. Of course not. That's why everyone did it: https://onnxruntime.ai/docs/execution-providers https://onnxruntime.ai/docs/execution-providers The problem with that "15 competing standards" XKCD is that normally one big proprietary standard wins. Nvidia has the history, the stability, the multi-OS and multi-arch support. The industry can definitely overturn it, but they have to work together to obsolete it.
- skocznymroczny 3y ago
  Perhaps RDNA3 GPUs get comparable results, but RDNA2 GPUs are behind. I bought a RX 6800XT to do some AI work because of the 16GB VRAM, and while the VRAM allows me to do stuff that my 6GB RTX 2060 wasn't able to, on performance side it's actually a downgrade in many aspects. But the main issue is software support. To get acceptable performance you need to use ROCm, which is Linux only. There was some Windows release of ROCm few weeks ago, but I am not sure how usable it is and none of the libraries have picked up on it yet. Even with a Linux installed, most frameworks still assume CUDA and it's an effort to get them to use ROCm. For some tools all it takes is uninstalling PyTorch or Tensorflow and installing a special ROCm enabled version of those libraries. Sometimes it will be enough, sometimes it wasn't. Sometimes the project uses some auxiliary library like bitsandbytes which doesn't have an official ROCm fork, so you have to use unofficial ones (that you have to compile manually and Makefiles quickly get out of date). Which once again, may work or may not. I have things set up for stable diffusion and text generation (oobabooga), and things mostly work, but sometimes they still don't. For example I can train stable diffusion embeddings and dreambooth checkpoints, but for some reason it crashes when I attempt to train a LORA. And I don't have enough expertise to debug it myself. For things like video encoding most tools also assume CUDA will be present so you're stuck with CPU encoding which takes forever. If you're lucky, some tools may have a DirectML backend, which kinda works under Windows for AMD, but it's performance is usually far behind a ROCm implementation.
  - paulmd 3y ago
    Video encoding doesn’t use CUDA but rather NVENC. However AMD is still as terrible at H264 as ever, and their AV1 encoder also has a hardware defect they’ve patched by forcing it to round up to 16 line multiples, so it is incapable of encoding a 1080p video and instead outputs 1082p that won’t be ingested properly when streaming. http://freedesktop.org/mesa/mesa/-/issues/9185#note_1954937 http://freedesktop.org/mesa/mesa/-/issues/9185#note_1954937 Also the AV1 quality is not as good as intel+nvidia even with resolutions that aren’t glitched. AMD seemingly went big on HEVC (supposedly because of stadia?) but everything else is a mess. And most places won’t touch HEVC because of the licensing costs. Microsoft makes it a windows store plugin you have to buy separately etc. somewhat odd that google picked HEVC for stadia but I guess those customers are actually directly paying you vs YouTube being a minus on their balance sheet (at least until recently possibly)
kccqzy 3y ago
The moniker of "hard" engineering is neither precise nor useful. What makes engineering hard? Is solving problems with distributed systems, even if these systems are for ads, hard? Or do you mean hardware? In that case even Nvidia is not hard enough since they don't fabricate their own chips. Or do you mean designing hardware? Then what makes writing system verilog at a desk hard but writing Python not hard?
- TechnicolorByte 3y ago
  I admit that was a glib comment and unnecessary. I’m really speaking about Nvidia’s ability to perform well in both hardware and software, at chip-scale and datacenter-scale. Also speaking of their product/business direction that revolutionizes multiple industries (leaders in graphics with ray tracing and AI frame/resolution sacking; leaders in AI infra and datacenter systems, etc.) all resulting in big impacts to their respective industries. You’re right that many of those software-only companies do very real engineering with distributed systems and such. I should’ve been more precise and was really complaining about the SV hype of the 2010s focusing on regulating-breaking companies like Airbnb, Uber, wework, etc. and on companies like Meta and Google who focus on pushing ads for their revenue.
- omniglottal 3y ago
  I suppose the difference is engineering something deterministic (i.e., physics, electronics, logic) versus something soft and indistinct (SEO, ad impressions, customer conversion rate).
- nsteel 3y ago
  It's hard to get complex systems correct. There's far less margin for error when you get a hardware design wrong. Correcting a Python software mistake is orders of magnitude easier and cheaper to resolve, it doesn't cost multiple billions and take 6 months to iterate. You might consider the hardware design harder in that respect.
xwdv 3y ago
Hard times make hard companies. Hard companies make good times. Good times make soft companies. Soft companies make hard times.
systemvoltage 3y ago
Yeah. NVidia was a docile looking company and in 2012, they were merely a gaming oriented hardware shop. These companies exist today. Which small or ignored companies do you think have a bright future?

epolanski 3y ago

Are they so far ahead? AMD GPUs get comparable results as of late on Stable Diffusion. Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science.

johnvanommen 3y ago
> Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science. I made the mistake of buying an A770 from Intel, based on the spec sheet. Hardware is comparable to what Nvidia is selling, for 70% of the price. It's basically a useless paperweight. The AI software crashes constantly, and when it's not crashing, it performs at half the level of Nvidia's cards. Turns out that drivers and software compatibility are a big deal, and Intel is way way behind in that arena.
- epolanski 3y ago
  Sure, but there's lots of room for improvement and plenty of financial benefits to do so.
  - andromeduck 3y ago
    The problem is HW companies like Intel are dominated by EEs and bean counters. Both see software as cost center.
- geodel 3y ago
  Its similar story with iPhone/ iOS vs Android. Endless talk about how Android is about to get so much better in performance compared to iPhone didn't yield much. I guess lately people have accepted perf will never match up. At least Android massive market share in rest of the world so with perf/pricing/compatibily/ localization it will remain competitive in some sense. I hope but do really foresee that AI/ML systems will have similar competitive stack like Nvidia.
  - callalex 3y ago
    I haven’t crossed over to the other side in a while, what do you find lacking in Android performance? I was under the impression that these days it’s solidly in “good enough not to notice” territory.
- ant6n 3y ago
  Has it been improving?
david-gpu 3y ago
> Software and hardware from competitors will catch up, crunching 4/8/16 bit width numbers is no rocket science. I used to think like that, until I got a job there and... Oh, boy! I left five years later still amazed at all the ever more mind bending ways you can multiply two damn matrices. It was the most tedious yet also most intellectually challenging work I've ever done. My coworkers there were also the brightest group of engineers I've ever met.
- kortilla 3y ago
  > was the most tedious yet also most intellectually challenging work I've ever done What does this mean? Tedious is pretty much the opposite of “intellectually challenging” when I think of careers.
  - em500 3y ago
    Trying to beat a good chess engine at 2500 Elo is probably both tedious and intellectually challenging.
smoldesu 3y ago
Nvidia has a small lead on the industry in a few places, adding up to super attractive backend hardware options. They aren't invincible, but they profit off the hostility between their competitors. Until those companies gang up to fund an open alternative, it's open season for Nvidia and HPC customers. The recent Stable Diffusion results are great news, but also don't include comparisons to an Nvidia card using the same optimizations. Nvidia claims that Microsoft Olive doubles performance on their cards too, so it might be a bit of a wash: https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvidia-ai-windows-rtx/ https://blogs.nvidia.com/blog/2023/05/23/microsoft-build-nvi... Plus, none of those optimizations were any more open than CUDA (since it used DirectML). > crunching 4/8/16 bit width numbers is no rocket science. Of course not. That's why everyone did it: https://onnxruntime.ai/docs/execution-providers https://onnxruntime.ai/docs/execution-providers The problem with that "15 competing standards" XKCD is that normally one big proprietary standard wins. Nvidia has the history, the stability, the multi-OS and multi-arch support. The industry can definitely overturn it, but they have to work together to obsolete it.
skocznymroczny 3y ago
Perhaps RDNA3 GPUs get comparable results, but RDNA2 GPUs are behind. I bought a RX 6800XT to do some AI work because of the 16GB VRAM, and while the VRAM allows me to do stuff that my 6GB RTX 2060 wasn't able to, on performance side it's actually a downgrade in many aspects. But the main issue is software support. To get acceptable performance you need to use ROCm, which is Linux only. There was some Windows release of ROCm few weeks ago, but I am not sure how usable it is and none of the libraries have picked up on it yet. Even with a Linux installed, most frameworks still assume CUDA and it's an effort to get them to use ROCm. For some tools all it takes is uninstalling PyTorch or Tensorflow and installing a special ROCm enabled version of those libraries. Sometimes it will be enough, sometimes it wasn't. Sometimes the project uses some auxiliary library like bitsandbytes which doesn't have an official ROCm fork, so you have to use unofficial ones (that you have to compile manually and Makefiles quickly get out of date). Which once again, may work or may not. I have things set up for stable diffusion and text generation (oobabooga), and things mostly work, but sometimes they still don't. For example I can train stable diffusion embeddings and dreambooth checkpoints, but for some reason it crashes when I attempt to train a LORA. And I don't have enough expertise to debug it myself. For things like video encoding most tools also assume CUDA will be present so you're stuck with CPU encoding which takes forever. If you're lucky, some tools may have a DirectML backend, which kinda works under Windows for AMD, but it's performance is usually far behind a ROCm implementation.
- paulmd 3y ago
  Video encoding doesn’t use CUDA but rather NVENC. However AMD is still as terrible at H264 as ever, and their AV1 encoder also has a hardware defect they’ve patched by forcing it to round up to 16 line multiples, so it is incapable of encoding a 1080p video and instead outputs 1082p that won’t be ingested properly when streaming. http://freedesktop.org/mesa/mesa/-/issues/9185#note_1954937 http://freedesktop.org/mesa/mesa/-/issues/9185#note_1954937 Also the AV1 quality is not as good as intel+nvidia even with resolutions that aren’t glitched. AMD seemingly went big on HEVC (supposedly because of stadia?) but everything else is a mess. And most places won’t touch HEVC because of the licensing costs. Microsoft makes it a windows store plugin you have to buy separately etc. somewhat odd that google picked HEVC for stadia but I guess those customers are actually directly paying you vs YouTube being a minus on their balance sheet (at least until recently possibly)

kccqzy 3y ago

The moniker of "hard" engineering is neither precise nor useful. What makes engineering hard? Is solving problems with distributed systems, even if these systems are for ads, hard? Or do you mean hardware? In that case even Nvidia is not hard enough since they don't fabricate their own chips. Or do you mean designing hardware? Then what makes writing system verilog at a desk hard but writing Python not hard?

TechnicolorByte 3y ago
I admit that was a glib comment and unnecessary. I’m really speaking about Nvidia’s ability to perform well in both hardware and software, at chip-scale and datacenter-scale. Also speaking of their product/business direction that revolutionizes multiple industries (leaders in graphics with ray tracing and AI frame/resolution sacking; leaders in AI infra and datacenter systems, etc.) all resulting in big impacts to their respective industries. You’re right that many of those software-only companies do very real engineering with distributed systems and such. I should’ve been more precise and was really complaining about the SV hype of the 2010s focusing on regulating-breaking companies like Airbnb, Uber, wework, etc. and on companies like Meta and Google who focus on pushing ads for their revenue.
omniglottal 3y ago
I suppose the difference is engineering something deterministic (i.e., physics, electronics, logic) versus something soft and indistinct (SEO, ad impressions, customer conversion rate).
nsteel 3y ago
It's hard to get complex systems correct. There's far less margin for error when you get a hardware design wrong. Correcting a Python software mistake is orders of magnitude easier and cheaper to resolve, it doesn't cost multiple billions and take 6 months to iterate. You might consider the hardware design harder in that respect.

xwdv 3y ago

Hard times make hard companies. Hard companies make good times. Good times make soft companies. Soft companies make hard times.

systemvoltage 3y ago

Yeah. NVidia was a docile looking company and in 2012, they were merely a gaming oriented hardware shop. These companies exist today. Which small or ignored companies do you think have a bright future?