3 ms·

It always cracks me up when people use the word "stupid" to insult other's intelligence. What a pathetically low-effort word to use.

nickysielicki 1y ago
When you’re responsible for supporting people who refuse to receive patches like this one [1], and those same people have the power to page your phone at 11pm on the weekend… you quickly learn how to call a spade, a spade. [1]: https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/20240924112933.242923-4-magali.lemes@canonical.com/ https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/202...
- jahsome 1y ago
  There is undoubtedly a better word than stupid. They're very likely not stupid. Careless, maybe. Inept, maybe. Irresponsible, maybe. Stubborn, maybe. More generously: overworked. Just probably not stupid.
  - xboxnolifes 1y ago
    What is the material difference here in the difference between inept and stupid?
    - jahsome 1y ago
      A dictionary is an easy way to find out, but in the interest of good faith: stupid is a lack of intelligence, inept is a lack of skills. To the point: I'd argue ineptitude is both more damning and accurate than stupidity in this particular case.
gosub100 1y ago
That wasn't what he used the word for. I understood his point perfectly: there are AI teams that are not knowledgeable or skilled enough to modify and enhance the docker images or toolkits that train/run the models. It takes some medium to advanced skills to get drivers to work properly. He used shorthand "too stupid to" instead of what I wrote above.
- jahsome 1y ago
  I fully understand. My issue is not with the point, my issue is being too lazy to articulate the point, and instead just saying "stupid." Address the behavior, not the people.
- siliconpotato 1y ago
  Still, it adds an air of arrogance to the whole post. For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image. Perhaps not stupidity but lack of docs from nvidia
  - nickysielicki 1y ago
    > For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image. I'm sorry to come across as arrogant, but it's really just frustration, because being surrounded by this kind of cargo-culting "special sauce" talk, even from so-called principal engineers, is what drove me to burnout and out of the industry into the northwoods. Furthermore, you're completely wrong. There is no special sauce, you just didn't look at the list of ingredients. There never has been any special sauce. NVIDIA builds their NGC base containers from open source scripts available on their gitlab instance: https://gitlab.com/nvidia/container-images/cuda https://gitlab.com/nvidia/container-images/cuda The build scripts for the base container are incredibly straightforward: they add the apt/yum repos and then install packages from that repo. The pytorch containers are constructed atop these base containers. The specific pytorch commit they use in their NGC pytorch containers are directly linked in their release notes for the container: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/index.html https://docs.nvidia.com/deeplearning/frameworks/pytorch-rele... That is: 25.08: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90d2a2b991deea1eef188260 https://github.com/pytorch/pytorch/commit/5228986c395dc79f90... 25.06: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90d2a2b991deea1eef188260 https://github.com/pytorch/pytorch/commit/5228986c395dc79f90... 25.05: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90d2a2b991deea1eef188260 https://github.com/pytorch/pytorch/commit/5228986c395dc79f90... 25.04: https://github.com/pytorch/pytorch/commit/79aa17489c3fc5ed6d5e972e9ffddf73e6dd0a5c https://github.com/pytorch/pytorch/commit/79aa17489c3fc5ed6d... 25.03: https://github.com/pytorch/pytorch/commit/7c8ec84dab7dc10d4ef90afc93a49b97bbd04503 https://github.com/pytorch/pytorch/commit/7c8ec84dab7dc10d4e... 25.02: https://github.com/pytorch/pytorch/commit/6c54963f75e9dfdae34c44f71081b5d3972b6b8d https://github.com/pytorch/pytorch/commit/6c54963f75e9dfdae3... 25.01: https://github.com/pytorch/pytorch/commit/ecf3bae40a6f2f0f3b237bde1fc4b2492765ab13 https://github.com/pytorch/pytorch/commit/ecf3bae40a6f2f0f3b... 24.12: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bdb592c184176e84669a7157 https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd... 24.11: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bdb592c184176e84669a7157 https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd... 24.10: https://github.com/pytorch/pytorch/commit/e000cf0ad980e5d140dc895a646174e9b945cf26 https://github.com/pytorch/pytorch/commit/e000cf0ad980e5d140... 24.09: https://github.com/pytorch/pytorch/commit/b465a5843b92f33fe3e89ff7ee91c6833df6aec0 https://github.com/pytorch/pytorch/commit/b465a5843b92f33fe3... 24.08: https://github.com/pytorch/pytorch/commit/872d972e41596a9ac94dfd343f40bfc12b340a74 https://github.com/pytorch/pytorch/commit/872d972e41596a9ac9... 24.07: https://github.com/pytorch/pytorch/commit/3bcc3cddb580bf0f0f1958cfe27001f236eac2c1 https://github.com/pytorch/pytorch/commit/3bcc3cddb580bf0f0f... 24.06: https://github.com/pytorch/pytorch/commit/f70bd71a4883c4d6243bc447f7e9af3fc8f83efd https://github.com/pytorch/pytorch/commit/f70bd71a4883c4d624... 24.05: https://github.com/pytorch/pytorch/commit/07cecf4168503a5b3defef9b2ecaeb3e075f4761 https://github.com/pytorch/pytorch/commit/07cecf4168503a5b3d... 24.04: https://github.com/pytorch/pytorch/commit/6ddf5cf85e3c27c596175aba7bf5affb5426255f https://github.com/pytorch/pytorch/commit/6ddf5cf85e3c27c596... 24.03: https://github.com/pytorch/pytorch/commit/40ec155e58ee1a1921377ff921b55e61502e4fb3 https://github.com/pytorch/pytorch/commit/40ec155e58ee1a1921... 24.02: https://github.com/pytorch/pytorch/commit/ebedce24ab578036dd9257e4928eea9ee38d1192 https://github.com/pytorch/pytorch/commit/ebedce24ab578036dd... 24.01: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de2c3b63206ca090927e203a https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de... 23.12: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de2c3b63206ca090927e203a https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de... 23.11: https://github.com/pytorch/pytorch/commit/6a974bec5d779ec10fafe53adbbd563aff4907d9 https://github.com/pytorch/pytorch/commit/6a974bec5d779ec10f... 23.10: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55a057061154c094a3eecb6f https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55... 23.09: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55a057061154c094a3eecb6f https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55... 23.08: https://github.com/pytorch/pytorch/commit/29c30b1db8129b57168fde2c17baa6b1bb70d8c7 https://github.com/pytorch/pytorch/commit/29c30b1db8129b5716... Do I need to keep going? Every single one of these commits is on pytorch/pytorch@main. So when you say: > For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. That's provably false. Unless you're suggesting that upstream pytorch continually rebased (eg: force pushed, breaking the worktree of every pytorch developer) atop unmerged code from nvidia, the commit ishes would not match. Meaning all of these commits were merged into pytorch/pytorch@main, and were available in pytorch nightlies, prior to the release of those NGC pytorch containers. No secret sauce, no man behind the curtain, just pure cargo culting and superstition.

nickysielicki 1y ago

When you’re responsible for supporting people who refuse to receive patches like this one [1], and those same people have the power to page your phone at 11pm on the weekend… you quickly learn how to call a spade, a spade. [1]: https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/20240924112933.242923-4-magali.lemes@canonical.com/ https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/202...

jahsome 1y ago
There is undoubtedly a better word than stupid. They're very likely not stupid. Careless, maybe. Inept, maybe. Irresponsible, maybe. Stubborn, maybe. More generously: overworked. Just probably not stupid.
- xboxnolifes 1y ago
  What is the material difference here in the difference between inept and stupid?
  - jahsome 1y ago
    A dictionary is an easy way to find out, but in the interest of good faith: stupid is a lack of intelligence, inept is a lack of skills. To the point: I'd argue ineptitude is both more damning and accurate than stupidity in this particular case.

gosub100 1y ago

That wasn't what he used the word for. I understood his point perfectly: there are AI teams that are not knowledgeable or skilled enough to modify and enhance the docker images or toolkits that train/run the models. It takes some medium to advanced skills to get drivers to work properly. He used shorthand "too stupid to" instead of what I wrote above.

jahsome 1y ago
I fully understand. My issue is not with the point, my issue is being too lazy to articulate the point, and instead just saying "stupid." Address the behavior, not the people.
siliconpotato 1y ago
Still, it adds an air of arrogance to the whole post. For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image. Perhaps not stupidity but lack of docs from nvidia
- nickysielicki 1y ago
  > For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. The upstream ecosystem hadn't caught up yet and Nvidia were adding their special sauce in their image. I'm sorry to come across as arrogant, but it's really just frustration, because being surrounded by this kind of cargo-culting "special sauce" talk, even from so-called principal engineers, is what drove me to burnout and out of the industry into the northwoods. Furthermore, you're completely wrong. There is no special sauce, you just didn't look at the list of ingredients. There never has been any special sauce. NVIDIA builds their NGC base containers from open source scripts available on their gitlab instance: https://gitlab.com/nvidia/container-images/cuda https://gitlab.com/nvidia/container-images/cuda The build scripts for the base container are incredibly straightforward: they add the apt/yum repos and then install packages from that repo. The pytorch containers are constructed atop these base containers. The specific pytorch commit they use in their NGC pytorch containers are directly linked in their release notes for the container: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/index.html https://docs.nvidia.com/deeplearning/frameworks/pytorch-rele... That is: 25.08: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90d2a2b991deea1eef188260 https://github.com/pytorch/pytorch/commit/5228986c395dc79f90... 25.06: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90d2a2b991deea1eef188260 https://github.com/pytorch/pytorch/commit/5228986c395dc79f90... 25.05: https://github.com/pytorch/pytorch/commit/5228986c395dc79f90d2a2b991deea1eef188260 https://github.com/pytorch/pytorch/commit/5228986c395dc79f90... 25.04: https://github.com/pytorch/pytorch/commit/79aa17489c3fc5ed6d5e972e9ffddf73e6dd0a5c https://github.com/pytorch/pytorch/commit/79aa17489c3fc5ed6d... 25.03: https://github.com/pytorch/pytorch/commit/7c8ec84dab7dc10d4ef90afc93a49b97bbd04503 https://github.com/pytorch/pytorch/commit/7c8ec84dab7dc10d4e... 25.02: https://github.com/pytorch/pytorch/commit/6c54963f75e9dfdae34c44f71081b5d3972b6b8d https://github.com/pytorch/pytorch/commit/6c54963f75e9dfdae3... 25.01: https://github.com/pytorch/pytorch/commit/ecf3bae40a6f2f0f3b237bde1fc4b2492765ab13 https://github.com/pytorch/pytorch/commit/ecf3bae40a6f2f0f3b... 24.12: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bdb592c184176e84669a7157 https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd... 24.11: https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bdb592c184176e84669a7157 https://github.com/pytorch/pytorch/commit/df5bbc09d191fff3bd... 24.10: https://github.com/pytorch/pytorch/commit/e000cf0ad980e5d140dc895a646174e9b945cf26 https://github.com/pytorch/pytorch/commit/e000cf0ad980e5d140... 24.09: https://github.com/pytorch/pytorch/commit/b465a5843b92f33fe3e89ff7ee91c6833df6aec0 https://github.com/pytorch/pytorch/commit/b465a5843b92f33fe3... 24.08: https://github.com/pytorch/pytorch/commit/872d972e41596a9ac94dfd343f40bfc12b340a74 https://github.com/pytorch/pytorch/commit/872d972e41596a9ac9... 24.07: https://github.com/pytorch/pytorch/commit/3bcc3cddb580bf0f0f1958cfe27001f236eac2c1 https://github.com/pytorch/pytorch/commit/3bcc3cddb580bf0f0f... 24.06: https://github.com/pytorch/pytorch/commit/f70bd71a4883c4d6243bc447f7e9af3fc8f83efd https://github.com/pytorch/pytorch/commit/f70bd71a4883c4d624... 24.05: https://github.com/pytorch/pytorch/commit/07cecf4168503a5b3defef9b2ecaeb3e075f4761 https://github.com/pytorch/pytorch/commit/07cecf4168503a5b3d... 24.04: https://github.com/pytorch/pytorch/commit/6ddf5cf85e3c27c596175aba7bf5affb5426255f https://github.com/pytorch/pytorch/commit/6ddf5cf85e3c27c596... 24.03: https://github.com/pytorch/pytorch/commit/40ec155e58ee1a1921377ff921b55e61502e4fb3 https://github.com/pytorch/pytorch/commit/40ec155e58ee1a1921... 24.02: https://github.com/pytorch/pytorch/commit/ebedce24ab578036dd9257e4928eea9ee38d1192 https://github.com/pytorch/pytorch/commit/ebedce24ab578036dd... 24.01: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de2c3b63206ca090927e203a https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de... 23.12: https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de2c3b63206ca090927e203a https://github.com/pytorch/pytorch/commit/81ea7a489a85d6f6de... 23.11: https://github.com/pytorch/pytorch/commit/6a974bec5d779ec10fafe53adbbd563aff4907d9 https://github.com/pytorch/pytorch/commit/6a974bec5d779ec10f... 23.10: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55a057061154c094a3eecb6f https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55... 23.09: https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55a057061154c094a3eecb6f https://github.com/pytorch/pytorch/commit/32f93b1c689954aa55... 23.08: https://github.com/pytorch/pytorch/commit/29c30b1db8129b57168fde2c17baa6b1bb70d8c7 https://github.com/pytorch/pytorch/commit/29c30b1db8129b5716... Do I need to keep going? Every single one of these commits is on pytorch/pytorch@main. So when you say: > For a while the only pytorch code that worked on newly released hopper GPUs we had was the Nvidia ngc container, not Pytorch nightly. That's provably false. Unless you're suggesting that upstream pytorch continually rebased (eg: force pushed, breaking the worktree of every pytorch developer) atop unmerged code from nvidia, the commit ishes would not match. Meaning all of these commits were merged into pytorch/pytorch@main, and were available in pytorch nightlies, prior to the release of those NGC pytorch containers. No secret sauce, no man behind the curtain, just pure cargo culting and superstition.