10 ms·

Impressions from last week’s CVPR, a conference with 12k attendees on computer vision - Pretty much everyone is using NVIDIA GPUs, and pretty much everyone isn’

by w-m 2y ago

Impressions from last week’s CVPR, a conference with 12k attendees on computer vision - Pretty much everyone is using NVIDIA GPUs, and pretty much everyone isn’t happy with the prices, and would like some competition in the space: NVIDIA was there with 57 papers, a website dedicated to their research presented at the conference, a full day tutorial on accelerating deep learning, and ever present with shirts and backpacks in the corridors and at poster presentations. AMD had a booth at the expo part, where they were raffling off some GPUs. I went up to them to ask what framework I should look into, when writing kernels (ideally from Python) for GPGPU. They referred me to the “technical guy”, who it turns out had a demo on inference on an LLM. Which he couldn’t show me, as the laptop with the APU had crashed and wouldn’t reboot. He didn’t know about writing kernels, but told me there was a compiler guy who might be able to help, but he wasn’t to be found at that moment, and I couldn’t find him when returning to the booth later. I’m not at all happy with this situation. As long as AMDs investment into software and evangelism remains at ~$0, I don’t see how any hardware they put out will make a difference. And you’ll continue to hear people walking away from their booth, saying “oh when I win it I’m going to sell it to buy myself an NVIDIA GPU”.

cstejerean 2y ago
Completely agree. It's been 18 years since Nvidia released CUDA. AMD has had a long time to figure this out so I'm amazed at how they continue to fumble this.
- bryanlarsen 2y ago
  10 years ago they were basically broke and bet the farm on Zen. That bet paid off. I doubt a bet on CUDA would have paid off in time to save the company. They definitely didn't have the resources to split that bet.
- jsheard 2y ago
  It's not like the specific push for AI on GPUs came out of nowhere either, Nvidia first shipped cuDNN in 2014.
- kimixa 2y ago
  CUDA of 18 years ago is very different to CUDA of today. Back then AMD/ATI were actually at the forefront on the GPGPU side - things like the early brook language and CTM lead pretty quickly into things like OpenCL. Lots of work went on using the xbox360 gpu in real games for GPGPU tasks. But CUDA steadily improved iteratively, and AMD kinda just... stopped developing their equivalents? Considering a good part of that time they were near bankruptcy it might have not have been surprising though. But saying Nvidia solely kicked off everything with CUDA is rather a-historical.
  - yvdriess 2y ago
    Yep! I used BrookGPU for my GPGPU master thesis, before CUDA was a thing. AMD lacked followthrough on yhe software side as you said, but a big factor was also NV handing out GPUs to researchers.
  - userabchn 2y ago
    > CUDA of 18 years ago is very different to CUDA of today. I've been writing CUDA since 2008 and it doesn't seem that different to me. They even still use some of the same graphics in the user guide.
  - dagw 2y ago
    AMD kinda just... stopped developing their equivalents? I wasn't so much that they stopped developing, rather they kept throwing everything out and coming out with new and non backwards compatible replacements. I knew people working in the GPU Compute field back in those days who were trying to support both AMD/ATI and NVidia. While their CUDA code just worked from release to release and every new release of CUDA just got better and better, AMD kept coming up with new breaking APIs and forcing rewrite and rewrite until they just gave up and dropped AMD.
- dragontamer 2y ago
  10 years ago AMD was selling its own headquarters so that it could stave off bankruptcy for another few weeks (https://arstechnica.com/information-technology/2013/03/amd-sells-its-austin-hq-for-164-million-to-raise-some-quick-cash/ https://arstechnica.com/information-technology/2013/03/amd-s...). AMD's software investments have begun in earnest a few years ago, but AMD really did progress more than pretty much everyone else aside from NVidia IMO. AMD further made a few bad decisions where they "split the bet", relying upon Microsoft and others to push software forward. (I did like C++ Amp for what its worth). The underpinnings of C++Amp led to Boltzmann which led to ROCm, which then needed to be ported away from C++Amp and into CUDA-like Hip. So its a bit of a misstep there for sure. But its not like AMD has been dilly dallying. And for what its worth, I would have personally preferred C++ Amp (a C++11 standardized way to represent GPU functions as []-lambdas rather than CUDA-specific <<<extensions>>>). Obviously everyone else disagrees with me but there's some elegance to parallel_for_each([](param1, param2){magically a GPU function executing in parallel}), where the compiler figures out the details of how to get param1 and param2 from CPU RAM into GPU (or you use GPU-specific allocators to make param1/param2 in the GPU codespace already to bypass the automagic).
  - pjmlp 2y ago
    Nowadays you can write regular C++ in CUDA if you so wish, and contrary to AMD, NVidia employs several WG21 contributors.
CuriousCosmic 2y ago
> As long as AMDs investment into software and evangelism remains at ~$0 Last time I checked they have been trying to hire a ton of software engineers for improving the applied stacks (CV, ML, DSP, compute, etc) at the location near where I'm located. It seems like there's a big push to improve the stacks but given that less than 10 years ago they were practically at death's door it's not terribly surprising that their software is in the state it is. It's been getting better gradually but quality software doesn't just show up over night and especially so when things are as complex and arcane as they are in the GPU world.
- benreesman 2y ago
  With margins that high? There is always financing, there are always people willing to go to the competitor at some wage, there is always a way if the leadership wants to. If it was just a straight up fab bottleneck? Yeah maybe you buy that for a year or two. “During Q1, Nvidia reported $5.6 billion in cost of goods sold (COGS). This resulted in a gross profit of $20.4 billion, or a margin profile of 78.4%.” That’s called an “induced market failure”.
  - almostgotcaught 2y ago
    > With margins that high? There is always financing, there are always people willing to go to the competitor at some wage, there is always a way if the leadership wants to. People love to pop-off on stuff they really know anything about. Let me ask you: what financing do you imagine is available? Like literally what financing do you propose for a publically traded company? Like do you realize they can't actually issue new shares without putting it to a shareholder vote? Should they issue bonds? No I know they should run an ICO!!! And then what margins exactly? Do you know what the margin is on MI300? No. Do you know whether they're currently selling at a loss to win marketshare? No. I would the happiest boy if hn, in addition to policing jokes and memes, could police arrogance.
    - JohnPrine 2y ago
      Are you saying that companies lose the ability to secure financing once they go public?
qaq 2y ago
Well if Mojo and Modular Max Platform take off I guess there will be a path for AMD
- pjmlp 2y ago
  Well, "Modular to bring NVIDIA Accelerated Computing to the MAX Platform" https://www.modular.com/blog/modular-partners-with-nvidia-to-bring-gpus-to-the-max-platform https://www.modular.com/blog/modular-partners-with-nvidia-to...
  - qaq 2y ago
    The whole point of Max is that you can compile same code to multiple targets without manually optimizing for a given target. They are obviously going to support NVIDIA as a target.
    - pjmlp 2y ago
      Yet you haven't seen any AMD or Intel deal from them.
      - qaq 2y ago
        Cause they start with the target with largest user base?
monkeydust 2y ago
As more a business person than engineer, help me understand why AMD are not getting this, what's the counter argument? Is CUDA just too far ahead, are they lacking the right people in senior leadership roles to see this through?
- cyanydeez 2y ago
  CUDA is a software moat. If you want to use any gpu other than nvidia, you need to double your engineering budget because theres no easy to bootstrap projects at any level. The hardware prices are meaninglesz if you need a 200k engineer, if they exist, just.to bootstrap a product.
  - rbanffy 2y ago
    Depending on your hardware budget, the engineering one can look like a rounding error.
    - cyanydeez 2y ago
      Sure, but then youre still on the.side.of NVIDIA because you jave the.budget.
      - sangnoir 2y ago
        Why give any additional money to Nvidia when you can announce more profits (or get more compute if you're a government agency) by hiring more engineers to enable AMD hardware for less than a few million per year? It's not like Microsoft loves the idea of handing over money to Nvidia if there is a cheaper alternative that can make $MSFT go up.
        sliken 2y ago
        Say your success rate for replicating CUDA+Nvidia hardware on AMD is 60%. But it will take 2 years. That's not going to be compelling for any large org, especially when the MI300x is cheaper, but not crazy cheaper than an h100. Especially since CUDA is still rolling out new functionality and optimizations, so the goal posts will keep moving.
make3 2y ago
99%+ of people aren't writing kernels man, this doesn't mean anything, this is just silly
- deleted 2y ago
  [deleted]
jwuphysics 2y ago
Have you looked into TinyCorp [0]/tinygrad [1], one of the latest endeavors by George Hotz? I've been pretty impressed by the performance. [2] [0] https://tinygrad.org/ https://tinygrad.org/ [1] https://github.com/tinygrad/tinygrad https://github.com/tinygrad/tinygrad [2] https://x.com/realGeorgeHotz/status/1800932122569343043?t=Y6zE71-DHpFvWNdCvlHgVw&s=19 https://x.com/realGeorgeHotz/status/1800932122569343043?t=Y6...
- arghwhat 2y ago
  He also shakes his fist at the software stack, but loudly enough that it has AMD react to it.
- anthonix1 2y ago
  I have not been impressed by the perf. Slower than PyTorch for LLMs, and PyTorch is actually stable on AMD (I've trained 7B/13B models).. so the stability issues seem to be more of a tinygrad problem and less of an AMD problem, despite George's ramblings [0][1] [0] https://github.com/tinygrad/tinygrad/issues/4301 https://github.com/tinygrad/tinygrad/issues/4301 [1] https://x.com/realAnthonix/status/1800993761696284676 https://x.com/realAnthonix/status/1800993761696284676
slavik81 2y ago
MIVisionX is probably the library you want for computer vision. As for kernels, you would generally write HIP, which is very similar to CUDA. To my knowledge, there's no equivalent to cupy for writing kernels in Python. For what it's worth, your post has cemented my decision to submit a few conference talks. I've felt too busy writing code to go out and speak, but I really should make time.
- hyperbovine 2y ago
  The equivalent to cupy is ... cupy: https://docs.cupy.dev/en/v13.2.0/install.html#using-cupy-on-amd-gpu-experimental https://docs.cupy.dev/en/v13.2.0/install.html#using-cupy-on-...
  - slavik81 2y ago
    Oh cool! It appears that I've already packaged cupy's required dependencies for AMD GPU support in the Debian 13 'main' and Ubuntu 24.04 'universe' repos. I also extended the enabled architectures to cover all discrete AMD GPUs from Vega onwards (aside from MI300, ironically). It might be nice to get python3-cupy-rocm added to Debian 13 if this is a library that people find useful.
- pjmlp 2y ago
  HIP isn't similar to CUDA, in the set of available languages that target PTX, existing library ecosystem, IDE plugins and graphical debuggers. This is the kind of stuff AMD keeps missing out, even OneAPI from Intel looks better in that regard.
sangnoir 2y ago
> I’m not at all happy with this situation. As long as AMDs investment into software and evangelism remains at ~$0, I don’t see how any hardware they put out will make a difference. It appears AMD initial strategy is courting the HPC crowd and hyperscalers, they have big budgets, lower support overhead and are willing and able to write code that papers-over AMDs not-great software while appreciating lower-than-Nvidia TCO. I think this this incremental strategy is sensible, considering where most of the money is. As a first mover, Nvidia had to start from the bottom up; CUDA used to run only/mostly on consumer GPUs - AMD is going top-down, starting with high-margin DC hardware, before trickling down rack-level users, and eventually APUs later as revenue growth allows more re-investment.
- antupis 2y ago
  That is wrong move personally would start from localllm/llama folks who crave more memory and build up from there.
  - sangnoir 2y ago
    Seeing that they don't have a mature software stack, I think for now AMD would prefer one customer who brings in $10m revenue over 10'000 customers at $1000 a pop.
- landryraccoon 2y ago
  They’re making the wrong strategic play. They will fail if they go after the highest margin customers. Nvidia has every advantage and every motivation to keep those customers. They would need a trillion dollars in capital to have a chance imho. It would be like trying to go after Intel in the early 2000s by trying to target server cpus, or going after the desktop operating system market in the 90s against Microsoft. Its aiming for your competition where they are strongest and you are weakest. Their only chance to disrupt is to try to get some of the customers that Nvidia doesn’t care about, like consumer level inference / academic or hobbyist models. Intel failed when they got beaten in a market they didn’t care about, i.e mobile / small power devices.
  - 2y ago
deleted 2y ago
[deleted]
xadhominemx 2y ago
If you are looking for attention from an evangelist, I'm sorry but you are not the target customer for MI300. They are courting the Hyperscalers for heavy duty production inference workloads.
ColonelPhantom 2y ago
Did you talk to anyone from Intel? It seems they were also present: https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Intel-Labs-Presents-24-Papers-on-Innovative-AI-and-Computer/post/1606587 https://community.intel.com/t5/Blogs/Tech-Innovation/Artific...
lostmsu 2y ago
I also stopped by their booth and talked about trial access, and right away asked for easy access a la Google Collab, specifically without bureaucracy. And they are like "yeah, we are making it, but nah man, you can't just login and use it, you gotta fill a form and wait for us to approve it". Was very disappointed at that point. That was a marketing guy BTW. I don't think they realize their marketing strategies suck.

by w-m 2y ago

cstejerean 2y ago

Completely agree. It's been 18 years since Nvidia released CUDA. AMD has had a long time to figure this out so I'm amazed at how they continue to fumble this.

bryanlarsen 2y ago
10 years ago they were basically broke and bet the farm on Zen. That bet paid off. I doubt a bet on CUDA would have paid off in time to save the company. They definitely didn't have the resources to split that bet.
jsheard 2y ago
It's not like the specific push for AI on GPUs came out of nowhere either, Nvidia first shipped cuDNN in 2014.
kimixa 2y ago
CUDA of 18 years ago is very different to CUDA of today. Back then AMD/ATI were actually at the forefront on the GPGPU side - things like the early brook language and CTM lead pretty quickly into things like OpenCL. Lots of work went on using the xbox360 gpu in real games for GPGPU tasks. But CUDA steadily improved iteratively, and AMD kinda just... stopped developing their equivalents? Considering a good part of that time they were near bankruptcy it might have not have been surprising though. But saying Nvidia solely kicked off everything with CUDA is rather a-historical.
- yvdriess 2y ago
  Yep! I used BrookGPU for my GPGPU master thesis, before CUDA was a thing. AMD lacked followthrough on yhe software side as you said, but a big factor was also NV handing out GPUs to researchers.
- userabchn 2y ago
  > CUDA of 18 years ago is very different to CUDA of today. I've been writing CUDA since 2008 and it doesn't seem that different to me. They even still use some of the same graphics in the user guide.
- dagw 2y ago
  AMD kinda just... stopped developing their equivalents? I wasn't so much that they stopped developing, rather they kept throwing everything out and coming out with new and non backwards compatible replacements. I knew people working in the GPU Compute field back in those days who were trying to support both AMD/ATI and NVidia. While their CUDA code just worked from release to release and every new release of CUDA just got better and better, AMD kept coming up with new breaking APIs and forcing rewrite and rewrite until they just gave up and dropped AMD.
dragontamer 2y ago
10 years ago AMD was selling its own headquarters so that it could stave off bankruptcy for another few weeks (https://arstechnica.com/information-technology/2013/03/amd-sells-its-austin-hq-for-164-million-to-raise-some-quick-cash/ https://arstechnica.com/information-technology/2013/03/amd-s...). AMD's software investments have begun in earnest a few years ago, but AMD really did progress more than pretty much everyone else aside from NVidia IMO. AMD further made a few bad decisions where they "split the bet", relying upon Microsoft and others to push software forward. (I did like C++ Amp for what its worth). The underpinnings of C++Amp led to Boltzmann which led to ROCm, which then needed to be ported away from C++Amp and into CUDA-like Hip. So its a bit of a misstep there for sure. But its not like AMD has been dilly dallying. And for what its worth, I would have personally preferred C++ Amp (a C++11 standardized way to represent GPU functions as []-lambdas rather than CUDA-specific <<<extensions>>>). Obviously everyone else disagrees with me but there's some elegance to parallel_for_each([](param1, param2){magically a GPU function executing in parallel}), where the compiler figures out the details of how to get param1 and param2 from CPU RAM into GPU (or you use GPU-specific allocators to make param1/param2 in the GPU codespace already to bypass the automagic).
- pjmlp 2y ago
  Nowadays you can write regular C++ in CUDA if you so wish, and contrary to AMD, NVidia employs several WG21 contributors.

CuriousCosmic 2y ago

> As long as AMDs investment into software and evangelism remains at ~$0 Last time I checked they have been trying to hire a ton of software engineers for improving the applied stacks (CV, ML, DSP, compute, etc) at the location near where I'm located. It seems like there's a big push to improve the stacks but given that less than 10 years ago they were practically at death's door it's not terribly surprising that their software is in the state it is. It's been getting better gradually but quality software doesn't just show up over night and especially so when things are as complex and arcane as they are in the GPU world.

benreesman 2y ago
With margins that high? There is always financing, there are always people willing to go to the competitor at some wage, there is always a way if the leadership wants to. If it was just a straight up fab bottleneck? Yeah maybe you buy that for a year or two. “During Q1, Nvidia reported $5.6 billion in cost of goods sold (COGS). This resulted in a gross profit of $20.4 billion, or a margin profile of 78.4%.” That’s called an “induced market failure”.
- almostgotcaught 2y ago
  > With margins that high? There is always financing, there are always people willing to go to the competitor at some wage, there is always a way if the leadership wants to. People love to pop-off on stuff they really know anything about. Let me ask you: what financing do you imagine is available? Like literally what financing do you propose for a publically traded company? Like do you realize they can't actually issue new shares without putting it to a shareholder vote? Should they issue bonds? No I know they should run an ICO!!! And then what margins exactly? Do you know what the margin is on MI300? No. Do you know whether they're currently selling at a loss to win marketshare? No. I would the happiest boy if hn, in addition to policing jokes and memes, could police arrogance.
  - JohnPrine 2y ago
    Are you saying that companies lose the ability to secure financing once they go public?

qaq 2y ago

Well if Mojo and Modular Max Platform take off I guess there will be a path for AMD

pjmlp 2y ago
Well, "Modular to bring NVIDIA Accelerated Computing to the MAX Platform" https://www.modular.com/blog/modular-partners-with-nvidia-to-bring-gpus-to-the-max-platform https://www.modular.com/blog/modular-partners-with-nvidia-to...
- qaq 2y ago
  The whole point of Max is that you can compile same code to multiple targets without manually optimizing for a given target. They are obviously going to support NVIDIA as a target.
  - pjmlp 2y ago
    Yet you haven't seen any AMD or Intel deal from them.
    - qaq 2y ago
      Cause they start with the target with largest user base?

monkeydust 2y ago

As more a business person than engineer, help me understand why AMD are not getting this, what's the counter argument? Is CUDA just too far ahead, are they lacking the right people in senior leadership roles to see this through?

cyanydeez 2y ago
CUDA is a software moat. If you want to use any gpu other than nvidia, you need to double your engineering budget because theres no easy to bootstrap projects at any level. The hardware prices are meaninglesz if you need a 200k engineer, if they exist, just.to bootstrap a product.
- rbanffy 2y ago
  Depending on your hardware budget, the engineering one can look like a rounding error.
  - cyanydeez 2y ago
    Sure, but then youre still on the.side.of NVIDIA because you jave the.budget.
    - sangnoir 2y ago
      Why give any additional money to Nvidia when you can announce more profits (or get more compute if you're a government agency) by hiring more engineers to enable AMD hardware for less than a few million per year? It's not like Microsoft loves the idea of handing over money to Nvidia if there is a cheaper alternative that can make $MSFT go up.
      - sliken 2y ago
        Say your success rate for replicating CUDA+Nvidia hardware on AMD is 60%. But it will take 2 years. That's not going to be compelling for any large org, especially when the MI300x is cheaper, but not crazy cheaper than an h100. Especially since CUDA is still rolling out new functionality and optimizations, so the goal posts will keep moving.

make3 2y ago

99%+ of people aren't writing kernels man, this doesn't mean anything, this is just silly

deleted 2y ago
[deleted]

jwuphysics 2y ago

Have you looked into TinyCorp [0]/tinygrad [1], one of the latest endeavors by George Hotz? I've been pretty impressed by the performance. [2] [0] https://tinygrad.org/ https://tinygrad.org/ [1] https://github.com/tinygrad/tinygrad https://github.com/tinygrad/tinygrad [2] https://x.com/realGeorgeHotz/status/1800932122569343043?t=Y6zE71-DHpFvWNdCvlHgVw&s=19 https://x.com/realGeorgeHotz/status/1800932122569343043?t=Y6...

arghwhat 2y ago
He also shakes his fist at the software stack, but loudly enough that it has AMD react to it.
anthonix1 2y ago
I have not been impressed by the perf. Slower than PyTorch for LLMs, and PyTorch is actually stable on AMD (I've trained 7B/13B models).. so the stability issues seem to be more of a tinygrad problem and less of an AMD problem, despite George's ramblings [0][1] [0] https://github.com/tinygrad/tinygrad/issues/4301 https://github.com/tinygrad/tinygrad/issues/4301 [1] https://x.com/realAnthonix/status/1800993761696284676 https://x.com/realAnthonix/status/1800993761696284676

slavik81 2y ago

MIVisionX is probably the library you want for computer vision. As for kernels, you would generally write HIP, which is very similar to CUDA. To my knowledge, there's no equivalent to cupy for writing kernels in Python. For what it's worth, your post has cemented my decision to submit a few conference talks. I've felt too busy writing code to go out and speak, but I really should make time.

hyperbovine 2y ago
The equivalent to cupy is ... cupy: https://docs.cupy.dev/en/v13.2.0/install.html#using-cupy-on-amd-gpu-experimental https://docs.cupy.dev/en/v13.2.0/install.html#using-cupy-on-...
- slavik81 2y ago
  Oh cool! It appears that I've already packaged cupy's required dependencies for AMD GPU support in the Debian 13 'main' and Ubuntu 24.04 'universe' repos. I also extended the enabled architectures to cover all discrete AMD GPUs from Vega onwards (aside from MI300, ironically). It might be nice to get python3-cupy-rocm added to Debian 13 if this is a library that people find useful.
pjmlp 2y ago
HIP isn't similar to CUDA, in the set of available languages that target PTX, existing library ecosystem, IDE plugins and graphical debuggers. This is the kind of stuff AMD keeps missing out, even OneAPI from Intel looks better in that regard.

sangnoir 2y ago

> I’m not at all happy with this situation. As long as AMDs investment into software and evangelism remains at ~$0, I don’t see how any hardware they put out will make a difference. It appears AMD initial strategy is courting the HPC crowd and hyperscalers, they have big budgets, lower support overhead and are willing and able to write code that papers-over AMDs not-great software while appreciating lower-than-Nvidia TCO. I think this this incremental strategy is sensible, considering where most of the money is. As a first mover, Nvidia had to start from the bottom up; CUDA used to run only/mostly on consumer GPUs - AMD is going top-down, starting with high-margin DC hardware, before trickling down rack-level users, and eventually APUs later as revenue growth allows more re-investment.

antupis 2y ago
That is wrong move personally would start from localllm/llama folks who crave more memory and build up from there.
- sangnoir 2y ago
  Seeing that they don't have a mature software stack, I think for now AMD would prefer one customer who brings in $10m revenue over 10'000 customers at $1000 a pop.
landryraccoon 2y ago
They’re making the wrong strategic play. They will fail if they go after the highest margin customers. Nvidia has every advantage and every motivation to keep those customers. They would need a trillion dollars in capital to have a chance imho. It would be like trying to go after Intel in the early 2000s by trying to target server cpus, or going after the desktop operating system market in the 90s against Microsoft. Its aiming for your competition where they are strongest and you are weakest. Their only chance to disrupt is to try to get some of the customers that Nvidia doesn’t care about, like consumer level inference / academic or hobbyist models. Intel failed when they got beaten in a market they didn’t care about, i.e mobile / small power devices.
- 2y ago

deleted 2y ago

[deleted]

xadhominemx 2y ago

If you are looking for attention from an evangelist, I'm sorry but you are not the target customer for MI300. They are courting the Hyperscalers for heavy duty production inference workloads.

ColonelPhantom 2y ago

Did you talk to anyone from Intel? It seems they were also present: https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Intel-Labs-Presents-24-Papers-on-Innovative-AI-and-Computer/post/1606587 https://community.intel.com/t5/Blogs/Tech-Innovation/Artific...

lostmsu 2y ago

I also stopped by their booth and talked about trial access, and right away asked for easy access a la Google Collab, specifically without bureaucracy. And they are like "yeah, we are making it, but nah man, you can't just login and use it, you gotta fill a form and wait for us to approve it". Was very disappointed at that point. That was a marketing guy BTW. I don't think they realize their marketing strategies suck.