4 ms·

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wants to take the risk of buying a whole bunch of hardware I do! Nobody has stated it yet, but this is probably great news for tenstorrent. Disclosure: building a cloud compute provider starting with AMD MI300x, and eventually any other high end hardware that our customers are asking for.

rnts08 3y ago
That's interesting, something that I've been really wanting to get into as well, but where I am there is literally no venture capital to raise for this at the moment. I'd be interested to know more and/or bounce some ideas though.
- latchkey 3y ago
  Extremely capital intensive, but also requires relationships in the industry at many levels. Luckily, I happen to have both and they are crazy enough to put their trust in me. I feel very grateful for that.
htrp 3y ago
that or if you put chips in the hands of your customers, they may start to benchmark it against other equivalent solutions
- latchkey 3y ago
  Funny you should mention that. ;-) https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_amd_mi300x_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_... I've got about a dozen people signed up. Just working through some hardware issues right now (see above about high rate of failures), and hope to have this resolved next week, so that I can get people onto them and doing their testing.
  - htrp 3y ago
    Shiny! You have you guys gotten MI300X's working for non-inference use cases?
    - latchkey 3y ago
      We got them, then gave them to a customer to use for a week, got them back and now we are having some hardware issues that we are in the process of sorting out. I literally haven't had more than a few hours time on them yet. They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay as unbiased as possible. noticing now: strange that my previous comment was downvoted. Would be nice to understand what someone didn't like about what I said!
      - lostmsu 3y ago
        That sounds like your machine broke after 1 week of intensive use.
        latchkey 3y ago
        Sadly, not even intensive. The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here. If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn. This will be less of an issue as we grow as we will have plenty of stock to pull from, but it is a real bummer as we are starting as a proof of concept first. We started working on this business last August, before anyone knew whether or not AMD would even change course on AI. The good news is that we onboarded a customer the day that we announced our availability, we passed that PoC challenge with flying colors and closed significant additional funding immediately after that. Onwards and upwards, just have to roll with the punches.
        3y ago
aleph_minus_one 3y ago
> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?
- themoonisachees 3y ago
  A lot. It's the same reason amazon doesn't sell servers and instead gives you access to a single instance that everyone pretends is the same but in reality is massively transient.
  - latchkey 3y ago
    We give full bare metal access. If you just want one GPU, we give you a VM with PCIe pass through. If you take a whole box, we can give BMC and can give you access to the PDU itself, to hard reboot things remotely. It is as if you own the whole machine yourself.
- latchkey 3y ago
  Good question. Not much. If anything, what I'm doing is even harder because I will have multiple sources for the hardware. I have to deal with all of the hardware and data center issues, as well as the customers who rely on us to provide them access. Good thing that I'm a glutton for punishment.
- alephnerd 3y ago
  > What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription Margins. Pricing for cloud compute is much higher and servicing and management for the provider is much cheaper. If I sold hardware directly, then I'm often on the hook for support contracts which can get pricy with hardware and distract from shipping future facing product features, as customers who purchase directly have longer upgrade windows due to logistical overhead.
  - latchkey 3y ago
ukd1 3y ago
How do y'all compare to https://tensorwave.com https://tensorwave.com?
- latchkey 3y ago
  To be totally honest, I have no idea. When I first learned about them, I reached out to the CEO privately on LinkedIn, he asked what I was up to, I told him probably more than I should have (I come from an open source and transparent background), then he stopped talking to me entirely. Since then, one of the co-founders blocked me on Twitter for pointing out that despite their claims, they were not the first MI300x to production. Neither were we, ElioVP gets that trophy, then Lamini, then GigaIO. Making us 4th and them 5th. I could go on and on with weird stuff I've seen them do, but it just isn't productive here. Anyway, I think we have some overlap since we both are one of the few startups on the planet that actually has MI300x. But beyond that, I believe strongly that this space is large enough for multiple players and I don't see a need to be weird with each other. Apparently, I'm not on the same page though. ¯\_(ツ)_/¯
  - ametrau 3y ago
    For what it’s worth, to me, your approach is the one I’d prefer as a customer.
    - latchkey 3y ago
      Thank you. I've been on HN since 2009. The most successful people I've seen here, are the ones that are transparent, honest and ethical. I'm not trying to point fingers, I'm just focused on building a sustainable business and listening to my customers needs. The only way I can do that is by communicating with everyone around me as clearly and openly as I can. All our customers will know exactly where they stand, at all times. I post a lot of open information on r/AMD_Stock and the feedback that I've gotten there has been exceptional. People are excited to see if AMD can claw back a bit of the market. For the safety and success of AI, we don't need team blue vs. team red, we need everyone to work towards having as many options as possible. This is one way that I think we are going to differentiate ourselves. We won't just have MI300x, we will have every best-of-the-best chunk of hardware that we can get our hands on. No longer will super computers be tied up behind govt/edu grants. We want to democratize it. It has long been a goal of mine to build a super computer, and here is my chance. I'm excited. One thing that sets us apart is that my co-founder and I have a ton of experience deploying, managing and optimizing 150,000 AMD GPUs and 20PB+ of storage. We did it ourselves, all through covid and all of the supply chain issues. I'm not sure many others have done that and this is something that we are well versed at doing. I'm also seeing my competitors hiring a ton, while we are staying lean and mean with a very small team. I'd rather automate everything we deploy and focus all of our investors money on buying compute. We also have a pool of previous people we can hire from, which I think is quite an advantage over blanket hiring.
winwang 3y ago
How's the overall software support for MI300 series? The hardware itself looks great. (also, +100 to valuing honesty and transparency)
- latchkey 3y ago
  The hardware is actually pretty amazing. 192GB (or 1.5TB in a chassis), is a game changer. I'll let you know once I get my hands on them again. There really isn't enough public information about them at all. So far, my friends at ElioVP [0] have published a blog post. Still with not enough detail for my taste, but I'm pretty sure he is limited by what he can talk about. Luckily, I am not. I mention in another comment below that my current goal is to get a bunch of people to perform testing on them and then publish blog posts along with open source code. This way, we can start a repository of CI/CD tests to see how things improve with time. ROCm 6.1 is rumored to be quite an improvement. [0] https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing https://www.evp.cloud/post/diving-deeper-insights-from-our-l...
  - winwang 3y ago
    Nice! I'm pretty interested in GPGPU applications and MI300A, but I'm also just glad for more competition. Love that you hit up the LocalLLaMa sub. Do you know if anyone's tested CuPy stuff on MI300X?
    - latchkey 3y ago
      We haven't spec'd to buy A's quite yet as you're actually the first person I've heard even suggest them. If you're truly interested, hit me up personally. By default, we are putting dual 9754's in the chassis, along with 3TB ram and 155TB nvme. A pretty beefy box. However, if you want to work with us, we can customize this to whatever customers need. Effectively, we are the capex/opex for something that requires a lot of upfront funding and want to work with businesses that would rather focus on the software side of things.
      - 3y ago
gpapilion 3y ago
It’s basically their minimum cluster size for a reasonable model requires 8ish racks of compute. Semi analysis did some cost estimates, and I did some but you’re likely paying somewhere in the 12 million dollar range for the equipment to serve a single query using llama-70b. Compare that to a couple of gpus, and it’s easy to see why they are struggling to sell hardware, they can’t scale down. Since they didn’t use hbm, you need to stich enough cards together to get the memory to hold your model. It takes a lot of 256mb cards to get to 64gb, and there isn’t a good way to try the tech out since a single rack really can’t serve an LLM. The cloud provider path sounds riskier since that’s two capital intensive businesses, chip design and production and running a cloud service provider.
- latchkey 3y ago
  This is some fantastic insight. That would also inflate the opex (power/space/staffing) needs as well as people need to consider all the networking gear to hook this stuff together. 400G nic/switches/cables aren't cheap and in some cases, very hard to obtain in any sort of quantity. It does seem like an odd move in that case. I liken this to a company like Bitmain. Why sell the miners when you could just run them yourselves? Well, fact is that they do both. But in this case, Groq is turning off the sales. Who knows, maybe it just ends up being a temporary thing until they can sort all of the pieces out.
c-c-c-c-c 3y ago
The GroqCard (RS-GQ-GC1-0109) was in stock at mouser a few weeks ago and they are still taking orders.

rnts08 3y ago

That's interesting, something that I've been really wanting to get into as well, but where I am there is literally no venture capital to raise for this at the moment. I'd be interested to know more and/or bounce some ideas though.

latchkey 3y ago
Extremely capital intensive, but also requires relationships in the industry at many levels. Luckily, I happen to have both and they are crazy enough to put their trust in me. I feel very grateful for that.

htrp 3y ago

that or if you put chips in the hands of your customers, they may start to benchmark it against other equivalent solutions

latchkey 3y ago
Funny you should mention that. ;-) https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_amd_mi300x_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_... I've got about a dozen people signed up. Just working through some hardware issues right now (see above about high rate of failures), and hope to have this resolved next week, so that I can get people onto them and doing their testing.
- htrp 3y ago
  Shiny! You have you guys gotten MI300X's working for non-inference use cases?
  - latchkey 3y ago
    We got them, then gave them to a customer to use for a week, got them back and now we are having some hardware issues that we are in the process of sorting out. I literally haven't had more than a few hours time on them yet. They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay as unbiased as possible. noticing now: strange that my previous comment was downvoted. Would be nice to understand what someone didn't like about what I said!
    - lostmsu 3y ago
      That sounds like your machine broke after 1 week of intensive use.
      - latchkey 3y ago
        Sadly, not even intensive. The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here. If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn. This will be less of an issue as we grow as we will have plenty of stock to pull from, but it is a real bummer as we are starting as a proof of concept first. We started working on this business last August, before anyone knew whether or not AMD would even change course on AI. The good news is that we onboarded a customer the day that we announced our availability, we passed that PoC challenge with flying colors and closed significant additional funding immediately after that. Onwards and upwards, just have to roll with the punches.
        3y ago

aleph_minus_one 3y ago

> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?

themoonisachees 3y ago
A lot. It's the same reason amazon doesn't sell servers and instead gives you access to a single instance that everyone pretends is the same but in reality is massively transient.
- latchkey 3y ago
  We give full bare metal access. If you just want one GPU, we give you a VM with PCIe pass through. If you take a whole box, we can give BMC and can give you access to the PDU itself, to hard reboot things remotely. It is as if you own the whole machine yourself.
latchkey 3y ago
Good question. Not much. If anything, what I'm doing is even harder because I will have multiple sources for the hardware. I have to deal with all of the hardware and data center issues, as well as the customers who rely on us to provide them access. Good thing that I'm a glutton for punishment.
alephnerd 3y ago
> What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription Margins. Pricing for cloud compute is much higher and servicing and management for the provider is much cheaper. If I sold hardware directly, then I'm often on the hook for support contracts which can get pricy with hardware and distract from shipping future facing product features, as customers who purchase directly have longer upgrade windows due to logistical overhead.
- latchkey 3y ago

ukd1 3y ago

How do y'all compare to https://tensorwave.com https://tensorwave.com?

latchkey 3y ago
To be totally honest, I have no idea. When I first learned about them, I reached out to the CEO privately on LinkedIn, he asked what I was up to, I told him probably more than I should have (I come from an open source and transparent background), then he stopped talking to me entirely. Since then, one of the co-founders blocked me on Twitter for pointing out that despite their claims, they were not the first MI300x to production. Neither were we, ElioVP gets that trophy, then Lamini, then GigaIO. Making us 4th and them 5th. I could go on and on with weird stuff I've seen them do, but it just isn't productive here. Anyway, I think we have some overlap since we both are one of the few startups on the planet that actually has MI300x. But beyond that, I believe strongly that this space is large enough for multiple players and I don't see a need to be weird with each other. Apparently, I'm not on the same page though. ¯\_(ツ)_/¯
- ametrau 3y ago
  For what it’s worth, to me, your approach is the one I’d prefer as a customer.
  - latchkey 3y ago
    Thank you. I've been on HN since 2009. The most successful people I've seen here, are the ones that are transparent, honest and ethical. I'm not trying to point fingers, I'm just focused on building a sustainable business and listening to my customers needs. The only way I can do that is by communicating with everyone around me as clearly and openly as I can. All our customers will know exactly where they stand, at all times. I post a lot of open information on r/AMD_Stock and the feedback that I've gotten there has been exceptional. People are excited to see if AMD can claw back a bit of the market. For the safety and success of AI, we don't need team blue vs. team red, we need everyone to work towards having as many options as possible. This is one way that I think we are going to differentiate ourselves. We won't just have MI300x, we will have every best-of-the-best chunk of hardware that we can get our hands on. No longer will super computers be tied up behind govt/edu grants. We want to democratize it. It has long been a goal of mine to build a super computer, and here is my chance. I'm excited. One thing that sets us apart is that my co-founder and I have a ton of experience deploying, managing and optimizing 150,000 AMD GPUs and 20PB+ of storage. We did it ourselves, all through covid and all of the supply chain issues. I'm not sure many others have done that and this is something that we are well versed at doing. I'm also seeing my competitors hiring a ton, while we are staying lean and mean with a very small team. I'd rather automate everything we deploy and focus all of our investors money on buying compute. We also have a pool of previous people we can hire from, which I think is quite an advantage over blanket hiring.

winwang 3y ago

How's the overall software support for MI300 series? The hardware itself looks great. (also, +100 to valuing honesty and transparency)

latchkey 3y ago
The hardware is actually pretty amazing. 192GB (or 1.5TB in a chassis), is a game changer. I'll let you know once I get my hands on them again. There really isn't enough public information about them at all. So far, my friends at ElioVP [0] have published a blog post. Still with not enough detail for my taste, but I'm pretty sure he is limited by what he can talk about. Luckily, I am not. I mention in another comment below that my current goal is to get a bunch of people to perform testing on them and then publish blog posts along with open source code. This way, we can start a repository of CI/CD tests to see how things improve with time. ROCm 6.1 is rumored to be quite an improvement. [0] https://www.evp.cloud/post/diving-deeper-insights-from-our-llm-inference-testing https://www.evp.cloud/post/diving-deeper-insights-from-our-l...
- winwang 3y ago
  Nice! I'm pretty interested in GPGPU applications and MI300A, but I'm also just glad for more competition. Love that you hit up the LocalLLaMa sub. Do you know if anyone's tested CuPy stuff on MI300X?
  - latchkey 3y ago
    We haven't spec'd to buy A's quite yet as you're actually the first person I've heard even suggest them. If you're truly interested, hit me up personally. By default, we are putting dual 9754's in the chassis, along with 3TB ram and 155TB nvme. A pretty beefy box. However, if you want to work with us, we can customize this to whatever customers need. Effectively, we are the capex/opex for something that requires a lot of upfront funding and want to work with businesses that would rather focus on the software side of things.
    - 3y ago

gpapilion 3y ago

It’s basically their minimum cluster size for a reasonable model requires 8ish racks of compute. Semi analysis did some cost estimates, and I did some but you’re likely paying somewhere in the 12 million dollar range for the equipment to serve a single query using llama-70b. Compare that to a couple of gpus, and it’s easy to see why they are struggling to sell hardware, they can’t scale down. Since they didn’t use hbm, you need to stich enough cards together to get the memory to hold your model. It takes a lot of 256mb cards to get to 64gb, and there isn’t a good way to try the tech out since a single rack really can’t serve an LLM. The cloud provider path sounds riskier since that’s two capital intensive businesses, chip design and production and running a cloud service provider.

latchkey 3y ago
This is some fantastic insight. That would also inflate the opex (power/space/staffing) needs as well as people need to consider all the networking gear to hook this stuff together. 400G nic/switches/cables aren't cheap and in some cases, very hard to obtain in any sort of quantity. It does seem like an odd move in that case. I liken this to a company like Bitmain. Why sell the miners when you could just run them yourselves? Well, fact is that they do both. But in this case, Groq is turning off the sales. Who knows, maybe it just ends up being a temporary thing until they can sort all of the pieces out.

c-c-c-c-c 3y ago

The GroqCard (RS-GQ-GC1-0109) was in stock at mouser a few weeks ago and they are still taking orders.