5 ms·

I work for a company (Lambda Labs) that would span all these categories. We build GPUs and sell them, sending them off to the customer. We also build them and

I work for a company (Lambda Labs) that would span all these categories. We build GPUs and sell them, sending them off to the customer. We also build them and sell them, but then host it for the customer. We also buy them for ourselves and rent them either bare metal or virtualized. The "build it and ship it to the customer" business is waning. The reason is because even the most deep pocketed customers no longer have data centers that can support the power and cooling needs of current GPUs. The latest from NVIDIA requires liquid cooling, tons of power, and solid floors (the racks are too heavy for raised floors). Being able to run a modern server is becoming a very niche skill, one that most companies don't want or need to invest in.

echelon 3y ago
> The latest from NVIDIA requires liquid cooling, tons of power, and solid floors We won't be running on GPUs forever. The innovation in the chip space will continue. With as much margin as cloud takes, there will be increased interest in on-prem and mixed environments. We run our own GPUs in production as well as having a cloud footprint.
- jedberg 3y ago
  > We won't be running on GPUs forever. The innovation in the chip space will continue. Sure, but I'm seeing a divergence. There are a group of chips that are getting smaller and require less power and cooling that can do a lot of workloads that you used to need huge chips for, but there are the most advanced chips for the most advanced workloads getting bigger and more power hungry. There will always be the big customers who need the big chips, and that is what will be nearly impossible for them to self host.
  - PLenz 3y ago
    This is the same argument that mainframe companies made in the 80s. Mass commodity systems will eventually kill the big gpu except for legacy and niche workloads. It's the tech infra circle of life.
    - cogman10 3y ago
      I'm less optimistic. Unlike the 80s we are quickly approaching the limits of node feature sizes. Barring radically new designs and materials, I think it's reasonable to assume that we are reaching the power:computational limits of silicon.
      - latchkey 3y ago
        Maybe in the case of CPUs. But, there is a fundamental technical shift going on away from computationally expensive things being done on CPUs and moving to AIAs. I suspect that this is why Jensen said Moore's law is dead.
        refulgentis 3y ago
        Jensen said Moore's Law is dead because of the dichotomy explained a few times -- the free lunches are coming fewer and far between, you can't get a substantial performance boost and power decrease every 18 months, per Moore's Law. We are reaching the power:computational limits of silicon. Both GPU and CPU.
        latchkey 3y ago
        Here is the quote: “Moore’s Law’s dead,” Huang said, referring to the standard that the number of transistors on a chip doubles every two years. “And the ability for Moore’s Law to deliver twice the performance at the same cost, or at the same performance, half the cost, every year and a half, is over. It’s completely over, and so the idea that a chip is going to go down in cost over time, unfortunately, is a story of the past. Computing is a not a chip problem, it’s a software and chip problem,” Huang said. What we are seeing now are software engineers offloading computationally expensive workloads to AIA's, more and more. This is enabled through the use of libraries like PyTorch and access to HPC levels of compute that were not broadly accessible before.
        MichaelZuo 3y ago
- marcosdumay 3y ago
  > With as much margin as cloud takes, there will be increased interest in on-prem and mixed environments. Or cheaper clouds... I wonder how much lock-in the current clouds actually have. And if they'll fully take customers down or entire industries hostage like the mainframes did.
  - latchkey 3y ago
    GPU clouds use long term reserved cloud contracts as lock-in. Lambda started off kind of bragging about shorter term contracts on their website and then changed that to one year after the demand picked up (and investors piled in tons of money). https://lambdalabs.com/blog/voltron-data-case-study-why-ml-teams-using-reserved-cloud-clusters https://lambdalabs.com/blog/voltron-data-case-study-why-ml-t... "While most other cloud companies offer contracts starting at one year, you can reserve a Lambda Reserved Cloud Cluster starting at six months."
- latchkey 3y ago
  GPUs are no longer GPUs... ai accelerators (AIA) is a better term. My MI300x don't have display ports. ;-) I actually see a lot of parallels to bitcoin mining. It started on cpus, then gpus, then fpgas and soon after asics. We are starting to see something similar in the AI space with groq and tenstorrent. The problem with asic's today is that in order to get to the same compute density of a AIA, due to AI's need for memory, you need significantly more hardware. Right now, MI300x is the leader with 192gb, if you do the math on groq, it just doesn't work out. This is part of the reason why the stopped selling to retail and are just focused on hosting services. It will be a while before the smaller chipmakers can get their hands on enough hbm to make a dent in the AIA market (due to AMD/NV hogging it all). Don't forget that AMD/NV can and will just come out with better designs too.
- nradov 3y ago
  The demand for compute utility capacity is effectively infinite. Eventually we'll move on from GPUs to some new architecture but the servers will still have to run in data centers that can provide enough power and cooling.
dylan604 3y ago
Do these solid floor racks still have troughs between the racks for flooding?
- wmf 3y ago
  Not that I've seen. It's just a concrete slab.
- ytjohn 3y ago
  Datacenters I've been in that were built with "no raised floors" and flooding in mind do have a "raised floor", but it just happens to be solid concrete. This is very much the same as a warehouse or factory floor. It's built several feet higher than the surrounding parking lot, so that everything is level with loading docks. If you're talking in-building flooding, not much going on there other than the racks themselves keep the equipment off the floor. All power is ran from overhead. A lot of modern "rack groups" (OCS, TSCIF, etc) have water limited water protection in mind.
juliangoldsmith 3y ago
Is any of this related to newer standards like Open Rack?
candiddevmike 3y ago
With the rise of WFH, I wonder if there could be a model (with some kind of trusted computing angle to prevent physical tampering) where you host your servers at employee locations/homes in a very distributed manner. Probably wouldn't fly with things like SOC2, but it's probably cheaper to pay/reimburse employees than pay for rack and network, especially with a small footprint.
- countvonbalzac 3y ago
  I wonder if that's less efficient though because now every employee has to manage a server, rather than having a small dedicated team that can manage hundreds or thousands of servers.
  - candiddevmike 3y ago
    Just have a fleet of NUC like things (that have ECC RAM ones) that you ship around if they break. You could have them boot from network/CDN and never worry about imaging, too. With 1Gb+ residential internet speeds, you could probably build a fairly robust and scalable platform like this.
    - solarpunk 3y ago
      If companies are seriously considering this, they should instead figure out how to distribute applications across their workstations to begin with, no need to distribute extra hardware if you can ensure security, connectivity, and uptime of a single laptop at each employees house.
    - ianburrell 3y ago
      Rack servers are significantly more powerful than NUC. This means going to need more little servers. And more employees or employees that host lots of servers and turn into point of failure. There are also powerful servers, like database, that can’t be split. How many people do you think have symmetric gigabit? The company is going to have to pay for employees to upgrade. And do some separation to keep from saturating link from either work or home.

echelon 3y ago

> The latest from NVIDIA requires liquid cooling, tons of power, and solid floors We won't be running on GPUs forever. The innovation in the chip space will continue. With as much margin as cloud takes, there will be increased interest in on-prem and mixed environments. We run our own GPUs in production as well as having a cloud footprint.

jedberg 3y ago
> We won't be running on GPUs forever. The innovation in the chip space will continue. Sure, but I'm seeing a divergence. There are a group of chips that are getting smaller and require less power and cooling that can do a lot of workloads that you used to need huge chips for, but there are the most advanced chips for the most advanced workloads getting bigger and more power hungry. There will always be the big customers who need the big chips, and that is what will be nearly impossible for them to self host.
- PLenz 3y ago
  This is the same argument that mainframe companies made in the 80s. Mass commodity systems will eventually kill the big gpu except for legacy and niche workloads. It's the tech infra circle of life.
  - cogman10 3y ago
    I'm less optimistic. Unlike the 80s we are quickly approaching the limits of node feature sizes. Barring radically new designs and materials, I think it's reasonable to assume that we are reaching the power:computational limits of silicon.
    - latchkey 3y ago
      Maybe in the case of CPUs. But, there is a fundamental technical shift going on away from computationally expensive things being done on CPUs and moving to AIAs. I suspect that this is why Jensen said Moore's law is dead.
      - refulgentis 3y ago
        Jensen said Moore's Law is dead because of the dichotomy explained a few times -- the free lunches are coming fewer and far between, you can't get a substantial performance boost and power decrease every 18 months, per Moore's Law. We are reaching the power:computational limits of silicon. Both GPU and CPU.
        latchkey 3y ago
        Here is the quote: “Moore’s Law’s dead,” Huang said, referring to the standard that the number of transistors on a chip doubles every two years. “And the ability for Moore’s Law to deliver twice the performance at the same cost, or at the same performance, half the cost, every year and a half, is over. It’s completely over, and so the idea that a chip is going to go down in cost over time, unfortunately, is a story of the past. Computing is a not a chip problem, it’s a software and chip problem,” Huang said. What we are seeing now are software engineers offloading computationally expensive workloads to AIA's, more and more. This is enabled through the use of libraries like PyTorch and access to HPC levels of compute that were not broadly accessible before.
        MichaelZuo 3y ago
marcosdumay 3y ago
> With as much margin as cloud takes, there will be increased interest in on-prem and mixed environments. Or cheaper clouds... I wonder how much lock-in the current clouds actually have. And if they'll fully take customers down or entire industries hostage like the mainframes did.
- latchkey 3y ago
  GPU clouds use long term reserved cloud contracts as lock-in. Lambda started off kind of bragging about shorter term contracts on their website and then changed that to one year after the demand picked up (and investors piled in tons of money). https://lambdalabs.com/blog/voltron-data-case-study-why-ml-teams-using-reserved-cloud-clusters https://lambdalabs.com/blog/voltron-data-case-study-why-ml-t... "While most other cloud companies offer contracts starting at one year, you can reserve a Lambda Reserved Cloud Cluster starting at six months."
latchkey 3y ago
GPUs are no longer GPUs... ai accelerators (AIA) is a better term. My MI300x don't have display ports. ;-) I actually see a lot of parallels to bitcoin mining. It started on cpus, then gpus, then fpgas and soon after asics. We are starting to see something similar in the AI space with groq and tenstorrent. The problem with asic's today is that in order to get to the same compute density of a AIA, due to AI's need for memory, you need significantly more hardware. Right now, MI300x is the leader with 192gb, if you do the math on groq, it just doesn't work out. This is part of the reason why the stopped selling to retail and are just focused on hosting services. It will be a while before the smaller chipmakers can get their hands on enough hbm to make a dent in the AIA market (due to AMD/NV hogging it all). Don't forget that AMD/NV can and will just come out with better designs too.
nradov 3y ago
The demand for compute utility capacity is effectively infinite. Eventually we'll move on from GPUs to some new architecture but the servers will still have to run in data centers that can provide enough power and cooling.

dylan604 3y ago

Do these solid floor racks still have troughs between the racks for flooding?

wmf 3y ago
Not that I've seen. It's just a concrete slab.
ytjohn 3y ago
Datacenters I've been in that were built with "no raised floors" and flooding in mind do have a "raised floor", but it just happens to be solid concrete. This is very much the same as a warehouse or factory floor. It's built several feet higher than the surrounding parking lot, so that everything is level with loading docks. If you're talking in-building flooding, not much going on there other than the racks themselves keep the equipment off the floor. All power is ran from overhead. A lot of modern "rack groups" (OCS, TSCIF, etc) have water limited water protection in mind.

juliangoldsmith 3y ago

Is any of this related to newer standards like Open Rack?

candiddevmike 3y ago

With the rise of WFH, I wonder if there could be a model (with some kind of trusted computing angle to prevent physical tampering) where you host your servers at employee locations/homes in a very distributed manner. Probably wouldn't fly with things like SOC2, but it's probably cheaper to pay/reimburse employees than pay for rack and network, especially with a small footprint.

countvonbalzac 3y ago
I wonder if that's less efficient though because now every employee has to manage a server, rather than having a small dedicated team that can manage hundreds or thousands of servers.
- candiddevmike 3y ago
  Just have a fleet of NUC like things (that have ECC RAM ones) that you ship around if they break. You could have them boot from network/CDN and never worry about imaging, too. With 1Gb+ residential internet speeds, you could probably build a fairly robust and scalable platform like this.
  - solarpunk 3y ago
    If companies are seriously considering this, they should instead figure out how to distribute applications across their workstations to begin with, no need to distribute extra hardware if you can ensure security, connectivity, and uptime of a single laptop at each employees house.
  - ianburrell 3y ago
    Rack servers are significantly more powerful than NUC. This means going to need more little servers. And more employees or employees that host lots of servers and turn into point of failure. There are also powerful servers, like database, that can’t be split. How many people do you think have symmetric gigabit? The company is going to have to pay for employees to upgrade. And do some separation to keep from saturating link from either work or home.