5 ms·

I wonder if there will come a cloud provider that solely uses consumer grade compute, local storage, and internal networking, with some kick ass network connect

I wonder if there will come a cloud provider that solely uses consumer grade compute, local storage, and internal networking, with some kick ass network connectivity and steal the cloud provider show?! Most applications that run on AWS and GCP don’t really need that premium Xeon/EPYC CPUs, enterprise-grade storage that is 10X the cost of consumer grade drives. Of course, those that do may justify the premium cost. But, the rest that just runs a few web servers behind a load balancer and non-critical services could use stuff on the cheap... Say cheaper digitalocean built upon bare metal consumer grade hardware - Ryzen CPU, consumer grade memory, this Local NVMe storage, and the onboard network card... may be I am crazy to think that’s enough. Heck such hardware may even perform better than several enterprise grade VMs provided by cloud providers.

Jonnax 6y ago
Well the consumer grade stuff generally has higher clocks so things may be faster. I think a lot of Devs have had the surprise that their laptop has higher single core performance than a server. I don't know to what extent but space is certainly a premium. For a unit of rack space you want to maximise the cores and memory. With a server grade multi socket CPU, you're also likely to pair it with at least 512gb of ram. Consumer CPUs have a much lower limit. Though Threadripper is a bit higher. There's also the availability side of things. How many CPUs can you get to populate your datacenter? Will it be easy to order 500 consumer CPUs if you're not a retailer? Intel/AMD do want to steer their enterprise customers to higher volume parts.
- reacharavindh 6y ago
  Yes, there certainly are operational reasons to go big with non consumer-grade servers. In many cases, it is warranted. I was only imagining the lower end of infrastructure needs where a provider can get away with consumer grade hardware and achieve substantially lower costs. Say, a provider building their own racks of ITX motherboards with consumer CPUs. I’m sure there are enough Retail SKUs to be purchased in open market for small scale... I think backblaze did just that with hard drives when they were starting out
spamizbad 6y ago
> I wonder if there will come a cloud provider that solely uses consumer grade compute, local storage, and internal networking, with some kick ass network connectivity and steal the cloud provider show?! Forget consumer grade: even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale. Most major cloud providers roll their own machines for a reason. I'm guessing any hardware savings would be evaporated by increased labor, power, and footprint costs, likely making you more expensive than other providers.
- moonbug 6y ago
  > even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale not if you know what you are doing.
  - insaneirish 6y ago
    LOL. Yes it is a pain. All these incumbent vendors have atrocious tools, management interfaces, and dealing with firmware is basically the same as 1990.
    - moonbug 6y ago
      and in your estimation, 30 years (actually closer to 20) isn't enough time to get good at a thing?
  - spamizbad 6y ago
    “Knowing what you are doing” — yes, and what they are doing is throwing more labor, more machines and more electricity at the problem than either Google or Facebook ever would. Major cloud providers got off “commodity enterprise” over a decade ago. There’s a reason for that.... the stuff is just bad.
bayindirh 6y ago
With today's tech, it doesn't make sense. I'll try to explain: > consumer grade compute Non-ECC compute is a no go from start. You won't believe how computers make mistakes when they're driven to their limits. Even with good cooling, frying RAM, CPUs and ethernet gear is ordinary events. > local storage Consumer grade HDDs are slow. SAS is not consumer grade. Consumer grade SSDs cannot handle the write cycles. The ones which can handle neither cheap nor consumer grade. > internal networking Gigabit ethernet is cheap but, is not fast enough anymore. 10Gig+, Infiniband and other stuff needs reliable high bandwidth PCIe, which is not in abdunance in consumer grade hardware. You need more lanes, then we come to next point. > Most applications that run on AWS and GCP don’t really need that premium Xeon/EPYC CPUs Server CPUs are not about processing power all the time. Reliability, PCIe lanes, expansion and advanced virtualization which allows that high performance network to be connected in abundance and shared between VMs without performance penalties. So you can scale the number of jobs running on less hardware in smaller spaces. > enterprise-grade storage that is 10X the cost of consumer grade drives. Again, it's not about performance but about reliability at least half the time. Performance is required to be able to scale, not for independent jobs.
- maccard 6y ago
  > Non-ECC compute is a no go from start. Many modern amd offerings support ecc memory, but I can't comment on intel. > Consumer grade HDDs are slow. Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s. Getting a medium performing SSD seems to cost in the realm of $250/month just for the storage, before computer costs. For many uses, users don't care about true reliability. I don't care if my CI agent has data loss, I'd _much_ much rather it ran on consumer grade hardware at speed and failed 2-3% of the time than what I've got now (as long as I have the choice. I want my build servers to run on fast unreliable hardware, for example) Agreed re: pcie though!
  - gruez 6y ago
    >Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s That's because they're not bare hard drives directly connected to the server, they're a virtual hard drive over SAN.
johnjungles 6y ago
Isn’t reliability a huge deal for servers thou? There must be stats on hardware failure rates where the cloud provider cpus should have significantly less failure rates than consumer grade ones, I’m going to guess that that’s gotta be a trade off. Consumer CPUs clock faster but the thermal controls are dependent on a single machine, where as the data center has to deal with packed server farms, thus sacrificing performance for long term thermal management.
deleted 6y ago
[deleted]
jgalt212 6y ago
I'm more thinking along the lines of arm server (low power) + VNMe storage + WAF = end of cloud needs for many current cloud consumers
daper 6y ago
Some providers are doing that. For example you can rent such bare metal servers from OVH's cheap brand So You Start. But that doesn't make sense if you need something more that one cheapest server. Consider the total cost of buying and hosting such servers compared to the resources they provide. You are paying more but mainly when the capabilities are better. Example for SSD: Samsung 860 PRO 1TB and "for Data Center" Samsung PM883 1TB both have about 1200 TBW lifespan, both cost about $180. On the other hand 8-core Xeon 4215 is almost 2 times more expensive than 8-core consumer CPU but it has 3x the memory bandwidth (6 channel controller vs 2 channel), 1TB of RAM per processor vs 128GB, 48 PCIe lanes vs 20 and supports 2-way SMP. That means you can put few times more capacity (cores, memory, in 1U rack enclosure than with consumer hardware. That additional capacity can also share 1 PSU, 1 network card. I'm not even talking about ECC memory and remote management capabilities available only in server grade hardware.
- ta988 6y ago
  ECC works on consumer motherboards nowadays, my X570 motherboard works really well with ECC.
christophilus 6y ago
I’m pretty sure Backblaze uses consumer grade hard drives.

Jonnax 6y ago

Well the consumer grade stuff generally has higher clocks so things may be faster. I think a lot of Devs have had the surprise that their laptop has higher single core performance than a server. I don't know to what extent but space is certainly a premium. For a unit of rack space you want to maximise the cores and memory. With a server grade multi socket CPU, you're also likely to pair it with at least 512gb of ram. Consumer CPUs have a much lower limit. Though Threadripper is a bit higher. There's also the availability side of things. How many CPUs can you get to populate your datacenter? Will it be easy to order 500 consumer CPUs if you're not a retailer? Intel/AMD do want to steer their enterprise customers to higher volume parts.

reacharavindh 6y ago
Yes, there certainly are operational reasons to go big with non consumer-grade servers. In many cases, it is warranted. I was only imagining the lower end of infrastructure needs where a provider can get away with consumer grade hardware and achieve substantially lower costs. Say, a provider building their own racks of ITX motherboards with consumer CPUs. I’m sure there are enough Retail SKUs to be purchased in open market for small scale... I think backblaze did just that with hard drives when they were starting out

spamizbad 6y ago

> I wonder if there will come a cloud provider that solely uses consumer grade compute, local storage, and internal networking, with some kick ass network connectivity and steal the cloud provider show?! Forget consumer grade: even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale. Most major cloud providers roll their own machines for a reason. I'm guessing any hardware savings would be evaporated by increased labor, power, and footprint costs, likely making you more expensive than other providers.

moonbug 6y ago
> even normal Dell/HP/Supermicro stuff is a PITA to run at datacenter scale not if you know what you are doing.
- insaneirish 6y ago
  LOL. Yes it is a pain. All these incumbent vendors have atrocious tools, management interfaces, and dealing with firmware is basically the same as 1990.
  - moonbug 6y ago
    and in your estimation, 30 years (actually closer to 20) isn't enough time to get good at a thing?
- spamizbad 6y ago
  “Knowing what you are doing” — yes, and what they are doing is throwing more labor, more machines and more electricity at the problem than either Google or Facebook ever would. Major cloud providers got off “commodity enterprise” over a decade ago. There’s a reason for that.... the stuff is just bad.

bayindirh 6y ago

With today's tech, it doesn't make sense. I'll try to explain: > consumer grade compute Non-ECC compute is a no go from start. You won't believe how computers make mistakes when they're driven to their limits. Even with good cooling, frying RAM, CPUs and ethernet gear is ordinary events. > local storage Consumer grade HDDs are slow. SAS is not consumer grade. Consumer grade SSDs cannot handle the write cycles. The ones which can handle neither cheap nor consumer grade. > internal networking Gigabit ethernet is cheap but, is not fast enough anymore. 10Gig+, Infiniband and other stuff needs reliable high bandwidth PCIe, which is not in abdunance in consumer grade hardware. You need more lanes, then we come to next point. > Most applications that run on AWS and GCP don’t really need that premium Xeon/EPYC CPUs Server CPUs are not about processing power all the time. Reliability, PCIe lanes, expansion and advanced virtualization which allows that high performance network to be connected in abundance and shared between VMs without performance penalties. So you can scale the number of jobs running on less hardware in smaller spaces. > enterprise-grade storage that is 10X the cost of consumer grade drives. Again, it's not about performance but about reliability at least half the time. Performance is required to be able to scale, not for independent jobs.

maccard 6y ago
> Non-ECC compute is a no go from start. Many modern amd offerings support ecc memory, but I can't comment on intel. > Consumer grade HDDs are slow. Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s. Getting a medium performing SSD seems to cost in the realm of $250/month just for the storage, before computer costs. For many uses, users don't care about true reliability. I don't care if my CI agent has data loss, I'd _much_ much rather it ran on consumer grade hardware at speed and failed 2-3% of the time than what I've got now (as long as I have the choice. I want my build servers to run on fast unreliable hardware, for example) Agreed re: pcie though!
- gruez 6y ago
  >Meanwhile cloud provider hdd storage is slower than usb2. I have a 32 core machine deployed in azure right now, and the hdd that was deployed with it has a max read speed of 10MB/s That's because they're not bare hard drives directly connected to the server, they're a virtual hard drive over SAN.

johnjungles 6y ago

Isn’t reliability a huge deal for servers thou? There must be stats on hardware failure rates where the cloud provider cpus should have significantly less failure rates than consumer grade ones, I’m going to guess that that’s gotta be a trade off. Consumer CPUs clock faster but the thermal controls are dependent on a single machine, where as the data center has to deal with packed server farms, thus sacrificing performance for long term thermal management.

deleted 6y ago

[deleted]

jgalt212 6y ago

I'm more thinking along the lines of arm server (low power) + VNMe storage + WAF = end of cloud needs for many current cloud consumers

daper 6y ago

Some providers are doing that. For example you can rent such bare metal servers from OVH's cheap brand So You Start. But that doesn't make sense if you need something more that one cheapest server. Consider the total cost of buying and hosting such servers compared to the resources they provide. You are paying more but mainly when the capabilities are better. Example for SSD: Samsung 860 PRO 1TB and "for Data Center" Samsung PM883 1TB both have about 1200 TBW lifespan, both cost about $180. On the other hand 8-core Xeon 4215 is almost 2 times more expensive than 8-core consumer CPU but it has 3x the memory bandwidth (6 channel controller vs 2 channel), 1TB of RAM per processor vs 128GB, 48 PCIe lanes vs 20 and supports 2-way SMP. That means you can put few times more capacity (cores, memory, in 1U rack enclosure than with consumer hardware. That additional capacity can also share 1 PSU, 1 network card. I'm not even talking about ECC memory and remote management capabilities available only in server grade hardware.

ta988 6y ago
ECC works on consumer motherboards nowadays, my X570 motherboard works really well with ECC.

christophilus 6y ago

I’m pretty sure Backblaze uses consumer grade hard drives.