6 ms·

Seeing the really just puny "provisioned IOPS" numbers on hugely expensive cloud instances made me chuckle (first in disbelief, then in horror) when I joined a

by c0l0 3y ago

Seeing the really just puny "provisioned IOPS" numbers on hugely expensive cloud instances made me chuckle (first in disbelief, then in horror) when I joined a "cloud-first" enterprise shop in 2020 (having come from a company that hosted their own hardware at a colo). It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. Ultra-fast (I'm not parroting some marketing speak here - I just take a look at IOPS numbers, and compare them to those from highest-end storage some 10-12 years ago) NVMe storage is a big part of that astonishing magic.

Aurornis 3y ago
> It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Many know what hardware can do. There’s no need to dunk on young people. Anyway, the cloud performance realities are well know to anyone who works in cloud performance. It’s part of the game and it’s learned by anyone scaling a system. It doesn’t really matter what you could do if you build a couple RUs yourself and hauled them down to the data center, because beyond simple single-purpose applications with flexible uptime requirements, that’s not a realistic option.
- zten 3y ago
  > On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Yes, this is often a big surprise. You can test out some disk-heavy app locally on your laptop and observe decent performance, and then have your day completely ruined when you provision a slice of an NVMe SSD instance type (like, i4i.2xlarge) and discover you're only paying for SATA SSD performance.
  - seabrookmx 3y ago
    This doesn't stop at SSD's. Spin up an E2 VM in Google Cloud and there's a good chance you'll get a nearly 9 year Broadwell architecture chip running your workload!
    - bmicraft 3y ago
      What this tells me is that the price of running inefficient cpus seemingly isn't nearly as high as I thought it would or should be (in terms of usd/kWh)
      - seabrookmx 3y ago
        Well they bill you for the instance not for some unit of computation. I'd imagine many users of E2 instances don't realize that they could be getting much much worse performance per vcore than if they picked a different instance type. From Google's perspective, if the hardware is paid for, still reliable, and they can still make money on it, they can put new hardware in new racks rather than replacing the old hardware. This suggests Google's DC's aren't space constrained but I'm not surprised after looking at a few via satellite images!
        bmicraft 3y ago
        Well not exactly. In my mind the price of running such old cpus for say the last (say, 4?) years would have been higher than buying new+new runtime costs. Those would definitely be considered opportunity costs that ought to be avoided.
        seabrookmx 3y ago
        > the price of running such old cpus for say the last (say, 4?) years would have been higher than buying new+new runtime costs I don't think this is true, because the old chips don't use more power outright[1][2][3]. In fact in many cases new chips use more power due to the higher core density. The new chips are way more efficient because they do more work per watt, but like I said in my previous comment you aren't paying for a unit of work. The billing model for the cloud providers is that of a rental: you pay per minute for the instance. There's complexity here like being able to pack more "instances" (VM's) onto a physical host with the higher core count machines, but simply saying the new hardware is cheaper to run I don't think is clear cut. [1]: https://cloud.google.com/compute/docs/cpu-platforms#intel_processors https://cloud.google.com/compute/docs/cpu-platforms#intel_pr... [2]: https://www.intel.com/content/www/us/en/products/sku/93792/intel-xeon-processor-e78880-v4-55m-cache-2-20-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/93792/i... [3]: https://www.intel.com/content/www/us/en/products/sku/231746/intel-xeon-platinum-8480-processor-105m-cache-2-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/231746/...
- EB66 3y ago
  > because beyond simple single-purpose applications with flexible uptime requirements, that’s not a realistic option. I frequently hear this point expressed in cloud vs colo debates. The notion that you can't achieve high availability with simple colo deploys is just nonsense. Two colo deploys in two geographically distinct datacenters, two active physical servers with identical builds (RAIDed drives, dual NICs, A+B power) in both datacenters, a third server racked up just sitting as a cold spare, pick your favorite container orchestration scheme, rig up your database replication, script the database failover activation process, add HAProxy (or use whatever built-in scheme your orchestration system offers), sprinkle in a cloud service for DNS load balancing/failover (Cloudflare or AWS Route 53), automate and store backups off-site and you're done. Yes it's a lot of work, but so is configuring a similar level of redundancy and high availability in AWS. I've done it both ways and I prefer the bare metal colo approach. With colo you get vastly more bang for your buck and when things go wrong, you have a greater ability to get hands on, understand exactly what's going on and fix it immediately.
  - joshstrange 3y ago
    I doubt you’ll find anyone who disagrees that colo is much cheaper and that it’s possible to have failover with little to no downtime. Same with higher performance on bare metal vs a public cloud. Or at least I’ve never thought differently. The difference is setting up all of that and maintaining it/debugging when something goes wrong is not a small task IMHO. For some companies with that experience in-house I can understand doing it all yourself. As a solo founder and an employee of a small company we don’t have the bandwidth to do all of that without hiring 1+ more people which are more expensive than the cloud costs. If we were drive-speed-constrained and getting that speed just wasn’t possible then maybe the math would shift further in favor of colo but we aren’t. Also upgrading the hardware our servers run on is fairly straightforward vs replacing a server on a rack or dealing with failing/older hardware.
jauntywundrkind 3y ago
NVMe has been ridiculously great. I'm excited to see what happens to prices as E1 form factor ramps up! Much physically bigger drives allows for consolidation of parts, a higher ratio of flash chips to everything else, which seems promising. It's more a value line, but Intel's P5315 is 15TB at a quite low $0.9/GB. It might not help much with oops though. Amazing that we have PCIe 5.0 16GB/s and already are so near theoretical max (some lost to overhead), even on consumer cards. Going enterprise for the drive-writes-per-day (DWPD) is 100% worth it for most folks, but I am morbidly curious how different the performance profile would be running enterprise vs non these days. But reciprocally the high DWPD drives (Kioxia CD8P-V for example is DWPD of 3) seems to often come with somewhat more mild sustained 4k write oops, making me think maybe there's a speed vs reliability tradeoff that could be taken advantage of from consumer drives in some cases; not sure who wants tons of iops but doesn't actually intend to hit their Total Drive Writes, but it save you some iops/$ if so. That said, I'm shocked to see the enterprise premium is a lot less absurd than it used to be! (If you can find stock.)
- bcaxis 3y ago
  The main problem with consumer drives is the missing power loss protection (plp). M.2 drives just don't have space for the caps like an enterprise 2.5 u.2/u.3 drive will have. This matters when the DB calls a sync and it's expecting the data to be written safely to disk before it returns. A consumer drive basically stops everything until it can report success and your IOPS falls to like 1/100th of what the drive is capable of if it's happening alot. An enterprise drive with plp will just report success knowing it has the power to finish the pending writes. Full speed ahead. You can "lie" to the process at the VPS level by enabling unsafe write back cache. You can do it at the OS level by launching the DB with "eatmydata". You will get the full performance of your SSD. In the event of power loss you may well end up in an unrecoverable corrupted condition with these enabled. I believe that if you buy all consumer parts - an enterprise drive is the best place to up spend your money profitably on an enterprise bit.
  - tumult 3y ago
    My experience lately is that consumer drives will also lie and use a cache, but then drop your data on the floor if the power is lost or there’s a kernel panic / BSOD. (Samsung and others.)
dboreham 3y ago
Some of us are making a good living offboarding workloads from cloud onto bare metal with on-node NVMe storage.
- dijit 3y ago
  Really? I'd like to do this as a job. Are you hiring? Cloud is great for prototyping or randomly elastic workloads, but it feels like people are pushing highly static workloads from on-prem to cloud. I'd love to be part of the change going the other way. Especially since the skills for doing so seem to have dried up completely.

by c0l0 3y ago

Aurornis 3y ago

> It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Many know what hardware can do. There’s no need to dunk on young people. Anyway, the cloud performance realities are well know to anyone who works in cloud performance. It’s part of the game and it’s learned by anyone scaling a system. It doesn’t really matter what you could do if you build a couple RUs yourself and hauled them down to the data center, because beyond simple single-purpose applications with flexible uptime requirements, that’s not a realistic option.

zten 3y ago
> On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Yes, this is often a big surprise. You can test out some disk-heavy app locally on your laptop and observe decent performance, and then have your day completely ruined when you provision a slice of an NVMe SSD instance type (like, i4i.2xlarge) and discover you're only paying for SATA SSD performance.
- seabrookmx 3y ago
  This doesn't stop at SSD's. Spin up an E2 VM in Google Cloud and there's a good chance you'll get a nearly 9 year Broadwell architecture chip running your workload!
  - bmicraft 3y ago
    What this tells me is that the price of running inefficient cpus seemingly isn't nearly as high as I thought it would or should be (in terms of usd/kWh)
    - seabrookmx 3y ago
      Well they bill you for the instance not for some unit of computation. I'd imagine many users of E2 instances don't realize that they could be getting much much worse performance per vcore than if they picked a different instance type. From Google's perspective, if the hardware is paid for, still reliable, and they can still make money on it, they can put new hardware in new racks rather than replacing the old hardware. This suggests Google's DC's aren't space constrained but I'm not surprised after looking at a few via satellite images!
      - bmicraft 3y ago
        Well not exactly. In my mind the price of running such old cpus for say the last (say, 4?) years would have been higher than buying new+new runtime costs. Those would definitely be considered opportunity costs that ought to be avoided.
        seabrookmx 3y ago
        > the price of running such old cpus for say the last (say, 4?) years would have been higher than buying new+new runtime costs I don't think this is true, because the old chips don't use more power outright[1][2][3]. In fact in many cases new chips use more power due to the higher core density. The new chips are way more efficient because they do more work per watt, but like I said in my previous comment you aren't paying for a unit of work. The billing model for the cloud providers is that of a rental: you pay per minute for the instance. There's complexity here like being able to pack more "instances" (VM's) onto a physical host with the higher core count machines, but simply saying the new hardware is cheaper to run I don't think is clear cut. [1]: https://cloud.google.com/compute/docs/cpu-platforms#intel_processors https://cloud.google.com/compute/docs/cpu-platforms#intel_pr... [2]: https://www.intel.com/content/www/us/en/products/sku/93792/intel-xeon-processor-e78880-v4-55m-cache-2-20-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/93792/i... [3]: https://www.intel.com/content/www/us/en/products/sku/231746/intel-xeon-platinum-8480-processor-105m-cache-2-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/231746/...
EB66 3y ago
> because beyond simple single-purpose applications with flexible uptime requirements, that’s not a realistic option. I frequently hear this point expressed in cloud vs colo debates. The notion that you can't achieve high availability with simple colo deploys is just nonsense. Two colo deploys in two geographically distinct datacenters, two active physical servers with identical builds (RAIDed drives, dual NICs, A+B power) in both datacenters, a third server racked up just sitting as a cold spare, pick your favorite container orchestration scheme, rig up your database replication, script the database failover activation process, add HAProxy (or use whatever built-in scheme your orchestration system offers), sprinkle in a cloud service for DNS load balancing/failover (Cloudflare or AWS Route 53), automate and store backups off-site and you're done. Yes it's a lot of work, but so is configuring a similar level of redundancy and high availability in AWS. I've done it both ways and I prefer the bare metal colo approach. With colo you get vastly more bang for your buck and when things go wrong, you have a greater ability to get hands on, understand exactly what's going on and fix it immediately.
- joshstrange 3y ago
  I doubt you’ll find anyone who disagrees that colo is much cheaper and that it’s possible to have failover with little to no downtime. Same with higher performance on bare metal vs a public cloud. Or at least I’ve never thought differently. The difference is setting up all of that and maintaining it/debugging when something goes wrong is not a small task IMHO. For some companies with that experience in-house I can understand doing it all yourself. As a solo founder and an employee of a small company we don’t have the bandwidth to do all of that without hiring 1+ more people which are more expensive than the cloud costs. If we were drive-speed-constrained and getting that speed just wasn’t possible then maybe the math would shift further in favor of colo but we aren’t. Also upgrading the hardware our servers run on is fairly straightforward vs replacing a server on a rack or dealing with failing/older hardware.

jauntywundrkind 3y ago

NVMe has been ridiculously great. I'm excited to see what happens to prices as E1 form factor ramps up! Much physically bigger drives allows for consolidation of parts, a higher ratio of flash chips to everything else, which seems promising. It's more a value line, but Intel's P5315 is 15TB at a quite low $0.9/GB. It might not help much with oops though. Amazing that we have PCIe 5.0 16GB/s and already are so near theoretical max (some lost to overhead), even on consumer cards. Going enterprise for the drive-writes-per-day (DWPD) is 100% worth it for most folks, but I am morbidly curious how different the performance profile would be running enterprise vs non these days. But reciprocally the high DWPD drives (Kioxia CD8P-V for example is DWPD of 3) seems to often come with somewhat more mild sustained 4k write oops, making me think maybe there's a speed vs reliability tradeoff that could be taken advantage of from consumer drives in some cases; not sure who wants tons of iops but doesn't actually intend to hit their Total Drive Writes, but it save you some iops/$ if so. That said, I'm shocked to see the enterprise premium is a lot less absurd than it used to be! (If you can find stock.)

bcaxis 3y ago
The main problem with consumer drives is the missing power loss protection (plp). M.2 drives just don't have space for the caps like an enterprise 2.5 u.2/u.3 drive will have. This matters when the DB calls a sync and it's expecting the data to be written safely to disk before it returns. A consumer drive basically stops everything until it can report success and your IOPS falls to like 1/100th of what the drive is capable of if it's happening alot. An enterprise drive with plp will just report success knowing it has the power to finish the pending writes. Full speed ahead. You can "lie" to the process at the VPS level by enabling unsafe write back cache. You can do it at the OS level by launching the DB with "eatmydata". You will get the full performance of your SSD. In the event of power loss you may well end up in an unrecoverable corrupted condition with these enabled. I believe that if you buy all consumer parts - an enterprise drive is the best place to up spend your money profitably on an enterprise bit.
- tumult 3y ago
  My experience lately is that consumer drives will also lie and use a cache, but then drop your data on the floor if the power is lost or there’s a kernel panic / BSOD. (Samsung and others.)

dboreham 3y ago

Some of us are making a good living offboarding workloads from cloud onto bare metal with on-node NVMe storage.

dijit 3y ago
Really? I'd like to do this as a job. Are you hiring? Cloud is great for prototyping or randomly elastic workloads, but it feels like people are pushing highly static workloads from on-prem to cloud. I'd love to be part of the change going the other way. Especially since the skills for doing so seem to have dried up completely.