4 ms·

EBS costs crazy crazy amounts for reasonable iops We pay 7k per month for RDS that can do barely 2k iops.. in the same time a machine at hetzner does 2 million

EBS costs crazy crazy amounts for reasonable iops We pay 7k per month for RDS that can do barely 2k iops.. in the same time a machine at hetzner does 2 million iops for 250 euro per month (not to mention it also have 4x more codes and 5x more ram). So, even though I/O is no longer the bottle neck physically, it still is a considerable issue and design challenge on the cloud.

nix23 4y ago
Well yes it's a total ripoff I installed a DB-Server for a Customer around 2years ago, in a DC near him with 16 cores 48GB Ram and ~6TB -> 12 SSD, vDevs mirror with 2, and Stripe over the mirrored vDevs (kind of a Raid10 but zfs), compression zstd (1GB could be compressed down to ~200MB so 5 times less reading/writing, and in theory ~30TB of pure DB-Data, 20TB realistic, remember never fill a zpool over 72%) record-size 16kb (postgresql). After 3 month the machine was paid (compared to the "cloud"-price) and the performance kind of 10-12 times higher. Called the customer about a two month ago and he said the DB-Server is still to fast and maybe he wants another one who uses less power... ;)
- samb1729 4y ago
  > remember never fill a zpool over 72% Could you please explain where this number comes from?
  - ewwhite 4y ago
    Yeah, that's a myth now. It's not current advice.
    - nix23 4y ago
      72% is my rule of thumb for write heavy production stuff (my absolute limit would be 75%) but it depends on record-size, raidz-level, if you have mostly write or mostly read workloads, how big your files are, how many snapshots you have, if you have a dedicated ZIL-Device and much more. For a Home-NAS (Movies etc) you can easy go up to 85%...if it's a "~WORM" workload maybe 90%...but resilvering can then be a thing of days (weeks?), depends on the raidz-level or mirror etc. >Yeah, that's a myth now. It's not current advice. It's not and you know it, keep it under 72% believe me if you want a performant zfs (especially if you delete files and have many snapshots...check the YT linked at the end) >>Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-96% utilization range. Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer. https://web.archive.org/web/20150905142644/http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#General_Storage_Pool_Performance_Considerations https://web.archive.org/web/20150905142644/http://www.solari... And under no circumstances go over 90%: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#free-space https://openzfs.github.io/openzfs-docs/Performance%20and%20T... >An introduction to the implementation of ZFS - Kirk McKusick https://www.youtube.com/watch?v=TQe-nnJPNF8 https://www.youtube.com/watch?v=TQe-nnJPNF8
- fomine3 4y ago
  Virtualized storage system increases latency (QD1 IOPS). Naively built non-parallel apps tend to rely on QD1 IOPS performance, so it runs very slowly on cloud platform, compared to dev machine with direct attached NVMe.
  - nix23 4y ago
    >Naively built non-parallel apps tend to rely on QD1 IOPS performance PostgreSQL is pretty much parallel but i know what you mean...the beehive ;)
- ilyt 4y ago
  The cloud costs really are between "unreasonable" and "very unreasonable". The only time when it gets cheaper is if workload would be so spiky we coul'dve turned most of it off for 2/3 of the day but most of what we have is many smaller projects that don't even have enough traffic to even need scaling and the big ones, well, can't scale database size down off-peak... Over last ~6 years we did "is it worth going to cloud" calculation few times and it was always ridiculously more expensive.
LastTrain 4y ago
Agree! And at the end of the day, we are optimizing for cost. Although, the EBS portion of that 7k RDS bill is going to be tiny, right?
- throwaway71271 4y ago
  well its tiny if you keep it 2k, and make sure you dont touch disk much, god forbid someone makes a query that requires a temp table.. a query you wouldnt even notice in a baremetal machine brings the whole rds setup down, cant even read the writeahead log and cant replicate and etc.. its like watching a slow motion train wreck from 2 queries per second

nix23 4y ago

Well yes it's a total ripoff I installed a DB-Server for a Customer around 2years ago, in a DC near him with 16 cores 48GB Ram and ~6TB -> 12 SSD, vDevs mirror with 2, and Stripe over the mirrored vDevs (kind of a Raid10 but zfs), compression zstd (1GB could be compressed down to ~200MB so 5 times less reading/writing, and in theory ~30TB of pure DB-Data, 20TB realistic, remember never fill a zpool over 72%) record-size 16kb (postgresql). After 3 month the machine was paid (compared to the "cloud"-price) and the performance kind of 10-12 times higher. Called the customer about a two month ago and he said the DB-Server is still to fast and maybe he wants another one who uses less power... ;)

samb1729 4y ago
> remember never fill a zpool over 72% Could you please explain where this number comes from?
- ewwhite 4y ago
  Yeah, that's a myth now. It's not current advice.
  - nix23 4y ago
    72% is my rule of thumb for write heavy production stuff (my absolute limit would be 75%) but it depends on record-size, raidz-level, if you have mostly write or mostly read workloads, how big your files are, how many snapshots you have, if you have a dedicated ZIL-Device and much more. For a Home-NAS (Movies etc) you can easy go up to 85%...if it's a "~WORM" workload maybe 90%...but resilvering can then be a thing of days (weeks?), depends on the raidz-level or mirror etc. >Yeah, that's a myth now. It's not current advice. It's not and you know it, keep it under 72% believe me if you want a performant zfs (especially if you delete files and have many snapshots...check the YT linked at the end) >>Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-96% utilization range. Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer. https://web.archive.org/web/20150905142644/http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#General_Storage_Pool_Performance_Considerations https://web.archive.org/web/20150905142644/http://www.solari... And under no circumstances go over 90%: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#free-space https://openzfs.github.io/openzfs-docs/Performance%20and%20T... >An introduction to the implementation of ZFS - Kirk McKusick https://www.youtube.com/watch?v=TQe-nnJPNF8 https://www.youtube.com/watch?v=TQe-nnJPNF8
fomine3 4y ago
Virtualized storage system increases latency (QD1 IOPS). Naively built non-parallel apps tend to rely on QD1 IOPS performance, so it runs very slowly on cloud platform, compared to dev machine with direct attached NVMe.
- nix23 4y ago
  >Naively built non-parallel apps tend to rely on QD1 IOPS performance PostgreSQL is pretty much parallel but i know what you mean...the beehive ;)
ilyt 4y ago
The cloud costs really are between "unreasonable" and "very unreasonable". The only time when it gets cheaper is if workload would be so spiky we coul'dve turned most of it off for 2/3 of the day but most of what we have is many smaller projects that don't even have enough traffic to even need scaling and the big ones, well, can't scale database size down off-peak... Over last ~6 years we did "is it worth going to cloud" calculation few times and it was always ridiculously more expensive.

LastTrain 4y ago

Agree! And at the end of the day, we are optimizing for cost. Although, the EBS portion of that 7k RDS bill is going to be tiny, right?

throwaway71271 4y ago
well its tiny if you keep it 2k, and make sure you dont touch disk much, god forbid someone makes a query that requires a temp table.. a query you wouldnt even notice in a baremetal machine brings the whole rds setup down, cant even read the writeahead log and cant replicate and etc.. its like watching a slow motion train wreck from 2 queries per second