3 ms·

Well yes it's a total ripoff I installed a DB-Server for a Customer around 2years ago, in a DC near him with 16 cores 48GB Ram and ~6TB -> 12 SSD, vDevs mirror

by nix23 4y ago

Well yes it's a total ripoff I installed a DB-Server for a Customer around 2years ago, in a DC near him with 16 cores 48GB Ram and ~6TB -> 12 SSD, vDevs mirror with 2, and Stripe over the mirrored vDevs (kind of a Raid10 but zfs), compression zstd (1GB could be compressed down to ~200MB so 5 times less reading/writing, and in theory ~30TB of pure DB-Data, 20TB realistic, remember never fill a zpool over 72%) record-size 16kb (postgresql). After 3 month the machine was paid (compared to the "cloud"-price) and the performance kind of 10-12 times higher. Called the customer about a two month ago and he said the DB-Server is still to fast and maybe he wants another one who uses less power... ;)

samb1729 4y ago
> remember never fill a zpool over 72% Could you please explain where this number comes from?
- ewwhite 4y ago
  Yeah, that's a myth now. It's not current advice.
  - nix23 4y ago
    72% is my rule of thumb for write heavy production stuff (my absolute limit would be 75%) but it depends on record-size, raidz-level, if you have mostly write or mostly read workloads, how big your files are, how many snapshots you have, if you have a dedicated ZIL-Device and much more. For a Home-NAS (Movies etc) you can easy go up to 85%...if it's a "~WORM" workload maybe 90%...but resilvering can then be a thing of days (weeks?), depends on the raidz-level or mirror etc. >Yeah, that's a myth now. It's not current advice. It's not and you know it, keep it under 72% believe me if you want a performant zfs (especially if you delete files and have many snapshots...check the YT linked at the end) >>Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-96% utilization range. Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer. https://web.archive.org/web/20150905142644/http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#General_Storage_Pool_Performance_Considerations https://web.archive.org/web/20150905142644/http://www.solari... And under no circumstances go over 90%: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#free-space https://openzfs.github.io/openzfs-docs/Performance%20and%20T... >An introduction to the implementation of ZFS - Kirk McKusick https://www.youtube.com/watch?v=TQe-nnJPNF8 https://www.youtube.com/watch?v=TQe-nnJPNF8
fomine3 4y ago
Virtualized storage system increases latency (QD1 IOPS). Naively built non-parallel apps tend to rely on QD1 IOPS performance, so it runs very slowly on cloud platform, compared to dev machine with direct attached NVMe.
- nix23 4y ago
  >Naively built non-parallel apps tend to rely on QD1 IOPS performance PostgreSQL is pretty much parallel but i know what you mean...the beehive ;)
ilyt 4y ago
The cloud costs really are between "unreasonable" and "very unreasonable". The only time when it gets cheaper is if workload would be so spiky we coul'dve turned most of it off for 2/3 of the day but most of what we have is many smaller projects that don't even have enough traffic to even need scaling and the big ones, well, can't scale database size down off-peak... Over last ~6 years we did "is it worth going to cloud" calculation few times and it was always ridiculously more expensive.

by nix23 4y ago

samb1729 4y ago

> remember never fill a zpool over 72% Could you please explain where this number comes from?

ewwhite 4y ago
Yeah, that's a myth now. It's not current advice.
- nix23 4y ago
  72% is my rule of thumb for write heavy production stuff (my absolute limit would be 75%) but it depends on record-size, raidz-level, if you have mostly write or mostly read workloads, how big your files are, how many snapshots you have, if you have a dedicated ZIL-Device and much more. For a Home-NAS (Movies etc) you can easy go up to 85%...if it's a "~WORM" workload maybe 90%...but resilvering can then be a thing of days (weeks?), depends on the raidz-level or mirror etc. >Yeah, that's a myth now. It's not current advice. It's not and you know it, keep it under 72% believe me if you want a performant zfs (especially if you delete files and have many snapshots...check the YT linked at the end) >>Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-96% utilization range. Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer. https://web.archive.org/web/20150905142644/http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#General_Storage_Pool_Performance_Considerations https://web.archive.org/web/20150905142644/http://www.solari... And under no circumstances go over 90%: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html#free-space https://openzfs.github.io/openzfs-docs/Performance%20and%20T... >An introduction to the implementation of ZFS - Kirk McKusick https://www.youtube.com/watch?v=TQe-nnJPNF8 https://www.youtube.com/watch?v=TQe-nnJPNF8

fomine3 4y ago

Virtualized storage system increases latency (QD1 IOPS). Naively built non-parallel apps tend to rely on QD1 IOPS performance, so it runs very slowly on cloud platform, compared to dev machine with direct attached NVMe.

nix23 4y ago
>Naively built non-parallel apps tend to rely on QD1 IOPS performance PostgreSQL is pretty much parallel but i know what you mean...the beehive ;)

ilyt 4y ago

The cloud costs really are between "unreasonable" and "very unreasonable". The only time when it gets cheaper is if workload would be so spiky we coul'dve turned most of it off for 2/3 of the day but most of what we have is many smaller projects that don't even have enough traffic to even need scaling and the big ones, well, can't scale database size down off-peak... Over last ~6 years we did "is it worth going to cloud" calculation few times and it was always ridiculously more expensive.