5 ms·

I saw the section on EBS, but it didn't offer much advice. Getting good performance on networked storage is the biggest challenge to me. The last time I asked a

I saw the section on EBS, but it didn't offer much advice. Getting good performance on networked storage is the biggest challenge to me. The last time I asked about that here [1], I got this answer: nasalgoat 161 days ago [-] The secret to EBS is to use General SSD, not Provisioned, but use a RAID stripe. The reason this works is because IOPS are provisioned per EBS drive and by the size of the drive. So a RAID0 stripe of, say, ten General SSD drives will outperform the more expensive PIOPS single drive. That sounds like a great approach, although I haven't had time to try it out yet. I'm curious if anyone else has done anything like that. [1] https://news.ycombinator.com/item?id=12609172 https://news.ycombinator.com/item?id=12609172

deleted 10y ago
[deleted]
user5994461 10y ago
Use multiple big* SSD volumes in RAID 0 on big* instances. * IOPS and bandwidth are proportional to the size of volumes * instance types have a hard cap on network and IO, proportional to the size of the instance. Otherwise, you can try local SSD instances but there are only a few and they are incredibly expensive.
kinghajj 10y ago
I'm working on an approach using ZFS on K8s. The idea is to write a custom "flexvolume" driver that takes set of EBS volumes an options, attaches them, create a zpool, set options, etc. In tandem, however, I'm adding to each node host a systemd service to create dm-crypt+LVM devices atop of each NVMe drive (as in the i3 instance types). The flexvolume driver will support adding SLOG and/or L2ARC devices by provisioning LVM volumes from those. Something I'm still unclear, is whether EBS does any data checksums, in which case I could disable ZFS' to spare CPU cycles (but keep compression, of course.)
- weitzj 10y ago
  Sounds great. Is this approach somehow related to ClusterHQ's?
- illamint 10y ago
  > I'm adding to each node host a systemd service to create dm-crypt+LVM devices atop of each NVMe drive That sounds cool; can you describe how you're using systemd for this in a bit more detail? I do this with an Ansible playbook when the machines are provisioned.
fdr 10y ago
Citus Cloud uses that approach. I designed it with the arithmetic from the page about gp2, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolume.... A useful inflection is the "minimum size for max throughput" at 214GiB. That same disk would also offers 38 minutes at the maximum throughput at 3000PIOPs. After that, performance will drop to about a fifth: that's the baseline for a 214GiB disk. Another important thing to consider that large contiguous reads and writes on EBS are up to 256KiB per IOP: most bulk transfers take up far fewer IOPS than people normally calculate. For most workloads, Burst Is Good: for a fixed amount of money, you'll be able to obtain meaningfully better responsiveness in a program. Few workloads are pushing the maximum IOPS all the time. If you need more baseline IOPS you can make gp2 disks bigger or add more disk, depending on whether you find bandwidth has been saturated or not.
- jacobscott 10y ago
  Are Citus Cloud customers generally IOPS constrained rather than throughput constrained? You're also constrained by the EBS-optimized interconnect, right? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimi...
  - jacobscott 10y ago
    Also, have you considered using i3 instances for their superior price per throughput/IOP? Obviously, you can't rely on EBS snapshot based backup/restore workflows on those instances, but maybe the performance gain is compelling enough to make the exercise worthwhile?
    - fdr 10y ago
      I have rejected this on the basis that stopping and starting an instance is too useful for solving problems. Even with a few striped EBS disks, instance issues solvable by re-attaching the disks via stop/start are probably ten to one hundred times more likely to occur than a disk failure. Still, I know some people do things this way, relying on archives and HA for redundancy, and it can be pretty good.

deleted 10y ago

[deleted]

user5994461 10y ago

Use multiple big* SSD volumes in RAID 0 on big* instances. * IOPS and bandwidth are proportional to the size of volumes * instance types have a hard cap on network and IO, proportional to the size of the instance. Otherwise, you can try local SSD instances but there are only a few and they are incredibly expensive.

kinghajj 10y ago

I'm working on an approach using ZFS on K8s. The idea is to write a custom "flexvolume" driver that takes set of EBS volumes an options, attaches them, create a zpool, set options, etc. In tandem, however, I'm adding to each node host a systemd service to create dm-crypt+LVM devices atop of each NVMe drive (as in the i3 instance types). The flexvolume driver will support adding SLOG and/or L2ARC devices by provisioning LVM volumes from those. Something I'm still unclear, is whether EBS does any data checksums, in which case I could disable ZFS' to spare CPU cycles (but keep compression, of course.)

weitzj 10y ago
Sounds great. Is this approach somehow related to ClusterHQ's?
illamint 10y ago
> I'm adding to each node host a systemd service to create dm-crypt+LVM devices atop of each NVMe drive That sounds cool; can you describe how you're using systemd for this in a bit more detail? I do this with an Ansible playbook when the machines are provisioned.

fdr 10y ago

Citus Cloud uses that approach. I designed it with the arithmetic from the page about gp2, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolume.... A useful inflection is the "minimum size for max throughput" at 214GiB. That same disk would also offers 38 minutes at the maximum throughput at 3000PIOPs. After that, performance will drop to about a fifth: that's the baseline for a 214GiB disk. Another important thing to consider that large contiguous reads and writes on EBS are up to 256KiB per IOP: most bulk transfers take up far fewer IOPS than people normally calculate. For most workloads, Burst Is Good: for a fixed amount of money, you'll be able to obtain meaningfully better responsiveness in a program. Few workloads are pushing the maximum IOPS all the time. If you need more baseline IOPS you can make gp2 disks bigger or add more disk, depending on whether you find bandwidth has been saturated or not.

jacobscott 10y ago
Are Citus Cloud customers generally IOPS constrained rather than throughput constrained? You're also constrained by the EBS-optimized interconnect, right? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimi...
- jacobscott 10y ago
  Also, have you considered using i3 instances for their superior price per throughput/IOP? Obviously, you can't rely on EBS snapshot based backup/restore workflows on those instances, but maybe the performance gain is compelling enough to make the exercise worthwhile?
  - fdr 10y ago
    I have rejected this on the basis that stopping and starting an instance is too useful for solving problems. Even with a few striped EBS disks, instance issues solvable by re-attaching the disks via stop/start are probably ten to one hundred times more likely to occur than a disk failure. Still, I know some people do things this way, relying on archives and HA for redundancy, and it can be pretty good.