4 ms·

The problem with heavy swapping on NVMe (or other flash memory) is that it wears out the flash storage very quickly, even for seemingly "reasonable" workloads.

The problem with heavy swapping on NVMe (or other flash memory) is that it wears out the flash storage very quickly, even for seemingly "reasonable" workloads. In a way, the high performance of NVMe can work against you. Definitely something you want to check out via SMART or similar wearout stats.

man8alexd 1y ago
[dead]
p_ing 1y ago
While what you stated is overall not true, who cares with a 97% cost savings vs RAM? Just pop in another NVMe when one fails.
jitl 1y ago
Let’s say we’re spending $1 million on hardware hypothetically with the swap setup. At that price point, either we use swap and let the kernel engineers move data from RAM to disk and back, or we disable swap and need user space code to move the same data to disk and back. We’d need to price out writing & maintaining the user space implementation (mmap perhaps?) for it to be fair price comparison. To avoid SSD wear and tear, we could spend $29 million a year more to put the data in RAM only. Not worth! (We rent EC2 instances from AWS, so SSD wear is baked into the pricing)
inkyoto 1y ago
Not an issue for the commenter – since they have mentioned S3, they are either using AWS EBS or instance attached scratch NVMe's which the vendor (AWS) takes care of. The AWS control plane will detect an ailing SSD backing up the EBS and will proactively evacuate the data before the physical storage goes pear shaped. If it is an EC2 instance with an instance attached NVMe, the control plane will issue an alert that can be automatically acted upon, and the instance can be bounced with a new EC2 instance allocated from a pool of the same instance type and get a new NVMe. Provided, of course, the design and implementation of the running system are stateless and can rebuild the working set upon a restart.
- jitl 1y ago
  EBS is slow. No way we would use it for swap. Gotta be instance storage device. And yes, we can rebuild a node from source data, we do so regularly to release changes anyways.
  - inkyoto 1y ago
    I figured that you were using instance attached NVMe's since you mentioned the scale of your load – an EBS even with the io2 Express storage class can't keep up with a physical NVMe drive on high intensity I/O tasks. Regardless, AWS takes care the hardware cycling / migration in either case.
ciupicri 1y ago
For what it's worth, these are the lifetime estimates for the Micron 7450 SSD [1]: Model Capacity 4K Rand 128K Seq [GB] [TBW] [TBW] PRO 3840 7_300 24_400 PRO 7680 14_000 48_800 MAX 3200 17_500 30_900 MAX 6400 35_000 61_800 > Values represent the theoretical maximum endurance for the given transfer size and type. Actual lifetime will vary by workload … > Total bytes written calculated assuming drive is 100% full (user capacity) with workload of 100% random aligned 4KB writes. [1]: page 6/17, https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b-b36c-4b17-b768-659acc4d4bca/renditions/original/as/7450-nvme-ssd-tech-prod-spec.pdf https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b...
justsomehnguy 1y ago
> that it wears out the flash storage very quickly Only if you use a consumer grade flash with a non-consumer grade usage. For anything with DPWD >= 1 it's not an issue, eg: https://news.ycombinator.com/item?id=45273937 https://news.ycombinator.com/item?id=45273937

man8alexd 1y ago

[dead]

p_ing 1y ago

While what you stated is overall not true, who cares with a 97% cost savings vs RAM? Just pop in another NVMe when one fails.

jitl 1y ago

Let’s say we’re spending $1 million on hardware hypothetically with the swap setup. At that price point, either we use swap and let the kernel engineers move data from RAM to disk and back, or we disable swap and need user space code to move the same data to disk and back. We’d need to price out writing & maintaining the user space implementation (mmap perhaps?) for it to be fair price comparison. To avoid SSD wear and tear, we could spend $29 million a year more to put the data in RAM only. Not worth! (We rent EC2 instances from AWS, so SSD wear is baked into the pricing)

inkyoto 1y ago

Not an issue for the commenter – since they have mentioned S3, they are either using AWS EBS or instance attached scratch NVMe's which the vendor (AWS) takes care of. The AWS control plane will detect an ailing SSD backing up the EBS and will proactively evacuate the data before the physical storage goes pear shaped. If it is an EC2 instance with an instance attached NVMe, the control plane will issue an alert that can be automatically acted upon, and the instance can be bounced with a new EC2 instance allocated from a pool of the same instance type and get a new NVMe. Provided, of course, the design and implementation of the running system are stateless and can rebuild the working set upon a restart.

jitl 1y ago
EBS is slow. No way we would use it for swap. Gotta be instance storage device. And yes, we can rebuild a node from source data, we do so regularly to release changes anyways.
- inkyoto 1y ago
  I figured that you were using instance attached NVMe's since you mentioned the scale of your load – an EBS even with the io2 Express storage class can't keep up with a physical NVMe drive on high intensity I/O tasks. Regardless, AWS takes care the hardware cycling / migration in either case.

ciupicri 1y ago

For what it's worth, these are the lifetime estimates for the Micron 7450 SSD [1]: Model Capacity 4K Rand 128K Seq [GB] [TBW] [TBW] PRO 3840 7_300 24_400 PRO 7680 14_000 48_800 MAX 3200 17_500 30_900 MAX 6400 35_000 61_800 > Values represent the theoretical maximum endurance for the given transfer size and type. Actual lifetime will vary by workload … > Total bytes written calculated assuming drive is 100% full (user capacity) with workload of 100% random aligned 4KB writes. [1]: page 6/17, https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b-b36c-4b17-b768-659acc4d4bca/renditions/original/as/7450-nvme-ssd-tech-prod-spec.pdf https://assets.micron.com/adobe/assets/urn:aaid:aem:d133a40b...

justsomehnguy 1y ago

> that it wears out the flash storage very quickly Only if you use a consumer grade flash with a non-consumer grade usage. For anything with DPWD >= 1 it's not an issue, eg: https://news.ycombinator.com/item?id=45273937 https://news.ycombinator.com/item?id=45273937