7 ms·

Right. So, when swapping pages back into RAM, a typical OS will not actually remove the pages from swap despite them now being in RAM as well? So if the memory

by Delk 3y ago

Right. So, when swapping pages back into RAM, a typical OS will not actually remove the pages from swap despite them now being in RAM as well? So if the memory needs to be freed again for another purpose before the pages in memory have been modified in the interim, the OS can just drop the pages from RAM without having to write them into swap again? And thus thrashing causes more swap writes when there's limited swap space. That's the only scenario I can think of. I didn't think about pages not being removed from swap when swapping in. > I've seen a server with 850/860 Evos Well, that's desktop-grade hardware in a server. Might work, but if it doesn't, you get what you asked for. > and LUKS (so the worst case: cheap drives and all writes are new) How does LUKS affect that? > people are forgetting what 3MB/s * 3600 * 24 is whooping 259GB a day, and you need to write ~259GB a day for 3 years to totally deplete the [official] write endurance (YssdMV). Right, this was my point. That's not an impossible amount of writes but it is a lot more than almost anybody does on a laptop/desktop.

justsomehnguy 3y ago
> OS will not actually remove the pages from swap despite them now being in RAM as well Now the hard part: I don't have an idea, because I'm no that versed in OS VMM, but it does make sense even if it is 1993. But this is the basics and I doubt it works some other way. I would be happy if someone more familiar would chime in (and the reason why or not), but in my experience this is what happening. > Well, that's desktop-grade hardware in a server. Might work, but if it doesn't, you get what you asked for. Extremely often and works not just 'quite well', but well enough. Similiar servers (literally, just without LUKS) were fine, with > 90% of 'SSD health' > How does LUKS affect that Every write is different, because the bytes on the storage are already encrypted. Ie you write the same bytes to the same blovk pn the FS, but the underlaying, encrypted, bytes are not the same => new write. > Right, this was my point. Yep.
- Delk 3y ago
  If the OS didn't keep the pages around in swap after they were swapped back into RAM, I don't see how having more space in swap could reduce writes to it. > Every write is different, because the bytes on the storage are already encrypted. Ie you write the same bytes to the same blovk pn the FS, but the underlaying, encrypted, bytes are not the same => new write. How are they not the same as the previous encrypted version of the same bytes if the encryption key stays the same? Are you sure TRIM/discard just wasn't enabled on the LUKS?
  - justsomehnguy 3y ago
    > If the OS didn't keep the pages around in swap after they were swapped back into RAM Well, why it shouldn't? Don't forget, while the RAM can be prepared (zeroed) fast, the on-disk swap can't be prepared that fast (compared to RAM ofc), and you already under some memory and disk constraints (that's how you got to be using the swap in the first place) so adding another workload for cleaning up the swap is... not a good thing. If you only would do 'zero on allocate' then sooner or later you would be in a position where you would need to stall the whole system until enough pages in the swap are available. And nor the users nor the programs like that. > I don't see how having more space in swap could reduce writes to it. Swap is just 'slow memory' part of your overall virtual memory allocation of the whole OS. If you have a small swap then you would be pressured to evict the data from it more often. If you have a big swap then less pressure => less evictions => less overwriting the same LBAs assigned to the swap file/partition => less writes overall. > How are they not the same as the previous encrypted version of the same bytes if the encryption key stays the same? Even if you write 00000000 to the filesystem block (a typical situation would be deleting a file and on the flash/SMR drives you would just call TRIM on those bytes^W LBAs) it's not 00000000 down there, it's some ciphertext. > Are you sure TRIM/discard just wasn't enabled on the LUKS? See above, no such thing as TRIM on an encrypted storage device.
    - Delk 3y ago
      > Well, why it shouldn't? Yeah, maybe it should. I don't know how exactly that's typically implemented, hence the question mark. > Don't forget, while the RAM can be prepared (zeroed) fast, the on-disk swap can't be prepared that fast (compared to RAM ofc), and you already under some memory and disk constraints (that's how you got to be using the swap in the first place) so adding another workload for cleaning up the swap is... not a good thing. I doubt any of that would typically involve the OS specifically zeroing anything apart from just marking the corresponding parts of the swap space as free. And that's probably in the in-memory data structures keeping track of pages in swap. No need to zero any of the page contents, just mark those areas of swap as unallocated. An OS might want to keep the pages around for the sake of having thrm still around but unless otherwise shown, I kind of doubt there's a great cost to the deallocation itself. > If you have a small swap then you would be pressured to evict the data from it more often. I get that if indeed the OS keeps pages available (and in the books) in swap after swapping them back into RAM, having a larger swap can reduce writes in case those same pages end up getting swapped back out again before they've been modified. It may be that's how it works. I'm not sure I follow the logic if that's not the case. I don't understand why the OS would "evict" pages from swap just because swap is getting full -- there's nowhere to evict them to except RAM. If you've got thing A currently in swap and you're needing to "evict" it to make space in the swap for thing B, that means you're wanting to get rid of B in RAM. That means you're already needing more space in RAM for some third thing C, so why would you solve that by swapping pages of thing A from swap into RAM? My understanding is that swapping pages back in would be initiated by needing those pages back in RAM, so that's not really a case of eviction. Anyway, since it doesn't seem like either of us knows for an actual fact how it works in any particular OS, I doubt speculation will lead to anything better. I believe what you've seen as a phenomenon, I'm just not sure the explanation of that phenomenon makes quite enough sense to me. > See above, no such thing as TRIM on an encrypted storage device. TRIM may have security implications for encrypted drives, e.g. in terms of plausible deniability for forensics, that's true, and I didn't think of that. Though I'm not sure if that's what you meant.

by Delk 3y ago

justsomehnguy 3y ago

> OS will not actually remove the pages from swap despite them now being in RAM as well Now the hard part: I don't have an idea, because I'm no that versed in OS VMM, but it does make sense even if it is 1993. But this is the basics and I doubt it works some other way. I would be happy if someone more familiar would chime in (and the reason why or not), but in my experience this is what happening. > Well, that's desktop-grade hardware in a server. Might work, but if it doesn't, you get what you asked for. Extremely often and works not just 'quite well', but well enough. Similiar servers (literally, just without LUKS) were fine, with > 90% of 'SSD health' > How does LUKS affect that Every write is different, because the bytes on the storage are already encrypted. Ie you write the same bytes to the same blovk pn the FS, but the underlaying, encrypted, bytes are not the same => new write. > Right, this was my point. Yep.

Delk 3y ago
If the OS didn't keep the pages around in swap after they were swapped back into RAM, I don't see how having more space in swap could reduce writes to it. > Every write is different, because the bytes on the storage are already encrypted. Ie you write the same bytes to the same blovk pn the FS, but the underlaying, encrypted, bytes are not the same => new write. How are they not the same as the previous encrypted version of the same bytes if the encryption key stays the same? Are you sure TRIM/discard just wasn't enabled on the LUKS?
- justsomehnguy 3y ago
  > If the OS didn't keep the pages around in swap after they were swapped back into RAM Well, why it shouldn't? Don't forget, while the RAM can be prepared (zeroed) fast, the on-disk swap can't be prepared that fast (compared to RAM ofc), and you already under some memory and disk constraints (that's how you got to be using the swap in the first place) so adding another workload for cleaning up the swap is... not a good thing. If you only would do 'zero on allocate' then sooner or later you would be in a position where you would need to stall the whole system until enough pages in the swap are available. And nor the users nor the programs like that. > I don't see how having more space in swap could reduce writes to it. Swap is just 'slow memory' part of your overall virtual memory allocation of the whole OS. If you have a small swap then you would be pressured to evict the data from it more often. If you have a big swap then less pressure => less evictions => less overwriting the same LBAs assigned to the swap file/partition => less writes overall. > How are they not the same as the previous encrypted version of the same bytes if the encryption key stays the same? Even if you write 00000000 to the filesystem block (a typical situation would be deleting a file and on the flash/SMR drives you would just call TRIM on those bytes^W LBAs) it's not 00000000 down there, it's some ciphertext. > Are you sure TRIM/discard just wasn't enabled on the LUKS? See above, no such thing as TRIM on an encrypted storage device.
  - Delk 3y ago
    > Well, why it shouldn't? Yeah, maybe it should. I don't know how exactly that's typically implemented, hence the question mark. > Don't forget, while the RAM can be prepared (zeroed) fast, the on-disk swap can't be prepared that fast (compared to RAM ofc), and you already under some memory and disk constraints (that's how you got to be using the swap in the first place) so adding another workload for cleaning up the swap is... not a good thing. I doubt any of that would typically involve the OS specifically zeroing anything apart from just marking the corresponding parts of the swap space as free. And that's probably in the in-memory data structures keeping track of pages in swap. No need to zero any of the page contents, just mark those areas of swap as unallocated. An OS might want to keep the pages around for the sake of having thrm still around but unless otherwise shown, I kind of doubt there's a great cost to the deallocation itself. > If you have a small swap then you would be pressured to evict the data from it more often. I get that if indeed the OS keeps pages available (and in the books) in swap after swapping them back into RAM, having a larger swap can reduce writes in case those same pages end up getting swapped back out again before they've been modified. It may be that's how it works. I'm not sure I follow the logic if that's not the case. I don't understand why the OS would "evict" pages from swap just because swap is getting full -- there's nowhere to evict them to except RAM. If you've got thing A currently in swap and you're needing to "evict" it to make space in the swap for thing B, that means you're wanting to get rid of B in RAM. That means you're already needing more space in RAM for some third thing C, so why would you solve that by swapping pages of thing A from swap into RAM? My understanding is that swapping pages back in would be initiated by needing those pages back in RAM, so that's not really a case of eviction. Anyway, since it doesn't seem like either of us knows for an actual fact how it works in any particular OS, I doubt speculation will lead to anything better. I believe what you've seen as a phenomenon, I'm just not sure the explanation of that phenomenon makes quite enough sense to me. > See above, no such thing as TRIM on an encrypted storage device. TRIM may have security implications for encrypted drives, e.g. in terms of plausible deniability for forensics, that's true, and I didn't think of that. Though I'm not sure if that's what you meant.