3 ms·

The package is available in Alpine so I tried it out. It seems to work best if the terminal is using en_US.UTF-8. My default is C but I have a function to set

The package is available in Alpine so I tried it out. It seems to work best if the terminal is using en_US.UTF-8. My default is C but I have a function to set UTF-8. I was curious if it could handle stdin and it does. It saves the output in ~/.config/lnav/ something to probably add to my clean up script. It is convenient that it can prepend timestamps to logs. I could see this being nice for people that do QA on build logs all day or that have to debug access logs often. If used with sensitive data one should probably add a shell function and log rotate scripts to clean up ~/.config/lnav/ using the shred command.

c0wb0yc0d3r 4y ago
I was under the impression that the shred command was pretty hard on SSDs. Did that change?
- LinuxBender 4y ago
  It would be as hard as writing the file {n} number of times again. So if one created a 1MB log file and then did a 3 pass wipe that would write 3MB. Number of passes would really just depend on the system being worked on. If PCI I would expect 7 passes and I am sure that would just be factored into the cost of the servers for that environment. Shred also isn't perfect as it has no concept of the file systems journal and does not clean that data but I think it is still good practice for sensitive data, in addition to filesystem or file level encryption on systems with highly sensitive data. With sensitive data one must look at the value of the data vs the value of the SSD. If I go out of my way to extend the life of my SSD, do I risk losing $5billion? That is how I factor in whether or not extra wear-and-tear on a SSD makes sense.
  - electroly 4y ago
    Shred doesn't work at all on SSDs because wear leveling will spray the writes all over the drive, rather than overwriting the blocks you intended to. Your random bytes will end up in new blocks, not overwriting the original blocks. Trying to shred individual files is a totally pointless exercise on SSDs. It isn't doing what you want it to do.
    - LinuxBender 4y ago
      I've also seen people say that and to instead use "blkdiscard -sz" but that also does not write to specific blocks. I still can not find a definitive source saying that shred is not able to write to specific blocks on a SSD and it is not clear to me that wear leveling actually prevents shred from writing to specific blocks. But I agree that I have seen people say this repeatedly on StackExchange, ServerFault and Reddit. Either way one should also use filesystem and file level encryption for sensitive files and encrypt sensitive attributes inside databases. Swap must also be encrypted as it may contain sensitive data. Datacenter bad-drive mishaps do happen as in not shredding the physical drive as a few government and military agencies have recently been embarrassed by. I do agree with pinkorchid that ultimately drives should be destroyed physically.
      - electroly 4y ago
        Wear leveling prevents anyone from writing to specific blocks. It decouples the OS's block numbering from the physical blocks. That indirection allows it to distribute writes evenly across the drive even if the OS is writing to consecutive blocks (or the same block). This behavior is fundamental; it's what wear leveling is and why it exists. Wear leveling exists to prevent you from writing to the same block repeatedly, which is exactly what you're trying to do with shred. I'm not sure what other source you're looking for, but I hope you find it. Perhaps, at least, the number of people "on StackExchange, ServerFault and Reddit" and now also HackerNews telling you the same thing is sufficient evidence to stop spreading the idea that shred might work on SSDs. Do what you want on your own systems, of course, but this is dangerously misleading guidance to be offering on an open forum.
        sgc 4y ago
        Does that mean that partitions on an SSD therefore do not affect the long term wear on the SSD?
        LinuxBender 4y ago
        So my concern with following this logic is that such tools that are writing to specific block locations would then be wiping unrelated inodes depending on the low level logic. I tried emailing Colin but his last known email is invalid now. He may be working for the NSA so I probably won't be able to reach him. Some people there are responsive, some are not and a few are on here. So even if wear leveling prevents overwriting a file then such tools should in theory be a risk of data corruption. If this is the case then the tool should be updated to detect if the target is an SSD and abort with a scary message. Perhaps another route is to reach out to the coreutils team.
pinkorchid 4y ago
This will depend on the specific threat model for the sensitive data you are dealing with, but generally files in an SSD can't be destroyed securely (and shred specifically does not help due to wear leveling). Instead, you may want to encrypt your disk so that as soon as the key is gone it all becomes unreadable. For a bigger threat, maybe you need to follow the NIST destroy guidelines [1] to "Disintegrate, Pulverize, Melt, and Incinerate" the media. 1: https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-88r1.pdf https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.s...
tstack 4y ago
The data consumed from stdin is stored in a file to support scrollback. > something to probably add to my clean up script. These capture files are deleted if they are older than a day the next time lnav is run. (I realize they should've been stored in ~/.local/state instead of ~/.config ...)

c0wb0yc0d3r 4y ago

I was under the impression that the shred command was pretty hard on SSDs. Did that change?

LinuxBender 4y ago
It would be as hard as writing the file {n} number of times again. So if one created a 1MB log file and then did a 3 pass wipe that would write 3MB. Number of passes would really just depend on the system being worked on. If PCI I would expect 7 passes and I am sure that would just be factored into the cost of the servers for that environment. Shred also isn't perfect as it has no concept of the file systems journal and does not clean that data but I think it is still good practice for sensitive data, in addition to filesystem or file level encryption on systems with highly sensitive data. With sensitive data one must look at the value of the data vs the value of the SSD. If I go out of my way to extend the life of my SSD, do I risk losing $5billion? That is how I factor in whether or not extra wear-and-tear on a SSD makes sense.
- electroly 4y ago
  Shred doesn't work at all on SSDs because wear leveling will spray the writes all over the drive, rather than overwriting the blocks you intended to. Your random bytes will end up in new blocks, not overwriting the original blocks. Trying to shred individual files is a totally pointless exercise on SSDs. It isn't doing what you want it to do.
  - LinuxBender 4y ago
    I've also seen people say that and to instead use "blkdiscard -sz" but that also does not write to specific blocks. I still can not find a definitive source saying that shred is not able to write to specific blocks on a SSD and it is not clear to me that wear leveling actually prevents shred from writing to specific blocks. But I agree that I have seen people say this repeatedly on StackExchange, ServerFault and Reddit. Either way one should also use filesystem and file level encryption for sensitive files and encrypt sensitive attributes inside databases. Swap must also be encrypted as it may contain sensitive data. Datacenter bad-drive mishaps do happen as in not shredding the physical drive as a few government and military agencies have recently been embarrassed by. I do agree with pinkorchid that ultimately drives should be destroyed physically.
    - electroly 4y ago
      Wear leveling prevents anyone from writing to specific blocks. It decouples the OS's block numbering from the physical blocks. That indirection allows it to distribute writes evenly across the drive even if the OS is writing to consecutive blocks (or the same block). This behavior is fundamental; it's what wear leveling is and why it exists. Wear leveling exists to prevent you from writing to the same block repeatedly, which is exactly what you're trying to do with shred. I'm not sure what other source you're looking for, but I hope you find it. Perhaps, at least, the number of people "on StackExchange, ServerFault and Reddit" and now also HackerNews telling you the same thing is sufficient evidence to stop spreading the idea that shred might work on SSDs. Do what you want on your own systems, of course, but this is dangerously misleading guidance to be offering on an open forum.
      - sgc 4y ago
        Does that mean that partitions on an SSD therefore do not affect the long term wear on the SSD?
      - LinuxBender 4y ago
        So my concern with following this logic is that such tools that are writing to specific block locations would then be wiping unrelated inodes depending on the low level logic. I tried emailing Colin but his last known email is invalid now. He may be working for the NSA so I probably won't be able to reach him. Some people there are responsive, some are not and a few are on here. So even if wear leveling prevents overwriting a file then such tools should in theory be a risk of data corruption. If this is the case then the tool should be updated to detect if the target is an SSD and abort with a scary message. Perhaps another route is to reach out to the coreutils team.

pinkorchid 4y ago

This will depend on the specific threat model for the sensitive data you are dealing with, but generally files in an SSD can't be destroyed securely (and shred specifically does not help due to wear leveling). Instead, you may want to encrypt your disk so that as soon as the key is gone it all becomes unreadable. For a bigger threat, maybe you need to follow the NIST destroy guidelines [1] to "Disintegrate, Pulverize, Melt, and Incinerate" the media. 1: https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-88r1.pdf https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.s...

tstack 4y ago

The data consumed from stdin is stored in a file to support scrollback. > something to probably add to my clean up script. These capture files are deleted if they are older than a day the next time lnav is run. (I realize they should've been stored in ~/.local/state instead of ~/.config ...)