6 ms·

Badblocks writes easily compressible patterns and I assume harddrive manufacturers try to compress and deduplicate data, masking any defects. Especially on SSDs

• Dylan16807 7y ago
  SSDs have enough remapping logic and get enough benefit from reduced data size to make it worth it. SMR maybe, I don't know if it's really worth it. For a hard drive dealing with 4KB sectors, that's a lot of trouble for almost no benefit.
• vxNsr 7y ago
  Wouldn't doing this sorta thing to SSDs reduce their life significantly?
  • wtallis 7y ago
    Consumer SSDs are typically rated for 0.1–0.5 drive writes per day, for a warranty period that's usually 5 years but sometimes just 3 years on low-end drives. So a single pass of sequential writes uses up somewhere between 2 and 10 days of warrantied write endurance, which is a more conservative measure than actual expected functional lifespan.
  • Arnt 7y ago
    No. SSD blocks can be written 10k, 100k, maybe a million times. If one block is rewritten every time the most frequently changed directory is changed, that block might be exhaustible. But starting out with writing each block once won't make a difference.
    • wtallis 7y ago
      > SSD blocks can be written 10k, 100k, maybe a million times. It's not 2007 anymore. SSDs also have wear leveling; you cannot wear out a particular block of flash by writing to the same LBA a bunch of times.
    • kabdib 7y ago
      In 1991 or so, Intel was speccing devices with 100K erase cycles. In private conversation we were told "Oh, they'll last millions, it's just that erases take longer and longer to the point where you run out of patience and just declare the block dead." I don't remember the part sizes, but they were in the low megabit range. Challenge accepted, I tried to wear a device out (my setup had direct bus access to the flash part in question, no remapping layer). Went for months without an error and I eventually repurposed that corner of my desk. 30 years later, they've pushed the technology a little :-) You won't get 10K out of an SLC cell (ten years ago those would suffer around 3,000 erase cycles, and it has to be much worse now). I don't know what QLC endurance is (16 voltage levels in a cell, run away), but I wouldn't be surprised if it was a few hundred cycles.
      • wtallis 7y ago
        > You won't get 10K out of an SLC cell (ten years ago those would suffer around 3,000 erase cycles, and it has to be much worse now). 10k P/E cycles is the right ballpark for SLC 3D NAND these days, and it's never been anywhere as low as 3k—that's more like what good TLC 3D NAND gets. QLC is in the 500-1000 cycle range.
        • kabdib 7y ago
          Yup, I had mis-remembered the SLC versus TLC numbers. I'm surprised QLC gets as high as 1000, frankly. It's like the cells are on a first-name basis with the electrons they hold.
          • wtallis 7y ago
            I think it was worse with TLC right before the transition to 3D NAND. They were down to about 8 electrons between voltage states. 3D NAND memory cells are physically much larger than 15nm planar NAND, and most are a lot less leaky.
            • kabdib 7y ago
              EIGHT electrons? Indistinguishable from magic, man. [Cue future archeologists: "Remember when our grandparents could only get a few gigabytes on a quark? How primitive, how sooo last-femtocentury."]
• JensRex 7y ago
  Per recent postings on HN about /dev/urandom, wouldn't it fine to use data from that, rather than bothering with AES output. /dev/urandom should be cryptographically safe.
  • jpdaigle 7y ago
    But how would you detect corruption in this case? If you write random bytes then read them back, you have nothing to compare them to.
    • tzs 7y ago
      Let b be the number of bytes per block. 1. Test N blocks of the disk, where N is small enough that you can keep a 128-bit hash of each block of random test data in memory for the duration of the test. You can use the hashes to verify that read data is correct. 2. Then test another Nb/8 blocks using the N blocks from #1 to store 128-bit hashes of the Nb/8 test blocks. 3. Then test N(1+b/8)/8 blocks, using the blocks from #1 and #2 to hold the hashes. 4. Then test N(1+b/8 + b/8^2)/8 blocks, using the blocks from #1 and #2 and #3 to hold the hashes. ... (Maybe replace the 8's in that with 7's, and in each block of hashes include a hash of the 7 hashes, so you can check that hash blocks are reading back fine?)
      • jeremysalwen 7y ago
        So basically you are demonstrating that the method is more complex than OPs, with absolutely no benefit, which I think was the point.
    • JoachimSchipper 7y ago
      tzs' solution is clever, but if you're planning to simply toss a bad drive (and don't particularly care which blocks are bad), something like tee /dev/the/disk </dev/urandom | sha256sum and sha256sum /dev/the/disk should also work fine.
    • JensRex 7y ago
      You're right, I don't know what I was thinking with that comment.
• bityard 7y ago
  > and I assume harddrive manufacturers try to compress and deduplicate data You assume incorrectly. There have been attempts, like SandForce drives in the early days of SSDs but there are a bunch of reasons why you don't want to do opportunistic general-purpose compression at the lowest possible level of the storage chain. The problem with compression* is that instead of having one drive capacity number (say, 500 GB), you have two: the drive's physical capacity and it's "logical" capacity. Which is fine and dandy, except that the logical capacity varies wildly with what kind of data is stored on the drive. Highly patterned data (text and most executables) compresses very well. Data which is already compressed (most images, videos, archives, etc) does not. So how do you report to the user how much space they have left on the drive? Even if you know the existing contents and current compression ratio, you don't know what they're going to put on the drive in the _future_. Your best guess would be, "250 GB or maybe about a terabyte, lol i dunno". There's also the fact that application-specific compression algorithms tend to do FAR better than general-purpose compression algorithms, which is why almost all of our media storage formats default to using them. JPEG, HEVC, and so on. Plus you get the benefit of having the thing compressed all the time (even over the I/O channel or network) instead of just on disk. Compressing data which is _already_ compressed often results in additional overhead. So unless the drive is testing the data to see how well it compresses before writing it to disk (which would murder performance), your 500 GB drive could actually end up being a 450 GB drive. Further, always-on compression would result in a substantial performance penalty unless special silicon is crafted to handle it. Storage is already an industry with razor-thin margins, companies are not going to add to the BOM cost for a feature that could ultimately make people buy _fewer_ drives. In the case of deduplication, there's no OS-level standard for it which makes current implementations far less efficient than they could be. That said, data compression is very popular in the enterprise storage space, but it is typically done at the pool or volume level (large groups of disks) rather than per-disk. These arrays usually combine compression and deduplication with other strategies like thin provisioning to optimize storage to an almost absurd level. It typically requires trained storage engineers to manage them. * _I lump deduplication into compression because dedeuplication is actually just one kind of compression strategy, even though lots of things treat it like a separate feature._