8 ms·
New hard drive rituals (2018)
- rini17 7y agoBadblocks writes easily compressible patterns and I assume harddrive manufacturers try to compress and deduplicate data, masking any defects. Especially on SSDs and SMR. So I prefer to fill the drive with AES-encrypted data like: yes | openssl enc -aes-256-cbc -out /dev/test-drive and then decrypt and grep. It's almost as fast as badblocks.
- Dylan16807 7y agoSSDs have enough remapping logic and get enough benefit from reduced data size to make it worth it. SMR maybe, I don't know if it's really worth it. For a hard drive dealing with 4KB sectors, that's a lot of trouble for almost no benefit.
- vxNsr 7y agoWouldn't doing this sorta thing to SSDs reduce their life significantly?
- wtallis 7y agoConsumer SSDs are typically rated for 0.1–0.5 drive writes per day, for a warranty period that's usually 5 years but sometimes just 3 years on low-end drives. So a single pass of sequential writes uses up somewhere between 2 and 10 days of warrantied write endurance, which is a more conservative measure than actual expected functional lifespan.
- Arnt 7y agoNo. SSD blocks can be written 10k, 100k, maybe a million times. If one block is rewritten every time the most frequently changed directory is changed, that block might be exhaustible. But starting out with writing each block once won't make a difference.
- wtallis 7y ago> SSD blocks can be written 10k, 100k, maybe a million times. It's not 2007 anymore. SSDs also have wear leveling; you cannot wear out a particular block of flash by writing to the same LBA a bunch of times.
- kabdib 7y agoIn 1991 or so, Intel was speccing devices with 100K erase cycles. In private conversation we were told "Oh, they'll last millions, it's just that erases take longer and longer to the point where you run out of patience and just declare the block dead." I don't remember the part sizes, but they were in the low megabit range. Challenge accepted, I tried to wear a device out (my setup had direct bus access to the flash part in question, no remapping layer). Went for months without an error and I eventually repurposed that corner of my desk. 30 years later, they've pushed the technology a little :-) You won't get 10K out of an SLC cell (ten years ago those would suffer around 3,000 erase cycles, and it has to be much worse now). I don't know what QLC endurance is (16 voltage levels in a cell, run away), but I wouldn't be surprised if it was a few hundred cycles.
- wtallis 7y ago> You won't get 10K out of an SLC cell (ten years ago those would suffer around 3,000 erase cycles, and it has to be much worse now). 10k P/E cycles is the right ballpark for SLC 3D NAND these days, and it's never been anywhere as low as 3k—that's more like what good TLC 3D NAND gets. QLC is in the 500-1000 cycle range.
- kabdib 7y agoYup, I had mis-remembered the SLC versus TLC numbers. I'm surprised QLC gets as high as 1000, frankly. It's like the cells are on a first-name basis with the electrons they hold.
- wtallis 7y agoI think it was worse with TLC right before the transition to 3D NAND. They were down to about 8 electrons between voltage states. 3D NAND memory cells are physically much larger than 15nm planar NAND, and most are a lot less leaky.
- JensRex 7y agoPer recent postings on HN about /dev/urandom, wouldn't it fine to use data from that, rather than bothering with AES output. /dev/urandom should be cryptographically safe.
- jpdaigle 7y agoBut how would you detect corruption in this case? If you write random bytes then read them back, you have nothing to compare them to.
- tzs 7y agoLet b be the number of bytes per block. 1. Test N blocks of the disk, where N is small enough that you can keep a 128-bit hash of each block of random test data in memory for the duration of the test. You can use the hashes to verify that read data is correct. 2. Then test another Nb/8 blocks using the N blocks from #1 to store 128-bit hashes of the Nb/8 test blocks. 3. Then test N(1+b/8)/8 blocks, using the blocks from #1 and #2 to hold the hashes. 4. Then test N(1+b/8 + b/8^2)/8 blocks, using the blocks from #1 and #2 and #3 to hold the hashes. ... (Maybe replace the 8's in that with 7's, and in each block of hashes include a hash of the 7 hashes, so you can check that hash blocks are reading back fine?)
- jeremysalwen 7y agoSo basically you are demonstrating that the method is more complex than OPs, with absolutely no benefit, which I think was the point.
- JoachimSchipper 7y agotzs' solution is clever, but if you're planning to simply toss a bad drive (and don't particularly care which blocks are bad), something like tee /dev/the/disk </dev/urandom | sha256sum and sha256sum /dev/the/disk should also work fine.
- JensRex 7y agoYou're right, I don't know what I was thinking with that comment.
- bityard 7y ago> and I assume harddrive manufacturers try to compress and deduplicate data You assume incorrectly. There have been attempts, like SandForce drives in the early days of SSDs but there are a bunch of reasons why you don't want to do opportunistic general-purpose compression at the lowest possible level of the storage chain. The problem with compression* is that instead of having one drive capacity number (say, 500 GB), you have two: the drive's physical capacity and it's "logical" capacity. Which is fine and dandy, except that the logical capacity varies wildly with what kind of data is stored on the drive. Highly patterned data (text and most executables) compresses very well. Data which is already compressed (most images, videos, archives, etc) does not. So how do you report to the user how much space they have left on the drive? Even if you know the existing contents and current compression ratio, you don't know what they're going to put on the drive in the _future_. Your best guess would be, "250 GB or maybe about a terabyte, lol i dunno". There's also the fact that application-specific compression algorithms tend to do FAR better than general-purpose compression algorithms, which is why almost all of our media storage formats default to using them. JPEG, HEVC, and so on. Plus you get the benefit of having the thing compressed all the time (even over the I/O channel or network) instead of just on disk. Compressing data which is _already_ compressed often results in additional overhead. So unless the drive is testing the data to see how well it compresses before writing it to disk (which would murder performance), your 500 GB drive could actually end up being a 450 GB drive. Further, always-on compression would result in a substantial performance penalty unless special silicon is crafted to handle it. Storage is already an industry with razor-thin margins, companies are not going to add to the BOM cost for a feature that could ultimately make people buy _fewer_ drives. In the case of deduplication, there's no OS-level standard for it which makes current implementations far less efficient than they could be. That said, data compression is very popular in the enterprise storage space, but it is typically done at the pool or volume level (large groups of disks) rather than per-disk. These arrays usually combine compression and deduplication with other strategies like thin provisioning to optimize storage to an almost absurd level. It typically requires trained storage engineers to manage them. * _I lump deduplication into compression because dedeuplication is actually just one kind of compression strategy, even though lots of things treat it like a separate feature._
- numpad0 7y agoWould it work with SMR drives?
- cmurf 7y agoFor badblocks, I suggest a block size larger than 4KiB, it'll go faster. The legacy reason for 4KiB: badblocks can produce a badblocks file that mke2fs can use to avoid using known bad sectors, however the block size for badblocks and mkfs must be the same. Since bad sectors are managed by the firmware these days, you're best off not using this legacy feature, therefore you can up the badblocks block size well above 4KiB. Another idea for a generic approach, whether HDD or SSD, is to use f3. It'll check for fake flash (firmware loops to report a bigger device size than real size), corruptions, and read/write errors.
- spartas 7y agohttps://unix.stackexchange.com/questions/202541/how-can-i-choose-optimal-values-for-block-size-and-blocks-number-for-badblocks https://unix.stackexchange.com/questions/202541/how-can-i-ch...
- jolmg 7y agoOn the topic of new rituals for storage devices, whenever I buy a microSD card or a USB flash drive and before I setup the filesystem, I've been filling it with random data and reading it back to ensure it does have the storage capacity it's supposed to have. I started doing this after reading comments online (probably here) on fraudulent cards and drives circulating around Amazon. Some comments mentioned that there are even some devices that have their firmware set-up such that writes don't fail even when you've written more than what the device can actually hold. They supposedly manage this by reusing previously used blocks for new files. I suppose in other words, each block has multiple addresses. The addresses would probably loop around the same blocks until the fake size is reached. That would give it the most realistic appearance that it's of the reported size while silently corrupting previously written storage. In any case, so far I've only gotten 4 cards/drives since then, and I haven't found them to be frauds. I've since wondered how much I'm subtracting from their product lifespan with my testing. I mean, I imagine that microSD cards and USB flash drives do wear leveling on unused blocks, but without TRIM support on them, after testing them, I'm probably eliminating the devices' ability to do wear leveling since I'm occupying all blocks without the ability to tell the device that they're unoccupied now. As far as I understand, the problem with lacking TRIM support on USB flash drives is not that there's no protocol for it through USB, since there are USB-SATA adapters that specify support for UASP. Admittedly, though, I'm assuming that UASP is enough to get TRIM support. I don't remember testing that. In any case, I wonder if this new ritual I'm doing is really worth it, and if it's not, when would it be? EDIT: I wonder if it's realistic to expect that one day we'll get USB flash drives and microSD cards with TRIM support. I don't expect the need to check for fraudulent storage devices to go away.
- yjftsjthsd-h 7y ago> In any case, I wonder if this new ritual I'm doing is really worth it, and if it's not, when would it be? Depends on the chances and impact of getting a fake/faulty device, yeah? I've heard claims that most online marketplaces have rampant fakes, although you're 4 for 4 with good ones. On the other hand, the failure mode is data loss, which seems pretty bad. So personally I think your approach is a good idea.
- 7y ago
- RobertRoberts 7y ago[Wrote a comment on the wrong thread... sorry]
- yjftsjthsd-h 7y agoAre you on the right thread?
- RobertRoberts 7y agohaha, nope. Ty
- deleted 7y ago[deleted]
- nickcw 7y agoHere is a little program I wrote to stress test disks. It was originally written for HDD but it works well on SSD and Flash too. I test all my media with it for 24 hours. In fact I'm testing two 5TB external Seagate drives as I type! https://github.com/ncw/stressdisk https://github.com/ncw/stressdisk
- 3JPLW 7y agoI'm curious about the efficacy here; how many "bad" drives will this find?
- mceachen 7y agoI've found several in the past couple years (I test them more rigorously before I put them into a NAS).
- mceachen 7y agoThis article has an unfortunate omission by ignoring drives' S.M.A.R.T. reporting system which can give a surprisingly thorough view into drive metadata, like how many blocks have been remapped due to media corruption. After running `badblocks`, run `smartctl -t long $DEVICE`, which is a "long" self-test. This will take upwards of a day to complete, depending on the drive. After that completes, check the status by running `smartctl --all $DEVICE`, and verify that there are only zeroes in the RAW_VALUE column for Reallocated_Sector_Ct, Used_Rsvd_Blk_Cnt_Tot, and Uncorrectable_Error_Cnt.
- rsync 7y agoIt's also very illuminating to check SMART stats on "new" drives that show hundreds of on/off cycles or other evidence that the drives are not, in fact, new. Pro tip: Many, many drives sold on Amazon by large, "reputable" suppliers are not new in the way you think they are - they are "new pulls" from systems integrators that received them as part of the complete PC and then pulled them, en masse, to sell in bulk. Those "new pulls" may have hundreds and hundreds of hours on them ... We (rsync.net) have started trending towards B&H Photo for larger batches of drives ... they pack them cluefully and there is no "new pull" monkey-business ... unfortunately they don't always have the quantity on-hand that we require, but that's not a knock against them ...
- oh_sigh 7y agoWhy not just buy direct from the manufacturer?
- Spooky23 7y agoYou need really high quantities. At moderate quantities, Intel and SSD makers do backend deals to get rebate credits to a PC OEM. When I bought lots of computers, we would usually get -1-3% margins.
- tomc1985 7y agoWow, so I guess people are buying new PCs and essentially scrapping them for parts? That's....... actually kind of funny
- rkagerer 7y agoBack around 2010 I got a great deal on eight PACER SSD's from China. At the time, SSD's were still quite expensive compared to spinning platters. My performance benchmarks were fantastic, but I started seeing some corruption issues. I wrote a C# program to fill the entire disk up with pseudorandom bytes (starting with a known seed), then read it back and compare. Nearly all of them gave back different bytes at some point in the test. These were silent errors, and didn't trigger any read warnings or CRC/ECC failures reported by the drive's controller. I suspect they achieved the performance by simply ignoring any errors and steamrolling right along.
- jojo2000 7y agoBad blocks bad blocks what ya gonna do, what ya gonna do when they come for you. Hard drives fail hard or smooth, SSDs fail hard ! On a hard drive, we can picture how the blocks are stored (sequentially). On an SSD, there is a remapping of blocks in silicon+firmware, and this added layer adds its own bugs [0] [1]. [0] https://blog.elcomsoft.com/2019/01/why-ssds-die-a-sudden-death-and-how-to-deal-with-it/ https://blog.elcomsoft.com/2019/01/why-ssds-die-a-sudden-dea... [1] https://www.anandtech.com/show/6503/second-update-on-samsung-ssd-840840-pro-failures https://www.anandtech.com/show/6503/second-update-on-samsung... (edit : added one reference more)
- proactivesvcs 7y agoFor hard disk drives I leave the drive aside for 24 hours, before powering it up, if it's likely to have been in below-zero temperatures before it got to me. I run a short SMART test and check the results, as well as attributes, in case it's faulty already. Then do similar to the OP, afterwards running a short, conveyance, and long test.
- dijit 7y ago> For hard disk drives I leave the drive aside for 24 hours, before powering it up, if it's likely to have been in below-zero temperatures before it got to me. Regarding temperatures: it used to be somewhat common practice to freeze drives that were clicking. Funny to hear someone waiting to them to thaw, when I have intentionally frozen a bunch of drives before.
- proactivesvcs 7y agoThe way I figure: they're fragile as it is, so if I can help make their life easier, I do :-)
- maccam94 7y agoI recommend using Flexible I/O tester (fio) to run burn-in new disks: https://fio.readthedocs.io/en/latest/fio_doc.html#running-fio https://fio.readthedocs.io/en/latest/fio_doc.html#running-fi... It will give you both performance statistics and catch checksum verification errors.
- zepearl 7y agoI'm thinking since a while about replacing mdadm's raid5 with ZFS' RAIDZ in my two NAS: if I understood correctly, in the case of single/few block failures ZFS would transparently remap the bad block(s) present on the bad HDD to a good block on the same HDD (therefore without having to immediately replace the partially faulty HDD and rebuild the whole raid) => is this correct? I admit that continuing to use the partially faulty HDD might be a bad idea, but at least having a raid 100% healthy before replacing the partially faulty HDD gives me a good feeling... .
- userbinator 7y agoMy preferred software for scanning a newly purchased disk is MHDD: https://hddguru.com/software/2005.10.02-MHDD/ https://hddguru.com/software/2005.10.02-MHDD/ Because it times how long it takes to read/write each sector, it has the advantage of being able to detect "weak" sectors before they even become bad ones --- HDDs will retry accesses multiple times if an error occurs before giving up, and to a program that simply reads and writes data, weak sectors that successfully read after a small number of retries aren't noticeable. They will show up in MHDD, as taking longer than usual to read.
- HenryBemis 7y agoOn every new hard drive I always run SpinRite when I hook it up on my machine(s). I know that for the repair mode it is sloooooooooow, but Steve Gibson is working on a newer version that would significantly improve speed. (not affiliated, just save my butt a couple of times).
- thanksforfish 7y agoI'm curious about what prevents the manufacturers from running badblocks and eliminating some of the downstream pain. Obviously some users care and will spend the effort to check their drives, and some unlucky users will have a data loss and a bad experience. Is not running badblocks before shipping a way to exploit users who won't RMA or won't realize their drive is busted? Or are drives getting damaged in transit? Or something else?
- grizzles 7y ago25 in the last 10 years. Must be nice. I buy that many each year. I often buy new drives that can't detect their own errors that ZFS finds. Because a faulty device by definition can't test itself. SMART doesn't work anymore if it ever did. Manufacturer firmware blobs have a huge incentive to report a successful test no matter what. Also as someone else mentioned, there are plenty of used drives being resold as new in the market. This industry has skated for way too long in my opinion. They benefit from a big reluctance on the part of people & companies to return faulty drives to protect sensitive company data. I have seriously considered starting an ecommerce shop that sells only hard drives. But also does backblaze like tracking / reporting of individual drives, has a good return policy and degausses returned drives. Speaking as a consumer I have zero interest in the current system, I want a system that lets me take the TCO of a drive/drive class into account before I make a purchasing decision.
- fred_is_fred 7y agoThere are at least two comments in here about special processes people use to validate Amazon purchases. This is because you cannot trust that what you are buying is new and because you cannot trust that it is the advertised size. If this is true, why not just buy computer equipment somewhere else?
- thatoneguy 7y agoRitual I've had since at least 1994: naming a new hard drive volume under MS-DOS/Windows "PLEASE_WORK"