15 ms·
I tested four NVMe SSDs from four vendors – half lose FLUSH'd data on power loss (2022)
- AlbertoGP 3y agoWell, yes, but which were those 2 out of 4 vendors?
- whitepoplar 3y agoSK Hynix and Sabrent: https://x.com/xenadu02/status/1496290369184874497?s=20 https://x.com/xenadu02/status/1496290369184874497?s=20
- mjevans 3y agoDid what, for those of us without Twitter accounts? Did they Pass or Fail the flush test? (Saved or Lost data respectively?)
- slabity 3y ago> Update 2: models that lost writes: > SK Hynix Gold P31 2TB SHGP31-2000GM-2, FW 31060C20 > Sabrent Rocket 512 (Phison PH-SBT-RKT-303 controller, no version or date codes listed) Looks like those two models failed. Would like to note that these tweets are from Feb 22, 2022. This entire thread should have (2022) on it.
- alright2565 3y agoThere were a few more listed deeper in the thread. Fyi, the nitter version is actually readable: https://nitter.net/xenadu02/status/1495693475584557056#m https://nitter.net/xenadu02/status/1495693475584557056#m Samsung 970 Evo Plus: MZ-V7S2T0, 2021.10: Pass WD Red: WDS100T1R0C-68BDK0, 04Sept2021: Pass Crucial P2 250GB CT250P2SSD8, FW P2CR046: Pass Samsung 980 250GB MZ-V8V250, 2021/11/07: Pass WD Black SN750 1TB WDS100T1B0E, 09Jan2022: Pass WD Green SN350 240GB WDS240G20C, 02Aug2021: Pass SK Hynix Gold P31 2TB SHGP31-2000GM-2, FW 31060C20: Fail Sabrent Rocket 512 (Phison PH-SBT-RKT-303 controller, no version or date codes listed): Fail
- dboreham 3y agoSo...vendors no sane person would store valuable data on their drives were the only ones that failed. The sound of a dog biting a man...
- Espressosaurus 3y agoFWIW, I've had SK Hynix parts in my Dell laptops before. I agree about Sabrent.
- electroglyph 3y agoSK Hynix is the #3 flash manufacturer (acquired Intel's NAND biz), and their RAM is quite decent. i wouldn't think twice about buying if it wasn't for this report.
- Shekelphile 3y agoHynix's flash is just fine. It's the controller at fault for this.
- yread 3y agoIn one of the tweets he says another drive with the same controller passed so it's not just the controller
- lights0123 3y agoI would expect the world's second largest DRAM manufacturer to have trustworthy memory products.
- wannacboatmovie 3y agoApple has been using SK Hynix products (some rebranded as their own) for years.
- Shekelphile 3y agoPretty much every brand uses phison controllers in at least some of their products, even wd/samsung/intel who design controllers in house use them for their cheapest offerings because phison is all about making the cheapest product possible.
- java-man 3y agoName the offenders please. I am sure it might be easy to see visually - a lack of substantial capacitor on the board would indicate a high likelihood of data loss.
- whitepoplar 3y agoSK Hynix and Sabrent: https://x.com/xenadu02/status/1496290369184874497?s=20 https://x.com/xenadu02/status/1496290369184874497?s=20
- java-man 3y agoWhich did "pass" the test?
- whitepoplar 3y agohttps://x.com/xenadu02/status/1496765378592083968?s=20 https://x.com/xenadu02/status/1496765378592083968?s=20 https://x.com/xenadu02/status/1496770278612819968?s=20 https://x.com/xenadu02/status/1496770278612819968?s=20 https://x.com/xenadu02/status/1495875479475298324?s=20 https://x.com/xenadu02/status/1495875479475298324?s=20
- java-man 3y agoありがとう
- potatopatch 3y agoI'd be curious how well it actually correlates. It would be hard to make the most performant system that's always consistent with flushed data but there are probably a lot of firmwares out there with untested performance ideas, etc.
- wmf 3y agoNone of the tested drives use power loss protection capacitors.
- lxgr 3y agoI guess it's time for `fsync_but_really_actually_sync_it_please(2)` (and the lower level equivalents in SATA, NVMe etc.)?
- wmf 3y agoBuggy firmware could also screw that up.
- throw0101b 3y ago> (and the lower level equivalents in SATA, NVMe etc.)? This is not a technical problem that needs yet another SATA/SAS/etc command to be standardized. It's a 'social' problem that there's no real incentives for firmware writers to tell the truth 100% of the time. The best you can hope for is if you buy a fancy-pants enterprise storage solution with compatibility lists and approved firmware versions.
- lxgr 3y agoWell said. Not even OS vendors are immune to the temptations of higher performance through somewhat relaxed interpretation of interfaces: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html https://developer.apple.com/library/archive/documentation/Sy...
- loloquwowndueo 3y agoTwitter yuk, can somebody just post the names of the four tested drives and which passed/failed please?
- Dalewyn 3y ago[flagged]
- RankingMember 3y agoimo hackernews should just automatically replace twitter.com with nitter.net even if just for readability without logging in's sake: https://nitter.net/xenadu02/status/1495693475584557056 https://nitter.net/xenadu02/status/1495693475584557056
- neoecos 3y agoThe Harmonic HN reader for Android does this
- arbitrandomuser 3y agoUhm nope , atleast not the one on playstore , is it there somewhere in the settings ?
- arbitrandomuser 3y agoOooh yes , found it in the settings
- usr1106 3y agoWe should just learn to ignore people still posting on Twitter (which is called something else now).
- saagarjha 3y agoThis is from last year.
- babberman 3y agoThat is unfortunate, but I guess those SSDs performed really well and outclassed all others in performance benchmarks? lol
- sashk 3y agoThis is (2022). Wondering if anything changed since the original tests...
- throw0101b 3y ago> Wondering if anything changed since the original tests... You're wondering if firmware writers lie to layers higher up in the stack? I think it's a 100% certainly that there's drive firmware that lies. There's a reason why many vendors have compatibility lists, approved firmware versions, and even their "own" (rebranded from an OEM) drives that you have to buy if you want official support (and it's not entirely a money grab: a QA testing infrastructure does cost money).
- Tarball10 3y agoI'm curious whether any of the brands which failed this test owned up to the issue and released firmware updates.
- babberman 3y ago[flagged]
- layer8 3y agoMultiple reports in the Twitter thread that this also happens for server-grade HDDs.
- RedShift1 3y agoI couldn't find this? Can you link this claim?
- layer8 3y agohttps://nitter.net/john_p_looney/status/1495823128546746372#m https://nitter.net/john_p_looney/status/1495823128546746372#... https://nitter.net/lgerbarg/status/1495822123772018688#m https://nitter.net/lgerbarg/status/1495822123772018688#m I interpreted those as being server-grade but maybe they’re not.
- RedShift1 3y agoThis whole Twitter thread is garbage. "2 out of 35 drives passed". Alright, but which drives are that?
- jauntywundrkind 3y agoLosing flushes is obviously bad. I wonder how much perf is on the table in various scenarios when we can give up needing to flush. If you know the drive has some resilience, say, 0.5s of time it can safely writeback during, maybe you can give up flushes (in some cases). How much faster is the app then? It's be neat to see some low-cost improvements here. Obviously in most cases, just get an enterprise drive with supercapa or batteries onboard. But an ATX power rail that has extra resilience from the supply, or an add-in/pass-through 6-pin sata power supercap... that could be useful too.
- invalidator 3y agoIf the write-cache is reordering requests (and it does, that's the whole point), you can't guarantee that $milliseconds will be enough unless you stop all requests, wait $milliseconds, write your commit record, wait $milliseconds, then resume requests. This is essentially re-implementing write-barriers in an ad-hoc, buggy way which requires stalling requests even longer. Flush+FUA requires the data to be stored to non-volatile media. Capacitor-backed RAM dumping to flash is non-volatile. When a drive knows it has enough capacitor-time to finish flushing all preceding writes from the cache, it can immediately say the flush was completed. This can all be handled on the device without the software having to make guesses at how long something has to be written before it's durable.
- supersour 3y agoPerformance gains wouldn’t be that large as enterprise SSDs already have internal capacitors to flush pending writes to NAND. During typical usage the flash controller is constantly journaling LBA to physical addresses in the background, so that the entire logical to physical table isn’t lost when the drive loses power. With a larger capacitor you could potentially remove this background process and instead flush the entire logical to physical table when the drive registers power loss. But as this area makes up ~2% of the total NAND, that’s at absolute best a 2% performance benefit we are potentially missing out on.
- hurryer 3y agoYou could gain much more by coalescing repeated writes to the same address - database scenarios for example
- handedness 3y agoPrevious Discussion: https://news.ycombinator.com/item?id=30419618 https://news.ycombinator.com/item?id=30419618
- CoastalCoder 3y agoDoes advertising a product as adhering to some standard, but secretly knowing that it doesn't 100%, count as e.g. fraud? I.e., is there any established case law on the matter? I'm thinking of this example, but also more generally USB devices, Bluetooth devices, etc.
- hsbauauvhabzb 3y agoHardware vendors are known to swap to cheaper lower performance hardware after reviews are out which in my eyes is fraud, whether or not the law agrees is a different story.
- matheusmoreira 3y agoPlease name the vendors that do this so I can avoid them.
- FridgeSeal 3y agoSamsung was caught doing this with their 970 pro( plus? Their naming is awful) - they swapped out the controller in a good portion of devices which resulted in significantly lower read and write performance.
- kawsper 3y agoLTT documented it happening to at least: ADATA, Kingston and PNY in this video: https://www.youtube.com/watch?v=K07sEM6y4Uc https://www.youtube.com/watch?v=K07sEM6y4Uc
- yjftsjthsd-h 3y agoI was under the impression that a lot of off-brand USB devices didn't use the USB logo specifically to get around certification requirements. Basically, they just aren't advertising adherence to a standard. No idea about NVMe or BT.
- adhesive_wombat 3y ago
- hackerfactor1 3y agoThe posting is from Feb 2022, nearly 2 years ago. How is this suddenly trending on Hacker News?
- laluser 3y agoDoesn’t take much for a story to trend. Also, it’s an interesting topic.
- IngvarLynn 3y agoThere is a flood of fake SSDs currently, mostly big brands. I've recently purchased counterfeit 1TB. It passes all the tests, performance is ok, it works... except it gets episodes where ioping would be anything between 0.7 ms and 15 seconds, that is under zero load. And these are quality fakes from a physical appearance perspective. The only way I could tell mine was fake is that the official Kingston firmware update tool would not recognize this drive.
- jwells89 3y agoThat’s wild. Is this limited to specific distribution channels or can you get them from anywhere?
- loeg 3y agoWhere are you seeing counterfeits? AliExpress, Ebay, Amazon?
- Shekelphile 3y agoProbably chinese sellers on all those sites. I've noticed a common thread with people who complain about counterfeits is that they're literally buying alphabet soup brand fakes from chinese FBA sellers instead of buying products directly sold by amazon or from more traditional retail channels.
- vineyardmike 3y agoThere’s definitely a problem with my grandma or some less-technically educated person buying “alphabet soup” fakes, BUT Amazon does commingle inventory. This means that lots of people can end up with fakes sold by 3rd parties when buying from a reputable brand. There were even stories of those crazy coupon people reselling on Amazon, and some cases of returned retail products ending up as “new” on Amazon. Which gets problematic with certain things like consumables (the WSJ did an article on toothpastes iirc).
- traceroute66 3y ago> I've noticed a common thread with people who complain about counterfeits is that they're literally buying alphabet soup brand fakes from chinese FBA sellers instead of buying products directly sold by amazon AMEN to that ! And the most annoying thing is that those of us who know to avoid FBA is that Amazon have removed the "sold by Amazon" search filter tick-box. So whilst in the past you could tick a box and be presented with a list of products which are direct-sold rather than FBA, you cannot do that anymore. According to some Reddit posts, you can still do it if you hack the URL and add an "emi=$obscure_value" GET-param. But I'm guessing sooner or later Amazon will kill this work-around too.
- ricardobeat 3y agoMisleading headline since after testing eight more drives, none more failed. 2/12 is not nearly as dramatic as “half”, and the ones that lost data are the cheap brands as one would expect.
- alanfranz 3y agoYou can either not editorialize the title, and accept that the thread contains updates, or editorialize it and violate HN guidelines. Either choice will lead somebody to complain
- wannacboatmovie 3y agoClearly they should only editorialize the ones that are wrong.
- lmm 3y agoThat doesn't help, HN mods still "correct" it back to the wrong title even in that case.
- wannacboatmovie 3y agoThat was sarcasm, as what's "wrong" is often subjective and up to interpretation.
- Dylan16807 3y agoAnd it's often not.
- wannacboatmovie 3y agoThen it gets flagged and removed. The end.
- 3y ago
- r1ch 3y agoWe shipped a shader cache in the latest release of OBS and quickly had reports come in that the cached data was invalid. After investigating, the cache files were the correct size on disk but the contents were all zero. On a journaled file system this seems like it should be impossible, so the current guess is that some users have SSDs that are ignoring flushes and experience data corruption on crash / power loss.
- bugfix 3y agoI had this exact experience with my workstation SSD (NTFS) after a short power loss while NPM was running. After I turned the computer back on, several files (package.json, package-lock.json and many others inside node_modules) had the correct size on disk but were filled with zeros. I think the last time I had corrupted files after a power loss was in a FAT32 disk on Win98, but you'd usually get garbage data, not all zeros.
- wannacboatmovie 3y agoThey may be pointing to unallocated space which on a SSD running TRIM would return all zeros. NTFS is an extremely resilient yet boring filesystem, I cannot remember the last time I had to run chkdsk even after an improper shutdown.
- Sakos 3y agoAs somebody who worked as a PC technician for a while until very recently, I've run chkdsk and had to repair errors on NTFS filesystems very, very, very often. It's almost an everyday thing. Anecdotal evidence is less than useful here.
- dspillett 3y agoSo anecdotal evidence is not useful, as proven by your anecdotal evidence? :) FWIW I've found NTFS and ext3/4 to be of similar reliability over the years, in general use and in the face of improper shutdown. Metadata journaling does a lot to preserve the filesystem in such circumstances. Most of the few significant problems I've had have been due to hardware issues, which few filesystems on their own will help you with. It is worth noting that when you run tools like chkdsk or fsck, some of the issues reported and fixed are not data damaging, or structurally dangerous, or at least not immediately so. For instance free areas marked in such a way that makes them look used to the allocation algorithms.
- pleoxy 3y agoIf you need PLP use an enterprise drive. That's what they're for.
- RecycledEle 3y agoPLP = Power Loss Prevention
- Dylan16807 3y agoWell, I don't need PLP. I just need to know when the data is written and when it isn't.
- anarazel 3y agoI've seen lost completed FUA writes on enterprise drives quite a few times.
- schleyreuth 3y agoIf stuck with consumer drives, you can add cheap PLP via riser cards that have a supercapacitor. Here's a post on the TrueNAS forums that tested one out.[1] [1] https://www.truenas.com/community/threads/x4-pcie-to-nvme-adapter-with-supercapacitor.58810/ https://www.truenas.com/community/threads/x4-pcie-to-nvme-ad...
- tripdout 3y agoFlushing in this case is from the SSDs internal DRAM cache to the actual NAND flash?
- MBCook 3y agoIt’s the computer telling the drive “write everything to durable storage (as opposed to some kind of in-drive cache/RAM) and tell me when it’s done”. After that command it should be 100% safe to pull the power because everything SHOULD have been written to flash. That’s the point of the command. It’s interesting that the drives that do it wrong still take time indicating they’re doing something.
- supertrope 3y agoThe DRAM cache does not hold user data. It holds the flash transition layer that links LBAs to NAND pages. Higher performance drives use 1GB of DRAM per 1TB of NAND. In cheap DRAM-less drives if the I/O to be serviced is not cached in the 1MB or so of SRAM it has to do a double lookup. Once to retrieve the full FTL table from NAND and a second lookup to actually service the I/O.
- caycep 3y agoThe model I’d be interested in would be the SK Hynix/Solidigm P44 Pro, as that model competes w the Samsung 9xx evo and pro models
- caycep 3y agoAlso…would modern journaling file systems protect against this sort of data loss?
- GrayShade 3y agoThey won't, since they rely on the drive to make sure their data was actually persisted.
- jbverschoor 3y agoBrands please. It’s time they have some pressure to fix these data corruption issues
- Joel_Mckay 3y agoCheap drives don't include large dram caches, lack fast SLC areas, and leave off super-capacitors that allow chips to drain buffers during a power-failure. "Buy cheap, buy twice" as they say... =)
- tw1984 3y agoDon't use home-grade SSDs for storing anything that is considered critical. The rule is not that hard to remember.
- kristopolous 3y agoUnder long term heavy duty, I've routinely seen cheap modern platter outperform cheap brand name NVME. There's some cost cutting somewhere. The NVMEs can't seem to sustain throughput. It's been pretty disappointing to move I/O bound workloads over and not see notable improvements. The magnitude of data I'm talking about is 500-~3000GB I've only got two NVME machines for what I'm doing so I'll gladly accept that it's coincidentally flaky bus hardware on two machines, but I haven't been impressed except for the first few seconds. I know Everyone says otherwise which is why I brought it up. Someone tell me why I'm crazy Edit: no, I'm not crazy. https://htwingnut.com/2022/03/06/review-leven-2tb-2-5-sata-ssd/ https://htwingnut.com/2022/03/06/review-leven-2tb-2-5-sata-s... this is similar to what I'm seeing with Crucial and Adata hardware, almost binary performance
- efxhoy 3y agoI think cheaper QLC chips use a part of their storage space as SLC, which is fast to write. But once you’ve written the fast part that fits in the SLC cache write throughput quickly tanks as it has to push the data further in to the slower QLC parts.
- kristopolous 3y agoYeah I guess it works well for how most people use computers which is not actually for computation... Modern platter is actually pretty decent and cheap. It's probably still the way to go for large loads unless you have a grove of money trees
- Sakos 3y ago1) Nobody says otherwise about cheap anything NVMe. They're pretty terrible once they've exhausted the write cache. This is well-known and addressed in every decent review by reputable sites. 2) Sustaining throughput seems the least of our problems when some unknown number of NVMe SSDs might be literally losing flushed data.
- kristopolous 3y agoIs this expected with say Samsung evo 9X0 pro? Or is there another tier above consumer level gear? Is there something I should go with?
- nik736 3y agoThat's what PLP is for.
- arglebargle123 3y agoMeanwhile I'm over here jamming Micron 7450 pros into my work laptop for better sync write performance. I have very little trust in consumer flash these days after seeing the firmware shortcuts and stealth hardware replacements manufacturers resort to to cut costs.
- CobaltFire 3y agoHave a solid vendor for these that isn't insanely priced (for home use)? The last couple I tried to buy they sent 7300's and tried to buy me off with a small refund (eBay).
- xbmcuser 3y agothis is from Feb 2022
- kmxdm 3y agoWrites are completed to the host when they land on the SSD controller, not when written to Flash. The SSD controller has to accumulate enough data to fill its write unit to Flash (the absolute minimum would be a Flash page, typically 16kB). If it waited for the write to Flash to send a completion, the latency would be unbearable. If it wrote every write to Flash as quickly as possible, it could waste much of the drive's capacity padding Flash pages. If a host tried to flush after every write to force the latter behavior, it would end up with the same problem. Non-consumer drives solve the problem with back-up capacitance. Consumer drives do not have this. Also, if the author repeated this test 10 or 100 times on each drive, I suspect that he would uncover a failure rate for each consumer drive. It's a game of chance.
- gumby 3y agoThe whole point of explicit flush is to tell the drive that you want the write at the expense of performance. Either the drive should not accept the flush command or it should fulfill it, not lie. (BTW this points out the crappy use of the word “performance” in computing to mean nothing but “speed”. The machine should “perform” what the user requests — if you hired someone to do a task and they didn’t do it, we’d say they failed to perform. That’s what’s going on here.)
- kmxdm 3y agoThe more dire problem is the case where the drive runs out of physical capacity before logical capacity. If the host flushes data that is smaller than the physical write unit of the SSD, capacity is lost to padding (if the SSD honors every Flush). A "reasonable" amount of Flush would not make too much of a difference, but a pathological case like flush-after-every-4k would cause the SSD to run out of space prematurely. There should be a better interface to handle all this, but the IO stack would need to be modified to solve what amounts to a cost issue at the SSD level. It's a race to the bottom selling 1TB consumer SSDs for less than $100.
- nolist_policy 3y agoI still don't think this is the problem, the drive can just slow down accepting writes until it has reclaimed enough space. The bigger problem is manufacturers chasing the performance. Generally you get the feeling they just hit their firmware with a hammer so it barely doesn't break NTFS. See also the drama around btrfs' "unreliability", which is all traced back to drives with broken firmware. I fully expect bcachefs will get exactly the same problems.
- 39 3y agoThis has always been the case? At least it was a course learning when we wrote our own device drivers for minux, even the controllers on spinning metal fib about flush.
- martincmartin 3y ago2022
- dtx1 3y agoI am a bit annoyed that everyone here takes this at face value. There's 0 evidence given, not even the vendors and models are named to confirm this. On a related note I tested 4 DDR5 Ram kits from major vendors - half of them corrupt data when exposed to UV light.
- deleted 3y ago[deleted]
- naasking 3y agoAt this point, any storage vendor should be required to pass the Sqlite test suite before they can sell their product.
- pajko 3y agoWithout any more information this post is just bullshit. For example, it's not documented how the flushing has been done. On Linux, even issuing 'sync' is not enough: https://unix.stackexchange.com/questions/98568/difference-between-blockdev-flushbufs-and-sync-on-linux https://unix.stackexchange.com/questions/98568/difference-be... The bottom answer especially states that "blockdev --flushbufs may still be required if there is a large write cache and you're disconnecting the device immediately after" The hpdarm utility has a parameter for syncing and flushing device buffers themselves. Seems like all three should be done for a complete flush at all levels.
- nijave 3y agoIt'd be nice if there were a database of known bad/known good hardware to reference. I know there's been some spreadsheets and special purpose like the USB-C cables Benson Leung tested. Especially for consumer hardware on Linux--there's a lot of stuff that "works" but is not necessarily stable long term or that required a lot of hacking on the kernel side to work around issues