68 ms·

I tested four NVMe SSDs from four vendors – half lose FLUSH’d data on power loss

797 points by ahachete 5y ago

nurettin 5y ago
I love the dedication on this guy, buying a dozen 2TB SSDs just to test them for FLUSH consistency.
eptcyka 5y ago
Hey, now I want/need to know if my SSD is flushing correctly or not.
mjw1007 5y ago
« Data loss occured with a Korean and US brand, but it will turn into a whole "thing" if I name them so please forgive me. »
- LeifCarrotson 5y ago
  > The models that never lost data: Samsung 970 EVO Pro 2TB and WD Red SN700 1TB. The others would probably be SK Hynix and Micron/Crucial, right? Curious why he's reluctant to name and shame. A drive not conforming to requirements and losing data is a legitimate problem that should be a "thing"!
  - 420official 5y ago
    > Curious why he's reluctant to name and shame My sense is he wants to shame review sites for not paying attention to this rather than shame manufacturers directly at this point.
  - nijave 5y ago
    Looks like he works at Apple. Maybe what he's testing is work related or covered by some sort of NDA (e.g. doesn't want to risk harming supplier relations for the brands misbehaving)
  - alliao 5y ago
    I thought Crucial specifically designed some power loss protection as a differentiating selling point? Well at least that was the reason why I bought one back in M.2 days (gosh my PC is ancient...)
    - iforgotpassword 5y ago
      Back in M.2 days? I know I'm getting old, but what did miss?
      - cestith 5y ago
        M.2 SATA as opposed to NVMe perhaps?
      - alliao 5y ago
        it's the storage form factor of my current PC.. which is probably close to 10yrs old :P https://en.wikipedia.org/wiki/M.2 https://en.wikipedia.org/wiki/M.2
        sascha_sl 5y ago
        It is still being used in very recent PCIe Gen 4 NVMe drives. Though you likely have a different key only taking SATA drives.
    - wtallis 5y ago
      I think the most they ever promised for some of their consumer drives was that a write interrupted by a power failure would not cause corruption of data that was already at rest. Such a failure can be possible when storing multiple bits per memory cell but programming the cells in multiple passes, especially if later passes are writing data from an entirely different host write command.
    - ryan29 5y ago
      It changed [1] from capacitors and "power loss protection" to something else described as "power loss immunity" with the MX500. I don't think I've ever seen it explained very well. > With the release of the MX500, Crucial has included a new replacement for the traditional power loss protection feature, power loss immunity. Instead of relying on a bank of capacitors for power loss protection, Crucial was able to work the new 3D TLC NAND and the code to allow for more efficient NAND programming so that the capacitors are no longer needed. That's just a regurgitated press release IMO. A lot of consumer drives also stopped reporting DRAT/RZAT [2] around the Crucial MX500, Samsung 850 timeframe. They swap internals as others in this thread have pointed out and the write endurance has dropped since reviewers stopped reporting on it. I have a Crucial MX500 in my system right now with 11% life remaining and only 37TBW even though it's advertised as having 180TBW of endurance. Edit: I actually found [3] an explanation of "power loss immunity". > The impact is still the same: you don't get the full protection that is standard for enterprise SSDs, but data that has already been written to the flash will not be corrupted if the drive loses power while writing a second pass of more data to the same cells. I always thought write operations on SSDs were more or less to write a new page or block or whatever the terminology is and to flag the old one(s) for garbage collection. I don't understand how it would be possible to lose the old data by doing that. Did they just invent a term that sounds like power loss protection, but doesn't actually do anything special? 1. https://www.thessdreview.com/our-reviews/crucial-mx500-ssd-review-1tb-the-best-value-in-sata/ https://www.thessdreview.com/our-reviews/crucial-mx500-ssd-r... 2. https://forums.anandtech.com/threads/looking-for-new-ssd-that-correctly-reports-drat-or-rzat-support.2556431/ https://forums.anandtech.com/threads/looking-for-new-ssd-tha... 3. https://www.anandtech.com/show/12165/the-crucial-mx500-1tb-ssd-review https://www.anandtech.com/show/12165/the-crucial-mx500-1tb-s...
  - 0xcde4c3db 5y ago
    Crucial seems plausible, but there's a surprising number of US brands for NVMe SSDs. I was able to find: Crucial, Corsair, Seagate, Plextor, Sandisk, Intel, Kingston, Mushkin, PNY, Patriot Memory, and VisionTek.
    - LeifCarrotson 5y ago
      Micron/Crucial is the 3rd largest manufacturer of flash memory, most of the other brands in your list just make the PCB and whitelabel other flash chips and controllers (perhaps with some firmware customization, but they're usually not responsible for implementing features like FLUSH). Toshiba/Kioxia is another big one, but they're based in Japan. The US brand could be Intel instead of Crucial, I suppose.
hughrr 5y ago
The important quote: > The models that never lost data: Samsung 970 EVO Pro 2TB and WD Red SN700 1TB. I always buy the EVO Pro’s for external drives and use TB to NVMe bridges and they are pretty good.
- Trellmor 5y ago
  There is a 970 Evo, a 970 Pro and a 970 Evo Plus, but no 970 Evo Pro as far as I am aware. Would be interesting what model OP is actually talking about and if it is the same for other Samsung NMVe SSDs. I also prefer Samsung SSDs because they are reliably and they usually don't change parts to lower spec ones while keeping the same model number like some other vendors.
  - hughrr 5y ago
    Sorry I should have said EVO plus there in my original post. I’ll leave the error in so your comment makes sense.
    - fuzzybear3965 5y ago
      How do you know the model? People are asking the same question in Twitter and OP doesn't seem to have supplied an answer, then.
    - zargon 5y ago
      The "EVO Pro" error was made by the OP. So it would be nice to know which drive OP actually tested.
      - hughrr 5y ago
        Indeed. I use the EVO Plus NVMe's though.
      - cgriswald 5y ago
        OP has since appended to his post: > Correction: “Plus” not “Pro”. Exact model and date codes: > Samsung 970 Evo Plus: MZ-V7S2T0, 2021.10 > WD Red: WDS100T1R0C-68BDK0, 04Sept2021
dboreham 5y ago
Surprised this is being rediscovered. Years ago only "enterprise" or "data center" SSDs had the supercap necessary to provide power for the controller to finish pending writes. Did we ever expect consumer SSDs to not lose writes on power fail?
- evancox100 5y ago
  It's not losing pending writes, it's the drive saying there are no pending writes, but losing them anyways. ie the drive is most likely lying
  - monocasa 5y ago
    As I said in the recent Apple discussion, pretty much all drives are lying and have been for decades at this point. The good brands just spec out enough capacitance that you don't see the difference externally. https://news.ycombinator.com/item?id=30370551#30374585 https://news.ycombinator.com/item?id=30370551#30374585
    - brigade 5y ago
      So if SSDs rely solely on capacitors for data integrity and lie about flushes, what do they do on a flush that takes any amount of time? Are they just taking a speed hit for funsies? Heck, from this test, the magnitude of the speed hit isn't even correlated with whether they lose writes...
      - monocasa 5y ago
        At one point it was different barriers on the different submission queues inside the drive. Not externally visible queues, but between internal implementation layers. It's been a few years since I've checked up on this and it was for the most part pre SSDs though.
      - supermatt 5y ago
        Probably implementing a barrier for ordering io…
monocasa 5y ago
Relevant recent discussion about Apple's NVMe being very slow on FLUSH. Apple's custom NVMes are amazingly fast – if you don't care about data integrity https://news.ycombinator.com/item?id=30370551 https://news.ycombinator.com/item?id=30370551
- hughrr 5y ago
  Not really a problem when your computer has a large UPS built into it. Desktop macs, not so good. But really isn’t the point of a journaling file system to make sure it is consistent at one guaranteed point in time, not necessarily without incidental data loss.
  - monocasa 5y ago
    Journaling file systems depend on commands like FLUSH to create the appropriate barriers to build their journal's semantics.
    - rowanG077 5y ago
      I don't have a clue how a journaling FS works. But any ordering should not be observable unless you have a power outage. Can you give an example how a journaling FS could observe something that should be observable?
      - b112 5y ago
        A crash, lockup, are the same as a power failure.
        colanderman 5y ago
        No, during a crash or lockup, acknowledged writes are not lost. (Because the drive has acknowledged them, they are in the drive's internal queue and thus need no further action from the OS to be committed to durable storage.) Only power loss/power cycle causes this.
formerly_proven 5y ago
> I guess review sites don’t test this stuff. Most review sites don't really test much at all.
- sokoloff 5y ago
  Except for their affiliate links; they do test those.
post_break 5y ago
The problem is you can't trust a model number of SSD. They change controllers, chips, etc after the reviews are out and they can start skimping on components. https://www.tomshardware.com/news/adata-and-other-ssd-makers-swapping-parts https://www.tomshardware.com/news/adata-and-other-ssd-makers...
- infogulch 5y ago
  This needs to be cracked down on from a consumer protection lens. Like, any product revision that could potentially produce a different behavior must have a discernable revision number published as part of the model number.
  - sokoloff 5y ago
    While I agree with the sentiment, even a firmware revision could cause a difference in behavior and it seems unreasonable to change the model number on every firmware release.
    - jay_kyburz 5y ago
      I don't think its unreasonable to just add a date next to the serial number that has when the firmware was released. It's 6 extra numbers
      - yjftsjthsd-h 5y ago
        But firmware can be updated after the unit was shipped; a date stamped on the hardware can't promise accuracy.
        lillecarl 5y ago
        Is this the definition of whataboutism?
        yjftsjthsd-h 5y ago
supermatt 5y ago
So, seems those drives may have been ignoring the F_FULLFSYNC after all… https://news.ycombinator.com/item?id=30371857 https://news.ycombinator.com/item?id=30371857 The Samsung EVO drives are interesting because they have a few GB of SLC that they use as a secondary buffer before they reflush to the MLC.
- verall 5y ago
  > reflush to the MLC I'm nitpicking, but an EVO has TLC. Also an SLC write cache is the norm for any high performance consumer ssd, it's not just Samsung.
  - fire 5y ago
    > I'm nitpicking, but an EVO has TLC. b...but the M in MLC stands for multi... as in multiple... right? checks Oh... uh; Apparently the obvious catch-all term MLC actually only refers to dual layer cells, but they didn't call it DLC, and now there's no catch-all term for > SLC. TIL.
    - marcan_42 5y ago
      The "L" stands for level, and that makes it even more wrong. MLC should have been been QLC or DBC, and TLC should have been OLC or TBC. It's two bits and four levels, or three bits and eight levels. The latest "QLC" flash has 16 levels (and is a massive step down in performance and reliability; it seems TLC is the sweet spot, at least right now, unless you really want the absolute cheapest, performance be damned). Interestingly enough, Apple has a patent for 8-bit cell flash (256 levels!), going full blown analog processing and error correction, but I don't think that ever became a product.
  - fomine3 5y ago
    I believe now SLC cache is available on every TLC consumer SSDs, because it's slow without it.
    - wtallis 5y ago
iforgotpassword 5y ago
I think this is something LTT could handle with their new test lab. They already said they want to set new standards when it comes to hardware testing, so if they can hold up to what they promised and hire enough experts this should be a trivial thing to add to a test Parcours for disk drives.
- balls187 5y ago
  LTT's commentary makes it difficult to trust they are objective (even if they are). I loved seeing how giddy Linus got while testing Valve's Steamdeck, but when it comes to actual benchmarks and testing, I would appreciate if they dropped the entertainment aspect.
  - KennyBlanken 5y ago
    GamersNexus seems to really be trying to work on improving and expanding their testing methodology as much as possible. I feel like they've successfully developed enough clout/trust that they have escaped the hell of having to pull punches in order to assure they get review samples. They eviscerated AMD for the 6500xt. They called out NZXT repeatedly for a case that was a fire hazard (!). Most recently they've been kicking Newegg in the teeth for trying to scam them over a damaged CPU. They've called out some really overpriced CPU coolers that underperform compared to $25-30 coolers. Etc. I bet they'd go for testing this sort of thing, if they haven't already started working on it already. They'd test it and then describe for what use cases it would be unlikely to be a problem vs what cases would be fine. For example, a game-file-only drive where if there's an error you can just verify the game files via the store application. Or a laptop that's not overclocked and only is used by someone to surf the web and maybe check their email.
    - nezgar 5y ago
      already already
    - throwaway2037 5y ago
      This is a good idea. You should send a suggestion to them!
timbit42 5y ago
What is an EVO Pro?
- tromp 5y ago
  See comment above; it's an EVO Plus
  - timbit42 5y ago
    I don't have twitter. Someone should tell the tweeter to fix their tweet.
    - closewith 5y ago
      Tweets can't be edited, unfortunately.
      - timbit42 5y ago
        Since I don't have Twitter, I didn't know that. Yet people down voted me. Nice. Stay classy HN.
    - deleted 5y ago
      [deleted]
shadowgovt 5y ago
As a frame of reference, how much loss of FLUSH'd data should be expected on power loss for a semi-permanent storage device (including spinning-platter hard drives, if anyone still installs them in machines these days)? I'm far more used to the mainframe space where the rule is "Expect no storage reliability; redundancy and checksums or you didn't want that data anyway" and even long-term data is often just stored in RAM (and then periodically cold-storage'd to tape). I've lost sight of what expected practice is for desktop / laptop stuff anymore.
- xenadu02 5y ago
  The semantics of a FLUSH command (per NVMe spec) is that all previously sent write commands along with any internal metadata must be written to durable storage before returning success. Basically the drive is saying "yup, it's all on NAND - not in some internal buffer. You can power off or whatever you want, nothing will be lost". Some drives are doing work in response to that FLUSH but still lose data on power loss.
  - benlwalker 5y ago
    A flush command only guarantees, upon completion, that all writes COMPLETED prior to submission of the flush are non-volatile. Not all previously sent writes. NVMe base specification 2.0b section 7.1. That's a very important distinction. You can't assume just because a write completed before the flush that it's actually durable. Only if it completed before you sent the flush. I'm not very confident that software is actually getting this right all that often, although it probably is in this fsync test.
    - wmf 5y ago
      Is there a separate barrier command so you don't have to track all the writes individually in software?
      - benlwalker 5y ago
        Nope. The reason this is so complex is that these devices are actually highly parallel machines with multiple queues accepting commands. It's quite difficult to even define "before" in terms of command sequence. For example, if you have a device with two hardware queues for submitting commands and a software thread for each, if you submit a flush on one queue, which commands on the other queue does it affect? Or what if the device issues a pci write to the completion entry that passes a flush command being submitted on the wire? I think the only interpretation that makes sense is from the perspective of a single software thread. If that particular thread has seen the completion via any mechanism and then that thread issues the flush, then you know the write is durable. Other than that, the device makes no promises.
gruez 5y ago
Correct me if I'm wrong, but if these drives are used for consumer applications, this behavior is probably not a big deal? If you made changes to a document, pressed control-S, and then 1 second later the power went out, then you might lose that last save. That'd suck, but you would have lost the data anyways if the power loss occurred 2s before, so it's not that bad. As long as other properties weren't violated (eg. ordering), your data should mostly be okay, aside from that 1s of data. It's a much bigger issue for enterprise applications, eg. a bank's mainframe responsible for processing transactions told a client that the transaction went through, but a power loss occurred and the transaction was lost.
- __david__ 5y ago
  > As long as other properties weren't violated (eg. ordering), your data should mostly be okay, aside from that 1s of data. That's the thing though—ordering isn't guaranteed as far as I remember. If you want ordering you do syncs/flushes, and if the drive isn't respecting those, then ordering is out of the window. That means FS corruption and such. Not good.
  - gruez 5y ago
    The tweet only mentioned data loss when you yanked the power cable. That doesn't say anything about whether the ordering is preserved. It's possible to have a drive that lies about data written to persistent storage, but still keeps the writes in order.
- colanderman 5y ago
  > As long as other properties weren't violated (eg. ordering) That is primarily what fsync is used to ensure. (SCSI provides other means of ensuring ordering, but AFAIK they're not widely implemented.) EDIT: per your other reply, yes, it's possible the drives maintain ordering of FLUSHed writes, but not durability. I'm curious to see that tested as well. (Still an integrity issue for any system involving more than just one single drive though.)
  - deleted 5y ago
    [deleted]
colanderman 5y ago
I'm curious whether the drives are at least maintaining write-after-ack ordering of FLUSHed writes in spite of a power failure. (I.e., whether the contents of the drives after power loss are nonetheless crash consistent.) That still isn't great, as it messes with consistency between systems, but at least a system solely dependent on that drive would not suffer loss of integrity.
jrockway 5y ago
I always liked the embedded system model where you get flash hardware that has two operations -- erase block and write block. GC, RAID, error correction, etc. are then handled at the application level. It was never clear to me that the current tradeoff with consumer-grade SSDs was right. On the one hand, things like the error correction, redundancy, and garbage collection don't require the attention from CPU (and more importantly, doesn't tie up any bus). On the other hand, the user has no control over what the software on the SSD's chip does. Clearly vendors and users are at odds with each other here; vendors want the best benchmarks (so you can sort by speed descending and pick the first one), but users want their files to exist after their power goes out. It would be nice if we could just buy dumb flash and let the application do whatever it wants (I guess that application would be your filesystem; but it could also be direct access for specialized use cases like databases). If you want maximum speed, adjust your settings for that. If you want maximum write durability, adjust your settings for that. People are always looking for that one size fits all use case, but it's hard here. Some people may be running cloud providers and already have software to store that block on 3 different continents. Some people may be an embedded system with a fixed disk image that changes once a year, with some temporary storage for logs. There probably isn't a single setting that gets optimal use out of the flash memory for both use cases. The cloud provider doesn't care if a block, flash chip, drive, server rack, availability zone, or continent goes away. The embedded system may be happy to lose logs in exchange for having enough writes left to install the next security update. It's all a mess, but the constraints have changed since we made the mess. You used to be happy to get 1/6th of a PCI Express lane for all your storage. Now processors directly expose 128 PCIe lanes and have a multitude of underused efficiency cores waiting to be used. Maybe we could do all the "smart" stuff in the OS and application code, and just attach commodity dumb flash chips to our computer.
- bob1029 5y ago
  I would absolutely love to have access to "dumb" flash from my application logic. I've got append only systems where I could be writing to disk many times faster if the controller weren't trying to be clever in anticipation of block updates.
  - 5y ago
hardwaresofton 5y ago
I've actually run into some data loss running simple stuff like pgbench on Hetzner due to this -- I ended up just turning off write-back caching at the device level for all the machines in my cluster: https://vadosware.io/post/everything-ive-seen-on-optimizing-postgres-on-zfs-on-linux/#solution-force-proper-disk-syncing https://vadosware.io/post/everything-ive-seen-on-optimizing-... Granted I was doing something highly questionable (running postgres with fsync off on ZFS) It was very painful to get to the actual issue, but I'm glad I found out. I've always wondered if it was worth pursuing to start a simple data product with tests like these on various cloud providers to know where these corners are and what you're really getting for the money (or lack thereof). [EDIT] To save people some time (that post is long), the command to set the feature is the following: nvme set-feature -f 6 -v 0 /dev/nvme0n1 The docs for `nvme` (nvme-cli package, if you're Ubuntu based) can be pieced together across some man pages: https://man.archlinux.org/man/nvme.1 https://man.archlinux.org/man/nvme.1 https://man.archlinux.org/man/nvme-set-feature.1.en https://man.archlinux.org/man/nvme-set-feature.1.en It's a bit hard to find all the NVMe features but 6 is the one for controlling write-back caching. https://unix.stackexchange.com/questions/472211/list-features-of-nvme-drive-like-hdparm-i-for-non-nvme https://unix.stackexchange.com/questions/472211/list-feature...
- hrgiger 5y ago
  I dont have ide in this machine but I found this in the source code [1] probably pointing to [2] and thanks for the tip! [1]https://github.com/linux-nvme/nvme-cli/blob/master/nvme-print.c#L6272 https://github.com/linux-nvme/nvme-cli/blob/master/nvme-prin... [2]https://github.com/linux-nvme/libnvme/blob/master/src/nvme/types.h#L6155 https://github.com/linux-nvme/libnvme/blob/master/src/nvme/t...
  - hardwaresofton 5y ago
    ah thanks this is perfect, saved those links!
    - wtallis 5y ago
      Also: https://nvmexpress.org/developers/nvme-specification/ https://nvmexpress.org/developers/nvme-specification/ Unlike eg. ATA and SCSI, the NVMe specs are freely available to the public. They're a little more complicated to read now that the spec has been split into a few modules, but finding the descriptions of all the optional features isn't too hard.
nyc_pizzadev 5y ago
I’m actually interested in testing this scenario, a drive getting power loss. Is there a thing which will cut power to a server device on command? Or do you just pull the drive out of its bay?
- klysm 5y ago
  I don’t know of a way on most systems to cut the power directly. It seems fairly straightforward to wire up a relay though.
- toast0 5y ago
  If your server has IPMI or similar, you can use that to cut power (not to the BMC though), otherwise, network PDUs or many UPSes or consumer level networked outlets or something fun with a relay (be careful with mains voltages). Pulling the drive is also worth testing though; you might get different results. Requires more human involvement though.
tr33house 5y ago
I'm a systems engineer but I've never done low level optimizations on drives. How does one even go about even testing something like this? It sounds like something cool that I'd like to be able to do
- klysm 5y ago
  Write and flush and unplug the cable!
  - StillBored 5y ago
    which is more difficult (and sometimes slower) than STONITH style devices which just kill power to the entire machine. The latter allow you to program the whole thing and run test cycle after test cycle where the device kills itself the moment it gets a successful flush.
- xenadu02 5y ago
  My script repeatedly writes a counter value "lines=$counter" to a file, then calls fcntl() with F_FULLFSYNC against that file descriptor which on macOS ends up doing an NVMe FLUSH to the drive (after sending in-memory buffers and filesystem metadata to the drive). Once those calls succeed it increments the counter and tries again. As soon as the write() or fcntl() fail it prints the last successfully written counter value which can be checked against the contents of the file. Remember: the semantics of the API and the NVMe spec require that a successful return from fcntl(fd, F_FULLFSYNC) on macOS require that data is durable at that point no matter what filesystem metadata OR drive internal metadata is needed to make that happen. In my test while the script is looping doing that as fast as possible I yank the TB cable. The enclosure is bus powered so it is an unceremonious disconnect and power off. Two of the tested drives always matched up: whatever the counter was when write()+fcntl() succeeded is what I read back from the file. Two of the drives sometimes failed by reporting counter values < the most recent successful value, meaning the write()+ fcntl() reported success but upon remount the data was gone. Anytime a drive reported a counter value +1 from what was expected I still counted at that as success... after all there's a race window where the fcntl() has succeeded but the kernel hasn't gotten the ACK yet. If disconnect happens at that moment fcntl() will report failure even though it succeeded. No data is lost so that's not a "real" error.
throwaway984393 5y ago
Aren't there filesystem options that affect this? Wasn't there a whole controversy over ext4 and another filesystem not committing changes even after flush (under specific options/scenarios)?
- wmf 5y ago
  Yes, but this is worse. Even if your filesystem does everything perfectly, these SSDs will still lose data.
- avar 5y ago
  The ext4 "issue" was in userspace. Certain software wasn't calling fsync() and had assumed the data was written anyway, because ext3 was more forgiving. IIRC the "solution" was to give ext4 the ext3 semantics, i.e. not insist that every broken userspace program needed to be fixed.
  - wmf 5y ago
    This is an old topic, but I disagree that any program that doesn't grind your disk into dust is broken. Consistency without durability is a useful choice.
    - avar 5y ago
      It is, but those programs are definitely "broken" if they expect any sort of durability guarantees, which was the problem with the programs that ran into problems with early versions of ext4. I.e. it's fine to open(), write() and close() without an appropriate fsync() (note that you'll also need to fsync() the relevant directory entri(es), which most people get wrong). It's not fine to do so and come complaining to the OS maintainers when the kernel lost your data, you didn't tell it you wanted the data flushed to disk, and you didn't wait around for that to happen until you reported success to the user. If you don't want to deal with any of that there's a widely supported way to get around it: Just mount your filesystems with the "sync" option, or run sync(1) after your program finishes, but before reporting success. You'll grind your "disks into dust", but you'll have data safety, sans any HW or kernel issues being discussed in this thread. But hey, consumer hardware sucks. News at 11! :) All that being said we live in the real world. IIRC the ext4 issue was that the delay for the implicit sync was changed from single-digit to double-digit seconds. So people were experiencing data loss due to API (mis)use that they were getting away with before. After the "do it like ext3" change they might still be, it's just that the implicit sync window was narrowed again. There's simply no way around not needing to care about any of this and still having some items from the "performance" column as well as the "data safety" column. All of modern computing is structured around these trade-offs, even SSD I/O is glacially slow compared to the CPU throughput. You need to juggle performance and data safety to get any reasonable I/O throughput, just like you can't have a reasonably performing OS without something like the OOM killer, "swap of death" or similar. You can always opt-out of it, but it means mounting your FS with sync, replacing your performant kernel with a glacially slow (but guaranteed to be predictable) RTOS etc.
lmilcin 5y ago
And that kids is how you bullshit your customers your product has more performance than in reality.
OnlyMortal 5y ago
Flush on Linux is not always a synchronous activity.
- marcan_42 5y ago
  It is if your disk works properly.
ed25519FUUU 5y ago
Would this affect filesystems like zfs? It seems a user space positive return syscall would mess all kinds of things up.
- davewritescode 5y ago
  At best ZFS could detect the failure but a file system can’t save you from drives lying about whether data is actually on the disk.
unnouinceput 5y ago
Quote: "Then I manually yanked the cable. Boom, data gone." He never said what cable he yanked. If he did it to internal one that goes from PSU to the drive then that's a very niche test, not relevant. But if he did it to the one from wall socket to the PC then yeah, then that's a good test. Regarding this, first of all Raymond Chen warned us more than a decade ago that vendors lie all the time about their hardware capabilities. He had one test with exactly this on flushing, where the HDD driver (written by the manufacturer of course) was always returning S_OK, regardless of whatever. Do note that this was in a time when HDD's were common, not SSD's. Secondly, I always buy a PSU that is more than double in power for the system. If, let's say, the system has a 500W requirement, my PSU for that system will be at least 1200W. This will definitely have big capacitors to keep your system alive 2 seconds after the power goes off. Those 2 seconds might seem small to us, but the drives, as much as lying assholes as they are to OS, will still correctly flush pending data. Never experienced data loss going this route.
allisdust 5y ago
Samsung has a hardware testing lab where all new storage products (SSDs/memory cards) are rigorously put to (automated) tests through a ridiculous number of reads, writes and power scenarios. The numbers are then averaged out and dialed down a bit to provide some buffer and finally advertised on the models. I'm not surprised that they maintain data integrity. They also own their entire stack (software and hardware) so there is less scope for a untested OEM bug to slip through.
yellowapple 5y ago
"Data loss occurred with a Korean and US brand, but it will turn into a whole "thing" if I name them so please forgive me." This does a disservice to those who might be running drives from those vendors with an expectation that they don't lose data post-flush. That said, this narrows one of the data losers down to Hynix. Curious about the other one, considering how many US-based SSD vendors there are.
- PhantomGremlin 5y ago
  That said, this narrows one of the data losers down to Hynix. Not really. Samsung builds a plethora of SSDs.
  - cerved 5y ago
    since they mentioned a Samsung working it's unlikely
  - yellowapple 5y ago
    Per the title, four vendors were tested. Samsung was already mentioned as a non-loser, so it can't be one of the two losers (or else the title would be wrong and the SSDs would be from 3 vendors at most).
    - PhantomGremlin 5y ago
      Yeah you got me. I didn't pay careful attention to the wording of the submitted title. I may have been confused because of the wording of the actual tweet: I tested a random selection of four NVMe SSDs from four vendors. The word "random" meant to me that Samsung drives could have been selected twice. But, yes, then there wouldn't be four distinct vendors. Unstated but implied by you is there are only two (major) Korean vendors to choose from. So if Samsung is a Korean winner then Hynix must be the Korean loser. Which is now clear to me. Is it possible there's a third (minor) Korean player? Could I possibly still have a chance? :)
      - yellowapple 5y ago
JoeAltmaier 5y ago
No surprise here. I've never encountered a redundancy feature in storage that worked. Power failure, drive controller failure, connection failure - and data is kaput. Regardless of the promised. How can it be so bleak? Can it be that nobody's data redundancy is real? Sure. If you don't test it, regularly, then by the hoary rules of computing it doesn't work.
- wmf 5y ago
  It would be... interesting to run Jepsen on some million-dollar SANs and discover that none of them pass.
  - JoeAltmaier 5y ago
    Don't know what a million-dollar SAN is. Everybody's data is worth a million to them. But any consumer-grade redundancy scheme (mirror, raid set, automatic backup) is likely useless.
pkaye 5y ago
I used to developed SSD firmware in the past and our team always used to make sure it would write the data and check the write status. We also used to used to analyze competitor products using bus analyzers and could determine some wouldn't do that. Also in the past many OS filesystems would ignore many errors we returned anyway. Edit: Here is an old paper on the subject of OS filesystem error handling. https://research.cs.wisc.edu/wind/Publications/iron-sosp05.pdf https://research.cs.wisc.edu/wind/Publications/iron-sosp05.p...
tpetry 5y ago
He is trsting consumer grade devices which don‘t have power loss protection by design. That is a „feature“ for enterprise devices so they can increase the price for datacenter usage. https://twitter.com/xenadu02/status/1496006341579751426?s=21 https://twitter.com/xenadu02/status/1496006341579751426?s=21
- willis936 5y ago
  He is trusting that drives are conformant to their specs. This is an issue of non-conformance that increases marketable performance at the cost of data security. PLP is great, but in lieu of that the drives should be honest about the state of writes. How can you trust your data will be there after an ACPI shut down?
- xenadu02 5y ago
  This has nothing to do with PLP. If the drive reports PLP then Flush is allowed to be a no-op because all acknowledged writes are durable by design - the OS need only wait for the data write and FS metadata writes to complete without needing to issue a special IO command. This is covered in 5.24.1.4 in the NVMe spec 2.0b
nightsd01 5y ago
In my opinion, as a consumer, this is up to you. If you need this, get a UPS battery backup (or a laptop which has its own battery). Or, you can get a super specialized SSD. Ultimately though, most consumer SSDs DON’T need this feature. And if they did include it by default, it would likely be environmentally questionable for a feature most people will never use (because most consumer SSDs these days go into laptops with their own batteries).
- iforgotpassword 5y ago
  You didn't understand the issue. It's not that these drives lose data with sudden power loss. It's that you tell the drive "please write all data that is currently in your write cache to persistent storage now" and then the drive says "ok I'm all done, your data is safe" and then when you cut power AFTER this, your data is still sometimes gone. This has nothing to do with any batteries, or complicated technology. It just means make your drive not lie.
gcoguiec 5y ago
It's worth noting PLP (Power Loss Protection) exists on Enterprise NVMEs to mitigate those issues.
_vvhw 5y ago
How do these NVMe SSDs fare when setting the FUA or Force Unit Access bit for write through on Linux (O_DIRECT | O_DSYNC) instead of F_FULLFSYNC on macOS? I imagine that different firmware machinery would be activated for FUA, and knowing whether FUA works properly would provide comfort to DB developers.
Cthulhu_ 5y ago
I'm confident this is a means to cheat performance benchmarks, because some of those will run tests using those flushes to try and get 'real' hardware performance instead of buffered/cached performance. I wonder if a small battery or capacitor on these devices would work to avoid data loss.
erosenbe0 5y ago
Most likely not valid. What was the external enclosure? You need an ATX 3.3V rail that has a 15ms hold up time and whatever other sequencing these devices were designed for. The buck converter in a USB enclosure isn't going to cut it for a valid test.
- undecisive 5y ago
  Could you explain the question? As far as I understand (which is little more than this twitter thread) the flush command should only return a success response once any data has been written to non-volatile storage. If the storage still requires power after that point to maintain the data, that storage area is volatile, no? So if the device has returned success (and I'm not going to claim that they've ensured that it was the device returning success and not the adapter, or that they even verified what the response was - those seem like valid questions) presumably the power wind-down should not be an issue? That said, I presumed by "disconnect the cable" the test involved some extension cable from the motherboard straight to drive to make it easier to disconnect - would that therefore make it a valid test of the NVMe?
  - erosenbe0 5y ago
    You understand incorrectly. Flush means data is in volatile DRAM in the device. That's how SSDs work. Extension cable from motherboard would certainly make it invalid. These devices are not hot swap and may expect power hold up and sequencing from the supply.
    - xenadu02 5y ago
      You're wrong. Please read the NVMe spec, section 7.1: > the Flush command shall commit data and metadata associated with the specified namespace(s) to non-volatile media It has nothing to do with DRAM. The spec is explicit about what Flush does. The drive is not allowed to ack the flush until the data and drive metadata are on durable storage. For a discussion on what happens when the drive has power loss protection, see 5.24.2.1 - in that case the write cache (aka DRAM) is considered non-volatile and Flush can be a no-op. None of the drives I tested fell into that category however.
markonen 5y ago
Enterprise drives with PLP (power loss protection) are surprisingly affordable. I would absolutely choose them for workstation / home use. The new Micron 7400 Pro M.2 960GB is $200, for example. Sure, the published IOPS figures are nothing to write home about, but drives like these 1) hit their numbers every time, in every condition, and 2) can just skip flushes altogether, making them much faster in uses where data integrity is important (and flushes would otherwise be issued).
eddybradson72 5y ago
More complex systems are liable to create more complex problems... I don't think you can get away from this - yes, can solve a problem, but if you model problems as entropy, increasing complexity increases entropy. It's like the messy room problem - you can clean your room (arguably high entropy state), but unless you are exceedingly careful doing so increases entropy. You merely move whatever mess to the garbage bin, expend extra heat, increase your consumption in your diet, possibly break your ankle, stress your muscles. https://drbrainpharma.com/product/buy-mdma-crystal-in-europe/ https://drbrainpharma.com/product/buy-mdma-crystal-in-europe...
vinay_ys 5y ago
Laptops, especially the likes of MacOS with T2 chip in which all I/o goes through T2 can do some clever things. It can essentially turn the underlying NVMe SSD into a battery backed storage. Even if the OS on main CPU crashes and dies, the T2 chip with its own independent OS can ensure SSD does a full flush before the battery runs out of power. Now, I don't know if Apple does this, but I sure hope they do. It would be great if they published the details so that even Linux on MacBook can do this well.
rob_c 5y ago
I'd be much more worried if it wasn't for one key piece of data 'MacOS' I'm not sure I trust the Xnu kernel-space to be reporting properly. That being said I think enough others said it, buy proper hardware if you don't trust that your family photos are on your corporate cloud account.
xenadu02 5y ago
I have an update for everyone: Models that lost writes in my test: SK Hynix Gold P31 2TB SHGP31-2000GM-2, FW 31060C20 Sabrent Rocket 512 (Phison PH-SBT-RKT-303 controller, no version or date codes listed) I've ordered more drives and will report back once I have results: Intel 670p Samsung 980 WD Black SN750 WD Green SN350 Kingston NV1 Seagate Firecuda 530 Crucial P2 Crucial P5 Plus These are just my results in my specific test configuration, done by me personally for fun in my own time. I may have made mistakes or the results might be invalid for reasons not yet know. No warranties expressed or implied.
- xenadu02 5y ago
  Crucial P5 Plus 1TB CT1000P5PSSD8, FW P7CR402: Pass Crucial P2 250GB CT250P2SSD8, FW P2CR046: Pass Kingston SNVS/250G, 012.A005: Pass Seagate Firecuda 530 PCIe Gen 4 1TB ZP1000GM30013, FW SU6SM001: Pass Intel 670p 1TB, SSDPEKNU010TZ, FW 002C: Pass Samsung 970 Evo Plus: MZ-V7S2T0, 2021.10: Pass Samsung 980 250GB MZ-V8V250, 2021/11/07: Pass WD Red: WDS100T1R0C-68BDK0, 04Sept2021: Pass WD Black SN750 1TB WDS100T1B0E, 09Jan2022: Pass WD Green SN350 240GB WDS240G20C, 02Aug2021: Pass Flush performance varies by 6x and is not necessarily correlated with overall perf or price. If you are doing lots of database writes or other workloads where durability matters don't just look at the random/sustained read/write performance! High flush perf: Crucial P5 Plus (fastest) and WD Red Despite being a relatively high end consumer drive the Seagate had really low flush performance. And despite being a budget drive the WD Green was really fast, almost as good as the WD Red in my test. The SK Hynix drive had fast flush perf at times, then at other times it would slow down. But it sometimes lost flushed data so it doesn't matter much.

797 points by ahachete 5y ago

nurettin 5y ago

I love the dedication on this guy, buying a dozen 2TB SSDs just to test them for FLUSH consistency.

eptcyka 5y ago

Hey, now I want/need to know if my SSD is flushing correctly or not.

mjw1007 5y ago

« Data loss occured with a Korean and US brand, but it will turn into a whole "thing" if I name them so please forgive me. »

LeifCarrotson 5y ago
> The models that never lost data: Samsung 970 EVO Pro 2TB and WD Red SN700 1TB. The others would probably be SK Hynix and Micron/Crucial, right? Curious why he's reluctant to name and shame. A drive not conforming to requirements and losing data is a legitimate problem that should be a "thing"!
- 420official 5y ago
  > Curious why he's reluctant to name and shame My sense is he wants to shame review sites for not paying attention to this rather than shame manufacturers directly at this point.
- nijave 5y ago
  Looks like he works at Apple. Maybe what he's testing is work related or covered by some sort of NDA (e.g. doesn't want to risk harming supplier relations for the brands misbehaving)
- alliao 5y ago
  I thought Crucial specifically designed some power loss protection as a differentiating selling point? Well at least that was the reason why I bought one back in M.2 days (gosh my PC is ancient...)
  - iforgotpassword 5y ago
    Back in M.2 days? I know I'm getting old, but what did miss?
    - cestith 5y ago
      M.2 SATA as opposed to NVMe perhaps?
    - alliao 5y ago
      it's the storage form factor of my current PC.. which is probably close to 10yrs old :P https://en.wikipedia.org/wiki/M.2 https://en.wikipedia.org/wiki/M.2
      - sascha_sl 5y ago
        It is still being used in very recent PCIe Gen 4 NVMe drives. Though you likely have a different key only taking SATA drives.
  - wtallis 5y ago
    I think the most they ever promised for some of their consumer drives was that a write interrupted by a power failure would not cause corruption of data that was already at rest. Such a failure can be possible when storing multiple bits per memory cell but programming the cells in multiple passes, especially if later passes are writing data from an entirely different host write command.
  - ryan29 5y ago
    It changed [1] from capacitors and "power loss protection" to something else described as "power loss immunity" with the MX500. I don't think I've ever seen it explained very well. > With the release of the MX500, Crucial has included a new replacement for the traditional power loss protection feature, power loss immunity. Instead of relying on a bank of capacitors for power loss protection, Crucial was able to work the new 3D TLC NAND and the code to allow for more efficient NAND programming so that the capacitors are no longer needed. That's just a regurgitated press release IMO. A lot of consumer drives also stopped reporting DRAT/RZAT [2] around the Crucial MX500, Samsung 850 timeframe. They swap internals as others in this thread have pointed out and the write endurance has dropped since reviewers stopped reporting on it. I have a Crucial MX500 in my system right now with 11% life remaining and only 37TBW even though it's advertised as having 180TBW of endurance. Edit: I actually found [3] an explanation of "power loss immunity". > The impact is still the same: you don't get the full protection that is standard for enterprise SSDs, but data that has already been written to the flash will not be corrupted if the drive loses power while writing a second pass of more data to the same cells. I always thought write operations on SSDs were more or less to write a new page or block or whatever the terminology is and to flag the old one(s) for garbage collection. I don't understand how it would be possible to lose the old data by doing that. Did they just invent a term that sounds like power loss protection, but doesn't actually do anything special? 1. https://www.thessdreview.com/our-reviews/crucial-mx500-ssd-review-1tb-the-best-value-in-sata/ https://www.thessdreview.com/our-reviews/crucial-mx500-ssd-r... 2. https://forums.anandtech.com/threads/looking-for-new-ssd-that-correctly-reports-drat-or-rzat-support.2556431/ https://forums.anandtech.com/threads/looking-for-new-ssd-tha... 3. https://www.anandtech.com/show/12165/the-crucial-mx500-1tb-ssd-review https://www.anandtech.com/show/12165/the-crucial-mx500-1tb-s...
- 0xcde4c3db 5y ago
  Crucial seems plausible, but there's a surprising number of US brands for NVMe SSDs. I was able to find: Crucial, Corsair, Seagate, Plextor, Sandisk, Intel, Kingston, Mushkin, PNY, Patriot Memory, and VisionTek.
  - LeifCarrotson 5y ago
    Micron/Crucial is the 3rd largest manufacturer of flash memory, most of the other brands in your list just make the PCB and whitelabel other flash chips and controllers (perhaps with some firmware customization, but they're usually not responsible for implementing features like FLUSH). Toshiba/Kioxia is another big one, but they're based in Japan. The US brand could be Intel instead of Crucial, I suppose.

hughrr 5y ago

The important quote: > The models that never lost data: Samsung 970 EVO Pro 2TB and WD Red SN700 1TB. I always buy the EVO Pro’s for external drives and use TB to NVMe bridges and they are pretty good.

Trellmor 5y ago
There is a 970 Evo, a 970 Pro and a 970 Evo Plus, but no 970 Evo Pro as far as I am aware. Would be interesting what model OP is actually talking about and if it is the same for other Samsung NMVe SSDs. I also prefer Samsung SSDs because they are reliably and they usually don't change parts to lower spec ones while keeping the same model number like some other vendors.
- hughrr 5y ago
  Sorry I should have said EVO plus there in my original post. I’ll leave the error in so your comment makes sense.
  - fuzzybear3965 5y ago
    How do you know the model? People are asking the same question in Twitter and OP doesn't seem to have supplied an answer, then.
  - zargon 5y ago
    The "EVO Pro" error was made by the OP. So it would be nice to know which drive OP actually tested.
    - hughrr 5y ago
      Indeed. I use the EVO Plus NVMe's though.
    - cgriswald 5y ago
      OP has since appended to his post: > Correction: “Plus” not “Pro”. Exact model and date codes: > Samsung 970 Evo Plus: MZ-V7S2T0, 2021.10 > WD Red: WDS100T1R0C-68BDK0, 04Sept2021

dboreham 5y ago

Surprised this is being rediscovered. Years ago only "enterprise" or "data center" SSDs had the supercap necessary to provide power for the controller to finish pending writes. Did we ever expect consumer SSDs to not lose writes on power fail?

evancox100 5y ago
It's not losing pending writes, it's the drive saying there are no pending writes, but losing them anyways. ie the drive is most likely lying
- monocasa 5y ago
  As I said in the recent Apple discussion, pretty much all drives are lying and have been for decades at this point. The good brands just spec out enough capacitance that you don't see the difference externally. https://news.ycombinator.com/item?id=30370551#30374585 https://news.ycombinator.com/item?id=30370551#30374585
  - brigade 5y ago
    So if SSDs rely solely on capacitors for data integrity and lie about flushes, what do they do on a flush that takes any amount of time? Are they just taking a speed hit for funsies? Heck, from this test, the magnitude of the speed hit isn't even correlated with whether they lose writes...
    - monocasa 5y ago
      At one point it was different barriers on the different submission queues inside the drive. Not externally visible queues, but between internal implementation layers. It's been a few years since I've checked up on this and it was for the most part pre SSDs though.
    - supermatt 5y ago
      Probably implementing a barrier for ordering io…

monocasa 5y ago

Relevant recent discussion about Apple's NVMe being very slow on FLUSH. Apple's custom NVMes are amazingly fast – if you don't care about data integrity https://news.ycombinator.com/item?id=30370551 https://news.ycombinator.com/item?id=30370551

hughrr 5y ago
Not really a problem when your computer has a large UPS built into it. Desktop macs, not so good. But really isn’t the point of a journaling file system to make sure it is consistent at one guaranteed point in time, not necessarily without incidental data loss.
- monocasa 5y ago
  Journaling file systems depend on commands like FLUSH to create the appropriate barriers to build their journal's semantics.
  - rowanG077 5y ago
    I don't have a clue how a journaling FS works. But any ordering should not be observable unless you have a power outage. Can you give an example how a journaling FS could observe something that should be observable?
    - b112 5y ago
      A crash, lockup, are the same as a power failure.
      - colanderman 5y ago
        No, during a crash or lockup, acknowledged writes are not lost. (Because the drive has acknowledged them, they are in the drive's internal queue and thus need no further action from the OS to be committed to durable storage.) Only power loss/power cycle causes this.

formerly_proven 5y ago

> I guess review sites don’t test this stuff. Most review sites don't really test much at all.

sokoloff 5y ago
Except for their affiliate links; they do test those.

post_break 5y ago

The problem is you can't trust a model number of SSD. They change controllers, chips, etc after the reviews are out and they can start skimping on components. https://www.tomshardware.com/news/adata-and-other-ssd-makers-swapping-parts https://www.tomshardware.com/news/adata-and-other-ssd-makers...

infogulch 5y ago
This needs to be cracked down on from a consumer protection lens. Like, any product revision that could potentially produce a different behavior must have a discernable revision number published as part of the model number.
- sokoloff 5y ago
  While I agree with the sentiment, even a firmware revision could cause a difference in behavior and it seems unreasonable to change the model number on every firmware release.
  - jay_kyburz 5y ago
    I don't think its unreasonable to just add a date next to the serial number that has when the firmware was released. It's 6 extra numbers
    - yjftsjthsd-h 5y ago
      But firmware can be updated after the unit was shipped; a date stamped on the hardware can't promise accuracy.
      - lillecarl 5y ago
        Is this the definition of whataboutism?
        yjftsjthsd-h 5y ago

supermatt 5y ago

So, seems those drives may have been ignoring the F_FULLFSYNC after all… https://news.ycombinator.com/item?id=30371857 https://news.ycombinator.com/item?id=30371857 The Samsung EVO drives are interesting because they have a few GB of SLC that they use as a secondary buffer before they reflush to the MLC.

verall 5y ago
> reflush to the MLC I'm nitpicking, but an EVO has TLC. Also an SLC write cache is the norm for any high performance consumer ssd, it's not just Samsung.
- fire 5y ago
  > I'm nitpicking, but an EVO has TLC. b...but the M in MLC stands for multi... as in multiple... right? checks Oh... uh; Apparently the obvious catch-all term MLC actually only refers to dual layer cells, but they didn't call it DLC, and now there's no catch-all term for > SLC. TIL.
  - marcan_42 5y ago
    The "L" stands for level, and that makes it even more wrong. MLC should have been been QLC or DBC, and TLC should have been OLC or TBC. It's two bits and four levels, or three bits and eight levels. The latest "QLC" flash has 16 levels (and is a massive step down in performance and reliability; it seems TLC is the sweet spot, at least right now, unless you really want the absolute cheapest, performance be damned). Interestingly enough, Apple has a patent for 8-bit cell flash (256 levels!), going full blown analog processing and error correction, but I don't think that ever became a product.
- fomine3 5y ago
  I believe now SLC cache is available on every TLC consumer SSDs, because it's slow without it.
  - wtallis 5y ago

iforgotpassword 5y ago

I think this is something LTT could handle with their new test lab. They already said they want to set new standards when it comes to hardware testing, so if they can hold up to what they promised and hire enough experts this should be a trivial thing to add to a test Parcours for disk drives.

balls187 5y ago
LTT's commentary makes it difficult to trust they are objective (even if they are). I loved seeing how giddy Linus got while testing Valve's Steamdeck, but when it comes to actual benchmarks and testing, I would appreciate if they dropped the entertainment aspect.
- KennyBlanken 5y ago
  GamersNexus seems to really be trying to work on improving and expanding their testing methodology as much as possible. I feel like they've successfully developed enough clout/trust that they have escaped the hell of having to pull punches in order to assure they get review samples. They eviscerated AMD for the 6500xt. They called out NZXT repeatedly for a case that was a fire hazard (!). Most recently they've been kicking Newegg in the teeth for trying to scam them over a damaged CPU. They've called out some really overpriced CPU coolers that underperform compared to $25-30 coolers. Etc. I bet they'd go for testing this sort of thing, if they haven't already started working on it already. They'd test it and then describe for what use cases it would be unlikely to be a problem vs what cases would be fine. For example, a game-file-only drive where if there's an error you can just verify the game files via the store application. Or a laptop that's not overclocked and only is used by someone to surf the web and maybe check their email.
  - nezgar 5y ago
    already already
  - throwaway2037 5y ago
    This is a good idea. You should send a suggestion to them!

timbit42 5y ago

What is an EVO Pro?

tromp 5y ago
See comment above; it's an EVO Plus
- timbit42 5y ago
  I don't have twitter. Someone should tell the tweeter to fix their tweet.
  - closewith 5y ago
    Tweets can't be edited, unfortunately.
    - timbit42 5y ago
      Since I don't have Twitter, I didn't know that. Yet people down voted me. Nice. Stay classy HN.
  - deleted 5y ago
    [deleted]

shadowgovt 5y ago

As a frame of reference, how much loss of FLUSH'd data should be expected on power loss for a semi-permanent storage device (including spinning-platter hard drives, if anyone still installs them in machines these days)? I'm far more used to the mainframe space where the rule is "Expect no storage reliability; redundancy and checksums or you didn't want that data anyway" and even long-term data is often just stored in RAM (and then periodically cold-storage'd to tape). I've lost sight of what expected practice is for desktop / laptop stuff anymore.

xenadu02 5y ago
The semantics of a FLUSH command (per NVMe spec) is that all previously sent write commands along with any internal metadata must be written to durable storage before returning success. Basically the drive is saying "yup, it's all on NAND - not in some internal buffer. You can power off or whatever you want, nothing will be lost". Some drives are doing work in response to that FLUSH but still lose data on power loss.
- benlwalker 5y ago
  A flush command only guarantees, upon completion, that all writes COMPLETED prior to submission of the flush are non-volatile. Not all previously sent writes. NVMe base specification 2.0b section 7.1. That's a very important distinction. You can't assume just because a write completed before the flush that it's actually durable. Only if it completed before you sent the flush. I'm not very confident that software is actually getting this right all that often, although it probably is in this fsync test.
  - wmf 5y ago
    Is there a separate barrier command so you don't have to track all the writes individually in software?
    - benlwalker 5y ago
      Nope. The reason this is so complex is that these devices are actually highly parallel machines with multiple queues accepting commands. It's quite difficult to even define "before" in terms of command sequence. For example, if you have a device with two hardware queues for submitting commands and a software thread for each, if you submit a flush on one queue, which commands on the other queue does it affect? Or what if the device issues a pci write to the completion entry that passes a flush command being submitted on the wire? I think the only interpretation that makes sense is from the perspective of a single software thread. If that particular thread has seen the completion via any mechanism and then that thread issues the flush, then you know the write is durable. Other than that, the device makes no promises.

gruez 5y ago

Correct me if I'm wrong, but if these drives are used for consumer applications, this behavior is probably not a big deal? If you made changes to a document, pressed control-S, and then 1 second later the power went out, then you might lose that last save. That'd suck, but you would have lost the data anyways if the power loss occurred 2s before, so it's not that bad. As long as other properties weren't violated (eg. ordering), your data should mostly be okay, aside from that 1s of data. It's a much bigger issue for enterprise applications, eg. a bank's mainframe responsible for processing transactions told a client that the transaction went through, but a power loss occurred and the transaction was lost.

__david__ 5y ago
> As long as other properties weren't violated (eg. ordering), your data should mostly be okay, aside from that 1s of data. That's the thing though—ordering isn't guaranteed as far as I remember. If you want ordering you do syncs/flushes, and if the drive isn't respecting those, then ordering is out of the window. That means FS corruption and such. Not good.
- gruez 5y ago
  The tweet only mentioned data loss when you yanked the power cable. That doesn't say anything about whether the ordering is preserved. It's possible to have a drive that lies about data written to persistent storage, but still keeps the writes in order.
colanderman 5y ago
> As long as other properties weren't violated (eg. ordering) That is primarily what fsync is used to ensure. (SCSI provides other means of ensuring ordering, but AFAIK they're not widely implemented.) EDIT: per your other reply, yes, it's possible the drives maintain ordering of FLUSHed writes, but not durability. I'm curious to see that tested as well. (Still an integrity issue for any system involving more than just one single drive though.)
- deleted 5y ago
  [deleted]

colanderman 5y ago

I'm curious whether the drives are at least maintaining write-after-ack ordering of FLUSHed writes in spite of a power failure. (I.e., whether the contents of the drives after power loss are nonetheless crash consistent.) That still isn't great, as it messes with consistency between systems, but at least a system solely dependent on that drive would not suffer loss of integrity.

jrockway 5y ago

I always liked the embedded system model where you get flash hardware that has two operations -- erase block and write block. GC, RAID, error correction, etc. are then handled at the application level. It was never clear to me that the current tradeoff with consumer-grade SSDs was right. On the one hand, things like the error correction, redundancy, and garbage collection don't require the attention from CPU (and more importantly, doesn't tie up any bus). On the other hand, the user has no control over what the software on the SSD's chip does. Clearly vendors and users are at odds with each other here; vendors want the best benchmarks (so you can sort by speed descending and pick the first one), but users want their files to exist after their power goes out. It would be nice if we could just buy dumb flash and let the application do whatever it wants (I guess that application would be your filesystem; but it could also be direct access for specialized use cases like databases). If you want maximum speed, adjust your settings for that. If you want maximum write durability, adjust your settings for that. People are always looking for that one size fits all use case, but it's hard here. Some people may be running cloud providers and already have software to store that block on 3 different continents. Some people may be an embedded system with a fixed disk image that changes once a year, with some temporary storage for logs. There probably isn't a single setting that gets optimal use out of the flash memory for both use cases. The cloud provider doesn't care if a block, flash chip, drive, server rack, availability zone, or continent goes away. The embedded system may be happy to lose logs in exchange for having enough writes left to install the next security update. It's all a mess, but the constraints have changed since we made the mess. You used to be happy to get 1/6th of a PCI Express lane for all your storage. Now processors directly expose 128 PCIe lanes and have a multitude of underused efficiency cores waiting to be used. Maybe we could do all the "smart" stuff in the OS and application code, and just attach commodity dumb flash chips to our computer.

bob1029 5y ago
I would absolutely love to have access to "dumb" flash from my application logic. I've got append only systems where I could be writing to disk many times faster if the controller weren't trying to be clever in anticipation of block updates.
- 5y ago

hardwaresofton 5y ago

I've actually run into some data loss running simple stuff like pgbench on Hetzner due to this -- I ended up just turning off write-back caching at the device level for all the machines in my cluster: https://vadosware.io/post/everything-ive-seen-on-optimizing-postgres-on-zfs-on-linux/#solution-force-proper-disk-syncing https://vadosware.io/post/everything-ive-seen-on-optimizing-... Granted I was doing something highly questionable (running postgres with fsync off on ZFS) It was very painful to get to the actual issue, but I'm glad I found out. I've always wondered if it was worth pursuing to start a simple data product with tests like these on various cloud providers to know where these corners are and what you're really getting for the money (or lack thereof). [EDIT] To save people some time (that post is long), the command to set the feature is the following: nvme set-feature -f 6 -v 0 /dev/nvme0n1 The docs for `nvme` (nvme-cli package, if you're Ubuntu based) can be pieced together across some man pages: https://man.archlinux.org/man/nvme.1 https://man.archlinux.org/man/nvme.1 https://man.archlinux.org/man/nvme-set-feature.1.en https://man.archlinux.org/man/nvme-set-feature.1.en It's a bit hard to find all the NVMe features but 6 is the one for controlling write-back caching. https://unix.stackexchange.com/questions/472211/list-features-of-nvme-drive-like-hdparm-i-for-non-nvme https://unix.stackexchange.com/questions/472211/list-feature...

hrgiger 5y ago
I dont have ide in this machine but I found this in the source code [1] probably pointing to [2] and thanks for the tip! [1]https://github.com/linux-nvme/nvme-cli/blob/master/nvme-print.c#L6272 https://github.com/linux-nvme/nvme-cli/blob/master/nvme-prin... [2]https://github.com/linux-nvme/libnvme/blob/master/src/nvme/types.h#L6155 https://github.com/linux-nvme/libnvme/blob/master/src/nvme/t...
- hardwaresofton 5y ago
  ah thanks this is perfect, saved those links!
  - wtallis 5y ago
    Also: https://nvmexpress.org/developers/nvme-specification/ https://nvmexpress.org/developers/nvme-specification/ Unlike eg. ATA and SCSI, the NVMe specs are freely available to the public. They're a little more complicated to read now that the spec has been split into a few modules, but finding the descriptions of all the optional features isn't too hard.

nyc_pizzadev 5y ago

I’m actually interested in testing this scenario, a drive getting power loss. Is there a thing which will cut power to a server device on command? Or do you just pull the drive out of its bay?

klysm 5y ago
I don’t know of a way on most systems to cut the power directly. It seems fairly straightforward to wire up a relay though.
toast0 5y ago
If your server has IPMI or similar, you can use that to cut power (not to the BMC though), otherwise, network PDUs or many UPSes or consumer level networked outlets or something fun with a relay (be careful with mains voltages). Pulling the drive is also worth testing though; you might get different results. Requires more human involvement though.

tr33house 5y ago

I'm a systems engineer but I've never done low level optimizations on drives. How does one even go about even testing something like this? It sounds like something cool that I'd like to be able to do

klysm 5y ago
Write and flush and unplug the cable!
- StillBored 5y ago
  which is more difficult (and sometimes slower) than STONITH style devices which just kill power to the entire machine. The latter allow you to program the whole thing and run test cycle after test cycle where the device kills itself the moment it gets a successful flush.
xenadu02 5y ago
My script repeatedly writes a counter value "lines=$counter" to a file, then calls fcntl() with F_FULLFSYNC against that file descriptor which on macOS ends up doing an NVMe FLUSH to the drive (after sending in-memory buffers and filesystem metadata to the drive). Once those calls succeed it increments the counter and tries again. As soon as the write() or fcntl() fail it prints the last successfully written counter value which can be checked against the contents of the file. Remember: the semantics of the API and the NVMe spec require that a successful return from fcntl(fd, F_FULLFSYNC) on macOS require that data is durable at that point no matter what filesystem metadata OR drive internal metadata is needed to make that happen. In my test while the script is looping doing that as fast as possible I yank the TB cable. The enclosure is bus powered so it is an unceremonious disconnect and power off. Two of the tested drives always matched up: whatever the counter was when write()+fcntl() succeeded is what I read back from the file. Two of the drives sometimes failed by reporting counter values < the most recent successful value, meaning the write()+ fcntl() reported success but upon remount the data was gone. Anytime a drive reported a counter value +1 from what was expected I still counted at that as success... after all there's a race window where the fcntl() has succeeded but the kernel hasn't gotten the ACK yet. If disconnect happens at that moment fcntl() will report failure even though it succeeded. No data is lost so that's not a "real" error.

throwaway984393 5y ago

Aren't there filesystem options that affect this? Wasn't there a whole controversy over ext4 and another filesystem not committing changes even after flush (under specific options/scenarios)?

wmf 5y ago
Yes, but this is worse. Even if your filesystem does everything perfectly, these SSDs will still lose data.
avar 5y ago
The ext4 "issue" was in userspace. Certain software wasn't calling fsync() and had assumed the data was written anyway, because ext3 was more forgiving. IIRC the "solution" was to give ext4 the ext3 semantics, i.e. not insist that every broken userspace program needed to be fixed.
- wmf 5y ago
  This is an old topic, but I disagree that any program that doesn't grind your disk into dust is broken. Consistency without durability is a useful choice.
  - avar 5y ago
    It is, but those programs are definitely "broken" if they expect any sort of durability guarantees, which was the problem with the programs that ran into problems with early versions of ext4. I.e. it's fine to open(), write() and close() without an appropriate fsync() (note that you'll also need to fsync() the relevant directory entri(es), which most people get wrong). It's not fine to do so and come complaining to the OS maintainers when the kernel lost your data, you didn't tell it you wanted the data flushed to disk, and you didn't wait around for that to happen until you reported success to the user. If you don't want to deal with any of that there's a widely supported way to get around it: Just mount your filesystems with the "sync" option, or run sync(1) after your program finishes, but before reporting success. You'll grind your "disks into dust", but you'll have data safety, sans any HW or kernel issues being discussed in this thread. But hey, consumer hardware sucks. News at 11! :) All that being said we live in the real world. IIRC the ext4 issue was that the delay for the implicit sync was changed from single-digit to double-digit seconds. So people were experiencing data loss due to API (mis)use that they were getting away with before. After the "do it like ext3" change they might still be, it's just that the implicit sync window was narrowed again. There's simply no way around not needing to care about any of this and still having some items from the "performance" column as well as the "data safety" column. All of modern computing is structured around these trade-offs, even SSD I/O is glacially slow compared to the CPU throughput. You need to juggle performance and data safety to get any reasonable I/O throughput, just like you can't have a reasonably performing OS without something like the OOM killer, "swap of death" or similar. You can always opt-out of it, but it means mounting your FS with sync, replacing your performant kernel with a glacially slow (but guaranteed to be predictable) RTOS etc.

lmilcin 5y ago

And that kids is how you bullshit your customers your product has more performance than in reality.

OnlyMortal 5y ago

Flush on Linux is not always a synchronous activity.

marcan_42 5y ago
It is if your disk works properly.

ed25519FUUU 5y ago

Would this affect filesystems like zfs? It seems a user space positive return syscall would mess all kinds of things up.

davewritescode 5y ago
At best ZFS could detect the failure but a file system can’t save you from drives lying about whether data is actually on the disk.

unnouinceput 5y ago

Quote: "Then I manually yanked the cable. Boom, data gone." He never said what cable he yanked. If he did it to internal one that goes from PSU to the drive then that's a very niche test, not relevant. But if he did it to the one from wall socket to the PC then yeah, then that's a good test. Regarding this, first of all Raymond Chen warned us more than a decade ago that vendors lie all the time about their hardware capabilities. He had one test with exactly this on flushing, where the HDD driver (written by the manufacturer of course) was always returning S_OK, regardless of whatever. Do note that this was in a time when HDD's were common, not SSD's. Secondly, I always buy a PSU that is more than double in power for the system. If, let's say, the system has a 500W requirement, my PSU for that system will be at least 1200W. This will definitely have big capacitors to keep your system alive 2 seconds after the power goes off. Those 2 seconds might seem small to us, but the drives, as much as lying assholes as they are to OS, will still correctly flush pending data. Never experienced data loss going this route.

allisdust 5y ago

Samsung has a hardware testing lab where all new storage products (SSDs/memory cards) are rigorously put to (automated) tests through a ridiculous number of reads, writes and power scenarios. The numbers are then averaged out and dialed down a bit to provide some buffer and finally advertised on the models. I'm not surprised that they maintain data integrity. They also own their entire stack (software and hardware) so there is less scope for a untested OEM bug to slip through.

yellowapple 5y ago

"Data loss occurred with a Korean and US brand, but it will turn into a whole "thing" if I name them so please forgive me." This does a disservice to those who might be running drives from those vendors with an expectation that they don't lose data post-flush. That said, this narrows one of the data losers down to Hynix. Curious about the other one, considering how many US-based SSD vendors there are.

PhantomGremlin 5y ago
That said, this narrows one of the data losers down to Hynix. Not really. Samsung builds a plethora of SSDs.
- cerved 5y ago
  since they mentioned a Samsung working it's unlikely
- yellowapple 5y ago
  Per the title, four vendors were tested. Samsung was already mentioned as a non-loser, so it can't be one of the two losers (or else the title would be wrong and the SSDs would be from 3 vendors at most).
  - PhantomGremlin 5y ago
    Yeah you got me. I didn't pay careful attention to the wording of the submitted title. I may have been confused because of the wording of the actual tweet: I tested a random selection of four NVMe SSDs from four vendors. The word "random" meant to me that Samsung drives could have been selected twice. But, yes, then there wouldn't be four distinct vendors. Unstated but implied by you is there are only two (major) Korean vendors to choose from. So if Samsung is a Korean winner then Hynix must be the Korean loser. Which is now clear to me. Is it possible there's a third (minor) Korean player? Could I possibly still have a chance? :)
    - yellowapple 5y ago

JoeAltmaier 5y ago

No surprise here. I've never encountered a redundancy feature in storage that worked. Power failure, drive controller failure, connection failure - and data is kaput. Regardless of the promised. How can it be so bleak? Can it be that nobody's data redundancy is real? Sure. If you don't test it, regularly, then by the hoary rules of computing it doesn't work.

wmf 5y ago
It would be... interesting to run Jepsen on some million-dollar SANs and discover that none of them pass.
- JoeAltmaier 5y ago
  Don't know what a million-dollar SAN is. Everybody's data is worth a million to them. But any consumer-grade redundancy scheme (mirror, raid set, automatic backup) is likely useless.

pkaye 5y ago

I used to developed SSD firmware in the past and our team always used to make sure it would write the data and check the write status. We also used to used to analyze competitor products using bus analyzers and could determine some wouldn't do that. Also in the past many OS filesystems would ignore many errors we returned anyway. Edit: Here is an old paper on the subject of OS filesystem error handling. https://research.cs.wisc.edu/wind/Publications/iron-sosp05.pdf https://research.cs.wisc.edu/wind/Publications/iron-sosp05.p...

tpetry 5y ago

He is trsting consumer grade devices which don‘t have power loss protection by design. That is a „feature“ for enterprise devices so they can increase the price for datacenter usage. https://twitter.com/xenadu02/status/1496006341579751426?s=21 https://twitter.com/xenadu02/status/1496006341579751426?s=21

willis936 5y ago
He is trusting that drives are conformant to their specs. This is an issue of non-conformance that increases marketable performance at the cost of data security. PLP is great, but in lieu of that the drives should be honest about the state of writes. How can you trust your data will be there after an ACPI shut down?
xenadu02 5y ago
This has nothing to do with PLP. If the drive reports PLP then Flush is allowed to be a no-op because all acknowledged writes are durable by design - the OS need only wait for the data write and FS metadata writes to complete without needing to issue a special IO command. This is covered in 5.24.1.4 in the NVMe spec 2.0b

nightsd01 5y ago

In my opinion, as a consumer, this is up to you. If you need this, get a UPS battery backup (or a laptop which has its own battery). Or, you can get a super specialized SSD. Ultimately though, most consumer SSDs DON’T need this feature. And if they did include it by default, it would likely be environmentally questionable for a feature most people will never use (because most consumer SSDs these days go into laptops with their own batteries).

iforgotpassword 5y ago
You didn't understand the issue. It's not that these drives lose data with sudden power loss. It's that you tell the drive "please write all data that is currently in your write cache to persistent storage now" and then the drive says "ok I'm all done, your data is safe" and then when you cut power AFTER this, your data is still sometimes gone. This has nothing to do with any batteries, or complicated technology. It just means make your drive not lie.

gcoguiec 5y ago

It's worth noting PLP (Power Loss Protection) exists on Enterprise NVMEs to mitigate those issues.

_vvhw 5y ago

How do these NVMe SSDs fare when setting the FUA or Force Unit Access bit for write through on Linux (O_DIRECT | O_DSYNC) instead of F_FULLFSYNC on macOS? I imagine that different firmware machinery would be activated for FUA, and knowing whether FUA works properly would provide comfort to DB developers.

Cthulhu_ 5y ago

I'm confident this is a means to cheat performance benchmarks, because some of those will run tests using those flushes to try and get 'real' hardware performance instead of buffered/cached performance. I wonder if a small battery or capacitor on these devices would work to avoid data loss.

erosenbe0 5y ago

Most likely not valid. What was the external enclosure? You need an ATX 3.3V rail that has a 15ms hold up time and whatever other sequencing these devices were designed for. The buck converter in a USB enclosure isn't going to cut it for a valid test.

undecisive 5y ago
Could you explain the question? As far as I understand (which is little more than this twitter thread) the flush command should only return a success response once any data has been written to non-volatile storage. If the storage still requires power after that point to maintain the data, that storage area is volatile, no? So if the device has returned success (and I'm not going to claim that they've ensured that it was the device returning success and not the adapter, or that they even verified what the response was - those seem like valid questions) presumably the power wind-down should not be an issue? That said, I presumed by "disconnect the cable" the test involved some extension cable from the motherboard straight to drive to make it easier to disconnect - would that therefore make it a valid test of the NVMe?
- erosenbe0 5y ago
  You understand incorrectly. Flush means data is in volatile DRAM in the device. That's how SSDs work. Extension cable from motherboard would certainly make it invalid. These devices are not hot swap and may expect power hold up and sequencing from the supply.
  - xenadu02 5y ago
    You're wrong. Please read the NVMe spec, section 7.1: > the Flush command shall commit data and metadata associated with the specified namespace(s) to non-volatile media It has nothing to do with DRAM. The spec is explicit about what Flush does. The drive is not allowed to ack the flush until the data and drive metadata are on durable storage. For a discussion on what happens when the drive has power loss protection, see 5.24.2.1 - in that case the write cache (aka DRAM) is considered non-volatile and Flush can be a no-op. None of the drives I tested fell into that category however.

markonen 5y ago

Enterprise drives with PLP (power loss protection) are surprisingly affordable. I would absolutely choose them for workstation / home use. The new Micron 7400 Pro M.2 960GB is $200, for example. Sure, the published IOPS figures are nothing to write home about, but drives like these 1) hit their numbers every time, in every condition, and 2) can just skip flushes altogether, making them much faster in uses where data integrity is important (and flushes would otherwise be issued).

eddybradson72 5y ago

More complex systems are liable to create more complex problems... I don't think you can get away from this - yes, can solve a problem, but if you model problems as entropy, increasing complexity increases entropy. It's like the messy room problem - you can clean your room (arguably high entropy state), but unless you are exceedingly careful doing so increases entropy. You merely move whatever mess to the garbage bin, expend extra heat, increase your consumption in your diet, possibly break your ankle, stress your muscles. https://drbrainpharma.com/product/buy-mdma-crystal-in-europe/ https://drbrainpharma.com/product/buy-mdma-crystal-in-europe...

vinay_ys 5y ago

Laptops, especially the likes of MacOS with T2 chip in which all I/o goes through T2 can do some clever things. It can essentially turn the underlying NVMe SSD into a battery backed storage. Even if the OS on main CPU crashes and dies, the T2 chip with its own independent OS can ensure SSD does a full flush before the battery runs out of power. Now, I don't know if Apple does this, but I sure hope they do. It would be great if they published the details so that even Linux on MacBook can do this well.

rob_c 5y ago

I'd be much more worried if it wasn't for one key piece of data 'MacOS' I'm not sure I trust the Xnu kernel-space to be reporting properly. That being said I think enough others said it, buy proper hardware if you don't trust that your family photos are on your corporate cloud account.

xenadu02 5y ago

I have an update for everyone: Models that lost writes in my test: SK Hynix Gold P31 2TB SHGP31-2000GM-2, FW 31060C20 Sabrent Rocket 512 (Phison PH-SBT-RKT-303 controller, no version or date codes listed) I've ordered more drives and will report back once I have results: Intel 670p Samsung 980 WD Black SN750 WD Green SN350 Kingston NV1 Seagate Firecuda 530 Crucial P2 Crucial P5 Plus These are just my results in my specific test configuration, done by me personally for fun in my own time. I may have made mistakes or the results might be invalid for reasons not yet know. No warranties expressed or implied.

xenadu02 5y ago
Crucial P5 Plus 1TB CT1000P5PSSD8, FW P7CR402: Pass Crucial P2 250GB CT250P2SSD8, FW P2CR046: Pass Kingston SNVS/250G, 012.A005: Pass Seagate Firecuda 530 PCIe Gen 4 1TB ZP1000GM30013, FW SU6SM001: Pass Intel 670p 1TB, SSDPEKNU010TZ, FW 002C: Pass Samsung 970 Evo Plus: MZ-V7S2T0, 2021.10: Pass Samsung 980 250GB MZ-V8V250, 2021/11/07: Pass WD Red: WDS100T1R0C-68BDK0, 04Sept2021: Pass WD Black SN750 1TB WDS100T1B0E, 09Jan2022: Pass WD Green SN350 240GB WDS240G20C, 02Aug2021: Pass Flush performance varies by 6x and is not necessarily correlated with overall perf or price. If you are doing lots of database writes or other workloads where durability matters don't just look at the random/sustained read/write performance! High flush perf: Crucial P5 Plus (fastest) and WD Red Despite being a relatively high end consumer drive the Seagate had really low flush performance. And despite being a budget drive the WD Green was really fast, almost as good as the WD Red in my test. The SK Hynix drive had fast flush perf at times, then at other times it would slow down. But it sometimes lost flushed data so it doesn't matter much.