28 ms·
Apple's custom NVMes are amazingly fast – if you don't care about data integrity
- steeve 5y agoAs the article mentions, on laptops, this is pretty clever. On desktops though... Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss? I think I heard some enterprise motherboards/controllers/computers did just that, given the upside in normal operation.
- throw3838 5y agoHow exactly this is clever? Maybe on some toy, not on workstation!
- marcan_42 5y agoThese machines are actually low-power enough that you could implement a last-gasp flush mechanism. The Mac Mini already survives 1-2 seconds without AC power (at least if idle). You could plausibly detect AC power being yanked and immediately power down all downstream USB/TB3 devices and the display (on iMacs), freeze all CPUs into idle, and have plenty enough reservoir cap to let NVMe issue a flush. But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown. This does require a last-gasp indication from the PSU to the rest of the system, so if they don't have that, it's not something they could add in a firmware update.
- fauigerzigerk 5y ago>But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown. That's unfortunate. My Mac Mini crashes every other night during sleep. I guess I'm going to have to shut it down to avoid any data corruption.
- marcan_42 5y agoIt should be flushing the drive cache on sleep. This is mostly an issue for sudden AC power loss.
- fauigerzigerk 5y agoAh, thanks! That's good to know.
- sgt 5y agoWhy does it crash? Mac Minis are some of the most reliable machines on the market, in my experience. Maybe a faulty unit?
- salamandersauce 5y agoShitty software? My 2018 Mac Mini would crash every single time going to sleep on the last version of Mojave. I'm not alone in this as there's huge threads on MacRumors and Apple's support forum about it. Apple's "fix" was to just update to Catalina which indeed fixes it but doesn't really help if you want to run 32 bit software. Wouldn't surprise me if they did something similar again.
- fauigerzigerk 5y agoIt has started crashing the night after I upgraded to macOS 12.2.0. The latest update to 12.2.1 hasn't fixed it. I'm pretty sure it's not hardware related as I had no issues before the OS upgrade. Edit: Here's the first line of the crash log (which I'm sending to Apple every time): panic(cpu 3 caller 0xfffffe0023be8be0): [data.kalloc.16]: element modified after free (off:0, val:0x0000000000000030, sz:16, ptr:0xfffffe2fffc9bb00) Looks like a use after free bug.
- yborg 5y agoThis is why I'm still on Catalina. I used to be in the "trail by one point release" mode on macOS, now I'm in the "trail by 2 major releases" camp.
- rubatuga 5y agoI mean the ATX standard has this signal built in, so Apple could just copy it: https://en.wikipedia.org/wiki/Power_good_signal https://en.wikipedia.org/wiki/Power_good_signal
- dale_glass 5y ago> The ATX specification requires that the power-good signal ("PWR_OK") go high no sooner than 100 ms after the power rails have stabilized, and remain high for 16 ms after loss of AC power, and fall (to less than 0.4 V) at least 1 ms before the power rails fall out of specification (to 95% of their nominal value). I don't think that quite works for the purpose. What you'd want is a second signal that goes low as soon as possible after loss of AC power. My reading here is that PWR_OK going low is an indication that the PSU has stopped providing good power, and the CPU must shut down immediately, or it might miscompute something due to low voltage. At this point you absolutely don't want to do any last-minute writing, you'd be risking corruption. What you need here is an early warning signal that you can react to while the PSU is still coasting on the internal capacitors.
- fps-hero 5y agoPower OK signals are used to prevent latch ups in silicon due to power glitches. The signals will route to power management ICs to ensure a full reset with proper bringing up of the power rails on any power glitch.
- jaidan 5y ago16ms is just longer than one AC cycle at 60Hz and less than one AC cycle at 50Hz. I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss. Of course there is industrial hardware that can detect quicker than this but it would add a LOT of cost for arguably little gain, or something that could be solved in another manner.
- dale_glass 5y ago> I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss. Doubtful. 16ms is an awfully long time these days. There's no reason why you couldn't detect power loss much sooner, given a good input signal. The concept also gets used quite often, in the form of SSRs with zero crossing detection. Those are used for dimmers. The reason is likely related to the awful waveforms produced by some UPSes and inverters: https://www.christidis.info/images/blog/scope_20.png https://www.christidis.info/images/blog/scope_20.png Unlike a nice sine wave, those spend a good while hovering near zero volts, so the PSU has to be able to tolerate that. Detecting loss of power sooner in this case isn't a question of cost, it's a question of that you don't have a good signal to do the detection on to start with.
- FooBarWidget 5y agoEven on laptops I feel uncomfortable. My macOS freezes or kernel panics on me from time to time.
- marcan_42 5y agoI believe the NVMe driver has a kernel panic hook; I would hope it is used to issue a flush. OTOH, if you have watchdog timeouts (I've seen this from bad drivers), those would certainly not give the kernel a chance to do that.
- ladyanita22 5y agoWhat would you implement in Asahi? Would you follow Apple's approach and defer flushes, implementing a kernel panic hook and having some kind of F_FULLFSYNC or just keep Linux' current implementation?
- marcan_42 5y agoWe're probably going to have a knob to defer flushes (but still do them, unlike Apple, after a max timeout) that will be on by default on laptops, and make sure panics flush the cache if we can. Also apparently we need to do something for the power button too, as I just tested how macOS handles that. There is a warning before the system shuts down but we need to listen to it. Same with critical battery states.
- ladyanita22 5y agoThen I misunderstood. Do you mean that Apple doesn't implement ANY timeout? So they only flush when the cache is full or when a shutdown routine has started?
- marcan_42 5y agoThey flush the cache when something requests the cache be flushed; I don't know if there is a timeout, because presumably it's not difficult for some random process to issue a FULLFSYNC and flush everything prior as a side-effect (the flush is global). But I've seen at least 5-10 seconds of data loss from drive cache loss on the Mac Mini, so if they do do deferred flushes the timeout is longer than that.
- dzhiurgis 5y agoDoes disk gets flushed in case of kernel panic?
- coldtea 5y ago>Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss? Or just add a UPS?
- ladyanita22 5y agoThis is such an ugly hack...
- headmelted 5y agoHonestly I don’t know. The order-of-magnitude performance difference in deferring the flush feels worth it to me if the risk is mitigated to sudden power loss. I would think when the last of Apple’s hardware moves to ARM they’ll ensure there’s enough onboard battery to ensure the flushes happen reliably across form factors even if there’s a power cut. If anything, now that the reason for the performance difference has been identified, I’d hope to see numbers for Linux and Windows storage access come up to par with these numbers as they go down this road too (e.g. via the NVME flush toggle mentioned in the article).
- rfoo 5y agoYeah. If the same thing happens to a brand-less garbage SSD you purchased from Aliexpress it's clearly cheat and plainly malicious and incompetence, but the Apple tag certainly made us believe there is a second reason. Trading correctness for performance without shouting at the users "YOUR DATA IS NOT SAFE WHEN YOU DO THIS" multiple times a day in a storage is benchmark-snake-oil. Period.
- coldtea 5y agoThe same thing happens in POSIX, most unices, including Linux itself until recently. So there's that. It's rather Apple's slow drive firmware checks though, that is problematic.
- headmelted 5y agoI’m not arguing for or against, I’m just pointing out that trading the possibility of data loss in the few seconds after a power cut for a difference of this magnitude actually makes sense in a lot of use cases. My point above was that the same “cheat” (to use your word) could be applied to the unbranded SSD too, with similar performance gains. I’m not giving Apple a pass for low flush performance, I’m saying there’s nothing I can see here that’s uniquely available to Apple that would prevent others from deferring flushes in the same way for similar performance gains - which would make sense in many cases.
- vgel 5y agoWhat confuses me about this is why are they so slow with F_FULLSYNC? Since that's the equivalent of what non-Apple NVMEs do under, say, Linux, and they manage to be much faster.
- marcan_42 5y agoThe OS does not matter; it's strictly about the drive. macOS on a non-Apple SSD should be equally fast with F_FULLSYNC. Indeed, I would very much like to know what on earth the ANS firmware is doing on flushes to make them so hideously slow. We do have the firmware blobs (for both the NVMe/ANS side and the downstream S5C NAND device controllers), so if someone is bored enough they could try to reverse engineer it... it also seems there's a bunch of debug mode options, so maybe we can even get some logs at some point.
- supermatt 5y agoDrives are known to ignore that hint... Thats why you should use vendor approved hardware if such things matter to you.
- formerly_proven 5y agoDeferring flushes on the NVMe level could also corrupt a journaling FS itself, not just the contents of files written with proper fsync incantations.
- marcan_42 5y agoIndeed, though that is somewhat rare. For our distro, I would opt to enable it by default on laptops (which is quite safe) and disable it on desktops.
- zeotroph 5y agoAPFS at least has metadata checksums to prevent that. However it does not do data checksums (weird decision...), despite being a CoW fs with snapshotting, similar to ZFS and btrfs.
- vbezhenar 5y agoThey rely on hardware storing checksums and on protocol using checksums to prevent data corruption on all levels.
- formerly_proven 5y agoEveryone else does that as well and it's not a substitute for end-to-end data integrity.
- etaioinshrdlu 5y agoAs a laptop user I would probably opt to make the same choice as Apple here. I like the idea mentioned to allow a tunable parameter to only allow ever losing 1 second of data. Although, I also have the seemingly rare opinion here that ECC ram doesn't really matter on a laptop or desktop.
- rowanG077 5y agoYou think it's oke that restarting your PC leads to data loss or corruption? That's basically a product killer for me. I reboot my laptop everyday.
- marcan_42 5y agoYou presumably don't reboot your laptop by connecting a USB-PD gadget that issues a hard reset. A normal OS reboot is fine, that will flush the cache. The most common situation where this would affect laptops, in my experience so far, would be a broken driver causing a kernel lockup (not a panic) which triggers a watchdog reboot. That situation wouldn't allow for an NVMe flush.
- rowanG077 5y agoDepends what exactly is a hard reboot. I don't reboot my laptop by issueing USB-PD command. But I do by holding the power button.
- Cthulhu_ 5y agoSee, that's a forced shutdown, a last resort measure; it's using a sledgehammer to tap in a nail. You shouldn't do that as a habit, even if this particular optimization issue wasn't a thing. I mean I grew up diligently turning off my PC by parking the disk and using the various operating system level shutdown procedures. Nowadays I smack the off button, but that still just triggers the OS shutdown procedure. I don't turn my Mac off as a rule, its sleep mode actually works. ish.
- Thomaschaaf 5y ago
- yumraj 5y agoThis sounds like laptops are fine, but iMacs and Minis are effed. Curious, what's the real world risk of full OS level corruption and not just data loss?
- marcan_42 5y agoGood question. I just started up a loop doing USB-PD hard reboots on my MBA every 18 seconds (that's about one second into the desktop with autologin on, where it should still be doing stuff in the background). Let's see if it eats itself.
- bestouff 5y agoFamous last words
- marcan_42 5y agoThis is just a test machine I also sometimes use as a dumb terminal around the house; I'm not going to cry if the OS eats itself :P
- masklinn 5y agoHopefully the ssd doesn’t either though, bricking it would be hilarious but not ideal.
- marcan_42 5y agoFinding out if a DFU restore can recover a corrupted SSD storage would be an interesting test in and of itself! But to be honest, if I end up really bricking a machine for science, that will be worth it for the information it gives us. Obviously I'm not trying to destroy my hardware, but I'm very grateful that I can afford it if it happens thanks to all the support I'm getting from folks for the project.
- joshspankit 5y ago
- rowanG077 5y agoI think most SSD have dram cache on board. Could they design issue here be Apple doesn't have that and instead using system RAM as SSD dram cache?
- marcan_42 5y agoThey do use system RAM as cache, but that has no effect on performance. If anything it should be way faster than the puny RAM cache chips on typical SSDs. It doesn't explain the slow flush perf.
- rowanG077 5y agoAfaik if you fsync an SSD with dram cache it won't hit the NAND cells. Those SSD do have some way to flush before they lose juice though. This was for enterprise SSD though a few years back.
- marcan_42 5y agoThat is only for enterprise SSDs. Consumer SSDs do not have capacitor banks to do a full DRAM flush. Some have capacitor banks to ensure data at rest integrity and some use another mechanism for that, but I'm not aware of any that will guarantee full cache stability.
- rossmohax 5y agoDocs [1] suggests that even F_FULLFSYNC might not be enough. Quote: > Note that F_FULLFSYNC represents a best-effort guarantee that iOS writes data to the disk, but data can still be lost in the case of sudden power loss. [1] https://developer.apple.com/documentation/xcode/reducing-disk-writes/ https://developer.apple.com/documentation/xcode/reducing-dis...
- ayende 5y agoWhen building databases, we care about durability, so database authors are usually well aware that you _have_ to use `F_FULLSYNC` for safety. The fact that `F_FULLSYNC` isn't safe means that you cannot write a transactional database on Mac, it is also a surprise to me. Note that the man page for `F_FULLSYNC` itself doesn't mention that it is not reliable: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html https://developer.apple.com/library/archive/documentation/Sy... Having a separate syscall is annoying, but workable. Having a scenario where we call flush and cannot ensure that this is the case is BAD. Note that handling flush failures is expected, but all databases require that flushing successfully will make the data durable. Without that, there are no way to ensure durable writes and you might get data loss or data corruption.
- stefan_ 5y agoI checked a few and they seem to do F_FULLFSYNC (sic), except MySQL, they deleted it to make it run faster: https://github.com/mysql/mysql-server/commit/3cb16e9c3879d1790159d2856cb7aa0bd3d201fd https://github.com/mysql/mysql-server/commit/3cb16e9c3879d17...
- nickpeterson 5y agoOh MySQL, I’m a world turned upside down you are my North Star.
- rat9988 5y ago"the possible durability gain is slim to none. This also makes OS X behave similar to other platforms." You didn't report the full reasoning.
- rossmohax 5y agoIn Apple defense, sloppy fsync behaviour is clearly documented: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html https://developer.apple.com/library/archive/documentation/Sy...
- lbriner 5y agoThat's not defence. It fails the principle of least-surprise. If everyone's experience is that fsync is flushing then why would somebody think to look up the docs for Mac in case they do it differently?
- cutthegrass2 5y agoWonder whether the AWS Mac EC2 instance types are affected too, anyone know?
- aaaaaaaaaaab 5y agoDo you really think those machines are just plugged into the mains socket in Amazon's data centers?
- my123 5y agoYou get an OS drive backed by EBS on those, through the AWS Nitro System. As such, they share the same storage infrastructure as other EC2 instances.
- supermatt 5y agoThis F_FULLFSYNC behaviour has been like this on OSX for as long as I can remember. It is a hint to ensures that the data in the write buffer has been flushed to stable storage - this is historically a limitation of fsync that is being accounted for - are you 1000% sure it does as you expect on other OSes? POSIX spec says no: https://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/f... Maybe unrealistic expectation for all OSes to behave like linux. Maybe linux fsync is more like F_BARRIERFSYNC than F_FULLFSYNC. You can retry with those for your benchmarks. Also note that 3rd party drives are known to ignore F_FULLFSYNC, which is why there is an approved list of drives for mac pros. This could explain why you are seeing different figures if you are supplying F_FULLFSYNC in your benchmarks using those 3rd party drives.
- marcan_42 5y agoYes. fsync() on Linux pushes down to stable storage, not just drive cache. OpenBSD, though, apparently behaves like macOS. I'm not sure I like that.
- supermatt 5y agoSo its basically implementation specific, and macOS has its own way of handling it. That doesnt make it worse - in fact it permits the flexibility you are now struggling with. edit: downvotes for truth? nice. go read the posix spec then come back and remove your downvotes...
- shellac 5y agoVariants of the FSYNC story have been going on for decades now. The framing varies, but typically somebody is benchmarking IO (often in the context of database benchmarking) and discovers a curious variance by OS. On NVMes I wonder whether this really matters, but it's a serious issue on spinning disks: do you really need to flush everything to the disk (and interrupt more efficient access patterns)?
- marcan_42 5y agoOn this NVMe, flushing is slower than on some spinning disks, so it apparently matters.
- shellac 5y agoYes, I would have skipped the fsync thing, which carries a lot of baggage, and concentrate on this. Btw, are you sure those spinning disks are actually flushing to rust? Caches all the way down... ;-)
- marcan_42 5y agoI mean, typical seek time on rust is O(10ms) and these controllers are spending 20ms flushing a few sectors. Obviously rust would do worse if you have the cache full of random writes, though. The problem here is the huge base cost.
- tlamponi 5y ago> On NVMes I wonder whether this really matters, but it's a serious issue on spinning disks: do you really need to flush everything to the disk (and interrupt more efficient access patterns)? That depends on the drive having power loss protection, which comes most of the time in the form of a capacitor that powers the drive long enough to guarantee that its buffers are flushed to persistent storage. Consumer SSDs often do not have that, so flushing is really important there, at least if your data, or no FS corruption is important to you. Enterprise SSDs almost always have power loss protection, so there it isn't required for consistency’s sake, albeit in-flight data that didn't hit the block device yet is naturally not protected by that, most FS handle that fine by default though. Note that Linux, for example, does by default a periodic flush every 30s independent of caching/flush settings, so that's normally the upper limit you'd lose, depending on the workload it can be still a relatively long time frame. https://sysctl-explorer.net/vm/dirty_expire_centisecs/ https://sysctl-explorer.net/vm/dirty_expire_centisecs/
- midrus 5y agoI've been using Macs (both desktop and laptops) since I have memory. I've had the M1 since launch day, and I use it all day, both for work and personal use. Why this never happened to me? Why I don't know anyone which had this problem? Why nobody is complaining as it happened with the previous gen keyboards? I think we might be missing something in this analysis. I don't think Apple engineers are idiots.
- wolf550e 5y agoOn laptop, you would get data loss / corruption on sudden power loss. This is rare. With "flush to storage device's RAM", even a kernel panic would not lose data if you let the storage device flush to flash without power loss.
- marcan_42 5y agoMost people don't unplug their Mac Mini in the middle of working, and most users who do lose data after that happens would just think it's normal and not realize there is an underlying problem and modern OSes aren't supposed to do that. I've seen APFS filesystems eat themselves in production (and had to do data recovery), twice. Apple don't have a perfect data integrity track record.
- EugeneOZ 5y agoClick-bait title again. “You can lose some of your file changes in case of hard-reboot” is more correct. It was a given truth for me all the time and I can tolerate some data losses if power was accidentally turned off for my desktop, or if OS panicked (it happens ~ once per year to me). If this is a price for a 1000x speed increase - I’m more than happy they have implemented it this way.
- marcan_42 5y agoYou can lose some file changes even after asking the OS to make sure they don't get lost, the normal way. That's a problem. It means e.g. transactional databases (which cannot afford to lose data like that) have a huge performance hit on these machines, since they have to use F_FULLFSYNC. And since that "no really, save my data" feature is not the standard fsync(), it means any portable software compiled for Linux will be safe, but will be unsafe on macOS, by default. That is a significant gotcha. The question is why do other NVMe manufacturers not have such a performance penalty? 10x is fine; 1000x is not. This is something Apple should fix. It's a firmware problem.
- EugeneOZ 5y agoNo, it’s not a problem, it is expected. If you are running a transactional database on your desktop - at least add a UPS to your system.
- olliej 5y agoThe whole point of a transactional database is that even in the case of a power loss you do not lose data. If you UPS blows up, and so you lose power, you should not lose data. The point here is that on the apple systems if you do the correct thing your performance drops to that of spinning disks.
- EugeneOZ 5y agoAdd a secondary UPS. What will be the next excuse? It's ridiculous to think that in case of power loss you expect 100% data integrity - it might happen in the middle of the command execution. If the system should be unkillable, it should have an unkillable power source in the first place.
- matthewmacleod 5y agoThat's interesting! I'd be fascinated to know what the underlying cause is – these full-sync numbers are amazingly low.
- marcan_42 5y agoI dug a bit further and the NVMe controller is doing about 6.2MB/s of DRAM reads and 10MB/s of DRAM writes while doing a flush loop like this (which it isn't doing with the same traffic sans the flushes). I wonder if it's doing something dumb like linear scanning a cache hash table to find things to flush... or maybe something with bad cache locality? I'm pretty sure, whatever it is, Apple could fix it in a firmware update.
- fmajid 5y agoThere’s a reason why Apple uses Linux for its server infrastructure.
- BirAdam 5y agoSure, but I do not think it’s due to their feeling that their own software is inferior. I think much more of that is cost. They needn’t pay to develop yet another OS variant, and instead benefit off of the open source community and their past contributions to said community.
- sudosysgen 5y agoThey wouldn't really need to develop a variant. Plenty of people used to run servers on macOS just configured to be headless. It just doesn't meet the standard anymore.
- fmajid 5y agoAnd they used to have the XServe line of servers and storage. I used XServe RAID storage as inexpensive JBODs for some Sun servers a decade ago, they were quite nice machines.
- jb1991 5y ago> Of course, in normal usage, this is basically never an issue on laptops; given the right software hooks, they should never run out of power before the OS has a chance to issue a disk flush command
- simonh 5y agoBut since I only use Mac desktops.....
- jb1991 5y agoI guess a UPS powerbackup would be useful. Laptops basically have built-in UPS which is perhaps why Apple has gone in that direction. I wonder if their high-end desktops with Apple Silicon will do something different there.
- freemint 5y agoAn UPS won't help though unless you program your Mac to check the UPS status and flush accordingly.
- jb1991 5y agoI'm not entirely sure how a UPS works with a computer, nor exactly how this flushing works, but doesn't a UPS run for awhile in the event of a power disruption, certainly more than the several seconds delay with this flushing?
- acdha 5y agoThat’s been built in to macOS since the 2000s - there’s a standard USB protocol which most UPS vendors support.
- kevincox 5y agoUnless of course your kernel panics. (Although there may still be a best-effort flush here, so it probably depends on how exactly it dies)
- deleted 5y ago[deleted]
- perryizgr8 5y agoFor their problem, they can easily solve it by replacing the SSD. Then fsync will be normal speed.
- borodi 5y agoI wonder if you hit the drive hard enough, so that the cache gets filled, does the performance degrade by that same magnitude?
- BirAdam 5y agoIn my use. Yes. I didn’t realize this was the reason until I saw this thread, and now I’ve tested it. Luckily, I don’t do massive data transfers nor do I do any large data work. When I got my M1 Mac Mini, however, I did and had immediate buyer’s remorse. I thought that I/O must be terrible on this thing, and I felt cheated. After the initial stand-up, I wasn’t so angry. For most tasks, it’s faster than my old TR4 1950X.
- trasz 5y agotl;dr OSX handles fsync() the way Linux used to, by not flushing to hardware
- lazerl0rd 5y agoThe funny thing here is that battery-backed enterprise systems are worse off in that manner, because you're much more likely to notice a dying battery that your entire device relies on than the little battery pack hooked up to your RAID array. Sure, you could write a program that periodically checks the battery rate (you'd have to poll since there's no ACPI notification like with a "device battery") and sends an email to the admin or something. However that's a tool that doesn't "exist" (as in, there isn't notable program that does so) which possibly hints that this isn't something system admins often do. The above also requires there to be an interface available from userland, not only in the management firmware or BIOS/UEFI. That exists for HP, but I'm not sure all other OEMs do so.
- claudex 5y agoEnterprise system have monitoring through the BIOS which will send an email, expose the status via SNMP and other method of monitoring (same as having a faulty fan).
- lazerl0rd 5y agoCorrect me if I'm wrong, but I wouldn't call the management engine (eg. HP iLO) the BIOS. Whilst those may support such warnings: 1) Not everyone wants to use iLO or whatever equivalent another OEM provides. 2) Whilst such systems do support sending warnings about system components via email, dashboards, etc. that doesn't mean they'll necessarily warn about a RAID controller's battery being depleted. If I remember correctly, iLO4 doesn't. 3) What about RAID cards like the P420 (*not* the P420i) that either aren't hooked up to a management engine or are from an entirely separate OEM?
- selectodude 5y ago>1) Not everyone wants to use iLO or whatever equivalent another OEM provides. Then you aren't an enterprise because they're absurdly useful for managing dozens/hundreds/thousands of systems. >3) What about RAID cards like the P420 (not the P420i) that either aren't hooked up to a management engine or are from an entirely separate OEM? There's a reason enterprises standardize on a common infrastructure from an OEM that supports everything in the box even though you could go on Newegg and build your own systems for thousands of dollars less.
- vbezhenar 5y agoIt's still not clear why Apple SSD so slow. Surely there's more to it. May be other SSDs are cheating in firmware? Or may be it's just bug in Apple firmware? I'm really interested if there will be follow ups on Apple side.
- marcan_42 5y agoSince this design is inherited from iDevices, my guess is they never bothered to optimize this command since software on a battery-powered device would almost never need to issue it. It should be something they can improve in firmware.
- claudex 5y agoMaybe it allow to design a device with less power consumption.
- joshspankit 5y agoFrom my understanding, the thing that’s slow is writing data to “permanent storage” (aka the layer under all the caching). Some storage tech is just slow at that, and manufacturers muddy the water by rating some (SSDs|Micro SDs|whatever) in GB/s overall when much of those big numbers are a combination of caches and trickery. I would not be surprised if Apple is using a tech that just has slow write speeds in trade for fast read speeds since most Apple users will be happy with faster read speeds.
- serguzest 5y agomaybe apple Nvmes have some sort of short time battery/capacitor gives it a time to finish the write once the power lost?
- viraptor 5y agoSlightly related: if a drive runs with a properly journaled, fully checksummed filesystem, for example zfs or btrfs - does the write-through mode guarantee that you can only lose new data and not corrupt the old?
- viraptor 5y agoFound it kind of answered in the side thread: https://mobile.twitter.com/marcan42/status/1494227803327598598 https://mobile.twitter.com/marcan42/status/14942278033275985... In short - no, you'll still see corruption.
- floatboth 5y agoNo, you won't see corruption on ZFS. Cutting power to the drive is always safe, you can slice a SATA cable with a guillotine if you want, you'll always see a consistent state of the filesystem. ZFS transactions are entirely atomic. ZFS (and btrfs) is not "journaled", it's copy-on-write.
- viraptor 5y agoYou won't see corruption of the filesystem itself, but you'll see data corruption as described in the thread. If the writes are delayed, the write ordering can get messed up. + Zfs has ZIL, which is basically journal equivalent.
- ryao 5y agoJournals are used to protect filesystem metadata. The ZIL is used only to protect data. You will not see any data corruption on ZFS as long as the underlying hardware implements REQ_PREFLUSH correctly and the software uses proper POSIX semantics. If no filesystem corruption is happening, then the stuff under ZFS is doing its job correctly and your problem is in userspace. Following a crash, ZFS returns to a past good state. Any completed synchronous writes or writes protected by a completed fsync will be there. Any of those that did not complete can be expected to be lost (unless it was moments away from returning to userspace) and any non-synchronous IO that occurred in the last several seconds is allowed to disappear. By default, non-sync IO is flushed to permanent storage every 5 seconds. A past good state is not something that I would call corruption and software is expected to be able to resume from the past under POSIX.
- throwawaylinux 5y agoThe dirty secret about today's high density NAND is that tPROG is not fast. It's an order of magnitude slower than the heyday of SLC. Now that doesn't really matter for enterprise drives, they complete writes into very fast storage that is made durable one way or another (e.g., flush on power fail), and this small store gets streamed out to the NAND log asynchronously. This is why random single queue depth durable writes can actually be faster than reads on enterprise drives, because random reads have to come from NAND (tREAD is still very fast, just not as fast as writing to DRAM). Apple may not implement such a durable cache, that's fine it's not an enterprise device and it's a cost tradeoff. So they might have to flush to NAND on any FUA, and that's slow as we've said, but not 25ms slow. Modern QLC NAND tPROG latency is more like 2.5ms-5ms, which could just about explain the EVO results when you include the OS and SATA stack and drive controller. There's pretty close to 0% chance Apple would have messed this up accidentally though, in my opinion. It would have been a deliberate design choice for some reason. One possible reason that comes to mind is that some drives gang a bunch of chips in parallel end you end up with pretty big "logical" pages. Flushing a big logical page on a 4kB write is going to cause a lot of write amp and drive wear, so you might delay for a short period (20ms) to try pick up other writes and reduce your inefficiency.
- mhh__ 5y agoI don't know what tPROG is (or anything else), is there a "What every programmer should know about storage" a la Drepper's work on memory?
- throwawaylinux 5y agotPROG is time it takes to program a NAND page from when you put the "program page" command on the pins to when you read off a successful status. Some of the basic NAND guides they put out are simple enough to understand the basics of operation https://www.micron.com/-/media/client/global/documents/products/technical-note/nand-flash/tn2914.pdf https://www.micron.com/-/media/client/global/documents/produ... The details get very complicated and proprietary. NAND wears out as you use it. But it also has a retention time. It gradually loses charge and won't read back if you leave it unpowered for long enough. This is actually where enterprise drives can be speced worse than consumer. So durability / lifetime is specified as meeting specified uncorrected error rates at the given retention period. The physics of NAND are pretty interesting too and how it translates into how a controller optimizes these parameters. Temperature at various stages of operation and retention changes properties, time between erase and program does too. You can adjust voltages on read, program, erase, and those can help you read data out or change the profile of the data. Reading can disturb parts of other pages (similar to rowhammer). Multilevel cells are actually interesting some of them you program in passes so that's a whole other spanner in the works. I don't know of a good place that covers all that, but much beyond "read/program/erase + wear + retention" is probably beyond "what every programmer should know". The way you turn a bunch of NAND chips that have a "read/program/erase" programming model into something that has a read/write model (the flash translation layer or FTL) is a whole other thing again though. And all the endurance management and optimization, error correction... Pretty fascinating details really. The basic details though is that they use the same concepts as the "log structured filesystem", turns out a log structure with garbage collection is about a perfect it for turning the program/erase model into a random write model. That's probably what every programmer should know about that (assuming you know something about LSFs -- garbage collection, write amplification, forward and reverse mapping schemes, etc).
- joshspankit 5y agoApple doesn’t care about on-disk data integrity. They just direct people to use Time Machine or iCloud, then look quizzically at you when have issue with writing off lost hours of work as cost of doing business.
- StillBored 5y agoHmm as slow as that is, does the controller support VERIFY? Because there is FUA in verify which forces the range to flush as well, and it could be used as a range flush. Depending on how they implement the disk cache its possible that is faster than a full cache walk (which is likely what they are doing). This is one of those things that SCSI was much better at, SYNC CACHE had a range option which could be used to flush say particular files/database tables/objects/whatever to nonvolatile storage. Of course out of the box Linux (and most other OSs) don't track their page/buffer caches closely enough to pull this off, so that fsync(fileno) is closer to sync(). So, few storage systems implemented it properly anyway. The choice of ignoring flushes vaguely makes sense if you assume the mac's SSD is in a laptop with a battery. In theory then the disk cache is non-volatile (and this assumption is made on various enterprise storage arrays with battery backup as well, although frequently its a controller setting). But i'm guessing someone just ignored the case of the mac mini without a battery.
- supermatt 5y agoI assumed the barrier was doing something like that, but marcan was able to inspect the actual nvme commands issued and has confirmed thats not the case. But that would be awesome, especially with these ever growing cache capacities.
- gok 5y agoAmusingly XNU adopted this behavior because it's what Linux did in the early 2000s and people complained that fsync was too slow without it.
- duxup 5y agoIs it possible to quantify how likely you are to hit a data integrity issue because of this?
- _vvhw 5y agoFantastic thread. The history is also interesting. It's not that "macOS cheats", but that it sincerely inherited the status quo of many years, then tried to go further by adding F_FULLFSYNC. However, Linux since got better, leaving macOS stuck in the past and everybody surprised. It's a big problem. Here's Dominic Giampaolo from Apple discussing this back in 2005, before Linux fixed fsync() to flush past the disk cache: https://lists.apple.com/archives/darwin-dev/2005/Feb/msg00087.html https://lists.apple.com/archives/darwin-dev/2005/Feb/msg0008... And here's TigerBeetle's Twitter thread with more of the history and how projects like LevelDB, SQLite and various language std libs were also affected: https://twitter.com/TigerBeetleDB/status/1422854779009654785 https://twitter.com/TigerBeetleDB/status/1422854779009654785
- yalogin 5y ago> fsync() will both flush writes to the drive, and ask it to flush its write cache to stable storage. Can someone explain what "flushing write cache to stable storage" means? Isn't that the same as "writes to the drive". I am obviously not well versed in this area. Also what is stable storage? Never heard that term before.
- fred123 5y agoSSDs and other storage drives have two layers (or more). The last layer is stable storage (= when you disconnect power no data is lost or corrupted). When you write to such a device your writes are first made in an earlier layer that is more like your computer’s main memory than actual storage (when you lose power your data is gone or corrupted). Only after time or when the cache is full an actual persistent write is made.
- deleted 5y ago[deleted]
- lowbloodsugar 5y agoI'm not here to defend apple, but if you have a desktop and you don't want to lose data then get a UPS. Proper write handling on the disk wont help if you haven't saved your doc in ten minutes.
- belter 5y agoFor some here comparing and contrasting both documentation ambiguity and fsync behaviour between OSX and Linux, these two are probably useful: "Linux Fsync Issue for Buffered IO and Its Preliminary Fix for PostgreSQL" https://news.ycombinator.com/item?id=19238121 https://news.ycombinator.com/item?id=19238121 "PostgreSQL used fsync incorrectly for 20 years (2019) [video] (fosdem.org)" https://news.ycombinator.com/item?id=30131165 https://news.ycombinator.com/item?id=30131165
- brokenmachine 5y agoInteresting post, but oh my god twitter is a garbage platform for what should be a blog post.
- mikewarot 5y agoThink about what's going on in the controller running any page access SSD. You have wear leveling trying to keep things from blowing holes in certain physical pages. In certain cell architectures you can only write to pages that have previously been erased. Once you do write the data to the silicon... it's not really written anyway, because the tables and data structures that map that to the virtual table the host sees on boot also have to be written. It is entirely reasonable that a system that does 100k honest sustained write I/O per second would come to its knees if you're insistent enough to actually want a full, real, power cycle proof, sync. To do an actual full sync, where it could come back from power off... requires flushing all of those layers. Nothing is optimized to do that. I'm amazed that it can happen 40 times per second. It's possible that you could speed this up a bit, but somewhere there's an actual non-wear leveled single page of data that tells the drive how to remap things to be useful... I strongly suspect writing that page frequently would eat the drive life up in somewhere between 0.1 and 20 million cycles. After that point, the drive would be toast. I agree with the other thread that actually flushing is likely to be a very, very well guarded bit of info.