5 ms·

So its basically implementation specific, and macOS has its own way of handling it. That doesnt make it worse - in fact it permits the flexibility you are now

So its basically implementation specific, and macOS has its own way of handling it. That doesnt make it worse - in fact it permits the flexibility you are now struggling with. edit: downvotes for truth? nice. go read the posix spec then come back and remove your downvotes...

marcan_42 5y ago
What is worse is their NVMe controller having 50x worse flush performance than the competition.
- supermatt 5y ago
  The competitions controller may be ignoring the F_FULLFSYNC. This is a known issue which is why apple have approved vendors for mac pro drives.
  - marcan_42 5y ago
    It isn't, because otherwise it would be showing the ~same performance with and without sync commands, as I showed in the thread. There is a significant performance loss for every drive, but Apple's is way worse. There is no real excuse for a single sector write to take ~20ms to flush to NAND, all the while the NAND controller is generating some 10MB/s of DRAM traffic. This is a dumb firmware design issue.
    - supermatt 5y ago
      It may be interpreting it differently. You arent comparing apples to apples, quite literally. Why not compare macOS and linux on approved x86 mac hardware. i.e. fusion drive or whatever. Also, as suggested - try F_BARRIERFSYNC, which flushes anything before the barrier (used for WAL IIRC).
      - throwawaylinux 5y ago
        It seems to be pretty apples to apples, they're running the same benchmark using equivalent data storage APIs on both systems. What are you thinking might be different? The Linux+WD drive isn't making the data durable? Or that OSX does something stupid which could be the cause of the slowdown rather than the drive? Both seem implausible.
      - marcan_42 5y ago
        This affects T2 Macs too, which use the same NVMe controller design as M1 Macs. We've looked at NVMe command traces from running macOS under a transparent hypervisor. We've issued NVMe commands outside of Linux from a bare-metal environment. The 20ms flush penalty is there for Apple's NVMe implementation. It's not some OS thing. And other drives don't have it. And I checked and Apple's NVMe controller is doing 10MB/s of DRAM memory traffic when issued flushes, for some reason (yes, we can get those stats). And we know macOS does not properly flush with just fsync() because it actively loses data on hard shutdowns. We've been fighting this issue for a while now, it's just that it only just hit us yesterday/today that there is no magic in macOS - it just doesn't flush, and doesn't guarantee data persistence, on fsync().
        supermatt 5y ago
        Ive just been scanning through linux kernel code (inc ext4). Are you sure that its not issuing a PREFLUSH? What are your barrier options on the mount? I think you will find these are going to be more like F_BARRIERFSYNC. I couldnt find much info about it - but the official docs are here: https://kernel.org/doc/html/v5.17-rc3/block/writeback_cache_control.html https://kernel.org/doc/html/v5.17-rc3/block/writeback_cache_...
        marcan_42 5y ago
        Those are Linux concepts. What you're looking for is the actual NVMe commands. There's two things: FLUSH (which flushes the whole cache), and a WRITE with the FUA bit set (which basically turns that write into write-through, but does not guarantee anything about other commands). The latter isn't very useful for most cases, since you usually want at least barrier semantics if not a full flush for previously completed writes. And that leaves you with FLUSH. Which is the one that takes 20ms on these drives.
dathinab 5y ago
Probably more like downvoted because missing the point. Sure fsync allows that behavior, but also it's so widely misunderstood that a lot of programs which should do a "full" flush only do a fsync, including Benchmarks. In which case they are not comparable and doing so is cheating. But that's not the point! The point is that with the M1 Macs SSDs the performance with fully flushing to disk is abysmal bad. And as such any application with cares for data integrity and does a full flush can expect noticable performance degradation. The fact that Apple neither forces frequent full syncs or at least full syncs when a Application is closed doesn't make it better. Though it is also not surprising as it's not the first time Apple set things up under the assumption their hardware is unfailable. And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice.
- ClumsyPilot 5y ago
  "And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice" Does the battery last forever? Do they never shut down from overheating, shut down from being too cold, freeze up, they are water and coffee proof? Talk to anyone that repairs mac about how high-end and reliable their designs trully are - they are better than bottomn of the barrel craptops, sure, but not particularly amazing and have some astounding design flaws.
  - BenjiWiebe 5y ago
    How cold is too cold for a computer?
    - michaelmrose 5y ago
      The CPU can't possibly get too cold. See for example overclocking performed by cooling the CPU with liquid nitrogen. Condensation is a factor as is lost of ductility of plastic at low temp making it brittle. Expansion and contraction of materials especially when different materials expand to different degrees.
rsync 5y ago
I downvoted you because you complained about your downvotes.
- supermatt 5y ago
  How did that work out for you?

marcan_42 5y ago

What is worse is their NVMe controller having 50x worse flush performance than the competition.

supermatt 5y ago
The competitions controller may be ignoring the F_FULLFSYNC. This is a known issue which is why apple have approved vendors for mac pro drives.
- marcan_42 5y ago
  It isn't, because otherwise it would be showing the ~same performance with and without sync commands, as I showed in the thread. There is a significant performance loss for every drive, but Apple's is way worse. There is no real excuse for a single sector write to take ~20ms to flush to NAND, all the while the NAND controller is generating some 10MB/s of DRAM traffic. This is a dumb firmware design issue.
  - supermatt 5y ago
    It may be interpreting it differently. You arent comparing apples to apples, quite literally. Why not compare macOS and linux on approved x86 mac hardware. i.e. fusion drive or whatever. Also, as suggested - try F_BARRIERFSYNC, which flushes anything before the barrier (used for WAL IIRC).
    - throwawaylinux 5y ago
      It seems to be pretty apples to apples, they're running the same benchmark using equivalent data storage APIs on both systems. What are you thinking might be different? The Linux+WD drive isn't making the data durable? Or that OSX does something stupid which could be the cause of the slowdown rather than the drive? Both seem implausible.
    - marcan_42 5y ago
      This affects T2 Macs too, which use the same NVMe controller design as M1 Macs. We've looked at NVMe command traces from running macOS under a transparent hypervisor. We've issued NVMe commands outside of Linux from a bare-metal environment. The 20ms flush penalty is there for Apple's NVMe implementation. It's not some OS thing. And other drives don't have it. And I checked and Apple's NVMe controller is doing 10MB/s of DRAM memory traffic when issued flushes, for some reason (yes, we can get those stats). And we know macOS does not properly flush with just fsync() because it actively loses data on hard shutdowns. We've been fighting this issue for a while now, it's just that it only just hit us yesterday/today that there is no magic in macOS - it just doesn't flush, and doesn't guarantee data persistence, on fsync().
      - supermatt 5y ago
        Ive just been scanning through linux kernel code (inc ext4). Are you sure that its not issuing a PREFLUSH? What are your barrier options on the mount? I think you will find these are going to be more like F_BARRIERFSYNC. I couldnt find much info about it - but the official docs are here: https://kernel.org/doc/html/v5.17-rc3/block/writeback_cache_control.html https://kernel.org/doc/html/v5.17-rc3/block/writeback_cache_...
        marcan_42 5y ago
        Those are Linux concepts. What you're looking for is the actual NVMe commands. There's two things: FLUSH (which flushes the whole cache), and a WRITE with the FUA bit set (which basically turns that write into write-through, but does not guarantee anything about other commands). The latter isn't very useful for most cases, since you usually want at least barrier semantics if not a full flush for previously completed writes. And that leaves you with FLUSH. Which is the one that takes 20ms on these drives.

dathinab 5y ago

Probably more like downvoted because missing the point. Sure fsync allows that behavior, but also it's so widely misunderstood that a lot of programs which should do a "full" flush only do a fsync, including Benchmarks. In which case they are not comparable and doing so is cheating. But that's not the point! The point is that with the M1 Macs SSDs the performance with fully flushing to disk is abysmal bad. And as such any application with cares for data integrity and does a full flush can expect noticable performance degradation. The fact that Apple neither forces frequent full syncs or at least full syncs when a Application is closed doesn't make it better. Though it is also not surprising as it's not the first time Apple set things up under the assumption their hardware is unfailable. And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice.

ClumsyPilot 5y ago
"And maybe for a desktop focused high end designs where most devices sold are battery powered that is a reasonable design choice" Does the battery last forever? Do they never shut down from overheating, shut down from being too cold, freeze up, they are water and coffee proof? Talk to anyone that repairs mac about how high-end and reliable their designs trully are - they are better than bottomn of the barrel craptops, sure, but not particularly amazing and have some astounding design flaws.
- BenjiWiebe 5y ago
  How cold is too cold for a computer?
  - michaelmrose 5y ago
    The CPU can't possibly get too cold. See for example overclocking performed by cooling the CPU with liquid nitrogen. Condensation is a factor as is lost of ductility of plastic at low temp making it brittle. Expansion and contraction of materials especially when different materials expand to different degrees.

rsync 5y ago

I downvoted you because you complained about your downvotes.

supermatt 5y ago
How did that work out for you?