8 ms·

As the article mentions, on laptops, this is pretty clever. On desktops though... Perhaps real macs should be equipped with internal batteries to flush to disk

by steeve 5y ago

As the article mentions, on laptops, this is pretty clever. On desktops though... Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss? I think I heard some enterprise motherboards/controllers/computers did just that, given the upside in normal operation.

throw3838 5y ago
How exactly this is clever? Maybe on some toy, not on workstation!
marcan_42 5y ago
These machines are actually low-power enough that you could implement a last-gasp flush mechanism. The Mac Mini already survives 1-2 seconds without AC power (at least if idle). You could plausibly detect AC power being yanked and immediately power down all downstream USB/TB3 devices and the display (on iMacs), freeze all CPUs into idle, and have plenty enough reservoir cap to let NVMe issue a flush. But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown. This does require a last-gasp indication from the PSU to the rest of the system, so if they don't have that, it's not something they could add in a firmware update.
- fauigerzigerk 5y ago
  >But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown. That's unfortunate. My Mac Mini crashes every other night during sleep. I guess I'm going to have to shut it down to avoid any data corruption.
  - marcan_42 5y ago
    It should be flushing the drive cache on sleep. This is mostly an issue for sudden AC power loss.
    - fauigerzigerk 5y ago
      Ah, thanks! That's good to know.
  - sgt 5y ago
    Why does it crash? Mac Minis are some of the most reliable machines on the market, in my experience. Maybe a faulty unit?
    - salamandersauce 5y ago
      Shitty software? My 2018 Mac Mini would crash every single time going to sleep on the last version of Mojave. I'm not alone in this as there's huge threads on MacRumors and Apple's support forum about it. Apple's "fix" was to just update to Catalina which indeed fixes it but doesn't really help if you want to run 32 bit software. Wouldn't surprise me if they did something similar again.
    - fauigerzigerk 5y ago
      It has started crashing the night after I upgraded to macOS 12.2.0. The latest update to 12.2.1 hasn't fixed it. I'm pretty sure it's not hardware related as I had no issues before the OS upgrade. Edit: Here's the first line of the crash log (which I'm sending to Apple every time): panic(cpu 3 caller 0xfffffe0023be8be0): [data.kalloc.16]: element modified after free (off:0, val:0x0000000000000030, sz:16, ptr:0xfffffe2fffc9bb00) Looks like a use after free bug.
      - yborg 5y ago
        This is why I'm still on Catalina. I used to be in the "trail by one point release" mode on macOS, now I'm in the "trail by 2 major releases" camp.
    - david_allison 5y ago
      I get regular crashes restoring from sleep on my 2014 Mac Mini (running Monterey)
- rubatuga 5y ago
  I mean the ATX standard has this signal built in, so Apple could just copy it: https://en.wikipedia.org/wiki/Power_good_signal https://en.wikipedia.org/wiki/Power_good_signal
  - dale_glass 5y ago
    > The ATX specification requires that the power-good signal ("PWR_OK") go high no sooner than 100 ms after the power rails have stabilized, and remain high for 16 ms after loss of AC power, and fall (to less than 0.4 V) at least 1 ms before the power rails fall out of specification (to 95% of their nominal value). I don't think that quite works for the purpose. What you'd want is a second signal that goes low as soon as possible after loss of AC power. My reading here is that PWR_OK going low is an indication that the PSU has stopped providing good power, and the CPU must shut down immediately, or it might miscompute something due to low voltage. At this point you absolutely don't want to do any last-minute writing, you'd be risking corruption. What you need here is an early warning signal that you can react to while the PSU is still coasting on the internal capacitors.
    - fps-hero 5y ago
      Power OK signals are used to prevent latch ups in silicon due to power glitches. The signals will route to power management ICs to ensure a full reset with proper bringing up of the power rails on any power glitch.
    - jaidan 5y ago
      16ms is just longer than one AC cycle at 60Hz and less than one AC cycle at 50Hz. I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss. Of course there is industrial hardware that can detect quicker than this but it would add a LOT of cost for arguably little gain, or something that could be solved in another manner.
      - dale_glass 5y ago
        > I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss. Doubtful. 16ms is an awfully long time these days. There's no reason why you couldn't detect power loss much sooner, given a good input signal. The concept also gets used quite often, in the form of SSRs with zero crossing detection. Those are used for dimmers. The reason is likely related to the awful waveforms produced by some UPSes and inverters: https://www.christidis.info/images/blog/scope_20.png https://www.christidis.info/images/blog/scope_20.png Unlike a nice sine wave, those spend a good while hovering near zero volts, so the PSU has to be able to tolerate that. Detecting loss of power sooner in this case isn't a question of cost, it's a question of that you don't have a good signal to do the detection on to start with.
FooBarWidget 5y ago
Even on laptops I feel uncomfortable. My macOS freezes or kernel panics on me from time to time.
- marcan_42 5y ago
  I believe the NVMe driver has a kernel panic hook; I would hope it is used to issue a flush. OTOH, if you have watchdog timeouts (I've seen this from bad drivers), those would certainly not give the kernel a chance to do that.
  - ladyanita22 5y ago
    What would you implement in Asahi? Would you follow Apple's approach and defer flushes, implementing a kernel panic hook and having some kind of F_FULLFSYNC or just keep Linux' current implementation?
    - marcan_42 5y ago
      We're probably going to have a knob to defer flushes (but still do them, unlike Apple, after a max timeout) that will be on by default on laptops, and make sure panics flush the cache if we can. Also apparently we need to do something for the power button too, as I just tested how macOS handles that. There is a warning before the system shuts down but we need to listen to it. Same with critical battery states.
      - ladyanita22 5y ago
        Then I misunderstood. Do you mean that Apple doesn't implement ANY timeout? So they only flush when the cache is full or when a shutdown routine has started?
        marcan_42 5y ago
        They flush the cache when something requests the cache be flushed; I don't know if there is a timeout, because presumably it's not difficult for some random process to issue a FULLFSYNC and flush everything prior as a side-effect (the flush is global). But I've seen at least 5-10 seconds of data loss from drive cache loss on the Mac Mini, so if they do do deferred flushes the timeout is longer than that.
dzhiurgis 5y ago
Does disk gets flushed in case of kernel panic?
coldtea 5y ago
>Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss? Or just add a UPS?

by steeve 5y ago

throw3838 5y ago

How exactly this is clever? Maybe on some toy, not on workstation!

marcan_42 5y ago

These machines are actually low-power enough that you could implement a last-gasp flush mechanism. The Mac Mini already survives 1-2 seconds without AC power (at least if idle). You could plausibly detect AC power being yanked and immediately power down all downstream USB/TB3 devices and the display (on iMacs), freeze all CPUs into idle, and have plenty enough reservoir cap to let NVMe issue a flush. But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown. This does require a last-gasp indication from the PSU to the rest of the system, so if they don't have that, it's not something they could add in a firmware update.

fauigerzigerk 5y ago
>But they aren't doing that. I tested it on the Mac Mini. It loses several seconds of fsync()ed data on hard shutdown. That's unfortunate. My Mac Mini crashes every other night during sleep. I guess I'm going to have to shut it down to avoid any data corruption.
- marcan_42 5y ago
  It should be flushing the drive cache on sleep. This is mostly an issue for sudden AC power loss.
  - fauigerzigerk 5y ago
    Ah, thanks! That's good to know.
- sgt 5y ago
  Why does it crash? Mac Minis are some of the most reliable machines on the market, in my experience. Maybe a faulty unit?
  - salamandersauce 5y ago
    Shitty software? My 2018 Mac Mini would crash every single time going to sleep on the last version of Mojave. I'm not alone in this as there's huge threads on MacRumors and Apple's support forum about it. Apple's "fix" was to just update to Catalina which indeed fixes it but doesn't really help if you want to run 32 bit software. Wouldn't surprise me if they did something similar again.
  - fauigerzigerk 5y ago
    It has started crashing the night after I upgraded to macOS 12.2.0. The latest update to 12.2.1 hasn't fixed it. I'm pretty sure it's not hardware related as I had no issues before the OS upgrade. Edit: Here's the first line of the crash log (which I'm sending to Apple every time): panic(cpu 3 caller 0xfffffe0023be8be0): [data.kalloc.16]: element modified after free (off:0, val:0x0000000000000030, sz:16, ptr:0xfffffe2fffc9bb00) Looks like a use after free bug.
    - yborg 5y ago
      This is why I'm still on Catalina. I used to be in the "trail by one point release" mode on macOS, now I'm in the "trail by 2 major releases" camp.
  - david_allison 5y ago
    I get regular crashes restoring from sleep on my 2014 Mac Mini (running Monterey)
rubatuga 5y ago
I mean the ATX standard has this signal built in, so Apple could just copy it: https://en.wikipedia.org/wiki/Power_good_signal https://en.wikipedia.org/wiki/Power_good_signal
- dale_glass 5y ago
  > The ATX specification requires that the power-good signal ("PWR_OK") go high no sooner than 100 ms after the power rails have stabilized, and remain high for 16 ms after loss of AC power, and fall (to less than 0.4 V) at least 1 ms before the power rails fall out of specification (to 95% of their nominal value). I don't think that quite works for the purpose. What you'd want is a second signal that goes low as soon as possible after loss of AC power. My reading here is that PWR_OK going low is an indication that the PSU has stopped providing good power, and the CPU must shut down immediately, or it might miscompute something due to low voltage. At this point you absolutely don't want to do any last-minute writing, you'd be risking corruption. What you need here is an early warning signal that you can react to while the PSU is still coasting on the internal capacitors.
  - fps-hero 5y ago
    Power OK signals are used to prevent latch ups in silicon due to power glitches. The signals will route to power management ICs to ensure a full reset with proper bringing up of the power rails on any power glitch.
  - jaidan 5y ago
    16ms is just longer than one AC cycle at 60Hz and less than one AC cycle at 50Hz. I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss. Of course there is industrial hardware that can detect quicker than this but it would add a LOT of cost for arguably little gain, or something that could be solved in another manner.
    - dale_glass 5y ago
      > I would has a guess that 16ms is the physical limit for most consumer hardware (and maybe commercial computing) to detect mains loss. Doubtful. 16ms is an awfully long time these days. There's no reason why you couldn't detect power loss much sooner, given a good input signal. The concept also gets used quite often, in the form of SSRs with zero crossing detection. Those are used for dimmers. The reason is likely related to the awful waveforms produced by some UPSes and inverters: https://www.christidis.info/images/blog/scope_20.png https://www.christidis.info/images/blog/scope_20.png Unlike a nice sine wave, those spend a good while hovering near zero volts, so the PSU has to be able to tolerate that. Detecting loss of power sooner in this case isn't a question of cost, it's a question of that you don't have a good signal to do the detection on to start with.

FooBarWidget 5y ago

Even on laptops I feel uncomfortable. My macOS freezes or kernel panics on me from time to time.

marcan_42 5y ago
I believe the NVMe driver has a kernel panic hook; I would hope it is used to issue a flush. OTOH, if you have watchdog timeouts (I've seen this from bad drivers), those would certainly not give the kernel a chance to do that.
- ladyanita22 5y ago
  What would you implement in Asahi? Would you follow Apple's approach and defer flushes, implementing a kernel panic hook and having some kind of F_FULLFSYNC or just keep Linux' current implementation?
  - marcan_42 5y ago
    We're probably going to have a knob to defer flushes (but still do them, unlike Apple, after a max timeout) that will be on by default on laptops, and make sure panics flush the cache if we can. Also apparently we need to do something for the power button too, as I just tested how macOS handles that. There is a warning before the system shuts down but we need to listen to it. Same with critical battery states.
    - ladyanita22 5y ago
      Then I misunderstood. Do you mean that Apple doesn't implement ANY timeout? So they only flush when the cache is full or when a shutdown routine has started?
      - marcan_42 5y ago
        They flush the cache when something requests the cache be flushed; I don't know if there is a timeout, because presumably it's not difficult for some random process to issue a FULLFSYNC and flush everything prior as a side-effect (the flush is global). But I've seen at least 5-10 seconds of data loss from drive cache loss on the Mac Mini, so if they do do deferred flushes the timeout is longer than that.

dzhiurgis 5y ago

Does disk gets flushed in case of kernel panic?

coldtea 5y ago

>Perhaps real macs should be equipped with internal batteries to flush to disk in the case of power loss? Or just add a UPS?