29 ms·

"ERROR: could not open temporary file" after upgrade to Mac OS Big Sur

227 points by devops000 6y ago

threatofrain 6y ago
Ouch. > The issue seems to be caused by some new security APIs in Big Sur. Apple has apparently started returning with errno=EINTR from some system calls that previously never returned this error. Fixing this problem on the PostgreSQL side is not trivial, since it seems to require changes in both PostgreSQL itself and possibly in extensions as well, so it's unlikely to be fixed quickly. > These are the possiblities: > Try to disable security software (eg. Antivirus products) that might trigger the problem Downgrade to macOS 10.15 Send feedback to Apple that they should fix the issue. PostgreSQL is unlikely to be the only affected software. Hope that someone fixes the error in PostgreSQL https://github.com/PostgresApp/PostgresApp/issues/610#issuecomment-770178375 https://github.com/PostgresApp/PostgresApp/issues/610#issuec...
- wheresmycraisin 6y ago
  .
  - deleted 6y ago
    [deleted]
    - 1_player 6y ago
      s/Postgres/macOS/
- jb1991 6y ago
  Those proposed fixes don't make any sense for macOS. Nearly no one runs antivirus software on a Mac, and downgrading to prior OS version is basically impossible for most people.
  - wilsonrocks 6y ago
    We've been made to use Macs at work now and they have McAfee on that we can't disable, causes no end of performance issues
    - KingOfCoders 6y ago
      We also had anti-virus software on every Mac in large enterprise. Took a lot of effort to at least except source directories.
      - wilsonrocks 6y ago
        Oooh I wonder if I could get that here - things like npm install(presume any package manager, certainly homebrew) or git commands take obscenely long times.
        joosters 6y ago
        Commands that download random 3rd party code and dependencies from the internet are precisely the kind of thing that should be included in a virus scan, surely?
        foepys 6y ago
        Especially since any npm package can execute any code at install time in npm's default configuration. On Linux this is extra juicy since you cannot globally install any package without root unless you explicitly changed the directory's permissions and/or location.
        jessaustin 6y ago
        By default, npm installs into the local project directory, as it should. Only the OS package manager should touch system directories. Before using the "--global" flag, think about what you're actually trying to do, and what the better way to do that would be. One conventional workaround is to install commonly used tools to "~/bin". Root is not required for that. Of course, it's a good idea to keep the username used for development separate from the one used for browsing the web. It wouldn't be surprising for rogue npm packages to search for e.g. credit card details. I'm sure the browsers try to obfuscate that somehow, but how much can they really defend against code that is allowed to read the disk?
- tpetry 6y ago
  Running docker with homebrew and on docker, no problems. I am not using antivirus as they produce many problems on mac (and are hellish slow), i guess the problem is really on triggered by antivirus software.
foepys 6y ago
.NET (Core) has also an issue with this. [0] Apple seems to intentionally break quite a lot of core APIs lately. 0: https://github.com/dotnet/runtime/issues/47584 https://github.com/dotnet/runtime/issues/47584
- my123 6y ago
  Returning EINTR is valid behaviour.
  - deleted 6y ago
    [deleted]
  - foepys 6y ago
    Even if it's valid behavior, it's still a breaking change. The function never set this error code before, now it does, and at least 3 large projects are not working anymore.
    - matwood 6y ago
      Could Apple be better at communicating changes like this? Absolutely. Is Apple in the wrong here? No. This is a documented return code that has been ignored. What's the point of a standard if they can't use it?
      - SubjectToChange 6y ago
        Apple isn't following POSIX by ignoring SA_RESTART. What is the point indeed.
    - bobbylarrybobby 6y ago
      I don’t think violating Hyrum’s law should be considered a breaking change, but I suppose that that’s a matter of opinion.
      - SubjectToChange 6y ago
toyg 6y ago
Sigh. Looks more and more likely Mojave will be the last OSX version I’ll ever use.
- my123 6y ago
  > Many system calls will report the EINTR error code if a signal occurred while the system call was in progress. No error actually occurred, it's just reported that way because the system isn't able to resume the system call automatically. This coding pattern simply retries the system call when this happens, to ignore the interrupt. > For instance, this might happen if the program makes use of alarm() to run some code asynchronously when a timer runs out. If the timeout occurs while the program is calling write(), we just want to retry the system call (aka read/write, etc). It's perfectly valid behaviour, and the solution is to retry in that case.
  - tobiasu 6y ago
    Your solution will hang postgres at 100% CPU, with no indication what is wrong.
  - jorams 6y ago
    Setting the SA_RESTART flag using sigaction[1], which Postgres does, makes it invalid behaviour. Even according to Apple's (confusingly worded) documentation[2]. [1]:https://pubs.opengroup.org/onlinepubs/9699919799/functions/sigaction.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/s... [2]: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/sigaction.2.html https://developer.apple.com/library/archive/documentation/Sy...
  - toyg 6y ago
    The point is that this obsession with more and more restrictions is death by a thousand cuts.
- drunkpotato 6y ago
  Same. Much as I’d hate to do it I’ve been eyeing a Windows system, with Linux either through the OS layer or virtualized.
ainar-g 6y ago
I wonder if EINTRs are now returned from syscalls that were always documented to (sometimes) return them in POSIX/SUS/other documentation but PostgreSQL simply ignored that possibility with Mach, or if they are returned from some completely unexpected places?
- dfox 6y ago
  This seems to be the case. On the other hand mostly everybody ignores the fact that open(2) is documented as possibly returning -EINTR, because it essentially never happens (in particular, network filesystem are usually implemented in a way that it will not happen even in situations when it clearly should happen).
  - deleted 6y ago
    [deleted]
- viraptor 6y ago
  It's returned from open() https://www.postgresql.org/message-id/CAFHbkwjPXHgWb15wsmaCMuYOOF2Fm6sBVj9Sv1LbwRav75KsRg%40mail.gmail.com https://www.postgresql.org/message-id/CAFHbkwjPXHgWb15wsmaCM... https://pubs.opengroup.org/onlinepubs/9699919799/ https://pubs.opengroup.org/onlinepubs/9699919799/ Posix is happy with returning EINTR.
  - cesarb 6y ago
    That documentation says "[EINTR] A signal was caught during open().", and the documentation there for sigaction with SA_RESTART says "If set, and a function specified as interruptible is interrupted by this signal, the function shall restart and shall not fail with [EINTR] unless otherwise specified.", so as long as all signals were installed with SA_RESTART (which is probably the most common case since nobody likes dealing with EINTR), open() should never fail with EINTR.
  - tzs 6y ago
    This from that PostgreSQL mailing list link you give is interesting: > AUTH means that the system call is blocked (on that condition variable I mentioned about), and the user mode daemon is asked about the generated event: "postgres, pid 999, open() on file /path/file with flags 0x400003" - something like that. The usermode can either allow or deny the event by replying. The use of a user mode daemon to make decisions about whether file access is allowed reminds me of the way TOPS-10 on DEC PDP-10 did access control lists over 40 years ago. It's a method I'd like to see for ACLs on modern systems. On modern systems the ACL is part of the per file metadata. On TOPS-10 the ACL information was in a separate file that contained the ACL information for multiple files. I don't remember if is was one ACL file per user or one per directory. An ACL entry could specify file name (wildcards allowed), account and group requesting access (wildcards allowed), program that was requesting access, and what kind of access. The wildcards and the centralization of the ACLs for multiple files is what made this system excellent to use. When an access was attempted the system first checked if it was allowed by the ordinary per file permissions. If it was access was granted and the ACL daemon was not consulted. If the ordinary permissions denied access and the caller had not set a flag indicating it did not want to check the ACLs the access was denied. If that flag was not set the kernel would consult the access daemon, which would find the ACL file and check the ACLs within to find the ones that applied and make its decision.
jarym 6y ago
I run Postgres in docker containers on my Mac but I know that won’t be viable for a lot of people.
- starbugs 6y ago
  I have been running Postgres in Docker on my M1 MBP for a while now. No issues so far.
- vetinari 6y ago
  M1 docker is still in preview, so that's not an option for many folks running newer Macs.
- fernandotakai 6y ago
  i run linux and i prefer running all these kind of services (postgres, redis, rabbitmq and the likes) on docker containers. it's much easier to manage, easier to add/remove/clean and it doesn't pollute my local system with random stuff.
- bottled_poe 6y ago
  Yeah, if you are running Postgres natively on OSX instead of in Docker you are behind the times. I’m sure this will be downvoted, but it needs to be said.
  - Doctor_Fegg 6y ago
    No it doesn’t. There are several Mac-native apps, such as QGIS, for which Postgres is a sensible local datastore. Running it in a container for this is an entirely unnecessary complication.
  - jdboyd 6y ago
    How does permeance compare? Also that doesn't look like a good option for M1 mac users.
  - slezyr 6y ago
    > you are behind the times Seriously? What is wrong with running programs natively without any layers in between? Also, IIRC PostgreSQL isn't recommended to be run in docker due to a possible data loss?
Etheryte 6y ago
PostgreSQL isn't alone in this, VirtualBox (and I'm sure other virtualization products too) ran into the new API errors head first all over the place initially as well. The common use cases more or less work now, but there are still specific setups that you simply can't run on Big Sur.
shreddit 6y ago
Quite interesting to read old stackoverflow questions to this topic which recommended “not to care about it”. https://stackoverflow.com/questions/4959524/when-to-check-for-eintr-and-repeat-the-function-call https://stackoverflow.com/questions/4959524/when-to-check-fo...
- raverbashing 6y ago
  Yes, those things are usually dealt with a "don't care about it..." attitude (not condoning it!) until you eventually have to.
- cesarb 6y ago
  That question is about Linux, which doesn't have this macOS-specific issue.
m463 6y ago
Obligatory reference to "Worse is Better" https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html specifically the section discussing MIT vs BSD system call return codes.
layoutIfNeeded 6y ago
Seems like they're not handling EINTR for some syscalls. I remember when I was first learning Unix programming, and read about this stuff about signals and EINTR, I was baffled how much of a steaming pile of crap Unices are. Later learning about the inability to do truly non-blocking I/O was another similar moment.
ddevault 6y ago
And while they're busy breaking their current set of APIs, they're dragging their feet on adding anything designed in the last 15 years. Someone just wrote to me with a complaint because my software doesn't build on macOS due to missing getline and fmemopen implementations. Naturally I told them to use an open source operating system and declined to update my code to accomodate Apple.
- my123 6y ago
  getline and fmemopen are both supported on macOS. [/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk/usr/include] -> % grep -r -n getline stdio.h stdio.h:355:ssize_t getline(char \* __restrict __linep, size_t \* __restrict __linecapp, FILE \* __restrict [/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk/usr/include] -> % grep -r -n fmemopen \* stdio.h:356:FILE *fmemopen(void * __restrict __buf, size_t __size, const char \* __restrict __mode) __API_AVAILABLE(macos(10.13), ios(11.0), tvos(11.0), watchos(4.0)); getline became available on OS X 10.7 (in 2011) and fmemopen on macOS 10.13 (2017).
  - ddevault 6y ago
    They were on OSX Lepoard, maybe that's not the most recent one? I don't use Mac. When were they introduced? In any case, it's a trend with Apple, even if this particular example is wrong. See also: Vulkan.
    - bombcar 6y ago
      Leopard is the final version of macOS to support the PowerPC architecture - and released in 2007. Honestly, I wouldn't expect even Linux or BSD from 2007 to work well with modern software.
      - IggleSniggle 6y ago
        This is where I really need to pay respects to Windows. It’s a heaping pile but their commitment to backwards compatibility is remarkable. I recently installed a complex application that last updated in 2007, and while performance was absolutely terrible, it worked.
DiabloD3 6y ago
So when do average developers just start considering macOS toxic and stop supporting it? About once a week, a story similar to this hits the HN front page: "$majorsoftwareproduct doesn't work on new OSX because they broke documented APIs in undocumented and unexpected ways, without any warning". What the hell, Apple?
- cheph 6y ago
  > So when do average developers just start considering macOS toxic and stop supporting it? Since forever ago? Nothing new here. I have 0 sympathy for people who use it. They paid dearly for their walled garden.
  - treeman79 6y ago
    We love our walled garden. Don’t upgrade for a few weeks, and the hardcore fanboys will have fixes for all the annoying little breaks.
- tpush 6y ago
  > So when do average developers just start considering macOS toxic and stop supporting it? Presumably at the same time breathless and misinformed comments stop being posted to HN. In this case returning EINTR is documented behavior and the bug is thusly on PostgreSQL not handling it (correctly or at all).
- tinus_hn 6y ago
  How is returning EINTR undocumented? It’s part of the POSIX interface. It is true that typically Apple doesn’t care if they change things within the limit of the specification and 3rd party applications fail. That’s why they release beta versions.
  - Sesse__ 6y ago
    Standards aside, returning EINTR on a write to local file is extremely rare in POSIX. Almost any syscall can return EINTR, but in practice, for all previous OSes, it has been reserved for those that can hang for an undetermined amount of time -- that is, anything from the network. Local I/O should complete in a bounded amount of time, so that kind of sleep is usually implemented as uninterruptable. There's one highly relevant edge case: NFS. NFS looks like local I/O, but if the NFS server goes away, what do you do? Traditionally, the process just hangs in the hope that the server will come back, which leads to the dreaded issue of processes being stuck and unkillable. On Linux, can add the “intr” flag to the mount (possibly via a remount) to allow signals to interrupt the syscall and break the deadlock -- but then you break a ton of software, because, again, EINTR on local file I/O is unheard of.
gnfargbl 6y ago
As a long-term Linux user, I have been forced on occasion over the last few years to use MacOS. I am at an almost complete loss as to what anyone sees in it. Standard utilities that work seamlessly on Linux are regularly broken, there's a weird security system that requires me to do all sorts of boot-time nonsense just to have control over my own machine... and now, apparently, they're violating the first rule of kernel development by breaking userspace. The hardware is nice, but as a platform, I just don't get it.
- jolux 6y ago
  Not breaking userspace is the first rule of Linux kernel development, but macOS has never had that policy as far as I know. Apple care much more about being able to move fast and clean up bad interfaces in their software, which is a priority I value. I’m not saying that’s what happened here, but I do know that Apple ultimately expects developers to pay attention to new releases and make sure their software behaves properly with them. As for standard utilities being broken, I’ve never experienced that on macOS. I do know that Macs get their standard utilities from FreeBSD and not coreutils by default.
  - albertopv 6y ago
    There are liberally millions of app, Apple can't expect all of them to be always up to date on day one, to even sw like postgres. You can't just change the semantic of an API, it's just wrong. Apple should introduce a new API, deprecate the older one and remove it two or three versions later minimum. They just don't care until people is going to buy their hw.
    - my123 6y ago
      Traditionally, Apple does not guarantee total ABI stability. They push betas instead to allow people to fix the issues, and then support older releases for multiple years. Yes, unmaintained software might or might not stay fully functional on macOS for long, but that's not an Apple priority.
  - my123 6y ago
monadic3 6y ago
What is the point of posix, Unix, etc if companies can just break compatibility like this without losing certification? If they don't cover useful behavior they aren't useful.
sub7 6y ago
osx has sucked big dongs for years. Everything is virtualized and slow, nothing is backwards compatible and Windows is just so much better at this point. If only those idiots at MS would disable telemetry by default
pdkl95 6y ago
Does macOS respect SA_RESTART? [1] Apple's manpage[2] for sigaction is confusing... Regardless, doesn't anybody at Apple use postgres? Breaking Postgres should be hard to miss. [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/sigaction.html#tag_03_680_06 https://pubs.opengroup.org/onlinepubs/009695399/functions/si... [2] https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/sigaction.2.html https://developer.apple.com/library/archive/documentation/Sy...
- joseph_grobbles 6y ago
  "Regardless, doesn't anybody at Apple use postgres?" This problem occurs if you use anti-virus that delays file activities, and is 100% API consistent. Anyone saying "well it didn't do it before!" is being idiotic, similar to saying you don't need to check the return of malloc because it's usually going to succeed. pgsql runs perfectly fine.
CoffeeDregs 6y ago
I'm baffled that any developer uses a Mac. I know, I know... BSD, 3-finger swipe, but-they-wrote-the-ux-guide. But I've been running Debian Testing for 15 years now with nary a problem (on thinkpads, latitudes and desktops) I guess I'm weird in that I think gnome 3 is swell. CI handles xCode. Otherwise, my 2x 42" 4k monitors, Gnome and works-here-same-as-the-cloud self reads these articles on puzzlement. I can't count the times I've helped developers deal with oh-yeah-brew-is-weird-about-libpq. Now this? It's the avocado-toast of developer workstations...
- TheRealDunkirk 6y ago
  > I've been running Debian Testing for 15 years now with nary a problem I ran Linux on the desktop for 19 years, through years-long stretches of Slackware, RedHat, SuSE, Gentoo, and Ubuntu. Obviously, I was (and still am) a huge fan, but if you've managed 15 years of Linux on the desktop with "nary a problem," you have had the most unbelievable luck of any human on the planet, and should start buying lottery tickets. After running Gentoo for several years, and switching to Ubuntu, I was amazed at how much time I was saving not dealing with portage. Fair enough. I mean, I was asking for it by using a source distro, but I had that same sigh of relief when I finally bought a Mac, and realized that I was still doing a lot of maintenance to keep Ubuntu happy, even though it was (obviously) a huge improvement over Gentoo. I just couldn't see it until it was gone.
- Toutouxc 6y ago
  I held on to a Dell running Ubuntu for two years because I couldn't convince myself to shell out north of $2k for a comparable MacBook (I'd had a MacBook before). Then after one of the kernel updates the "resetting rcs0 for hang on rcs0" bug started happening, I had to downgrade to an earlier kernel, then Ubuntu 20 came with Snap everywhere, then covid came and I started having Zoom meetings all over the place and my mic would randomly stop working and I started using a 4K screen and noticed how choppy scrolling is and that YouTube stutters hard in fullscreen, then there was a memory leak in Firefox, semi-functional display scaling, the machine sometimes wouldn't wake up from deep sleep... Then the M1 Macs came out and I got an Air. I've been using it for a month and it's heaven. It's like coming home. It gets work done, lets me have fun, watch videos, play some games and gets out of the way.
varispeed 6y ago
Apple keeps throwing spanners in other companies wheels. Why spending time on new product development, if you have to commit teams to solve whatever this time Apple decided to break? This is anti-competitive behaviour that keeps companies working with Apple eco-system in check. Unfortunately thanks to excellent marketing, the user base is not going anywhere so companies need to put up with this.
- IggleSniggle 6y ago
  N of one, but this user is in the process of leaving Apple.
- netdur 6y ago
  Jan 23 I have started a rarely used windows laptop to do C++ development work, because my app depends on library that depends on Microsoft library that was not signed, therefor won't run with new update. we develop on macos and deploy on linux, macos is supported as side effect, windows was not supported... a single policy change without a workaround... made us support windows and linux, and macos left to dust.
joseph_grobbles 6y ago
Apple does exactly what the API is supposed to do, specifically in combination with antivirus software, and incorrectly written programs have small faults. Now read the comments before. This borders on parody at this point. It would be cool if all of the "this is the last straw!" and "welp, guess this is when I leave Mac" people who appear in each of these threads just, like, fuck off and leave then?
quotemstr 6y ago
Apple is right here and people complaining about the new behavior are wrong. If a system call can return EINTR, you must retry or deal with the error some other way. A retry is trivial too. It's irresponsible to depend on an API happening not to EINTR when it's documented to do so.
vemv 6y ago
macOS and importantly Homebrew's direction (which now doesn't support brewing from a sha-addressed github link) are leaving me at a crossroads. I value stability and autonomy over everything else - I expect the setups I work hard on scripting to work for years, freeing me of this continuous keeping up with the magical incantation du jour. As a hacker, it's particularly frustrating to feel at the whims of someone else. I think my next setup will be using my (company-issued) macbook as a thin client against a beefy unix machine. That way I'd keep a number of affordances (trackpad, iTerm, fancy Emacs etc) while having my core tooling absolutely stable. (I have the vague impression that Linux distros have also dramas and instability of their own... looks like the BSD family is more minimalistic?)
- dschuler 6y ago
  That's a good way of verbalizing how I've felt with the last few iterations of macOS as well as Homebrew's perpetually changing behavior. Most of it feels like change for the sake of change. I've also used FreeBSD for several years, and it's the opposite approach - everything that used to work still does, and things are well-documented. Usually the man page is enough, or the FreeBSD handbook. It's still all just Fire and Motion [0]. [0] https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ https://www.joelonsoftware.com/2002/01/06/fire-and-motion/
  - astrange 6y ago
    The interesting thing about Homebrew is that there were perfectly good systems before it that were more stable and better designed (Fink and Macports, neither of which install to /usr/local) and apparently the only reason people switched to Homebrew is that it was "cool" because it was written in Ruby.
    - hollerith 6y ago
      When I switched to MacOS (about 11 years ago) I tried Macports first, but ended up with Homebrew because Macports required me to learn more things than Homebrew required of me before I could successfully install the 10 or so small command-line packages I wanted to install. (At that time, Fink was lacking a maintainer IIRC.) Note that I didn't want to take the time to try to determine which one had the better design and engineering. (I didn't not know or want to learn Ruby when I made the decision.)
overgard 6y ago
I'm amazed how many things Big Sur broke. Arduino development just doesn't really work any more because they broke the kernel extension for virtual com ports. Also one app locking up seems to break the entire system (ive had dev tools freeze up and then activity monitor couldn't kill them)
- ehutch79 6y ago
  The fact that kexts were going away was not a secret, and has been a thing generally known about for YEARS.
Shadonototro 6y ago
what a shitty title... correction: "PostgreSQL needs to be updated to comply with New macOS Big Sur security API"
- maxlybbert 6y ago
  I can only speak for myself, but when I see a headline that some software update breaks another program, I don’t assign blame. I consider it shorthand for “X doesn’t work on Y, but it works in other environments.” I have to dig a little to determine if Y has a good reason for its behavior.
jandrewrogers 6y ago
It appears to be a PostgreSQL bug from my reading. The return code causing the problem has always been legitimate, even though you never expected to see it in practice. I did a quick check of the source of two other storage engines that run on both Linux and macOS to see how common this oversight is. They both had code for handling this case correctly for years. It was probably never used, but it was implemented because the syscall docs allowed for the possibility. This is the kind of thing I would have assumed PostgreSQL handles correctly as a matter of pedantry. Given how syscall heavy PostgreSQL is, it will be a fair amount of work to fix it.
- jorams 6y ago
  While open[1] is documented for macOS to be able to return EINTR, the documentation for sigaction[2] seems to specify (in confusing wording) that setting SA_RESTART prevents that. It seems to me like a bug in macOS, since they're going against their own documentation. [1]: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/open.2.html#//apple_ref/doc/man/2/open https://developer.apple.com/library/archive/documentation/Sy... [2]: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/sigaction.2.html https://developer.apple.com/library/archive/documentation/Sy... specifically: > If a signal is caught during the system calls listed below, the call may be forced to terminate with the error EINTR, the call may return with a data transfer shorter than requested, or the call may be restarted. Restart of pending calls is requested by setting the SA_RESTART bit in sa_flags. The affected system calls include open(2), read(2), write(2), sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a communications channel or a slow device (such as a terminal, but not a regular file) and during a wait(2) or ioctl(2). However, calls that have already committed are not restarted, but instead return a partial success (for example, a short read count).
  - fanf2 6y ago
    The Apple documentation matches POSIX so if open() is returning EINTR for reasons other than a signal, that is a standards conformance bug.
rsanheim 6y ago
Huh, I’ve been running PostgreSQL (version 10 tho) on an M1 since they came out and have had no problems (knock on wood). Maybe it’s only the case when importing large datasets? Or when running antivirus?installed?
commandlinefan 6y ago
My work computer upgraded itself to Big Sur last month and killed the network card: neither WiFi nor wired worked after the upgrade, no matter what I tried. IT believes I had something installed already that conflicted with the new OS, but they weren’t able to figure out what it was and ended up having to wipe the OS and downgrade back to Catalina. Say what you like about M$ or Linux, but I’ve never seen an OS upgrade with so many problems in 30 years.

227 points by devops000 6y ago

Ouch. > The issue seems to be caused by some new security APIs in Big Sur. Apple has apparently started returning with errno=EINTR from some system calls that previously never returned this error. Fixing this problem on the PostgreSQL side is not trivial, since it seems to require changes in both PostgreSQL itself and possibly in extensions as well, so it's unlikely to be fixed quickly. > These are the possiblities: > Try to disable security software (eg. Antivirus products) that might trigger the problem Downgrade to macOS 10.15 Send feedback to Apple that they should fix the issue. PostgreSQL is unlikely to be the only affected software. Hope that someone fixes the error in PostgreSQL https://github.com/PostgresApp/PostgresApp/issues/610#issuecomment-770178375 https://github.com/PostgresApp/PostgresApp/issues/610#issuec...

wheresmycraisin 6y ago
.
- deleted 6y ago
  [deleted]
  - 1_player 6y ago
    s/Postgres/macOS/
jb1991 6y ago
Those proposed fixes don't make any sense for macOS. Nearly no one runs antivirus software on a Mac, and downgrading to prior OS version is basically impossible for most people.
- wilsonrocks 6y ago
  We've been made to use Macs at work now and they have McAfee on that we can't disable, causes no end of performance issues
  - KingOfCoders 6y ago
    We also had anti-virus software on every Mac in large enterprise. Took a lot of effort to at least except source directories.
    - wilsonrocks 6y ago
      Oooh I wonder if I could get that here - things like npm install(presume any package manager, certainly homebrew) or git commands take obscenely long times.
      - joosters 6y ago
        Commands that download random 3rd party code and dependencies from the internet are precisely the kind of thing that should be included in a virus scan, surely?
        foepys 6y ago
        Especially since any npm package can execute any code at install time in npm's default configuration. On Linux this is extra juicy since you cannot globally install any package without root unless you explicitly changed the directory's permissions and/or location.
        jessaustin 6y ago
        By default, npm installs into the local project directory, as it should. Only the OS package manager should touch system directories. Before using the "--global" flag, think about what you're actually trying to do, and what the better way to do that would be. One conventional workaround is to install commonly used tools to "~/bin". Root is not required for that. Of course, it's a good idea to keep the username used for development separate from the one used for browsing the web. It wouldn't be surprising for rogue npm packages to search for e.g. credit card details. I'm sure the browsers try to obfuscate that somehow, but how much can they really defend against code that is allowed to read the disk?
tpetry 6y ago
Running docker with homebrew and on docker, no problems. I am not using antivirus as they produce many problems on mac (and are hellish slow), i guess the problem is really on triggered by antivirus software.

foepys 6y ago

.NET (Core) has also an issue with this. [0] Apple seems to intentionally break quite a lot of core APIs lately. 0: https://github.com/dotnet/runtime/issues/47584 https://github.com/dotnet/runtime/issues/47584

my123 6y ago
Returning EINTR is valid behaviour.
- deleted 6y ago
  [deleted]
- foepys 6y ago
  Even if it's valid behavior, it's still a breaking change. The function never set this error code before, now it does, and at least 3 large projects are not working anymore.
  - matwood 6y ago
    Could Apple be better at communicating changes like this? Absolutely. Is Apple in the wrong here? No. This is a documented return code that has been ignored. What's the point of a standard if they can't use it?
    - SubjectToChange 6y ago
      Apple isn't following POSIX by ignoring SA_RESTART. What is the point indeed.
  - bobbylarrybobby 6y ago
    I don’t think violating Hyrum’s law should be considered a breaking change, but I suppose that that’s a matter of opinion.
    - SubjectToChange 6y ago

toyg 6y ago

Sigh. Looks more and more likely Mojave will be the last OSX version I’ll ever use.

my123 6y ago
> Many system calls will report the EINTR error code if a signal occurred while the system call was in progress. No error actually occurred, it's just reported that way because the system isn't able to resume the system call automatically. This coding pattern simply retries the system call when this happens, to ignore the interrupt. > For instance, this might happen if the program makes use of alarm() to run some code asynchronously when a timer runs out. If the timeout occurs while the program is calling write(), we just want to retry the system call (aka read/write, etc). It's perfectly valid behaviour, and the solution is to retry in that case.
- tobiasu 6y ago
  Your solution will hang postgres at 100% CPU, with no indication what is wrong.
- jorams 6y ago
  Setting the SA_RESTART flag using sigaction[1], which Postgres does, makes it invalid behaviour. Even according to Apple's (confusingly worded) documentation[2]. [1]:https://pubs.opengroup.org/onlinepubs/9699919799/functions/sigaction.html https://pubs.opengroup.org/onlinepubs/9699919799/functions/s... [2]: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/sigaction.2.html https://developer.apple.com/library/archive/documentation/Sy...
- toyg 6y ago
  The point is that this obsession with more and more restrictions is death by a thousand cuts.
drunkpotato 6y ago
Same. Much as I’d hate to do it I’ve been eyeing a Windows system, with Linux either through the OS layer or virtualized.

ainar-g 6y ago

I wonder if EINTRs are now returned from syscalls that were always documented to (sometimes) return them in POSIX/SUS/other documentation but PostgreSQL simply ignored that possibility with Mach, or if they are returned from some completely unexpected places?

dfox 6y ago
This seems to be the case. On the other hand mostly everybody ignores the fact that open(2) is documented as possibly returning -EINTR, because it essentially never happens (in particular, network filesystem are usually implemented in a way that it will not happen even in situations when it clearly should happen).
- deleted 6y ago
  [deleted]
viraptor 6y ago
It's returned from open() https://www.postgresql.org/message-id/CAFHbkwjPXHgWb15wsmaCMuYOOF2Fm6sBVj9Sv1LbwRav75KsRg%40mail.gmail.com https://www.postgresql.org/message-id/CAFHbkwjPXHgWb15wsmaCM... https://pubs.opengroup.org/onlinepubs/9699919799/ https://pubs.opengroup.org/onlinepubs/9699919799/ Posix is happy with returning EINTR.
- cesarb 6y ago
  That documentation says "[EINTR] A signal was caught during open().", and the documentation there for sigaction with SA_RESTART says "If set, and a function specified as interruptible is interrupted by this signal, the function shall restart and shall not fail with [EINTR] unless otherwise specified.", so as long as all signals were installed with SA_RESTART (which is probably the most common case since nobody likes dealing with EINTR), open() should never fail with EINTR.
- tzs 6y ago
  This from that PostgreSQL mailing list link you give is interesting: > AUTH means that the system call is blocked (on that condition variable I mentioned about), and the user mode daemon is asked about the generated event: "postgres, pid 999, open() on file /path/file with flags 0x400003" - something like that. The usermode can either allow or deny the event by replying. The use of a user mode daemon to make decisions about whether file access is allowed reminds me of the way TOPS-10 on DEC PDP-10 did access control lists over 40 years ago. It's a method I'd like to see for ACLs on modern systems. On modern systems the ACL is part of the per file metadata. On TOPS-10 the ACL information was in a separate file that contained the ACL information for multiple files. I don't remember if is was one ACL file per user or one per directory. An ACL entry could specify file name (wildcards allowed), account and group requesting access (wildcards allowed), program that was requesting access, and what kind of access. The wildcards and the centralization of the ACLs for multiple files is what made this system excellent to use. When an access was attempted the system first checked if it was allowed by the ordinary per file permissions. If it was access was granted and the ACL daemon was not consulted. If the ordinary permissions denied access and the caller had not set a flag indicating it did not want to check the ACLs the access was denied. If that flag was not set the kernel would consult the access daemon, which would find the ACL file and check the ACLs within to find the ones that applied and make its decision.

jarym 6y ago

I run Postgres in docker containers on my Mac but I know that won’t be viable for a lot of people.

starbugs 6y ago
I have been running Postgres in Docker on my M1 MBP for a while now. No issues so far.
vetinari 6y ago
M1 docker is still in preview, so that's not an option for many folks running newer Macs.
fernandotakai 6y ago
i run linux and i prefer running all these kind of services (postgres, redis, rabbitmq and the likes) on docker containers. it's much easier to manage, easier to add/remove/clean and it doesn't pollute my local system with random stuff.
bottled_poe 6y ago
Yeah, if you are running Postgres natively on OSX instead of in Docker you are behind the times. I’m sure this will be downvoted, but it needs to be said.
- Doctor_Fegg 6y ago
  No it doesn’t. There are several Mac-native apps, such as QGIS, for which Postgres is a sensible local datastore. Running it in a container for this is an entirely unnecessary complication.
- jdboyd 6y ago
  How does permeance compare? Also that doesn't look like a good option for M1 mac users.
- slezyr 6y ago
  > you are behind the times Seriously? What is wrong with running programs natively without any layers in between? Also, IIRC PostgreSQL isn't recommended to be run in docker due to a possible data loss?

Etheryte 6y ago

PostgreSQL isn't alone in this, VirtualBox (and I'm sure other virtualization products too) ran into the new API errors head first all over the place initially as well. The common use cases more or less work now, but there are still specific setups that you simply can't run on Big Sur.

shreddit 6y ago

Quite interesting to read old stackoverflow questions to this topic which recommended “not to care about it”. https://stackoverflow.com/questions/4959524/when-to-check-for-eintr-and-repeat-the-function-call https://stackoverflow.com/questions/4959524/when-to-check-fo...

raverbashing 6y ago
Yes, those things are usually dealt with a "don't care about it..." attitude (not condoning it!) until you eventually have to.
cesarb 6y ago
That question is about Linux, which doesn't have this macOS-specific issue.

m463 6y ago

Obligatory reference to "Worse is Better" https://dreamsongs.com/RiseOfWorseIsBetter.html https://dreamsongs.com/RiseOfWorseIsBetter.html specifically the section discussing MIT vs BSD system call return codes.

layoutIfNeeded 6y ago

Seems like they're not handling EINTR for some syscalls. I remember when I was first learning Unix programming, and read about this stuff about signals and EINTR, I was baffled how much of a steaming pile of crap Unices are. Later learning about the inability to do truly non-blocking I/O was another similar moment.

ddevault 6y ago

And while they're busy breaking their current set of APIs, they're dragging their feet on adding anything designed in the last 15 years. Someone just wrote to me with a complaint because my software doesn't build on macOS due to missing getline and fmemopen implementations. Naturally I told them to use an open source operating system and declined to update my code to accomodate Apple.

my123 6y ago
getline and fmemopen are both supported on macOS. [/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk/usr/include] -> % grep -r -n getline stdio.h stdio.h:355:ssize_t getline(char \* __restrict __linep, size_t \* __restrict __linecapp, FILE \* __restrict [/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk/usr/include] -> % grep -r -n fmemopen \* stdio.h:356:FILE *fmemopen(void * __restrict __buf, size_t __size, const char \* __restrict __mode) __API_AVAILABLE(macos(10.13), ios(11.0), tvos(11.0), watchos(4.0)); getline became available on OS X 10.7 (in 2011) and fmemopen on macOS 10.13 (2017).
- ddevault 6y ago
  They were on OSX Lepoard, maybe that's not the most recent one? I don't use Mac. When were they introduced? In any case, it's a trend with Apple, even if this particular example is wrong. See also: Vulkan.
  - bombcar 6y ago
    Leopard is the final version of macOS to support the PowerPC architecture - and released in 2007. Honestly, I wouldn't expect even Linux or BSD from 2007 to work well with modern software.
    - IggleSniggle 6y ago
      This is where I really need to pay respects to Windows. It’s a heaping pile but their commitment to backwards compatibility is remarkable. I recently installed a complex application that last updated in 2007, and while performance was absolutely terrible, it worked.

DiabloD3 6y ago

So when do average developers just start considering macOS toxic and stop supporting it? About once a week, a story similar to this hits the HN front page: "$majorsoftwareproduct doesn't work on new OSX because they broke documented APIs in undocumented and unexpected ways, without any warning". What the hell, Apple?

cheph 6y ago
> So when do average developers just start considering macOS toxic and stop supporting it? Since forever ago? Nothing new here. I have 0 sympathy for people who use it. They paid dearly for their walled garden.
- treeman79 6y ago
  We love our walled garden. Don’t upgrade for a few weeks, and the hardcore fanboys will have fixes for all the annoying little breaks.
tpush 6y ago
> So when do average developers just start considering macOS toxic and stop supporting it? Presumably at the same time breathless and misinformed comments stop being posted to HN. In this case returning EINTR is documented behavior and the bug is thusly on PostgreSQL not handling it (correctly or at all).
tinus_hn 6y ago
How is returning EINTR undocumented? It’s part of the POSIX interface. It is true that typically Apple doesn’t care if they change things within the limit of the specification and 3rd party applications fail. That’s why they release beta versions.
- Sesse__ 6y ago
  Standards aside, returning EINTR on a write to local file is extremely rare in POSIX. Almost any syscall can return EINTR, but in practice, for all previous OSes, it has been reserved for those that can hang for an undetermined amount of time -- that is, anything from the network. Local I/O should complete in a bounded amount of time, so that kind of sleep is usually implemented as uninterruptable. There's one highly relevant edge case: NFS. NFS looks like local I/O, but if the NFS server goes away, what do you do? Traditionally, the process just hangs in the hope that the server will come back, which leads to the dreaded issue of processes being stuck and unkillable. On Linux, can add the “intr” flag to the mount (possibly via a remount) to allow signals to interrupt the syscall and break the deadlock -- but then you break a ton of software, because, again, EINTR on local file I/O is unheard of.

gnfargbl 6y ago

As a long-term Linux user, I have been forced on occasion over the last few years to use MacOS. I am at an almost complete loss as to what anyone sees in it. Standard utilities that work seamlessly on Linux are regularly broken, there's a weird security system that requires me to do all sorts of boot-time nonsense just to have control over my own machine... and now, apparently, they're violating the first rule of kernel development by breaking userspace. The hardware is nice, but as a platform, I just don't get it.

jolux 6y ago
Not breaking userspace is the first rule of Linux kernel development, but macOS has never had that policy as far as I know. Apple care much more about being able to move fast and clean up bad interfaces in their software, which is a priority I value. I’m not saying that’s what happened here, but I do know that Apple ultimately expects developers to pay attention to new releases and make sure their software behaves properly with them. As for standard utilities being broken, I’ve never experienced that on macOS. I do know that Macs get their standard utilities from FreeBSD and not coreutils by default.
- albertopv 6y ago
  There are liberally millions of app, Apple can't expect all of them to be always up to date on day one, to even sw like postgres. You can't just change the semantic of an API, it's just wrong. Apple should introduce a new API, deprecate the older one and remove it two or three versions later minimum. They just don't care until people is going to buy their hw.
  - my123 6y ago
    Traditionally, Apple does not guarantee total ABI stability. They push betas instead to allow people to fix the issues, and then support older releases for multiple years. Yes, unmaintained software might or might not stay fully functional on macOS for long, but that's not an Apple priority.
- my123 6y ago

monadic3 6y ago

What is the point of posix, Unix, etc if companies can just break compatibility like this without losing certification? If they don't cover useful behavior they aren't useful.

sub7 6y ago

osx has sucked big dongs for years. Everything is virtualized and slow, nothing is backwards compatible and Windows is just so much better at this point. If only those idiots at MS would disable telemetry by default

pdkl95 6y ago

Does macOS respect SA_RESTART? [1] Apple's manpage[2] for sigaction is confusing... Regardless, doesn't anybody at Apple use postgres? Breaking Postgres should be hard to miss. [1] https://pubs.opengroup.org/onlinepubs/009695399/functions/sigaction.html#tag_03_680_06 https://pubs.opengroup.org/onlinepubs/009695399/functions/si... [2] https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/sigaction.2.html https://developer.apple.com/library/archive/documentation/Sy...

joseph_grobbles 6y ago
"Regardless, doesn't anybody at Apple use postgres?" This problem occurs if you use anti-virus that delays file activities, and is 100% API consistent. Anyone saying "well it didn't do it before!" is being idiotic, similar to saying you don't need to check the return of malloc because it's usually going to succeed. pgsql runs perfectly fine.

CoffeeDregs 6y ago

I'm baffled that any developer uses a Mac. I know, I know... BSD, 3-finger swipe, but-they-wrote-the-ux-guide. But I've been running Debian Testing for 15 years now with nary a problem (on thinkpads, latitudes and desktops) I guess I'm weird in that I think gnome 3 is swell. CI handles xCode. Otherwise, my 2x 42" 4k monitors, Gnome and works-here-same-as-the-cloud self reads these articles on puzzlement. I can't count the times I've helped developers deal with oh-yeah-brew-is-weird-about-libpq. Now this? It's the avocado-toast of developer workstations...

TheRealDunkirk 6y ago
> I've been running Debian Testing for 15 years now with nary a problem I ran Linux on the desktop for 19 years, through years-long stretches of Slackware, RedHat, SuSE, Gentoo, and Ubuntu. Obviously, I was (and still am) a huge fan, but if you've managed 15 years of Linux on the desktop with "nary a problem," you have had the most unbelievable luck of any human on the planet, and should start buying lottery tickets. After running Gentoo for several years, and switching to Ubuntu, I was amazed at how much time I was saving not dealing with portage. Fair enough. I mean, I was asking for it by using a source distro, but I had that same sigh of relief when I finally bought a Mac, and realized that I was still doing a lot of maintenance to keep Ubuntu happy, even though it was (obviously) a huge improvement over Gentoo. I just couldn't see it until it was gone.
Toutouxc 6y ago
I held on to a Dell running Ubuntu for two years because I couldn't convince myself to shell out north of $2k for a comparable MacBook (I'd had a MacBook before). Then after one of the kernel updates the "resetting rcs0 for hang on rcs0" bug started happening, I had to downgrade to an earlier kernel, then Ubuntu 20 came with Snap everywhere, then covid came and I started having Zoom meetings all over the place and my mic would randomly stop working and I started using a 4K screen and noticed how choppy scrolling is and that YouTube stutters hard in fullscreen, then there was a memory leak in Firefox, semi-functional display scaling, the machine sometimes wouldn't wake up from deep sleep... Then the M1 Macs came out and I got an Air. I've been using it for a month and it's heaven. It's like coming home. It gets work done, lets me have fun, watch videos, play some games and gets out of the way.

varispeed 6y ago

Apple keeps throwing spanners in other companies wheels. Why spending time on new product development, if you have to commit teams to solve whatever this time Apple decided to break? This is anti-competitive behaviour that keeps companies working with Apple eco-system in check. Unfortunately thanks to excellent marketing, the user base is not going anywhere so companies need to put up with this.

IggleSniggle 6y ago
N of one, but this user is in the process of leaving Apple.
netdur 6y ago
Jan 23 I have started a rarely used windows laptop to do C++ development work, because my app depends on library that depends on Microsoft library that was not signed, therefor won't run with new update. we develop on macos and deploy on linux, macos is supported as side effect, windows was not supported... a single policy change without a workaround... made us support windows and linux, and macos left to dust.

joseph_grobbles 6y ago

Apple does exactly what the API is supposed to do, specifically in combination with antivirus software, and incorrectly written programs have small faults. Now read the comments before. This borders on parody at this point. It would be cool if all of the "this is the last straw!" and "welp, guess this is when I leave Mac" people who appear in each of these threads just, like, fuck off and leave then?

quotemstr 6y ago

Apple is right here and people complaining about the new behavior are wrong. If a system call can return EINTR, you must retry or deal with the error some other way. A retry is trivial too. It's irresponsible to depend on an API happening not to EINTR when it's documented to do so.

vemv 6y ago

macOS and importantly Homebrew's direction (which now doesn't support brewing from a sha-addressed github link) are leaving me at a crossroads. I value stability and autonomy over everything else - I expect the setups I work hard on scripting to work for years, freeing me of this continuous keeping up with the magical incantation du jour. As a hacker, it's particularly frustrating to feel at the whims of someone else. I think my next setup will be using my (company-issued) macbook as a thin client against a beefy unix machine. That way I'd keep a number of affordances (trackpad, iTerm, fancy Emacs etc) while having my core tooling absolutely stable. (I have the vague impression that Linux distros have also dramas and instability of their own... looks like the BSD family is more minimalistic?)

dschuler 6y ago
That's a good way of verbalizing how I've felt with the last few iterations of macOS as well as Homebrew's perpetually changing behavior. Most of it feels like change for the sake of change. I've also used FreeBSD for several years, and it's the opposite approach - everything that used to work still does, and things are well-documented. Usually the man page is enough, or the FreeBSD handbook. It's still all just Fire and Motion [0]. [0] https://www.joelonsoftware.com/2002/01/06/fire-and-motion/ https://www.joelonsoftware.com/2002/01/06/fire-and-motion/
- astrange 6y ago
  The interesting thing about Homebrew is that there were perfectly good systems before it that were more stable and better designed (Fink and Macports, neither of which install to /usr/local) and apparently the only reason people switched to Homebrew is that it was "cool" because it was written in Ruby.
  - hollerith 6y ago
    When I switched to MacOS (about 11 years ago) I tried Macports first, but ended up with Homebrew because Macports required me to learn more things than Homebrew required of me before I could successfully install the 10 or so small command-line packages I wanted to install. (At that time, Fink was lacking a maintainer IIRC.) Note that I didn't want to take the time to try to determine which one had the better design and engineering. (I didn't not know or want to learn Ruby when I made the decision.)

overgard 6y ago

I'm amazed how many things Big Sur broke. Arduino development just doesn't really work any more because they broke the kernel extension for virtual com ports. Also one app locking up seems to break the entire system (ive had dev tools freeze up and then activity monitor couldn't kill them)

ehutch79 6y ago
The fact that kexts were going away was not a secret, and has been a thing generally known about for YEARS.

Shadonototro 6y ago

what a shitty title... correction: "PostgreSQL needs to be updated to comply with New macOS Big Sur security API"

maxlybbert 6y ago
I can only speak for myself, but when I see a headline that some software update breaks another program, I don’t assign blame. I consider it shorthand for “X doesn’t work on Y, but it works in other environments.” I have to dig a little to determine if Y has a good reason for its behavior.

jandrewrogers 6y ago

It appears to be a PostgreSQL bug from my reading. The return code causing the problem has always been legitimate, even though you never expected to see it in practice. I did a quick check of the source of two other storage engines that run on both Linux and macOS to see how common this oversight is. They both had code for handling this case correctly for years. It was probably never used, but it was implemented because the syscall docs allowed for the possibility. This is the kind of thing I would have assumed PostgreSQL handles correctly as a matter of pedantry. Given how syscall heavy PostgreSQL is, it will be a fair amount of work to fix it.

jorams 6y ago
While open[1] is documented for macOS to be able to return EINTR, the documentation for sigaction[2] seems to specify (in confusing wording) that setting SA_RESTART prevents that. It seems to me like a bug in macOS, since they're going against their own documentation. [1]: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/open.2.html#//apple_ref/doc/man/2/open https://developer.apple.com/library/archive/documentation/Sy... [2]: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/sigaction.2.html https://developer.apple.com/library/archive/documentation/Sy... specifically: > If a signal is caught during the system calls listed below, the call may be forced to terminate with the error EINTR, the call may return with a data transfer shorter than requested, or the call may be restarted. Restart of pending calls is requested by setting the SA_RESTART bit in sa_flags. The affected system calls include open(2), read(2), write(2), sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a communications channel or a slow device (such as a terminal, but not a regular file) and during a wait(2) or ioctl(2). However, calls that have already committed are not restarted, but instead return a partial success (for example, a short read count).
- fanf2 6y ago
  The Apple documentation matches POSIX so if open() is returning EINTR for reasons other than a signal, that is a standards conformance bug.

rsanheim 6y ago

Huh, I’ve been running PostgreSQL (version 10 tho) on an M1 since they came out and have had no problems (knock on wood). Maybe it’s only the case when importing large datasets? Or when running antivirus?installed?

commandlinefan 6y ago

My work computer upgraded itself to Big Sur last month and killed the network card: neither WiFi nor wired worked after the upgrade, no matter what I tried. IT believes I had something installed already that conflicted with the new OS, but they weren’t able to figure out what it was and ended up having to wipe the OS and downgrade back to Catalina. Say what you like about M$ or Linux, but I’ve never seen an OS upgrade with so many problems in 30 years.