15 ms·

Bug in macOS 14 Sonoma prevents our app from working

1020 points by eptcyka 3y ago

buildbot 3y ago
Great writeup, very succinct and informative, they even have a simple reproduction of the bug. I love Mullvad! Tangentially, MacOS has had a lot of weird firewall bugs in the last few releases in general, I wonder what drive them to rip up and redo (I assume? so much of it recently.
- nhubbard 3y ago
  The rewrite was definitely influenced by the mandatory migration from kernel extensions to userspace System Extensions, specifically NetworkExtension, between Catalina and Big Sur: https://developer.apple.com/documentation/networkextension https://developer.apple.com/documentation/networkextension
  - hulitu 3y ago
    One would expect they have a test suite.
    - dclowd9901 3y ago
      Oh you poor sweet summer child
    - londons_explore 3y ago
      One would suspect their test suite is lacking...
      - MichaelZuo 3y ago
        There are a lot of errors and faults that show up in Console on a brand new MacBook just sitting on the desktop. And with every version the number seems to increase. So it's not even their test suite that's the issue.
        dmix 3y ago
        I'm curious, why do you look at random OS errors in Console to the point you noticed such a thing?
        MichaelZuo 3y ago
        When my computer can't even stay up for a full week without crashing, or suffer an inexplicable lag spike, etc., I become more motivated to closely examine everything to see what's causing the crash.
        callalex 3y ago
        Are you plugging/unplugging monitors or other hardware? With macs it’s always that for some reason.
        MichaelZuo 3y ago
        No, literally just sitting there with no peripherals attached, doing nothing more complex then playing a 4k video and web browsing.
        astrange 3y ago
        File a bug in Feedback Assistant, if you want.
        MichaelZuo 3y ago
        Okay, so?
        Moomoomoo309 3y ago
        If there's an issue, how are they supposed to know about it? Via osmosis? Even if they should have found it earlier, they didn't, so now what? You should tell them, so they can fix it.
    - lIIllIIllIIllII 3y ago
      For a firewall I'd ideally want some degree of formal verification. It's not something you fuck around with if you're serious about creating a secure product - like an operating system.
      - thghtihadanacct 3y ago
        [flagged]
      - rcme 3y ago
        But how do you verify the formal verification?
        mejutoco 3y ago
        Proving in Math that the building blocks have certain properties.
        tialaramex 3y ago
        While this is worthwhile, there is an opportunity for the math to violate your assumptions in a way you didn't understand. Let's talk briefly about TLS 1.3 Selfie attacks. TLS 1.3 is the first TLS in which experts built a formally verified model of the protocol before they shipped the standard. There are models of earlier TLS versions, but nothing learned from them could be incorporated into the corresponding standard because they were an afterthought - like writing unit tests after shipping version 1.0 TLS 1.3 has a mode intended for applications where some nodes which have a pre-existing relationship communicate, in this case we don't need the Web PKI ("SSL certificates") instead we use Pre-shared Keys (PSKs) - that is all the parties know the keys anyway, which clearly couldn't work for the open web but is fine for my boiler and its remote thermostat since duh, I don't want some random other person's thermostat talking to my boiler. PSKs are also used when you connect to the same server again later, but that doesn't matter here. The mathematical proof seemed to say exactly what the designers wanted, and it passed. So TLS 1.3 is exactly what we wanted... right? Well, almost. Designers were thinking of symmetries like "Alice talks to Bob" and "Bob talks to Alice" as one conversation, but the proof thinks that's two conversations. So the proof thinks Alice and Bob need two keys, the Alice->Bob key and the Bob->Alice key, while humans assumed they only need a single key, an Alice-x-Bob key. And so the RFC documents (prior to errata) the human assumption but the protocol actually requires the machine assumption. The result is the Selfie Attack. In our scenario Alice, and Bob have a Cat. The Cat, like many cats, would like more food, but Alice and Bob use a secure protocol to ensure they only feed the cat once. Bob has fed the cat and left. The cat is sat by an empty food bowl looking plaintive. Alice sends Bob an encrypted message, "Did you feed the cat?". The Cat intercepts the message, it doesn't have the Alice-x-Bob key so it can't decrypt this message nor directly answer it, but it doesn't need to. The Cat simply sends Alice's own message back to her. Receiving a message encrypted with the Alice-x-Bob key, Alice concludes it's from Bob. "Did you feed the cat?". So she replies "No, I did not feed the cat". The cat receives this message too, and directs it back to Alice as well, now Alice has what appears to be a reply to her first question, "No, I did not feed the cat" encrypted with the Alice-x-Bob key. So, Alice feeds the cat again because the cat was able to trick her into answering her own question.
    - skhr0680 3y ago
      from my past experience with Mac OS, the users of the .0 version are the test suite
      - rjzzleep 3y ago
        I think this pretty accurate. After messing around with the .0 versions of macOS a few times I also learned that the common wisdom is to only install OS updates on .1 versions.
        detourdog 3y ago
        It has been true since the NeXT days. .3 is for speed. Anything above .3 is a usually a bug fix.
- isodev 3y ago
  It would have been even better if they had included the rdar/Feedback number.
- voytec 3y ago
  [flagged]
  - jxf 3y ago
    If Apple is shipping that implementation as part of their OS, doesn't it make sense to let Apple know they should pick a different upstream target?
    - jojobas 3y ago
      No, you see grasshopper, Apple is infallible. If there's a problem with any of their devices or software, it's always someone else's fault. /s
  - eptcyka 3y ago
    To the best of my knowledge, OpenBSD's pf wouldn't exhibit such pathological behavior.
    - voytec 3y ago
      Exactly - it's a screwup on Apple's part, not OpenBSD's.
      - alpaca128 3y ago
        And Mullvad very clearly indicated it's connected to Apple/macOS: > bug in the macOS firewall, packet filter (PF)
        voytec 3y ago
        It's not macOS firewall, but Apple's implementation of OpenBSD's pf used in Apple's macOS. Mullvad is clearly pointing at a bug in OpenBSD's "packet filter", mentioning that it's used in macOS. Mullvad's article lacks proper wording and shits on the wrong target.
        3y ago
- tiffanyh 3y ago
  macOS has attempted to progress its networking stack for years but would run into regressions and then revert back. Old article on the topic. https://9to5mac.com/2015/05/26/apple-drops-discoveryd-in-latest-os-x-beta-following-months-of-complaints-about-network-issues-with-yosemite/ https://9to5mac.com/2015/05/26/apple-drops-discoveryd-in-lat...
  - cptcobalt 3y ago
    Drudging up 8 year old architectural decisions that Apple rightfully reverted is hardly a charitable comment. A bug can just be a bug.
- keepamovin 3y ago
  If you're concerned about these kinds of bugs on your local OS platform you may consider "abstracting away" your local connection point via a remote browser. This way, whatever your local machine and OS, you can have a dedicated server that you run your browsing through. Granted it doesn't enclose your entire network connection: only your browsing, but what it does there is change your IP address, mask your location, and add protection from browser 0 days. We're constantly adding new features add BrowserBox to respect and protect privacy and improve the overall experience. It's open source so you can change it how you want too. If you don't like AGPL-3.0 you can get a commercial license. Come take us for a spin: https://github.com/dosyago/BrowserBoxPro https://github.com/dosyago/BrowserBoxPro If you don't want something open source, but prefer the joy of a large company I think Mullvad also has their Mullvad Browser which does something similar!
  - scarmig 3y ago
    I've always imagined RBI as something primarily targeting enterprise; thanks for making it clearer the advantages for a personal end user. Do many websites end up blocking traffic if it's originating from e.g. an EC2 instance?
    - keepamovin 3y ago
      Yeah, thank you. It does actually have use cases for regular folk! About the IP: No actually. I'm not sure why that is, but I assume because a lot of premium VPN services and proxies need to end up going through IP blocks used by cloud providers as well these days. Most of the blocking I've seen comes down to how well the browser presents as a regular browser. If it looks like an automated bot browser (just raw headless, or whatever) it's more likely to get blocked in my experience than from the IP address. Although you certainly get some IP blocks or individual IPs that are more likely to get blocked. No offence to DO, but I've noticed their IPs are less reliable. I've never really had an issue with AWS, GCP, Vultr, Linode or Hetzner, except for the odd machine that somehow has an IP address that must be on a ban list, I'd estimate it around 5% probability of getting such an IP from the regular providers tho.
olliej 3y ago
[edit my scanning the article missed that they answered this :D] The obvious answer question is whether they reported this to apple already and are using this post to draw attention to it, or if they’ve found the bug in the betas (which is why betas exist) but then not reported it directly (defeating the purpose of betas)
- finitestateuni 3y ago
  If you read the article, they mention that they’ve reported the bug in previous versions of the beta and it has still not been fixed in the latest version. They’re cautioning their users against upgrading in two weeks when the release comes out of beta unless there is confirmation that the bug has been fixed.
  - rollcat 3y ago
    Beware of point 0 releases. I've had such a bad experience with iOS 13 / macOS 10.15 that I'm reluctant with the point 1's as well.
    - icedchai 3y ago
      I typically don't upgrade until the first minor macOS release. With Ventura, I waited much longer (at least 6 months.)
- deleted 3y ago
  [deleted]
- dinkblam 3y ago
  from the article: we have investigated this issue after the 6th beta was released and reported the bug to Apple
- Exuma 3y ago
  That is not an obvious question if you read the article
dinkblam 3y ago
not surprised. we've filed dozens of bug reports as every new macOS release gets worse and worse. i've given up on filing reports now, since they won't get fixed or even looked at anyway. i see the problem with triaging when around 4k issues are filed per day, but it's not like Apple is hurting for cash.
- deleted 3y ago
  [deleted]
- riscy 3y ago
  how can you know they’re not looked at? is there a read receipt?
  - eschaton 3y ago
    They can’t, they’re just presuming because Apple doesn’t communicate anything unless more information is needed or the bug is addressed in a build the reporter has access to.
- baz00 3y ago
  They actually fixed 3 bugs I raised!
  - bpoyner 3y ago
    I reported a bug in the iOS Home app and they fixed it. Seems to be hit or miss on what they'll fix.
- planb 3y ago
  I raised a bug in the image capture framework which prevented scanning from sandboxed apps and it was fixed 3 betas later. But probably because Preview.app was also affected and I asked all users of my software to file a bug for the Preview app.
- superlupo 3y ago
  I've basically given up reporting bugs with Apple as they just seem to be ignored and either never fixed, or fixed some years later when the corresponding component is completely rewritten. I basically resent filing bugs with companies that have enough money to do proper testing, I don't want to work for them for free, especially if there is no answer, or a 1st-level answer who hasn't even tried the filed repro case. However, I am happily reporting bugs with open source projects.
Szpadel 3y ago
I don't see any action points in blog post so I'm not sure if they do it, but some warning in app when specific version of MacOS is detected or even blocking functionality that is known to be leaky would be great for anyone that might track blog posts. plus maybe link to this blog post in case MacOS resolves issue but app will not be updated
Terretta 3y ago
This repro is a thing of beauty.
- coldtea 3y ago
  Isn't it merely setting a simple firewall rule and trying a query that violates it? Which is the scope of the bug sure. But doesn't make the check particularly elaborate or beautiful!
  - jihadjihad 3y ago
    > But doesn't make the check particularly elaborate or beautiful! I think GP is saying it's beautiful precisely because it needs such a simple and not elaborate test
- jjcm 3y ago
  I love how simple it is, but also that it has a cleanup step as well! Such a missed element in many of these.
joomooru 3y ago
No wonder, I recently installed the latest sonoma beta and couldn't for the life of me get Mullvad to work. Glad to hear Mullvad is working on a workaround. I even considered downgrading back to Ventura this morning. I feel validated!
- deleted 3y ago
  [deleted]
- sleepybrett 3y ago
  I assume the tailscale/mullvad stuff still works. Might be a nice workaround until apple gets it fixed.
  - joomooru 3y ago
    Went to look at tailscale vpn and didn't realize it was a service entirely contained within tailscale, so I can't use my existing Mullvad account or credit tailscale with my already-purchased Mullvad credits :(
    - sleepybrett 3y ago
      Aw, that sucks. Maybe possible through Mullvad support?
      - ezfe 3y ago
        They've said they don't plan on allowing that
        gbraad 3y ago
        I have seen a case where they substituted the credits on Mullvad to the add-on for Tailscale, but this might not be a regular thing. Although, it is also still in Beta and issues have been reported/known, so similarly if you look for a stable solution it might be best to hold out for now.
        3y ago
callmeal 3y ago
[flagged]
- detourdog 3y ago
  or the process they expect is to boot oustisde SIP.
- LeoPanthera 3y ago
  This kind of cynicism is tiresome. The test case involves pinging Mullvad, not Apple. If Apple wanted no filtering to be possible, they would simply remove pf entirely.
paws 3y ago
The more macOS seems to break user control of networking, the more I wonder what kind of "separate box" solutions are out there that can intermediate _outgoing_ traffic. e.g. Something like LittleSnitch on a router, where it notifies the Mac when it detects a new outgoing connection. Do things e.g. pfSense support that already? "Hold" an outgoing connection from the moment the SYN is observed, notify whatever client, and only allow if the user clicks?
- _boffin_ 3y ago
  Interesting
- fiddlerwoaroof 3y ago
  Most of the alternatives that aren’t marketed to the consumer immediately have something. I ran openwrt for years and used its firewall to block a bunch of traffic and now I’ve switched to Ubiquiti because of wifi issues.
- deleted 3y ago
  [deleted]
- smashed 3y ago
  > Do things e.g. pfSense support that already? "Hold" an outgoing connection from the moment the SYN is observed, notify whatever client, and only allow if the user clicks? Not that I am aware of. This is a desktop centric workflow where the user can react live to an application that is sending traffic. Your typical network firewall will apply a set of static rules and the decision to log/reject/drop is done ASAP. Waiting for user input is impossible. Some systems can show logs of recent blocked traffic, and allow an admin to quickly generate an exception/allow rule for blocked traffic but that's pretty much it.
  - jerf 3y ago
    With some work you could integrate the two; create an external box paired with an app on your source machine. When the source machine can see more info about the connection it can cooperate to pass it along to the middlebox, and if the middlebox sees something the source system has no idea about, well, that's useful info to annotate the network connection with.
frankjr 3y ago
Speaking of macOS's firewall being broken, there's a bug (or "feature"?) in the NetworkExtension framework which causes connections to get initiated (SYN, leaks your IP address) even if there's an explicit rule to deny that connection. This affects LittleSnitch, Lulu, and all the other apps building on top of the framework. Bug reports have been filed and as usual ignored by Apple. More: Little Snitch "denied" connections leak your IP address - https://lapcatsoftware.com/articles/2023/3/4.html https://lapcatsoftware.com/articles/2023/3/4.html Follow-up to Little Snitch "denied" connections leak your IP address - https://lapcatsoftware.com/articles/2023/3/5.html https://lapcatsoftware.com/articles/2023/3/5.html Little Snitch "denied" connections leak your IP address: Developer response - https://lapcatsoftware.com/articles/2023/6/3.html https://lapcatsoftware.com/articles/2023/6/3.html
- deleted 3y ago
  [deleted]
- sneak 3y ago
  The situation is even worse on iOS where certain connections that exist in connection tracking early in boot will forever bypass your VPN. Apple doesn't seem to care that much about privacy applications of their OSes.
  - frankjr 3y ago
    > Apple doesn't seem to care that much about privacy applications of their OSes. If I remember correctly, macOS (iOS as well?) also sends information about the application you are launching back to Apple from time to time. It doesn't contain the application's name directly but it does contain information about the developer of the app (their certificate; multiple apps can technically share the same developer though). They do this to be able to prevent the app from launching in case the developer has been done something nefarious. Bonus points go to Apple for making these requests over plain text connections, no pesky TLS - let the whole network see!
    - sneak 3y ago
      That was probably from my website: https://sneak.berlin/20201112/your-computer-isnt-yours/ https://sneak.berlin/20201112/your-computer-isnt-yours/ Apple committed on April 30, 2021 to sending these revocation checks over a new encrypted protocol within one year. AFAIK that turned out to be a lie, as there have been two OS releases since then and plaintext OCSP remains a thing. https://support.apple.com/en-us/HT202491 https://support.apple.com/en-us/HT202491 ARM macOS also still gets boot tickets from the TSS server (which include permanent HW serials like chip ECID) via plaintext tcp/80 HTTP on every OS update.
Pesthuf 3y ago
Normally, when software doesn't have big changes between releases, like macOS, there's a feature freeze and devs are working on bugfixes. Yet, macOS seems to get buggier and buggier between releases. Something about the way it's being developed right now is going very wrong.
- rollcat 3y ago
  What's interesting is that there was a PF bug in FreeBSD not long ago as well: <https://www.enricobassetti.it/2023/09/cve-2023-4809-freebsd-pf-bypass-when-using-ipv6/ https://www.enricobassetti.it/2023/09/cve-2023-4809-freebsd-...>; <https://news.ycombinator.com/item?id=37437530 https://news.ycombinator.com/item?id=37437530>.
- deleted 3y ago
  [deleted]
- hulitu 3y ago
  > Yet, macOS seems to get buggier and buggier between releases. Something about the way it's being developed right now is going very wrong This is a common trend in the SW development in the last decade. Aparently bug fixing is hard and expensive, that's why they concentrate on new features or, in extreme cases, complete rewrites (GTK).
  - steve1977 3y ago
    https://www.jwz.org/doc/cadt.html https://www.jwz.org/doc/cadt.html
- pohl 3y ago
  I see more and more complaints about bugs, anyway. I use it all day every day without issues, though. Not sure what to make of that. It’s not that I don’t believe those who are affected. I’m just not sure one can conclude that it’s getting more buggy with each release. Maybe the number of unusual use cases increases with popularity?
  - deergomoo 3y ago
FollowingTheDao 3y ago
[flagged]
- deleted 3y ago
  [deleted]
nvahalik 3y ago
Anyone else get an invalid certificate on this site? What's the deal? ETA: Hm. OpenDNS (family) blocks them...
- deleted 3y ago
  [deleted]
- ezfe 3y ago
  makes sense that blocking software would block VPN software
- mzs 3y ago
  https://web.archive.org/web/20230913161315/https://mullvad.net/en/blog/2023/9/13/bug-in-macos-14-sonoma-prevents-our-app-from-working/ https://web.archive.org/web/20230913161315/https://mullvad.n...
keehun 3y ago
I'm glad Mullvad is raising the public temperature on this! This one has definitely been noticed and been very concerning.
- scosman 3y ago
  Has this been noted elsewhere? Sounds like Mulvad reported after the 6th which is pretty close to the RC. From source: "we have investigated this issue after the 6th beta was released and reported the bug to Apple"
  - keehun 3y ago
    Not publicly that I have seen, but I can assure you networking and cybersecurity companies (and others) saw this pretty quickly when the bug was first released. I was just glad to see a relatively big company calling out this rather egregious issue.
    - LeoNatan25 3y ago
      Security companies should be much more open about these issues, rather than quake the notion that if they go public, they’d lose their hush hush secret contacts at Apple that give them private entitlements for private functionality. (Source: first hand experience)
      - pdimitar 3y ago
        Your comment is pretty vague but intriguing. Are you allowed to share an example?
        mrcode007 3y ago
        they're called managed capabilities and require apple's approval for unlocking access. CarPlay is an example: https://developer.apple.com/documentation/carplay/requesting_carplay_entitlements https://developer.apple.com/documentation/carplay/requesting... edit: tap to pay is another: https://developer.apple.com/documentation/proximityreader/setting-up-the-entitlement-for-tap-to-pay-on-iphone https://developer.apple.com/documentation/proximityreader/se...
unnouinceput 3y ago
We, the old Windows developers, welcome you, the current Apple developers, to the 90's, when Windows was shittier and shittier with each version. Get ready for the next decade when workarounds and basically underground techniques will be your only survivability. As MacOS becomes more popular, it seems it has to go to this shitty phase, as Windows did back in the day. We got rid of this phase with Windows XP release, so around 7 years. For you, who knows, hopefully shorter.
- whyenot 3y ago
  It's been 23 years (to the day!) since the release of the OS X public beta, and it's a mature product. I'm not sure it's getting "shittier and shittier," I think there are still refinements and improvements, they just aren't as big as they used to be.
  - sumuyuda 3y ago
    The UI has definitely gotten shittier and shittier.
    - heyoni 3y ago
      And system settings panel is so unresponsive! I think it’s written in react or something?
    - whyenot 3y ago
      I don't know about that. Aqua with it's pin stripes jewel-like buttons and other quirks was significantly worse than what we have today.
      - can16358p 3y ago
        Yup. I absolutely hated that skeuomorphic blurry 3D-like design. Flat design looks much, much cleaner. Same for iOS. iOS 7 was the first version that I actually liked looking at.
      - BizarreByte 3y ago
zshrc 3y ago
Just a note, while I experienced issues connecting with the Mullvad.app, running a Mullvad Wireguard config in Wireguard.app worked fine.
voytec 3y ago
[flagged]
- thedanbob 3y ago
  But it's not a misconfiguration, it's a bug as the article explains. And it's not the author's fault that Apple named their firewall the same as OpenBSD.
  - voytec 3y ago
    It's not a case of naming something similarily to other software. OSX used FreeBSD's ipfw and around the time they renamed the OS to macOS, they switched to OpenBSD's pf. Now they've screwed up either configuration or implementation but to me - it doesn't read like a bug in pf.
    - fullspectrumdev 3y ago
      It’s a bug in the program named pf on macOS. It’s not that deep. Nobody is blaming OpenBSD’s pf here.
  - Khaine 3y ago
    I believe Apple used FreeBSD's implementation of pf, as it also has the same syntax. OpenBSD pf has evolved since then and their are minor syntactic differences for some rules between freebsd pf and openbsd pf.
CharlesW 3y ago
Mullvad has been working okay for me in Sonoma betas (I had to click "Connect" twice), and appears to work perfectly in the final 14.0 release (23A339). A test with ipleak.net looks normal. What am I doing right/wrong?
- deleted 3y ago
  [deleted]
- contact9879 3y ago
  it works but requires local network connections to be enabled which I assume is the "leak" they note
panosv 3y ago
Not completely relevant, but another long standing bug: An 11 Year Old Bug in the macOS Popen(): https://news.ycombinator.com/item?id=37238433 https://news.ycombinator.com/item?id=37238433
- saagarjha 3y ago
  If this is your bug, consider also sending in a feedback with your patch. The open source projects don’t usually take PRs.
  - panosv 3y ago
    We had a discussion with one of Apple's moderators here: https://developer.apple.com/forums/thread/726713 https://developer.apple.com/forums/thread/726713 If you have a better way to reach them, lmk.
    - krackers 3y ago
      That's more than "one of Apple's moderators", that's the legendary Quinn (and the _only_ person within Apple I've seen to respond on the dev forums). Also if you do need to reach someone at apple, I think filing a DTS incident would do that.
      - saagarjha 3y ago
        There are a handful of Apple engineers who will reply in the forums for their products.
    - saagarjha 3y ago
      That's about the best thing short of knowing someone on that team, I think.
  - dolmen 3y ago
    From the forum discussion, this has been reported as FB12144217.
m3kw9 3y ago
New OSs can sink your company
thehours 3y ago
I don’t know if related, but immediately the last two MacOS upgrades I was unable to get my networking to work. I could connect to Wi-Fi / Ethernet / Hotspot. But nothing would actually connect (e.g. browser, pings, etc) , not even to my router. The fix both times was to open the Mullvad VPN app just once and everything worked again. No idea why just opening the app would fix the issue.
- oefrha 3y ago
  VPN apps make changes to the routing table, I suppose your traffic was routed to a nonexistent interface until the VPN app was launched.
deleted 3y ago
[deleted]
throwaway914 3y ago
How does one do the equivalent in Linux with nftables? It's very cool that you can create a logging interface.
- dharmab 3y ago
  In Linux you can use eBPF. See https://github.com/iovisor/bcc https://github.com/iovisor/bcc for an easy way to write eBPF, or look for something in the tools/ dir that does what you want. You distro might have these packaged in bcc-tools or similar.
mannyv 3y ago
Question: if pflog1 isn't created, does pf work as expected? Since tcpdump presumably isn't logging anything going out of pflog1, it implies that the data isn't being sent to pflog1. That implies that it's something upstream from there. Was pflog1 plumbed and/or pulled up? Back in the day you used to have to plumb and pull up an interface manually. Does MacOS do that for you? Granted, isolating this isn't the bug reporter's job. But at some point the engineer will be asking these questions, so you might as well answer them.
- MuffinFlavored 3y ago
  Is there something you could add to the reproduce case to workaround this, like plumb/pull up after pfrules are loaded as you mentioned?
  - mannyv 3y ago
    On Ventura 13.5.2 ifconfig create creates the interface UP: bash-3.2# ifconfig pflog1 create bash-3.2# ifconfig -a pflog1: flags=1<UP> mtu 33072 But I don't think tcpdump will see anything because there's no IP associated with that interface. On some devices (Broadcom) the OS will just dump stuff out of an interface, but I'm not sure what MacOS does in this case. I presume this was tested to work on 13.5.2?
    - AdamJacobMuller 3y ago
      tcpdump can see traffic fine without an IP on an interface (how much traffic the interface has traffic without one is another question) pflog interface is not a "real" interface, there's no point in putting an IP address on it
      - mannyv 3y ago
        Oh, pflog interfaces are a special device that's known by tcpdump. Interesting.
        3y ago
OJFord 3y ago
If you use their wireguard servers, you can use wireguard directly (no Mullvad) app, and this is probably not an issue? I'm not certain and don't have a (modern) Mac, so I can't test. Just a reminder that it's nothing proprietary and the app is optional I suppose.
- derefr 3y ago
  I would presume that the thing Mullvad are (rightfully) doing with firewall rules in their app, is to prevent non-VPN traffic from flowing whenever the VPN is enabled. The whole “leaking your IP through DNS / privileged connections to OS update servers / etc.” issue.
  - OJFord 3y ago
    I'm not saying there's no value in the app, but there must be other ways of achieving that (though, maybe they're broken by the same). Do you have `ip netns` on macOS? Fwmark? There's the WireGuard app too of course, I wonder if that's similarly broken.
    - fastball 3y ago
      The way of achieving that is through the Packet Filter, which is what is broken. It seems unlikely you can block system-level traffic with anything except system-level APIs.
      - hnarn 3y ago
        I'm not in any way excusing the bug, but what about routing? Even if the firewall is broken, if you have a wg0 interface and your only route is through that interface, wouldn't that also achieve the same goal?
kdrag0n 3y ago
The same PF bug is breaking one of OrbStack's networking features. I found it hard to believe when I narrowed it down to this, but I guess I'm not alone. Really hoping this is fixed before the stable release. I didn't get a chance to report this to Apple until yesterday, but I think it's a fairly recent regression, probably from around beta 6. PF is supposed to be a last-match firewall but it's almost like macOS is doing first-match now: an earlier "block" rule (without "quick") is overriding "pass" rules in a different anchor, which obviously breaks things.
- bjoli 3y ago
  "really hope" is too weak. This has to be fixed before release. A stable release with this would be unacceptable and would be reason for me to abandon Mac OS. You don't deliver a product with this kind of regression if you want it to be used for any kind of serious business
  - nikanj 3y ago
    Maybe you don't deliver products like that, but Apple delivers them all the time. Then you have to desperately raise noise on all possible forums, and hope to catch the eye of someone who actually works at a relevant team at Apple, before your customers get too tired of the issues caused by the bug and abandon you. This is not an Apple-specific issue either, Microsoft works just the same. Large OS releases are a massive undertaking, and the release train don't stop for a firewall bug - no matter how severe the bug is for the people it does affect.
    - szundi 3y ago
      I remember the time around iOS 5-6, they made some delays and probably it fired back on them and since they have delay related PTSD as an organization. I think since then they just have to push whatever it takes. I remember after iOS 7 I almost left the ecosystem it was so buggy and almost unusable. They probably measured it because after maybe iOS 9-10 they started to be very proud of fixing and stabilizing stuff even in product announcement and iOS is very stable now, not recognizing an issue for years maybe. But this had a price, they paid it, now they seem very aggressive on schedules until the next major clusterfucky year.
TheBinaryGuy 3y ago
For me the solution was to download Wireguard configs and use the Wireguard app.
- cyclecount 3y ago
  But is the Wireguard app leaking data and just less secure than Mullvad's app?
- computerfriend 3y ago
  This is only a solution if you don't set PostUp and PostDown firewall rules in your WireGuard config. Which isn't much of a solution.
maxehmookau 3y ago
Does this also apply to the new iOS? Since upgrading to the beta, I've been unable to use Mullvad.
anacrolix 3y ago
Mullvad VPN breaks DNS over IPv6 https://github.com/golang/go/issues/61060 https://github.com/golang/go/issues/61060

1020 points by eptcyka 3y ago

Great writeup, very succinct and informative, they even have a simple reproduction of the bug. I love Mullvad! Tangentially, MacOS has had a lot of weird firewall bugs in the last few releases in general, I wonder what drive them to rip up and redo (I assume? so much of it recently.

nhubbard 3y ago
The rewrite was definitely influenced by the mandatory migration from kernel extensions to userspace System Extensions, specifically NetworkExtension, between Catalina and Big Sur: https://developer.apple.com/documentation/networkextension https://developer.apple.com/documentation/networkextension
- hulitu 3y ago
  One would expect they have a test suite.
  - dclowd9901 3y ago
    Oh you poor sweet summer child
  - londons_explore 3y ago
    One would suspect their test suite is lacking...
    - MichaelZuo 3y ago
      There are a lot of errors and faults that show up in Console on a brand new MacBook just sitting on the desktop. And with every version the number seems to increase. So it's not even their test suite that's the issue.
      - dmix 3y ago
        I'm curious, why do you look at random OS errors in Console to the point you noticed such a thing?
        MichaelZuo 3y ago
        When my computer can't even stay up for a full week without crashing, or suffer an inexplicable lag spike, etc., I become more motivated to closely examine everything to see what's causing the crash.
        callalex 3y ago
        Are you plugging/unplugging monitors or other hardware? With macs it’s always that for some reason.
        MichaelZuo 3y ago
        No, literally just sitting there with no peripherals attached, doing nothing more complex then playing a 4k video and web browsing.
        astrange 3y ago
        File a bug in Feedback Assistant, if you want.
        MichaelZuo 3y ago
        Okay, so?
        Moomoomoo309 3y ago
        If there's an issue, how are they supposed to know about it? Via osmosis? Even if they should have found it earlier, they didn't, so now what? You should tell them, so they can fix it.
  - lIIllIIllIIllII 3y ago
    For a firewall I'd ideally want some degree of formal verification. It's not something you fuck around with if you're serious about creating a secure product - like an operating system.
    - thghtihadanacct 3y ago
      [flagged]
    - rcme 3y ago
      But how do you verify the formal verification?
      - mejutoco 3y ago
        Proving in Math that the building blocks have certain properties.
        tialaramex 3y ago
        While this is worthwhile, there is an opportunity for the math to violate your assumptions in a way you didn't understand. Let's talk briefly about TLS 1.3 Selfie attacks. TLS 1.3 is the first TLS in which experts built a formally verified model of the protocol before they shipped the standard. There are models of earlier TLS versions, but nothing learned from them could be incorporated into the corresponding standard because they were an afterthought - like writing unit tests after shipping version 1.0 TLS 1.3 has a mode intended for applications where some nodes which have a pre-existing relationship communicate, in this case we don't need the Web PKI ("SSL certificates") instead we use Pre-shared Keys (PSKs) - that is all the parties know the keys anyway, which clearly couldn't work for the open web but is fine for my boiler and its remote thermostat since duh, I don't want some random other person's thermostat talking to my boiler. PSKs are also used when you connect to the same server again later, but that doesn't matter here. The mathematical proof seemed to say exactly what the designers wanted, and it passed. So TLS 1.3 is exactly what we wanted... right? Well, almost. Designers were thinking of symmetries like "Alice talks to Bob" and "Bob talks to Alice" as one conversation, but the proof thinks that's two conversations. So the proof thinks Alice and Bob need two keys, the Alice->Bob key and the Bob->Alice key, while humans assumed they only need a single key, an Alice-x-Bob key. And so the RFC documents (prior to errata) the human assumption but the protocol actually requires the machine assumption. The result is the Selfie Attack. In our scenario Alice, and Bob have a Cat. The Cat, like many cats, would like more food, but Alice and Bob use a secure protocol to ensure they only feed the cat once. Bob has fed the cat and left. The cat is sat by an empty food bowl looking plaintive. Alice sends Bob an encrypted message, "Did you feed the cat?". The Cat intercepts the message, it doesn't have the Alice-x-Bob key so it can't decrypt this message nor directly answer it, but it doesn't need to. The Cat simply sends Alice's own message back to her. Receiving a message encrypted with the Alice-x-Bob key, Alice concludes it's from Bob. "Did you feed the cat?". So she replies "No, I did not feed the cat". The cat receives this message too, and directs it back to Alice as well, now Alice has what appears to be a reply to her first question, "No, I did not feed the cat" encrypted with the Alice-x-Bob key. So, Alice feeds the cat again because the cat was able to trick her into answering her own question.
  - skhr0680 3y ago
    from my past experience with Mac OS, the users of the .0 version are the test suite
    - rjzzleep 3y ago
      I think this pretty accurate. After messing around with the .0 versions of macOS a few times I also learned that the common wisdom is to only install OS updates on .1 versions.
      - detourdog 3y ago
        It has been true since the NeXT days. .3 is for speed. Anything above .3 is a usually a bug fix.
isodev 3y ago
It would have been even better if they had included the rdar/Feedback number.
voytec 3y ago
[flagged]
- jxf 3y ago
  If Apple is shipping that implementation as part of their OS, doesn't it make sense to let Apple know they should pick a different upstream target?
  - jojobas 3y ago
    No, you see grasshopper, Apple is infallible. If there's a problem with any of their devices or software, it's always someone else's fault. /s
- eptcyka 3y ago
  To the best of my knowledge, OpenBSD's pf wouldn't exhibit such pathological behavior.
  - voytec 3y ago
    Exactly - it's a screwup on Apple's part, not OpenBSD's.
    - alpaca128 3y ago
      And Mullvad very clearly indicated it's connected to Apple/macOS: > bug in the macOS firewall, packet filter (PF)
      - voytec 3y ago
        It's not macOS firewall, but Apple's implementation of OpenBSD's pf used in Apple's macOS. Mullvad is clearly pointing at a bug in OpenBSD's "packet filter", mentioning that it's used in macOS. Mullvad's article lacks proper wording and shits on the wrong target.
        3y ago
tiffanyh 3y ago
macOS has attempted to progress its networking stack for years but would run into regressions and then revert back. Old article on the topic. https://9to5mac.com/2015/05/26/apple-drops-discoveryd-in-latest-os-x-beta-following-months-of-complaints-about-network-issues-with-yosemite/ https://9to5mac.com/2015/05/26/apple-drops-discoveryd-in-lat...
- cptcobalt 3y ago
  Drudging up 8 year old architectural decisions that Apple rightfully reverted is hardly a charitable comment. A bug can just be a bug.
keepamovin 3y ago
If you're concerned about these kinds of bugs on your local OS platform you may consider "abstracting away" your local connection point via a remote browser. This way, whatever your local machine and OS, you can have a dedicated server that you run your browsing through. Granted it doesn't enclose your entire network connection: only your browsing, but what it does there is change your IP address, mask your location, and add protection from browser 0 days. We're constantly adding new features add BrowserBox to respect and protect privacy and improve the overall experience. It's open source so you can change it how you want too. If you don't like AGPL-3.0 you can get a commercial license. Come take us for a spin: https://github.com/dosyago/BrowserBoxPro https://github.com/dosyago/BrowserBoxPro If you don't want something open source, but prefer the joy of a large company I think Mullvad also has their Mullvad Browser which does something similar!
- scarmig 3y ago
  I've always imagined RBI as something primarily targeting enterprise; thanks for making it clearer the advantages for a personal end user. Do many websites end up blocking traffic if it's originating from e.g. an EC2 instance?
  - keepamovin 3y ago
    Yeah, thank you. It does actually have use cases for regular folk! About the IP: No actually. I'm not sure why that is, but I assume because a lot of premium VPN services and proxies need to end up going through IP blocks used by cloud providers as well these days. Most of the blocking I've seen comes down to how well the browser presents as a regular browser. If it looks like an automated bot browser (just raw headless, or whatever) it's more likely to get blocked in my experience than from the IP address. Although you certainly get some IP blocks or individual IPs that are more likely to get blocked. No offence to DO, but I've noticed their IPs are less reliable. I've never really had an issue with AWS, GCP, Vultr, Linode or Hetzner, except for the odd machine that somehow has an IP address that must be on a ban list, I'd estimate it around 5% probability of getting such an IP from the regular providers tho.

olliej 3y ago

[edit my scanning the article missed that they answered this :D] The obvious answer question is whether they reported this to apple already and are using this post to draw attention to it, or if they’ve found the bug in the betas (which is why betas exist) but then not reported it directly (defeating the purpose of betas)

finitestateuni 3y ago
If you read the article, they mention that they’ve reported the bug in previous versions of the beta and it has still not been fixed in the latest version. They’re cautioning their users against upgrading in two weeks when the release comes out of beta unless there is confirmation that the bug has been fixed.
- rollcat 3y ago
  Beware of point 0 releases. I've had such a bad experience with iOS 13 / macOS 10.15 that I'm reluctant with the point 1's as well.
  - icedchai 3y ago
    I typically don't upgrade until the first minor macOS release. With Ventura, I waited much longer (at least 6 months.)
deleted 3y ago
[deleted]
dinkblam 3y ago
from the article: we have investigated this issue after the 6th beta was released and reported the bug to Apple
Exuma 3y ago
That is not an obvious question if you read the article

dinkblam 3y ago

not surprised. we've filed dozens of bug reports as every new macOS release gets worse and worse. i've given up on filing reports now, since they won't get fixed or even looked at anyway. i see the problem with triaging when around 4k issues are filed per day, but it's not like Apple is hurting for cash.

deleted 3y ago
[deleted]
riscy 3y ago
how can you know they’re not looked at? is there a read receipt?
- eschaton 3y ago
  They can’t, they’re just presuming because Apple doesn’t communicate anything unless more information is needed or the bug is addressed in a build the reporter has access to.
baz00 3y ago
They actually fixed 3 bugs I raised!
- bpoyner 3y ago
  I reported a bug in the iOS Home app and they fixed it. Seems to be hit or miss on what they'll fix.
planb 3y ago
I raised a bug in the image capture framework which prevented scanning from sandboxed apps and it was fixed 3 betas later. But probably because Preview.app was also affected and I asked all users of my software to file a bug for the Preview app.
superlupo 3y ago
I've basically given up reporting bugs with Apple as they just seem to be ignored and either never fixed, or fixed some years later when the corresponding component is completely rewritten. I basically resent filing bugs with companies that have enough money to do proper testing, I don't want to work for them for free, especially if there is no answer, or a 1st-level answer who hasn't even tried the filed repro case. However, I am happily reporting bugs with open source projects.

Szpadel 3y ago

I don't see any action points in blog post so I'm not sure if they do it, but some warning in app when specific version of MacOS is detected or even blocking functionality that is known to be leaky would be great for anyone that might track blog posts. plus maybe link to this blog post in case MacOS resolves issue but app will not be updated

Terretta 3y ago

This repro is a thing of beauty.

coldtea 3y ago
Isn't it merely setting a simple firewall rule and trying a query that violates it? Which is the scope of the bug sure. But doesn't make the check particularly elaborate or beautiful!
- jihadjihad 3y ago
  > But doesn't make the check particularly elaborate or beautiful! I think GP is saying it's beautiful precisely because it needs such a simple and not elaborate test
jjcm 3y ago
I love how simple it is, but also that it has a cleanup step as well! Such a missed element in many of these.

joomooru 3y ago

No wonder, I recently installed the latest sonoma beta and couldn't for the life of me get Mullvad to work. Glad to hear Mullvad is working on a workaround. I even considered downgrading back to Ventura this morning. I feel validated!

deleted 3y ago
[deleted]
sleepybrett 3y ago
I assume the tailscale/mullvad stuff still works. Might be a nice workaround until apple gets it fixed.
- joomooru 3y ago
  Went to look at tailscale vpn and didn't realize it was a service entirely contained within tailscale, so I can't use my existing Mullvad account or credit tailscale with my already-purchased Mullvad credits :(
  - sleepybrett 3y ago
    Aw, that sucks. Maybe possible through Mullvad support?
    - ezfe 3y ago
      They've said they don't plan on allowing that
      - gbraad 3y ago
        I have seen a case where they substituted the credits on Mullvad to the add-on for Tailscale, but this might not be a regular thing. Although, it is also still in Beta and issues have been reported/known, so similarly if you look for a stable solution it might be best to hold out for now.
      - 3y ago

callmeal 3y ago

[flagged]

detourdog 3y ago
or the process they expect is to boot oustisde SIP.
LeoPanthera 3y ago
This kind of cynicism is tiresome. The test case involves pinging Mullvad, not Apple. If Apple wanted no filtering to be possible, they would simply remove pf entirely.

paws 3y ago

The more macOS seems to break user control of networking, the more I wonder what kind of "separate box" solutions are out there that can intermediate _outgoing_ traffic. e.g. Something like LittleSnitch on a router, where it notifies the Mac when it detects a new outgoing connection. Do things e.g. pfSense support that already? "Hold" an outgoing connection from the moment the SYN is observed, notify whatever client, and only allow if the user clicks?

_boffin_ 3y ago
Interesting
fiddlerwoaroof 3y ago
Most of the alternatives that aren’t marketed to the consumer immediately have something. I ran openwrt for years and used its firewall to block a bunch of traffic and now I’ve switched to Ubiquiti because of wifi issues.
deleted 3y ago
[deleted]
smashed 3y ago
> Do things e.g. pfSense support that already? "Hold" an outgoing connection from the moment the SYN is observed, notify whatever client, and only allow if the user clicks? Not that I am aware of. This is a desktop centric workflow where the user can react live to an application that is sending traffic. Your typical network firewall will apply a set of static rules and the decision to log/reject/drop is done ASAP. Waiting for user input is impossible. Some systems can show logs of recent blocked traffic, and allow an admin to quickly generate an exception/allow rule for blocked traffic but that's pretty much it.
- jerf 3y ago
  With some work you could integrate the two; create an external box paired with an app on your source machine. When the source machine can see more info about the connection it can cooperate to pass it along to the middlebox, and if the middlebox sees something the source system has no idea about, well, that's useful info to annotate the network connection with.

frankjr 3y ago

Speaking of macOS's firewall being broken, there's a bug (or "feature"?) in the NetworkExtension framework which causes connections to get initiated (SYN, leaks your IP address) even if there's an explicit rule to deny that connection. This affects LittleSnitch, Lulu, and all the other apps building on top of the framework. Bug reports have been filed and as usual ignored by Apple. More: Little Snitch "denied" connections leak your IP address - https://lapcatsoftware.com/articles/2023/3/4.html https://lapcatsoftware.com/articles/2023/3/4.html Follow-up to Little Snitch "denied" connections leak your IP address - https://lapcatsoftware.com/articles/2023/3/5.html https://lapcatsoftware.com/articles/2023/3/5.html Little Snitch "denied" connections leak your IP address: Developer response - https://lapcatsoftware.com/articles/2023/6/3.html https://lapcatsoftware.com/articles/2023/6/3.html

deleted 3y ago
[deleted]
sneak 3y ago
The situation is even worse on iOS where certain connections that exist in connection tracking early in boot will forever bypass your VPN. Apple doesn't seem to care that much about privacy applications of their OSes.
- frankjr 3y ago
  > Apple doesn't seem to care that much about privacy applications of their OSes. If I remember correctly, macOS (iOS as well?) also sends information about the application you are launching back to Apple from time to time. It doesn't contain the application's name directly but it does contain information about the developer of the app (their certificate; multiple apps can technically share the same developer though). They do this to be able to prevent the app from launching in case the developer has been done something nefarious. Bonus points go to Apple for making these requests over plain text connections, no pesky TLS - let the whole network see!
  - sneak 3y ago
    That was probably from my website: https://sneak.berlin/20201112/your-computer-isnt-yours/ https://sneak.berlin/20201112/your-computer-isnt-yours/ Apple committed on April 30, 2021 to sending these revocation checks over a new encrypted protocol within one year. AFAIK that turned out to be a lie, as there have been two OS releases since then and plaintext OCSP remains a thing. https://support.apple.com/en-us/HT202491 https://support.apple.com/en-us/HT202491 ARM macOS also still gets boot tickets from the TSS server (which include permanent HW serials like chip ECID) via plaintext tcp/80 HTTP on every OS update.

Pesthuf 3y ago

Normally, when software doesn't have big changes between releases, like macOS, there's a feature freeze and devs are working on bugfixes. Yet, macOS seems to get buggier and buggier between releases. Something about the way it's being developed right now is going very wrong.

rollcat 3y ago
What's interesting is that there was a PF bug in FreeBSD not long ago as well: <https://www.enricobassetti.it/2023/09/cve-2023-4809-freebsd-pf-bypass-when-using-ipv6/ https://www.enricobassetti.it/2023/09/cve-2023-4809-freebsd-...>; <https://news.ycombinator.com/item?id=37437530 https://news.ycombinator.com/item?id=37437530>.
deleted 3y ago
[deleted]
hulitu 3y ago
> Yet, macOS seems to get buggier and buggier between releases. Something about the way it's being developed right now is going very wrong This is a common trend in the SW development in the last decade. Aparently bug fixing is hard and expensive, that's why they concentrate on new features or, in extreme cases, complete rewrites (GTK).
- steve1977 3y ago
  https://www.jwz.org/doc/cadt.html https://www.jwz.org/doc/cadt.html
pohl 3y ago
I see more and more complaints about bugs, anyway. I use it all day every day without issues, though. Not sure what to make of that. It’s not that I don’t believe those who are affected. I’m just not sure one can conclude that it’s getting more buggy with each release. Maybe the number of unusual use cases increases with popularity?
- deergomoo 3y ago

FollowingTheDao 3y ago

[flagged]

deleted 3y ago
[deleted]

nvahalik 3y ago

Anyone else get an invalid certificate on this site? What's the deal? ETA: Hm. OpenDNS (family) blocks them...

deleted 3y ago
[deleted]
ezfe 3y ago
makes sense that blocking software would block VPN software
mzs 3y ago
https://web.archive.org/web/20230913161315/https://mullvad.net/en/blog/2023/9/13/bug-in-macos-14-sonoma-prevents-our-app-from-working/ https://web.archive.org/web/20230913161315/https://mullvad.n...

keehun 3y ago

I'm glad Mullvad is raising the public temperature on this! This one has definitely been noticed and been very concerning.

scosman 3y ago
Has this been noted elsewhere? Sounds like Mulvad reported after the 6th which is pretty close to the RC. From source: "we have investigated this issue after the 6th beta was released and reported the bug to Apple"
- keehun 3y ago
  Not publicly that I have seen, but I can assure you networking and cybersecurity companies (and others) saw this pretty quickly when the bug was first released. I was just glad to see a relatively big company calling out this rather egregious issue.
  - LeoNatan25 3y ago
    Security companies should be much more open about these issues, rather than quake the notion that if they go public, they’d lose their hush hush secret contacts at Apple that give them private entitlements for private functionality. (Source: first hand experience)
    - pdimitar 3y ago
      Your comment is pretty vague but intriguing. Are you allowed to share an example?
      - mrcode007 3y ago
        they're called managed capabilities and require apple's approval for unlocking access. CarPlay is an example: https://developer.apple.com/documentation/carplay/requesting_carplay_entitlements https://developer.apple.com/documentation/carplay/requesting... edit: tap to pay is another: https://developer.apple.com/documentation/proximityreader/setting-up-the-entitlement-for-tap-to-pay-on-iphone https://developer.apple.com/documentation/proximityreader/se...

unnouinceput 3y ago

We, the old Windows developers, welcome you, the current Apple developers, to the 90's, when Windows was shittier and shittier with each version. Get ready for the next decade when workarounds and basically underground techniques will be your only survivability. As MacOS becomes more popular, it seems it has to go to this shitty phase, as Windows did back in the day. We got rid of this phase with Windows XP release, so around 7 years. For you, who knows, hopefully shorter.

whyenot 3y ago
It's been 23 years (to the day!) since the release of the OS X public beta, and it's a mature product. I'm not sure it's getting "shittier and shittier," I think there are still refinements and improvements, they just aren't as big as they used to be.
- sumuyuda 3y ago
  The UI has definitely gotten shittier and shittier.
  - heyoni 3y ago
    And system settings panel is so unresponsive! I think it’s written in react or something?
  - whyenot 3y ago
    I don't know about that. Aqua with it's pin stripes jewel-like buttons and other quirks was significantly worse than what we have today.
    - can16358p 3y ago
      Yup. I absolutely hated that skeuomorphic blurry 3D-like design. Flat design looks much, much cleaner. Same for iOS. iOS 7 was the first version that I actually liked looking at.
    - BizarreByte 3y ago

zshrc 3y ago

Just a note, while I experienced issues connecting with the Mullvad.app, running a Mullvad Wireguard config in Wireguard.app worked fine.

voytec 3y ago

[flagged]

thedanbob 3y ago
But it's not a misconfiguration, it's a bug as the article explains. And it's not the author's fault that Apple named their firewall the same as OpenBSD.
- voytec 3y ago
  It's not a case of naming something similarily to other software. OSX used FreeBSD's ipfw and around the time they renamed the OS to macOS, they switched to OpenBSD's pf. Now they've screwed up either configuration or implementation but to me - it doesn't read like a bug in pf.
  - fullspectrumdev 3y ago
    It’s a bug in the program named pf on macOS. It’s not that deep. Nobody is blaming OpenBSD’s pf here.
- Khaine 3y ago
  I believe Apple used FreeBSD's implementation of pf, as it also has the same syntax. OpenBSD pf has evolved since then and their are minor syntactic differences for some rules between freebsd pf and openbsd pf.

CharlesW 3y ago

Mullvad has been working okay for me in Sonoma betas (I had to click "Connect" twice), and appears to work perfectly in the final 14.0 release (23A339). A test with ipleak.net looks normal. What am I doing right/wrong?

deleted 3y ago
[deleted]
contact9879 3y ago
it works but requires local network connections to be enabled which I assume is the "leak" they note

panosv 3y ago

Not completely relevant, but another long standing bug: An 11 Year Old Bug in the macOS Popen(): https://news.ycombinator.com/item?id=37238433 https://news.ycombinator.com/item?id=37238433

saagarjha 3y ago
If this is your bug, consider also sending in a feedback with your patch. The open source projects don’t usually take PRs.
- panosv 3y ago
  We had a discussion with one of Apple's moderators here: https://developer.apple.com/forums/thread/726713 https://developer.apple.com/forums/thread/726713 If you have a better way to reach them, lmk.
  - krackers 3y ago
    That's more than "one of Apple's moderators", that's the legendary Quinn (and the _only_ person within Apple I've seen to respond on the dev forums). Also if you do need to reach someone at apple, I think filing a DTS incident would do that.
    - saagarjha 3y ago
      There are a handful of Apple engineers who will reply in the forums for their products.
  - saagarjha 3y ago
    That's about the best thing short of knowing someone on that team, I think.
- dolmen 3y ago
  From the forum discussion, this has been reported as FB12144217.

m3kw9 3y ago

New OSs can sink your company

thehours 3y ago

I don’t know if related, but immediately the last two MacOS upgrades I was unable to get my networking to work. I could connect to Wi-Fi / Ethernet / Hotspot. But nothing would actually connect (e.g. browser, pings, etc) , not even to my router. The fix both times was to open the Mullvad VPN app just once and everything worked again. No idea why just opening the app would fix the issue.

oefrha 3y ago
VPN apps make changes to the routing table, I suppose your traffic was routed to a nonexistent interface until the VPN app was launched.

deleted 3y ago

[deleted]

throwaway914 3y ago

How does one do the equivalent in Linux with nftables? It's very cool that you can create a logging interface.

dharmab 3y ago
In Linux you can use eBPF. See https://github.com/iovisor/bcc https://github.com/iovisor/bcc for an easy way to write eBPF, or look for something in the tools/ dir that does what you want. You distro might have these packaged in bcc-tools or similar.

mannyv 3y ago

Question: if pflog1 isn't created, does pf work as expected? Since tcpdump presumably isn't logging anything going out of pflog1, it implies that the data isn't being sent to pflog1. That implies that it's something upstream from there. Was pflog1 plumbed and/or pulled up? Back in the day you used to have to plumb and pull up an interface manually. Does MacOS do that for you? Granted, isolating this isn't the bug reporter's job. But at some point the engineer will be asking these questions, so you might as well answer them.

MuffinFlavored 3y ago
Is there something you could add to the reproduce case to workaround this, like plumb/pull up after pfrules are loaded as you mentioned?
- mannyv 3y ago
  On Ventura 13.5.2 ifconfig create creates the interface UP: bash-3.2# ifconfig pflog1 create bash-3.2# ifconfig -a pflog1: flags=1<UP> mtu 33072 But I don't think tcpdump will see anything because there's no IP associated with that interface. On some devices (Broadcom) the OS will just dump stuff out of an interface, but I'm not sure what MacOS does in this case. I presume this was tested to work on 13.5.2?
  - AdamJacobMuller 3y ago
    tcpdump can see traffic fine without an IP on an interface (how much traffic the interface has traffic without one is another question) pflog interface is not a "real" interface, there's no point in putting an IP address on it
    - mannyv 3y ago
      Oh, pflog interfaces are a special device that's known by tcpdump. Interesting.
      - 3y ago

OJFord 3y ago

If you use their wireguard servers, you can use wireguard directly (no Mullvad) app, and this is probably not an issue? I'm not certain and don't have a (modern) Mac, so I can't test. Just a reminder that it's nothing proprietary and the app is optional I suppose.

derefr 3y ago
I would presume that the thing Mullvad are (rightfully) doing with firewall rules in their app, is to prevent non-VPN traffic from flowing whenever the VPN is enabled. The whole “leaking your IP through DNS / privileged connections to OS update servers / etc.” issue.
- OJFord 3y ago
  I'm not saying there's no value in the app, but there must be other ways of achieving that (though, maybe they're broken by the same). Do you have `ip netns` on macOS? Fwmark? There's the WireGuard app too of course, I wonder if that's similarly broken.
  - fastball 3y ago
    The way of achieving that is through the Packet Filter, which is what is broken. It seems unlikely you can block system-level traffic with anything except system-level APIs.
    - hnarn 3y ago
      I'm not in any way excusing the bug, but what about routing? Even if the firewall is broken, if you have a wg0 interface and your only route is through that interface, wouldn't that also achieve the same goal?

kdrag0n 3y ago

The same PF bug is breaking one of OrbStack's networking features. I found it hard to believe when I narrowed it down to this, but I guess I'm not alone. Really hoping this is fixed before the stable release. I didn't get a chance to report this to Apple until yesterday, but I think it's a fairly recent regression, probably from around beta 6. PF is supposed to be a last-match firewall but it's almost like macOS is doing first-match now: an earlier "block" rule (without "quick") is overriding "pass" rules in a different anchor, which obviously breaks things.

bjoli 3y ago
"really hope" is too weak. This has to be fixed before release. A stable release with this would be unacceptable and would be reason for me to abandon Mac OS. You don't deliver a product with this kind of regression if you want it to be used for any kind of serious business
- nikanj 3y ago
  Maybe you don't deliver products like that, but Apple delivers them all the time. Then you have to desperately raise noise on all possible forums, and hope to catch the eye of someone who actually works at a relevant team at Apple, before your customers get too tired of the issues caused by the bug and abandon you. This is not an Apple-specific issue either, Microsoft works just the same. Large OS releases are a massive undertaking, and the release train don't stop for a firewall bug - no matter how severe the bug is for the people it does affect.
  - szundi 3y ago
    I remember the time around iOS 5-6, they made some delays and probably it fired back on them and since they have delay related PTSD as an organization. I think since then they just have to push whatever it takes. I remember after iOS 7 I almost left the ecosystem it was so buggy and almost unusable. They probably measured it because after maybe iOS 9-10 they started to be very proud of fixing and stabilizing stuff even in product announcement and iOS is very stable now, not recognizing an issue for years maybe. But this had a price, they paid it, now they seem very aggressive on schedules until the next major clusterfucky year.

TheBinaryGuy 3y ago

For me the solution was to download Wireguard configs and use the Wireguard app.

cyclecount 3y ago
But is the Wireguard app leaking data and just less secure than Mullvad's app?
computerfriend 3y ago
This is only a solution if you don't set PostUp and PostDown firewall rules in your WireGuard config. Which isn't much of a solution.

maxehmookau 3y ago

Does this also apply to the new iOS? Since upgrading to the beta, I've been unable to use Mullvad.

anacrolix 3y ago

Mullvad VPN breaks DNS over IPv6 https://github.com/golang/go/issues/61060 https://github.com/golang/go/issues/61060