7 ms·

I work at Mullvad. (co-CEO, co-founder) Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in th

by kfreds 5mo ago

I work at Mullvad. (co-CEO, co-founder) Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in the blog post. As for mitigation, we are already testing a patch of the unintended behavior on a subset of our infrastructure. If any of you try to reproduce the blog post's findings you may get confusing results throughout the day. We will also re-evaluate whether the intended behaviors are acceptable or not. Some of this is a trade-off between multiple aspects of privacy, and multiple aspects of user experience. Please note that this is my current understanding, which may change. I was only made aware of this an hour ago, and most of that time was spent talking with Ops, considering what to do immediately, and writing this post. Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away.

ignoramous 5mo ago
> Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings How to report a bug or vulnerability ... we (currently) have no bug bounty program ... send an email to support@mullvadvpn.net https://mullvad.net/en/help/how-report-bug-or-vulnerability https://mullvad.net/en/help/how-report-bug-or-vulnerability / https://archive.vn/BeHhr https://archive.vn/BeHhr
- wren6991 5mo ago
  To support? Oof.
  - kfreds 5mo ago
    I'm not sure what you mean by "Oof". We don't have a dedicated security team because security and privacy are integral to all aspects of our service. It doesn't make sense to centralise it. As for our support team they are responsive and experienced. Several of them have worked with us for many years and do offensive security research in their free time. Unlike many organisations we don't see customer support as a cost center, just like we don't see security as a cost center. Our support team represent our customers, and as a consequence contribute a lot to how we prioritise our roadmap.
    - traceroute66 5mo ago
      > I'm not sure what you mean by "Oof". I second this. Clearly the person who wrote "Oof" has never emailed Mullvad support. Whenever I have emailed Mullvad support I have received a prompt reply from a human being who clearly actually cares about taking ownership of the question and seeing it through to resolution. I have also witnessed first-hand the support person taking the question to an internal team member where it requires additional input. So there are clear paths for escalation if circumstances require it. Finally the support mail allows for PGP encryption of communications too. (I am not a Mullvad shill. Not a Mullvad employee. Just a satisfied customer)
    - fragmede 5mo ago
      Human psychology is weird and some things are just cultural. If you have the ops team make the security@ email alias just forward to support, you could avoid having to go into all that. "Just email support@" feels like you don't care. That you do, and that your support team is awesome, doesn't change the fact that there are other companies out there who's aren't. Security people are human with human egos, and they want to feel special, so giving them a special way to reach you, even if it's the same thing behind the scene, makes a world of difference.
    - nananana9 5mo ago
      It still probably makes sense to alias it to security@mullvadvpn.net for privacy/security concerns. I'm not familiar with how you run your company -- without the context you gave most people would hesitate emailing support@ for security issues.
    - wren6991 5mo ago
      You're right, that was a knee-jerk reaction. Sorry, I take it back.
    - Stefan-H 5mo ago
      "We don't have a dedicated security team because security and privacy are integral to all aspects of our service". Do you have people whose role is explicitly security? Who are the security SMEs in your organization if not? I personally find the "Security is so important to us that we don't have a team dedicated to it" argument weak, and often results in misaligned incentives - if individuals have to alternate hats from "deliver results" to "properly vet security", the business push to deliver tends to win out. I'd be very curious to hear how you ensure your team doesn't fall into that trap.
- dust-jacket 5mo ago
  Not having a bug bounty or dedicated email address does not make it OK to go public immediately
  - mvdtnz 5mo ago
    Yes it does actually.
    - dust-jacket 5mo ago
      I don't feel like its hard to come up with examples where (I would say) its ethically wrong to disclose immediately. If you spotted a company's mistake that might endanger their user's lives or safety, would you put those users at risk simply because there was no obvious financial reward? If so, I guess we just have different opinions on the ethics involved here.
      - alt227 5mo ago
        If you are talking about some open source project then I would fully agree. But when it comes to money making corporations then personally I dont agree that revealing flaws in their product comes into ethics at all. A companies paid product is flawed, their own paid engineers didnt figure that out, why should I do it for free becasue 'ethics'? This is the entire reason bug bounty programs exist in the first place.
        fn-mote 5mo ago
        You seem to have a very bright line between the acceptable behavior for “no money involved” and “money involved”. For me, it’s more subtle than that. Everybody (“almost all software”) has exploitable bugs. Are you a fool for not finding the ones in yours? Maybe. Sometimes. There is a huge difference between Project Zero finding a trivial vulnerability almost identical to one reported months earlier (close to negligence) and Mullvad having the CEO personally posting a response here in a very calm tone.
        5mo ago
- abanana 5mo ago
  Are you seriously suggesting people shouldn't operate with a bit of common decency unless they're going to get some money out of it?
  - embedding-shape 5mo ago
    Most of HN readers/writers are American, of course they won't do anything unless they personally profit off it, the entire culture is built around this mindset. Meanwhile, Mullvad is Swedish, and we tend to assume we all want to help build a better world together. Mix the two, and you get this conversation :)
    - trelane 5mo ago
      > Most of HN readers/writers are American, of course they won't do anything unless they personally profit off it, the entire culture is built around this mindset American culture is highly varied. For some this is true, for others this is wrong and highly insulting. Maybe try a narrower brush next time.
      - mrpopo 5mo ago
        I think the "others" should open their eyes to the world around them, then.
        trelane 5mo ago
        > should open their eyes to the world around them Amazingly brazen assumption right there.
      - embedding-shape 5mo ago
        It's OK for the country to have a pervasive culture yet not every resident or citizen of the country to be a part of that culture, or even actively work against it. If you're not one of them matching that description, it shouldn't be insulting, as it's not about you in the first place. Maybe not everything is aimed towards you, especially if you don't feel like the description actually matches you :)
azalemeth 5mo ago
You really do provide a reassuring, good service -- thank you. It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours. `mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post. I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.
lionkor 5mo ago
I just want to say I absolutely love Mullvad! You guys did a fantastic job at designing a genuinely good and trustworthy (as much as possible) VPN vendor. You communicating here is just another data point towards this.
- alex1138 5mo ago
  I almost want the people doing the mandatory VPN product placement ("This video is sponsored by NordVPN!") to do Mullvad for once. My jaw would hit the floor from unfamiliarity
  - consumer451 5mo ago
    If Mullvad was suddenly in that ad scene, I would get worried. This is not anything specific against Nord, I don't know anything about them. However, at this point, I take YouTube/influencer ads as a very negative signal towards the product being pushed. I am not sure if that's fair, but that's just my gut feeling given the entire YouTube ad scene. I think it's the cost per viewer, where "scams" are more profitable than a honest business, and that makes my gut tingle. Again, to be fair, I may be being a jerk here with my judgment.
999900000999 5mo ago
Can we have an Open Suse client. Sorta odd you don't support one of Europe's most popular distros.
- c0balt 5mo ago
  It already has official packaging for Tumbleweed, see https://github.com/mullvad/mullvadvpn-app/issues/2242 https://github.com/mullvad/mullvadvpn-app/issues/2242 for the upstream issue. Leap can use the normal Linux application, you will just have to provide the dependencies yourself.
  - 999900000999 5mo ago
    https://mullvad.net/en/help/install-mullvad-app-linux https://mullvad.net/en/help/install-mullvad-app-linux >The Mullvad VPN app is available in our repository for the following supported Linux distributions: Ubuntu (24.04+) Debian (12+) Fedora (42+) The only thing I see on the issue you linked is a way to jerry-rig the fedora package. When I tried that I kept getting untrusted key warnings. You can skip them of course, but it kind of undermines any type of trust here
    - aschar 5mo ago
      > When I tried that I kept getting untrusted key warnings. You can skip them of course, but it kind of undermines any type of trust here Yes, the expected procedure would be to trust those keys for that package instead of disabling integrity checks. This is an issue between you and your package manager and not something Mullvad or any other packager (except OpenSUSE maintainers) can fix for you. You complain about the packaging and support of mullvad maintainers when you are having skill issues with your distro. https://github.com/mullvad/mullvadvpn-app/issues/2242#issuecomment-282271864 https://github.com/mullvad/mullvadvpn-app/issues/2242#issuec...
      - 999900000999 5mo ago
        It's a skill issue that they decide to not list open suse as a supported distro on their own help page ? It's a skill issue that the thread has a bunch of different solutions and none of them are definitive and endorsed by the company I'm paying $5 a month too ?
BenFranklin100 5mo ago
Thanks for the reassurances. Love your product.
GardenLetter27 5mo ago
Any chance on port forwarding coming back?
- riley_dog 5mo ago
  I'd leave Proton and go back if they offered it again. I doubt it'll happen, but you never know, I guess.
drewfax 5mo ago
Thank you for being on our side Mullvad. I'm using Mullvad through Tailscale. It's been an awesome combination for my self-hosting and privacy needs.
Icathian 5mo ago
I deeply appreciate Mullvad's thorough approach to privacy and ethics. In this day and age, you all are an absolute breath of fresh air. Thanks for that.
HumblyTossed 5mo ago
Long time Mullvad customer. Thank you for this.
datadrivenangel 5mo ago
I enjoy the print ads in the DC subway by the way.
dennysora 5mo ago
Thank you! As a staunch supporter of Mullvad, you are the only VPN provider I recommend for cybersecurity, especially in today's landscape where VPNs are facing increasing regulatory restrictions. Thank you so much!
dongcarl 5mo ago
Carl here (Obscura CEO, one of Mullvad's partners) This was an interesting finding, though as kfreds mentioned it would have been better to notify the vendor before publishing. The main finding (IP-position-in-pool correlation between servers) seems to include genuinely unintended behaviour. Given our great experience with the Mullvad team, I'm sure this will be addressed soon. In general, if you want different "identities", you should make sure to rotate or use different WireGuard keys. One small thing from the article I'll comment on: > Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key, which rotates every 1 to 30 days (unless you use a third-party client, in which case it never rotates). Context: WireGuard is by design[1] a "Connection-less Protocol", there's no concept of a connection, there's only a "re-keying handshake" (key here refers to the ephemeral Diffie-Hellman key, not the WireGuard key) every 2-3 minutes ONLY IF there's traffic flowing. The above statement is not too surprising if you consider the counterfactual: What would happen if, even with the same WireGuard key, the exit IP were randomized each time you "connect" to the server (say each time there is a "re-keying handshake" or at more frequent cadence (e.g. every 15 minutes) than the WireGuard key rotation). In this scenario, ~every 15 minutes: - At the Transport layer, all your in-tunnel connections that are on non-roaming protocols (basically everything except QUIC) would be disrupted, and the connections would have to be re-established. - At the Application layer, many application-level sessions that treat "same cookie, new IP" as suspicious would trigger logouts, CAPTCHAs, or risk scoring. Both are terrible UX, and what's worse would also make users much more uniquely fingerprintable ("this person keeps reconnecting from a different IP, they must be using Mullvad"). [1]: https://www.wireguard.com/protocol/ https://www.wireguard.com/protocol/
- Cider9986 5mo ago
  Apparently I already use Obscura despite never paying for it :) https://obscura.com/check/ https://obscura.com/check/
  - dongcarl 5mo ago
    Heh yeah we just check for a Mullvad exit IP, it's one way you know we're actually relaying to Mullvad!

by kfreds 5mo ago

ignoramous 5mo ago

> Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings How to report a bug or vulnerability ... we (currently) have no bug bounty program ... send an email to support@mullvadvpn.net https://mullvad.net/en/help/how-report-bug-or-vulnerability https://mullvad.net/en/help/how-report-bug-or-vulnerability / https://archive.vn/BeHhr https://archive.vn/BeHhr

wren6991 5mo ago
To support? Oof.
- kfreds 5mo ago
  I'm not sure what you mean by "Oof". We don't have a dedicated security team because security and privacy are integral to all aspects of our service. It doesn't make sense to centralise it. As for our support team they are responsive and experienced. Several of them have worked with us for many years and do offensive security research in their free time. Unlike many organisations we don't see customer support as a cost center, just like we don't see security as a cost center. Our support team represent our customers, and as a consequence contribute a lot to how we prioritise our roadmap.
  - traceroute66 5mo ago
    > I'm not sure what you mean by "Oof". I second this. Clearly the person who wrote "Oof" has never emailed Mullvad support. Whenever I have emailed Mullvad support I have received a prompt reply from a human being who clearly actually cares about taking ownership of the question and seeing it through to resolution. I have also witnessed first-hand the support person taking the question to an internal team member where it requires additional input. So there are clear paths for escalation if circumstances require it. Finally the support mail allows for PGP encryption of communications too. (I am not a Mullvad shill. Not a Mullvad employee. Just a satisfied customer)
  - fragmede 5mo ago
    Human psychology is weird and some things are just cultural. If you have the ops team make the security@ email alias just forward to support, you could avoid having to go into all that. "Just email support@" feels like you don't care. That you do, and that your support team is awesome, doesn't change the fact that there are other companies out there who's aren't. Security people are human with human egos, and they want to feel special, so giving them a special way to reach you, even if it's the same thing behind the scene, makes a world of difference.
  - nananana9 5mo ago
    It still probably makes sense to alias it to security@mullvadvpn.net for privacy/security concerns. I'm not familiar with how you run your company -- without the context you gave most people would hesitate emailing support@ for security issues.
  - wren6991 5mo ago
    You're right, that was a knee-jerk reaction. Sorry, I take it back.
  - Stefan-H 5mo ago
    "We don't have a dedicated security team because security and privacy are integral to all aspects of our service". Do you have people whose role is explicitly security? Who are the security SMEs in your organization if not? I personally find the "Security is so important to us that we don't have a team dedicated to it" argument weak, and often results in misaligned incentives - if individuals have to alternate hats from "deliver results" to "properly vet security", the business push to deliver tends to win out. I'd be very curious to hear how you ensure your team doesn't fall into that trap.
dust-jacket 5mo ago
Not having a bug bounty or dedicated email address does not make it OK to go public immediately
- mvdtnz 5mo ago
  Yes it does actually.
  - dust-jacket 5mo ago
    I don't feel like its hard to come up with examples where (I would say) its ethically wrong to disclose immediately. If you spotted a company's mistake that might endanger their user's lives or safety, would you put those users at risk simply because there was no obvious financial reward? If so, I guess we just have different opinions on the ethics involved here.
    - alt227 5mo ago
      If you are talking about some open source project then I would fully agree. But when it comes to money making corporations then personally I dont agree that revealing flaws in their product comes into ethics at all. A companies paid product is flawed, their own paid engineers didnt figure that out, why should I do it for free becasue 'ethics'? This is the entire reason bug bounty programs exist in the first place.
      - fn-mote 5mo ago
        You seem to have a very bright line between the acceptable behavior for “no money involved” and “money involved”. For me, it’s more subtle than that. Everybody (“almost all software”) has exploitable bugs. Are you a fool for not finding the ones in yours? Maybe. Sometimes. There is a huge difference between Project Zero finding a trivial vulnerability almost identical to one reported months earlier (close to negligence) and Mullvad having the CEO personally posting a response here in a very calm tone.
        5mo ago
abanana 5mo ago
Are you seriously suggesting people shouldn't operate with a bit of common decency unless they're going to get some money out of it?
- embedding-shape 5mo ago
  Most of HN readers/writers are American, of course they won't do anything unless they personally profit off it, the entire culture is built around this mindset. Meanwhile, Mullvad is Swedish, and we tend to assume we all want to help build a better world together. Mix the two, and you get this conversation :)
  - trelane 5mo ago
    > Most of HN readers/writers are American, of course they won't do anything unless they personally profit off it, the entire culture is built around this mindset American culture is highly varied. For some this is true, for others this is wrong and highly insulting. Maybe try a narrower brush next time.
    - mrpopo 5mo ago
      I think the "others" should open their eyes to the world around them, then.
      - trelane 5mo ago
        > should open their eyes to the world around them Amazingly brazen assumption right there.
    - embedding-shape 5mo ago
      It's OK for the country to have a pervasive culture yet not every resident or citizen of the country to be a part of that culture, or even actively work against it. If you're not one of them matching that description, it shouldn't be insulting, as it's not about you in the first place. Maybe not everything is aimed towards you, especially if you don't feel like the description actually matches you :)

azalemeth 5mo ago

You really do provide a reassuring, good service -- thank you. It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours. `mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post. I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.

lionkor 5mo ago

I just want to say I absolutely love Mullvad! You guys did a fantastic job at designing a genuinely good and trustworthy (as much as possible) VPN vendor. You communicating here is just another data point towards this.

alex1138 5mo ago
I almost want the people doing the mandatory VPN product placement ("This video is sponsored by NordVPN!") to do Mullvad for once. My jaw would hit the floor from unfamiliarity
- consumer451 5mo ago
  If Mullvad was suddenly in that ad scene, I would get worried. This is not anything specific against Nord, I don't know anything about them. However, at this point, I take YouTube/influencer ads as a very negative signal towards the product being pushed. I am not sure if that's fair, but that's just my gut feeling given the entire YouTube ad scene. I think it's the cost per viewer, where "scams" are more profitable than a honest business, and that makes my gut tingle. Again, to be fair, I may be being a jerk here with my judgment.

999900000999 5mo ago

Can we have an Open Suse client. Sorta odd you don't support one of Europe's most popular distros.

c0balt 5mo ago
It already has official packaging for Tumbleweed, see https://github.com/mullvad/mullvadvpn-app/issues/2242 https://github.com/mullvad/mullvadvpn-app/issues/2242 for the upstream issue. Leap can use the normal Linux application, you will just have to provide the dependencies yourself.
- 999900000999 5mo ago
  https://mullvad.net/en/help/install-mullvad-app-linux https://mullvad.net/en/help/install-mullvad-app-linux >The Mullvad VPN app is available in our repository for the following supported Linux distributions: Ubuntu (24.04+) Debian (12+) Fedora (42+) The only thing I see on the issue you linked is a way to jerry-rig the fedora package. When I tried that I kept getting untrusted key warnings. You can skip them of course, but it kind of undermines any type of trust here
  - aschar 5mo ago
    > When I tried that I kept getting untrusted key warnings. You can skip them of course, but it kind of undermines any type of trust here Yes, the expected procedure would be to trust those keys for that package instead of disabling integrity checks. This is an issue between you and your package manager and not something Mullvad or any other packager (except OpenSUSE maintainers) can fix for you. You complain about the packaging and support of mullvad maintainers when you are having skill issues with your distro. https://github.com/mullvad/mullvadvpn-app/issues/2242#issuecomment-282271864 https://github.com/mullvad/mullvadvpn-app/issues/2242#issuec...
    - 999900000999 5mo ago
      It's a skill issue that they decide to not list open suse as a supported distro on their own help page ? It's a skill issue that the thread has a bunch of different solutions and none of them are definitive and endorsed by the company I'm paying $5 a month too ?

BenFranklin100 5mo ago

Thanks for the reassurances. Love your product.

GardenLetter27 5mo ago

Any chance on port forwarding coming back?

riley_dog 5mo ago
I'd leave Proton and go back if they offered it again. I doubt it'll happen, but you never know, I guess.

drewfax 5mo ago

Thank you for being on our side Mullvad. I'm using Mullvad through Tailscale. It's been an awesome combination for my self-hosting and privacy needs.

Icathian 5mo ago

I deeply appreciate Mullvad's thorough approach to privacy and ethics. In this day and age, you all are an absolute breath of fresh air. Thanks for that.

HumblyTossed 5mo ago

Long time Mullvad customer. Thank you for this.

datadrivenangel 5mo ago

I enjoy the print ads in the DC subway by the way.

dennysora 5mo ago

Thank you! As a staunch supporter of Mullvad, you are the only VPN provider I recommend for cybersecurity, especially in today's landscape where VPNs are facing increasing regulatory restrictions. Thank you so much!

dongcarl 5mo ago

Carl here (Obscura CEO, one of Mullvad's partners) This was an interesting finding, though as kfreds mentioned it would have been better to notify the vendor before publishing. The main finding (IP-position-in-pool correlation between servers) seems to include genuinely unintended behaviour. Given our great experience with the Mullvad team, I'm sure this will be addressed soon. In general, if you want different "identities", you should make sure to rotate or use different WireGuard keys. One small thing from the article I'll comment on: > Surprisingly, the exit IP you are given is not randomized each time you connect to the server, but deterministically picked based on your WireGuard key, which rotates every 1 to 30 days (unless you use a third-party client, in which case it never rotates). Context: WireGuard is by design[1] a "Connection-less Protocol", there's no concept of a connection, there's only a "re-keying handshake" (key here refers to the ephemeral Diffie-Hellman key, not the WireGuard key) every 2-3 minutes ONLY IF there's traffic flowing. The above statement is not too surprising if you consider the counterfactual: What would happen if, even with the same WireGuard key, the exit IP were randomized each time you "connect" to the server (say each time there is a "re-keying handshake" or at more frequent cadence (e.g. every 15 minutes) than the WireGuard key rotation). In this scenario, ~every 15 minutes: - At the Transport layer, all your in-tunnel connections that are on non-roaming protocols (basically everything except QUIC) would be disrupted, and the connections would have to be re-established. - At the Application layer, many application-level sessions that treat "same cookie, new IP" as suspicious would trigger logouts, CAPTCHAs, or risk scoring. Both are terrible UX, and what's worse would also make users much more uniquely fingerprintable ("this person keeps reconnecting from a different IP, they must be using Mullvad"). [1]: https://www.wireguard.com/protocol/ https://www.wireguard.com/protocol/

Cider9986 5mo ago
Apparently I already use Obscura despite never paying for it :) https://obscura.com/check/ https://obscura.com/check/
- dongcarl 5mo ago
  Heh yeah we just check for a Mullvad exit IP, it's one way you know we're actually relaying to Mullvad!