12 ms·

Mullvad: Diskless infrastructure using stboot in beta

423 points by NabiDev 5y ago

oefrha 5y ago
For people wondering how the hell a user can audit the server is diskless or whatever, the goal appears to be using TPM to provide remote attestation for all code in the boot path. See https://www.system-transparency.org/ https://www.system-transparency.org/.
- Gigachad 5y ago
  How do we audit that the TPM chip is secure? What happened when a bug in the silicone is found later?
  - sanxiyn 5y ago
    We assume it is, just like we assume CPU works as advertised. In other words, TPM is part of TCB.
    - Gigachad 5y ago
      So what is the point? I already assume the code on their server is not malicious by using it. What extra trust does an untrusted TPM chip give me?
      - sanxiyn 5y ago
        It reduces TCB. TPM is smaller than entire server.
      - kfreds 5y ago
        System Transparency reduces your trust assumptions on us. As a VPN provider we are in an immense position of power over you. We aim to reduce your trust assumptions on us to a few things that we would need to explicitly lie about in order to betray you. As an example, let's say that we offered any of our users to at any time during the year show up at our office and inspect our VPN hardware, without warning us beforehand. In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense?
        gravypod 5y ago
        > In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense? Have you ever thought about doing something like that with some big youtube personalities? Maybe have them hire some pen testers, randomly show up to one of your datacenters, and post recordings of what is done and attacks that could be possible. Since your software is open pen testers could prepare some things to try to attack days in advanced. I'd love to see something like this with Level1Techs or something.
        nopcode 5y ago
        > big youtube personalities You mean the same people that are being "sponsored" by VPN providers for years? Why would we want to trust youtubers of all people?
- cmeacham98 5y ago
  I hate to be this skeptical, but let's say this is 100% possible (I have my doubts, see previous attacks on things like TPMs and SGX, but I digress). You probably could get 90% of the logging capability by putting monitoring in front of and behind the server, and associating connections by traffic/time. It just seems like this goal of using technology to prove they're trustworthy is unlikely to actually work for a VPN company due to the threat models.
  - kfreds 5y ago
    You are correct that System Transparency is not a universal remedy for all threat models. Indeed the word "secure" is undefined until you have a threat model. Most threat models are implied and undocumented assumptions. At some stage in an R&D project one should shift from exploration to threat model-driven development. Most people, myself included, tend to focus on technical solutions, and argue back and forth how "oh, but it can be broken using X". System Transparency aims to provide remote auditability assuming (1) the server hardware specification is correct, (2) a correct cryptographic hash of the contents of the SPI flash containing the platform firmware, and (3) a keypair generated on and only accessible to the platform. This is very simplified of course. An attacker aiming to tap incoming and outgoing network traffic from our servers, who has physical access to the VPN server's Ethernet port, or an upstream router, isn't in the scope of System Transparency to protect against. We need to use other means for that.
  - azalemeth 5y ago
    Traffic analysis and correlation analysis is indeed a powerful tool, and in general only communicating at a constant bandwidth between all nodes at all times is the only way to completely defeat it (which is what, I understand, some military systems do). That's inherently highly wasteful, however. To get around this, Mullvad offer very transparent comprehensive multi-hop routing systems [1]; you can bounce your wireguard tunnels around in layered wireguard tunnels (a bit á-la tor) by just choosing a series of ports to tunnel on and to. My understanding is that each one of these adds non-deterministic latency to your connection and probably would help to make such attacks harder at the very least, because from the point of view of an "all seeing" adversary the fact that all of these servers talk to each other all the time makes it very much harder to know where any packet could have gone. Yes, you can see each individual link but the metadata is lost. I signed up for Mullvad when the UK's Snooper's Charter came into force and the local health inspectors suddenly had the rights to see my DNS record. Since then, I've had it installed on my router and just route everything through a custom wireguard (originally openvpn) tunnel. I've had some issues with my ISP randomly bandwidth limiting traffic on the odd port to 1 MByte/s, but frankly that makes me more inclined to put everything behind an encrypted tunnel. I don't want my ISP to do traffic shaping and I do want them to just leave me alone and let me communicate in peace. I have absolutely nothing to hide, but now have to accept that I partly live in a country where everything is surveilled all the time, and warantless, unaccountable investigation of my (highly personal!) online habits may be happening. I think Mullvad's excellent product, sensible architecture and reasonable price is worth paying. I'm an academic, unlikely to be of interest to three-letter acronyms, and therefore it matches my needs very well. [1] https://mullvad.net/en/help/multihop-wireguard/ https://mullvad.net/en/help/multihop-wireguard/
- kfreds 5y ago
  Correct! Thank you for highlighting that. Here are some additional details for those interested. We intend to make use of TPM for remote attestation of the current boot chain, reproducible builds to provide a strong link from source code to build artifacts, and a transparency log for a historical record of previously used boot chains, artifacts, WireGuard server keys, and related signatures. As dtx1 mentioned elsewhere in this thread, diskless VPN infrastructure is currently in use by many other VPN providers. That is not a novel feature of course. What is novel is user-auditability of running VPN infrastructure. We were the first VPN provider to state our intention to make our infrastructure user-auditable AND provide a realistic roadmap with the specific technologies needed to do so. See the link above. I believe the technologies we use in System Transparency will ultimately reshape the VPN provider industry into a highly competitive space focused on maximizing the transparency of VPN infrastructure. Or not, but at least OUR users will be able to audit us. :) Either way we’re looking forward to the future. The opportunity for improvement is immense.
  - azalemeth 5y ago
    This all sounds very exciting. Where will you draw the line between public and private – at the moment your consumer-facing app is on github, but less "server side" stuff (in common with many other VPN providers). I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently. I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled)
    - kfreds 5y ago
      > Where will you draw the line between public and private – at the moment your consumer-facing app is on github, but less "server side" stuff (in common with many other VPN providers). All source code for all software on our VPN servers must eventually be public, and all build artifacts must be reproducible by 3rd parties. > I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently. Exactly, but we will also have to measure each artifact in the boot chain into the platform TPM, and allow anyone to issue a challenge to the TPM to get a signed quote of the boot chain measurements. > I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled) We’ll have to constrain our own ability to access the VPN servers. We cannot be allowed arbitrary root access as that would make the TPM measurements meaningless from an audit perspective. Well, you’d be able to conclude we have root access, so not totally meaningless.
- franga2000 5y ago
  Damn it, this is a really cool use case for TPMs! So far, every use case I've heard has made me wish they were never invented, but this made me reconsider...
sascha_sl 5y ago
Pretty similar to the default (diskless) mode in Alpine, though it lacks the tooling to verify persisted data and the sources apkvols can be applied from at boot aren't that well documented.
dtx1 5y ago
This is hardly a new thing in VPN providers though. I know that perfect privacy[1] and azire vpn[2] both advertise this feature already. [1]https://www.perfect-privacy.com/en/features/without-logs https://www.perfect-privacy.com/en/features/without-logs [2]https://www.azirevpn.com/docs/environment https://www.azirevpn.com/docs/environment
- pilif 5y ago
  if only there was any proof of this actually being the case and there not being some "accidental" debug log enabled, or some other network level component having "accidental" access to the keys. There's just no good answer to perfect trust-no-one private internet access. If you need to hide all of your traffic from other users in your local network, you can accomplish that in a trust-no-one fashion by running your own VPN endpoint on a server you control which provides better privacy guarantees compared to a centralised commercial VPN whose business model will eventually involve selling your data (once user growth stops but shareholders demand continued revenue growth). But if you need to hide your traffic from anybody but your peer on the internet and you need to hide the fact that you talked to that peer, then, I'm afraid, your out of luck.
  - minerva23 5y ago
    Is no-trust ever possible? I thought people create their threat models and verify they can trust those they have to trust.
    - zeepzeep 5y ago
      > people create their threat models and verify they can trust those they have to trust What kind of people? How do you verify you can trust some company?
      - minerva23 5y ago
        Everyone does it informally to varying degrees for varying problems (often times subconsciously). E.g. "What do I know about this person? Can I trust them around my kids?" You verify trustworthiness by research. Who is involved? Do I trust anyone who trusts them? What are their motivations? What would cause them to take action against me? What causes them to protect my interests? What laws are they subjected to (i.e. who can coerce them)? What do they say for themselves? Where do their words fall on the credibility to BS scale? What is their reputation in the community? What do their competitors/adversaries say? What would cause their behavior to change? I won't enumerate all my research on Mullvad. I can say Mozilla attaching their brand to Mullvad's services helped me a lot (trust by proxy). I'll also say that some of their product decisions give credibility to their anonymity claims. Lastly, I found someone who shared a competitive analysis across many providers. I found the analysis trustworthy. Mullvad has some weak points, but was still the best provider for my particular use case.
yamrzou 5y ago
A bit tangential to the main post, but I'd to share a recent positive experience with Mullvad: I am a regular user of Mullvad and recently wanted to try a different VPN, that only provides Wireguard configs (i.e. no native app). I used the default setup. For some reason, my internet connection was flaky, and when it disconnected and reconnected, my traffic leaked. That never happened to me with Mullvad as the app comes with an "Always require VPN" option out of the box and it has always worked reliably.
- suyjuris 5y ago
  On linux you can create a network namespace exposing only the wireguard network device, so that applications in that namespace cannot leak traffic. Setting this up, however, is quite fiddly in my experience.
  - Koffiepoeder 5y ago
    In addition it is probably not a bad idea to block all traffic on wlo1 / eth0, except that to the mullvad server ip's, through some ufw rules. If you forget to configure the namespace for some applications then, it is highly unlikely the app has internet access (ie, it would need its own mullvad/vpn implementation included).
  - encryptluks2 5y ago
    I would suggest vopono to do this automatically.
  - ComputerGuru 5y ago
    It’s easier and more secure to just create a VM that’s bridged to the VPN interface (regardless of protocol) if you don’t use the VPN for everything but the things you do use it for absolutely must go through it.
    - Melatonic 5y ago
      I think I like this idea the best - simple, effective, and unbreakable due to config changes or updates. Plus it gives you a psychological separation between "VPN related activities" and not. Or you just do everything in the VM. Adds a layer security wise as well to protect your physical system. If you wanted to get really fancy you could have a few different VM's and each one on a different companies VPN
sydthrowaway 5y ago
The next step: dedicated VPN over FPGAs
- rekoil 5y ago
  *drools*
  - sydthrowaway 5y ago
    Wait, Cisco already do this
    - rekoil 5y ago
      They do? Do you have any product names you can share?
joker197cinque 5y ago
The server configuration (and therefore customers account numbers) is stored in Server OS images I suppose, right ? It shouldn't be an issue as far as inspection is concerned, should it ? Also, isn't there a law that enforces logs to be kept for n years ? How is it compatible with diskless setup ?
- encryptluks2 5y ago
  No, they would not need to be stored in the images. They are deployed as part of provisioning package according to the post. If they were part of the OS image, they would have to update the image anytime a new customer is added, which would be far less optimal than having Ansible or whatever push changes. There is no law in Sweden that requires logs, and the point of a VPN provider is to not store logs. If your VPN provider is keeping logs of your connections then you should look for a new provider.
abcdef009 5y ago
The VPN provider I use have had diskless systems for years. Is this the same thing? https://www.ovpn.com/en/blog/improvement-of-the-physical-security https://www.ovpn.com/en/blog/improvement-of-the-physical-sec...
TacticalCoder 5y ago
> If the computer is powered off, moved or confiscated, there is no data to retrieve. Don't forget to add insta-shutdown when any USB device is connected to the system!
- amq 5y ago
  Or disable usb in bios entirely
  - mschuster91 5y ago
    Disabling USB in BIOS only disables the emulation of classic PS2 keyboards and IDE storage so that old OSes or bootloaders without USB stacks can work with modern equipment. As soon as the OS kernel initializes the PCI bus, USB will work again - however they could go and remove the xHCI modules from the kernel and image.
    - rekoil 5y ago
      Mullvad has a custom-built bare metal UEFI implementation based on coreboot, I assume stboot is an evolution of that, which means it takes as close as you can get to full responsibility for initialization of all system components like processor, chipset, Ethernet, USB, everything. As a result they can absolutely disable USB entirely by never exposing those parts of the device tree to Linux.
      - mschuster91 5y ago
        x86 devices do not have device trees, and for ARM I'd take a guess and say that as long as the PCI root port is exposed to the OS, a PCI re-scan will be enough to wake the USB chipset.
- howdydoo 5y ago
  And a tilt sensor that cuts power if the chasis is moved
codethief 5y ago
I've been following Mullvad for a long time and my impression (from countless reviews and comments here on HN) has been quite positive. But here's what I don't understand: Why are the servers located in Sweden, a country that's known for online surveillance[0] like no other country in the EU? From the Wikipedia article[1]: > The law permits the signals intelligence agency, National Defense Radio Establishment, to monitor the content of all cross-border cable-based Internet traffic to combat "external threats" such as terrorism and organized crime. [0]: https://www.opendemocracy.net/en/can-europe-make-it/didier-bigo-sergio-carrera-nicholas-hernanz-julien-jeandesboz-joanna-parkin-fra-1/ https://www.opendemocracy.net/en/can-europe-make-it/didier-b... [1]: https://en.wikipedia.org/wiki/Internet_in_Sweden#Internet_censorship_and_surveillance https://en.wikipedia.org/wiki/Internet_in_Sweden#Internet_ce...
- y4mi 5y ago
  Because it's a swedish company? The location of the servers is kinda irrelevant in that regard. They'd have to provide government access if there is a lawsuit that demands it. If there isn't one than your critique is entirely pointless
  - codethief 5y ago
    > Because it's a swedish company? I'm not sure what point you're trying to make. If online privacy is as important to them as they say, they could have easily registered their company (or a subsidiary) in a different EU member state. > They'd have to provide government access if there is a lawsuit that demands it. IANAL but I am not entirely sure this is true in the EU. Either way, my question was "Why are the servers located in Sweden?" Whether or not this due to the company owning the servers being in Sweden is irrelevant.
    - akerl_ 5y ago
      I think you’re overthinking this. The people who run the company are based in Sweden. So they registered the company in Sweden, because that’s where they are. Then they hosted the servers in Sweden, because that’s where they are and where the company is registered. Registering the company somewhere else wouldn’t do them any good when they’re living in Sweden, because the legal system isn’t fooled by sleight of hand like that. Likewise, hosting the servers elsewhere from where they’re based. Both would expand the number of entities with the ability to compel them to disclose data, because as long as the company owners live in Sweden, Sweden has that ability. Unless you’re asking why they didn’t move to another country to start their company, which is surely a larger ask than the “easily” you suggest.
throwaway984393 5y ago
It's a trade-off. If you have no disk, the disk can't fail, but the network can, and the remote PXE server can, and the remote SAN can. You can get into a state where you have to pray no servers reboot. Intermittent errors can be real annoying when it makes provisioning fail. (used to work a server farm that'd do server rebuilds over PXE, and ran a few diskless cluster projects) An alternative is you use a RAID array and mount your disks in read-only mode, or use physically read-only disks and when you have to replace a disk, you pre-mirror the replacement disk. In this way the local disks can be replaced as they fail and there's never a point when the server is at risk of not being able to boot. ......or they could boot from CDROM :)
- encryptluks2 5y ago
  A network or PXE server can fail regardless, so this are things that always have to be taken into consideration and in those instances then you address those issues. With this type of setup you do not need a remote SAN as it would defeat the purpose of not having external storage that could store logs. Mullvad has servers all over the world, so a temporary failure in one location will not bring down their entire infrastructure.
  - deleted 5y ago
    [deleted]
  - throwaway984393 5y ago
    It's not just a temporary failure, it's potentially the entire AZ going down hard. High Availability network boot without local storage is very difficult/expensive. They can still use local disks to provision the OS over a network but boot from local storage, and prevent writing to disks from the booted OS (hell, they can completely remove the disk drivers from the kernel!). It just doesn't make sense to ditch the drives from a reliability standpoint. They're going to have a big outage one day just because they didn't want to deal with drives.
    - encryptluks2 5y ago
      Mullvad and similar providers often colocate or rent servers from multiple local hosting providers. A group of servers going down for them would not be a big deal. Network boot is not difficult/expensive. Many of their servers are using 10Gbit uplinks+ so I take it they get pretty good deals for bandwidth. It isn't like Amazon or other cloud providers that charge an arm/leg for egress. The point of not using local disks is again fairly straightforward, to show that they do not have a stateful storage medium to write logs to. Whether it significantly helps or not is beside the point, they have determined that it helps provide assurance to their customers and additionally showcases a feature for auditors. Network booting loads the OS into RAM, so even if there was a network outage they'd have to restart the severs to cause a problem. From what I know of most VPN solutions though, again a network outage would only affect the group of servers at that data center which isn't their entire operation.
deleted 5y ago
[deleted]
easton 5y ago
https://archive.is/HbjIQ https://archive.is/HbjIQ (it was blocked for me because it's a VPN service provider)
geek_at 5y ago
I love the concept and I even wrote a blog post about how to set up a fully pxe bootable server environment using Alpine Linux [1] (which by default boots from RAM) in 2019. I still use it and it's one of those things that makes recovery or testing so much faster because I don't even need a usb thumb drive [1] https://blog.haschek.at/2019/build-your-own-datacenter-with-pxe-and-alpine.html https://blog.haschek.at/2019/build-your-own-datacenter-with-...
vbezhenar 5y ago
Some information that could be of interest to those running VPN servers. I live in Kazakhstan and recently our government decided to shut down the Internet. But apparently there were ways to get out: they did not filter two TCP ports. My guess it was some "backdoor" put by employees who had to obey the orders but wanted to provide people some way to get around those blocks. Those ports were used to run VPN software. I used Outline VPN on my VPS and it allowed me and my friends to have a working Internet. TLDR: allow specifying port and protocol (TCP/UDP) as some kind of advanced option for those users who need it for some reason. Right now we've got Internet back and it works fine, but who knows when our government will decide to shut it down again. PS mullvad.net website apparently is blocked in Kazakhstan as well. I know that they block popular VPN provider websites, so that should not come as a surprise, but still. I have no idea whether actual VPN subnets are blocked or not.
- Melatonic 5y ago
  I think it is probably more likely those ports were the backdoor the government or their allies was using to function. Everyone always builds in a backdoor for themselves! When I hear about things like this it makes me glad I have a simple satellite communicator - it will only do short text messages but that is a hell of a lot better than nothing. Of course one could get a full satellite based mobile internet device or phone but the plans on those get quite expensive.
gzer0 5y ago
This is awesome, glad that Mullvad is heading in this direction. For reference, ExpressVPN (which has been audited by PwC) introduced this in 2019 [1]. Unfortunately, ever since ExpressVPN was purchased by Kape Technologies (they also own PIA, Cyberghost, Zenmate all of which do not have reliable histories); Mulvad has been the clear choice for a while now. They're also the backend for Mozilla VPN (mozilla just whitelabels from Mulvad [3]) [1] https://www.expressvpn.com/blog/introducing-trustedserver/ [2] https://www.expressvpn.com/blog/pwc-audits-expressvpn-servers-to-confirm-essential-privacy-protections/ [3] https://news.ycombinator.com/item?id=26646510
NaturalPhallacy 5y ago
How do you debug when stuff goes wrong?
jarbus 5y ago
Every year I feel more and more proud to renew my subscription. What a great company.
stjohnswarts 5y ago
Isn't this the same as just running a boot cd or PXE server and running all the data out of RAM drives? I mean we've been doing this for years on linux as hobbyists haven't we? Or does this bring somethign new to the table?

423 points by NabiDev 5y ago

For people wondering how the hell a user can audit the server is diskless or whatever, the goal appears to be using TPM to provide remote attestation for all code in the boot path. See https://www.system-transparency.org/ https://www.system-transparency.org/.

Gigachad 5y ago
How do we audit that the TPM chip is secure? What happened when a bug in the silicone is found later?
- sanxiyn 5y ago
  We assume it is, just like we assume CPU works as advertised. In other words, TPM is part of TCB.
  - Gigachad 5y ago
    So what is the point? I already assume the code on their server is not malicious by using it. What extra trust does an untrusted TPM chip give me?
    - sanxiyn 5y ago
      It reduces TCB. TPM is smaller than entire server.
    - kfreds 5y ago
      System Transparency reduces your trust assumptions on us. As a VPN provider we are in an immense position of power over you. We aim to reduce your trust assumptions on us to a few things that we would need to explicitly lie about in order to betray you. As an example, let's say that we offered any of our users to at any time during the year show up at our office and inspect our VPN hardware, without warning us beforehand. In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense?
      - gravypod 5y ago
        > In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense? Have you ever thought about doing something like that with some big youtube personalities? Maybe have them hire some pen testers, randomly show up to one of your datacenters, and post recordings of what is done and attacks that could be possible. Since your software is open pen testers could prepare some things to try to attack days in advanced. I'd love to see something like this with Level1Techs or something.
        nopcode 5y ago
        > big youtube personalities You mean the same people that are being "sponsored" by VPN providers for years? Why would we want to trust youtubers of all people?
cmeacham98 5y ago
I hate to be this skeptical, but let's say this is 100% possible (I have my doubts, see previous attacks on things like TPMs and SGX, but I digress). You probably could get 90% of the logging capability by putting monitoring in front of and behind the server, and associating connections by traffic/time. It just seems like this goal of using technology to prove they're trustworthy is unlikely to actually work for a VPN company due to the threat models.
- kfreds 5y ago
  You are correct that System Transparency is not a universal remedy for all threat models. Indeed the word "secure" is undefined until you have a threat model. Most threat models are implied and undocumented assumptions. At some stage in an R&D project one should shift from exploration to threat model-driven development. Most people, myself included, tend to focus on technical solutions, and argue back and forth how "oh, but it can be broken using X". System Transparency aims to provide remote auditability assuming (1) the server hardware specification is correct, (2) a correct cryptographic hash of the contents of the SPI flash containing the platform firmware, and (3) a keypair generated on and only accessible to the platform. This is very simplified of course. An attacker aiming to tap incoming and outgoing network traffic from our servers, who has physical access to the VPN server's Ethernet port, or an upstream router, isn't in the scope of System Transparency to protect against. We need to use other means for that.
- azalemeth 5y ago
  Traffic analysis and correlation analysis is indeed a powerful tool, and in general only communicating at a constant bandwidth between all nodes at all times is the only way to completely defeat it (which is what, I understand, some military systems do). That's inherently highly wasteful, however. To get around this, Mullvad offer very transparent comprehensive multi-hop routing systems [1]; you can bounce your wireguard tunnels around in layered wireguard tunnels (a bit á-la tor) by just choosing a series of ports to tunnel on and to. My understanding is that each one of these adds non-deterministic latency to your connection and probably would help to make such attacks harder at the very least, because from the point of view of an "all seeing" adversary the fact that all of these servers talk to each other all the time makes it very much harder to know where any packet could have gone. Yes, you can see each individual link but the metadata is lost. I signed up for Mullvad when the UK's Snooper's Charter came into force and the local health inspectors suddenly had the rights to see my DNS record. Since then, I've had it installed on my router and just route everything through a custom wireguard (originally openvpn) tunnel. I've had some issues with my ISP randomly bandwidth limiting traffic on the odd port to 1 MByte/s, but frankly that makes me more inclined to put everything behind an encrypted tunnel. I don't want my ISP to do traffic shaping and I do want them to just leave me alone and let me communicate in peace. I have absolutely nothing to hide, but now have to accept that I partly live in a country where everything is surveilled all the time, and warantless, unaccountable investigation of my (highly personal!) online habits may be happening. I think Mullvad's excellent product, sensible architecture and reasonable price is worth paying. I'm an academic, unlikely to be of interest to three-letter acronyms, and therefore it matches my needs very well. [1] https://mullvad.net/en/help/multihop-wireguard/ https://mullvad.net/en/help/multihop-wireguard/
kfreds 5y ago
Correct! Thank you for highlighting that. Here are some additional details for those interested. We intend to make use of TPM for remote attestation of the current boot chain, reproducible builds to provide a strong link from source code to build artifacts, and a transparency log for a historical record of previously used boot chains, artifacts, WireGuard server keys, and related signatures. As dtx1 mentioned elsewhere in this thread, diskless VPN infrastructure is currently in use by many other VPN providers. That is not a novel feature of course. What is novel is user-auditability of running VPN infrastructure. We were the first VPN provider to state our intention to make our infrastructure user-auditable AND provide a realistic roadmap with the specific technologies needed to do so. See the link above. I believe the technologies we use in System Transparency will ultimately reshape the VPN provider industry into a highly competitive space focused on maximizing the transparency of VPN infrastructure. Or not, but at least OUR users will be able to audit us. :) Either way we’re looking forward to the future. The opportunity for improvement is immense.
- azalemeth 5y ago
  This all sounds very exciting. Where will you draw the line between public and private – at the moment your consumer-facing app is on github, but less "server side" stuff (in common with many other VPN providers). I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently. I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled)
  - kfreds 5y ago
    > Where will you draw the line between public and private – at the moment your consumer-facing app is on github, but less "server side" stuff (in common with many other VPN providers). All source code for all software on our VPN servers must eventually be public, and all build artifacts must be reproducible by 3rd parties. > I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently. Exactly, but we will also have to measure each artifact in the boot chain into the platform TPM, and allow anyone to issue a challenge to the TPM to get a signed quote of the boot chain measurements. > I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled) We’ll have to constrain our own ability to access the VPN servers. We cannot be allowed arbitrary root access as that would make the TPM measurements meaningless from an audit perspective. Well, you’d be able to conclude we have root access, so not totally meaningless.
franga2000 5y ago
Damn it, this is a really cool use case for TPMs! So far, every use case I've heard has made me wish they were never invented, but this made me reconsider...

sascha_sl 5y ago

Pretty similar to the default (diskless) mode in Alpine, though it lacks the tooling to verify persisted data and the sources apkvols can be applied from at boot aren't that well documented.

dtx1 5y ago

This is hardly a new thing in VPN providers though. I know that perfect privacy[1] and azire vpn[2] both advertise this feature already. [1]https://www.perfect-privacy.com/en/features/without-logs https://www.perfect-privacy.com/en/features/without-logs [2]https://www.azirevpn.com/docs/environment https://www.azirevpn.com/docs/environment

pilif 5y ago
if only there was any proof of this actually being the case and there not being some "accidental" debug log enabled, or some other network level component having "accidental" access to the keys. There's just no good answer to perfect trust-no-one private internet access. If you need to hide all of your traffic from other users in your local network, you can accomplish that in a trust-no-one fashion by running your own VPN endpoint on a server you control which provides better privacy guarantees compared to a centralised commercial VPN whose business model will eventually involve selling your data (once user growth stops but shareholders demand continued revenue growth). But if you need to hide your traffic from anybody but your peer on the internet and you need to hide the fact that you talked to that peer, then, I'm afraid, your out of luck.
- minerva23 5y ago
  Is no-trust ever possible? I thought people create their threat models and verify they can trust those they have to trust.
  - zeepzeep 5y ago
    > people create their threat models and verify they can trust those they have to trust What kind of people? How do you verify you can trust some company?
    - minerva23 5y ago
      Everyone does it informally to varying degrees for varying problems (often times subconsciously). E.g. "What do I know about this person? Can I trust them around my kids?" You verify trustworthiness by research. Who is involved? Do I trust anyone who trusts them? What are their motivations? What would cause them to take action against me? What causes them to protect my interests? What laws are they subjected to (i.e. who can coerce them)? What do they say for themselves? Where do their words fall on the credibility to BS scale? What is their reputation in the community? What do their competitors/adversaries say? What would cause their behavior to change? I won't enumerate all my research on Mullvad. I can say Mozilla attaching their brand to Mullvad's services helped me a lot (trust by proxy). I'll also say that some of their product decisions give credibility to their anonymity claims. Lastly, I found someone who shared a competitive analysis across many providers. I found the analysis trustworthy. Mullvad has some weak points, but was still the best provider for my particular use case.

yamrzou 5y ago

A bit tangential to the main post, but I'd to share a recent positive experience with Mullvad: I am a regular user of Mullvad and recently wanted to try a different VPN, that only provides Wireguard configs (i.e. no native app). I used the default setup. For some reason, my internet connection was flaky, and when it disconnected and reconnected, my traffic leaked. That never happened to me with Mullvad as the app comes with an "Always require VPN" option out of the box and it has always worked reliably.

suyjuris 5y ago
On linux you can create a network namespace exposing only the wireguard network device, so that applications in that namespace cannot leak traffic. Setting this up, however, is quite fiddly in my experience.
- Koffiepoeder 5y ago
  In addition it is probably not a bad idea to block all traffic on wlo1 / eth0, except that to the mullvad server ip's, through some ufw rules. If you forget to configure the namespace for some applications then, it is highly unlikely the app has internet access (ie, it would need its own mullvad/vpn implementation included).
- encryptluks2 5y ago
  I would suggest vopono to do this automatically.
- ComputerGuru 5y ago
  It’s easier and more secure to just create a VM that’s bridged to the VPN interface (regardless of protocol) if you don’t use the VPN for everything but the things you do use it for absolutely must go through it.
  - Melatonic 5y ago
    I think I like this idea the best - simple, effective, and unbreakable due to config changes or updates. Plus it gives you a psychological separation between "VPN related activities" and not. Or you just do everything in the VM. Adds a layer security wise as well to protect your physical system. If you wanted to get really fancy you could have a few different VM's and each one on a different companies VPN

sydthrowaway 5y ago

The next step: dedicated VPN over FPGAs

rekoil 5y ago
*drools*
- sydthrowaway 5y ago
  Wait, Cisco already do this
  - rekoil 5y ago
    They do? Do you have any product names you can share?

joker197cinque 5y ago

The server configuration (and therefore customers account numbers) is stored in Server OS images I suppose, right ? It shouldn't be an issue as far as inspection is concerned, should it ? Also, isn't there a law that enforces logs to be kept for n years ? How is it compatible with diskless setup ?

encryptluks2 5y ago
No, they would not need to be stored in the images. They are deployed as part of provisioning package according to the post. If they were part of the OS image, they would have to update the image anytime a new customer is added, which would be far less optimal than having Ansible or whatever push changes. There is no law in Sweden that requires logs, and the point of a VPN provider is to not store logs. If your VPN provider is keeping logs of your connections then you should look for a new provider.

abcdef009 5y ago

The VPN provider I use have had diskless systems for years. Is this the same thing? https://www.ovpn.com/en/blog/improvement-of-the-physical-security https://www.ovpn.com/en/blog/improvement-of-the-physical-sec...

TacticalCoder 5y ago

> If the computer is powered off, moved or confiscated, there is no data to retrieve. Don't forget to add insta-shutdown when any USB device is connected to the system!

amq 5y ago
Or disable usb in bios entirely
- mschuster91 5y ago
  Disabling USB in BIOS only disables the emulation of classic PS2 keyboards and IDE storage so that old OSes or bootloaders without USB stacks can work with modern equipment. As soon as the OS kernel initializes the PCI bus, USB will work again - however they could go and remove the xHCI modules from the kernel and image.
  - rekoil 5y ago
    Mullvad has a custom-built bare metal UEFI implementation based on coreboot, I assume stboot is an evolution of that, which means it takes as close as you can get to full responsibility for initialization of all system components like processor, chipset, Ethernet, USB, everything. As a result they can absolutely disable USB entirely by never exposing those parts of the device tree to Linux.
    - mschuster91 5y ago
      x86 devices do not have device trees, and for ARM I'd take a guess and say that as long as the PCI root port is exposed to the OS, a PCI re-scan will be enough to wake the USB chipset.
howdydoo 5y ago
And a tilt sensor that cuts power if the chasis is moved

codethief 5y ago

I've been following Mullvad for a long time and my impression (from countless reviews and comments here on HN) has been quite positive. But here's what I don't understand: Why are the servers located in Sweden, a country that's known for online surveillance[0] like no other country in the EU? From the Wikipedia article[1]: > The law permits the signals intelligence agency, National Defense Radio Establishment, to monitor the content of all cross-border cable-based Internet traffic to combat "external threats" such as terrorism and organized crime. [0]: https://www.opendemocracy.net/en/can-europe-make-it/didier-bigo-sergio-carrera-nicholas-hernanz-julien-jeandesboz-joanna-parkin-fra-1/ https://www.opendemocracy.net/en/can-europe-make-it/didier-b... [1]: https://en.wikipedia.org/wiki/Internet_in_Sweden#Internet_censorship_and_surveillance https://en.wikipedia.org/wiki/Internet_in_Sweden#Internet_ce...

y4mi 5y ago
Because it's a swedish company? The location of the servers is kinda irrelevant in that regard. They'd have to provide government access if there is a lawsuit that demands it. If there isn't one than your critique is entirely pointless
- codethief 5y ago
  > Because it's a swedish company? I'm not sure what point you're trying to make. If online privacy is as important to them as they say, they could have easily registered their company (or a subsidiary) in a different EU member state. > They'd have to provide government access if there is a lawsuit that demands it. IANAL but I am not entirely sure this is true in the EU. Either way, my question was "Why are the servers located in Sweden?" Whether or not this due to the company owning the servers being in Sweden is irrelevant.
  - akerl_ 5y ago
    I think you’re overthinking this. The people who run the company are based in Sweden. So they registered the company in Sweden, because that’s where they are. Then they hosted the servers in Sweden, because that’s where they are and where the company is registered. Registering the company somewhere else wouldn’t do them any good when they’re living in Sweden, because the legal system isn’t fooled by sleight of hand like that. Likewise, hosting the servers elsewhere from where they’re based. Both would expand the number of entities with the ability to compel them to disclose data, because as long as the company owners live in Sweden, Sweden has that ability. Unless you’re asking why they didn’t move to another country to start their company, which is surely a larger ask than the “easily” you suggest.

throwaway984393 5y ago

It's a trade-off. If you have no disk, the disk can't fail, but the network can, and the remote PXE server can, and the remote SAN can. You can get into a state where you have to pray no servers reboot. Intermittent errors can be real annoying when it makes provisioning fail. (used to work a server farm that'd do server rebuilds over PXE, and ran a few diskless cluster projects) An alternative is you use a RAID array and mount your disks in read-only mode, or use physically read-only disks and when you have to replace a disk, you pre-mirror the replacement disk. In this way the local disks can be replaced as they fail and there's never a point when the server is at risk of not being able to boot. ......or they could boot from CDROM :)

encryptluks2 5y ago
A network or PXE server can fail regardless, so this are things that always have to be taken into consideration and in those instances then you address those issues. With this type of setup you do not need a remote SAN as it would defeat the purpose of not having external storage that could store logs. Mullvad has servers all over the world, so a temporary failure in one location will not bring down their entire infrastructure.
- deleted 5y ago
  [deleted]
- throwaway984393 5y ago
  It's not just a temporary failure, it's potentially the entire AZ going down hard. High Availability network boot without local storage is very difficult/expensive. They can still use local disks to provision the OS over a network but boot from local storage, and prevent writing to disks from the booted OS (hell, they can completely remove the disk drivers from the kernel!). It just doesn't make sense to ditch the drives from a reliability standpoint. They're going to have a big outage one day just because they didn't want to deal with drives.
  - encryptluks2 5y ago
    Mullvad and similar providers often colocate or rent servers from multiple local hosting providers. A group of servers going down for them would not be a big deal. Network boot is not difficult/expensive. Many of their servers are using 10Gbit uplinks+ so I take it they get pretty good deals for bandwidth. It isn't like Amazon or other cloud providers that charge an arm/leg for egress. The point of not using local disks is again fairly straightforward, to show that they do not have a stateful storage medium to write logs to. Whether it significantly helps or not is beside the point, they have determined that it helps provide assurance to their customers and additionally showcases a feature for auditors. Network booting loads the OS into RAM, so even if there was a network outage they'd have to restart the severs to cause a problem. From what I know of most VPN solutions though, again a network outage would only affect the group of servers at that data center which isn't their entire operation.

deleted 5y ago

[deleted]

easton 5y ago

https://archive.is/HbjIQ https://archive.is/HbjIQ (it was blocked for me because it's a VPN service provider)

geek_at 5y ago

I love the concept and I even wrote a blog post about how to set up a fully pxe bootable server environment using Alpine Linux [1] (which by default boots from RAM) in 2019. I still use it and it's one of those things that makes recovery or testing so much faster because I don't even need a usb thumb drive [1] https://blog.haschek.at/2019/build-your-own-datacenter-with-pxe-and-alpine.html https://blog.haschek.at/2019/build-your-own-datacenter-with-...

vbezhenar 5y ago

Some information that could be of interest to those running VPN servers. I live in Kazakhstan and recently our government decided to shut down the Internet. But apparently there were ways to get out: they did not filter two TCP ports. My guess it was some "backdoor" put by employees who had to obey the orders but wanted to provide people some way to get around those blocks. Those ports were used to run VPN software. I used Outline VPN on my VPS and it allowed me and my friends to have a working Internet. TLDR: allow specifying port and protocol (TCP/UDP) as some kind of advanced option for those users who need it for some reason. Right now we've got Internet back and it works fine, but who knows when our government will decide to shut it down again. PS mullvad.net website apparently is blocked in Kazakhstan as well. I know that they block popular VPN provider websites, so that should not come as a surprise, but still. I have no idea whether actual VPN subnets are blocked or not.

Melatonic 5y ago
I think it is probably more likely those ports were the backdoor the government or their allies was using to function. Everyone always builds in a backdoor for themselves! When I hear about things like this it makes me glad I have a simple satellite communicator - it will only do short text messages but that is a hell of a lot better than nothing. Of course one could get a full satellite based mobile internet device or phone but the plans on those get quite expensive.

gzer0 5y ago

This is awesome, glad that Mullvad is heading in this direction. For reference, ExpressVPN (which has been audited by PwC) introduced this in 2019 [1]. Unfortunately, ever since ExpressVPN was purchased by Kape Technologies (they also own PIA, Cyberghost, Zenmate all of which do not have reliable histories); Mulvad has been the clear choice for a while now. They're also the backend for Mozilla VPN (mozilla just whitelabels from Mulvad [3]) [1] https://www.expressvpn.com/blog/introducing-trustedserver/ [2] https://www.expressvpn.com/blog/pwc-audits-expressvpn-servers-to-confirm-essential-privacy-protections/ [3] https://news.ycombinator.com/item?id=26646510

NaturalPhallacy 5y ago

How do you debug when stuff goes wrong?

jarbus 5y ago

Every year I feel more and more proud to renew my subscription. What a great company.

stjohnswarts 5y ago

Isn't this the same as just running a boot cd or PXE server and running all the data out of RAM drives? I mean we've been doing this for years on linux as hobbyists haven't we? Or does this bring somethign new to the table?