5 ms·

Are these physical servers or VM's? I ask because some VM types can be frozen/shapshot/cloned or live replicated including memory contents as is done in some V

Are these physical servers or VM's? I ask because some VM types can be frozen/shapshot/cloned or live replicated including memory contents as is done in some VPS providers for lawful requests. From the bare metal host anything can be accessed in a VM or container. Do they have a diagram of their physical setup? [Edit] - Are there any Mullvad architects here that can help us avoid going down theoretical hypothetical rabbit holes and turtle stacks?

dijit 3y ago
Mullvad has been pushing system transparency a lot (which is a security system specifically designed for bare-metal), so it's fairly safe to say that it's based on bare-metal. https://www.system-transparency.org https://www.system-transparency.org
- LinuxBender 3y ago
  Maybe it should be safe to say? but do they actually have a diagram of their physical and logical setup that shows they use bare metal only and not VM's or containers? Is that documented in audited security controls?
  - dijit 3y ago
    Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security. It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security. Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question. They've also discussed booting UEFI directly to VPN nodes: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructure-beta-system-transparency-stboot https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur... It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.
    - kaszanka 3y ago
      > Containers, btw, cannot be snapshotted Why not?
      - anonacct37 3y ago
        criu has been used for live migration for a while. Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.
    - LinuxBender 3y ago
      Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security. I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question. Containers, btw, cannot be snapshotted. I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted. They've also discussed booting UEFI directly to VPN nodes: That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this. It's completely weird to argue that they might introduce a layer of insecurity here. Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentation unless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.
      - ziddoap 3y ago
        >They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question. They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit. >On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant. >After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information. https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subject-to-a-search-warrant-customer-data-not-compromised https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
  - tiagod 3y ago
    Even if they did... if mullvad was a honeypot, there is nothing that would stop them from posting a fake diagram.
k8sToGo 3y ago
Physical server can also be frozen (literally) and then data extracted from RAM.
- LinuxBender 3y ago
  It can but to my knowledge law enforcement in the US anyway have only done this once for a national security incident in the real world outside of a lab. If you have any links to this becoming common practice outside of a lab demo or security presentation on their own hardware I would be interested to save in my own documentation. They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went. [Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocation with a performance hit. Both are enabled by default on modern kernels now. init_on_alloc=1 init_on_free=1
  - mschuster91 3y ago
    There exist special UPSes that you can attach to a power line of a server, unplug and unmount it, and keep the server powered all the time, without the server being able to detect that. If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.
    - tln 3y ago
      Whoa! Any keyword suggestions to find those UPSes?
      - Eldt 3y ago
        I'm having trouble remembering what they're called, but I do remember reading about these. I saw a demonstration where they slightly pulled at the desktop power plug in order to hook a thin device around the plug connectors to provide power, and then they could transport the desktop around.
xyst 3y ago
If you are asking this, you probably need to self host your VPN
- LinuxBender 3y ago
  I do this but I do not use them for privacy. I use Tinc [1] to route around internet outages as it can do user-space dynamic mesh routing. Since they are all nodes I pay for they would not hide who I am. While I do control the logging of Tinc I do not control external logging of the server and VPS providers. For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi. [1] - https://www.tinc-vpn.org/ https://www.tinc-vpn.org/
WirelessGigabit 3y ago
One can fix that by deploying software with the VM pause functionality removed and / or remove disks the bare-metal servers and only PXE-boot them. That way there is nothing to pause.
LinuxBender 3y ago
The more I think about this VM's are not required. There are tools that can extract data streams based on specific parameters on demand using eBPF. Perhaps tools like Sysdig/Falco, etc... It might also be useful to know if they custom compile kernels to disable eBPF and if they do any additional specific kernel hardening to strip out or disable on demand debugging capabilities.

dijit 3y ago

Mullvad has been pushing system transparency a lot (which is a security system specifically designed for bare-metal), so it's fairly safe to say that it's based on bare-metal. https://www.system-transparency.org https://www.system-transparency.org

LinuxBender 3y ago
Maybe it should be safe to say? but do they actually have a diagram of their physical and logical setup that shows they use bare metal only and not VM's or containers? Is that documented in audited security controls?
- dijit 3y ago
  Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security. It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security. Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question. They've also discussed booting UEFI directly to VPN nodes: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructure-beta-system-transparency-stboot https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur... It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.
  - kaszanka 3y ago
    > Containers, btw, cannot be snapshotted Why not?
    - anonacct37 3y ago
      criu has been used for live migration for a while. Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.
  - LinuxBender 3y ago
    Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security. I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question. Containers, btw, cannot be snapshotted. I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted. They've also discussed booting UEFI directly to VPN nodes: That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this. It's completely weird to argue that they might introduce a layer of insecurity here. Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentation unless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.
    - ziddoap 3y ago
      >They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question. They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit. >On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant. >After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information. https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subject-to-a-search-warrant-customer-data-not-compromised https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
- tiagod 3y ago
  Even if they did... if mullvad was a honeypot, there is nothing that would stop them from posting a fake diagram.

k8sToGo 3y ago

Physical server can also be frozen (literally) and then data extracted from RAM.

LinuxBender 3y ago
It can but to my knowledge law enforcement in the US anyway have only done this once for a national security incident in the real world outside of a lab. If you have any links to this becoming common practice outside of a lab demo or security presentation on their own hardware I would be interested to save in my own documentation. They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went. [Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocation with a performance hit. Both are enabled by default on modern kernels now. init_on_alloc=1 init_on_free=1
- mschuster91 3y ago
  There exist special UPSes that you can attach to a power line of a server, unplug and unmount it, and keep the server powered all the time, without the server being able to detect that. If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.
  - tln 3y ago
    Whoa! Any keyword suggestions to find those UPSes?
    - Eldt 3y ago
      I'm having trouble remembering what they're called, but I do remember reading about these. I saw a demonstration where they slightly pulled at the desktop power plug in order to hook a thin device around the plug connectors to provide power, and then they could transport the desktop around.

xyst 3y ago

If you are asking this, you probably need to self host your VPN

LinuxBender 3y ago
I do this but I do not use them for privacy. I use Tinc [1] to route around internet outages as it can do user-space dynamic mesh routing. Since they are all nodes I pay for they would not hide who I am. While I do control the logging of Tinc I do not control external logging of the server and VPS providers. For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi. [1] - https://www.tinc-vpn.org/ https://www.tinc-vpn.org/

WirelessGigabit 3y ago

One can fix that by deploying software with the VM pause functionality removed and / or remove disks the bare-metal servers and only PXE-boot them. That way there is nothing to pause.

LinuxBender 3y ago

The more I think about this VM's are not required. There are tools that can extract data streams based on specific parameters on demand using eBPF. Perhaps tools like Sysdig/Falco, etc... It might also be useful to know if they custom compile kernels to disable eBPF and if they do any additional specific kernel hardening to strip out or disable on demand debugging capabilities.