5 ms·

If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the sa

If you're running in one of the big 3 cloud providers, the bottom-level hypervisors are not-linux. This is equivalent. Are you anti-AWS or anti-Azure for the same reason? This is the substrate upon which you will run any virtualized infrastructure.

qmarchi 3y ago
Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support
- refulgentis 3y ago
  Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
  - qmarchi 3y ago
    Correct, that the Hypervisor isn't running Linux. I think the only provider where that would make sense would be Microsoft, where they have their own OS.
  - p_l 3y ago
    Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
    - zokier 3y ago
      Nitro is supposed to be this super customized version of KVM.
  - bpye 3y ago
    Azure runs a version of Windows, see: https://techcommunity.microsoft.com/t5/windows-os-platform-blog/azure-host-os-cloud-host/ba-p/3709528 https://techcommunity.microsoft.com/t5/windows-os-platform-b...
  - bewaretheirs 3y ago
    When your programs are running on a VM, the linux that loads and runs your binaries is not at the bottom; that linux image runs inside a virtual machine which is constructed and supervised by a hypervisor which sits underneath it all. That hypervisor may run on the bare machine (or what passes for a bare machine what with all the sub-ring-zero crud out there), or may run on top of another OS which could be linux or something else. And even if there is linux in the middle and linux at the bottom they could be completely different versions of linux from releases made years apart.
  - antod 3y ago
    > Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything? Yup. eg with Xen the hypervisor wasn't Linux, even if the privileged management VM (dom0) was Linux (or optionally NetBSD in the early days). The very small Xen hypervisor running on the bare metal was not a general purpose OS, and didn't expose any interface itself - it was well hidden and relied on dom0 for administration.
- bewaretheirs 3y ago
  As I understand it, there's linux running on the Google Cloud hardware but the virtualized networking and storage stacks in Google Cloud are google proprietary and largely bypass linux -- in the case of networking see the "Snap: a Microkernel Approach to Host Networking" paper. In contrast, it appears that Oxide is committing to open-source the equivalent pieces of their virtualization platform.
wmf 3y ago
I suspect a lot of people would (irrationally) freak out if they saw how the public cloud works because it's so different from "best practices". Oxide would probably trigger people less if they never mentioned Illumos but that's not really an option when it's open source.
tptacek 3y ago
I don't about EC2 but Lambda and Fargate are presumably Firecracker, which is Linux KVM.
- zokier 3y ago
  AWS "Nitro" hypervisor which powers EC2 is their (very customized) KVM. https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-nitro-system-journey.html https://docs.aws.amazon.com/whitepapers/latest/security-desi...

qmarchi 3y ago

Small note, that's not true for Google Cloud, which runs on top of Linux, though modified. Disclaimer: Former Googler, Cloud Support

refulgentis 3y ago
Another Xoogler here: any idea what they mean by it's not Linux at the bottom for other providers? Like, surely it's _some_ common OS? Either my binaries wouldn't run or AWS is reimplementing Linux so they can, which seems odd. Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything?
- qmarchi 3y ago
  Correct, that the Hypervisor isn't running Linux. I think the only provider where that would make sense would be Microsoft, where they have their own OS.
- p_l 3y ago
  Old AWS used to be Xen, Nitro afaik uses customised VMM and I don't recall if it's not a custom OS or hosted on top of something. Azure is Hyper-V underneath IIRC, a custom variant at least (remember Windows Server Nano? IIRC it was the closest you could get to running it), with sometimes weird things like network cards running Linux and integrating with Windows' built-in SDN facility. Rest of the bigger ones is mainly Linux with occasional Xen and such, but sometimes you can encounter non-trivial VMware deployments.
  - zokier 3y ago
    Nitro is supposed to be this super customized version of KVM.
- bpye 3y ago
  Azure runs a version of Windows, see: https://techcommunity.microsoft.com/t5/windows-os-platform-blog/azure-host-os-cloud-host/ba-p/3709528 https://techcommunity.microsoft.com/t5/windows-os-platform-b...
- bewaretheirs 3y ago
  When your programs are running on a VM, the linux that loads and runs your binaries is not at the bottom; that linux image runs inside a virtual machine which is constructed and supervised by a hypervisor which sits underneath it all. That hypervisor may run on the bare machine (or what passes for a bare machine what with all the sub-ring-zero crud out there), or may run on top of another OS which could be linux or something else. And even if there is linux in the middle and linux at the bottom they could be completely different versions of linux from releases made years apart.
- antod 3y ago
  > Or are they just saying that the VM my binary runs on might be some predictable Linux version, but the underlying thing launching the VM could be anything? Yup. eg with Xen the hypervisor wasn't Linux, even if the privileged management VM (dom0) was Linux (or optionally NetBSD in the early days). The very small Xen hypervisor running on the bare metal was not a general purpose OS, and didn't expose any interface itself - it was well hidden and relied on dom0 for administration.
bewaretheirs 3y ago
As I understand it, there's linux running on the Google Cloud hardware but the virtualized networking and storage stacks in Google Cloud are google proprietary and largely bypass linux -- in the case of networking see the "Snap: a Microkernel Approach to Host Networking" paper. In contrast, it appears that Oxide is committing to open-source the equivalent pieces of their virtualization platform.

wmf 3y ago

I suspect a lot of people would (irrationally) freak out if they saw how the public cloud works because it's so different from "best practices". Oxide would probably trigger people less if they never mentioned Illumos but that's not really an option when it's open source.

tptacek 3y ago

I don't about EC2 but Lambda and Fargate are presumably Firecracker, which is Linux KVM.

zokier 3y ago
AWS "Nitro" hypervisor which powers EC2 is their (very customized) KVM. https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-nitro-system-journey.html https://docs.aws.amazon.com/whitepapers/latest/security-desi...