6 ms·

> Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN 1. Xen Type-1 hypervisor is smaller than KVM/QEMU. 2. Xen "dom0" = Linux/FreeBSD

> Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN 1. Xen Type-1 hypervisor is smaller than KVM/QEMU. 2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS. 3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware. 4. bhyve is Type-2. 5. Xen has Type-2 (uXen). 6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM. 7. pKVM (Arm/Android) is smaller than KVM/Xen. > The Service Management Facility (SMF) is responsible for the supervision of services under illumos.. a [Linux] robust infrastructure product would likely end up using few if any of the components provided by the systemd project, despite there now being something like a hundred of them. Instead, more traditional components would need to be revived, or thoroughly bespoke software would need to be developed, in order to avoid the technological and political issues with this increasingly dominant force in the Linux ecosystem. Is this an argument for Illumos over Linux, or for translating SMF to Linux?

bonzini 2y ago
Talking about "technological and political issues" without mentioning any, or without mentioning which components would need to be revived, sounds a lot like FUD unfortunately. Mixing and matching traditional and systemd components is super common, for example Fedora and RHEL use chrony instead of timesyncd, and NetworkManager instead of networkd.
- actionfromafar 2y ago
  I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".
  - bonzini 2y ago
    But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos. I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for what we know best", there's no need to fake that so much more research went into a decision.
    - panick21_ 2y ago
      The underlying hyperwiser on oxide isn't exposed to the consumers of the API. Just like on Amazon. I think arguably the bhyve over KVM was the more fundamental reason, and bhyve doesn't run on linux anyway.
      - bonzini 2y ago
        Exactly, then why would they be dragged into systemd-or-not-systemd discussion? If you want to use Linux, use either Debian or the CentOS hyperscaler spin (the one that Meta uses) and call it a day. I am obviously biased as I am a KVM (and QEMU) developer myself, but I don't see any other plausible reason other than "we know the Illumos userspace best". Founder mode and all that. As to their choice of hypervisor, to be honest KVM on Illumos was probably not a great idea to begin with, therefore they used bhyve.
        jclulow 2y ago
        FWIW, founder mode didn't exist five years ago when we were getting started! More seriously, though, this document (which I helped write) is an attempt specifically to avoid classic FUD tropes. It's not perfect, but it reflects certainly aspects of my lived experience in trying to get pieces of the Linux ecosystem to work in production settings. While it's true that I'm a dyed in the wool illumos person, being in the core team and so on, I have Linux desktops, and the occasional Linux system in lab environments. I have been supporting customers with all sorts of environments that I don't get to choose for most of my career, including Linux and Windows systems. At Joyent most of our customers were running hardware virtualised Linux and Windows guests, so it's not like I haven't had a fair amount of exposure. I've even spent several days getting SCO OpenServer to run under our KVM, for a customer, because I apparently make bad life choices! As for not discussing the social and political stuff in any depth, I felt at the time (and still do today) that so much ink had been split by all manner of folks talking about LKML or systemd project behaviour over the last decade that it was probably a distraction to do anything other than mention it in passing. As I believe I said in the podcast we did about this RFD recently: I'm not sure if this decision would be right for anybody else or not, but I believe it was and is right for us. I'm not trying to sell you, or anybody else, on making the same calls. This is just how we made our decision.
  - wmf 2y ago
    Instead people rave about Solaris.
- packetlost 2y ago
  The Oxide folks are rather vocal about their distaste for the Linux Foundation. FWIW I think they went with the right choice for them considering they'd rather sign up for maintaining the entire thing themselves than saddling themselves with the baggage of a Linux fork or upstreaming
- netbsdusers 2y ago
  > Talking about "technological and political issues" without mentioning any I don't know why you think none were mentioned - to name one, they link a GitHub issue created against the systemd repository by a Googler complaining that systemd is inappropriately using Google's NTP servers, which at the time were not a public service, and kindly asking for systemd to stop using them. This request was refused and the issue was closed and locked. Behaviour like this from the systemd maintainers can only appear bizarre, childish, and unreasonable to any unprejudiced observer, putting their character and integrity into question and casting doubt on whether they should be trusted with the maintenance of software so integral to at least a reasonably large minority of modern Linux systems.
  - inftech 2y ago
    And people forget that this behavior of systemd devs is present in lots of other core projects of the Linux ecosystem. Unfortunately this makes modern Linux not reliable.
  - suprjami 2y ago
    systemd made time servers a compile-time option and a warn if distros are using the default time servers: https://github.com/systemd/systemd/pull/554 https://github.com/systemd/systemd/pull/554 What's your suggested alternative? Using pool.ntp.org requires a vendor zone. systemd does not consider itself a vendor, it's the distros shipping systemd which are the vendor and should register and use their own vendor zone. I don't care about systemd either way, but your own false representation of facts makes your last paragraph apply to your "argument".
    - bigfatkitten 2y ago
      > What's your suggested alternative? That if they do not wish to ship a safe default, they do not ship a default at all.
      - suprjami 2y ago
        That would have been my preference too.
evandrofisico 2y ago
I've been using xen in production for at least 18 years, and although there is been some development, it is extremely hard to get actual documentation on how to do things with it. There is no place documenting how to integrate the Dom0less/Hyperlaunch in a distribution or how to build infrastructure with it, at best you will find a github repo, with the last commit dated 4 years ago, with little to no information on what to do with the code.
- transpute 2y ago
  > github repo, with the last commit dated 4 years ago Some preparatory work shipped in Xen 4.19. Aug 2024 v4 patch series [1] + Feb 2024 repo [2] has recent dev work. > hard to get actual documentation Hyperlaunch: this [3] repo looks promising, but it's probably easier to ask for help on xen-devel and/or trenchboot-devel [4]. Upstream acceptance is delayed by competing boot requirements for Arm, x86, RISC-V and Power. dom0less: ELC2022 slides [5] and video [6]. [1] https://lists.xenproject.org/archives/html/xen-devel/2024-08/msg01434.html https://lists.xenproject.org/archives/html/xen-devel/2024-08... [2] https://github.com/FidelisPlatform/xen https://github.com/FidelisPlatform/xen [3] https://github.com/apertussolutions/initrd-builder https://github.com/apertussolutions/initrd-builder [4] https://groups.google.com/g/trenchboot-devel https://groups.google.com/g/trenchboot-devel [5] https://www.slideshare.net/StefanoStabellini/static-partitioning-with-xen-linuxrt-and-zephyr-a-concrete-endtoend-example-elc-na-2022 https://www.slideshare.net/StefanoStabellini/static-partitio... [6] https://www.youtube.com/watch?v=CiELAJCuHJg https://www.youtube.com/watch?v=CiELAJCuHJg
dijit 2y ago
Honestly, SMF is superior to SystemD and it’s ironic it came earlier (and, that shows based on the fact that it uses XML as its configuration language.. ick). However, two things are an issue: 1) The CDDL license of SMF makes it difficult to use, or at least that’s what I was told when I asked someone why SMF wasn’t ported to Linux in 2009. 2) SystemD is it now. It’s too complicated to replace and software has become hopelessly dependent on its existence, which is what I mentioned was my largest worry with a monoculture and I was routinely dismissed. So, to answer your question. The argument must be: IllumOS over Linux.
- transpute 2y ago
  > software has become hopelessly dependent on its existence With some effort, Devuan has managed to support multiple init systems, at least for the software packaged by Devuan/Debian. > SMF is superior to SystemD ... [CDDL] OSS workalike opportunity, as new Devuan init system? > The argument must be: IllumOS over Linux. Thanks :)
  - jclulow 2y ago
    SMF is OSS. The CDDL is an OSI approved licence. I'm not aware of any reason one couldn't readily ship user mode CDDL software in a Linux distribution; you don't even have the usual (often specious) arguments about linking and derivative works and so on in that case.
- anonfordays 2y ago
  >Honestly, SMF is superior to SystemD Maybe 15 years ago, not by a mile now. systemd surpassed SMF years ago and it's not even close now. No one in their right mind would pick SMF over systemd in 2024.
  - cthalupa 2y ago
    I regularly pick significantly less featured init systems over systemd whenever it is feasible, because systemd and it's related components have caused some of the largest amounts of work for me over the past decade. I don't really want to litigate the systemd vs. everything else argument, but as someone that has issues with systemd but is not particularly in love with sysvinit derivatives, I wouldn't mind SMF as an alternative.
cthalupa 2y ago
> Is this an argument for Illumos over Linux, or for translating SMF to Linux? I'd certainly like that! I had spent some time working with Solaris a lifetime ago, and ran a good amount of SmartOS infrastructure slightly more recently. I really enjoyed working with SMF. I really do not enjoy working with the systemd sprawl. I will note the distinction between type-1/type-2 hypervisors never really made sense, and makes even less sense today. http://blog.codemonkey.ws/2007/10/myth-of-type-i-and-type-ii-hypervisors.html http://blog.codemonkey.ws/2007/10/myth-of-type-i-and-type-ii...

bonzini 2y ago

Talking about "technological and political issues" without mentioning any, or without mentioning which components would need to be revived, sounds a lot like FUD unfortunately. Mixing and matching traditional and systemd components is super common, for example Fedora and RHEL use chrony instead of timesyncd, and NetworkManager instead of networkd.

actionfromafar 2y ago
I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".
- bonzini 2y ago
  But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos. I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for what we know best", there's no need to fake that so much more research went into a decision.
  - panick21_ 2y ago
    The underlying hyperwiser on oxide isn't exposed to the consumers of the API. Just like on Amazon. I think arguably the bhyve over KVM was the more fundamental reason, and bhyve doesn't run on linux anyway.
    - bonzini 2y ago
      Exactly, then why would they be dragged into systemd-or-not-systemd discussion? If you want to use Linux, use either Debian or the CentOS hyperscaler spin (the one that Meta uses) and call it a day. I am obviously biased as I am a KVM (and QEMU) developer myself, but I don't see any other plausible reason other than "we know the Illumos userspace best". Founder mode and all that. As to their choice of hypervisor, to be honest KVM on Illumos was probably not a great idea to begin with, therefore they used bhyve.
      - jclulow 2y ago
        FWIW, founder mode didn't exist five years ago when we were getting started! More seriously, though, this document (which I helped write) is an attempt specifically to avoid classic FUD tropes. It's not perfect, but it reflects certainly aspects of my lived experience in trying to get pieces of the Linux ecosystem to work in production settings. While it's true that I'm a dyed in the wool illumos person, being in the core team and so on, I have Linux desktops, and the occasional Linux system in lab environments. I have been supporting customers with all sorts of environments that I don't get to choose for most of my career, including Linux and Windows systems. At Joyent most of our customers were running hardware virtualised Linux and Windows guests, so it's not like I haven't had a fair amount of exposure. I've even spent several days getting SCO OpenServer to run under our KVM, for a customer, because I apparently make bad life choices! As for not discussing the social and political stuff in any depth, I felt at the time (and still do today) that so much ink had been split by all manner of folks talking about LKML or systemd project behaviour over the last decade that it was probably a distraction to do anything other than mention it in passing. As I believe I said in the podcast we did about this RFD recently: I'm not sure if this decision would be right for anybody else or not, but I believe it was and is right for us. I'm not trying to sell you, or anybody else, on making the same calls. This is just how we made our decision.
- wmf 2y ago
  Instead people rave about Solaris.
packetlost 2y ago
The Oxide folks are rather vocal about their distaste for the Linux Foundation. FWIW I think they went with the right choice for them considering they'd rather sign up for maintaining the entire thing themselves than saddling themselves with the baggage of a Linux fork or upstreaming
netbsdusers 2y ago
> Talking about "technological and political issues" without mentioning any I don't know why you think none were mentioned - to name one, they link a GitHub issue created against the systemd repository by a Googler complaining that systemd is inappropriately using Google's NTP servers, which at the time were not a public service, and kindly asking for systemd to stop using them. This request was refused and the issue was closed and locked. Behaviour like this from the systemd maintainers can only appear bizarre, childish, and unreasonable to any unprejudiced observer, putting their character and integrity into question and casting doubt on whether they should be trusted with the maintenance of software so integral to at least a reasonably large minority of modern Linux systems.
- inftech 2y ago
  And people forget that this behavior of systemd devs is present in lots of other core projects of the Linux ecosystem. Unfortunately this makes modern Linux not reliable.
- suprjami 2y ago
  systemd made time servers a compile-time option and a warn if distros are using the default time servers: https://github.com/systemd/systemd/pull/554 https://github.com/systemd/systemd/pull/554 What's your suggested alternative? Using pool.ntp.org requires a vendor zone. systemd does not consider itself a vendor, it's the distros shipping systemd which are the vendor and should register and use their own vendor zone. I don't care about systemd either way, but your own false representation of facts makes your last paragraph apply to your "argument".
  - bigfatkitten 2y ago
    > What's your suggested alternative? That if they do not wish to ship a safe default, they do not ship a default at all.
    - suprjami 2y ago
      That would have been my preference too.

evandrofisico 2y ago

I've been using xen in production for at least 18 years, and although there is been some development, it is extremely hard to get actual documentation on how to do things with it. There is no place documenting how to integrate the Dom0less/Hyperlaunch in a distribution or how to build infrastructure with it, at best you will find a github repo, with the last commit dated 4 years ago, with little to no information on what to do with the code.

transpute 2y ago
> github repo, with the last commit dated 4 years ago Some preparatory work shipped in Xen 4.19. Aug 2024 v4 patch series [1] + Feb 2024 repo [2] has recent dev work. > hard to get actual documentation Hyperlaunch: this [3] repo looks promising, but it's probably easier to ask for help on xen-devel and/or trenchboot-devel [4]. Upstream acceptance is delayed by competing boot requirements for Arm, x86, RISC-V and Power. dom0less: ELC2022 slides [5] and video [6]. [1] https://lists.xenproject.org/archives/html/xen-devel/2024-08/msg01434.html https://lists.xenproject.org/archives/html/xen-devel/2024-08... [2] https://github.com/FidelisPlatform/xen https://github.com/FidelisPlatform/xen [3] https://github.com/apertussolutions/initrd-builder https://github.com/apertussolutions/initrd-builder [4] https://groups.google.com/g/trenchboot-devel https://groups.google.com/g/trenchboot-devel [5] https://www.slideshare.net/StefanoStabellini/static-partitioning-with-xen-linuxrt-and-zephyr-a-concrete-endtoend-example-elc-na-2022 https://www.slideshare.net/StefanoStabellini/static-partitio... [6] https://www.youtube.com/watch?v=CiELAJCuHJg https://www.youtube.com/watch?v=CiELAJCuHJg

dijit 2y ago

Honestly, SMF is superior to SystemD and it’s ironic it came earlier (and, that shows based on the fact that it uses XML as its configuration language.. ick). However, two things are an issue: 1) The CDDL license of SMF makes it difficult to use, or at least that’s what I was told when I asked someone why SMF wasn’t ported to Linux in 2009. 2) SystemD is it now. It’s too complicated to replace and software has become hopelessly dependent on its existence, which is what I mentioned was my largest worry with a monoculture and I was routinely dismissed. So, to answer your question. The argument must be: IllumOS over Linux.

transpute 2y ago
> software has become hopelessly dependent on its existence With some effort, Devuan has managed to support multiple init systems, at least for the software packaged by Devuan/Debian. > SMF is superior to SystemD ... [CDDL] OSS workalike opportunity, as new Devuan init system? > The argument must be: IllumOS over Linux. Thanks :)
- jclulow 2y ago
  SMF is OSS. The CDDL is an OSI approved licence. I'm not aware of any reason one couldn't readily ship user mode CDDL software in a Linux distribution; you don't even have the usual (often specious) arguments about linking and derivative works and so on in that case.
anonfordays 2y ago
>Honestly, SMF is superior to SystemD Maybe 15 years ago, not by a mile now. systemd surpassed SMF years ago and it's not even close now. No one in their right mind would pick SMF over systemd in 2024.
- cthalupa 2y ago
  I regularly pick significantly less featured init systems over systemd whenever it is feasible, because systemd and it's related components have caused some of the largest amounts of work for me over the past decade. I don't really want to litigate the systemd vs. everything else argument, but as someone that has issues with systemd but is not particularly in love with sysvinit derivatives, I wouldn't mind SMF as an alternative.

cthalupa 2y ago

> Is this an argument for Illumos over Linux, or for translating SMF to Linux? I'd certainly like that! I had spent some time working with Solaris a lifetime ago, and ran a good amount of SmartOS infrastructure slightly more recently. I really enjoyed working with SMF. I really do not enjoy working with the systemd sprawl. I will note the distinction between type-1/type-2 hypervisors never really made sense, and makes even less sense today. http://blog.codemonkey.ws/2007/10/myth-of-type-i-and-type-ii-hypervisors.html http://blog.codemonkey.ws/2007/10/myth-of-type-i-and-type-ii...