20 ms·

Why Oxide Chose Illumos

226 points by kblissett 2y ago

Rendello 2y ago
I like the RFDs. Oxide just did a podcast episode on the process: https://oxide.computer/podcasts/oxide-and-friends/2065190 https://oxide.computer/podcasts/oxide-and-friends/2065190
rtpg 2y ago
> There is not a significant difference in functionality between the illumos and FreeBSD implementations, since pulling patches downstream has not been a significant burden. Conversely, the more advanced OS primitives in illumos have resulted in certain bugs being fixed only there, having been difficult to upstream to FreeBSD. curious about what bugs are being thought of there. Sounds like a very interesting situation to be in
taspeotis 2y ago
I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-computer-company/ https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...
- gyre007 2y ago
  Yeah I came here to say that Bryan worked at Sun so why do they even need to write this post (yes, I appreciate the techinical reasons, just wanted to highlight the fact via a subtle dig :-))
  - sausagefeet 2y ago
    This isn't a blog post from an Oxide, it's a link to their internal RFD which they use to make decisions.
    - gyre007 2y ago
      I never said it was a post by Oxide.
      - sausagefeet 2y ago
        Might want to re-read your initial comment, then.
- sausagefeet 2y ago
  Early Oxide founders came from Joyent which was an illumos shop and Cantrill is quite vocal about the history of Solaris, OpenSolaris, and illumos.
  - codetrotter 2y ago
    > Joyent which was an illumos shop And before that, they used to run FreeBSD. Mentioned for example in this comment by Bryan Cantrill a decade ago: https://news.ycombinator.com/item?id=6254092 https://news.ycombinator.com/item?id=6254092 > […] Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that […] And I’ve seen some other primary sources (people who worked at Joyent) write that online too. And Bryan Cantrill, and several other people, came from Sun Microsystems to Joyent. Though I’ve never seen it mentioned which order that happened in; was it people from Sun that joined Joyent and then Joyent switched from FreeBSD to Illumos and creating SmartOS? Or had Joyent already switched to Illumos before the people that came from Sun joined? I would actually really enjoy a long documentary or talk from some people that worked at Joyent about the history of the company, how they were using FreeBSD and when they switched to Illumos and so on.
    - panick21_ 2y ago
      Joyent was using Solaris before Bryan worked there. Listen to the this podcast with Bryan and his co-founder about their origin story: https://www.youtube.com/watch?v=eVkIKm9pkPY https://www.youtube.com/watch?v=eVkIKm9pkPY This is about as good as you are gone get on the topic of Joyant history.
      - codetrotter 2y ago
        Thank you, I will watch that right away :)
    - selykg 2y ago
      Joyent also merged with TextDrive, which is where the FreeBSD part came from. TextDrive was an early Rails host, and could even do it in a shared hosting environment, which is where I think a lot of the original user base came from (also TextPattern) As I recall they were also the original host of Twitter, which if I recall was Rails back in the day.
- mzi 2y ago
  @bcantrill is the CTO of Oxide.
  - taspeotis 2y ago
    Yup, thanks
- panick21_ 2y ago
  Bryan Cantrill also ported KVM to Illumos. At Joyent they had plenty of experience with KVM. See: https://www.youtube.com/watch?v=cwAfJywzk8o https://www.youtube.com/watch?v=cwAfJywzk8o As far as I know, Bryan didn't personally work on the porting of bhyve (this might be wrong). So if anything, that would point to KVM as the 'familiar' thing given how many former Joyant people were there.
  - bonzini 2y ago
    KVM got more and more integrated with the rest of Linux as more virtualization features became general system features (e.g. posted interrupts). Also Google and Amazon are working more upstream and the pace of development increased a lot. Keeping a KVM port up to date is a huge effort compared to bhyve, and they probably had learnt that in the years between the porting of KVM and the founding of Oxide.
  - elijahwright 2y ago
    Where is Max Bruning these days?
- kafHas 2y ago
  [flagged]
bonzini 2y ago
> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf video - https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y
- _rs 2y ago
  I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?
  - bonzini 2y ago
    AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances. Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever it is called these days, crosvm for ChromeOS, and QEMU for Android Emulator.
    - tptacek 2y ago
      Lambda and Fargate.
      - dastbe 2y ago
        unless something has changed in the past year, fargate still runs each task in a single use ec2 vm with no further isolation around containers in a task.
        deleted 2y ago
        [deleted]
sausagefeet 2y ago
While it's fair to say this does describe why Illumos was chosen, the actual RFD title is not presented and it is about Host OS + Virtualization software choice. Even if you think it's a foregone conclusion given the history of bcantrill and other founders of Oxide, there absolutely is value in putting decision to paper and trying to provide a rational because then it can be challenged. The company I co-founded does an RFD process as well and even if there is 99% chance that we're going to use the thing we've always used, if you're a serious person, the act of expressing it is useful and sometimes you even change your own mind thanks to the process.
transpute 2y ago
> Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN 1. Xen Type-1 hypervisor is smaller than KVM/QEMU. 2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS. 3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware. 4. bhyve is Type-2. 5. Xen has Type-2 (uXen). 6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM. 7. pKVM (Arm/Android) is smaller than KVM/Xen. > The Service Management Facility (SMF) is responsible for the supervision of services under illumos.. a [Linux] robust infrastructure product would likely end up using few if any of the components provided by the systemd project, despite there now being something like a hundred of them. Instead, more traditional components would need to be revived, or thoroughly bespoke software would need to be developed, in order to avoid the technological and political issues with this increasingly dominant force in the Linux ecosystem. Is this an argument for Illumos over Linux, or for translating SMF to Linux?
- bonzini 2y ago
  Talking about "technological and political issues" without mentioning any, or without mentioning which components would need to be revived, sounds a lot like FUD unfortunately. Mixing and matching traditional and systemd components is super common, for example Fedora and RHEL use chrony instead of timesyncd, and NetworkManager instead of networkd.
  - actionfromafar 2y ago
    I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".
    - bonzini 2y ago
      But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos. I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for what we know best", there's no need to fake that so much more research went into a decision.
saagarjha 2y ago
Unrelated, but is this a homegrown publishing platform?
- crb 2y ago
  Yes; it's referred to in Oxide's RFD about RFDs [1] https://rfd.shared.oxide.computer/rfd/0001 https://rfd.shared.oxide.computer/rfd/0001 but the referenced URL is 404 unless you're an Oxide employee. [1] https://rfd.shared.oxide.computer/rfd/0001#_shared_rfd_rendered_site https://rfd.shared.oxide.computer/rfd/0001#_shared_rfd_rende... [2] https://github.com/oxidecomputer/rfd/blob/master/src https://github.com/oxidecomputer/rfd/blob/master/src
  - dcre 2y ago
    That link is out of date. The site and backend are now open source. Only the repo containing the RFD contents is private. https://github.com/oxidecomputer/rfd-site https://github.com/oxidecomputer/rfd-site https://github.com/oxidecomputer/rfd-api https://github.com/oxidecomputer/rfd-api
- benjaminleonard 2y ago
  Yep, you can see a little more on the [blog](https://oxide.computer/blog/a-tool-for-discussion https://oxide.computer/blog/a-tool-for-discussion) or the most recent [podcast](https://oxide.computer/podcasts/oxide-and-friends/2065190 https://oxide.computer/podcasts/oxide-and-friends/2065190). The API and site repos are also public.
ReleaseCandidat 2y ago
Instead of stating more or less irrelevant reasons, I'd prefer to read something like "I am (have been?) one of the core maintainers and know Illumos and Bhyve, so even if there would be 'objectively' better choices, our familiarity with the OS and hypervisor trump that". A "I like $A, always use $A and have experience using $A" is almost always a better argument than "$A is better than $B because $BLA", because that doesn't tell me anything about the depth of knowledge of using $A and $B or the knowledge of the subject of decision - there is a reason half of Google's results is some kind of "comparison" spam.
- actionfromafar 2y ago
  But everyone at Oxide already knows that back story. At least if you list some other reasons list you can have a discussion about technical merits if you want to.
  - ReleaseCandidat 2y ago
    But that doesn't make sense if you have specialists for $A that also like to work with $A. Why should I as a customer trust Illumos/Bhye developers that are using Linux/KVM instead of "real" Linux/KVM developers? The only thing that such a decision would tell me is to not even think about using Illumos or Bhyve. The difference between "Buy our Illumos/Bhye solution! Why? I have been an Illumos/Bhyve Maintainer!" and "Buy our Linux/KVM solution! Why? I have been an Illumos/Bhyve Maintainer!" should make my point a bit clearer
    - panick21_ 2y ago
      Those are not the only options. You can have KVM on Illumos, or Bhye on FreeBSD. And finding people to heir that know Linux/KVM wouldn't be a problem for them. This evaluation was done years ago and they added like 50 people since then. Saying 'We have a great KVM Team but our CEO was once an Illumos developer' is perfectly reasonable. And as I point out in my other comment, the former Joyant people like know more about KVM then anything else anyway. So it would be: "Buy our KVM Solution, we have KVM experts" But they evaluated that Bhyve was better then KVM despite that.
blinkingled 2y ago
I guess that's one way of keeping Solaris alive :)
daneel_w 2y ago
"• Emerging VMMs (OpenBSD’s vmm, etc): Haven’t been proven in production" It's a small operation, but https://openbsd.amsterdam/ https://openbsd.amsterdam/ have absolutely proven that OpenBSD's hypervisor is production-capable in terms of stability - but there are indeed other problems that rule against it on scale. For those who are unfamiliar with OpenBSD: the primary caveat is that its hypervisor can so far only provide guests with a single CPU core.
- jclulow 2y ago
  Yes, to be clear this is not meant to be a criticism of software quality at OpenBSD! Though I don't necessarily always agree with the leadership style I have big respect for their engineering efforts and obviously as another relatively niche UNIX I feel a certain kinship! That part of the document was also written some years ago, much closer to 2018 when that service got started than now, so it's conceivable that we wouldn't have said the same thing today. I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product!
  - notaplumber1 2y ago
    > I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product! Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development? From a secure design perspective, OpenBSD's vmd is a lot more appealing than bhyve is today. I saw recently that AMD SEV (Secure Encrypted Virtualization) was added, which seems compelling for Oxide's AMD based platform. Has Oxide added support for that to their bhyve fork yet?
    - pmooney 2y ago
      > Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development? Being that vmd's values are aligned with OpenBSD's (security above all else), it is probably not a good fit for what Oxide is trying to achieve. Last I looked at vmd (circa 2019), it was doing essentially all device emulation in userspace. While it makes total sense to keep as much logic as possible out of ring-0 (again, emphasis on security), doing so comes with some substantial performance costs. Heavily used devices, such as the APIC, will incur pretty significant overhead if the emulation requires round trips out to userspace on top of the cost of VM exits. > I saw recently that AMD SEV (Secure Encrypted Virtualization) was added, which seems compelling for Oxide's AMD based platform. Has Oxide added support for that to their bhyve fork yet? SEV complicates things like the ability to live-migrate guests between systems.
tonyg 2y ago
> Nested virtualisation [...] challenging to emulate the underlying interfaces with flawless fidelity [...] dreadful performance It is so sad that we've ended up with designs where this is the case. There is no intrinsic reason why nested virtualization should be hard to implement or should perform poorly. Path dependence strikes again.
- bonzini 2y ago
  It doesn't perform poorly in fact. It can be tuned at 90% of non-nested virtualization, and for workloads where it doesn't, that's more than anything else a testimony to how close virtualized performance is to bare-metal. That said, it does add a lot of complexity.
fefe23 2y ago
These sound like reason you retconned so it sounds like you didn't choose Illumos because your founder used to work at Sun and Joyent before. :-) Frankly I don't understand why they blogged that at all. It reeks of desperation, like they feel they need to defend their choice. They don't. It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.
- jclulow 2y ago
  It's not a blog post, it's an RFD. We have a strong focus on writing as part of thinking and making decisions, and when we can, we like to publish our decision making documents in the spirit of open source. This is not a defence of our position so much as a record of the process through which we arrived at it. This is true of our other RFDs as well, which you can see on the site there. > It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details. Yes, the whole product is definitely designed that way intentionally. Customers get abstracted control of compute and storage resources through cloud style APIs. From their perspective it's a cloud appliance. It's only from our perspective as the people building it that it's a UNIX system.
  - stonogo 2y ago
    So at no point did anyone even suspect that Illumos was under consideration because it's been corporate leadership's pet project for decades? That seems like a wild thing to omit from the "RFD" process. Or were some topics not open to the "RFD" process?
    - jclulow 2y ago
      We are trying to build a business here. The goal is to sell racks and racks of computers to people, not build a menagerie of curiosities and fund personal projects. Everything we've written here is real, at least from our point of view. If we didn't think it would work, why would we throw our own business, and equity, and so on, away? The reason I continue to invest myself, if nothing else, in illumos, is because I genuinely believe it represents a better aggregate trade off for production work than the available alternatives. This document is an attempt to distill why that is, not an attempt to cover up a personal preference. I do have a personal preference, and I'm not shy about it -- but that preference is based on tangible experiences over twenty years!
leoh 2y ago
I’d love to use Illumos, but a lack of arm64 support is a non-starter
- jclulow 2y ago
  Folks are working on it! I believe it boots on some small systems and under QEMU, but it's still relatively early days. I'm excited for the port to eventually make it into the gate, though!
  - geerlingguy 2y ago
    In before someone asks about riscv64 ;)
- ComputerGuru 2y ago
  I don’t mean to downplay the importance for you personally but I do want to clarify that while it might be a non-starter for you, all of arm64 is so new that it’s hardly a non-starter for anyone considering putting it into (traditional) production.
  - timenova 2y ago
    You're right, however I was looking for the same information to maybe try it on a RPi to learn more about Illumos.
BirAdam 2y ago
I think a bigger reason for Oxide using Illumos is that many of the people over there are former Sun folks.
throw0101b 2y ago
Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode: * https://www.youtube.com/watch?v=UvEKSqBBcZw https://www.youtube.com/watch?v=UvEKSqBBcZw Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: "Ceph is operated, not shipped [like ZFS]". There's more care-and-feeding required for it, and they probably don't want that as they want to treat product in a more appliance/toaster-like fashion.
- pclmulqdq 2y ago
  Ceph is sadly not very good at what it does. The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). ZFS solves a different problem, though. ZFS is a full-featured filesystem. Like Ceph it is also vulnerable to single points of failure.
  - throw0101b 2y ago
    > The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023: * https://www.youtube.com/watch?v=bl6H888k51w https://www.youtube.com/watch?v=bl6H888k51w Certainly there are some number of folks that need more than that, but I don't there are many. > Like Ceph it is also vulnerable to single points of failure. The SPOF for ZFS is the host (unless you replicate, e.g., zfs send). What is SPOF of Ceph? You can have multiple monitors, managers, and MDSes.
    - pclmulqdq 2y ago
      Single-monitor is a common way to run Ceph. On top of that, many cluster configurations cause the whole thing to slow to a crawl when a very small minority of nodes go down. Never mind packet loss, bad switches, and other sorts of weird failure mechanisms. Ceph in general is pretty bad at operating in degraded modes. ZFS and systems like Tectonic (FB) and Colossus (Google) do much better when things aren't going perfectly. Do you know how many administrators CERN has for its Ceph clusters? Google operates Colossus at ~1000x that size with a team of 20-30 SREs (almost all of whom aren't spending their time doing operations).
jonstewart 2y ago
Illumos makes sense as a host OS—it’s capable, they know it, they can make sure it works well on their hardware, and virtualization means users don’t need that much familiarity with it. If I were Oxide, though, I’d be sprinting to seamless VMWare support. Broadcom has turned into a modern-day Oracle (but dumber??) and many customers will migrate in the next two years. Even if those legacy VMs aren’t “hyperscale”, there’s going to be lots of budget devoted to moving off VMWare.
- parasubvert 2y ago
  Oracle is a $53 billion company, and never had a mass exodus, just less greenfield deployments. Broadcom also isn't all that dumb, VMware was fat and lazy and customers were coddled for a very long time. They've made a bet that it's sticky. The competition isn't as weak as they thought, that's true, but it will take 5+ years to catch up, not 2 years, in general. Broadcom was betting on it taking 10 years: plenty of time to squeeze out margins. Customers have been trying and failing to eliminate the vTax since OpenStack. Red Hat and Microsoft are the main viable alternatives.
  - jonstewart 2y ago
    I don’t disagree much. Still, there’s a sudden weakening of the main incumbent in the on-prem virtualization market… and that is _all_ Oxide does. It will be interesting to see whether Oxide can convert some VMWare customers.
    - meepmorp 2y ago
      Maybe some, but the main difference between Oxide and the others is that they sell an integrated platform, so if a customer is just looking to replace the VM software, that seems like a harder sell.
    - parasubvert 2y ago
      Oxide is a moonshot, it requires companies to fire a lot more suppliers than VMware. Which is easier said than done. A lot will depend on if Oxide can find their early niches: they're a long play (years) before they crack the mainstream or are truly competitive with, say, VCF on VxRail, for the average talent ops team. Long run it's clearly going to be better, but short run they have to pick their battles.
kayo_20211030 2y ago
Is the date on this piece correct? The section about Rust as a first class citizen seems to contain references to its potential use in Linux that are a few years out of date; with nothing more current than 2021. > As of March 2021, work on a prototype for writing Linux drivers in Rust is happening in the linux-next tree.
- kayo_20211030 2y ago
  nm, I read the postscript. The RFD was from 2021. I wonder how correct it was, and whether decisions made, based on it, were good ones or bad ones.
alberth 2y ago
Isn’t it simply Oxide founders are old Sun engineers, and Illumos is the open source spinoff of their old work.
- sophacles 2y ago
  According to the founders and early engineers on their podcast - no, they tried to fairly evaluate all the oses and were willing to go with other options. Practically speaking, its hard to do it completely objectively and the in-house expertise probably colored the decision.
  - pclmulqdq 2y ago
    Tried to, sure, but when you evaluate other products strictly against the criteria under which you built your own version, you know what the conclusion will be. Never mind that you are carrying your blind spots with you. I would say that there was an attempt to evaluate other products, but not so much an attempt to be objective in that evaluation. In general, being on your own private tech island is a tough thing to do, but many engineers would rather do that than swallow their pride.
Aissen 2y ago
Point 1.1 about QEMU seems even less relevant today, with QEMU adding support for the microvm machines, hence greatly reducing the amount of exposed code. And as bonzini said in the thread, the recent vulnerability track record is not so bad.
magicalhippo 2y ago
Been running Bhyve on FreeBSD (technically FreeNAS). Found PCIe pass-through of NMVe drives was fairly straight forward once the correct incantations were found, but network speed to host has been fairly abysmal. On my admittedly aging Threadripper 1920X, I can only get ~2-3 Gbps peak from a Linux guest. That's with virtio, the virtual intel "card" is even slower. They went with Illumos though, so curious if the poor performance is a FreeBSD-specific thing.
- bitfilped 2y ago
  It's been a minute since I messed with bhyve on FreeBSD, but I'm pretty sure you have to switch out the networking stack to something like Netgraph if you intend to use fast networking.
  - craftkiller 2y ago
    Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through multiple additional layers that may have slowed it down (off the top of my head: kubernetes with flannel, iptables in the VM, bhyve, and pf on the FreeBSD host).
    - bitfilped 2y ago
      Do you know if you're still using if_bridge? I remembered this article from klara that goes a bit more into the details. https://klarasystems.com/articles/using-netgraph-for-freebsds-bhyve-networking/ https://klarasystems.com/articles/using-netgraph-for-freebsd...
      - craftkiller 2y ago
        Thanks, but I am not using if_bridge. I am creating a netgraph bridge[0] which is connected to the host via a netgraph eiface. Then on the host, the packet passes to my real physical interface because I have gateway_enable and I have pf perform NAT[1]. It looks like that blog post connected the netgraph bridge directly to the external interface, so my guess is my slowdown is from either pf performing NAT or the packet forwarding from gateway_enable. [0] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/bhyve/files/bhyve_netgraph_bridge.bash#L214 [1] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/firewall/files/mrmanager_pf.conf#L35
tcdent 2y ago
Linux has a rich ecosystem, but the toolkit is haphazard and a little shakey. Sure, everyone uses it, because when we last evaluated our options (in like 2009) it was still the most robust solution. That may no longer be the case. Given all of that, and taking into account building a product on top of it, and thus needing to support it and stand behind it, Linux wasn't the best choice. Looking ahead (in terms of decades) and not just shipping a product now, it was found that an alternate ecosystem existed to support that. Culture of the community, design principles, maintainability are all things to consider beyond just "is it popular". Exciting times in computing once again!
craftkiller 2y ago
I wonder if CockroachDB abandoning the open source license[0] will have an impact on their choice to use it. It looks like the RFD was posted 1 day before the license switch[1], and the RFD has a section on licenses stating they intended to stick to the OSS build: > To mitigate all this, we’re intending to stick with the OSS build, which includes no CCL code. [0] https://news.ycombinator.com/item?id=41256222 [1] https://rfd.shared.oxide.computer/rfd/0110
- implr 2y ago
  They already have another RFD for this: https://rfd.shared.oxide.computer/rfd/0508 https://rfd.shared.oxide.computer/rfd/0508 and on HN: https://news.ycombinator.com/item?id=41268043 https://news.ycombinator.com/item?id=41268043
  - Rendello 2y ago
    And a podcast episode: https://oxide.computer/podcasts/oxide-and-friends/2052742 https://oxide.computer/podcasts/oxide-and-friends/2052742
yellowapple 2y ago
I'm surprised that KVM on Illumos wasn't in the running, especially with SmartOS setting that as precedent (even if bhyve is preferred nowadays).
mechanicker 2y ago
Wonder if this is more due to Bhyve being developed on FreeBSD and Illumos derives from a common ancestor BSD? I know NetApp (stack based on FreeBSD) contributed significantly to Bhyve when they were exploring options to virtualize Data ONTAP (C mode) https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervisor.77410/ https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervi...
- jclulow 2y ago
  While we have a common ancestor in the original UNIX, so much of illumos is really more from our SVR4 heritage -- but then also so much of that has been substantially reworked since then anyway.
computersuck 2y ago
Because CTO Bryan Cantrill, who was a core contributor to illumos
- deleted 2y ago
  [deleted]
_zagj 2y ago
Ctrl+f Cantrill >Phrase not found Bryan Cantrill, ex-Sun dev, ex-Joyent CTO, now CTO of Oxide, is the reason they chose Illumos. Oxide is primarily an attempt to give Solaris (albeit Rustified) a second life, similar to Joyent before. The company even cites Sun co-founder Scott McNealy for its principles: https://oxide.computer/principles https://oxide.computer/principles >"Kick butt, have fun, don't cheat, love our customers and change computing forever." >If this sounds familiar, it's because it's essentially Scott McNealy's coda for Sun Microsystems.
JonChesterfield 2y ago
Illumos and ZFS sounds completely sensible for a company that runs on specific hardware. They mention the specific epyc cpu their systems are running on which suggests they're all ~ identical. Linux has a massive advantage where it comes to hardware support for all kinds of esoteric devices. If you don't need that, and you've got engineers that are capable of patching the OS to support your hardware, yep, have at it. Good call.
rbatiati 2y ago
Letting licensing alone, I think they have a couple of killer reasons to choose Illumos: It's a fine operating system, and it's much easier for them to land the fixes/features they need on the Illumos kernel than if they've built on Linux.

226 points by kblissett 2y ago

I like the RFDs. Oxide just did a podcast episode on the process: https://oxide.computer/podcasts/oxide-and-friends/2065190 https://oxide.computer/podcasts/oxide-and-friends/2065190

rtpg 2y ago

> There is not a significant difference in functionality between the illumos and FreeBSD implementations, since pulling patches downstream has not been a significant burden. Conversely, the more advanced OS primitives in illumos have resulted in certain bugs being fixed only there, having been difficult to upstream to FreeBSD. curious about what bugs are being thought of there. Sounds like a very interesting situation to be in

taspeotis 2y ago

I kagi’d Illumos and apparently Bryan Cantrill was a maintainer. Bryan Cantrill is CTO of Oxide [1]. I assume that has no bearing on the choice, otherwise it would be mentioned in the discussion. [1] https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-computer-company/ https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

gyre007 2y ago
Yeah I came here to say that Bryan worked at Sun so why do they even need to write this post (yes, I appreciate the techinical reasons, just wanted to highlight the fact via a subtle dig :-))
- sausagefeet 2y ago
  This isn't a blog post from an Oxide, it's a link to their internal RFD which they use to make decisions.
  - gyre007 2y ago
    I never said it was a post by Oxide.
    - sausagefeet 2y ago
      Might want to re-read your initial comment, then.
sausagefeet 2y ago
Early Oxide founders came from Joyent which was an illumos shop and Cantrill is quite vocal about the history of Solaris, OpenSolaris, and illumos.
- codetrotter 2y ago
  > Joyent which was an illumos shop And before that, they used to run FreeBSD. Mentioned for example in this comment by Bryan Cantrill a decade ago: https://news.ycombinator.com/item?id=6254092 https://news.ycombinator.com/item?id=6254092 > […] Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that […] And I’ve seen some other primary sources (people who worked at Joyent) write that online too. And Bryan Cantrill, and several other people, came from Sun Microsystems to Joyent. Though I’ve never seen it mentioned which order that happened in; was it people from Sun that joined Joyent and then Joyent switched from FreeBSD to Illumos and creating SmartOS? Or had Joyent already switched to Illumos before the people that came from Sun joined? I would actually really enjoy a long documentary or talk from some people that worked at Joyent about the history of the company, how they were using FreeBSD and when they switched to Illumos and so on.
  - panick21_ 2y ago
    Joyent was using Solaris before Bryan worked there. Listen to the this podcast with Bryan and his co-founder about their origin story: https://www.youtube.com/watch?v=eVkIKm9pkPY https://www.youtube.com/watch?v=eVkIKm9pkPY This is about as good as you are gone get on the topic of Joyant history.
    - codetrotter 2y ago
      Thank you, I will watch that right away :)
  - selykg 2y ago
    Joyent also merged with TextDrive, which is where the FreeBSD part came from. TextDrive was an early Rails host, and could even do it in a shared hosting environment, which is where I think a lot of the original user base came from (also TextPattern) As I recall they were also the original host of Twitter, which if I recall was Rails back in the day.
mzi 2y ago
@bcantrill is the CTO of Oxide.
- taspeotis 2y ago
  Yup, thanks
panick21_ 2y ago
Bryan Cantrill also ported KVM to Illumos. At Joyent they had plenty of experience with KVM. See: https://www.youtube.com/watch?v=cwAfJywzk8o https://www.youtube.com/watch?v=cwAfJywzk8o As far as I know, Bryan didn't personally work on the porting of bhyve (this might be wrong). So if anything, that would point to KVM as the 'familiar' thing given how many former Joyant people were there.
- bonzini 2y ago
  KVM got more and more integrated with the rest of Linux as more virtualization features became general system features (e.g. posted interrupts). Also Google and Amazon are working more upstream and the pace of development increased a lot. Keeping a KVM port up to date is a huge effort compared to bhyve, and they probably had learnt that in the years between the porting of KVM and the founding of Oxide.
- elijahwright 2y ago
  Where is Max Bruning these days?
kafHas 2y ago
[flagged]

bonzini 2y ago

> QEMU is often the subject of bugs affecting its reliability and security. {{citation needed}}? When I ran the numbers in 2019, there hadn't been guest exploitable vulnerabilities that affected devices normally used for IaaS for 3 years. Pretty much every cloud outside the big three (AWS, GCE, Azure) runs on QEMU. Here's a talk I gave about it that includes that analysis: slides - https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf https://kvm-forum.qemu.org/2019/kvmforum19-bloat.pdf video - https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y https://youtu.be/5TY7m1AneRY?si=Sj0DFpRav7PAzQ0Y

_rs 2y ago
I thought AWS uses KVM, which is the same VM that QEMU would use? Or am I mistaken?
- bonzini 2y ago
  AWS uses KVM in the kernel but they have a different, non-open source userspace stack for EC2; plus Firecracker which is open source but is only used for Lambda, and runs on EC2 bare metal instances. Google also uses KVM with a variety of userspace stacks: a proprietary one (tied to a lot of internal Google infrastructure but overall a lot more similar to QEMU than Amazon's) for GCE, gVisor for AppEngine or whatever it is called these days, crosvm for ChromeOS, and QEMU for Android Emulator.
  - tptacek 2y ago
    Lambda and Fargate.
    - dastbe 2y ago
      unless something has changed in the past year, fargate still runs each task in a single use ec2 vm with no further isolation around containers in a task.
      - deleted 2y ago
        [deleted]

sausagefeet 2y ago

While it's fair to say this does describe why Illumos was chosen, the actual RFD title is not presented and it is about Host OS + Virtualization software choice. Even if you think it's a foregone conclusion given the history of bcantrill and other founders of Oxide, there absolutely is value in putting decision to paper and trying to provide a rational because then it can be challenged. The company I co-founded does an RFD process as well and even if there is 99% chance that we're going to use the thing we've always used, if you're a serious person, the act of expressing it is useful and sometimes you even change your own mind thanks to the process.

transpute 2y ago

> Xen: Large and complicated (by dom0) codebase, discarded for KVM by AMZN 1. Xen Type-1 hypervisor is smaller than KVM/QEMU. 2. Xen "dom0" = Linux/FreeBSD/OpenSolaris. KVM/bhyve also need host OS. 3. AMZN KVM-subset: x86 cpu/mem virt, blk/net via Arm Nitro hardware. 4. bhyve is Type-2. 5. Xen has Type-2 (uXen). 6. Xen dom0/host can be disaggregated (Hyperlaunch), unlike KVM. 7. pKVM (Arm/Android) is smaller than KVM/Xen. > The Service Management Facility (SMF) is responsible for the supervision of services under illumos.. a [Linux] robust infrastructure product would likely end up using few if any of the components provided by the systemd project, despite there now being something like a hundred of them. Instead, more traditional components would need to be revived, or thoroughly bespoke software would need to be developed, in order to avoid the technological and political issues with this increasingly dominant force in the Linux ecosystem. Is this an argument for Illumos over Linux, or for translating SMF to Linux?

bonzini 2y ago
Talking about "technological and political issues" without mentioning any, or without mentioning which components would need to be revived, sounds a lot like FUD unfortunately. Mixing and matching traditional and systemd components is super common, for example Fedora and RHEL use chrony instead of timesyncd, and NetworkManager instead of networkd.
- actionfromafar 2y ago
  I read it as "we can sit in this more quiet room where people don't rave about systemd all day long".
  - bonzini 2y ago
    But do they? Oxide targets the enterprise, and people there don't care that much about how the underlying OS works. It's been ten years since a RHEL release started using systemd and there has been no exodus to either Windows or Illumos. I don't mean FUD in a disparaging sense, more like literal fear of the unknown causing people to be excessively cautious. I wouldn't have any problem with Oxide saying "we went for what we know best", there's no need to fake that so much more research went into a decision.

saagarjha 2y ago

Unrelated, but is this a homegrown publishing platform?

crb 2y ago
Yes; it's referred to in Oxide's RFD about RFDs [1] https://rfd.shared.oxide.computer/rfd/0001 https://rfd.shared.oxide.computer/rfd/0001 but the referenced URL is 404 unless you're an Oxide employee. [1] https://rfd.shared.oxide.computer/rfd/0001#_shared_rfd_rendered_site https://rfd.shared.oxide.computer/rfd/0001#_shared_rfd_rende... [2] https://github.com/oxidecomputer/rfd/blob/master/src https://github.com/oxidecomputer/rfd/blob/master/src
- dcre 2y ago
  That link is out of date. The site and backend are now open source. Only the repo containing the RFD contents is private. https://github.com/oxidecomputer/rfd-site https://github.com/oxidecomputer/rfd-site https://github.com/oxidecomputer/rfd-api https://github.com/oxidecomputer/rfd-api
benjaminleonard 2y ago
Yep, you can see a little more on the [blog](https://oxide.computer/blog/a-tool-for-discussion https://oxide.computer/blog/a-tool-for-discussion) or the most recent [podcast](https://oxide.computer/podcasts/oxide-and-friends/2065190 https://oxide.computer/podcasts/oxide-and-friends/2065190). The API and site repos are also public.

ReleaseCandidat 2y ago

Instead of stating more or less irrelevant reasons, I'd prefer to read something like "I am (have been?) one of the core maintainers and know Illumos and Bhyve, so even if there would be 'objectively' better choices, our familiarity with the OS and hypervisor trump that". A "I like $A, always use $A and have experience using $A" is almost always a better argument than "$A is better than $B because $BLA", because that doesn't tell me anything about the depth of knowledge of using $A and $B or the knowledge of the subject of decision - there is a reason half of Google's results is some kind of "comparison" spam.

actionfromafar 2y ago
But everyone at Oxide already knows that back story. At least if you list some other reasons list you can have a discussion about technical merits if you want to.
- ReleaseCandidat 2y ago
  But that doesn't make sense if you have specialists for $A that also like to work with $A. Why should I as a customer trust Illumos/Bhye developers that are using Linux/KVM instead of "real" Linux/KVM developers? The only thing that such a decision would tell me is to not even think about using Illumos or Bhyve. The difference between "Buy our Illumos/Bhye solution! Why? I have been an Illumos/Bhyve Maintainer!" and "Buy our Linux/KVM solution! Why? I have been an Illumos/Bhyve Maintainer!" should make my point a bit clearer
  - panick21_ 2y ago
    Those are not the only options. You can have KVM on Illumos, or Bhye on FreeBSD. And finding people to heir that know Linux/KVM wouldn't be a problem for them. This evaluation was done years ago and they added like 50 people since then. Saying 'We have a great KVM Team but our CEO was once an Illumos developer' is perfectly reasonable. And as I point out in my other comment, the former Joyant people like know more about KVM then anything else anyway. So it would be: "Buy our KVM Solution, we have KVM experts" But they evaluated that Bhyve was better then KVM despite that.

blinkingled 2y ago

I guess that's one way of keeping Solaris alive :)

daneel_w 2y ago

"• Emerging VMMs (OpenBSD’s vmm, etc): Haven’t been proven in production" It's a small operation, but https://openbsd.amsterdam/ https://openbsd.amsterdam/ have absolutely proven that OpenBSD's hypervisor is production-capable in terms of stability - but there are indeed other problems that rule against it on scale. For those who are unfamiliar with OpenBSD: the primary caveat is that its hypervisor can so far only provide guests with a single CPU core.

jclulow 2y ago
Yes, to be clear this is not meant to be a criticism of software quality at OpenBSD! Though I don't necessarily always agree with the leadership style I have big respect for their engineering efforts and obviously as another relatively niche UNIX I feel a certain kinship! That part of the document was also written some years ago, much closer to 2018 when that service got started than now, so it's conceivable that we wouldn't have said the same thing today. I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product!
- notaplumber1 2y ago
  > I will say, though, that single VCPU guests would not have met our immediate needs in the Oxide product! Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development? From a secure design perspective, OpenBSD's vmd is a lot more appealing than bhyve is today. I saw recently that AMD SEV (Secure Encrypted Virtualization) was added, which seems compelling for Oxide's AMD based platform. Has Oxide added support for that to their bhyve fork yet?
  - pmooney 2y ago
    > Could Oxide not have helped push multi-vcpu guests out the door by sponsoring one of the main developers working on it, or contributing to development? Being that vmd's values are aligned with OpenBSD's (security above all else), it is probably not a good fit for what Oxide is trying to achieve. Last I looked at vmd (circa 2019), it was doing essentially all device emulation in userspace. While it makes total sense to keep as much logic as possible out of ring-0 (again, emphasis on security), doing so comes with some substantial performance costs. Heavily used devices, such as the APIC, will incur pretty significant overhead if the emulation requires round trips out to userspace on top of the cost of VM exits. > I saw recently that AMD SEV (Secure Encrypted Virtualization) was added, which seems compelling for Oxide's AMD based platform. Has Oxide added support for that to their bhyve fork yet? SEV complicates things like the ability to live-migrate guests between systems.

tonyg 2y ago

> Nested virtualisation [...] challenging to emulate the underlying interfaces with flawless fidelity [...] dreadful performance It is so sad that we've ended up with designs where this is the case. There is no intrinsic reason why nested virtualization should be hard to implement or should perform poorly. Path dependence strikes again.

bonzini 2y ago
It doesn't perform poorly in fact. It can be tuned at 90% of non-nested virtualization, and for workloads where it doesn't, that's more than anything else a testimony to how close virtualized performance is to bare-metal. That said, it does add a lot of complexity.

fefe23 2y ago

These sound like reason you retconned so it sounds like you didn't choose Illumos because your founder used to work at Sun and Joyent before. :-) Frankly I don't understand why they blogged that at all. It reeks of desperation, like they feel they need to defend their choice. They don't. It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details.

jclulow 2y ago
It's not a blog post, it's an RFD. We have a strong focus on writing as part of thinking and making decisions, and when we can, we like to publish our decision making documents in the spirit of open source. This is not a defence of our position so much as a record of the process through which we arrived at it. This is true of our other RFDs as well, which you can see on the site there. > It also should not matter to their customers. They get exposed APIs and don't have to care about the implementation details. Yes, the whole product is definitely designed that way intentionally. Customers get abstracted control of compute and storage resources through cloud style APIs. From their perspective it's a cloud appliance. It's only from our perspective as the people building it that it's a UNIX system.
- stonogo 2y ago
  So at no point did anyone even suspect that Illumos was under consideration because it's been corporate leadership's pet project for decades? That seems like a wild thing to omit from the "RFD" process. Or were some topics not open to the "RFD" process?
  - jclulow 2y ago
    We are trying to build a business here. The goal is to sell racks and racks of computers to people, not build a menagerie of curiosities and fund personal projects. Everything we've written here is real, at least from our point of view. If we didn't think it would work, why would we throw our own business, and equity, and so on, away? The reason I continue to invest myself, if nothing else, in illumos, is because I genuinely believe it represents a better aggregate trade off for production work than the available alternatives. This document is an attempt to distill why that is, not an attempt to cover up a personal preference. I do have a personal preference, and I'm not shy about it -- but that preference is based on tangible experiences over twenty years!

leoh 2y ago

I’d love to use Illumos, but a lack of arm64 support is a non-starter

jclulow 2y ago
Folks are working on it! I believe it boots on some small systems and under QEMU, but it's still relatively early days. I'm excited for the port to eventually make it into the gate, though!
- geerlingguy 2y ago
  In before someone asks about riscv64 ;)
ComputerGuru 2y ago
I don’t mean to downplay the importance for you personally but I do want to clarify that while it might be a non-starter for you, all of arm64 is so new that it’s hardly a non-starter for anyone considering putting it into (traditional) production.
- timenova 2y ago
  You're right, however I was looking for the same information to maybe try it on a RPi to learn more about Illumos.

BirAdam 2y ago

I think a bigger reason for Oxide using Illumos is that many of the people over there are former Sun folks.

throw0101b 2y ago

Somewhat related, they discussed why they chose to use ZFS for their storage backend as opposed to (say) Ceph in a podcast episode: * https://www.youtube.com/watch?v=UvEKSqBBcZw https://www.youtube.com/watch?v=UvEKSqBBcZw Certainly they already had experience with ZFS (as it is built into Illumos/Solaris), but as it was told to them by someone they trusted who ran a lot of Ceph: "Ceph is operated, not shipped [like ZFS]". There's more care-and-feeding required for it, and they probably don't want that as they want to treat product in a more appliance/toaster-like fashion.

pclmulqdq 2y ago
Ceph is sadly not very good at what it does. The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). ZFS solves a different problem, though. ZFS is a full-featured filesystem. Like Ceph it is also vulnerable to single points of failure.
- throw0101b 2y ago
  > The big clouds have internal versions of object store that are far better (no single point of failure, much better error recovery story, etc.). There are different levels of scalability needs. CERN has over a dozen (Ceph) clusters with over 100PB of total data as of 2023: * https://www.youtube.com/watch?v=bl6H888k51w https://www.youtube.com/watch?v=bl6H888k51w Certainly there are some number of folks that need more than that, but I don't there are many. > Like Ceph it is also vulnerable to single points of failure. The SPOF for ZFS is the host (unless you replicate, e.g., zfs send). What is SPOF of Ceph? You can have multiple monitors, managers, and MDSes.
  - pclmulqdq 2y ago
    Single-monitor is a common way to run Ceph. On top of that, many cluster configurations cause the whole thing to slow to a crawl when a very small minority of nodes go down. Never mind packet loss, bad switches, and other sorts of weird failure mechanisms. Ceph in general is pretty bad at operating in degraded modes. ZFS and systems like Tectonic (FB) and Colossus (Google) do much better when things aren't going perfectly. Do you know how many administrators CERN has for its Ceph clusters? Google operates Colossus at ~1000x that size with a team of 20-30 SREs (almost all of whom aren't spending their time doing operations).

jonstewart 2y ago

Illumos makes sense as a host OS—it’s capable, they know it, they can make sure it works well on their hardware, and virtualization means users don’t need that much familiarity with it. If I were Oxide, though, I’d be sprinting to seamless VMWare support. Broadcom has turned into a modern-day Oracle (but dumber??) and many customers will migrate in the next two years. Even if those legacy VMs aren’t “hyperscale”, there’s going to be lots of budget devoted to moving off VMWare.

parasubvert 2y ago
Oracle is a $53 billion company, and never had a mass exodus, just less greenfield deployments. Broadcom also isn't all that dumb, VMware was fat and lazy and customers were coddled for a very long time. They've made a bet that it's sticky. The competition isn't as weak as they thought, that's true, but it will take 5+ years to catch up, not 2 years, in general. Broadcom was betting on it taking 10 years: plenty of time to squeeze out margins. Customers have been trying and failing to eliminate the vTax since OpenStack. Red Hat and Microsoft are the main viable alternatives.
- jonstewart 2y ago
  I don’t disagree much. Still, there’s a sudden weakening of the main incumbent in the on-prem virtualization market… and that is _all_ Oxide does. It will be interesting to see whether Oxide can convert some VMWare customers.
  - meepmorp 2y ago
    Maybe some, but the main difference between Oxide and the others is that they sell an integrated platform, so if a customer is just looking to replace the VM software, that seems like a harder sell.
  - parasubvert 2y ago
    Oxide is a moonshot, it requires companies to fire a lot more suppliers than VMware. Which is easier said than done. A lot will depend on if Oxide can find their early niches: they're a long play (years) before they crack the mainstream or are truly competitive with, say, VCF on VxRail, for the average talent ops team. Long run it's clearly going to be better, but short run they have to pick their battles.

kayo_20211030 2y ago

Is the date on this piece correct? The section about Rust as a first class citizen seems to contain references to its potential use in Linux that are a few years out of date; with nothing more current than 2021. > As of March 2021, work on a prototype for writing Linux drivers in Rust is happening in the linux-next tree.

kayo_20211030 2y ago
nm, I read the postscript. The RFD was from 2021. I wonder how correct it was, and whether decisions made, based on it, were good ones or bad ones.

alberth 2y ago

Isn’t it simply Oxide founders are old Sun engineers, and Illumos is the open source spinoff of their old work.

sophacles 2y ago
According to the founders and early engineers on their podcast - no, they tried to fairly evaluate all the oses and were willing to go with other options. Practically speaking, its hard to do it completely objectively and the in-house expertise probably colored the decision.
- pclmulqdq 2y ago
  Tried to, sure, but when you evaluate other products strictly against the criteria under which you built your own version, you know what the conclusion will be. Never mind that you are carrying your blind spots with you. I would say that there was an attempt to evaluate other products, but not so much an attempt to be objective in that evaluation. In general, being on your own private tech island is a tough thing to do, but many engineers would rather do that than swallow their pride.

Aissen 2y ago

Point 1.1 about QEMU seems even less relevant today, with QEMU adding support for the microvm machines, hence greatly reducing the amount of exposed code. And as bonzini said in the thread, the recent vulnerability track record is not so bad.

magicalhippo 2y ago

Been running Bhyve on FreeBSD (technically FreeNAS). Found PCIe pass-through of NMVe drives was fairly straight forward once the correct incantations were found, but network speed to host has been fairly abysmal. On my admittedly aging Threadripper 1920X, I can only get ~2-3 Gbps peak from a Linux guest. That's with virtio, the virtual intel "card" is even slower. They went with Illumos though, so curious if the poor performance is a FreeBSD-specific thing.

bitfilped 2y ago
It's been a minute since I messed with bhyve on FreeBSD, but I'm pretty sure you have to switch out the networking stack to something like Netgraph if you intend to use fast networking.
- craftkiller 2y ago
  Hmmm I'm not the OP, but I run my personal site on a kubernetes cluster hosted in bhyve VMs running Debian on a FreeBSD machine using netgraph for the networking. I just tested by launching iperf3 on the FreeBSD host and launching an alpine linux pod in the cluster, and I only got ~4Gbit/s. This is surprising to me since netgraph is supposed to be capable of much faster networking but I guess this is going through multiple additional layers that may have slowed it down (off the top of my head: kubernetes with flannel, iptables in the VM, bhyve, and pf on the FreeBSD host).
  - bitfilped 2y ago
    Do you know if you're still using if_bridge? I remembered this article from klara that goes a bit more into the details. https://klarasystems.com/articles/using-netgraph-for-freebsds-bhyve-networking/ https://klarasystems.com/articles/using-netgraph-for-freebsd...
    - craftkiller 2y ago
      Thanks, but I am not using if_bridge. I am creating a netgraph bridge[0] which is connected to the host via a netgraph eiface. Then on the host, the packet passes to my real physical interface because I have gateway_enable and I have pf perform NAT[1]. It looks like that blog post connected the netgraph bridge directly to the external interface, so my guess is my slowdown is from either pf performing NAT or the packet forwarding from gateway_enable. [0] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/bhyve/files/bhyve_netgraph_bridge.bash#L214 [1] https://code.fizz.buzz/talexander/machine_setup/src/commit/20768edcf69eddae5cf65e30a0bde869f9ddd19b/ansible/roles/firewall/files/mrmanager_pf.conf#L35

tcdent 2y ago

Linux has a rich ecosystem, but the toolkit is haphazard and a little shakey. Sure, everyone uses it, because when we last evaluated our options (in like 2009) it was still the most robust solution. That may no longer be the case. Given all of that, and taking into account building a product on top of it, and thus needing to support it and stand behind it, Linux wasn't the best choice. Looking ahead (in terms of decades) and not just shipping a product now, it was found that an alternate ecosystem existed to support that. Culture of the community, design principles, maintainability are all things to consider beyond just "is it popular". Exciting times in computing once again!

craftkiller 2y ago

I wonder if CockroachDB abandoning the open source license[0] will have an impact on their choice to use it. It looks like the RFD was posted 1 day before the license switch[1], and the RFD has a section on licenses stating they intended to stick to the OSS build: > To mitigate all this, we’re intending to stick with the OSS build, which includes no CCL code. [0] https://news.ycombinator.com/item?id=41256222 [1] https://rfd.shared.oxide.computer/rfd/0110

implr 2y ago
They already have another RFD for this: https://rfd.shared.oxide.computer/rfd/0508 https://rfd.shared.oxide.computer/rfd/0508 and on HN: https://news.ycombinator.com/item?id=41268043 https://news.ycombinator.com/item?id=41268043
- Rendello 2y ago
  And a podcast episode: https://oxide.computer/podcasts/oxide-and-friends/2052742 https://oxide.computer/podcasts/oxide-and-friends/2052742

yellowapple 2y ago

I'm surprised that KVM on Illumos wasn't in the running, especially with SmartOS setting that as precedent (even if bhyve is preferred nowadays).

mechanicker 2y ago

Wonder if this is more due to Bhyve being developed on FreeBSD and Illumos derives from a common ancestor BSD? I know NetApp (stack based on FreeBSD) contributed significantly to Bhyve when they were exploring options to virtualize Data ONTAP (C mode) https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervisor.77410/ https://forums.freebsd.org/threads/bhyve-the-freebsd-hypervi...

jclulow 2y ago
While we have a common ancestor in the original UNIX, so much of illumos is really more from our SVR4 heritage -- but then also so much of that has been substantially reworked since then anyway.

computersuck 2y ago

Because CTO Bryan Cantrill, who was a core contributor to illumos

deleted 2y ago
[deleted]

_zagj 2y ago

Ctrl+f Cantrill >Phrase not found Bryan Cantrill, ex-Sun dev, ex-Joyent CTO, now CTO of Oxide, is the reason they chose Illumos. Oxide is primarily an attempt to give Solaris (albeit Rustified) a second life, similar to Joyent before. The company even cites Sun co-founder Scott McNealy for its principles: https://oxide.computer/principles https://oxide.computer/principles >"Kick butt, have fun, don't cheat, love our customers and change computing forever." >If this sounds familiar, it's because it's essentially Scott McNealy's coda for Sun Microsystems.

JonChesterfield 2y ago

Illumos and ZFS sounds completely sensible for a company that runs on specific hardware. They mention the specific epyc cpu their systems are running on which suggests they're all ~ identical. Linux has a massive advantage where it comes to hardware support for all kinds of esoteric devices. If you don't need that, and you've got engineers that are capable of patching the OS to support your hardware, yep, have at it. Good call.

rbatiati 2y ago

Letting licensing alone, I think they have a couple of killer reasons to choose Illumos: It's a fine operating system, and it's much easier for them to land the fixes/features they need on the Illumos kernel than if they've built on Linux.