10 ms·
Oxide builds servers as they should be [audio]
- fio_ini 4y agoAwesome product. How much $$$?
- nimbius 4y agoaccording to the website for oxide theyre "Hardware, with the software baked in" so I'm worried this is going to be more like Cisco's hamfisted UCS platform or dell/HP lock in with drac and ilo and "hyperconverged" embedded Java fuckery that can never be patched.
- sokoloff 4y agoI’ve been relatively satisfied with HP’s iLO offering, in that it’s been sufficient for my needs, avoids many pointless fly/drives to the colo (or dealing with the comic misnomer that is “smart hands”), and is still getting the occasional update on 8-year old servers. Even for servers in my building, I find it’s more convenient than a crash cart for most things (and of course my desk is a better work environment).
- mlindner 4y agoAll of the source will be open and re-flashable from what I've read. So I assume that won't be a problem.
- steveklabnik 4y agoI wrote a while back about the difference between "hyperconverged" and "hyperscalar": https://news.ycombinator.com/item?id=30688865 https://news.ycombinator.com/item?id=30688865
- brodo 4y agoHere is Bryans application video for YC120 2019: https://youtu.be/px9OjW7GB0Q https://youtu.be/px9OjW7GB0Q
- nynx 4y agoDefinitely worth watching
- mlindner 4y agoWhy do you say that? Comes off as rather high-and-mighty and out of touch.
- AceJohnny2 4y ago> Comes off as rather high-and-mighty Cantrill is, shall we say, very opinionated. And does not suffer fools gladly. Edit: but also, it's a YC (Anti)Application video, so being high-and-mighty is practically a prerequisite.
- alberth 4y agoWho’s the target buyer of Oxide? I ask genuinely … because I don’t understand who will spend the premium for a rack server that is easier to maintain? In a world of cloud and dedicated hosted servers (where the users of the server is not the buyer of the server) - who would buy Oxide? It seems like whoever can build the cheapest server wins now. Said differently, cloud has completely changed the value-chain in server buyers
- qaq 4y agoLarge enterprises running things on prem. Governments etc.
- walrus01 4y agoultimately somebody has to buy and own and operate the underlying hardware of the "cloud". obviously AWS, azure, etc do their own hardware procurement and totally bespoke stuff in house. so this would be targeted at any smaller players in hosting companies that are not at that massive size of azure, but need vast quantity of servers.
- alberth 4y ago> so this would be targeted at any smaller players in hosting companies that are not at that massive size of azure, but need vast quantity of servers. But my understanding Oxide is/will-be priced as a premium over a typical x86 server. It seems unlikely that any hosting company is going to pay a premium just for a bit more convenience. Note: I'm not trying to be a "hater". Just genuinely not understanding the market here.
- walrus01 4y agoyou are correct, margins are very thin in hosting company business and the absolute most server performance/number of nodes for the least money is very important. have been on the manufacturing side of this before. bulk x86-64 servers such as you could buy from the vendors you meet at the computex taipei are a big part of it. things along the same general idea as the supermicro that is 2U high and has four discrete nodes in it, but cheaper. long/narrow motherboards with dual socket xeons and other similar stuff... if the price for this is not market competitive with that, or just a bunch of 1U commodity dual socket servers, the only people buying it will be enterprise/specialty end user and not hosting operations.
- walrus01 4y agowhat does this accomplish that buying some open compute platform based x86-64 servers in whole-rack quantity does not? Or a bunch of the the 4-servers-in-a-single-2U-package setup from Supermicro? And then interconnecting them by your own choice of 100GbE switches at top of rack.
- panick21_ 4y agoThis company was created because somebody that actually built a large could with Dell/Supermicro suffered years and years of pain. If you do as you suggest you will have to do a whole lot more, at the minimum you need to set-up (and pay) VMWare. Then you will need to figure out how you do automated firmware upgrades, low level monitoring and if security is a concern you likely want to configure secure booting with attestation and things like that. Consider how much work it is to do what you suggest, vs buying this rack. And then consider how good (and repeatable) the end result is. That is at least the theory, see: https://www.youtube.com/watch?v=vvZA9n3e5pc https://www.youtube.com/watch?v=vvZA9n3e5pc
- stonogo 4y agoAsk questions about business model on an Oxide thread, and speculation abounds. Ask questions about podcasts, and all the founders show up with links.
- steveklabnik 4y agoThe business model is straightforward: we sell servers, a rack at a time. You put them in your data center and enjoy a cloud deployment model but a “I own the metal” ownership model. I write something longer in a previous thread: https://news.ycombinator.com/item?id=30678324 https://news.ycombinator.com/item?id=30678324
- bcantrill 4y agoThe business model is elaborated on it extensively in the linked conversation, for whatever it's worth.
- rejectfinite 4y agoWill Windows Server/HyperV and VMWare run on these?
- bcantrill 4y agoWindows Server yes -- as a hardware virtualized guest. HyperV and VMware, no: we provide the hypervisor and control plane (and that that's built into the cost of the rack is very much part of the value proposition!).
- deleted 4y ago[deleted]
- geertj 4y agoI wish the Oxide web site had a little more detail on the software running on this. Does This use a custom hypervisor? If so what is it based on, and what are its capabilities?
- steveklabnik 4y agoWe’re redoing the website, though I’m not sure if this specific information will be on it. The host is using Illumos and bhyve for hypervisor stuff. I don’t work on that part of the stack personally so that’s the most detail I can give you.
- bcantrill 4y agoJust to add some additional detail: the control plane (Omicron[0] -- named before the COVID variant!), the user-level portion of the hypervisor (Propolis[1]) and the storage subsystem (Crucible[2]) are all open source. And while our focus has been building them for our own product, we know that they are at least usable outside of Oxide![3][4] [0] https://github.com/oxidecomputer/omicron https://github.com/oxidecomputer/omicron [1] https://github.com/oxidecomputer/propolis https://github.com/oxidecomputer/propolis [2] https://github.com/oxidecomputer/crucible https://github.com/oxidecomputer/crucible [3] https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.html https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.htm... [4] https://artemis.sh/2022/06/14/oxide-crucible.html https://artemis.sh/2022/06/14/oxide-crucible.html
- maxwell86 4y ago> The elevation of the room where the rack is installed must be below 9,842 feet (3,000 meters). Why?
- lazylizard 4y agowith GPUs?
- brianzelip 4y agoSlightly off topic, but hey Jerod if you’re reading, here’s a vote to keep up the weekly brief Changelog episodes!
- simonw 4y agohttps://oxide.computer/product https://oxide.computer/product CPU: 2,048 x86 cores (AMD Milan) Memory: Up to 32TB DRAM Storage: 1024TB of raw storage in NVMe Network switch: Intel Tofino 2 Network speed: 100 Gbps That's a pretty meaty server!
- littlestymaar 4y agoNoob here: any idea of the approximate price tag that should be expected for such a monster?
- mjgerm 4y agoLooking at the hardware, I'd wag somewhere in the $2-3M range. Depends on how big of a profit margin they want to make or how scrappy they feel like being (could probably go as low as ~$1M if they can make it up in volume). If they're following the typical 1/5x pricing model for support, that'd be roughly $500k/yr/rack. But it's also hard to do that while simultaneously describing Dell as "rapacious".
- littlestymaar 4y agoThanks!
- Sirened 4y agoI suspect they are not going to be cheap at all lol, aside from being a very nice hardware package they also have an entire custom hypervisor platform which is intended to be one of the big plusses for the platform.
- hestefisk 4y agoImagine a Beowulf cluster of these …
- deleted 4y ago[deleted]
- boulos 4y ago
- chairmanwow1 4y agoListened to 18 min before stopping, not really sure companies values warranted so much time. Furthermore pretty leading question to compare Brian to Steve Jobs forced me out.
- trhway 4y agoBack from Sun times I remember Brian loving the limelight. Somebody comparing you to Jobs is the pinnacle of a limelight search in our industry.
- deleted 4y ago[deleted]
- 1123581321 4y agoHe demurred to the comparison and made a solid book recommendation. Seems better to just skip through uninteresting segments with the +45s button.
- deleted 4y ago[deleted]
- sgt 4y agoA lot of people have been compared to Jobs. At the very least, Cranston has some substance.
- smabie 4y agoSeems like this company has gotten a lot of hype but what so they actually do? Cloud hosting? Or selling physical servers? Both of those things sound like a terrible business tho..
- stingraycharles 4y agoFrom what I gather, it’s extremely beefy, physical servers with DPUs built-in. I.e. perfect for large on-premise deployments while maintaining the high performance that DPUs enable, all as a single appliance. It’s the kind of thing I would expect Nvidia or Dell to become big at as well, and Bryan Cantrill has enough leverage to pique my interest significantly here.
- wmf 4y agoI don't think they're using DPUs.
- jjtheblunt 4y agowhat's a DPU?
- unixhero 4y agoDatacenter processing unit, it's a rather new thing
- gertrunde 4y agoAlso known as: SmartNIC, and a bunch of other names, basically a NIC with an embedded CPU (usually ARM of some sort), so can offload some workload into the NIC. An example is nvida's bluefield, there are others That sort of thing is being used in a few areas, can even run ESXi on them. There was a fun article on servethehome a while back using them to build an arm cluster (Link: https://www.servethehome.com/building-the-ultimate-x86-and-arm-cluster-in-a-box/ https://www.servethehome.com/building-the-ultimate-x86-and-a... )
- qbasic_forever 4y ago
- labrador 4y agoCranston made a pretty good case for Oxide in a talk he gave - his argument as I understood it is that legacy BIOS is proprietary and insecure
- gruturo 4y ago> Cranston made a pretty good case for Oxide in a talk he gave Hmmm wrong Bryan? Breaking Bad wouldn't look the same if you swapped them, I'm afraid.
- labrador 4y agoThere was another commenter who made the same mistake. I was going to correct them but thought better of it, then made the same mistake myself!
- gfodor 4y agoBreaking Bad where Bryan Cantrill runs an illegal Bitcoin mining operation on Oxide servers under a NYC laundromat
- hinkley 4y agoTheir driers are really slow because they use the exhaust heat from the server room.
- labrador 4y agoHis nemesis is El Salvadoran volcano crypto cartels, but Cantrill mounts a 51% attack and destroys their Magmacoin
- shrubble 4y ago"I am the one who port-knocks!"
- zja 4y ago“Jesse, we have to compile”
- uthinter 4y agoIf anyone from Oxide is here, could you guys restart the "On the metal" podcast . It was the only CS related podcast I ever liked.
- thundergolfer 4y agoSignals and Threads is equally as good as a CS podcast.
- JoachimSchipper 4y ago"On the metal" was indeed lots of fun; I understand that it's been "succeeded" by Oxide's Twitter spaces, https://github.com/oxidecomputer/twitter-spaces https://github.com/oxidecomputer/twitter-spaces
- mlindner 4y agoTwitter spaces are awful however and are also not easily recorded and can't be used easily on any device. (You need to go to some kind of youtube link from one of the participants to listen to a recording.) Also their twitter talks are more unfocused rants by the participants with little focus. So it's hard to listen to. I'm sure it's more fun for the participants, but not so much for the listeners. You also lose the comfort of consistency of participants. Some of them have even been delving into some of the participants personal politics as well which is just something I'm not interested in hearing.
- sam_bristow 4y agoThe Oxide Twitter spaces are recorded and distributed as a podcast [1]. If you're not a fan of the content it won't help but it's at least more convenient to consume than through Twitter. [1] https://feeds.transistor.fm/oxide-and-friends https://feeds.transistor.fm/oxide-and-friends
- bcantrill 4y agoWe've turned it into a podcast as well, so you should be able to enjoy it wherever you consume podcasts.[0] While there may be some unfocused rants in there (sorry, I guess?), there's also a lot of extraordinary technical content -- certainly, if anyone else has described their board bringup experiences in as explicit technical detail as we have in [1] and [2], I would love to be pointed to it! [0] https://podbay.fm/p/oxide-and-friends https://podbay.fm/p/oxide-and-friends [1] Tales from the Bringup Lab: https://podbay.fm/p/oxide-and-friends/e/1638838800 https://podbay.fm/p/oxide-and-friends/e/1638838800 [2] More Tales from the Bringup Lab: https://podbay.fm/p/oxide-and-friends/e/1650326400 https://podbay.fm/p/oxide-and-friends/e/1650326400
- bombas 4y agoI don’t understand how Oxide is any different than what VCE attempted to accomplish from a converged infra perspective.
- mlindner 4y agoDidn't VCE still use consumer rack computers? All they were was basically a hardware reseller with their own racks. They re-sold Cisco stuff. They just repackaged everyone else's stuff as a value added service.
- rtp4me 4y agoDisclaimer: I worked for VCE (now Dell/EMC) VCE used Cisco C2xx and C4xx rackmount and BLxxx blade servers (before the EMC/Dell acquisition). Now, I believe they use all Dell hardware.
- Tuna-Fish 4y agoThey designed and built their own hardware and wrote all their own firmware.
- Klasiaster 4y agoWhat is missing is a middle ground: an improvement over the IPMI broken-state-of-things that is aimed for automated deployments, without demanding a certain network environment for PXE booting and configuration of IP addresses for management. I haven't used RedFish but it seems it does not address the problems of IPMI in this regard: To be fully automatable there shouldn't be an assumption about the network environment - rather I would like that to have an L2-based protocol (one has to use the MAC address as identifier anyway) or if IPv4/v6 is used then at least demand mandatory link-local auto-addressing by default. However, RedFish seems to support emulating a CD-ROM drive for booting an installation media, this is a nice idea but what I really would like to see is directly writing an image to the hard disk. Then of coure the hardware manufacturers would need to write high-quality firmware and not the regular quirk- and bug-ridden stuff that we know from IPMI implementations.
- bcantrill 4y agoYour last sentence reveals what you probably already know to be true: there isn't a middle ground. We tried for years to get hyperscaler-class software running on commodity-class servers -- and came to the conclusion that the hyperscalers came to (and as Alan Kay famously observed): to be really serious about elastic infrastructure, one needs to do one's own hardware. Now, ~2.5 years into Oxide, I can substantiate that with more detail that I could have ever imagined: there is no middle ground -- and indeed, this is why the infrastructure world has become so acutely bifurcated!
- mwcampbell 4y agoDo you think there's at least a place for a holistic system that's not a whole rack? Or do you think that those of us with smaller computing needs will just have to either rent from a hyperscaler or put up with PC servers?
- bcantrill 4y agoDefinitely, but it's not a market that Oxide intends to go after (at least not for the foreseeable future). Part of the challenge is that it really does require diverging from the reference designs (as these have effectively enshrined a proprietary BMC and everything that comes with it) -- and once you are away from reference designs, you are talking about a very different economic proposition. This is part of why we are so transparent and open about everything we're doing: our hope is that we can enable others to pick up the baton and serve those markets that we do not intend to address.