9 ms·
Who killed the network switch? A Hubris Bug Story
- 1970-01-01 3y ago>random crashes that go away when you add some debugging code are the worst kind of crash, but that was the situation we were in Oh my
- Lord-Jobo 3y agoI can think of very few scenarios that would send me into an 8 hour fury and caffeine fueled debug session as quickly as that. It's one of those problems that throws you into a fit of self doubt and paranoia. You question your basic assumptions, your hardware, your colleagues, and God himself before making any headway and it turns out its often someone else's "fault", who programmed some core feature of a library 25 years ago in an obscure way, because they never could have imagined the Frankenstein-esque behemoth their code would be embedded in nearly 3 decades later.
- nicetryguy 3y agoOk, haven't i been hearing Rust Evangelists SCREAMING at the top of their lungs for the last decade about memory safety, how every other language is unsafe, archaic, unusable, etc; and this bug is (checks notes) ...a memory overwrite?
- christop 3y agoIt seems more like the kernel was actively preventing safe memory access, albeit erroneously, and also has nothing to do with Rust per se?
- moosingin3space 3y agoThis is a fantastic in-depth look at debugging a complex problem, and the fact that the rest of the system remained stable is a testament to the quality of the engineering work that the Oxide team put into this. I'm personally quite inspired by this and plan on applying similar techniques in my day job!
- aus10d 3y agoThe work Oxide is doing is truly amazing
- brcmthrowaway 3y agoFirst tailscale, now Oxide is the darling of projects 99% of people will never need
- p_l 3y agoI can understand Oxide (though current vmware apocalypse might make their appeal bigger than expected), but Tailscale? Tailscale's biggest proposition is that their VPN software is, arguably, the easiest I've ever managed. There systems that are better integrated in some environments, systems that provide various niche features, etc. But Tailscale is simplest to use.
- always2slow 3y agoIt's ok to have heroes.
- maybeben 3y agoI look forward to scoring some beefy Milan boxes with weird rails for cheap on ebay because the firmware only loads OpenSolaris.
- aus10d 3y agoYou don't have to "need" it to admire the scope of what they created. Hardware AND software engineering at its finest solving some very hard problems.
- pstuart 3y agoI'm assuming that 1% will make their efforts worthwhile. It is impressive regardless of whether or not you're going to use it.
- StressedDev 3y agoOK, this comment was not really helpful. The article described some really interesting tech. Trashing tech or people who are excited by it does not really make things better.
- monocasa 3y agoFWIW, you can support more than 8 regions by treating that hardware more like a soft fill TLB.
- packetlost 3y agoI assume TLB is translation lookaside buffer, but what do you mean by "soft fill" here?
- Veserv 3y agoThey mean software-filled. On “page fault (really a illegal memory access)” you walk a software data structure to determine if there should actually be accessible memory there then you “page in (update the MPU to make the memory accessible, possibly evicting another entry)” the memory. This is how some older chips used to work. Hardware page table walkers can be viewed as just a hardware implementation of such code.
- chc4 3y agoI was horrified to learn that PowerPC uses an "inverted page table", which is essentially just a TLB implemented as a hash table. If the requested page isn't resident in the inverted page table it delivers an interrupt, where the kernel is supposed to walk it's own internal internal page table data structure and populate the hash table entry for the missing page. It was a big surprise coming from x86 or ARM on personal computer kernels where they have a fully populated page table as the canonical source of mappings, and all walking in hardware!
- saagarjha 3y agoIt does let you do some cute things in the software implementation though. I’d prefer a hardware walk of course but if I can’t have it might as well enjoy things.
- robocat 3y agoI would guess they (a) want soft realtime performance, and (b) would not want to introduce something critical that could interfere with debuggability or potentially reliability. I'd never do it unless it was the last choice available. Virtual paging is nasty and I wouldn't want the doubt.
- scottlamb 3y agoNice read! Nit: // Order the task's regions in ascending address order. // // THIS IS IMPORTANT. The kernel exploits this property to do cheaper // access tests. regions.sort_by_key(|i| region_table.get_index(*i).unwrap().1.base); I wouldn't put this comment here. It's not just some detail of this function; it's an invariant of the field that all writers have to respect (maybe this is the only one now but still) and all readers can take advantage of. So I'd add it to the `TaskDesc::regions` docstring. [1] [1] https://github.com/oxidecomputer/hubris/commit/b44e677fb39cde8be5b10bbf78a9f26c000f6ad6#diff-96e27941c050f6013357d7ad1716058d7171f2ab012e2bc5ee01a6c19701d826L45 https://github.com/oxidecomputer/hubris/commit/b44e677fb39cd...
- db48x 3y agoIt is nice to have the comment next to the sort though, which is otherwise going to be surprising. Probably the best thing to do is to make a constructor method for the TaskDesc that sorts the regions to enforce the invariants. The code is evidently getting more complex over time, so packing the complexity up into methods is probably now worth spending a little time on.
- sbt567 3y agoI adore whatever folks at Oxide does. And this is one of it
- orf 3y agoHubris is really, really nice. I've spent half an hour reading some of the kernel code and it’s exceptionally clear and well written - a far cry from ifdef macro soup, two letter variable name loving, comment starved C code I’ve seen previously. A good bit of bedtime reading! I recommend leafing through it: https://github.com/oxidecomputer/hubris/blob/b44e677fb39cde8be5b10bbf78a9f26c000f6ad6/sys/kern/src/ https://github.com/oxidecomputer/hubris/blob/b44e677fb39cde8...
- hinkley 3y agoIt bothers me deeply how much of C ethos can be boiled down to, "We can't be bothered to learn to type at a reasonable speed" Disk space for source code hasn't really been a problem for forty years (binaries definitely), and yet we are still being stingy with variable names.
- pif 3y agoCode quality has zero correlation with typing speed.
- postmodest 3y ago> It bothers me deeply how much of C ethos can be boiled down to, "We can't be bothered to learn to type at a reasonable speed" "My computer is a LITERAL TELETYPE, so I will enshrine typographical parsimony in every part of this OS/Language I am creating" -K&R
- djmips 3y agoYeah it wasn't about typing speed. Maybe you were being sarcastic.
- tjoff 3y agoIf it bothers you so much I'm happy to tell that that is not the reason for it.
- throwaway11460 3y agoWho cares though? Just use an IDE or Copilot or something.
- dralley 3y agoWhy/when would one use Hubris over something like embassy_rs, or vice-versa?
- steveklabnik 3y agoHubris has some particular design constraints that allow it to do interesting things, but also may disqualify it for certain projects. The main one is that everything is static, ahead of time: you set up tasks when you build the OS and the tasks, there's no "spawn new task" sort of APIs. Another difference is that Hubris does not use async Rust, and embassy does. That may be something you desire either way. We don't believe async Rust is bad (heck it's used further up the stack quite a bit) but it wasn't a fit for this project. The documentation explains the design goals: https://hubris.oxide.computer/reference/ https://hubris.oxide.computer/reference/ Doesn't mean embassy isn't good too, it's just different.
- StressedDev 3y agoThank you for the thoughtful reply. I really appreciated it.
- steveklabnik 3y agoYou're welcome!
- MrBuddyCasino 3y ago> Hubris does not use async Rust, and embassy does As much as I understand why you'd want async on an MCU for efficiency reasons, I'm very glad there is now a good option without it. I wonder what the Hubris team thinks of rtic and its design.
- steveklabnik 3y agoI can only speak for myself, but it’s sort of like “embassy without the HAL,” so closer to Hubris in some ways, and further away in others. I’m sure that other folks would say the same thing here though: also good, just different.
- mgerdts 3y agoAccording to [1] the prequel is found at [2]. 1. https://hachyderm.io/@mjk/112157472314396711 https://hachyderm.io/@mjk/112157472314396711 2. https://www.mattkeeter.com/blog/2024-03-25-packing/ https://www.mattkeeter.com/blog/2024-03-25-packing/
- quasarj 3y agoSomebody named an OS Hubris? Oh the.. I can't even say it
- hinkley 3y agoNo, Brian Cantrill named an OS hubris. Who in their right mind writes a new operating system today? Answer: someone trying to tackle a problem every operating system is ignoring (controllers on MB and expansion cards that the OS doesn’t and cannot control).
- steveklabnik 3y agoTo be clear, Bryan didn't name it: Cliff did. But yes, the idea is to gently poke fun at "wait you're writing your own OS?", which is admittedly a bit hubristic. Humor is a good way to stay grounded.
- hinkley 3y agoOh really.
- mkeeter 3y agoYou’ll be glad to learn that the debugger is named “humility”: https://github.com/oxidecomputer/humility https://github.com/oxidecomputer/humility
- k0stas 3y agoAnd the author of the post is named Cliff L. Biffle, which is one of the best names I have ever heard.
- cdchn 3y agoSeems pretty on-brand.
- 0xfedbee 3y agoMoral of the story: using Rust will not automatically make your code bug proof. It can actually introduce bugs that are of totally new breed and hard to debug.
- kibwen 3y agoI'm not sure I see where the bug here is due to Rust, or of a totally new breed. And the latter half of the blog post notes that it was fixed less than three hours after first being noticed.
- rablackburn 3y agoAnd to quote the article itself: >> (…) it’s a rare case of a kernel memory access check bug that had no security implications. My favourite type of kernel bug.
- eigenform 3y agoI know you're intentionally spinning this as a bad thing, but it's really not. Instead of worrying about simple and very common classes of bugs that can be solved statically with the help of the compiler, you are free to worry about whatever other non-trivial bugs in your program are remaining. You're obviously free to waste your own time if you'd like.
- nemothekid 3y agoI don't know, the fact that a buggy memory allocator implementation crashed rather than silently worked until it became a CVE seems like a win to me.
- toast0 3y agoThis example showed an introduced bug that was easy to debug, though.
- StressedDev 3y agoNope - This bug was not caused by the language. Also, no one has ever claimed Rust is perfect. What people are claiming is it is a good alternative to C/C++ which prevents a lot of common errors like buffer overflows.
- rablackburn 3y agoThis is one of the best job ads I’ve ever seen. Seamless segue into culture and “btw we’re hiring” at the end :) This is truly a fantastic post-mortem: even an application-level developer like myself could follow it. Though I am in the middle of Rust in Action at the moment so I was primed for this type of content (: And of course it’s always a pleasure seeing someone else who comments their code at such a high LoC ratio. Literate programming works!
- IshKebab 3y agoOnly in America though unfortunately.
- deleted 3y ago[deleted]
- dcre 3y agoNot true. We have team members in Europe and NZ. “Most of our team are based outside of the Bay Area. We do ask that your workday overlaps with Pacific Time for at least four hours.”
- IshKebab 3y agoThat's quite an ask though - it means I'd finish at 8:30pm at the earliest every day which is completely impossible with children.
- steveklabnik 3y agoYou are not wrong, but that's why we try to focus on the direct need: shared time to directly collaborate with others in the company, rather than secondary expressions of those needs, like "must be in the US." It's true that means that this won't be a fit for a lot of people, but that's always the case with every job. (This is also why our job descriptions tend to not say things like "must have X years of Rust experience" and instead say things like "you will thrive in this role if you appreciate the hard-won thrill of debugging a knotty problem to root cause," because that's focusing on the direct need we have for the position.)
- retSava 3y agoAnyone knows what they mean by "For instance, we have an internal board that serves as a useful I2C debugging probe and has 8 kiB of RAM and 32 kiB of flash.". What could that interesting thing be? An inline-I2C sniffer perhaps? Or an interface board to inject i2c commands? Or something else?
- syntheticgate 3y agoI believe this is referring to a little board with an STM32G031 part that exposes a PMOD i2c interface and we use it for programming serial numbers into FRUID EEPROMs on the manufacturing line. It runs hubris, can be controlled by humility via SWD, and has just enough space to do this function. We have other dev boards using larger STM32 parts and didn't want to consume those for our manufacturing line stations. Given the size of the part on here, and the limited I/O, it's not very useful outside of this use-case but works great here.
- WatchDog 3y agoRefactoring that `can_access` function into it's own crate, seems excessive, leftPad vibes.
- mycoliza 3y agoWe did it as a result of my suggestion, and I'll freely admit that it does seem pretty goofy! The motivation, though, was that we wanted to write tests for the function that would run on a developer's machine using `cargo test`, and the Hubris kernel currently only compiles for the targets that we actually run Hubris on (various Cortex-M targets). So, if we want to write tests for that function, we would either need to move it to a crate that doesn't contain Cortex-M-only code, or litter the whole kernel with `#[cfg(not(test))]` attributes so that most of it doesn't compile when building tests. This felt much less unpleasant than conditional compilation. We're hoping that, eventually, other complex-but-not-architecture-specific kernel code will end up in the `kerncore` crate as well so that we can write tests for it, too, so eventually it won't be a crate with only one function... I do think that there's room for tooling improvement to make writing host-platform tests for `#![no_std]` Rust that gets cross-compiled to another platform, but I don't have any particularly concrete ideas for what that would look like. For now, at least, putting it in a separate crate lets us have our tests --- and those tests let us ensure that some of the function's edge cases are handled correctly, so I do think it was worth it.
- steveklabnik 3y agoIf you’re going by “how many LOC is in a translation unit” then the whole thing has “leftpad vibes.” The kernel itself is 2k lines. There’s lots of drivers that are only dozens of lines.
- xelxebar 3y ago> Tight nonhierarchical integration of the team. This isn’t a Hubris feature, but it’s hard to separate Hubris from the team that built it. Oxide’s engineering team has essentially no internal silos. Our culture rewards openness, curiosity, and communication, and discourages defensiveness, empire-building, and gatekeeping. We’ve worked hard to create and defend this culture, and I think it shows in the way we organized horizontally, across the borders of what other organizations would call teams, to solve this mystery. This caught my attention. I'd love to hear more about the motivations for crafting such a culture as well as some particular implementation details. I'm curious if there are drawbacks to fostering "openness, curiosity, and communication" within an organization? Obviously, some go for more rigid hierarchical systems, and it occurs to me that an org chart can (and probably should) be strategically decided upon, but I am somewhat clueless as to the tradeoffs. HN have any insight here?
- rcxdude 3y agoHmmmm, the specific values stated here is hard to evaluate, but in general one disadvantage of having an organisation without a strongly defined structure is that generally there still is a power structure of sorts. But when it's not explicitly defined it's less open, and generally less deliberately chosen, and harder to understand (especially for those who are good at social interaction), so it can allow sometimes even more pathological behaviours because of the shadowy nature of it, but also even if it doesn't get particularly bad it can make co-ordination much harder. I've experienced this in some companies I've worked for (one was a very large company where there was an explicit power structure of sorts, but it was not particularly stuck to in practice. It was a consultancy which worked on a variety of different projects, and the way you got onto projects was basically by making friends with the people selling and managing them, not so much through any particular official channel. This worked great if you were good at forming the social network needed to do this, it didn't work so well if you weren't). For other examples, there's "The Tyranny of Structurelessness" which is a talk from a feminist who noticed the same kind of thing happen in the organisations she was in, which generally rejected hierarchy as patriarchal, and you can see similar discussions around how this does/doesn't work in Valve, which also doesn't have a particularly well-defined internal structure. Open-source projects can suffer from the same thing (I would characterise some of the rust drama as stemming from a similar problem). That said, an explicit power structure doesn't need to be hierarchical, even though that's the 'traditional' business organisation. It's possible Oxide's structure is explicit but non-hierarchical. Also, in general such an approach tends to work better at smaller scales than larger ones. Said consultancy is probably the largest company I'm aware of which still mostly operates in a freeform way there, and they do have a kind of scaffolding which keeps it kinda held in place. And of course, this is more of a spectrum than a dichotomy: even the most rigid of power structures on paper still has some implicit, more complex implicit structure underneath: it's the nature of groups of people. Don't take this as me believing explicit > implicit in general. This is just me talking about some of the observed disadvantages of less explicit organisation. More explicit power structures have their own problems as well (and there's the related 'seeing like a state'/'legibility' issues there).