5 ms·

This is not some sort of "new" feature invented by OpenBSD. It's literally "just" them making their kernel (finally) multi-core instead of locking everything e

This is not some sort of "new" feature invented by OpenBSD. It's literally "just" them making their kernel (finally) multi-core instead of locking everything every time a process does networking. Pretty much every other OS out there has done that for years, if not decades. (FreeBSD for almost 20 years, although arguably at first that release was quite unstable.)

gostsamo 2y ago
This is what surprised me. It looks as a common sense approach, but it is on the front page of hn.
- knorker 2y ago
  I like openbsd for what it is, but they do have a tendency to imply or outright say that the thing they just did is something they invented. Like when they launched W^X (which isn't even an accurate term, because that implies that read-only non-execute pages don't exist), they basically claimed to have invented it. And they said that it was "impossible" to do on 32bit x86. Which was a surprise for those of us who had been running that on 32bit x86 for years, using grsec patches. They defended themselves for this mis-statement by saying that they don't look at what other OSs, like Linux, do. Fair enough, but don't say you're the first to do it, then. OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded. And since they have fewer developers (and not a focus on performance), they've understandably been lagging behind in this space.
  - daneel_w 2y ago
    > "OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded." Which seems like an entirely reasonable choice if they felt there were bigger priorities. OpenBSD received SMP support in November 2004, right at the same time as the first mainstream dual-core CPU arrived on the market. I get what you mean by "late to the game", but I also feel that it's not really a fitting statement.
    - knorker 2y ago
      Yes, I said they focus on other things. It's not criticism. But it is true that they dismissed SMP, and did not see dual-core coming. They were late because they were reacting to dual core launching. > I also feel that it's not really a fitting statement. In what way? Linux got it almost a decade before that. Windows NT got it more than a decade before OpenBSD. SunOS approaching 15 years before OpenBSD. ChatGPT claims Mac OS X got it in 1999 with Mac OS X Server, but I could not immediately confirm that one. Again, this is not criticism. But I don't see how it would be inaccurate to say they were late, when I think they were literally last.
      - daneel_w 2y ago
        > "In what way?" They were last, but not late. SMP in the x86 world was miniscule in terms of multi-CPU setups. It didn't "happen" until the multi-core era, and OpenBSD was well in time for that.
        knorker 2y ago
        I think you're nitpicking semantics, and don't know what your point is. Anyway, this is not a productive semantics debate. Have a nice weekend.
        daneel_w 2y ago
        If you were the last person to get to the train station but the train still hadn't left, then you really were not late for the train.
- bluGill 2y ago
  It is insanely difficult to implement it. The idea sounds common sense, but the details are extremely difficult to get right and any mistake causes random crashes, or incorrect data (read security holes).
  - MuffinFlavored 2y ago
    Is the challenge solved by basically "a lot of mutexes"?
    - Joker_vD 2y ago
      Don't mutexes provide, well, mutual exclusion, so that some operations would be forced to be single-threaded?
      - bluGill 2y ago
        Right, but if done right you rarely are in a situation two threads want to share the thread, so in practice you are not. However doing this right is difficult.
      - toast0 2y ago
        Yeah, but most of the things that need exclusivity or consistency are relatively rare or very narrow. You need consitency when you update the socket tables. (Relatively rare, but can be a bottleneck for use cases with frequent open/close and maybe accept) You need exclusivity when send or recieve on tcp sockets. (Very narrow) You also need exclusivity when putting packets one of on the NIC's sendqueues. That one's not really rare, better NICs have more queues for less contention though. For really high volume packet processing, it's helpful to align socket processing so everything is pinned to the same core: nic interrupts, kernel rx, application read and processing, application send, kernel tx, etc. But last I checked OpenBSD doesn't do cpu pinning, so there's that.
    - bluGill 2y ago
      That is one way to solve it. However it isn't always a good answer. Mutexs - by definition - mean that you stop everyone else from doing something when you are doing it, which means if two threads are trying to do something one will be stopped. It is also very easy to forget to lock/unlock a mutex at the right time, both of which will cause problems. Mutexs also send a message out to other CPUs which means even if no other thread wants the mutex you still slowed your system down. In the real world mutexs become painfully slow when more than a few threads are accessing the same shared data. There are also atomics which don't lock, and so in some cases are faster that a mutex - but they can also be a lot slower if in some data contention patterns. While these are often faster, there are a lot of useful data structures that cannot be written with atomics. The correct answer is design everything to avoid the needs for writing shared data as much as possible. Then use atomics and/or mutexs in those final places where data sharing is needed. You always need to benchmark to see what works best.
- dijit 2y ago
  Usually when OpenBSD does something it's because there's a clean implementation. OpenBSD is very seldom the first to actually do something; but they focus on making their code able to be understood by a single person. Doing things is hard, doing them in a way that is clear, obvious and "simple" is even harder. So when OpenBSD finally manages to do something that others may have had for a while... it's a cause of celebration. Because if the implementation was a clean one it would have gone in much sooner.
  - nequo 2y ago
    Do you know of a write up that illustrates this extra simplicity by comparing OpenBSD’s implementation of something with a previous mainstream implementation?
    - detourdog 2y ago
      I would say compare the OpenBSD documentation for ground up install to network serving to any other documentation source for the same process. OpenBSD will definitely be straight forward and easy to follow all from a single "manual". I'm sure there are other OSs that compare favorable to OpenBSD but those other OSs would also be high quality.
    - toast0 2y ago
      OpenBSD recently built a simple httpd and I think a mailer (replaces sendmail) as well. There's probably a write up of one of those. The big difference on those is a very limited scope.
drewg123 2y ago
I was recently looking at porting a cool OpenBSD feature to FreeBSD and was wondering how the locking could possibly work in terms of ordering. It turns out that all the OpenBSD locks boiled down to a global netlock rw lock for the entire IPv4/IPv6 stack.
mzs 2y ago
But this is only for raw IP, the moment the packet is determined to be say UDP, it gets queued like before this patch.

gostsamo 2y ago

This is what surprised me. It looks as a common sense approach, but it is on the front page of hn.

knorker 2y ago
I like openbsd for what it is, but they do have a tendency to imply or outright say that the thing they just did is something they invented. Like when they launched W^X (which isn't even an accurate term, because that implies that read-only non-execute pages don't exist), they basically claimed to have invented it. And they said that it was "impossible" to do on 32bit x86. Which was a surprise for those of us who had been running that on 32bit x86 for years, using grsec patches. They defended themselves for this mis-statement by saying that they don't look at what other OSs, like Linux, do. Fair enough, but don't say you're the first to do it, then. OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded. And since they have fewer developers (and not a focus on performance), they've understandably been lagging behind in this space.
- daneel_w 2y ago
  > "OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded." Which seems like an entirely reasonable choice if they felt there were bigger priorities. OpenBSD received SMP support in November 2004, right at the same time as the first mainstream dual-core CPU arrived on the market. I get what you mean by "late to the game", but I also feel that it's not really a fitting statement.
  - knorker 2y ago
    Yes, I said they focus on other things. It's not criticism. But it is true that they dismissed SMP, and did not see dual-core coming. They were late because they were reacting to dual core launching. > I also feel that it's not really a fitting statement. In what way? Linux got it almost a decade before that. Windows NT got it more than a decade before OpenBSD. SunOS approaching 15 years before OpenBSD. ChatGPT claims Mac OS X got it in 1999 with Mac OS X Server, but I could not immediately confirm that one. Again, this is not criticism. But I don't see how it would be inaccurate to say they were late, when I think they were literally last.
    - daneel_w 2y ago
      > "In what way?" They were last, but not late. SMP in the x86 world was miniscule in terms of multi-CPU setups. It didn't "happen" until the multi-core era, and OpenBSD was well in time for that.
      - knorker 2y ago
        I think you're nitpicking semantics, and don't know what your point is. Anyway, this is not a productive semantics debate. Have a nice weekend.
        daneel_w 2y ago
        If you were the last person to get to the train station but the train still hadn't left, then you really were not late for the train.
bluGill 2y ago
It is insanely difficult to implement it. The idea sounds common sense, but the details are extremely difficult to get right and any mistake causes random crashes, or incorrect data (read security holes).
- MuffinFlavored 2y ago
  Is the challenge solved by basically "a lot of mutexes"?
  - Joker_vD 2y ago
    Don't mutexes provide, well, mutual exclusion, so that some operations would be forced to be single-threaded?
    - bluGill 2y ago
      Right, but if done right you rarely are in a situation two threads want to share the thread, so in practice you are not. However doing this right is difficult.
    - toast0 2y ago
      Yeah, but most of the things that need exclusivity or consistency are relatively rare or very narrow. You need consitency when you update the socket tables. (Relatively rare, but can be a bottleneck for use cases with frequent open/close and maybe accept) You need exclusivity when send or recieve on tcp sockets. (Very narrow) You also need exclusivity when putting packets one of on the NIC's sendqueues. That one's not really rare, better NICs have more queues for less contention though. For really high volume packet processing, it's helpful to align socket processing so everything is pinned to the same core: nic interrupts, kernel rx, application read and processing, application send, kernel tx, etc. But last I checked OpenBSD doesn't do cpu pinning, so there's that.
  - bluGill 2y ago
    That is one way to solve it. However it isn't always a good answer. Mutexs - by definition - mean that you stop everyone else from doing something when you are doing it, which means if two threads are trying to do something one will be stopped. It is also very easy to forget to lock/unlock a mutex at the right time, both of which will cause problems. Mutexs also send a message out to other CPUs which means even if no other thread wants the mutex you still slowed your system down. In the real world mutexs become painfully slow when more than a few threads are accessing the same shared data. There are also atomics which don't lock, and so in some cases are faster that a mutex - but they can also be a lot slower if in some data contention patterns. While these are often faster, there are a lot of useful data structures that cannot be written with atomics. The correct answer is design everything to avoid the needs for writing shared data as much as possible. Then use atomics and/or mutexs in those final places where data sharing is needed. You always need to benchmark to see what works best.
dijit 2y ago
Usually when OpenBSD does something it's because there's a clean implementation. OpenBSD is very seldom the first to actually do something; but they focus on making their code able to be understood by a single person. Doing things is hard, doing them in a way that is clear, obvious and "simple" is even harder. So when OpenBSD finally manages to do something that others may have had for a while... it's a cause of celebration. Because if the implementation was a clean one it would have gone in much sooner.
- nequo 2y ago
  Do you know of a write up that illustrates this extra simplicity by comparing OpenBSD’s implementation of something with a previous mainstream implementation?
  - detourdog 2y ago
    I would say compare the OpenBSD documentation for ground up install to network serving to any other documentation source for the same process. OpenBSD will definitely be straight forward and easy to follow all from a single "manual". I'm sure there are other OSs that compare favorable to OpenBSD but those other OSs would also be high quality.
  - toast0 2y ago
    OpenBSD recently built a simple httpd and I think a mailer (replaces sendmail) as well. There's probably a write up of one of those. The big difference on those is a very limited scope.

drewg123 2y ago

I was recently looking at porting a cool OpenBSD feature to FreeBSD and was wondering how the locking could possibly work in terms of ordering. It turns out that all the OpenBSD locks boiled down to a global netlock rw lock for the entire IPv4/IPv6 stack.

mzs 2y ago

But this is only for raw IP, the moment the packet is determined to be say UDP, it gets queued like before this patch.