7 ms·

There's a bunch in here that I think HN readers will find fascinating, but it may help to have a little context going in. At Oxide, we have developed our own s

There's a bunch in here that I think HN readers will find fascinating, but it may help to have a little context going in. At Oxide, we have developed our own switch[0][1] using switching silicon that implements P4[2]. P4 is really interesting -- you can think of it as a much cleaner eBPF -- and our programmable switch is effectively a P4 engine. This is really powerful; if you want to jump to why, listen to Ry expand on it about an hour into the conversation.[3] This conversation goes into the details about making all of this work. Along the way, we hit on pushing a cycle-accurate simulator to its limits; developing a new P4 compiler that has a Rust target; using that compiler to then develop an accurate hardware-virtualized model of the ASIC; developing a new routing protocol (delay driven multipathing); and how all of this adds to up solving some gnarly problems like microcongestion and flexible topology construction. [0] https://share.transistor.fm/s/c80a79d8 https://share.transistor.fm/s/c80a79d8 [1] https://share.transistor.fm/s/65a10522 https://share.transistor.fm/s/65a10522 [2] https://p4.org/ https://p4.org/ [3] https://www.youtube.com/watch?v=AkWh2Sms3aw#t=1h4m46s https://www.youtube.com/watch?v=AkWh2Sms3aw#t=1h4m46s

supriyo-biswas 4y ago
While I’m a big fan of your work and have been deeply both entertained and educated by your presentation skills, an unfortunate and concerning observation is that the products that you’ve championed generally failed to see adoption in the mass market, and it may be due to the overtly technical focus, which is not bad in of itself but lengthens the time to market. Perhaps a more pragmatic approach may be to release an iteration of the product before improving on it. I say this not to detract from your work but because I’d genuinely like to see companies opting into self-hosted hardware as Oxide envisions. The centralization of all internet services into three big cloud vendors does not bode well for society.
- naikrovek 4y ago
  it's ok for Bryan to work on things that he believes have value. given his skills and experience compared to anyone I've worked with (at least at the time that I worked with them) I would trust what Bryan puts his weight behind over the internet's techno-fetish of the month, any day.
  - supriyo-biswas 4y ago
    I don’t disagree with any part of your comment — however, it is sad that technologically superior solutions that Bryan has worked on “lost”. As an example, Manta/Triton had an objectively better model, and yet it is sad that we’re all stuck with S3 like storages today.
    - naikrovek 4y ago
      sometimes "good enough" wins, even when something better is available. you must always work on things that have value to you, even if those things are unknown or unpopular. not because they are unknown or unpopular, but because they are the right solution for you. the amount of homogeny that I see today staggers me. people don't seem to be solving their actual problems, but instead are twisting their problems so that the commonly deployed solutions fit. Oxide seem to be developing solutions for people that have problems which the common solutions can not address, no matter how much you twist them.
- bcantrill 4y ago
  While I appreciate the kind words, I don't really agree with your analysis of my career, for whatever it's worth. ;) In terms of Oxide: I can assure you that we are shipping the minimum viable product -- the challenge is that the minimum viable product is very (very!) large. (And indeed, the projections we made in our initial investor deck were far more accurate than they had any business being!) Glad to hear that you're cheering us on, and thanks again for the kind words!
  - panick21_ 4y ago
    Fishworks made a lot of money for Oracle, if that's not success I don't know what is. PS: I would really love an episode about Sun deciding to go to bed with AT&T and merge and BSD and System V. That seem to cast a long shadow and must have left some scars.
mtlmtlmtlmtl 4y ago
I love your podcasts, your talks, and most of all your technical work, but I would really appreciate it if you found the time to get a better microphone for these recordings. I get that this is something you just hit record on while chatting on a work laptop or something, but the 's' sounds in your audio track especially are so high pitched it's unbearable to listen to for me.
- bcantrill 4y ago
  So, I am on what is supposed to be a decent mic on this episode -- so this is hopefully a solvable problem. If it's not too much to ask, could you point me to a minute that's unbearable, and I'll do what I can to solve it?
  - mtlmtlmtlmtl 4y ago
    Might be you're a little too close to it, or something else like that(too much treble in the settings or something?). I will say that this recording does sound better than a much earlier one I tried a while back. So if you did change your mic at some point, it probably helped quite a bit. I just listened to the first couple minutes, but generally quite a few of your 's' sounds trigger the nails on chalkboard type reaction(not that bad though, but same type of reaction). Hope this makes sense to you. Im a couple decades younger than you, so could be I'm hearing some high frequencies you couldn't hear. Keep up the great work at Oxide!
    - bcantrill 4y ago
      Glad to hear that this one was better. If you can bear to indicate the exact offset of a grating "s" it would help, but otherwise we'll listen to the first few minutes and do our best; thanks for the feedback!
      - zoratu 4y ago
        0:14 of https://www.youtube.com/watch?v=NVZ80_tbkbc https://www.youtube.com/watch?v=NVZ80_tbkbc is a good example of previous recordings. It is helpful, if you think you have a good mic, to talk past it rather than into it.
sargun 4y ago
What switching silicon did you end up using? Given the speeds and feeds of the switching silicon you used, was it cost-effective compared to using Broadcom, and just overprovisioning the hell out of it?
- bcantrill 4y ago
  We're using Intel Tofino 2 -- and there are many reasons why we do not regret the choice!
  - pclmulqdq 4y ago
    I assume that since you're using P4, you can likely pivot to Mount Evans chips (or the successors in that line) and a dumber switch if you need to, and the fact that Google is buying them too gives the product some legs from Intel's perspective. I really like Intel's networking products, but they have an unfortunate habit of getting canceled.
    - jsolson 4y ago
      Mount Evans is NIC silicon -- great chip, but it doesn't have the lane count / topology to be an effective switch chip as far as I'm aware.
      - rygood 4y ago
        Yeah, definitely not a switch ASIC. But would love to see something like the E2000 with 1) an open underlying ISA to enable truly open P4 compilers and 2) no compute complex so we can focus the power budget on the programmable data plane.
      - pclmulqdq 4y ago
        I will disclaim that I have no idea how Oxide is using P4 for packet processing, but what you are currently doing with P4 on your switch can likely be moved to some rules on a dumber switch/router (eg what hyperscalers use) plus a special packet encapsulation on the NIC using similar P4 rules. That lets you push the intelligence out of the switch and toward the edge of the network. At least, this is the approach that the major hyperscalers are taking, and there doesn't seem to be a good reason to not do this (other than potentially the cost, since you don't have the scale to get a good discount on the NICs).
infogulch 4y ago
This was a great episode, thank you! If you allow me to query on a different oxide topic: I noticed that you provide a certificate to each HSM in every system that is signed by an Oxide Computing root cert. This is pretty neat because, combined with other security features, it enables clients to validate that the running firmware came from you. Have you considered publishing a certificate transparency log of all certificates generated for hardware root of trust HSMs? This would enable customers to ensure they are not using firmware signed using a compromised root certificate. https://eprint.iacr.org/2022/981 https://eprint.iacr.org/2022/981
- bcantrill 4y ago
  Yes, and we really need to do an entire episode (or several!) on how we're tackling that problem. Right now that team is very, very heads-down (as you can imagine, having this apparatus in place is a blocker to start manufacturing), but count on an episode on this in the not-so-distant future!
  - infogulch 4y ago
    I'm glad to hear that, and now I'm excited for a future episode about it!
fruffy 4y ago
Hi Bryan, As someone deeply involved with the P4 community and contributor to the P4 compiler it is great to see someone being bullish on the language. Have been following your work with interest. :) I see that you are developing your own compiler. If you have complaints or suggestions please feel free to reach out to p4-design@p4.org. The language design group is meeting monthly and is always interested in hearing grievances of consumers of the language. As a side-note, we are currently developing tools to ensure that P4 programs are compiled and executed correctly. Tools include packet-test generators, fuzzers, and general-purpose verification frameworks. Because the language is so restricted you can do really powerful validation. A concrete example is Google using P4 as a specification language for their network devices[0]. If that is something that interests you, shoot me a message (fruffy@nyu.edu). [0] https://research.google/pubs/pub51497/ https://research.google/pubs/pub51497/
- rygood 4y ago
  Hey! Thanks for reaching out. Primary author of the Oxide P4 compiler here. I joined the P4 Language Design WG a while back and have attended a few meetings as time and schedules allow. I'll join in on next week's meeting in case folks are interested in follow-up questions/discussions.

supriyo-biswas 4y ago

While I’m a big fan of your work and have been deeply both entertained and educated by your presentation skills, an unfortunate and concerning observation is that the products that you’ve championed generally failed to see adoption in the mass market, and it may be due to the overtly technical focus, which is not bad in of itself but lengthens the time to market. Perhaps a more pragmatic approach may be to release an iteration of the product before improving on it. I say this not to detract from your work but because I’d genuinely like to see companies opting into self-hosted hardware as Oxide envisions. The centralization of all internet services into three big cloud vendors does not bode well for society.

naikrovek 4y ago
it's ok for Bryan to work on things that he believes have value. given his skills and experience compared to anyone I've worked with (at least at the time that I worked with them) I would trust what Bryan puts his weight behind over the internet's techno-fetish of the month, any day.
- supriyo-biswas 4y ago
  I don’t disagree with any part of your comment — however, it is sad that technologically superior solutions that Bryan has worked on “lost”. As an example, Manta/Triton had an objectively better model, and yet it is sad that we’re all stuck with S3 like storages today.
  - naikrovek 4y ago
    sometimes "good enough" wins, even when something better is available. you must always work on things that have value to you, even if those things are unknown or unpopular. not because they are unknown or unpopular, but because they are the right solution for you. the amount of homogeny that I see today staggers me. people don't seem to be solving their actual problems, but instead are twisting their problems so that the commonly deployed solutions fit. Oxide seem to be developing solutions for people that have problems which the common solutions can not address, no matter how much you twist them.
bcantrill 4y ago
While I appreciate the kind words, I don't really agree with your analysis of my career, for whatever it's worth. ;) In terms of Oxide: I can assure you that we are shipping the minimum viable product -- the challenge is that the minimum viable product is very (very!) large. (And indeed, the projections we made in our initial investor deck were far more accurate than they had any business being!) Glad to hear that you're cheering us on, and thanks again for the kind words!
- panick21_ 4y ago
  Fishworks made a lot of money for Oracle, if that's not success I don't know what is. PS: I would really love an episode about Sun deciding to go to bed with AT&T and merge and BSD and System V. That seem to cast a long shadow and must have left some scars.

mtlmtlmtlmtl 4y ago

I love your podcasts, your talks, and most of all your technical work, but I would really appreciate it if you found the time to get a better microphone for these recordings. I get that this is something you just hit record on while chatting on a work laptop or something, but the 's' sounds in your audio track especially are so high pitched it's unbearable to listen to for me.

bcantrill 4y ago
So, I am on what is supposed to be a decent mic on this episode -- so this is hopefully a solvable problem. If it's not too much to ask, could you point me to a minute that's unbearable, and I'll do what I can to solve it?
- mtlmtlmtlmtl 4y ago
  Might be you're a little too close to it, or something else like that(too much treble in the settings or something?). I will say that this recording does sound better than a much earlier one I tried a while back. So if you did change your mic at some point, it probably helped quite a bit. I just listened to the first couple minutes, but generally quite a few of your 's' sounds trigger the nails on chalkboard type reaction(not that bad though, but same type of reaction). Hope this makes sense to you. Im a couple decades younger than you, so could be I'm hearing some high frequencies you couldn't hear. Keep up the great work at Oxide!
  - bcantrill 4y ago
    Glad to hear that this one was better. If you can bear to indicate the exact offset of a grating "s" it would help, but otherwise we'll listen to the first few minutes and do our best; thanks for the feedback!
    - zoratu 4y ago
      0:14 of https://www.youtube.com/watch?v=NVZ80_tbkbc https://www.youtube.com/watch?v=NVZ80_tbkbc is a good example of previous recordings. It is helpful, if you think you have a good mic, to talk past it rather than into it.

sargun 4y ago

What switching silicon did you end up using? Given the speeds and feeds of the switching silicon you used, was it cost-effective compared to using Broadcom, and just overprovisioning the hell out of it?

bcantrill 4y ago
We're using Intel Tofino 2 -- and there are many reasons why we do not regret the choice!
- pclmulqdq 4y ago
  I assume that since you're using P4, you can likely pivot to Mount Evans chips (or the successors in that line) and a dumber switch if you need to, and the fact that Google is buying them too gives the product some legs from Intel's perspective. I really like Intel's networking products, but they have an unfortunate habit of getting canceled.
  - jsolson 4y ago
    Mount Evans is NIC silicon -- great chip, but it doesn't have the lane count / topology to be an effective switch chip as far as I'm aware.
    - rygood 4y ago
      Yeah, definitely not a switch ASIC. But would love to see something like the E2000 with 1) an open underlying ISA to enable truly open P4 compilers and 2) no compute complex so we can focus the power budget on the programmable data plane.
    - pclmulqdq 4y ago
      I will disclaim that I have no idea how Oxide is using P4 for packet processing, but what you are currently doing with P4 on your switch can likely be moved to some rules on a dumber switch/router (eg what hyperscalers use) plus a special packet encapsulation on the NIC using similar P4 rules. That lets you push the intelligence out of the switch and toward the edge of the network. At least, this is the approach that the major hyperscalers are taking, and there doesn't seem to be a good reason to not do this (other than potentially the cost, since you don't have the scale to get a good discount on the NICs).

infogulch 4y ago

This was a great episode, thank you! If you allow me to query on a different oxide topic: I noticed that you provide a certificate to each HSM in every system that is signed by an Oxide Computing root cert. This is pretty neat because, combined with other security features, it enables clients to validate that the running firmware came from you. Have you considered publishing a certificate transparency log of all certificates generated for hardware root of trust HSMs? This would enable customers to ensure they are not using firmware signed using a compromised root certificate. https://eprint.iacr.org/2022/981 https://eprint.iacr.org/2022/981

bcantrill 4y ago
Yes, and we really need to do an entire episode (or several!) on how we're tackling that problem. Right now that team is very, very heads-down (as you can imagine, having this apparatus in place is a blocker to start manufacturing), but count on an episode on this in the not-so-distant future!
- infogulch 4y ago
  I'm glad to hear that, and now I'm excited for a future episode about it!

fruffy 4y ago

Hi Bryan, As someone deeply involved with the P4 community and contributor to the P4 compiler it is great to see someone being bullish on the language. Have been following your work with interest. :) I see that you are developing your own compiler. If you have complaints or suggestions please feel free to reach out to p4-design@p4.org. The language design group is meeting monthly and is always interested in hearing grievances of consumers of the language. As a side-note, we are currently developing tools to ensure that P4 programs are compiled and executed correctly. Tools include packet-test generators, fuzzers, and general-purpose verification frameworks. Because the language is so restricted you can do really powerful validation. A concrete example is Google using P4 as a specification language for their network devices[0]. If that is something that interests you, shoot me a message (fruffy@nyu.edu). [0] https://research.google/pubs/pub51497/ https://research.google/pubs/pub51497/

rygood 4y ago
Hey! Thanks for reaching out. Primary author of the Oxide P4 compiler here. I joined the P4 Language Design WG a while back and have attended a few meetings as time and schedules allow. I'll join in on next week's meeting in case folks are interested in follow-up questions/discussions.