4 ms·

Building big systems with remote hardware teams

105 points by irsagent 4y ago

panick21_ 4y ago
Pretty cool how fast prototyping PCBs is. In a few years maybe you can do that with chips as well, with FPGAs you already can. I bet in a few years the have costume RISC-V in a few days for a few 1000$ as well. Next step, in house mini fab.
- jasonwatkinspdx 4y ago
  Wafters spend something like 3 months in continuous processing in a fab. Considering the extreme difficulty of the physics of the task and the scale of investment into fabs where small improvements in speed would be worth billions, I would be very skeptical that any low hanging fruit remain for dramatic shortcuts of the process.
  - pjc50 4y ago
    I think the actual process time is a lot shorter than that, but the dynamics of trying to (1) keep the fab 100% busy and (2) satisfy the most lucrative customers first mean that your job gets done at some point in a 3 month window.
    - jasonwatkinspdx 4y ago
      I am not an EE but my understanding is the time varies a bit based on complexity of the design (ie how many layers) but that even if you're willing to pay a multiple of the basic price the shortest you can expect is 2 months. There's physical steps that simply cannot be sped up more than partial inventory simply sitting around in a queue. Googling real quick agrees with that impression. There's just no way the current approach will get to the days turn around the top comment was betting will happen in a few years. If there was anything even remotely promising that with commercialization in a few years we'd all have heard about it loud and clear. Likewise, the support equipment, the chemicals involved. There's just no chance of current fab tech becoming a 3d printer like table top device. That would require a revolution even bigger than the original chip revolution.
    - artemonster 4y ago
      Please dont assume, if you dont know. Growing oxide, doping silicon, depositing metal, grinding all take substantial time. Even if you do a magical „no logistics everything in one place machine fab“ it would still be months of non-stop operation to do 200-300 process steps required.
  - panick21_ 4y ago
    I didn't know that, thanks for the information.
c_o_n_v_e_x 4y ago
Love their hardware prototyping strategy. Unit testing makes so much sense, but I've rarely seen it done in hardware. I'm curious how big their hardware team is?
- mkeeter 4y ago
  The official hardware team is about 10 people, but there's another 5-10 people that somewhat overlap. Several of the boards discussed in the post were designed by people that are nominally on the embedded team, e.g. I designed the Gimletlet NIC and Root-of-Trust-Carrier-Carrier.
- syntheticgate 4y ago
  Author here. One thing I really enjoy about this team is that we're trying to avoid silos and build a cohesive team that can flex when needed. As Matt mentioned, a number of the boards discussed are a result of someone jumping in and helping where there was a need. The Gemini bring-up board was done before we even had an "official" hardware team.
stn8188 4y ago
This post was a great read and it's exciting to see companies prove that hardware design can be done remotely. The real gem here though is the silkscreen on those boards, that was great to explore and laugh!
drewcoo 4y ago
How does the hardware talk? If it's not via software, I'm willing to bet the hardware is not remote. Or if not exactly software, it's contractual signaling, which we can model and understand and MITM in software. If it is by software, we can model these problems just like software. Known problems with known solutions. With the understanding that individual systems may incur side effects (like hardware bugs or . . . UI!). Most of this is not about hardware and a lot of it is unproved conjecture. There is also a smattering of hardware-specific knowledge (this wears out that way!) that is not about building large systems, but about problems in the small.
- pjc50 4y ago
  It's not the hardware which is "remote" but the team, i.e. a set of people working from home in different places rather than lab workbenches.
  - svnt 4y ago
    I think the parent has a point as the author makes a big deal of how this hardware supports their remote work. It’s clear to me oxide are having fun, and while you could do some of this in software, you clearly couldn’t do that without some hardware. Could they have purchased the hardware instead? Probably. Would it have been as cost-effective or fun? Probably not cost effective, without shifting their staffing. Does anyone have fun purchasing OTS development hardware? You’re just gambling on what systems and complexity in their own stuff they’ve really tested. Changing staffing is hard, expensive and painful. Repurposing apparently excess hardware development folks to fill the gap here is interesting, not the way hardware is “usually done” and seems to be working.
    - oytis 4y ago
      OTS development hardware is fine for very initial prototyping/derisking phase, but as your product evolves you need your development hardware to match the hardware in the field more closely.
      - 4y ago
pjc50 4y ago
> All of that to say, the cost of building small boards is really low. A single prototype run that saves a "real" board re-spin pays for itself immediately. Enabling software development before "real" hardware lands pays for itself immediately. Even when things don’t work out, the cost of failure is low; we lost a few hundred dollars and some engineering time, but learned something in the process. Yes. This is the absolutely central aspect of modern software development, and it's good to see the same being applied to hardware through cost and iteration time reductions. The modularity is interesting as well: it's basically building their own Arduino-like system of pieces.
al2o3cr 4y ago
The test boards remind me of this old article from Sparkfun, about how they use pogo pins to build custom testing jigs to verify products. https://www.sparkfun.com/tutorials/138 https://www.sparkfun.com/tutorials/138
hbogert 4y ago
Listen to Oxide's podcast "Oxide & Friends", it's awesome if you're in this space of datacenters, servers, networking etc. Last episode was about doing their own networking, and it's not some whitelabel $hit, they go to the firmware level and are trying to open up stuff like a server's BMC. (jesus christ those things are horrendous.) [1]: https://www.techopedia.com/definition/15941/baseboard-management-controller-bmc https://www.techopedia.com/definition/15941/baseboard-manage...

105 points by irsagent 4y ago

Pretty cool how fast prototyping PCBs is. In a few years maybe you can do that with chips as well, with FPGAs you already can. I bet in a few years the have costume RISC-V in a few days for a few 1000$ as well. Next step, in house mini fab.

jasonwatkinspdx 4y ago
Wafters spend something like 3 months in continuous processing in a fab. Considering the extreme difficulty of the physics of the task and the scale of investment into fabs where small improvements in speed would be worth billions, I would be very skeptical that any low hanging fruit remain for dramatic shortcuts of the process.
- pjc50 4y ago
  I think the actual process time is a lot shorter than that, but the dynamics of trying to (1) keep the fab 100% busy and (2) satisfy the most lucrative customers first mean that your job gets done at some point in a 3 month window.
  - jasonwatkinspdx 4y ago
    I am not an EE but my understanding is the time varies a bit based on complexity of the design (ie how many layers) but that even if you're willing to pay a multiple of the basic price the shortest you can expect is 2 months. There's physical steps that simply cannot be sped up more than partial inventory simply sitting around in a queue. Googling real quick agrees with that impression. There's just no way the current approach will get to the days turn around the top comment was betting will happen in a few years. If there was anything even remotely promising that with commercialization in a few years we'd all have heard about it loud and clear. Likewise, the support equipment, the chemicals involved. There's just no chance of current fab tech becoming a 3d printer like table top device. That would require a revolution even bigger than the original chip revolution.
  - artemonster 4y ago
    Please dont assume, if you dont know. Growing oxide, doping silicon, depositing metal, grinding all take substantial time. Even if you do a magical „no logistics everything in one place machine fab“ it would still be months of non-stop operation to do 200-300 process steps required.
- panick21_ 4y ago
  I didn't know that, thanks for the information.

c_o_n_v_e_x 4y ago

Love their hardware prototyping strategy. Unit testing makes so much sense, but I've rarely seen it done in hardware. I'm curious how big their hardware team is?

mkeeter 4y ago
The official hardware team is about 10 people, but there's another 5-10 people that somewhat overlap. Several of the boards discussed in the post were designed by people that are nominally on the embedded team, e.g. I designed the Gimletlet NIC and Root-of-Trust-Carrier-Carrier.
syntheticgate 4y ago
Author here. One thing I really enjoy about this team is that we're trying to avoid silos and build a cohesive team that can flex when needed. As Matt mentioned, a number of the boards discussed are a result of someone jumping in and helping where there was a need. The Gemini bring-up board was done before we even had an "official" hardware team.

stn8188 4y ago

This post was a great read and it's exciting to see companies prove that hardware design can be done remotely. The real gem here though is the silkscreen on those boards, that was great to explore and laugh!

drewcoo 4y ago

How does the hardware talk? If it's not via software, I'm willing to bet the hardware is not remote. Or if not exactly software, it's contractual signaling, which we can model and understand and MITM in software. If it is by software, we can model these problems just like software. Known problems with known solutions. With the understanding that individual systems may incur side effects (like hardware bugs or . . . UI!). Most of this is not about hardware and a lot of it is unproved conjecture. There is also a smattering of hardware-specific knowledge (this wears out that way!) that is not about building large systems, but about problems in the small.

pjc50 4y ago
It's not the hardware which is "remote" but the team, i.e. a set of people working from home in different places rather than lab workbenches.
- svnt 4y ago
  I think the parent has a point as the author makes a big deal of how this hardware supports their remote work. It’s clear to me oxide are having fun, and while you could do some of this in software, you clearly couldn’t do that without some hardware. Could they have purchased the hardware instead? Probably. Would it have been as cost-effective or fun? Probably not cost effective, without shifting their staffing. Does anyone have fun purchasing OTS development hardware? You’re just gambling on what systems and complexity in their own stuff they’ve really tested. Changing staffing is hard, expensive and painful. Repurposing apparently excess hardware development folks to fill the gap here is interesting, not the way hardware is “usually done” and seems to be working.
  - oytis 4y ago
    OTS development hardware is fine for very initial prototyping/derisking phase, but as your product evolves you need your development hardware to match the hardware in the field more closely.
    - 4y ago

pjc50 4y ago

> All of that to say, the cost of building small boards is really low. A single prototype run that saves a "real" board re-spin pays for itself immediately. Enabling software development before "real" hardware lands pays for itself immediately. Even when things don’t work out, the cost of failure is low; we lost a few hundred dollars and some engineering time, but learned something in the process. Yes. This is the absolutely central aspect of modern software development, and it's good to see the same being applied to hardware through cost and iteration time reductions. The modularity is interesting as well: it's basically building their own Arduino-like system of pieces.

al2o3cr 4y ago

The test boards remind me of this old article from Sparkfun, about how they use pogo pins to build custom testing jigs to verify products. https://www.sparkfun.com/tutorials/138 https://www.sparkfun.com/tutorials/138

hbogert 4y ago

Listen to Oxide's podcast "Oxide & Friends", it's awesome if you're in this space of datacenters, servers, networking etc. Last episode was about doing their own networking, and it's not some whitelabel $hit, they go to the firmware level and are trying to open up stuff like a server's BMC. (jesus christ those things are horrendous.) [1]: https://www.techopedia.com/definition/15941/baseboard-management-controller-bmc https://www.techopedia.com/definition/15941/baseboard-manage...