5 ms·

I really wish hardware startups would at least hire people early on who have experience with actual manufacturing. Stuff like this really shouldn't happen and

I really wish hardware startups would at least hire people early on who have experience with actual manufacturing. Stuff like this really shouldn't happen and really shouldn't be a surprise, either. Temperature considerations are a standard part of the design criteria and testing in any mechanical device, and standard root cause analysis (did they even do FMEA?) would have forced them to find the problem the first time, not the second. When you're still in the early prototype phase, fine, but this is a 25? person team selling a product with a subscription and everything. Edit: just realized their stated fix is to remove two resistors. Did they actually test the full effects of that change? Seems like there's a decent chance of a third head-scratcher a few months from now...

fuzzfactor 4y ago
You shouldn't need to be an engineer to be familiar with Murphy's Law, but when you are it can make you a more effective designer. One of the most memorable corollaries to Murphy is: "Mother Nature always sides with the hidden flaw. Mother Nature is a bitch."
Tempest1981 4y ago
Yep, we used a temperature chamber that could ramp from -40°C to +80°C, along with humidity. We would push the limits until something broke, fix the design, and repeat. Surprised, but not surprised, that everyone doesn't do at least some stress testing. I was expecting something more complicated, like electromagnetic interference from a passing vehicle. Or ESD (static discharge).
exmadscientist 4y ago
> standard root cause analysis (did they even do FMEA?) would have forced them to find the problem the first time, not the second Not to mention that the standard checklist for solving "why is this joint having problems" is "is it cold or does it get worse when we make it cold".... > just realized their stated fix is to remove two resistors. Did they actually test the full effects of that change? Seems like there's a decent chance of a third head-scratcher a few months from now... As a half-decent analog guy I can actually believe this. I can imagine a lot of ways something like this could fix a problem, though of course I haven't seen any schematics here.
- pclmulqdq 4y ago
  I'm guessing that a robotics company actually has very few analog-competent engineers. It may well have solved the issue if one of them had built an analog circuit. A lot of engineers don't think about operating conditions when they first learn to build circuits, and run them near the edge of their specified tolerances, so moving the component into a "comfortable" range would fix it. However, if they were working with a COTS component and took some resistors off that, I doubt it was actually the right fix.
  - not_the_fda 4y ago
    Any experienced EE should know this stuff and one of the first questions on doing a design is "what is the operational environment?".
Game_Ender 4y ago
As a software person this is first I have heard of FMEA (Failure Mode and Effect Analysis) [0]. Sounds like a very rigorous way to move through a system and identify all the ways it can break and develop ways to fix it. 0 - https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis https://en.wikipedia.org/wiki/Failure_mode_and_effects_analy...
- CamperBob2 4y ago
  Not everybody needs to run formal FMEA procedures on everything, but if you ship hardware without at least putting it in the fridge for a few hours next to the Jolt Cola and leftover kung pao shrimp, you are tempting fate.
  - defrost 4y ago
    NB: If kung pao unavailable a good Madras curry will substitute.
  - IshKebab 4y ago
    Yeah I mean maybe it doesn't matter for robots intended to be used indoors, but they're making a robot intended to be used outside in all (dry) weather on construction sites!
pifm_guy 4y ago
What's the betting that on the hot weather their Texas customers with those resistors removed will find their robots nonfunctional again...?
steve_adams_86 4y ago
I’m just a bum with an automated hydroponic garden and I verified and tested all of my components for expected behaviour in the normal conditions the system will be in. I had to exclude several components as a result. On one hand I’m surprised they didn’t do that, but on the other, I have no time constraints and I’m very, very uncertain of what I’m doing most of the time. I knew I wanted to have accurate readings and reliable performance so my garden doesn’t die due to malfunctions. So, I goofed around and made sure stuff was right. I’d do the same with code in my spare time, but my employers have me cut corners all the time. People could criticize me for it, but it’s not as though I don’t know better. It might be similar for this team. By the time the bugs strike, it’s not clear what’s been properly vetted or who knows what about which components. Debugging becomes harder because the initial spec and how well it was met is no longer clear. I’m totally guessing as I don’t really know their team, process, or hardware in general. I also discovered the ruggeduino line of arduino boards in the process of component elimination, which are pretty cool. Overkill for my use case, but I hope to have a use for one some day. I’m thinking of making a robot shop vac and metal-picker-upper, and the ruggeduino would be great there I think.
- marmetio 4y ago
  If you buy a password manager from mechanical engineers but it doesn't encrypt anything because they didn't know it should, is the issue time constraints or that they didn't hire a software engineer?
  - steve_adams_86 4y ago
    Definitely that they didn't hire the right person. Do you mean to say that this company doesn't employ the right people to build robots?
    - marmetio 4y ago
      No. They sell product profitably. Who am I to say they're wrong? Same with their customers using the product profitably. If they develop a business need to avoid similar issues before they happen, then the answer would be yes. The work they unwittingly skipped (like design for environment) and struggled with (diagnosing an environment problem) are the daily tasks for many people, so they obviously don't have one of those people.
tempestn 4y ago
Ha, reminds me of something at a company I used to work at! One of their products included this little switched power supply of some kind. I don't recall the exact details, but basically it included a snubber circuit that was failing due to some transients being larger than designed for. So they fixed it by.. removing the snubber. That worked briefly, but then of course the transistor it was protecting started failing. So they installed a massive overkill transistor, which worked for a while, until that started overheating, so they redesigned the PCB to move it to the edge of the board where it could be heat-sinked to the case. Even with that they were still getting occasional returns with it failed, so when I joined the company I was asked to look at it. Added the snubber back in with an appropriately-sized resistor, and problem solved. (And was able to switch back from the $30 transistor to one that cost a few cents.)
- dreamcompiler 4y ago
  "The purpose of a transistor is to protect the fuse." -old hardware lore
indrora 4y ago
So, sometimes you really don't anticipate the kinds of failures you can have. A really good one to remember is the "Camera Shy" Pi 2. For those unfamiliar with this issue, or who forgot: The power regulator IC on the Pi 2 was a flip-silicon slab that was otherwise uncoated or encased. When engineers were taking pictures of it, they likely were using their cell phones and otherwise loose lighting under fluorescent or sunlight. When high intensity Xenon bulbs were used for flash photography, however, intense IR light would hit the silicon, excite something, and cause the power supply to go wobbly kneed, drop out like a Silicon Valley startup founder, and you'd have a bad day. This failure was only figured out when someone happened to compare two flashing lights (one Xenon and one a bike light) that they figured out what was going on. Engineers miss things all the time, even those who are supposedly really good. They are good, but humans err, because to err is human. Prime example: I diagnosed an arcade cabinet that only had issues on sustained playtime during the summer, but never during the winter. It was fine on the off day that there was a lot of cloud cover, and only seemed to get worse after a shade awning was taken down: before the removal, it would intermittently sputter back to life during the later afternoon. Nobody knew what was going on. I spent an afternoon watching it fail. At one point, I eventually went and grabbed an IR thermometer and pointed it at the cabinet. It was registering well into 100F. It was also outright refusing to work. I rolled down the window blinds, pointed a fan at the poor machine, and waited for a while. Miraculously, it woke up after a while and started working. I traced it to the three canned voltage regulators. The voltage regulators for the system were strapped to the metal chassis of the CRT, which contained the system's power supply. The machine happened to be right next to a window, and the cabinet had been painted black by the previous owner. As the daytime sun would wander over, the black paint would soak up the UV and absolutely cook the regulators. Once the sun had passed behind the shade, it was quickly cool enough inside the chassis to bring the regulators back into spec, which caused the machine to start working again. When the shade was removed, it only went back up once the sun was no longer cooking it like a roast chicken. I believe my recommendation was "Replace the side paneling with woodgrain and move it out of the sun." The engineer that designed that arcade machine probably never thought it'd be in a hot, New Mexico university faculty lounge next to a window that faced the sun. I remember going back about a year later and sure enough, it had moved about three feet and was "rarely, if ever broken" now. Engineers miss things all the time.

fuzzfactor 4y ago

You shouldn't need to be an engineer to be familiar with Murphy's Law, but when you are it can make you a more effective designer. One of the most memorable corollaries to Murphy is: "Mother Nature always sides with the hidden flaw. Mother Nature is a bitch."

Tempest1981 4y ago

Yep, we used a temperature chamber that could ramp from -40°C to +80°C, along with humidity. We would push the limits until something broke, fix the design, and repeat. Surprised, but not surprised, that everyone doesn't do at least some stress testing. I was expecting something more complicated, like electromagnetic interference from a passing vehicle. Or ESD (static discharge).

exmadscientist 4y ago

> standard root cause analysis (did they even do FMEA?) would have forced them to find the problem the first time, not the second Not to mention that the standard checklist for solving "why is this joint having problems" is "is it cold or does it get worse when we make it cold".... > just realized their stated fix is to remove two resistors. Did they actually test the full effects of that change? Seems like there's a decent chance of a third head-scratcher a few months from now... As a half-decent analog guy I can actually believe this. I can imagine a lot of ways something like this could fix a problem, though of course I haven't seen any schematics here.

pclmulqdq 4y ago
I'm guessing that a robotics company actually has very few analog-competent engineers. It may well have solved the issue if one of them had built an analog circuit. A lot of engineers don't think about operating conditions when they first learn to build circuits, and run them near the edge of their specified tolerances, so moving the component into a "comfortable" range would fix it. However, if they were working with a COTS component and took some resistors off that, I doubt it was actually the right fix.
- not_the_fda 4y ago
  Any experienced EE should know this stuff and one of the first questions on doing a design is "what is the operational environment?".

Game_Ender 4y ago

As a software person this is first I have heard of FMEA (Failure Mode and Effect Analysis) [0]. Sounds like a very rigorous way to move through a system and identify all the ways it can break and develop ways to fix it. 0 - https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis https://en.wikipedia.org/wiki/Failure_mode_and_effects_analy...

CamperBob2 4y ago
Not everybody needs to run formal FMEA procedures on everything, but if you ship hardware without at least putting it in the fridge for a few hours next to the Jolt Cola and leftover kung pao shrimp, you are tempting fate.
- defrost 4y ago
  NB: If kung pao unavailable a good Madras curry will substitute.
- IshKebab 4y ago
  Yeah I mean maybe it doesn't matter for robots intended to be used indoors, but they're making a robot intended to be used outside in all (dry) weather on construction sites!

pifm_guy 4y ago

What's the betting that on the hot weather their Texas customers with those resistors removed will find their robots nonfunctional again...?

steve_adams_86 4y ago

I’m just a bum with an automated hydroponic garden and I verified and tested all of my components for expected behaviour in the normal conditions the system will be in. I had to exclude several components as a result. On one hand I’m surprised they didn’t do that, but on the other, I have no time constraints and I’m very, very uncertain of what I’m doing most of the time. I knew I wanted to have accurate readings and reliable performance so my garden doesn’t die due to malfunctions. So, I goofed around and made sure stuff was right. I’d do the same with code in my spare time, but my employers have me cut corners all the time. People could criticize me for it, but it’s not as though I don’t know better. It might be similar for this team. By the time the bugs strike, it’s not clear what’s been properly vetted or who knows what about which components. Debugging becomes harder because the initial spec and how well it was met is no longer clear. I’m totally guessing as I don’t really know their team, process, or hardware in general. I also discovered the ruggeduino line of arduino boards in the process of component elimination, which are pretty cool. Overkill for my use case, but I hope to have a use for one some day. I’m thinking of making a robot shop vac and metal-picker-upper, and the ruggeduino would be great there I think.

marmetio 4y ago
If you buy a password manager from mechanical engineers but it doesn't encrypt anything because they didn't know it should, is the issue time constraints or that they didn't hire a software engineer?
- steve_adams_86 4y ago
  Definitely that they didn't hire the right person. Do you mean to say that this company doesn't employ the right people to build robots?
  - marmetio 4y ago
    No. They sell product profitably. Who am I to say they're wrong? Same with their customers using the product profitably. If they develop a business need to avoid similar issues before they happen, then the answer would be yes. The work they unwittingly skipped (like design for environment) and struggled with (diagnosing an environment problem) are the daily tasks for many people, so they obviously don't have one of those people.

tempestn 4y ago

Ha, reminds me of something at a company I used to work at! One of their products included this little switched power supply of some kind. I don't recall the exact details, but basically it included a snubber circuit that was failing due to some transients being larger than designed for. So they fixed it by.. removing the snubber. That worked briefly, but then of course the transistor it was protecting started failing. So they installed a massive overkill transistor, which worked for a while, until that started overheating, so they redesigned the PCB to move it to the edge of the board where it could be heat-sinked to the case. Even with that they were still getting occasional returns with it failed, so when I joined the company I was asked to look at it. Added the snubber back in with an appropriately-sized resistor, and problem solved. (And was able to switch back from the $30 transistor to one that cost a few cents.)

dreamcompiler 4y ago
"The purpose of a transistor is to protect the fuse." -old hardware lore

indrora 4y ago

So, sometimes you really don't anticipate the kinds of failures you can have. A really good one to remember is the "Camera Shy" Pi 2. For those unfamiliar with this issue, or who forgot: The power regulator IC on the Pi 2 was a flip-silicon slab that was otherwise uncoated or encased. When engineers were taking pictures of it, they likely were using their cell phones and otherwise loose lighting under fluorescent or sunlight. When high intensity Xenon bulbs were used for flash photography, however, intense IR light would hit the silicon, excite something, and cause the power supply to go wobbly kneed, drop out like a Silicon Valley startup founder, and you'd have a bad day. This failure was only figured out when someone happened to compare two flashing lights (one Xenon and one a bike light) that they figured out what was going on. Engineers miss things all the time, even those who are supposedly really good. They are good, but humans err, because to err is human. Prime example: I diagnosed an arcade cabinet that only had issues on sustained playtime during the summer, but never during the winter. It was fine on the off day that there was a lot of cloud cover, and only seemed to get worse after a shade awning was taken down: before the removal, it would intermittently sputter back to life during the later afternoon. Nobody knew what was going on. I spent an afternoon watching it fail. At one point, I eventually went and grabbed an IR thermometer and pointed it at the cabinet. It was registering well into 100F. It was also outright refusing to work. I rolled down the window blinds, pointed a fan at the poor machine, and waited for a while. Miraculously, it woke up after a while and started working. I traced it to the three canned voltage regulators. The voltage regulators for the system were strapped to the metal chassis of the CRT, which contained the system's power supply. The machine happened to be right next to a window, and the cabinet had been painted black by the previous owner. As the daytime sun would wander over, the black paint would soak up the UV and absolutely cook the regulators. Once the sun had passed behind the shade, it was quickly cool enough inside the chassis to bring the regulators back into spec, which caused the machine to start working again. When the shade was removed, it only went back up once the sun was no longer cooking it like a roast chicken. I believe my recommendation was "Replace the side paneling with woodgrain and move it out of the sun." The engineer that designed that arcade machine probably never thought it'd be in a hot, New Mexico university faculty lounge next to a window that faced the sun. I remember going back about a year later and sure enough, it had moved about three feet and was "rarely, if ever broken" now. Engineers miss things all the time.