5 ms·

I’m gonna snark on this one. There is no mystery or surprise here. It’s basic functional qualification. You buy or make a thermal chamber and cycle release ver

by svnt 4y ago

I’m gonna snark on this one. There is no mystery or surprise here. It’s basic functional qualification. You buy or make a thermal chamber and cycle release versions of your device before you ship one. This isn’t some uncatchable mystery, they just didn’t test adequately. Depending on the product size and cost you may also do this to every individual robot off the line. This is not uncommon. This isn’t “hardware is hard” this is “we thought it was software with screwdrivers.”

iancmceachern 4y ago
Agreed. In many more regulated industries (medical devices, automotive, aerospace) the kind of testing you mention is required by law. In all cases it's good form, good practice to test your product to make sure it works as promised by its labeling. Typically you put an operating temperature and humidity range in your manual or labeling. In many industries testing to those operating parameters is required by law.
- greenbit 4y ago
  "environmental qualification testing", good old EQT.
KennyBlanken 4y ago
Or even just drive down to Lake Tahoe and find a winter construction site and try it there. This is a general problem with all these California-based companies and inventors (especially SV inventors cranking out crowd-funded bike stuff.) They seem blissfully unaware of things like cold weather, water, dirt/mud, and road salt...or combinations of them. I laugh at all those stupid fucking delivery bots because they'll fall apart anywhere there's snow, and get completely stuck on the slightest bit of ice. For many years, driving a Model S in heavy rain would cause water to get into the drive unit via either seals or vents that weren't sufficiently designed to keep water out. It "totals" the drive unit, causing corrosion of the motor control boards. And Tesla denies warranty claims on such repairs, because of course they do - just like they did on the windows that randomly shattered in parked cars. Raise your hand if you've owned a car that had problems with water ingress issues affecting its transmission. Or windows randomly shattering. What's that? Nobody? Exactly.
- trasz2 4y ago
- lostlogin 4y ago
  It is not just SV startups. I have owned a couple of Philips bread makers. Basic ones and expensive ones. If you make sourdough with them (ie bread) the coating is stripped off the bowl and the stirrer corrodes. They will deny replacement and claim you sprayed something acid on it. Yes, fermentation is acid, but they don’t believe bread would damage their unit.
  - deleted 4y ago
    [deleted]
TooSmugToFail 4y ago
You are 100% right, but I would not be as dismissive to the engineering team. When you run a hardware startup, you can only hope for an experienced team that would do everything by the book and implement best practices from the very first production unit. Reality is: that’s a luxury for most hardware startup teams out there. Typically, there’s a frantic rush to get your device to market that you simply skip, or more likely don’t even have time to think about stuff like climate chamber cycling. One thing I’m almost sure of: these guys have learned something — the engineer’s way. Good chance there’s a guy there googling climate chambers to ask the CEO for a budget to buy one.
- toss1 4y ago
  >>Typically, there’s a frantic rush to get your device to market that you simply skip, or more likely don’t even have time to think about stuff like climate chamber cycling. And that, right there, is the difference between a company oriented around myopic management vs a company oriented around robust quality. Any company trying to build a quality reputation would spec this stuff out AT THE BEGINNING — what are the operational requirements, what loads will they see, in what environments will they run, etc??? Then spec every component, and test the whole lot against those requirements. Sure, this is more like the dreaded "waterfall" vs "agile", but the result is a quality product from the start that has far fewer of these problems (because they did this whole test & fix routine at the prototype or Alpha test stages), rather than showing up with stories like this of how they recovered from customer-reported problems. If you're telling your customers that they're the Alpha testers because they get early access, fine. If you're selling it as a finished product, then we know your company isn't prioritizing actual quality.
  - mschuster91 4y ago
    > If you're telling your customers that they're the Alpha testers because they get early access, fine. If you're selling it as a finished product, then we know your company isn't prioritizing actual quality. Sounds like Tesla... although it is public knowledge there that you are still alpha testers years after a model was introduced.
- 4y ago
SkyPuncher 4y ago
I'm arm-chairing a bit. This is the type of problem I'd expect on a prototype, but not on a production level device - especially on a construction robot that will be clearly out in the weather. It reads to me like they didn't properly spec/source components that were appropriate for the weather conditions these robots are likely to see. The fact that this issue was reproducible at refrigerator temperatures is even more shocking. 39F (taken from a photo) is not very cold.
- pitaj 4y ago
  > The fact that this issue was reproducible at refrigerator temperatures is even more shocking. 39F (taken from a photo) is not very cold. Yeah that's pretty inexcusable. As a comparison, common "industrial" semiconductors are often specified from -40C to 125C.
  - exmadscientist 4y ago
    Speaking very broadly, because it's not all standardized and many companies have their own definitions, in degrees C: Range Min Max Commercial 0 to + 70 Industrial 0 to + 85 Automotive -40 to +105 Extended Automotive -40 to +125 Military -55 to +125 But that's for the parts themselves, and just working at that. Weird things can and do happen at temperature extremes. If you're very lucky, they're documented in datasheets/manuals. If you're lucky, they're known enough to be in white papers or industry presentations, or known to one of the companies that do this stuff for a living. If not... well, hope someone's heard of it, or you've got a huge testing budget....
Aloha 4y ago
Based on the pictures of the robots, it looks like they were intended for indoor use, I guess no one planned for an indoor space outside of 50-90 degrees - which is a pretty reasonable supposition, better than 80% of indoor environments are within that range.
bsder 4y ago
> You buy or make a thermal chamber and cycle release versions of your device before you ship one. That is a total waste of time in a startup with a small number of units shipped. A component behaving out of spec due to temperature excursions simply isn't that common nowadays. If my system is mostly ADC to digital to DAC (standard for robotics controllers), testing for temperature is a waste until I'm shipping significant volumes. There is a video of one of the slightly famous YouTubers who has a high voltage thing that fails at the altitude of his lab. The manufacturer did check it for function at the elevation of Denver, but his lab is higher than that. There are limits to how much engineering effort you should put in until you get an actual failure. (Maybe someone can link the video as I can't remember it at this point.) You can waste infinite engineering effort covering all possibilities. Or you can ship the thing and fix the failures. "Good engineering" is about balancing the two--you need to ship, but you don't want to have too many failures in the field either.
- snovv_crash 4y ago
  No, often components are dependent on the stress they are under. If you have inconsistent reflow, or lots of rework happening, then each device will behave differently due to thermal contractions.
- mdorazio 4y ago
  Testing thermal performance is as simple as going to a restaurant down the street and asking if you can pay them $100 to borrow their walk-in fridge for an hour for cold, and leaving your device in a hot car for an hour for hot. It's also exactly the kind of thing you should be doing if you're selling an actual product to a customer instead of partnering with someone to test your prototype.
- justin66 4y ago
  If you specify an operating temperature range for your gadget, some testing at the extremes of that range is called for.
- negative_zero 4y ago
  Except that it isn't. They just proved it with their tweet. They wasted a ton of time and money on what they thought was the problem and what sounds like essentially a product recall. A $5k temperature chamber running over night or weekend would have caught this.
negative_zero 4y ago
Completely agree and will continue snarking. Chest freezers, dry ice, pie warmers are cheap. The fridge+freezer and oven in your company kitchen are free. HALT/HAS, 4-corners, thermal cycling and thermal shock testing are all things. They wasted resources on fixing what they "thought" was the problem. That money would have paid for some nice chambers and other equipment. Next thread: "We learned our robot doesn't work when sitting in a shipping container at -40C for three months because customs was being difficult." Thread after: "Hardware is hard! We learnt about EMC compliance."
not_the_fda 4y ago
The more I think about this the worse it gets. Trying to give the benefit of a doubt, but its really incompetency, and they decided to blog about it. If they didn't account for temperature, they certainly didn't account for moisture, and these boards probably don't have any conformal coating on them. I guess its good for the customer that its a robots as a service, but these things are going to end up failing spectacularly in the field. These things are outside at construction sites, they needs to account for the environmental conditions.

by svnt 4y ago

iancmceachern 4y ago

Agreed. In many more regulated industries (medical devices, automotive, aerospace) the kind of testing you mention is required by law. In all cases it's good form, good practice to test your product to make sure it works as promised by its labeling. Typically you put an operating temperature and humidity range in your manual or labeling. In many industries testing to those operating parameters is required by law.

greenbit 4y ago
"environmental qualification testing", good old EQT.

KennyBlanken 4y ago

Or even just drive down to Lake Tahoe and find a winter construction site and try it there. This is a general problem with all these California-based companies and inventors (especially SV inventors cranking out crowd-funded bike stuff.) They seem blissfully unaware of things like cold weather, water, dirt/mud, and road salt...or combinations of them. I laugh at all those stupid fucking delivery bots because they'll fall apart anywhere there's snow, and get completely stuck on the slightest bit of ice. For many years, driving a Model S in heavy rain would cause water to get into the drive unit via either seals or vents that weren't sufficiently designed to keep water out. It "totals" the drive unit, causing corrosion of the motor control boards. And Tesla denies warranty claims on such repairs, because of course they do - just like they did on the windows that randomly shattered in parked cars. Raise your hand if you've owned a car that had problems with water ingress issues affecting its transmission. Or windows randomly shattering. What's that? Nobody? Exactly.

trasz2 4y ago
lostlogin 4y ago
It is not just SV startups. I have owned a couple of Philips bread makers. Basic ones and expensive ones. If you make sourdough with them (ie bread) the coating is stripped off the bowl and the stirrer corrodes. They will deny replacement and claim you sprayed something acid on it. Yes, fermentation is acid, but they don’t believe bread would damage their unit.
- deleted 4y ago
  [deleted]

TooSmugToFail 4y ago

You are 100% right, but I would not be as dismissive to the engineering team. When you run a hardware startup, you can only hope for an experienced team that would do everything by the book and implement best practices from the very first production unit. Reality is: that’s a luxury for most hardware startup teams out there. Typically, there’s a frantic rush to get your device to market that you simply skip, or more likely don’t even have time to think about stuff like climate chamber cycling. One thing I’m almost sure of: these guys have learned something — the engineer’s way. Good chance there’s a guy there googling climate chambers to ask the CEO for a budget to buy one.

toss1 4y ago
>>Typically, there’s a frantic rush to get your device to market that you simply skip, or more likely don’t even have time to think about stuff like climate chamber cycling. And that, right there, is the difference between a company oriented around myopic management vs a company oriented around robust quality. Any company trying to build a quality reputation would spec this stuff out AT THE BEGINNING — what are the operational requirements, what loads will they see, in what environments will they run, etc??? Then spec every component, and test the whole lot against those requirements. Sure, this is more like the dreaded "waterfall" vs "agile", but the result is a quality product from the start that has far fewer of these problems (because they did this whole test & fix routine at the prototype or Alpha test stages), rather than showing up with stories like this of how they recovered from customer-reported problems. If you're telling your customers that they're the Alpha testers because they get early access, fine. If you're selling it as a finished product, then we know your company isn't prioritizing actual quality.
- mschuster91 4y ago
  > If you're telling your customers that they're the Alpha testers because they get early access, fine. If you're selling it as a finished product, then we know your company isn't prioritizing actual quality. Sounds like Tesla... although it is public knowledge there that you are still alpha testers years after a model was introduced.
4y ago

SkyPuncher 4y ago

I'm arm-chairing a bit. This is the type of problem I'd expect on a prototype, but not on a production level device - especially on a construction robot that will be clearly out in the weather. It reads to me like they didn't properly spec/source components that were appropriate for the weather conditions these robots are likely to see. The fact that this issue was reproducible at refrigerator temperatures is even more shocking. 39F (taken from a photo) is not very cold.

pitaj 4y ago
> The fact that this issue was reproducible at refrigerator temperatures is even more shocking. 39F (taken from a photo) is not very cold. Yeah that's pretty inexcusable. As a comparison, common "industrial" semiconductors are often specified from -40C to 125C.
- exmadscientist 4y ago
  Speaking very broadly, because it's not all standardized and many companies have their own definitions, in degrees C: Range Min Max Commercial 0 to + 70 Industrial 0 to + 85 Automotive -40 to +105 Extended Automotive -40 to +125 Military -55 to +125 But that's for the parts themselves, and just working at that. Weird things can and do happen at temperature extremes. If you're very lucky, they're documented in datasheets/manuals. If you're lucky, they're known enough to be in white papers or industry presentations, or known to one of the companies that do this stuff for a living. If not... well, hope someone's heard of it, or you've got a huge testing budget....

Aloha 4y ago

Based on the pictures of the robots, it looks like they were intended for indoor use, I guess no one planned for an indoor space outside of 50-90 degrees - which is a pretty reasonable supposition, better than 80% of indoor environments are within that range.

bsder 4y ago

> You buy or make a thermal chamber and cycle release versions of your device before you ship one. That is a total waste of time in a startup with a small number of units shipped. A component behaving out of spec due to temperature excursions simply isn't that common nowadays. If my system is mostly ADC to digital to DAC (standard for robotics controllers), testing for temperature is a waste until I'm shipping significant volumes. There is a video of one of the slightly famous YouTubers who has a high voltage thing that fails at the altitude of his lab. The manufacturer did check it for function at the elevation of Denver, but his lab is higher than that. There are limits to how much engineering effort you should put in until you get an actual failure. (Maybe someone can link the video as I can't remember it at this point.) You can waste infinite engineering effort covering all possibilities. Or you can ship the thing and fix the failures. "Good engineering" is about balancing the two--you need to ship, but you don't want to have too many failures in the field either.

snovv_crash 4y ago
No, often components are dependent on the stress they are under. If you have inconsistent reflow, or lots of rework happening, then each device will behave differently due to thermal contractions.
mdorazio 4y ago
Testing thermal performance is as simple as going to a restaurant down the street and asking if you can pay them $100 to borrow their walk-in fridge for an hour for cold, and leaving your device in a hot car for an hour for hot. It's also exactly the kind of thing you should be doing if you're selling an actual product to a customer instead of partnering with someone to test your prototype.
justin66 4y ago
If you specify an operating temperature range for your gadget, some testing at the extremes of that range is called for.
negative_zero 4y ago
Except that it isn't. They just proved it with their tweet. They wasted a ton of time and money on what they thought was the problem and what sounds like essentially a product recall. A $5k temperature chamber running over night or weekend would have caught this.

negative_zero 4y ago

Completely agree and will continue snarking. Chest freezers, dry ice, pie warmers are cheap. The fridge+freezer and oven in your company kitchen are free. HALT/HAS, 4-corners, thermal cycling and thermal shock testing are all things. They wasted resources on fixing what they "thought" was the problem. That money would have paid for some nice chambers and other equipment. Next thread: "We learned our robot doesn't work when sitting in a shipping container at -40C for three months because customs was being difficult." Thread after: "Hardware is hard! We learnt about EMC compliance."

not_the_fda 4y ago

The more I think about this the worse it gets. Trying to give the benefit of a doubt, but its really incompetency, and they decided to blog about it. If they didn't account for temperature, they certainly didn't account for moisture, and these boards probably don't have any conformal coating on them. I guess its good for the customer that its a robots as a service, but these things are going to end up failing spectacularly in the field. These things are outside at construction sites, they needs to account for the environmental conditions.