4 ms·

I would like to add most of the IoT problem is no patches at all. The firmware they get is usually bog standard with some very minor tweaks out of china somewh

I would like to add most of the IoT problem is no patches at all. The firmware they get is usually bog standard with some very minor tweaks out of china somewhere. It is a problem of vendor locked in products where you have to buy a hub to do an update. If there even is an update. If you want to get a good picture of how sideways updating can even be watch the linus tech tips on where he wanted (and has the tech ability) to patch his light switches. But could not even get them to give him the correct firmware or even say if he could. Also many devices there is literally no way to even do the update. They flash it on the line and that is the last update it ever gets. Also Supported and actually maintained in the hardware world can mean different things. So you will need to get your definitions up front correct. Supported could mean to a HW manufacture if the thing burns out ship a new one. The firmware is a secondary consideration. Another aspect you will run into is licensing. I can sell a device but may not have access to the code. Example: The vendor who makes that code went EoL on their code 5 years ago. They will not even sell me the code as they may or may not even have anyone who works for them to give it away if they could. They may or may not want to sell me that software anymore as they have a new shiny they want to sell me. So I am stuck even though I want to update I can not do it. I had one vendor flat out refuse to give me the older docs because the item was EoL and they had a replacement product that cost like 5x. That was just to communicate with the thing. Not even to update it to a later revision.

MarcoPerazaFCC 3y ago
Re: the licensing issue, companies wanting to put a label on their product would probably want to extract similar guarantees up their supply chain. Especially with a voluntary program like the one the FCC is proposing, good practices won't become the norm across the market overnight. But maybe, at the very least, the segment of product and component makers that take security seriously will begin to grow. I encourage you to share your thoughts in an official comment.
- structural 3y ago
  As someone who designs IoT devices like these for a living, the device manufacturers here are in many cases the smallest companies in the supply chain and have very little ability to influence things upstream of them, especially for specialty products or companies entering a new market. It's often a major win to get a chipmaker to pick up the phone and sell us their product, much less receive any support at all. I wish I could put a label like this on all of my products and I've been wishing for this for over twenty years, but the reality on the ground is that our support ends when the support for the individual parts in our product ends. We've looked at our supply chain periodically to see if we can replace parts with better documented/supported comparable parts, but frequently there just really aren't any better options. This is a great idea in concept, but I fear that the flaw in the FCC's proposed rulemaking is that only indirectly addresses the root cause (the software, documentation, and support/updates provided by chipmakers for their parts). Furthermore, by focusing on device manufacturers who are the weaker partners in the chain, the regulation is likely to punish smaller, more innovative manufacturers.
  - happymellon 3y ago
    If it was forward looking, rather than retroactive then it would at least mean that chip manufacturers wouldn't be able to sell their undocumented/unsupported crap because all the buyers have to have it? If there are no buyers then their attitude should change.
    - structural 3y ago
      This is incorrect, because you're assuming that all the buyers have to have it, when the chip manufacturer is selling into many industries/markets. Since the specific "IoT device for the USA market" set of buyers is actually a small percentage of sales for most of the parts they sell, they really don't care to support their product from the IoT security perspective. This support is expensive, so it would very likely be cheaper for them to ignore the market completely.
      - sumtechguy 3y ago
        > This support is expensive Most of IoT is that way. We had sales cycles that were 2-3 years long and they would in the end buy 300 units. I then go back to my suppliers and say 'hey support these 500 ic's that you sold me for 10 years from right now' They would laugh me out of the room unless I am showing up with big bags of cash. That instantly makes the whole project unviable to sell/support.
        structural 3y ago
        Yes, absolutely. This is the exact conditions of most of our higher-end products (500-1000 units sold of a particular configuration is common). It's funny to get laughed out of the room even asking some chipmakers "can you sell us 1000 parts, please?"
        sumtechguy 3y ago
        It is tough to explain to people that 1000 is not even alot for some of these guys. 1000 parts at say a fun price of 20 each. That is Maybe a 20-25k sale at most. For some of these companies that is a rounding error. You get lower priced parts and they just do not care much. There is no margin in it for them. Especially if you are not coming back every few months.
com2kid 3y ago
> I would like to add most of the IoT problem is no patches at all. The firmware they get is usually bog standard with some very minor tweaks out of china somewhere. I was on a team that worked with a firmware vendor, from the US, for a bluetooth chip. We would send in bug reports, they'd send us firmware with fixes. Except it was obvious they did not use source control because they would sometimes base patches off of old firmware versions that had the bugs they had fixed in newer versions. It was absolutely insane having to send emails like "hi, your latest patch is based on firmware from a year ago, can you please instead fix the firmware you sent us last month?"
- mannyv 3y ago
  Sounds like broadcom to me.
  - com2kid 3y ago
    Hilariously, not that time. I do understand why you might think that though. :-D
- sumtechguy 3y ago
  Those sorts of places are fun to interview at. 'So what sort of source control do you use'. You would think everyone does that by this point. An easy slam dunk question to ask and for them to answer. I had one say 'well sometimes we check it into sourcesafe but usually just copy it around the 5 of us on a fileshare' (this was like 4-5 years ago).
slavik81 3y ago
> If you want to get a good picture of how sideways updating can even be watch the linus tech tips on where he wanted (and has the tech ability) to patch his light switches. But could not even get them to give him the correct firmware or even say if he could. It was a mess, but it may not be a good example because part of the confusion was that there was no newer firmware. Their firmware version was being reported in hexadecimal, but the latest firmware version was listed in decimal.
- sumtechguy 3y ago
  He had a mix of random ones. They were telling him to buy a hub and hope for the best or go thru one of their vendors (more cost). Even if it that was slightly wrong that exact example could very easily happen. You have a group of devices in random levels of firmware states with no real nice way to tell what is what.

MarcoPerazaFCC 3y ago

Re: the licensing issue, companies wanting to put a label on their product would probably want to extract similar guarantees up their supply chain. Especially with a voluntary program like the one the FCC is proposing, good practices won't become the norm across the market overnight. But maybe, at the very least, the segment of product and component makers that take security seriously will begin to grow. I encourage you to share your thoughts in an official comment.

structural 3y ago
As someone who designs IoT devices like these for a living, the device manufacturers here are in many cases the smallest companies in the supply chain and have very little ability to influence things upstream of them, especially for specialty products or companies entering a new market. It's often a major win to get a chipmaker to pick up the phone and sell us their product, much less receive any support at all. I wish I could put a label like this on all of my products and I've been wishing for this for over twenty years, but the reality on the ground is that our support ends when the support for the individual parts in our product ends. We've looked at our supply chain periodically to see if we can replace parts with better documented/supported comparable parts, but frequently there just really aren't any better options. This is a great idea in concept, but I fear that the flaw in the FCC's proposed rulemaking is that only indirectly addresses the root cause (the software, documentation, and support/updates provided by chipmakers for their parts). Furthermore, by focusing on device manufacturers who are the weaker partners in the chain, the regulation is likely to punish smaller, more innovative manufacturers.
- happymellon 3y ago
  If it was forward looking, rather than retroactive then it would at least mean that chip manufacturers wouldn't be able to sell their undocumented/unsupported crap because all the buyers have to have it? If there are no buyers then their attitude should change.
  - structural 3y ago
    This is incorrect, because you're assuming that all the buyers have to have it, when the chip manufacturer is selling into many industries/markets. Since the specific "IoT device for the USA market" set of buyers is actually a small percentage of sales for most of the parts they sell, they really don't care to support their product from the IoT security perspective. This support is expensive, so it would very likely be cheaper for them to ignore the market completely.
    - sumtechguy 3y ago
      > This support is expensive Most of IoT is that way. We had sales cycles that were 2-3 years long and they would in the end buy 300 units. I then go back to my suppliers and say 'hey support these 500 ic's that you sold me for 10 years from right now' They would laugh me out of the room unless I am showing up with big bags of cash. That instantly makes the whole project unviable to sell/support.
      - structural 3y ago
        Yes, absolutely. This is the exact conditions of most of our higher-end products (500-1000 units sold of a particular configuration is common). It's funny to get laughed out of the room even asking some chipmakers "can you sell us 1000 parts, please?"
        sumtechguy 3y ago
        It is tough to explain to people that 1000 is not even alot for some of these guys. 1000 parts at say a fun price of 20 each. That is Maybe a 20-25k sale at most. For some of these companies that is a rounding error. You get lower priced parts and they just do not care much. There is no margin in it for them. Especially if you are not coming back every few months.

com2kid 3y ago

> I would like to add most of the IoT problem is no patches at all. The firmware they get is usually bog standard with some very minor tweaks out of china somewhere. I was on a team that worked with a firmware vendor, from the US, for a bluetooth chip. We would send in bug reports, they'd send us firmware with fixes. Except it was obvious they did not use source control because they would sometimes base patches off of old firmware versions that had the bugs they had fixed in newer versions. It was absolutely insane having to send emails like "hi, your latest patch is based on firmware from a year ago, can you please instead fix the firmware you sent us last month?"

mannyv 3y ago
Sounds like broadcom to me.
- com2kid 3y ago
  Hilariously, not that time. I do understand why you might think that though. :-D
sumtechguy 3y ago
Those sorts of places are fun to interview at. 'So what sort of source control do you use'. You would think everyone does that by this point. An easy slam dunk question to ask and for them to answer. I had one say 'well sometimes we check it into sourcesafe but usually just copy it around the 5 of us on a fileshare' (this was like 4-5 years ago).

slavik81 3y ago

> If you want to get a good picture of how sideways updating can even be watch the linus tech tips on where he wanted (and has the tech ability) to patch his light switches. But could not even get them to give him the correct firmware or even say if he could. It was a mess, but it may not be a good example because part of the confusion was that there was no newer firmware. Their firmware version was being reported in hexadecimal, but the latest firmware version was listed in decimal.

sumtechguy 3y ago
He had a mix of random ones. They were telling him to buy a hub and hope for the best or go thru one of their vendors (more cost). Even if it that was slightly wrong that exact example could very easily happen. You have a group of devices in random levels of firmware states with no real nice way to tell what is what.