5 ms·

Wow, somebody at Samsung is sure having an incredibly bad day... Flashing firmware to devices in the field is genuinely a really tough problem. Due to space, me

by bArray 6y ago

Wow, somebody at Samsung is sure having an incredibly bad day... Flashing firmware to devices in the field is genuinely a really tough problem. Due to space, memory and processing limitations you can end up backed into a corner as an engineer. It may be the case where there is genuinely no way to recover from this - turning a $10k remote software patch job into a several million dollar product recall. We used to have a simple rule where you always had two firmwares in storage, a primary image and secondary image. If you see tonnes of unplanned reboots recently you fallback to the secondary image. It can also be manually rolled back by holding some button during boot. (Other reasons to falling back could also be a failed hash check or bad signing for example.) To be clear, this method also had potential flaws and two bad software updates in a row could be bad (it tries to mark "good" firmwares, but it isn't foolproof). I've seen other methods also make use of a minimalist boot system that is just about enough to flash a new firmware. Stick a USB device in and hold a button during boot for example - if no button press you pay very little in startup time and the main application boots instead. You then * really * test this fallback firmware to make sure it will be very reliable in the field from cold boot. This gets even more troubling when OTA (over the air) methods are more widely employed too, where there may not be any exposed programming port on the device. I remember seeing somebody brick a few hundred devices via bluetooth before somebody said "have you tested that firmware works?" - turns out it was a bad unchecked-in local build...

MithrilTuxedo 6y ago
We had to get Samsung to update their firmware to stop the 6,000 TVs we installed on a cruise ship from saturating the physically isolated 10GbE network with attempts to check for firmware updates.
- slenk 6y ago
  1 Samsung TV causes an obscene amount of network calls, but 6000...wow
- dehrmann 6y ago
  I wonder if there are enough IoT devices with sloppy firmware that network operators see periodic new connection bursts.
dehrmann 6y ago
> We used to have a simple rule where you always had two firmwares in storage, a primary image and secondary image If you've got a 64MB image, that could be 50 cents per unit. That works out to a non-trivial drop in margins, and Blu-ray players have gotten commoditized enough that you might not be able to afford that. And I guarantee you it's not just this corner. Lots of corners were cut. I assume, anyway. I don't actually know for this model. The funny thing is I'd happily pay $10-20 more for a Blu-ray player that had less cost-centric engineering, the problem is consumers have no way to easily see this.
- userbinator 6y ago
  the problem is consumers have no way to easily see this. I remember seeing several motherboards advertising "dual BIOS" quite prominently, although the customer base there would understand the advantages of the feature far more than if something similar was advertised for a Blu-ray player.
- elcritch 6y ago
  But could be cheaper for Samsung than recalling all of the units, no? For a brand image the extra cost could be worth it as insurance.
  - bArray 6y ago
    FYI these companies will often set aside some % of yearly revenue for exactly this purpose.
- bArray 6y ago
  > If you've got a 64MB image, that could be 50 cents per > unit. I remember at times, even 1 cent savings was "rewarded". It really can get cut throat on cost savings. Luckily some of the things I worked on were automotive, so the potential cost of recall really was a consideration that we could hold over the heads of management, otherwise you would really struggle to make the case. > And I guarantee you it's not just this corner. Lots of > corners were cut. I think very large books could be filled with engineering stories about corner cutting. This is the sort of thing that wakes me up in the middle of the night in a cold sweat. > problem is consumers have no way to easily see this. This is why I end up buying slightly older hardware/software that has had some real user testing, but of course this doesn't always work.
deleted 6y ago
[deleted]

by bArray 6y ago

MithrilTuxedo 6y ago

We had to get Samsung to update their firmware to stop the 6,000 TVs we installed on a cruise ship from saturating the physically isolated 10GbE network with attempts to check for firmware updates.

slenk 6y ago
1 Samsung TV causes an obscene amount of network calls, but 6000...wow
dehrmann 6y ago
I wonder if there are enough IoT devices with sloppy firmware that network operators see periodic new connection bursts.

dehrmann 6y ago

> We used to have a simple rule where you always had two firmwares in storage, a primary image and secondary image If you've got a 64MB image, that could be 50 cents per unit. That works out to a non-trivial drop in margins, and Blu-ray players have gotten commoditized enough that you might not be able to afford that. And I guarantee you it's not just this corner. Lots of corners were cut. I assume, anyway. I don't actually know for this model. The funny thing is I'd happily pay $10-20 more for a Blu-ray player that had less cost-centric engineering, the problem is consumers have no way to easily see this.

userbinator 6y ago
the problem is consumers have no way to easily see this. I remember seeing several motherboards advertising "dual BIOS" quite prominently, although the customer base there would understand the advantages of the feature far more than if something similar was advertised for a Blu-ray player.
elcritch 6y ago
But could be cheaper for Samsung than recalling all of the units, no? For a brand image the extra cost could be worth it as insurance.
- bArray 6y ago
  FYI these companies will often set aside some % of yearly revenue for exactly this purpose.
bArray 6y ago
> If you've got a 64MB image, that could be 50 cents per > unit. I remember at times, even 1 cent savings was "rewarded". It really can get cut throat on cost savings. Luckily some of the things I worked on were automotive, so the potential cost of recall really was a consideration that we could hold over the heads of management, otherwise you would really struggle to make the case. > And I guarantee you it's not just this corner. Lots of > corners were cut. I think very large books could be filled with engineering stories about corner cutting. This is the sort of thing that wakes me up in the middle of the night in a cold sweat. > problem is consumers have no way to easily see this. This is why I end up buying slightly older hardware/software that has had some real user testing, but of course this doesn't always work.

deleted 6y ago

[deleted]