4 ms·

Yep, same problem with eMMC on Tesla's Tegra board. Solution: mount /var/log and other such partitions as ramdisks :)

walrus01 6y ago
That's really scary and a poor choice that something which is probably buried deep in the car, and costs thousands of dollars to remove and replace, is a soldered onboard flash drive which will fail from constant use. I'd be surprised if tesla didn't have embedded systems engineering responsible for this thing, and indeed even flash memory experts on staff, who are intimately familiar with all of the design problems of flash memory write wear leveling. And they went ahead and send that design to production anyways?
- meekrohprocess 6y ago
  It is quite embarrassing for them; Flash write endurance is something that most new graduates are aware of. There's an NHTSA investigation into the incident, but on the bright side, Tesla did remediate it in newer models...by using chips with twice as much eMMC memory to push the failures farther out.
- heavyset_go 6y ago
  Compared to other luxury cars, Tesla's cars have a lot of issues and design problems you wouldn't normally see. I'm not surprised that those problems extend into the internals of their cars, as well.
- freerobby 6y ago
  I believe they avoid socketed chips due to vibration issues. Definitely should have upped the physical capacity to distribute the writes, though.
- NotSammyHagar 6y ago
  I have a 5 year old model S and I ran into this failure. My car was about 4.5 years old, it was about 6 months after the initial 4 year warranty expired. I'd struggled with deciding to get an extended warranty or not - of course now I insist everyone should get one. They very generously let me purchase an extended warranty at repair time and let me use that toward that repair. That was a really stupid design decision of course for Tesla to use the emmc in that way. I love my car but that was a bad day when the problem hit me. It's been a great car even with a few problems (had one of the door handles replaced). Mine will be 6 years old soon, I will probably wait till it's about 8 years old for replacing it. On the other hand, I have a friend with an 8 year old model S and he had no extended warranty and he hasn't had that problem.
castratikron 6y ago
Where do you save the log on a reboot or power loss?
- walrus01 6y ago
  I'm absolutely not an expert on this, but you ideally provide a sufficiently large amount of storage area, with a flash memory controller, that over the expected (20 year plus hopefully!) lifespan of the product, the amount of periodic daily writes can be evenly distributed, and will not exceed the individual cell write lifespan. Some people who have intentionally done torture test writes on consumer grade SSDs have discovered the actual cumulative TB writes that a $50 to $100 SSD will take before ultimate failure. From back in 2015: https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/ https://techreport.com/review/27909/the-ssd-endurance-experi... One drive failed at 700TB written. If you were to write 2GB per day in an industrial/embedded application nonstop for 20 years, that is considerably less data than 700TB. With teslas that have a persistent LTE data connection for the car, you also have the option of doing something like 300KB of file upload per day to a remote server.
  - a1369209993 6y ago
    > If you were to write 2GB per day in an industrial/embedded application nonstop for 20 years, that is considerably less data than 700TB. Specifically: $ units You have: 2GiB/day * 20yr You want: TiB * 14.267273 You have: 2GiB/day * 20yr You want: 700TiB / 49.063334 14TB total, or about one fiftieth of your quota.
- inetknght 6y ago
  Either you attach dedicated storage for it or you send the logs elsewhere using a network-mounted /var/log or using a syslog daemon on another host.
- freerobby 6y ago
  I have a reverse-ssh tunnel + script to upload the logs to my home server on demand. If the MCU crashes or reboots unexpectedly, I would lose those logs. Ironically, my system for downloading the logs writes them to an SD card on a raspberry pi.

That's really scary and a poor choice that something which is probably buried deep in the car, and costs thousands of dollars to remove and replace, is a soldered onboard flash drive which will fail from constant use. I'd be surprised if tesla didn't have embedded systems engineering responsible for this thing, and indeed even flash memory experts on staff, who are intimately familiar with all of the design problems of flash memory write wear leveling. And they went ahead and send that design to production anyways?

meekrohprocess 6y ago
It is quite embarrassing for them; Flash write endurance is something that most new graduates are aware of. There's an NHTSA investigation into the incident, but on the bright side, Tesla did remediate it in newer models...by using chips with twice as much eMMC memory to push the failures farther out.
heavyset_go 6y ago
Compared to other luxury cars, Tesla's cars have a lot of issues and design problems you wouldn't normally see. I'm not surprised that those problems extend into the internals of their cars, as well.
freerobby 6y ago
I believe they avoid socketed chips due to vibration issues. Definitely should have upped the physical capacity to distribute the writes, though.
NotSammyHagar 6y ago
I have a 5 year old model S and I ran into this failure. My car was about 4.5 years old, it was about 6 months after the initial 4 year warranty expired. I'd struggled with deciding to get an extended warranty or not - of course now I insist everyone should get one. They very generously let me purchase an extended warranty at repair time and let me use that toward that repair. That was a really stupid design decision of course for Tesla to use the emmc in that way. I love my car but that was a bad day when the problem hit me. It's been a great car even with a few problems (had one of the door handles replaced). Mine will be 6 years old soon, I will probably wait till it's about 8 years old for replacing it. On the other hand, I have a friend with an 8 year old model S and he had no extended warranty and he hasn't had that problem.

castratikron 6y ago

Where do you save the log on a reboot or power loss?

walrus01 6y ago
I'm absolutely not an expert on this, but you ideally provide a sufficiently large amount of storage area, with a flash memory controller, that over the expected (20 year plus hopefully!) lifespan of the product, the amount of periodic daily writes can be evenly distributed, and will not exceed the individual cell write lifespan. Some people who have intentionally done torture test writes on consumer grade SSDs have discovered the actual cumulative TB writes that a $50 to $100 SSD will take before ultimate failure. From back in 2015: https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/ https://techreport.com/review/27909/the-ssd-endurance-experi... One drive failed at 700TB written. If you were to write 2GB per day in an industrial/embedded application nonstop for 20 years, that is considerably less data than 700TB. With teslas that have a persistent LTE data connection for the car, you also have the option of doing something like 300KB of file upload per day to a remote server.
- a1369209993 6y ago
  > If you were to write 2GB per day in an industrial/embedded application nonstop for 20 years, that is considerably less data than 700TB. Specifically: $ units You have: 2GiB/day * 20yr You want: TiB * 14.267273 You have: 2GiB/day * 20yr You want: 700TiB / 49.063334 14TB total, or about one fiftieth of your quota.
inetknght 6y ago
Either you attach dedicated storage for it or you send the logs elsewhere using a network-mounted /var/log or using a syslog daemon on another host.
freerobby 6y ago
I have a reverse-ssh tunnel + script to upload the logs to my home server on demand. If the MCU crashes or reboots unexpectedly, I would lose those logs. Ironically, my system for downloading the logs writes them to an SD card on a raspberry pi.