6 ms·

Proactively recall to replace with what? It's not like the regs and/or Tesla have fixed the discovered flaw in a new design, and will be replacing the old part

Proactively recall to replace with what? It's not like the regs and/or Tesla have fixed the discovered flaw in a new design, and will be replacing the old part with the newly designed part. In that case proactive recall would make sense. But that's not the case here. Here if you've only used 0.5 of 8Gb, and Tesla is required to make a recall, all Tesla is doing is replacing your old unit with the exact same model but now 8Gb free space instead of 7.5Gb free. Instead, if a new design is not developed, shouldn't it be treated like a recurring maintenance? like when the memory is 6 or 7Gb full, the user gets a warning to take it to the repair to have the unit replaced, or something. Also why isn't the memory emptied from time to time? old data should be of no use and should be backed up to Tesla cloud, or user's cloud storage outside the vehicle or something? In which case the whole exercise is moot.

iancarroll 6y ago
I think the article uses "capacity" a bit confusingly; the NAND chip is not at its storage limit, it is at its write cycle limit. Presumably there are NAND chips with much higher limits that they swapped it for, as this doesn't affect units after 2018 (according to the article).
reitzensteinm 6y ago
Surely Tesla has reduced the log write frequency in updates after this was discovered. My understanding is that the parts have decent endurance, it was just that a ton was being written there. If that's true, replacing the hardware is a long term fix. Edit: elsewhere in this thread points to 6000 cycle endurance, which means a best case of 42 TB lifetime writes, although write amplification will lower that. That's enough for 6gb of logs a day over a 20 year lifetime which seems plenty. Although that's also predicated on a controller that wear levels well.
- StillBored 6y ago
  The 3k writes in the article sound like a single page lifecycle number, and the computation indicates it fails after that many vehicle starts. Meaning they are either rewriting the entire flash on every start (8G) or it simply doesn't have a FTL. Which means its not even to the level of needing to worry about write amplification. Pretty rookie mistake there.
  - reitzensteinm 6y ago
    Ouch. Presumably the new hardware will be fine, though.
numpad0 6y ago
> why isn't the memory emptied from time to time? The problem isn’t lack of capacity, it’s that Flash storage(of all kinds, from microSD to NVMe SSD) bears only so many write operations. Reading is fine but reflashing takes away lifespan like what poisonous metals do to a person, they gets slower and slurred and eventually die. It’s not a problem had their software been actually tested and finalized before shipping so the cars don’t have to download Chromium patches every few weeks. Tesla being Tesla do that and break cars.
- yudlejoza 6y ago
  The memory system could be split in two: - One is the system memory. It'll have OS, apps, necessary data. This shouldn't change frequently. Like one update a month, or a few months. Should last a lifetime. - Second is the daily data memory. Logs, metrics, whatever. This would go bad more frequently, but the functioning of the car shouldn't depend on this part. If this stops working and user doesn't replace it, the rest of the car should work, except that no new logs will be generated. Or user is given the option to upload the logs straight to the cloud (is a Tesla vehicle always online?).
  - stefan_ 6y ago
    The typical design is to have redundant system partitions that are only ever mounted read-only and exist in a separate wear-leveling domain (through GPP partitioning). Application data is on its own partition and raw logs aren't written to flash storage at all, they go to a tmpfs in memory. Of course you will find few Linux distributions are set up like this, even if they would benefit tremendously (see Raspberry Pi SD card corruptions..). But your throwaway Android phone probably does it.
    - bostik 6y ago
      Funny how history repeats itself. Nokia had logbufd on their linux devices (N770/N800/...) to prevent NAND getting abused by constant writes. They realised very early on that writing logs during runtime is essential for debugging purposes but also that it would kill the internal flash in no time at all. With tmpfs backed log partition, you still need active log rotation. Logbufd was an in-memory ringbuffer that listened on the standard syslog socket, and if you spewed too much logs, you only killed your own ability to go back in time. If you fill up tmpfs before rotation gets a chance to rescue you, you will break all applications that don't handle out-of-disk-space gracefully.
creeble 6y ago
They’re replacing the chip (actually the daughterboard that hold the chip) with a much larger chip (32GB vs 8GB) and allocating half the chip for wear leveling. This is the same fix a southern Cal company has been performing for a few years, at car owner’s cost. They are not doing much in the way of preventive replacement, and their “predictive failure” notification is BS afaict. Maybe it’s new, but I’ve not yet seen it when my MCU starts acting up. It’s currently a $500 non-warranty service, but they won’t do it until the chip fails. Ask me how I know. Edit: mem sizes
- Goz3rr 6y ago
  > and their “predictive failure” notification is BS afaict. Maybe it’s new, but I’ve not yet seen it when my MCU starts acting up. If they are using eMMC, it should be trivial to just query the controller about the flash health which should be a pretty accurate indication of when it's going to fail.
- zaroth 6y ago
  The predictive notification is new as of 2020.40.9.2, which was released on November 24, 2020 after they extended the warranty. Just because it was release on 11/24 doesn’t mean all cars got it that day. Figure it can take a little while depending on how they roll these releases out to the fleet. https://tesletter.com/2020-40-9-2-update-emmc-failure/ https://tesletter.com/2020-40-9-2-update-emmc-failure/

iancarroll 6y ago

I think the article uses "capacity" a bit confusingly; the NAND chip is not at its storage limit, it is at its write cycle limit. Presumably there are NAND chips with much higher limits that they swapped it for, as this doesn't affect units after 2018 (according to the article).

reitzensteinm 6y ago

Surely Tesla has reduced the log write frequency in updates after this was discovered. My understanding is that the parts have decent endurance, it was just that a ton was being written there. If that's true, replacing the hardware is a long term fix. Edit: elsewhere in this thread points to 6000 cycle endurance, which means a best case of 42 TB lifetime writes, although write amplification will lower that. That's enough for 6gb of logs a day over a 20 year lifetime which seems plenty. Although that's also predicated on a controller that wear levels well.

StillBored 6y ago
The 3k writes in the article sound like a single page lifecycle number, and the computation indicates it fails after that many vehicle starts. Meaning they are either rewriting the entire flash on every start (8G) or it simply doesn't have a FTL. Which means its not even to the level of needing to worry about write amplification. Pretty rookie mistake there.
- reitzensteinm 6y ago
  Ouch. Presumably the new hardware will be fine, though.

numpad0 6y ago

> why isn't the memory emptied from time to time? The problem isn’t lack of capacity, it’s that Flash storage(of all kinds, from microSD to NVMe SSD) bears only so many write operations. Reading is fine but reflashing takes away lifespan like what poisonous metals do to a person, they gets slower and slurred and eventually die. It’s not a problem had their software been actually tested and finalized before shipping so the cars don’t have to download Chromium patches every few weeks. Tesla being Tesla do that and break cars.

yudlejoza 6y ago
The memory system could be split in two: - One is the system memory. It'll have OS, apps, necessary data. This shouldn't change frequently. Like one update a month, or a few months. Should last a lifetime. - Second is the daily data memory. Logs, metrics, whatever. This would go bad more frequently, but the functioning of the car shouldn't depend on this part. If this stops working and user doesn't replace it, the rest of the car should work, except that no new logs will be generated. Or user is given the option to upload the logs straight to the cloud (is a Tesla vehicle always online?).
- stefan_ 6y ago
  The typical design is to have redundant system partitions that are only ever mounted read-only and exist in a separate wear-leveling domain (through GPP partitioning). Application data is on its own partition and raw logs aren't written to flash storage at all, they go to a tmpfs in memory. Of course you will find few Linux distributions are set up like this, even if they would benefit tremendously (see Raspberry Pi SD card corruptions..). But your throwaway Android phone probably does it.
  - bostik 6y ago
    Funny how history repeats itself. Nokia had logbufd on their linux devices (N770/N800/...) to prevent NAND getting abused by constant writes. They realised very early on that writing logs during runtime is essential for debugging purposes but also that it would kill the internal flash in no time at all. With tmpfs backed log partition, you still need active log rotation. Logbufd was an in-memory ringbuffer that listened on the standard syslog socket, and if you spewed too much logs, you only killed your own ability to go back in time. If you fill up tmpfs before rotation gets a chance to rescue you, you will break all applications that don't handle out-of-disk-space gracefully.

creeble 6y ago

They’re replacing the chip (actually the daughterboard that hold the chip) with a much larger chip (32GB vs 8GB) and allocating half the chip for wear leveling. This is the same fix a southern Cal company has been performing for a few years, at car owner’s cost. They are not doing much in the way of preventive replacement, and their “predictive failure” notification is BS afaict. Maybe it’s new, but I’ve not yet seen it when my MCU starts acting up. It’s currently a $500 non-warranty service, but they won’t do it until the chip fails. Ask me how I know. Edit: mem sizes

Goz3rr 6y ago
> and their “predictive failure” notification is BS afaict. Maybe it’s new, but I’ve not yet seen it when my MCU starts acting up. If they are using eMMC, it should be trivial to just query the controller about the flash health which should be a pretty accurate indication of when it's going to fail.
zaroth 6y ago
The predictive notification is new as of 2020.40.9.2, which was released on November 24, 2020 after they extended the warranty. Just because it was release on 11/24 doesn’t mean all cars got it that day. Figure it can take a little while depending on how they roll these releases out to the fleet. https://tesletter.com/2020-40-9-2-update-emmc-failure/ https://tesletter.com/2020-40-9-2-update-emmc-failure/