6 ms·

This issue is best explained on Rich Rebuilds' recent video here: https://youtube.com/watch?v=o-7b1waoj9Q&t=525 https://youtube.com/watch?v=o-7b1waoj9Q&t=525 T

by BluSyn 7y ago

This issue is best explained on Rich Rebuilds' recent video here: https://youtube.com/watch?v=o-7b1waoj9Q&t=525 https://youtube.com/watch?v=o-7b1waoj9Q&t=525 This has been a known issue for some time, and it's amazing Tesla hasn't fixed it yet. As explained in the video, the issue is simply the base linux OS has full syslog's still enabled, which over time burn own the eMMC chip used for storage. Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards. The GOOD news is this issue is almost fully solvable through a software update. They just need to disable base syslogs, or at least only store logs in RAM while car is running (if they need those logs for debugging purposes, which I doubt). Long term, they should replace this eMMC with a proper SSD, or removable SD card that's easily serviced when it goes bad (as is the case in other parts of the car). I think this must be an issue of visibility in the company, and the right engineer just isn't aware of this issue yet. I hope this gets more visibility and gets patched in a future software update.

pfortuny 7y ago
It is a bit baffling that a company which enables “autopilot” OTA does not do this as well...
- nm2259 7y ago
  If this anonymous coward's characterization of Tesla is true and accurate, it's not surprising https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/former_tesla_programmers_anecdotes_about_problems/ https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...
  - arkades 7y ago
    Why did your choice of words include “anonymous coward”?
    - function_seven 7y ago
      On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in. The linked reddit post is from a user who claims to have worked for Tesla in the past, and is (presumably) not using their "real" reddit account to post it. I don't read it as an insult.
      - arkades 7y ago
        > On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in. Gotcha. That makes sense. Slashdot was never my thing. Thanks for clarifying; I had read it as bafflingly unjustified aggression.
  - village-idiot 7y ago
    The wording makes it sound like you're calling the parent comment an anonymous coward. I also suspect a lot of people didn't catch the Slashdot reference.
    - JaimeThompson 7y ago
      Maybe he was hit / smashed over the head with an OPEN SOURCE CD?
  - PhantomGremlin 7y ago
    Wow. Thanks for that link. Some amazing inside stuff. My favorite anecdote: most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do shit like add farting unicorns
semi-extrinsic 7y ago
Wow, this is exactly the tutorials that you get on "How to get your Rasberry Pi to run stable for extended periods of time". Exact same failure mode, exact same fix proposed.
castratikron 7y ago
Problems with an SD card is that it will wear out even faster than an eMMC due to lack of FTL, and also friction fit instead of soldered connections means the part is susceptible to wiggling out of the socket and also corrosion between the contactors. Not sure of the benefits of an SSD over an eMMC. Guessing the former will have a more sophisticated FTL and other things.
- JaimeThompson 7y ago
  Multiple companies make high endurance SD cards for things like dash cams and automobiles. Take, for example, WD https://www.westerndigital.com/products/embedded-removable-flash/automotive-sd-cards https://www.westerndigital.com/products/embedded-removable-f...
  - castratikron 7y ago
    Didn't know that, thanks. But there's still the problem of a friction fit connection in an environment with a lot of vibration and contaminants. Seems like these would be good for use cases where the part needs to be removable and also have high write cycles, like a dash cam or data logger that has no other interface to a PC. Also the eMMC interface exposes quite a bit of the underlying hardware (see mmc-utils on Linux). With SD cards I think you're generally stuck with some kind of SMART interface.
icelancer 7y ago
An SD card has a much different set of problems, including physical contact issues. eMMC is an acceptable tool for the job, it's just complete negligence to not use a batch logging technique if you even need full syslog data.
- rootusrootus 7y ago
  They use an SD card for the car logging, according to the Rich Rebuilds video, so it's doable. It just needs to not brick the car when it fails.
  - icelancer 7y ago
    That they use it in other applications doesn't make it automatically the best idea. SD cards are fairly unreliable compared to the other options, and as Rich's friend points out, smartphone manufacturers use eMMC and don't have issues because they understand wear cycles.
at-fates-hands 7y ago
> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards. Am I the only one thinking about the recycling and cost implications of doing this?
- silversconfused 7y ago
  Well, if you have some thoughts on it, I am curious what the other approach would look like. What are your thoughts? I'm thinking they would have all the labor of taking it out, and all the labor of putting it in, but then much additional labor and repair time incurred by doing the component level repair instead of just grabbing another radio off the shelf. Recycling would be possible on the unit later and it could be installed as a refurb unit in another vehicle.
rush340 7y ago
> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards. Do you know that they toss the old media units? Are you sure they don't swap the entire media unit to simplify service procedures, and have have the old units inspected and refurbished (i.e. eMMC module replaced) for future repairs? That sounds way more efficient than training all of the shop workers to diagnose and swap out individual parts of the media unit.
- deleted 7y ago
  [deleted]
joezydeco 7y ago
"Replacing a tiny part" is trivializing the work necessary to do this, in my opinion. eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip. You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without torching the rest of the board, missing a solder connection, or zapping any other component with static electricity. You can see a similar process done on an iPhone here: https://www.youtube.com/watch?v=y4M9uAZlbK4 https://www.youtube.com/watch?v=y4M9uAZlbK4 You can understand why Tesla would rather replace/refurbish the board than try to field-repair it.
- Crosseye_Jack 7y ago
  They have been replacing the whole media unit including the screen (and charging customers $2k-5k for the priv) where they could replace the processor module board (which is "just" a add-in board inside the media unit) keeping the rest of the media unit and lowering costs. Sure refurbish the processor module and replace the eMMC at a proper rework station but replacing the whole media unit esp when the failed part is on a removable board just seems like a huge waste to me esp when the customer is footing the bill.
- raverbashing 7y ago
  This is the right answer Replace the board, then you can refurbish it in a specialized lab Same reason why genius bars won't replace components, just boards
- solarkraft 7y ago
  Luckily Tesla isn't very space or money limited with this pretty ciritcal component. An SD card would make life easier for everyone, as the logs could be easily read out and the booting OS changed - yay for actually owning the device you paid for. But wait: That's not something Tesla wants.
  - joezydeco 7y ago
    eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s more mechanically stable and also way faster. In my experience, removable SD wouldn’t hold up to an application like this.
deleted 7y ago
[deleted]

by BluSyn 7y ago

pfortuny 7y ago

It is a bit baffling that a company which enables “autopilot” OTA does not do this as well...

nm2259 7y ago
If this anonymous coward's characterization of Tesla is true and accurate, it's not surprising https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/former_tesla_programmers_anecdotes_about_problems/ https://www.reddit.com/r/EnoughMuskSpam/comments/99sbwa/form...
- arkades 7y ago
  Why did your choice of words include “anonymous coward”?
  - function_seven 7y ago
    On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in. The linked reddit post is from a user who claims to have worked for Tesla in the past, and is (presumably) not using their "real" reddit account to post it. I don't read it as an insult.
    - arkades 7y ago
      > On Slashdot, "anonymous coward" was the default username for anyone posting without being logged in. Gotcha. That makes sense. Slashdot was never my thing. Thanks for clarifying; I had read it as bafflingly unjustified aggression.
- village-idiot 7y ago
  The wording makes it sound like you're calling the parent comment an anonymous coward. I also suspect a lot of people didn't catch the Slashdot reference.
  - JaimeThompson 7y ago
    Maybe he was hit / smashed over the head with an OPEN SOURCE CD?
- PhantomGremlin 7y ago
  Wow. Thanks for that link. Some amazing inside stuff. My favorite anecdote: most of the time me and the other firmware folks were chasing elon's whims about what to do with firmware. where i should have been fixing critical issues in the system i was pulled off to do shit like add farting unicorns

semi-extrinsic 7y ago

Wow, this is exactly the tutorials that you get on "How to get your Rasberry Pi to run stable for extended periods of time". Exact same failure mode, exact same fix proposed.

castratikron 7y ago

Problems with an SD card is that it will wear out even faster than an eMMC due to lack of FTL, and also friction fit instead of soldered connections means the part is susceptible to wiggling out of the socket and also corrosion between the contactors. Not sure of the benefits of an SSD over an eMMC. Guessing the former will have a more sophisticated FTL and other things.

JaimeThompson 7y ago
Multiple companies make high endurance SD cards for things like dash cams and automobiles. Take, for example, WD https://www.westerndigital.com/products/embedded-removable-flash/automotive-sd-cards https://www.westerndigital.com/products/embedded-removable-f...
- castratikron 7y ago
  Didn't know that, thanks. But there's still the problem of a friction fit connection in an environment with a lot of vibration and contaminants. Seems like these would be good for use cases where the part needs to be removable and also have high write cycles, like a dash cam or data logger that has no other interface to a PC. Also the eMMC interface exposes quite a bit of the underlying hardware (see mmc-utils on Linux). With SD cards I think you're generally stuck with some kind of SMART interface.

icelancer 7y ago

An SD card has a much different set of problems, including physical contact issues. eMMC is an acceptable tool for the job, it's just complete negligence to not use a batch logging technique if you even need full syslog data.

rootusrootus 7y ago
They use an SD card for the car logging, according to the Rich Rebuilds video, so it's doable. It just needs to not brick the car when it fails.
- icelancer 7y ago
  That they use it in other applications doesn't make it automatically the best idea. SD cards are fairly unreliable compared to the other options, and as Rich's friend points out, smartphone manufacturers use eMMC and don't have issues because they understand wear cycles.

at-fates-hands 7y ago

silversconfused 7y ago
Well, if you have some thoughts on it, I am curious what the other approach would look like. What are your thoughts? I'm thinking they would have all the labor of taking it out, and all the labor of putting it in, but then much additional labor and repair time incurred by doing the component level repair instead of just grabbing another radio off the shelf. Recycling would be possible on the unit later and it could be installed as a refurb unit in another vehicle.

rush340 7y ago

> Currently Tesla just replaces the entire media unit when this happens, which is extremely expensive for them. In reality they only need to replace a tiny part instead of tossing entire boards. Do you know that they toss the old media units? Are you sure they don't swap the entire media unit to simplify service procedures, and have have the old units inspected and refurbished (i.e. eMMC module replaced) for future repairs? That sounds way more efficient than training all of the shop workers to diagnose and swap out individual parts of the media unit.

deleted 7y ago
[deleted]

joezydeco 7y ago

"Replacing a tiny part" is trivializing the work necessary to do this, in my opinion. eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip. You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without torching the rest of the board, missing a solder connection, or zapping any other component with static electricity. You can see a similar process done on an iPhone here: https://www.youtube.com/watch?v=y4M9uAZlbK4 https://www.youtube.com/watch?v=y4M9uAZlbK4 You can understand why Tesla would rather replace/refurbish the board than try to field-repair it.

Crosseye_Jack 7y ago
They have been replacing the whole media unit including the screen (and charging customers $2k-5k for the priv) where they could replace the processor module board (which is "just" a add-in board inside the media unit) keeping the rest of the media unit and lowering costs. Sure refurbish the processor module and replace the eMMC at a proper rework station but replacing the whole media unit esp when the failed part is on a removable board just seems like a huge waste to me esp when the customer is footing the bill.
raverbashing 7y ago
This is the right answer Replace the board, then you can refurbish it in a specialized lab Same reason why genius bars won't replace components, just boards
solarkraft 7y ago
Luckily Tesla isn't very space or money limited with this pretty ciritcal component. An SD card would make life easier for everyone, as the logs could be easily read out and the booting OS changed - yay for actually owning the device you paid for. But wait: That's not something Tesla wants.
- joezydeco 7y ago
  eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s more mechanically stable and also way faster. In my experience, removable SD wouldn’t hold up to an application like this.

deleted 7y ago

[deleted]