8 ms·
I see four failures here on the part of tesla: 1) choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a
by walrus01 6y ago
I see four failures here on the part of tesla:
1) choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a larger amount of flash to spread the write endurance out over a period of up to 20, 25 years.
2) linux/operating system/software engineering implementation, for a certain volume of MB of writes per day, that would be known to wear out the flash write endurance over a short number of years. possibly a result of the software team not communicating properly with the hardware team. the software team very well may have been operating in a vacuum of assuming that they could do whatever they wanted within the CPU, I/O, RAM, and disk space limits of the architecture, and weren't even thinking at all of the consequences of their logging setup.
3) not making that a socketed/removable part.
4) asking the owners to pay for the replacement due to tesla's own screwup.
- TheSpiceIsLife 6y agoI'd argue you've put those points in the reverse priority. 1. Admit you screwed up and apologise 2. Moving forward: make repairability a priority and use durable components 3. Open source some of the software, or allow third parties to scrutinise it
- pound 6y agore 4) - this is what Tesla does a lot :( In Model3 you can move front passenger seat forward and up (to clean up in the back) all the way to the point when it pushes on sun visor and cracks the mirror in it. Just by using handles intended to move the seat, no manual pressure or anything, there are no limitation on how far it goes and it's strong enough.. Tesla service centers blame owners for that and charge full replacement cost for the visor :\
- jbob2000 6y agoHaha, if you think that's bad, try driving a jeep. You don't even have to drive it for it to break!
- TheSpiceIsLife 6y ago"There's only one Jeep" because all the others fell apart.
- camjohnson26 6y agoMy jeep has 60k miles and never any issues
- fl0wenol 6y agoEven as an anecdote, that's not impressive.
- camjohnson26 6y agoOk? Counter example to “breaks even if you don’t drive it”
- dylan604 6y agoIt is for a Jeep though
- SigmundA 6y agoMy Jeep Wrangler does not have any physical interference issues regarding seats breaking mirrors and the only issue I have had is the rear main seal started leaking and they happily replaced under warranty at 4 years and 6 months no questions asked.
- Foomf 6y agoThis is exactly my experience with the jeep my family had. However, there's a big caveat: the jeep engine itself was perfect. Over the 15 or so years we had it, that engine was solid. Everything else in the jeep failed at some point or another, and had nothing to do with driving. I have a few examples off the top of my head: The heated seats died after a few months. The insides of the car doors became detached because the glue wasn't strong enough. One by one the dashboard knobs all split apart at one point or another. We were a normal family, there were no crazy kids or pets in it that went nuts on the inside. Our jeep just kind of disintegrated around us, and none of the problems had to do with actually driving the jeep. So yes, we didn't even need to drive it for it to fall apart.
- walrus01 6y agoJust out of morbid curiosity, what does this visor cost to replace? I'm guessing $299
- pound 6y agoActually it wasn't too expensive - something like half of that (I don't remember exact number, but mid-100s). It was more of a feeling that I shouldn't be paying for it at all that displeased me :( (and reddit is full of stories when it happened with others (with the same service centers' experience))
- pound 6y agoOne more thing that has no solution now but I'm sure Tesla will not fix it at all or will charge a lot for it: White interior and black seatbelts. Despite Musks's promises how durable white interior is, seatbelts they use are dyed with I don't know what and they mar the seats. It's permanently baked in and cannot be removed.. Tesla responded long time ago that they are looking for solution, but whatever they will change will definitely not apply to any existing (screwed-up) white interior owners.
- adamcstephens 6y agoMusk really said the white interior is durable? Maybe they got a deal on white material and wanted to sucker people into buying it.
- pound 6y agoTbh, it actually is okay. After about two years most prominent marks we have on them are those left by seatbelts.
- buran77 6y agoOne thing people don't understand about cost saving is that it almost always looks good in the short term. You save a lot of money by using commercial grade parts (like eMMC, the dashboard screen, etc.), or by hiring SW and HW engineers who have never worked in the automotive or embedded environment. It's sacrificing quality for quantity and price. But it's usually the customers who pay for it all, one way or another. From Tesla's perspective the bet might have payed off, by the time this becomes an actual problem for them, they're already over the hump and weathered the worst of times, and can afford to spec them better (one would hope they learned something from this although I have my doubts). In reality probably the real reason this worked for Tesla is Musk's ability to build a fanatical fanbase that can drown out any other voice.
- pound 6y agoFunny thing is while I was okay with buying Tesla, it was Musk's personality attached to it that really bugged me.. So while there is a (quite loud) fanbase, there is also a group of potential Tesla owners who may go other route just due to desire to not be associated with "meme king"
- onethought 6y agoThe (actual) founders of Tesla are much more interesting and compelling than Elon in this regard
- buran77 6y agoThe idea was that there is a critical mass of people which are vocal enough to help Musk push Tesla's public image past any blunder. No other car manufacturer has this kind of clout.
- throwaway0a5e 6y ago>No other car manufacturer has this kind of clout. Toyota does. Though they did put out ~20yr of great cars (emphasis on cars) to get that.
- toddmorey 6y agoThat's really disappointing to hear. I've owned a Model S since before the Model 3 arrived and have always fretted that the level of service I received wasn't sustainable at scale. They AGGRESSIVELY attacked any problems and replaced parts, even when that's not what I asked for. It's kinda nuts. Just the things I can remember: - All 4 door handles were replaced with newer versions - I complained about a corner of stitching on the driver's seat and got an entirely new driver's seat - Glovebox cover replaced (to address latch) - Steering wheel controls replaced - Driver display unit replaced entirely - Sunroof replaced Granted, that's A LOT of parts to address, but their attitude was we'll do anything to make it right. Never once pushed back or questioned. Usually told me that they wanted to replace the part (rather than fix) because Tesla had engineered an improved version. And, to their credit, the new parts they put in seemed to have meaningful improvements and have worked really well.
- andykellr 6y agoAs a 2013 Model S owner and now a 2018 and 2021 Model 3 owner (Tesla only family), I agree 100%. I assumed there were a few reasons for the Model S service I received: 1. Small scale 2. Desire to keep early adopters happy 3. 2x the cost of the Model 3 However, my Model 3 service has been really poor. In my new 2021 the charge port bolt was loose and wouldn't charge. Known issue that took several days and many phone calls to get fixed while traveling. Eventually a service manager got involved and the fix took 5 minutes. Then they installed Homelink and forgot to plug the front sensors back in. Full self-driving was unavailable for 3 weeks until they returned to plug in the sensors. Also ordered snow tires from Tesla and the TPMS sensors don't work in my car (2021 uses BLE sensors). They're ordering new sensors and I will need to take my wheels to a tire store to have them installed. It's been less than 2 months since I took delivery. Love the car, love the company, an enthusiastic shareholder, but service needs to be fixed.
- camjohnson26 6y ago“Love the car, but” has become a Tesla quality issue meme at this point.
- tibbydudeza 6y agoThe software team probably never worked in an embedded automotive environment.
- walrus01 6y agoEver scarier they might have never worked in an embedded-anything environment.
- ChuckNorris89 6y agoThe sad truth is you might be right. At a former company we had wear leveling in the firmware of IoT devices that would cost under 20$/piece so the fact that a car lacks such a basic feature, is shocking and tells me all I need to know about the development practices at Tesla. As a firmware engineer with automotive background, having now seen how the "sausage" is made there makes me stay away from them for the safety of my own life. I know the rest of he car manufacturers aren't as innovative or as desirable but at least some I worked for have their firmware development quality processes down to a religion, and, as boring as that may be, it's the kind of mentality you want from something you put your family into every day.
- colejohnson66 6y agoIt did have wear leveling. They were just writing such an ungodly amount to it (more than the 8GB capacity) per day it defeated the purpose of wear leveling.
- eldaisfish 6y agowhich is beside the point. the point being that competent and experienced people would have seen this coming and devised a solution.
- MrMan 6y agoChevy bolt is a great car
- 6y ago
- NathanielK 6y ago> choosing that particular eMMC, likely not something specced for very high write endurance, or not speccing a larger amount of flash to spread the write endurance out over a period of up to 20, 25 years. The problem is that there isn't any wear leveling happening at all, so if you have an event that happens at every boot up and is logged, it wears the same block every time. > not making that a socketed/removable part. It's a very integrated module, I don't think socketed ICs have been a thing in automtive for a long time. edit: I'm wrong, daughter post has a nice run down If you're curious the very long forum thread is a lovely read.[1] You can see the chip levels itself, because just soldering in a bigger chip with the same code on it makes a big difference to many customers. https://teslamotorsclub.com/tmc/threads/consolidated-emmc-thread-mcu-repair-black-center-screen.181124/ https://teslamotorsclub.com/tmc/threads/consolidated-emmc-th...
- walrus01 6y agoI didn't mean literally socketed, but in 2012 things like msata interface cards (a predecessor of m.2) absolutely were a thing. Or implemented soldered on a custom small pcb to edge pin interface not very different in size and slot from a laptop sodimm.
- NathanielK 6y agoThat's pretty neat. Do you have any pictures? I have never seen that before from automotive OEMs.
- wmf 6y agoThe socketed version of eMMC is called SD; it's pretty popular.
- gorkish 6y ago> The problem is that there isn't any wear leveling happening at all, so if you have an event that happens at every boot up and is logged, it wears the same block every time. This isn't true. Very nearly all eMMC devices have hardware wear leveling. They aren't like MTD (raw flash) devices or very cheap USB/SD where you have to rely on a block remapping layer or special filesystem because there is no hardware wear leveling. But like any flash device, when they are completely full there are fewer free blocks for the wear leveling to operate. Tesla also may have had them grossly misconfigured, but I haven't seen any evidence that it was anything other than good-old-fashioned log churn on a very full filesystem. I also understand Google kind of had them over a barrel with policies related to the caching of map data and aerial imagery too: since they could not retain this for very long, caches were constantly invalidated and the same data was being redownloaded over and over. At this point, I don't even think they cache map imagery to storage anymore. Regardless of the technical details, it's clear that Tesla gave little consideration to this problem during design and did even further damage by wasting their small remaining space by filling it up with Elon's glorious vision of fart sounds, fireplace videos, and games -- Which are great fun, not gonna lie, but in hindsight a terrible decision that greatly exacerbated this problem. I'm not a professional embedded systems person by any stretch, but I was watching out for this problem even in my hobby projects back in the 90's which used early flash memory products like DiskOnChip modules. To this day, I still buy SSD's that are a minimum of twice as large as I think I will need for every application? For things like all-flash SAN I double it yet again. Anyone who would like to understand why can look at my fat stacks of dead Intel SSDs that all died basically because they filled up. My point is that this sort of thing is really well known and there must have been hundreds of times people within Tesla had questioned this internally.
- szczys 6y agoI mean, on your first point, it's an automotive grade part. I think any eMMC put under these conditions (75% full and the other 25% written heavily to with log files) is going to have a relatively short life expectancy. What baffles me is that who is reading these logs? Can't they just turn then off (pipe to /dev/null) and then have cars that need troubleshooting put into a debug mode at some point?
- kace91 6y agoI'm assuming they're some sort of airplane black box?
- amluto 6y agoThe OS was stock Ubuntu, I believe. It was not intended for this purpose.
- netsharc 6y agoI think even stock Ubuntu it's not that hard (famous last words) to disable/redirect logs to /dev/null. It would take a competent engineer probably 1 day to configure it, even if they need to read Stack Overflow. They either didn't have the time, or they didn't even know it would be a problem.
- rmtech 6y agoThey should have just paid for an eMMC that was 4× larger than needed. Even I wouldn't put an SSD under this kind of stress and I'm just a random developer.
- usr1106 6y ago> (75% full and the other 25% written heavily to with log files) Does an eMMC even know how full it is? If it were designed for write once it could certainly track that, but I doubt they are designed for such little usage. After having every block written once they would need TRIM to know what is in use, but to my knowledge eMMCs don't support that. (Not an expert in the area, just a random software developer.)
- toddmorey 6y agoAlways thought of this as a fascinating race to watch. What will happen first: (1) Tesla figuring out how to really build & manufacture cars or (2) the legacy automotive companies really figuring out build & deploy software? (It's not that black and white, all automotive companies have varying degrees of expertise in both software engineering and mechanical engineering. But it's certainly interesting to watch the industries learn from each other.)
- CharlesW 6y ago> Always thought of this as a fascinating race to watch. What will happen first: (1) Tesla figuring out how to really build & manufacture cars or (2) the legacy automotive companies really figuring out build & deploy software? Yes! And (3) is potentially: A company who's even better at building and deploying software and hardware enters the market.
- CarelessExpert 6y agoYou forgot the much more cynical (4) Tesla lowers the bar for success and quality diminishes across the industry in a race-to-the-bottom on price. ;)
- kcb 6y agoNothing that Tesla has built is anywhere near the bottom of the industry though.
- semi-extrinsic 6y agoA Dacia Sandero Stepway (5 seater small SUV/crossover) costs € 13 000 brand new. How much is the cheapest Tesla again?
- catmanjan 6y agoApples to oranges if it's not electric
- kwhitefoot 6y ago> 4) asking the owners to pay for the replacement due to tesla's own screwup. I had an email from Tesla yesterday about this recall and as far as I can see I will not be expected to pay anything. It even says that if someone had the part replaced at their own cost that they might be able to get a refund. I live in Norway and have a 2015 S 70D. The email says: "Tesla har besluttet å frivillig og proaktivt tilbakekalle enkelte Model S og Model X som er bygget før mars 2018 og som er utstyrt med et 8 GB innebygd MultiMediaCard (eMMC) i MCU som kan oppleve en funksjonsfeil på grunn av akkumulert slitasje. ... Hvis du allerede har betalt for reparasjoner som er tilknyttet til denne tilbakekallingen, kan du være kvalifisert for en refusjon." Google Translate does a good job on it: "Tesla has decided to voluntarily and proactively recall some Model S and Model X that were built before March 2018 and that are equipped with an 8 GB built-in MultiMediaCard (eMMC) in the MCU that may experience a malfunction due to accumulated wear. ... If you have already paid for repairs associated with this recall, you may be eligible for a refund." Hackaday's 'article' seems to be conflating what Tesla wanted to happen (and what they actually did before) with what is happening now. Or is Tesla in the US expecting owners to pay now as well as before?
- coolspot 6y agoYou expected to pay yourself if both: 1) Your car is out of warranty. 2) Your car’s VIN is not listed for the recall.
- kwhitefoot 6y agoThank you. I wonder how Tesla decides which cars to recall.