4 ms·

It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this c

It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car?

jschwartzi 7y ago
They probably used a reference design and then failed to perform the step of analyzing the design for weaknesses like this one. Or they did but decided it wasn't their problem.
- myself248 7y ago
  I assume the decision to leave datalogging turned on to this degree was not made by the same folks who decided to put datalogging and firmware in the same flash IC.
PinguTS 7y ago
In mass production every tenth of a penny pays. That is the thing in car mass production.
- taneq 7y ago
  Yeah but there's Toyota Corolla style mass production and Tesla Model S style mass production. In the former? Sure, save $5 here and there and it adds up. In the latter? Spend the money. Any small saving from corner-cutting will be far outweighed by the loss of reputation if your $100k car bricks itself for want of a $5 part.
  - HeWhoLurksLate 7y ago
    Maybe the Toyota Corolla isn't the best comparison?
    - smolder 7y ago
      Are you saying this because of the "stuck pedal" thing? Just pretend he means 90s Corollas then. They were properly robust in spite of cost minimization.
      - olyjohn 7y ago
        There was no "stuck pedal" thing. Not a single incident of "unintended acceleration" with Toyota was ever proven. Something like 70-80% of the incidents turned out to be intoxicated drivers, and the rest were driver error.
        bluGill 7y ago
        You can't exactly prove it though, the design meant that we can't say for sure it didn't happen. However there isn't enough information logged to say it did either.
        keithwinstein 7y ago
        You're right that the government did a big investigation and ultimately didn't find a major problem with the software/electronic throttle control system, but mistaken about the other causes of unintended acceleration. Ultimately Toyota recalled about 7 million cars (some twice). Dealers were installing the wrong floor mats, and these were trapping the accelerator pedal. This problem was well-documented as the cause of "unintended acceleration" incidents, including at least five deaths. Additionally, there was a lack of "brake override" system to idle the engine when the brake and accelerator were both applied (Toyota added one at the government's request), and some pedals manufactured by a supplier were well-documented as "sticky" (they recalled these cars and fixed this). See the NHTSA report, pages 4-10 (PDF pages 15-21): https://one.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pdf https://one.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pd... In summary, there were real "stuck pedal" problems with these cars that led to unintended acceleration and fatalities and, eventually, recalls and fixes. And, yes, some or many incidents were probably related to "pedal misapplication."
        7y ago
  - beerandt 7y ago
    And the $100s it will cost to send out a service guy.
- CalChris 7y ago
  In mass production every tenth of a penny pays if it pays. This is a cost benefit analysis and not just a cost analysis. If in the lifetime of the car, this part will fail then it will have to be replaced at great expense, $1,800 to $3,000. If that happens under warranty, that will cost Tesla a lot. If that happens out of warranty, that will cost the customer a lot and will cost Tesla a lot in customer good will. I sure hope Tesla has figured this out for 3 and the Y. (It has. According to the FA, this has to do with MCUv1 Tesla Model S and Model X units up to 2018.)
- vonmoltke 7y ago
  Depends on the definition of "mass". $0.001/unit doesn't pay when "mass production" is 500,000 units per year. That's a savings of $500/year.
  - perl4ever 7y ago
    I think Tesla is producing about half that rate currently.
    - vonmoltke 7y ago
      I was using their higher target rate for emphasis.
danesparza 7y ago
This twitter thread from last year might enlighten you: https://twitter.com/atomicthumbs/status/1032939617404645376 https://twitter.com/atomicthumbs/status/1032939617404645376
- pseudosavant 7y ago
  That thread is a trip.
- deleted 7y ago
  [deleted]
- microtherion 7y ago
  Discussed heavily on HN at the time: https://news.ycombinator.com/item?id=17835760 https://news.ycombinator.com/item?id=17835760
Slartie 7y ago
Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars. This is especially evident when looking at comparable cases of part failure, like the infamous touchscreen failures: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows-why-automotive-grade-matters https://www.thedrive.com/tech/27989/teslas-screen-saga-shows... You are absolutely right in that this failure should not have happened, because they should use some other medium to store their logs that is not getting cramped by increasingly bigger firmware versions and that is appropriately sized and engineered to withstand at least a century or two of heavy usage. But since cloud thermostat and media player box vendors also get away with just stuffing some embedded Linux distro on a single chip of the cheapest Flash they can find on the spot market, Tesla probably didn't put much thought into why they shouldn't just do the same. After all, this system is just a media player box mounted in a car, right? (Except that for some totally absurd reason, the car ceases to function entirely if its media system breaks, which is the next big WTF in this story...)
- perl4ever 7y ago
  "Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars." Who didn't assume this to be the case? Teslas are automotive iPhones, and I've always assumed they would be scrap in 3-5 years rather than 15-20. This is why I don't think it was ever valid to compare the purchase price to a regular car.
deleted 7y ago
[deleted]
com2kid 7y ago
> It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car? It is strange no one performed back of the envelope calculations on flash life vs logging size. I've worked in embedded, we had a time series based rolling log. Flash life calculation was done at regular intervals. IIRC with what we shipped, the flash chip would outlive the battery by 3x or 4x. And that was without wear leveling! We were smart enough to continue the rolling log at the last point on flash we uploaded from, avoiding the mistake I've seen some products do of starting over from the very "beginning" after every log sync. Do that and you end up with 10% of your flash worn down to nothing and 90% of your flash going unused. Also, unless needed, please stop using JSON. Use a binary format instead. Protobufs is trivial to get up and running almost anywhere. Other binary encoding formats may serve a product's needs better. JSON is fine if everything is low bandwidth web services talking now and then, or low friction HTTP APIs, but for other use cases... use something else. Worst I ever saw was XML being used to RPC invoke every single function call, even local ones. All params got packed up in XML, sent over to the destination, who'd then unpack the XML, execute, and package up the return value. A double digit % of the CPU was being used on XML serialization and de-serialization. (If your choice is that or JSON... probably go with JSON!) Logging flash being same as storage flash is super weird though. Though if their MCU only supports hanging a single flash chip off of it, I could see it happening.
- catalogia 7y ago
  Estimating log growth rate is troublesome. It might be easy enough to measure under normal operating conditions, but sometimes something breaks, maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate. Assuring yourself that under no circumstance is that possible can sometimes be tricky.
  - com2kid 7y ago
    > maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate. Sure, but sensor error reporting frequency should also be documented and calculated somewhere. Embedded requires more planning, this stuff isn't like AWS instances where the worst case is a larger bill. And from the sound of it just general usage is causing problems.
wyldfire 7y ago
FRU sizes are optimized to minimize warranty cost and must be balanced against things like design complexity, reliability and service procedure count/complexity. This was a no-brainer IMO: Tesla wanted their cars to be superior and time to market was key. Why do handwringing about the cars that might be still on the road in five or ten years when that's beyond the warranty and beyond Tesla's life in business if they don't deliver a Model S with the target specs? They made the right tradeoff. This is probably one of dozens of design weaknesses that Tesla cars have, ones that allowed them to get way ahead of the competition. Now that we have arrived at five or ten years later there is some nonzero risk to their brand. Do they have unreliable cars? How many Tesla owners still own the original car and how many of these were purchased second-hand? Folks who bought the original car and they encounter this failure mode might think about this expensive out-of-warranty repair when they go car shopping next. If there are a lot of them, Tesla could consider the financial tradeoff between extending the warranty to cover this failure mode and the impact on future sales.

jschwartzi 7y ago

They probably used a reference design and then failed to perform the step of analyzing the design for weaknesses like this one. Or they did but decided it wasn't their problem.

myself248 7y ago
I assume the decision to leave datalogging turned on to this degree was not made by the same folks who decided to put datalogging and firmware in the same flash IC.

PinguTS 7y ago

In mass production every tenth of a penny pays. That is the thing in car mass production.

taneq 7y ago
Yeah but there's Toyota Corolla style mass production and Tesla Model S style mass production. In the former? Sure, save $5 here and there and it adds up. In the latter? Spend the money. Any small saving from corner-cutting will be far outweighed by the loss of reputation if your $100k car bricks itself for want of a $5 part.
- HeWhoLurksLate 7y ago
  Maybe the Toyota Corolla isn't the best comparison?
  - smolder 7y ago
    Are you saying this because of the "stuck pedal" thing? Just pretend he means 90s Corollas then. They were properly robust in spite of cost minimization.
    - olyjohn 7y ago
      There was no "stuck pedal" thing. Not a single incident of "unintended acceleration" with Toyota was ever proven. Something like 70-80% of the incidents turned out to be intoxicated drivers, and the rest were driver error.
      - bluGill 7y ago
        You can't exactly prove it though, the design meant that we can't say for sure it didn't happen. However there isn't enough information logged to say it did either.
      - keithwinstein 7y ago
        You're right that the government did a big investigation and ultimately didn't find a major problem with the software/electronic throttle control system, but mistaken about the other causes of unintended acceleration. Ultimately Toyota recalled about 7 million cars (some twice). Dealers were installing the wrong floor mats, and these were trapping the accelerator pedal. This problem was well-documented as the cause of "unintended acceleration" incidents, including at least five deaths. Additionally, there was a lack of "brake override" system to idle the engine when the brake and accelerator were both applied (Toyota added one at the government's request), and some pedals manufactured by a supplier were well-documented as "sticky" (they recalled these cars and fixed this). See the NHTSA report, pages 4-10 (PDF pages 15-21): https://one.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pdf https://one.nhtsa.gov/staticfiles/nvs/pdf/NHTSA-UA_report.pd... In summary, there were real "stuck pedal" problems with these cars that led to unintended acceleration and fatalities and, eventually, recalls and fixes. And, yes, some or many incidents were probably related to "pedal misapplication."
        7y ago
- beerandt 7y ago
  And the $100s it will cost to send out a service guy.
CalChris 7y ago
In mass production every tenth of a penny pays if it pays. This is a cost benefit analysis and not just a cost analysis. If in the lifetime of the car, this part will fail then it will have to be replaced at great expense, $1,800 to $3,000. If that happens under warranty, that will cost Tesla a lot. If that happens out of warranty, that will cost the customer a lot and will cost Tesla a lot in customer good will. I sure hope Tesla has figured this out for 3 and the Y. (It has. According to the FA, this has to do with MCUv1 Tesla Model S and Model X units up to 2018.)
vonmoltke 7y ago
Depends on the definition of "mass". $0.001/unit doesn't pay when "mass production" is 500,000 units per year. That's a savings of $500/year.
- perl4ever 7y ago
  I think Tesla is producing about half that rate currently.
  - vonmoltke 7y ago
    I was using their higher target rate for emphasis.

danesparza 7y ago

This twitter thread from last year might enlighten you: https://twitter.com/atomicthumbs/status/1032939617404645376 https://twitter.com/atomicthumbs/status/1032939617404645376

pseudosavant 7y ago
That thread is a trip.
deleted 7y ago
[deleted]
microtherion 7y ago
Discussed heavily on HN at the time: https://news.ycombinator.com/item?id=17835760 https://news.ycombinator.com/item?id=17835760

Slartie 7y ago

Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars. This is especially evident when looking at comparable cases of part failure, like the infamous touchscreen failures: https://www.thedrive.com/tech/27989/teslas-screen-saga-shows-why-automotive-grade-matters https://www.thedrive.com/tech/27989/teslas-screen-saga-shows... You are absolutely right in that this failure should not have happened, because they should use some other medium to store their logs that is not getting cramped by increasingly bigger firmware versions and that is appropriately sized and engineered to withstand at least a century or two of heavy usage. But since cloud thermostat and media player box vendors also get away with just stuffing some embedded Linux distro on a single chip of the cheapest Flash they can find on the spot market, Tesla probably didn't put much thought into why they shouldn't just do the same. After all, this system is just a media player box mounted in a car, right? (Except that for some totally absurd reason, the car ceases to function entirely if its media system breaks, which is the next big WTF in this story...)

perl4ever 7y ago
"Even though it might sound absurd, Tesla actually applied a "cloud thermostat" approach to car engineering, by which I mean that they did not refrain from using basically consumer-grade equipment for some critical functions of their cars." Who didn't assume this to be the case? Teslas are automotive iPhones, and I've always assumed they would be scrap in 3-5 years rather than 15-20. This is why I don't think it was ever valid to compare the purchase price to a regular car.

deleted 7y ago

[deleted]

com2kid 7y ago

> It seems strange to me that data-logging flash wouldn't be a dedicated FRU module, instead of being in the same chip(s) as the actual firmware. I see how this could easily happen as a BOM-reducing or size-reducing measure for something like a cloud thermostat, but for a luxury car? It is strange no one performed back of the envelope calculations on flash life vs logging size. I've worked in embedded, we had a time series based rolling log. Flash life calculation was done at regular intervals. IIRC with what we shipped, the flash chip would outlive the battery by 3x or 4x. And that was without wear leveling! We were smart enough to continue the rolling log at the last point on flash we uploaded from, avoiding the mistake I've seen some products do of starting over from the very "beginning" after every log sync. Do that and you end up with 10% of your flash worn down to nothing and 90% of your flash going unused. Also, unless needed, please stop using JSON. Use a binary format instead. Protobufs is trivial to get up and running almost anywhere. Other binary encoding formats may serve a product's needs better. JSON is fine if everything is low bandwidth web services talking now and then, or low friction HTTP APIs, but for other use cases... use something else. Worst I ever saw was XML being used to RPC invoke every single function call, even local ones. All params got packed up in XML, sent over to the destination, who'd then unpack the XML, execute, and package up the return value. A double digit % of the CPU was being used on XML serialization and de-serialization. (If your choice is that or JSON... probably go with JSON!) Logging flash being same as storage flash is super weird though. Though if their MCU only supports hanging a single flash chip off of it, I could see it happening.

catalogia 7y ago
Estimating log growth rate is troublesome. It might be easy enough to measure under normal operating conditions, but sometimes something breaks, maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate. Assuring yourself that under no circumstance is that possible can sometimes be tricky.
- com2kid 7y ago
  > maybe a sensor goes bad, and all of a sudden you're spamming your logs at 10,000x the normal rate. Sure, but sensor error reporting frequency should also be documented and calculated somewhere. Embedded requires more planning, this stuff isn't like AWS instances where the worst case is a larger bill. And from the sound of it just general usage is causing problems.

wyldfire 7y ago

FRU sizes are optimized to minimize warranty cost and must be balanced against things like design complexity, reliability and service procedure count/complexity. This was a no-brainer IMO: Tesla wanted their cars to be superior and time to market was key. Why do handwringing about the cars that might be still on the road in five or ten years when that's beyond the warranty and beyond Tesla's life in business if they don't deliver a Model S with the target specs? They made the right tradeoff. This is probably one of dozens of design weaknesses that Tesla cars have, ones that allowed them to get way ahead of the competition. Now that we have arrived at five or ten years later there is some nonzero risk to their brand. Do they have unreliable cars? How many Tesla owners still own the original car and how many of these were purchased second-hand? Folks who bought the original car and they encounter this failure mode might think about this expensive out-of-warranty repair when they go car shopping next. If there are a lot of them, Tesla could consider the financial tradeoff between extending the warranty to cover this failure mode and the impact on future sales.