7 ms·

Toyota to restart Japan production on Wednesday after system failure

74 points by KnuthIsGod 3y ago

rtpg 3y ago
One could only speculate on the cause here without more info. I am reminded of reading The Toyota Way a while back, which does talk about how they would shut down lines for any defect. A coworker of mine was pretty adamant that this was A Good Idea(TM), but I thought about how much of IT best practices are about uptime and doing (often principled!) workarounds when you have operational issues, and moving forward in a degraded state. I wonder if there are IT operations that do try to aim for the ~no defect approach like this.
- housemusicfan 3y ago
  Toyota sounds like the kind of place that still runs Solaris on SPARC and has no immediate plans to change.
  - r00fus 3y ago
    You jest but back in the day (90s) one of my friends said they still ran Smalltalk VMs in production. That was 25 years ago but still possible…
    - deleted 3y ago
      [deleted]
  - kenhwang 3y ago
    From my conversations with some Toyota software engineers, they have a respectably sophisticated modern software/tech stack. There's a ton of C++ since they have to interface with hardware. For tooling and internal applications it's a lot of Rails/Elixir/Javascript. Web APIs are backed by Elixir/Go. Data processing in Scala/Spark. Infrastructure in AWS, orchestrated with k8s. Truly agile prototyping then polish, significant automated testing. Honestly, it makes "big tech" look like dinosaurs with their megasized-Java projects and 3:1 PM:engineer ratio waterfall planning once a year releases. What they're missing is the operational experience with operating the software at scale.
- upon_drumhead 3y ago
  I’m sure finance folks are quick to hit the big red stop button if it’s causing invalid transactions to clear. But generally the impact to the end user doesn’t really matter. There’s no reason to shut it all down if 1% of your request to a web server gets a 500. It might even make it harder to understand.
  - onion2k 3y ago
    There’s no reason to shut it all down if 1% of your request to a web server gets a 500. This is more like finding a bug in production and having all the devs stop working (including merging more changes) until it's fixed. Customers can still use the end product, you just stop making new things until you've solved the problem. There's some undeniable logic to doing that if you value bug free over fast.
- FirmwareBurner 3y ago
  >I am reminded of reading The Toyota Way a while back, which does talk about how they would shut down lines for any defect. Like a crack in the frame?[1] Shots fired! [1] https://news.ycombinator.com/item?id=37292321 https://news.ycombinator.com/item?id=37292321
- stubish 3y ago
  A lot of the Toyota methodologies ended up in various agile software practices. Not so much operations of a live system, but development practices. For example, all your teams push their changes to an integration branch which is automatically deployed to staging systems for testing and automatically deployed to production. And when a failure is found, that pipeline gets shut down until the problem is solved. And all eyes are looking at that problem, rather than tapping away in their own world ignoring the 'somebody else's problem'. Nobody is pushing new code unrelated to fixing the problem, making the testing and deployment of an update slower. And in theory, improves your uptime because the quality of code ending up on production is better, and when that fails, fixes will end up on production faster.
  - rtpg 3y ago
    That's interesting, I had never considered "main branch is red" as a similar fail state to the Toyota case, but I totally agree with that comparison!
- jacquesm 3y ago
  Instead you should do the opposite: aim to get the process completed even in the eye of failures. Because failures at all levels, including hardware are a fact of life in computing, more so in distributed computing and almost every problem is now a distributed problem.
  - rtpg 3y ago
    I would like to pose the claim that distributed computing is probably not more complicated than the distributed production of automobiles. Maybe The Toyota Way is actually bunk from the get go. But if you buy into the idea that it does work, then why would it not be applicable in software development? This is the core thing that interests me.
- monomers 3y ago
  In IT we (wrongly) use the word "production" to refer to the systems in operation serving customers: ie. the car that has left the factory. I don't know much about cars, but Toyota has a reputation for high reliability there. In manufacturing, production lines instead refer to a previous step in the lifecycle, still in the factory. That's where you can pull the andon cord.
jdkee 3y ago
"Toyota's share price was up 0.12% at 2,439 yen in afternoon trade after spending much of the morning in negative territory."
- 01100011 3y ago
  It's nearing end of month in a low-liquidity market(summer, pre-labor day weekend). Next month is historically bearish(probably because the pro's return from vacation and start correcting all the summertime madness).
happytiger 3y ago
My Toyota was the only car I ever bought that had exactly zero defects and worked perfectly in every capacity asked of it. They are the Debian Linux of vehicles — older libraries but stable. No cutting edge package managers to crash your workday. Therefore, for them to shut the line down means they are working through a serious issue, but as long as they keep that commitment to “quality, reliability and durability” and the reputation they’ve earned because of it, I’m sure the reason is worth it. That said, the new Land Cruiser has cup holders for a third row seat in the US (a 250 globally, not sure why they decided to do a Lc brand on a 250 just for the US markets), but it… doesn’t have a third row to use them. It’s baffling and very not Toyota. So maybe they are slipping?
- nsvd 3y ago
  With regards to the cupholder, it's possible that Toyota is re-using parts from other vehicles which do have a third row, to bring down costs on the assembly line. Pure speculation on my part.
  - jychang 3y ago
    Yep, this is the correct answer. The new Land Cruiser is built on the same platform as the Lexus GX, which has the 3rd row. Also, the 3rd row is being occupied by the hybrid battery system. In other countries where they offer a non-hybrid Land Cruiser, they have a 3rd row.
- rootsudo 3y ago
  every other market in the world the LC is 3 row seating. It isn't considered luxurious because the space in the trunk is more wanted. Look at LC in Philippines or APAC.
  - happytiger 3y ago
    Absolutely. You know what you’re talking about. So why didn’t they delete the body panel for the US? It’s a minor part.
mrweasel 3y ago
Oh how I do hate the word "glitch". It tells you nothing and makes the whole thing sound like a happy little accident. A glitch to me is "Hey every time we order 200 of these fuses, we also get five lug nuts". That's a glitch. It's weird, but production moves forward. This is an incident. No, it either a software bug, misconfiguration, network issue, hardware failure in Toyotas part ordering system or something to that nature.
- bmitc 3y ago
  Glitch just sits on two sides of the extreme. It's either too early in the investigation to call it anything else or it's after the investigation and there's not enough knowledge of the stack consisting of people, software, hardware (electrical, mechanical), and physics to figure it out.
- swarnie 3y ago
  Reminds me of my old college lecturer. > There is no such thing as a bug or a glitch. Computers will only do exactly what someone tells them to do. > You either have a mistake, or a lack of understanding.
  - imp0cat 3y ago
    Unless of course a random solar flare comes and flips a bit. ;)
    - bluepod4 3y ago
      Or something more simple like a browser update.
    - SAI_Peregrinus 3y ago
      Or any other error source like that. Thermal noise, background radioactivity, etc. As chip density increases the chance of bit flips also increases, and ECC RAM becomes more important.
  - rusk 3y ago
stubish 3y ago
Always headlines when this happens, but shutting down production like this is what Toyota believes is part of their magic sauce. They lose productivity when a problem shuts them down, but most of the time when things are working they gain due to lack of waste. But despite being a common part of curriculum around the world, it isn't that common to see, I think because intuitively it seems wrong somehow.
- sethammons 3y ago
  The Andon Cord: https://en.m.wikipedia.org/wiki/Andon_(manufacturing) https://en.m.wikipedia.org/wiki/Andon_(manufacturing) My best CEO borrowed that idea for our company and anyone could stop anything (or at least get serious consideration) by "pulling the andon cord" - if something was broken or being mismanaged, employees were empowered and encouraged and congratulated to fix it. Best engineering org I've been in.
  - bluepod4 3y ago
    Was there a mechanism in place to prevent people from abusing this process? (I’m assuming this is what you’re referring to when you said “at least serious consideration”) I ask because based on personal experience and from reading comments on HN, most engineering orgs tend to be ego-heavy.
    - dist-epoch 3y ago
      > Was there a mechanism in place to prevent people from abusing this process? Yes, it's called firing.
      - dtjohnnymonkey 3y ago
        No one would pull the Andon cord if they thought they might get fired for it.
    - sethammons 3y ago
      Were there times someone called something an Andon Cord situation and it turned out not to be? Of course! And that is fine. Were there "serial false andon cord pullers"? Not that I'm aware of - and if there were, the solution would have been to have an adult conversation with a focus on getting alignment. Pulling an Andon Cord was not for things you merely disagreed with, for that we relied on "disagree and commit" whereby you could bring data up later show it was a bad decision, but for now, work on making it work until you have that data.
ChrisArchitect 3y ago
Version of news at the Japan Times: (no account/paywall) https://news.ycombinator.com/item?id=37303807 https://news.ycombinator.com/item?id=37303807
lapcat 3y ago
Toyota has major supply chain problems. I wanted to buy a Corolla recently, but the wait time was 8+ months, so I ended up buying a Honda Civic instead, which also had a wait time, but it was much more tolerable at 1 month.
- justin66 3y ago
  The wait for a Toyota, if one wants any say in the options on the car, is quoted at six or eight months where I live. And as my friends have experienced, there's no penalty if the dealership takes longer than the quoted time to deliver a vehicle. It's weird.

74 points by KnuthIsGod 3y ago

rtpg 3y ago

One could only speculate on the cause here without more info. I am reminded of reading The Toyota Way a while back, which does talk about how they would shut down lines for any defect. A coworker of mine was pretty adamant that this was A Good Idea(TM), but I thought about how much of IT best practices are about uptime and doing (often principled!) workarounds when you have operational issues, and moving forward in a degraded state. I wonder if there are IT operations that do try to aim for the ~no defect approach like this.

housemusicfan 3y ago
Toyota sounds like the kind of place that still runs Solaris on SPARC and has no immediate plans to change.
- r00fus 3y ago
  You jest but back in the day (90s) one of my friends said they still ran Smalltalk VMs in production. That was 25 years ago but still possible…
  - deleted 3y ago
    [deleted]
- kenhwang 3y ago
  From my conversations with some Toyota software engineers, they have a respectably sophisticated modern software/tech stack. There's a ton of C++ since they have to interface with hardware. For tooling and internal applications it's a lot of Rails/Elixir/Javascript. Web APIs are backed by Elixir/Go. Data processing in Scala/Spark. Infrastructure in AWS, orchestrated with k8s. Truly agile prototyping then polish, significant automated testing. Honestly, it makes "big tech" look like dinosaurs with their megasized-Java projects and 3:1 PM:engineer ratio waterfall planning once a year releases. What they're missing is the operational experience with operating the software at scale.
upon_drumhead 3y ago
I’m sure finance folks are quick to hit the big red stop button if it’s causing invalid transactions to clear. But generally the impact to the end user doesn’t really matter. There’s no reason to shut it all down if 1% of your request to a web server gets a 500. It might even make it harder to understand.
- onion2k 3y ago
  There’s no reason to shut it all down if 1% of your request to a web server gets a 500. This is more like finding a bug in production and having all the devs stop working (including merging more changes) until it's fixed. Customers can still use the end product, you just stop making new things until you've solved the problem. There's some undeniable logic to doing that if you value bug free over fast.
FirmwareBurner 3y ago
>I am reminded of reading The Toyota Way a while back, which does talk about how they would shut down lines for any defect. Like a crack in the frame?[1] Shots fired! [1] https://news.ycombinator.com/item?id=37292321 https://news.ycombinator.com/item?id=37292321
stubish 3y ago
A lot of the Toyota methodologies ended up in various agile software practices. Not so much operations of a live system, but development practices. For example, all your teams push their changes to an integration branch which is automatically deployed to staging systems for testing and automatically deployed to production. And when a failure is found, that pipeline gets shut down until the problem is solved. And all eyes are looking at that problem, rather than tapping away in their own world ignoring the 'somebody else's problem'. Nobody is pushing new code unrelated to fixing the problem, making the testing and deployment of an update slower. And in theory, improves your uptime because the quality of code ending up on production is better, and when that fails, fixes will end up on production faster.
- rtpg 3y ago
  That's interesting, I had never considered "main branch is red" as a similar fail state to the Toyota case, but I totally agree with that comparison!
jacquesm 3y ago
Instead you should do the opposite: aim to get the process completed even in the eye of failures. Because failures at all levels, including hardware are a fact of life in computing, more so in distributed computing and almost every problem is now a distributed problem.
- rtpg 3y ago
  I would like to pose the claim that distributed computing is probably not more complicated than the distributed production of automobiles. Maybe The Toyota Way is actually bunk from the get go. But if you buy into the idea that it does work, then why would it not be applicable in software development? This is the core thing that interests me.
monomers 3y ago
In IT we (wrongly) use the word "production" to refer to the systems in operation serving customers: ie. the car that has left the factory. I don't know much about cars, but Toyota has a reputation for high reliability there. In manufacturing, production lines instead refer to a previous step in the lifecycle, still in the factory. That's where you can pull the andon cord.

jdkee 3y ago

"Toyota's share price was up 0.12% at 2,439 yen in afternoon trade after spending much of the morning in negative territory."

01100011 3y ago
It's nearing end of month in a low-liquidity market(summer, pre-labor day weekend). Next month is historically bearish(probably because the pro's return from vacation and start correcting all the summertime madness).

happytiger 3y ago

My Toyota was the only car I ever bought that had exactly zero defects and worked perfectly in every capacity asked of it. They are the Debian Linux of vehicles — older libraries but stable. No cutting edge package managers to crash your workday. Therefore, for them to shut the line down means they are working through a serious issue, but as long as they keep that commitment to “quality, reliability and durability” and the reputation they’ve earned because of it, I’m sure the reason is worth it. That said, the new Land Cruiser has cup holders for a third row seat in the US (a 250 globally, not sure why they decided to do a Lc brand on a 250 just for the US markets), but it… doesn’t have a third row to use them. It’s baffling and very not Toyota. So maybe they are slipping?

nsvd 3y ago
With regards to the cupholder, it's possible that Toyota is re-using parts from other vehicles which do have a third row, to bring down costs on the assembly line. Pure speculation on my part.
- jychang 3y ago
  Yep, this is the correct answer. The new Land Cruiser is built on the same platform as the Lexus GX, which has the 3rd row. Also, the 3rd row is being occupied by the hybrid battery system. In other countries where they offer a non-hybrid Land Cruiser, they have a 3rd row.
rootsudo 3y ago
every other market in the world the LC is 3 row seating. It isn't considered luxurious because the space in the trunk is more wanted. Look at LC in Philippines or APAC.
- happytiger 3y ago
  Absolutely. You know what you’re talking about. So why didn’t they delete the body panel for the US? It’s a minor part.

mrweasel 3y ago

Oh how I do hate the word "glitch". It tells you nothing and makes the whole thing sound like a happy little accident. A glitch to me is "Hey every time we order 200 of these fuses, we also get five lug nuts". That's a glitch. It's weird, but production moves forward. This is an incident. No, it either a software bug, misconfiguration, network issue, hardware failure in Toyotas part ordering system or something to that nature.

bmitc 3y ago
Glitch just sits on two sides of the extreme. It's either too early in the investigation to call it anything else or it's after the investigation and there's not enough knowledge of the stack consisting of people, software, hardware (electrical, mechanical), and physics to figure it out.
swarnie 3y ago
Reminds me of my old college lecturer. > There is no such thing as a bug or a glitch. Computers will only do exactly what someone tells them to do. > You either have a mistake, or a lack of understanding.
- imp0cat 3y ago
  Unless of course a random solar flare comes and flips a bit. ;)
  - bluepod4 3y ago
    Or something more simple like a browser update.
  - SAI_Peregrinus 3y ago
    Or any other error source like that. Thermal noise, background radioactivity, etc. As chip density increases the chance of bit flips also increases, and ECC RAM becomes more important.
- rusk 3y ago

stubish 3y ago

Always headlines when this happens, but shutting down production like this is what Toyota believes is part of their magic sauce. They lose productivity when a problem shuts them down, but most of the time when things are working they gain due to lack of waste. But despite being a common part of curriculum around the world, it isn't that common to see, I think because intuitively it seems wrong somehow.

sethammons 3y ago
The Andon Cord: https://en.m.wikipedia.org/wiki/Andon_(manufacturing) https://en.m.wikipedia.org/wiki/Andon_(manufacturing) My best CEO borrowed that idea for our company and anyone could stop anything (or at least get serious consideration) by "pulling the andon cord" - if something was broken or being mismanaged, employees were empowered and encouraged and congratulated to fix it. Best engineering org I've been in.
- bluepod4 3y ago
  Was there a mechanism in place to prevent people from abusing this process? (I’m assuming this is what you’re referring to when you said “at least serious consideration”) I ask because based on personal experience and from reading comments on HN, most engineering orgs tend to be ego-heavy.
  - dist-epoch 3y ago
    > Was there a mechanism in place to prevent people from abusing this process? Yes, it's called firing.
    - dtjohnnymonkey 3y ago
      No one would pull the Andon cord if they thought they might get fired for it.
  - sethammons 3y ago
    Were there times someone called something an Andon Cord situation and it turned out not to be? Of course! And that is fine. Were there "serial false andon cord pullers"? Not that I'm aware of - and if there were, the solution would have been to have an adult conversation with a focus on getting alignment. Pulling an Andon Cord was not for things you merely disagreed with, for that we relied on "disagree and commit" whereby you could bring data up later show it was a bad decision, but for now, work on making it work until you have that data.

ChrisArchitect 3y ago

Version of news at the Japan Times: (no account/paywall) https://news.ycombinator.com/item?id=37303807 https://news.ycombinator.com/item?id=37303807

lapcat 3y ago

Toyota has major supply chain problems. I wanted to buy a Corolla recently, but the wait time was 8+ months, so I ended up buying a Honda Civic instead, which also had a wait time, but it was much more tolerable at 1 month.

justin66 3y ago
The wait for a Toyota, if one wants any say in the options on the car, is quoted at six or eight months where I live. And as my friends have experienced, there's no penalty if the dealership takes longer than the quoted time to deliver a vehicle. It's weird.