18 ms·
Toyota blames factory shutdown in Japan on ‘insufficient disk space’
- thisisit 3y agoMy company is going through a cost cutting phase. And somehow cutting disk space ended being one of the priority items. One of my servers utilized 50% of the allocated space and the "low disk space" alert was set to go off at 80%. Management came up with an idea - why not reduce it to used space + 20% of used space. That way we save a lot of space. I had to explain them simple math - if the space is 100GB and I utilize 50GB then the proposal was to have 60GB on the server. But 50/60 is already beyond the 80% alert. And if the target was lower at say 70% alert then we needed at least 22GB i.e. used space + 44% of used space to make it work. It was mind boggling that people can't understand simple ratios. The most maddening part was that no one could explain the savings if the space was reduced. People making noise said we could "avoid costs" while forgetting that used space is already cost, you cannot "avoid" it.
- jiofj 3y agoDidn't they see the balloon?!? https://3.bp.blogspot.com/-Pv29dGQwIMI/UFcMD6hgcyI/AAAAAAAAFHI/g_z4acYUqT8/s1600/ttUntitled-2.jpg https://3.bp.blogspot.com/-Pv29dGQwIMI/UFcMD6hgcyI/AAAAAAAAF...
- mcny 3y agoTo spell it out, I think the joke here is the assertion that they are probably using old operating system and software. The screenshot is for windows XP.
- picadores 3y agoWhy is that a joke? Factory equipment is ordered and has a lifetime of 20 years. Of course it has old systems. These are grownup industries. They have no time for development hipster nonsense and extraction games for unwanted features to stave of unemployment in some glass-towers.
- scandox 3y agoYou're shooting the translator.
- BoxFour 3y ago> They have no time for development hipster nonsense https://support.microsoft.com/en-us/windows/windows-xp-support-has-ended-47b944b8-f4d3-82f2-9acc-21c79ee6ef5e https://support.microsoft.com/en-us/windows/windows-xp-suppo... > extraction games for unwanted features https://www.theverge.com/2022/7/12/23204950/bmw-subscriptions-microtransactions-heated-seats-feature https://www.theverge.com/2022/7/12/23204950/bmw-subscription...
- delecti 3y ago> If you continue to use Windows XP now that support has ended, your computer will still work but it might become more vulnerable to security risks and viruses It only matters if the machine is online, and they often aren't.
- BoxFour 3y ago> they often aren't. This was NOT my experience while working in the automotive industry.
- mcny 3y ago> This was NOT my experience while working in the automotive industry. Exactly. The only people who can climb the corporate ladder are people who care about business "impact" more than anything else.
- COM9 3y ago> It only matters if the machine is online, and they often aren't. No. Really, you shouldn't think that. It's really bad security practice. https://en.wikipedia.org/wiki/Stuxnet https://en.wikipedia.org/wiki/Stuxnet It is typically introduced to the target environment via an infected USB flash drive, thus crossing any air gap.
- giancarlostoro 3y agoYou're likely not wrong, but also the fact that Windows tells you you're running out of storage going as far back as XP if not further back. I wont be surprised if Mac and Linux do the same. My guess is, they kept ignoring it until it became impossible to ignore. I wonder if years passed until it was unavoidable.
- thelastparadise 3y agoA lot of server OS's/configurations will just silently fill up and take down any services depending on that volume with it.
- giancarlostoro 3y agoWell yeah, if your swap partitions on the same as the rest of your OS, eventually it will just make itself known as an issue.
- Ekaros 3y agoI just had much more fun experience on Debian. The VM simply froze after startup as disk was absolutely full... And figuring out how to actually extend disk through all of the indirection layers took a bit of time.
- jbverschoor 3y agoNever happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.
- Ekaros 3y agoIn this case I had to use rescue mode. As even different tty was too slow to respond. Also just removing some stuff up wasn't working.
- jbverschoor 3y agoSounds like swapping/trashing + an exception loop taking all resources
- codetrotter 3y agoCould it depend on what file system you use, and OS settings you have? For example, Google says that by default ext4 will reserve 5% of disk space. So depending on your system, a “100% full” drive might still have a (sometimes significant) bit of space left.
- throw0101c 3y ago> For example, Google says that by default ext4 will reserve 5% of disk space. Reserved 5% for root, but if the files/logs that are taking up space are written by the root user then it's a moot point: > Specify the percentage of the file system blocks reserved for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the file system. The default percentage is 5%. * https://man.archlinux.org/man/mke2fs.8 https://man.archlinux.org/man/mke2fs.8 One trick possible with ZFS is to take a pool (e.g., rpool) and create an 'extra' data set in it (rpool/reservation) and set a reservation for it, so that even if all the other areas are filled (rpool/root, rpool/var, rpool/home) you can simply lower the reservation to get some space back quickly.
- zoky 3y agoIf 14 geographically distinct factories can be shut down by a single server acting up, you’ve got problems that overnight shipping from Newegg can’t solve…
- b3lvedere 3y agoIt was a rainy day so the cloud was not to full capacity :)
- robertlagrant 3y agoWhy is the article tying just-in-time to this issue?
- mattw2121 3y agoI guess they would have us believe that adding disk space, when necessary, is also done "just-in-time" and wasn't done "just-in-time" enough.
- deleted 3y ago[deleted]
- junaru 3y agoBecause they wouldn't need to shut down factories if they had bigger stock of parts on site.
- riidom 3y agoI wonder, wouldn't it be more of a question of how much later to shut down. Or in other words, how big is the time buffer you get for not being "just in time"? The article suggests (the way I read it, at least) that it is some fundamental time span, but I feel like it's more like half a day?
- LeifCarrotson 3y agoIn some (American) automotive plants I've worked in, JIT lifetimes are about 10 hours. The order for a seat or mirror or door handle doesn't exist at the tier 1 suppliers where my equipment gets installed until 10 hours before it is needed. At that point, my PLC gets a request to induct a particular part number and paint color into the manifest, inducts it and acknowledges that request, the machine builds it, laser-marks a serial number onto it, tests it, uploads the test results with serial number to the database, packs it out into a foam-lined box or a dedicated nest and uploads the box barcode to the database, that box gets loaded onto a pallet, the pallet into a trailer, the trailer drives 20 minutes away to the OEM, and the trailer gets unloaded (in reverse order; truck size and the resulting last-in-first-out queue is one of the main drivers for JIT lifetimes) so the parts land at the assembly line just a couple hours before the vehicle they belong to is ready to receive them. You'll notice that this does not leave a lot of wiggle room for anything to go wrong. When I'm physically a 5 hour drive away from the plant, it's not like I can drive to the site, debug a wiring issue, order a replacement widget online - even by UPS Next Day Air Early A.M. - and then install it on-site before holding up literally millions of dollars worth of production. And my little mirror line is just one of hundreds of parts that need to arrive on time.
- tantalor 3y agoLeslie, I typed your symptoms into the thing up here, and it says you could have network connectivity problems
- wizerdrobe 3y agoAs we move to more abstract systems I wonder how well we (as in companies) are keeping basic systems management capabilities in place at a personnel level. At $DAY_JOB we recently scuttled most development efforts for a week for our teams. Our nightly backup job that sanitizes PHI ballooned overtime to, say, 20GB+1Byte and ran out of disk space. Because we are running Kubernetes on Fargate we don’t need a full time operations guy, right? Commence the company (me) scrambling to learn how to use a Peristent Volume and Peristent Volume Claim because a career programmer should be able to perform systems administration takes because DevOps has the word “Dev” in it, right? So we lost a week of productivity to disk space, but in reality we lost a week to poor personnel and capabilities.
- thelastparadise 3y agoIf it suits your setup, I find that offloading stateful services to managed cloud providers, then running the stateless stuff in k8s is a nice combo. It's not k8s or anything else that makes it hard/easy. Maintaining and keeping stateful services running the long run is what's a bit tricky.
- acdha 3y agoThat’s been my preferred model since the 2000s, too. I’ve seen a lot of projects where someone wanted to do something which they couldn’t do with RDS, etc. and every time the results didn’t match the early optimism.
- BowBun 3y agoI worry about the same thing - cloud provider and especially specialty hosted solutions (Heroku) are getting more expensive as dependency on them grows and smaller shops basically have to use them to get off the ground. In the end what you have is a stack of technology that the company cannot actually maintain on their own. Now I know that 'shoulders of giants' is a thing and that we shouldn't expect everyone to be able to do every part of work that came before them, but as your example shows it is a real business risk when things go south. Part of the original promise from these providers was that there would be a 90%+ reduction in supporting work, when we all know the reality is that you become specialized in solving issues with Heroku/AWS/providerX
- xeromal 3y agoI was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
- zimpenfish 3y agoAnecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.
- lostlogin 3y agoIt’s year ago now, but my father bought a second hand hard drive and found it full of banking data with customer names and all.
- deleted 3y ago[deleted]
- b112 3y agoUm, does he still have that data? Asking for a friend!
- hnbear 3y agoAs an ex-bank employee one time I needed space and was told I couldn’t get it because all the filers were full. When would we get new filers? Dunno, because the datacenter was also full.
- willcipriano 3y agoThere was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.
- archildress 3y agoJust-in-time will cost you far more than it will save you. The problem is that when it fails, you judge those failures as "backwards looking one offs" as opposed to a cost that is part and parcel to being JIT. Factory downtime is the single biggest cost a manufacturing shop can incur. No amount of working capital savings from carrying fewer screws, widgets, and bolts can offset the costs of taking 14 plants down due to penny wise, pound foolish math.
- hotstickyballs 3y agoFortunately Japanese (or generally Asian) culture makes it the individuals responsibility if things fail so corporate is safe as usual
- deleted 3y ago[deleted]
- davedx 3y agoIt would be nice if you could back that up with some facts, considering Toyota has been using some variant of this system since 1948. Please provide a citation for how JIT manufacturing cost TM "far more than it saved them"?
- 0cf8612b2e1e 3y agoBad business practices can survive for a long time. I do not have any skin in on the game as to how effective or not JIT is, but wouldn’t the correct question be, how did Toyota fair vs non-JIT companies? As GP was implying, JIT is fine day-to-day, but can make failure conditions worse.
- wil421 3y agoA family member worked for a large military aerospace company. In the 80s they were building a system to order parts and tell the factory what to produce and complete daily so airplane production was maximized. He said they had so many parts employees said they used JIC or Just In Case. Reducing the amount of parts on hand saved them money and made them more efficient.
- CodeCompost 3y agoThat's why kids, you shouldn't use Event Sourcing.
- Floegipoky 3y agoWhat does this have to do with event sourcing?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- ndhfkashdben 3y agoI worked at a phone factory once. One day, an app stopped working. It was running on an old Linux server. Nobody had access to it, or so I thought. I noticed the pings were showing high failure rates. I asked to check switch logs or the server's ifconfig output and got weird excuses why they couldn't check anything. One week goes by, people are living with the error and adjusting the factory to work around that. Many meetings, etc. Network engineer pulls me to the side and says "you seem like a nice guy, stop worrying about this issue. we won't fix it because if we do, this server becomes ours and we are responsible for this app.. so nobody will touch it. yes, the cable to that server needs to be redone. yeah we won't do that" I stayed there for 10 months and quit.
- datavirtue 3y agoTypical.
- syndicatedjelly 3y agoHe was low key telling you to quit and gtfo
- xwdv 3y agoWell if I knew I was leaving anyway then I would have just quietly fixed it on my own. And then immediately put in a two weeks notice and use my vacation time.
- hhhhhtime 3y agoAaah, plant shutdowns. I worked in IT in an automotive assembly plant at one point. Once, I was out on the plant floor with a colleague. We were diagnosing a new network drop and they plugged in a ping testing device into the cable, hit the button, and within seconds the entire assembly line went down. Cue lots of radio chatter and people driving up to us in their carts trying to figure out what was going on. Turns out the ping testing device was last configured with the IP of a critical application server that talks to every device in the plant. When the colleague pressed the button, every plant device stopped contacting the apps server and the plant went down. Unplugging the device eventually fixed it. The line was stopped for 7 minutes. I believe stoppages were usually quoted at tens of thousands of dollars per minute and billed to the responsible department. I remember how stressed everyone was after and meetings for root cause analysis in the following days. I was always surprised at this considering how frequently the line went down for small amounts of time, and they never seemed as big of a deal as that one specific time, though for the others we were not responsible.
- pnutjam 3y agoI once plugged a toner into a network patch panel to locate a drop and crashed the AIX system at a medical office. :(
- oasisbob 3y agoHope the RCA highlighted the risks of having a critical network server share the L2 segment with all its clients.
- grecy 3y agoA friend was a process engineer and studied and worked in a bunch of car assembly plants optimizing stuff. Every single time the line stopped, no matter why, it was estimated to cost, at absolutely minimum, $1 million dollars.
- albertgoeswoof 3y agoWow, my company (mailpace.com) recently had an outage for basically the same reason: https://blog.mailpace.com/blog/postgres-outage-post-mortem/ https://blog.mailpace.com/blog/postgres-outage-post-mortem/ Luckily, we only went offline for about 2 hours, but glad to see behemoths like these suffer from similar issues...
- Octokiddie 3y agoCan anyone offer evidence that this is not due to low demand, with insufficient disk space being used as an excuse?
- bluedino 3y agoHad a production system crash because of a similar issue somewhat recently. The cause? / filled. It was configured at somewhere around 80GB. There was a scratch volume on the VM as well, also 80GB. Recently the machine was upgraded to a newer version of Ubuntu, that used a Snap for Chromium. tmp on the Snap wasn't on the scratch drive like on the old version, it was in /tmp For some reason, ops never set a watch/alert on that disk. That would have prevented the issue.
- NKosmatos 3y agoIf it’s not an excuse for something more serious they definitely need to rethink their infrastructure. It’s not allowed to have a single server with no disk space, taking offline 14 factories. That should’ve used the old trick. Create a big empty file inside the server (i.e. delete_me_in_case_of_need.txt with size a few GBs) and delete that in case of emergency. This will buy you some time to take the necessary actions ;-)
- doublepg23 3y agoI learned the hard way about always using ZFS quotas.
- oasisbob 3y agoClassic scaling hack. There's a talk from someone at a high-volume adult entertainment site which used this trick too - can't remember which one. As told, the caching functionality in a key service had an artificial 10ms delay added. When things got tough, it was an easy change to disable the delay.
- marcosdumay 3y agoYes, when you fully optimize something by one metric, like cost, other metrics, like reliability, get worse and worse. Funny thing that if you get any book about Toyotism and Lean, this warning is around the first things you see. You only optimize until you start seeing problems, then you either stop or improve things so the problems don't happen anymore and you can move further. Well, looks like the management at the company that led this culture is forgetting the lesson.
- deleted 3y ago[deleted]
- danielfoster 3y agoI don’t see how a one-day factory stoppage aligns with the article’s teaser, “ Just in time’ production system minimises costs but technical glitch highlights risks.” It seems like Toyota identified the issue and got things running again quickly enough. This does not at all seem like a good case study for exploring the risks of JIT. I read The Guardian regularly and will continue to, but they sometimes go out of their way to find an anti-business angle.
- daviddever23box 3y agoSomeone needs to fall on a very sharp sword for this one.
- rawgabbit 3y agoThey didn't say which database software Toyota uses. But in general, if you are performing DML and you have configured backup retention to "normal".... that means the transactions file will balloon in size until the database is BACKEDUP. Because this caused an outage, I assume there was no DBA assigned and management assumes it will run faithfully like it always did.
- freedude 3y agoHmmm, after experiencing a similar "Insufficient Disk Space" situation once, there is now a folder with 10 Gb of "data" I can delete to "remediate" the situation in short order.
- momirlan 3y ago[flagged]
- dang 3y agoRecent and related: Toyota to restart Japan production on Wednesday after system failure - https://news.ycombinator.com/item?id=37303569 https://news.ycombinator.com/item?id=37303569 - Aug 2023 (66 comments)
- FreshStart 3y agoA well oiled car factory drops one of the belt ever n minutes. On a new car there are y% of profit. n x y > price of Harddisk is were the yelling and firing of administration began. Power trips and savings are nice. But factory stops have a price and when it turns out to be thiefdoms, those chiefthiefdomtains are goners. Can't run "the rules stated.." by the shareholders who know that the rules are made up for operations not against it.