11 ms·
Toshiba and WD NAND Production Hit by Power Outage: 6 Exabytes Lost
- gruez 7y agoSo... does that mean they'll be hiking NAND prices, just like with HDD prices after the Thai floods?
- jacquesm 7y agoNo, it means prices will go up because there is a shortage, like with everything else. Sugar, Gasoline and so on are good examples. HDD prices are just one more item that follows the supply/demand curve. Sure there will be some clever parties that will make some money anticipating this. But that's the same reason why the price of the gas at the pump that was already in the tank jumps up because of a shortage somewhere else. The whole stock is instantly valued at a different price.
- thesimp 7y agoLooking at the numbers it should not move that much. According to this article, https://www.businesswire.com/news/home/20190307005812/en/TRENDFOCUS-Combined-SSD-HDD-Storage-Shipped-Jumps https://www.businesswire.com/news/home/20190307005812/en/TRE..., in 2018 912 exabytes of HD and SSD storage was sold. 800 exabyte for HD and 112 exabyte for SSD. And the SSD market grew 45% in 2018. If manufacturers project to grow at the same rate then 2019 SSD shipments will be around 162 exabyte. This puts the 6 exabyte loss at around 3.5%. But we all know that markets are driven by emotion: losing 3.5% of your raw materials in a market that is projected to grow 45% will cause big fluctuations. But that is just my opinion.
- maxerickson 7y ago6 exabytes is the Western Digital side, the article says that Toshiba produces more there, so it is going to be 12+ exabytes.
- imtringued 7y agoHDD and RAM companies always use bullshit excuses to collude and raise prices.
- otakucode 7y agoNAND chips are a commodity, so you're actually undercounting their sales by a tremendous amount. Don't forget the integrated NAND chips in... well, nearly every product produced across every market segment for the past 20 years. If it runs on electricity, it probably has a NAND chip somewhere.
- blattimwind 7y agoMost of those "if it runs on electricity, it probably has a NAND chip somewhere" are very different chips from those used in SSDs; they are typically low density and manufactured on older processes (1xx μm) used for firmware/bitstream storage.
- otakucode 7y agoThat is true. However, those less performant chips, due to the way NAND storage functions, could easily be used in place of the newer chips. If you need twice as much performance, you just use 2 chips and you're done. Unless dealing with extreme size constraints like in a cell phone (where the high-performance NAND isn't even used for price reasons), there's no real reason to prefer a single chip over two. For the consumer, that is. If you're producing chips and trying to drive per-chip profit margin and trying to make your product look like CPUs and other products that have some complexity behind them, though, it's useful. I can understand how NAND prices don't look suspect if you're not terribly familiar with the history and low-level factors of the industry, but if you really look into it, it's kind of ridiculous. The same companies price-fixed DRAM chips and got busted. Then they price-fixed LCD panels and got busted. Then they price-fixed DRAM chips again and got busted again. They were being investigated for price-fixing NAND chips, but the South Korean president shut down the investigation. Shortly before being ousted for rampant corruption (and then being bailed out of prison by Samsung). Anything that is present in such a gigantic variety of devices should cost almost nothing. That's just economics. It becomes commoditized. The materials involved and their rarity become the primary drivers of price. Comparing price per terabyte of storage between mechanical drives and NAND-based storage is the most telling to me personally. The technology that goes into modern high density mechanical hard drives is utter madness. They should be, by all accounts, astronomically expensive. They use helium, of which there is a global shortage. They coat the platters with rubidium and other rare materials. They include neodymium magnets. They include high precision mechanical motors that spin platters fast enough that the surface tension against the air becomes a significant factor (leading to the use of helium) and still maintain enough precision to be able to seek to a very precise spot in nanoseconds. Also, you've got 'hybrid' drives that include both the mechanical and NAND storage... which incurs almost no premium over the pure mechanical solution. Now they're beginning to produce drives with integrated lasers for heat-assisted magnetic recording. And these are still many times cheaper on a $/TB basis compared to.... just a dumb parallel array of NAND gates that don't require anything rare?
- antpls 7y agoI believe you forgot Toshiba, which could be 9 exabytes lost according to the article for this quarter, so we are talking about 15 exabytes lost. According to your data, the total quaterly production is 41 exabytes for SSD, which would mean losing about 37% of the total SSD production this quarter. That being said, it is the first time I read about the scale of storage production worldwide. It makes you wonder what does the humanity store in those hundreds of exabytes per year. Probably many duplicated data or unused bytes.
- simcop2387 7y agoThese days a large portion of it is just basically logs. Logs of all the traffic we're generating looking at content on the internet, to be used to try to target ads. That and videos, youtube itself probably accounts for a significant amount of that storage use.
- owl57 7y agoAre the logs of this scale and less-popular videos usually stored on SSD? I thought HDDs are still cheaper and RAID gives enough throughput given enough disks?
- dodobirdlord 7y agoLogs are probably on HDD, video might be on SSD given how aggressively it gets edge-cached.
- londons_explore 7y agoHDD's access time (10milliseconds or more), means a hard disk can't really serve more than 100 concurrent users, assuming each wants to stream a chunk of video every second. That makes it a poor choice for serving anything but the rarest of YouTube videos.
- owl57 7y agoI believe a typical video, stored in all of the Youtube formats, uses on the order of a megabyte per second. So, a 10TB disk probably holds about 100 days of video. Seems fine for videos that are watched less than once a day, that's probably a vast majority of Youtube's storage.
- otakucode 7y agoI think that is entirely up to Toshiba/WD. The profit margin on NAND is so astronomical and the price charged for it is so completely decoupled from the cost of production (which is as close to nothing as anything gets) that they could afford to just absorb the 'loss', but it might mess with their projected schedules of how much they had expected to make, so I could see them jacking up the price to compensate. The market and society in general seems to be content with permitting the NAND manufacturers price-fixing even when it's become absurd (do a tally of the raw materials and processes involved in producing 1TB of modern mechanical hard drive storage compared to a dumb regular parallel array of 1TB of NAND gates... it's ludicrous) so they've got whatever flexibility they feel like using.
- tw04 7y agoAre you just leaving out the cost of building the Fab...? The whole reason it was a joint venture in the first place was to try to soften the up front investment for both companies. That's ignoring the R&D dollars on 3D and/stacking. If you think that's also easy, why were micron and intel over a year behind Samsung? It wasn't by choice.
- icefo 7y agoI wonder what failed in their redundant power supply because they surely have something. I hope the postmortem will be public !
- igravious 7y agoIf my reading comprehension has not let me down then a 13 minute power disruption can cause them to lose 1/2 of their output for a quarter. Given the massive consequences of quite a short disruption maybe they need to figure out how to weather disruptions more robustly?
- iamgopal 7y agoWhy individual process independently powered, if risk is this huge ? Why common power ?
- igravious 7y agoNot quite understanding you, sorry.
- Tuna-Fish 7y ago> Given the massive consequences of quite a short disruption maybe they need to figure out how to weather disruptions more robustly? If you mean "they should not lose so much product when equipment loses power", that's just not possible. Modern semiconductor manufacturing involves hundreds of steps where the wafers need to soak in a chemical bath for a very specific time, and missing deadlines by a few seconds causes the entire wafer to fail. The question is very much: "why did their UPS fail?".
- igravious 7y agoThat is indeed what I was implying.
- dgacmu 7y agoCycle times (the time it takes to process one wafer) can be in the range of a month. Any disruption therefore kills roughly a month (plus or minus) of output, at least for wafers in certain steps. It's brutal. Fabs are engineered to have redundant power, but what's interesting is that the same thing happened to Samsung last year: https://www.anandtech.com/show/12535/power-outage-at-samsungs-fab-destroys-3-percent-of-global-nand-flash-output https://www.anandtech.com/show/12535/power-outage-at-samsung...
- maheart 7y agoHere's an article from one month ago discussing the over-supply of NAND and DRAM (and the effect it has on pricing): https://www.forbes.com/sites/tomcoughlin/2019/05/25/nand-dram-supply-and-pricing https://www.forbes.com/sites/tomcoughlin/2019/05/25/nand-dra... I can't help but feel very skeptical about the timing of this event, given the history of price-fixing in the industry.
- GordonS 7y agoThese kind of issues do seem to hit with a suspicious degree of regularity - it seems every 1-2 years there is a shortage due to some calamity or such...
- wil421 7y agoYea like that time a suspicious typhoon knocked out all the Hard Drive fabs in Thailand.
- sbr464 7y agoI was there at the time, was pretty bad.
- landofmiles 7y agoI was there too. Not at the exact time but around those times. I had enough drive-space so I was never that hung up on it. I guess these things happen. Just as the LCD price fixing and everything else in this world. I was across the compound in the Seagate HDD scenario. Not living there but just sauntering along, just at those dates. Get some Western digital drives, 10tb's maybe, I hear they are shuckable and able now. But now my girl Mary is going to sleep. Signed off by Abraham Lincoln. I've got the 13th or23rd amendment on many drives. Plus in my hat. And also my wife's SCSI. One love. Too bad about Trump. May the Union win and persevere. I heard my favorite show is on Netflix. I may let my trusted Booth spin up the drive. Does anyone have popcorn. Your trusted Abraham L. The end. --- Fucking faggotry. Why all the downvotes. Fucking silicon valley cocksuckers. Enjoy your rent and shitty lives. What a bunch of faggots. Enjoy San Francisco and your end of life lives cocksuckers. Just kill yourselves now. You know the outcome anyways. Enjoy your shitty hells.
- loudouncodes 7y agoEvery time I see Godzilla he’s ensnared in and tearing down power lines. This was bound to happen sooner or later.
- ggm 7y agoThe fragility of the supply chain.
- deleted 7y ago[deleted]
- YayamiOmate 7y agoThis seems weird that 13 minute outage can kill month and a half production. I wonder if this is standat hi-tech factory process reliability.
- jotm 7y agoI guess it makes more sense to destroy everything affected by a power loss (even if some of it could be perfectly fine, or salvageable) than risk shipping products that will fail at a higher rate. That would cost way more in lost trust and lost sales.
- r00fus 7y agoNot to mention it's likely that it's insured against.
- meruru 7y agoThey could sell under a special brand.
- jotm 7y agoOh I just remembered! There used to be noname parts - motherboards, PCI cards, RAM. Literally no brand markings, no warranty either. I remember the RAM in particular - the chips had nothing etched on them, or a single 5char line. Even the firmware was unbranded, with strange timings, too (probably loosened because they would not work at standard specs). I'm guessing it was Chinese companies buying up "bad" or excess stock and reselling it. Haven't seen this in a while, either they tightened up regulations or the margins are too low to make a profit nowadays.
- taspeotis 7y agoI would be most grateful if someone could please explain what sort of tools are likely to be used here, and why a power loss to those tools would ruin days/weeks/months worth of output relative to the time they were offline?
- kop316 7y agoI imagine that the process is very highly pipelined and optimized, and I would imagine that they had some sort of backup (generator) that failed. One analogy is to think about if you had a batch script that you were working on that touches a lot of files (1000s). Now imagine power was cut and the batch script was interupted because the computer turned off, but that computer hasnt been turned off in a long time (say it was a server). First, you have to turn the server back on after a power outage. Was there corrupted files in it? You have to now get that server in a known working state, and if you have kept it on for years....then you may be in a world of hurt. NOw you got your server up and running. You have the option of going through each of the 1000s of files your script was working on....but that will take time. Does it make sense to start from scratch? You will have to through our all the files you were working on, but at least you can start that script again. You could attempt to salvage every file, but that will also take time too.
- paranoidrobot 7y agoAnother analogy is that you have a bakery that produces soufflés on an industrial scale. These soufflés take two months to bake. The baking has to be done in such a precise fashion that even fractions of a degree in variance results in the entire batch being ruined. Worse still, it takes a long time to bring the oven up to temperature and stabilise it on the precise temperature. You can't just scrap the batch that's ruined and start production again.
- watsocd 7y agoI like this analogy. Lets expand. These soufflés take two months to bake but you need soufflés every day for sales. What does this mean? You always have 2 months of soufflés in various states of production at all times. Now you lose power and all these as very fragile soufflés in production are lost because of the power failure. Furthermore, it will take you two months to get the first soufflés off the restarted production line.
- social_quotient 7y agoConspiracy: this is how the NSA buys their disk space.
- jamiek88 7y agoAt least 12 exabytes upgrade. Wow. Maybe they switched to using electron internally?!
- glbrew 7y agoThat is an interesting idea but it would be so much easier for them to constantly buy, say 5%, of the output.
- social_quotient 7y agoWell I’m not typically a real conspiracy guy but if I had to really think on this I’d say it’s easier to have a loss event like this and consequent writeoff vs some unexplainable long term 5% buyer. I also don’t think if I needed tons of storage like this I’d want to acquire it over a 20+ month period. Obviously I think I’m kidding but the thought is interesting.
- glbrew 7y agoNSA knows they need the storage, they didn't just figure out they needed to store a lot of data to perform a massive buy. It would much easier for the manufacturer to disguise 5% of constant purchases through various techniques, they could hide it in all sorts of ways. Also the NSA could set up shell companies to consistently buy output (frequently done during the cold war when USA needed to buy supplies from Soviet aligned nations, USSR itself). The NSA could hide behind major buyers like Google and Amazon. If the plant shut down for false reasons literally thousands of plants workers would know if was false. That wouldn't accomplish anything.
- otakucode 7y agoI know that NAND involves no exotic raw materials, so does that enable them to recycle any of the damaged/lost wafers? I don't know very much about the physical processing/preparation of the raw silicon and such that goes into making a wafer, could you simply grind up or perhaps chemically dissolve everything back to base components and re-create a fresh wafer?
- baybal2 7y agoAt least some scrap is now being bought by solar cell industry, but that material is forever lost for IC making because it's already contaminated with dopants and metals
- baybal2 7y agoI recall the story of Micron's first Chinese fab: 1 millisecond out phase brownout and they loose few megabucks instantly, and like that during every electrical event. Giant UPSes are not an option in the industry because fabs eat oodles of electricity, and it is cheaper to loose a megabuck once a year than build a stabilisation/ups plant
- vpribish 7y agomy friend - it's lose, not loose.
- agumonkey 7y agoFirst time I have to really think about Exa<unit>. Giga / Tera / Peta / Exa 6 Millions Terabytes of solid state memory.. quite a mass.
- pbhjpbhj 7y agoThis doesn't make sense: >Toshiba Memory and Western Digital on Friday disclosed that an unexpected power outage in the Yokkaichi province in Japan on June 15 affected the manufacturing facilities that are jointly operated. // Surely that's not the reason, it would have to be "and local [backup] power failed, and the failovers for that failed too"?? Toshiba manufacture generators too, it's not like they'd need to go far to get backup power designed for them. There must be more to this? (Which explains why people are assuming it's suspicious, I guess; and this site is making 35% of global NAND output). FWIW, I hadn't realised that it takes ~2months to process a wafer in to a chip.
- wyxuan 7y agoYeah I was surprised. Don't they have a ups for this kind of thing?
- HankB99 7y agoYes. My Google-fu is not up to finding them but I happen to be familiar with this company's products and here is one. https://www.energy-xprt.com/products/purewave-ups-systems-559832 https://www.energy-xprt.com/products/purewave-ups-systems-55... One application of this kind of product is chip fabs because they are so sensitive to power disruptions. Whether Toshiba/WD had this type of system and if so, why it didn't prevent loss of product was not mentioned in the linked article. I have heard that there is a glut in chips for SSDs so a reason to cut production can't be ruled out. However it seems like Toshiba/WD would pay the price for this outage while their competitors would reap the benefits (unless the competition agreed to somehow share the cost.)
- pault 7y agoI believe the memory industry is also known for price fixing, so it's not out of the question. The oversupply is ridiculous though; the last time I shopped for an nvme they were $1000 for 500gb and the other day I bought 1TB for $500.
- 7y ago
- ksec 7y agoImportant to quote from the comment section >Five fabs and an R&D center, outage was after the batteries also ran out. For perspective, the batteries at GF's leading fab can run the 1/3 of the systems for only a few minutes. That's the scale we're dealing with. I think before we do all sort of conspiracy theory, we need to look into reason for why was there an outage in Yokkaichi.
- jsjohnst 7y agoBatteries (and giant multi-to. spinning wheels, which serve the same purpose) are not a long term power supply. They are intended to only bridge the couple minutes until generators can come online and provide stable power. So yes, it’s expected that they drained, the question is why didn’t the generators come online?
- ksec 7y agoThat is a good question. but if I had to guess, Judging from the scale, the "Generator" would have to be a power plant? I.e It is not feasible to have generators to operate at this Scale?
- jsjohnst 7y agoFunny you should ask, but the parent company of said fab actually builds and sells generators of the appropriate size. Outside that, have you not seen a multi-building data center complex? The power demands aren’t that different.
- jsjohnst 7y ago> It is not feasible to have generators to operate at this Scale? Getting an exact figure on how much utility power they use is proving difficult, but let’s shoot on the very high side and say it’s 100MW. It’s fairly easy these days to buy generators that put out 10MW of power and are either diesel or natural gas powered. Price wildly varies based on a number of factors, but even on the very high end that would cost $50M for ten such generators. The facility itself was in the multiple billions range to build, so the added cost would be a rounding error. The environmental hazards alone due to losing containment, let alone how much the outage costs in lost business, seems pretty logical to me then that the generators existed. So the question really is, was it incompetence (unexpected failure of backup systems) or malice (good excuse to justify constraining supply)? We will likely never know.
- Rickvst 7y agoThe stock of these companies really took a hit. Sarcasm.