5 ms·
40000 (or even 40960) seems an odd number to fail at. 64k or 32k would make the cause pretty obvious, but 40000 doesn't seem all that round in binary. Perhaps a
by userbinator 4y ago
40000 (or even 40960) seems an odd number to fail at. 64k or 32k would make the cause pretty obvious, but 40000 doesn't seem all that round in binary. Perhaps a 12-bit counter incrementing every 10h? This is puzzling.
Of course, I am also entertaining the possibility that no one thought they would be in use for this long, which would certainly be evidence of planned obsolescence.
- twawaaay 4y agoVery strange understanding of the word "evidence". No sane SSD manufacturer would do such thing on purpose. You do it and you loose business, that's it. The simplest explanation is that somebody made an honest engineering mistake.
- fartcannon 4y agoGiven the power dynamic between a single customer and large corporations, the smart thing to do is to assume malice until prove otherwise. This puts the onus on the corporations and, if we're lucky, creates an environment where they compete with each other to be seen as the most honest. The worst thing that happens is the single customer has to buy an SSD from someone they don't trust. If we do the opposite, as you say, and assume everything is an honest mistake, that puts pressure on the single customer to prove that the organization with a huge marketing budget is doing something wrong. In this situation, the worst thing that happens is we all get taken advantage of. Our collective distrust is the only power we have against massive marketing/PR budgets. It doesn't have to be angry, or sour, or cranky, we just collectively need to not take their word until we have a reason to do so.
- charcircuit 4y agoAre you seriously saying that by default we should believe they intentionally planned to cause their customers to lose all of their data?
- deleted 4y ago[deleted]
- alliao 4y agoplanned obsolescence is quite a thing...?
- dtjb 4y agoIn some cases, but a product must fulfill its core purpose. If a SSD intentionally dumped data and self destructed at a set time, that would be disastrous for the brand. Same way a car doesn't adopt planned obsolescence by blowing up after 200k miles.
- landemva 4y agoYou are not a used car afficionado? 'This insulation prematurely disintegrates under normal use causing the wires it is designed to protect and insulate, to short causing many problems.' http://www.mercedesdefects.com/2008/04/wire-harness-defect.html http://www.mercedesdefects.com/2008/04/wire-harness-defect.h...
- cmeacham98 4y ago> If a SSD intentionally dumped data and self destructed at a set time, that would be disastrous for the brand. Other than "intentionally" (which we cannot know and makes no difference to whether you lose your data or not) that is literally what these SSDs are doing, and no SSD brand has been destroyed over it.
- bmicraft 4y agoWhat more could a manufacturer do to be "disastrous for the brand" than literally build an ssd that stops working after 40kh? Because this does not seem to qualify for you
- bayindirh 4y agoIf a spinning rust can run for ~8 years without any problems,a consumer SSD can hit beyond 40K hours reliably, and everything is checked and tested tens of times because of the complexity of flash storage, I'd get suspicious too. Also, enterprise drives get firmware updates (regardless of spinning or not), and this firmware is automatically applied via RAID controller, so it could be remedied easily before it got this big if it's an actual error.
- bayindirh 4y agoWhen you purchase a server (fleet), you get a long warranty with it. Generally 3 to 5 years. So you expect this fleet to stay in service for <=5 years mostly. Unless you burn through your SSDs, you're very unlikely to hit this event. When these servers' continue to be used and disks all start to fail at the same time, this will obviously stink. The bathtub curve is not like this. You can feel that.
- toast0 4y ago40k hours is a little more than 4.5 years. These drives deterministically fail at that uptime (unless firmware is updated) and most servers are on 24x7, so if you run your servers for 5 years, it's highly likely you'd run into this. If you run your SSDs hard and they fail early as a result, then you'd be spared from this mass death. Or if you use three year leases and replace on a schedule. Now more than ever, five year old server hardware isn't that far behind the curve unless you're on the bleeding edge. I've been looking for bottom of the barrel hosting lately, and there's lots of dedicated servers available with 10+ year old cpus, and probably most of the rest of the machine is a similar age.
- jstrong 4y agoyou have any recs for lower-end bare metal providers?
- toast0 4y agoI haven't used them yet, but take a look at dedicated server offers at LowEndTalk and Web Hosting Talk. Obviously, there's some offers that are sketchier than others, and a lot of resale, but if your needs are low, there's some neat stuff.
- justinsaccount 4y agoSomeone pointed out on the other thread that it could be 2^57 nanoseconds: >>> 2**57/10**9/3600 40031.996687737745
- AaronFriel 4y agoIf it were 53, I'd wonder "are they storing the time in the integer part of a double precision float?" That wouldn't go negative, it'd just start absorbing increments without changing the value. Though that might cause a divide by zero? What could cause unexpected behavior at 57 bits? Perhaps storing fractions of an hour, like incrementing it every 1/16th of an hour and calculating a relative rate of change, causing a divide by zero?
- mkl 4y agoDo embedded CPUs like the one in an SSD have floating point units? It seems more likely to me that the upper bits in a 64 bit integer counter were used for something else.
- danielheath 4y agoPacking a type flag into the upper bits of a 64 bit value is a reasonably common optimisation in dynamic language implementations (because it lets you use unboxed number arithmetic).
- KMag 4y agoOr sometimes the lower bits, as at least used to be the case for integers in v8. (Also OCaml, but that's not dynamically typed. It simplifies the garbage collector to at least some times not require a pointer map for each type, just a flag in the object header to indicate if it contains any pointers, and then everything that isn't ints or pointers needs to be boxed.)
- jonas21 4y agoMy overactive imagination thinks it went something like this: Engineer A: Gee, I need to store a few flags with each block, but there's nowhere to put them. Ah! We're storing timestamps as 64-bit microseconds. I can borrow a few of those bits and there'll still be enough to go for thousands of years without overflowing. Engineer B: Gee, our SSDs are getting so fast, soon we'll be able to hit 1M writes/sec. But we're storing timestamps as microseconds. How can we generate unique timestamps for each write? Ah! I'll switch to nanoseconds. It's a good thing we have plenty of space in this 64-bit int. BOOM!
- tyingq 4y agoFrom a related issue with a different vendor: "The fault fixed by the Dell EMC firmware concerns an Assert function which had a bad check to validate the value of a circular buffer’s index value. Instead of checking the maximum value as N, it checked for N-1. The fix corrects the assert check to use the maximum value as N." https://www.anandtech.com/show/15673/dell-hpe-updates-for-40k-hour-essd-failures https://www.anandtech.com/show/15673/dell-hpe-updates-for-40... Why the MAX value would be in an circular buffer, or what was being stored in N-1? No idea.
- userbinator 4y agocircular buffer The infamous Seagate firmware bug was due to the same thing.
- KMag 4y agoFrom my reading, it's checking the maximum index into the circular buffer. That is, when it hits the end of the circular buffer, there's an assertion to check that they're properly wrapping the index back to the start of the buffer, but the assertion has an off-by-one error. I presume you find a lot of circular buffers in SSD firmware, for wear-leveling reasons. Samsung's NILFS and NILFS2 are structured as circular buffer append-only logs, at least partly to avoid trusting the firmware wear-leveling.
- R0b0t1 4y agoIt's a power of two shifted by a decimal place value instead of a binary one. Unsure why.
- chaboud 4y ago2^57 nanoseconds is ~40032 hours. I wouldn't be surprised if someone out there was counting intervals in a 64-bit value and masking off some of the higher bits for flagging. Any time I see these sorts of issues (odometers that kill themselves, for example) I think of smaller units at higher bit depths. That's not the only way to get to this kind of concern, but it's a way that pretty-darn-competent engineers can leave ticking time bombs due to estimation failures. Always check types for overflow and/or precision loss. Always.