8 ms·

Reverse-engineering and analysis of SanDisk High Endurance microSDXC card (2020)

274 points by userbinator 2y ago

walrus01 2y ago
Previously, Bunnie Huang on the problems of counterfeit sd cards https://www.bunniestudios.com/blog/on-microsd-problems/ https://www.bunniestudios.com/blog/on-microsd-problems/ https://www.bunniestudios.com/blog/2013/on-hacking-microsd-cards/ https://www.bunniestudios.com/blog/2013/on-hacking-microsd-c...
- everybodyknows 2y ago
  2010, and 2013.
mdp2021 2y ago
Related: a few years ago had to select microSD models for reliability (longevity): turned out that because of scarcity of available technical details*, the best indicator seemed to be the length of the warranty (5yrs, 10yrs, lifetime...). Do you have better advice? * cpr the article: > [manufacturer] doesn’t publish any detailed specifications about the cards’ internal workings [...] I shouldn’t have to go this far in hardware reverse-engineering to just ask a simple question of what Flash SanDisk used in their high-endurance card -- About details in the article, instead: it states that 3D NAND should be more reliable (3D has less interference than planar) - but actually I read that 3D NAND can be more fragile physically, and that the bits-per-cell value remains crucial (the less the better)...
- deleted 2y ago
  [deleted]
- muragekibicho 2y ago
  For maximum clarity, is there a positive or negative relationship between warranty length and longevity?
  - Uvix 2y ago
    Positive. A longer warranty means they don’t think it will fail during that period (i.e. higher longevity).
  - ptero 2y ago
    Positive and strongly correlated, as long as the part is not a fake from a fly-by-night shop. I personally buy SD cards and USB memory sticks from bhphotovideo, whose customers (pro and enthusiast photographers), care greatly about not losing their images, so B&H watches their suppliers closely. My 2c.
- megous 2y ago
  That can just as well be marketing or a way to pad the price. 64 GiB microSD still costs ~$14 locally, but the same thing can be had for $4 from China without any warranty. They stick endless limited warranty on it and increase the price so much that it's economical for them to just replace a few cards that people will actually bother to return, and still make profit. And warranty will tell you nothing about quality. In fact on many products you can explicitly choose to pay more for longer warranty.
  - deleted 2y ago
    [deleted]
dekoruotas 2y ago
Very interesting. This longevity aspect frequently comes up in the Wii U modding community. It is tempting to plug in a simple USB stick to store the game files, but, because of the intense read/write nature, it is deemed to be prone to failure. Recently use of high-endurance SD cards has grown, but some say it is still not as safe as an external HDD. It would be interesting to hear thoughts from someone more experienced about the safest storage option as the last thing you want is for your save files to get corrupted.
- Neywiny 2y ago
  Safe is a multifaceted term. Essentially these storage media (I'm more experienced with solid state but HDDs may be included) the probability the data you wrote is the data you read is a function of how many program/erase cycles, how long ago that was, and naturally the part's specifics. For example, a lot of NOR flash is rated to 10 years at up to 100k cycles. But devices > 10 years old rarely make the news for their flash being dead. On the other hand, I believe there was a Tesla fiasco where their logs were wearing out the flash prematurely. There's usually trends to look for in regards to that third factor. The lower the # of bits per cell, the higher probability the voltage level is still working the right range. Which is why so much flash is still SLC or pSLC capable. Usually this is more industrial. Then you have entirely different technologies altogether. NVRAM/FRAM/MRAM are various terms for extremely high (or infinite) read/write technologies while still being non-volatile (keeps its data with power off). I don't know how much of a drop in replacement those are. I think LTT had one of those on a flash drive a while back https://youtu.be/oJ5fFph0AEM https://youtu.be/oJ5fFph0AEM, but it's so low capacity it'll probably be useless. It may be possible to hack something up with a MR5A16A. It's a whole 4 MB but has unlimited endurance and over 20 years of endurance. It looks like it has more of an SRAM interface than NAND, but should be capable of saturating a USB high speed link. The drive would likely cost $75? TBH if there was a market it may be a fun project. If you sacrifice some endurance you can go up to 1Gb per device which might be interesting. But the cost scales.
  - Scoundreller 2y ago
    > But devices > 10 years old rarely make the news for their flash being dead. Accelerated stability testing is fraught with potential issues, and any output is intentionally conservative. An issue with estimating lifespan on new products is that they'll expose them to more extreme conditions, but those more extreme conditions may trigger (exponentially faster) higher order reactions that are relative non-issues at regular conditions. Then you have things like activation energy requirements for a reaction that just might not be met at regular conditions, but happen at higher temperatures. And an IC is quite the soup of molecules in varying combinations unlike a straightforward solution.
PaulKeeble 2y ago
Its really hard to compare SD cards and especially durability because we get no information as to what they are doing differently. You can get a better idea of the performance characteristics than the broad categories (A1 or A2, largely useless) with a review on storagereview but they don't have anything further or a way to compare durability. It matters less in cameras but for Single board computers or dashcam uses it would be nice to have a better idea of the practical durability and the usage pattern that would preserve it.
thrdbndndn 2y ago
I don’t quite understand the sarcastic, if not slightly passive-aggressive, tone in this article. This kind of attitude seems quite common in hacking and reverse engineering write-ups. In my opinion, manufacturers aren’t obligated to disclose every technical detail beyond what a typical spec sheet would cover, such as the specific type of flash used. It’s incredibly impressive that someone as talented as the author can figure it out independently, but I don’t see why there’s a need for the constant tone of frustration or an “us vs. the company” mindset throughout the article.
- foxhop 2y ago
  When i buy 512G microsd it becomes my property. If i need to repair or replace my property, it should be disclosed what technology i bought, so that i or my repairman could understand my property enough to decide a proper path forward depending on my data & other environmental devices. The author needed to reverse engineer what could have been on the spec sheet...
- bean-weevil 2y ago
  As someone who's been in a position like this before, I suspect the author is angry at the manufacturers and feels they're not being honest by not providing proper specs. It's hard to spec out a project when no one will tell you exactly what their parts can do. It's like if you were trying to buy a light truck for work and intsead of telling you how much weight it could pull or how often maintenance would be needed, the manufacturer refused to say anything more than "it can tow a speedboat" and "it requires infrequent maintenance."
- hilbert42 2y ago
  The reason why we know how these media storage devices work is that our valuable data is being stored on them! We need to know how fragile that storage is and the only way we can do that is to have all the engineering data/test info. SSD and MicroSD/thumb drive and even HD manufacturers have a damn hide by being so secretive about their devices—of course it's never the manufactures who suffer the burden of data loss, it's the customer. What's desperately needed are open-source manufacturers who will publish the necessary data. This problem isn't new, sleazebag Kodak knowingly released shoddy unstable color stock in the 1950s and decades later precious family and wedding photos had faded to nothing. Let that be a lesson, this solid-state storage shit hasn't been around long enough yet to know whether we'll be seeing a repeat of that Kodak fuckup.
jqpabc123 2y ago
My experience: 1) All microSD cards have a high failure rate. 2) Cheap cards sourced directly from China are usually fake and will fail a format test. I tried multiple times and could never find a reliable source. 3) MS Bitlocker is a security measure that also seems to serve as a reliability test. Cards with physical memory reliability issues seem more prone to write failures with Bitlocker. It's better to know about this up front. 4) If your data is really important, always make a backup copy of any card. 5) Physically, microSD cards are fairly durable and even water resistant thanks to being encased in epoxy. I have one on my watch strap and take it wherever I go, even the shower and pool. Just make sure it is dry before plugging it in. https://www.thingiverse.com/thing:6784665 https://www.thingiverse.com/thing:6784665
- matsemann 2y ago
  Ref 2: if you buy from Amazon, you never know if you get a genuine card. Even if you shop from an official seller, as inventory is commingled so you might as well get a fake one from another seller with the same SKU. Big problem in action cam related circles, where lots of people have broken recordings due to fake cards.
  - everybodyknows 2y ago
    Are you saying that the counterfeiting of name brand packaging is so skillful that even careful inspection by us the consumers cannot reasonably hope to detect it?
    - AlexandrB 2y ago
      Unless you have the real and the fake side by side it can be really hard.
      - dboreham 2y ago
        Even then it's hard because the legit vendors change their packaging frequently.
    - 2y ago
casenmgreen 2y ago
Very good work.
takeda-kun 2y ago
MLC is the opposite of "high-endurance".
- jsheard 2y ago
  It's high endurance compared to TLC and QLC, which make up >99.9% of flash sold nowadays. Sure SLC is even better but it's all relative.
_benj 2y ago
Something I found interesting when learning about flash was how it is manufactured and how much reliable single layer cells are (SLC) There’s certainly a very visible increase in price and decrease in capacity but it’s certainly interesting when you get sd cards with a proper datasheet vs. customer level devices https://www.digikey.com/en/products/filter/memory-cards/501?s=N4IgjCBcoKwOwAYqgMZQC4CcCuBTANCAPZQDaIALAgMwzVwgC6hADulCAMpYCWAdgHMQAX0IBaAEzIQaSFjyESkcjCbDRIAGzSeAEw5iwSVu0ggQhdAE8WuDiOFA https://www.digikey.com/en/products/filter/memory-cards/501?...
- error9397 2y ago
  These don't use SLC, they use TLC. SLC is hard to find in SD cards and SSDs due to cost.
  - l11r 2y ago
    In the provided link price seems to be reasonable for SLC, no? 15 bucks for 512 MB (not GB). Also datashits clearly say it's SLC and not TLC: https://static6.arrow.com/aropdfconversion/55056b1a2a966f599450d16486375ea493d946b4/atp_microsd_datasheet.pdf https://static6.arrow.com/aropdfconversion/55056b1a2a966f599...
- msarnoff 2y ago
  Most eMMC chips (basically the chip version of an SD card) can be configured to work in pseudo-SLC (pSLC) mode. This halves the capacity but improves the write endurance by several times.
  - anthk 2y ago
    Guess what the PocketCHIP guys did in a upgrade. Exactly the oppsite. Yes, ubifs died every time.
    - Dylan16807 2y ago
      They switched it from SLC to MLC? I wouldn't expect 2 bits per cell to cause problems... how bad was the flash? Unless you're saying the switch corrupted the existing data, in which case wow what a screwup.
las_balas_tres 2y ago
We have hundreds of raspberry pi s out in the field that experience sdcard failures far too often even though the filesystems are setup as readonly using a ram based overlay filesystem. I suspect something happens during reboot of these pi's and that the SoC generates spurious signals on the bus lines causing havoc with some of the sdcards. It doesn't seem to happen with high end cards or the same setup but with a compute module (which we now use exclusively) with an on board emmc chip.
- dezgeg 2y ago
  Are there other SBCs / SoMs that are as widely used with SD cards though? That's certainly a source of bias.
  - imtringued 2y ago
    99% of devices that use SD cards are battery powered. It's an issue with the power supply.
- OptionOfT 2y ago
  This is interesting, I got gifted an one of those hydroponic systems. The thing was 2 years old. Runs on a Raspberry Zero. The issue? SD card corrupt. I got an image from someone else and fixed it with that. What's worse is that these things are connected to the internet (with VNC installed!), and they don't do updates... The system is awesome, but I VERY quickly moved it to a separate VLAN.
- lysace 2y ago
  These haven't failed for me: https://www.digikey.com/en/product-highlight/a/atp/advancedmlc-amlc-microsd-microsdhc-cards https://www.digikey.com/en/product-highlight/a/atp/advancedm... I'm cynically guessing it's down to a relatively simple workaround in the SD card firmware that fixes this very common corruption on power loss while writing. Or maybe that combined with a 0.1 cent capacitor.
- lxgr 2y ago
  Writes aren't the only things that can degrade data stored on NAND flash. Not sure how SD cards are mitigating read disturb errors, for example.
ck2 2y ago
When are we going to get tiny RAID10 microSDXC USB3 devices? Just plug in four microsd cards and the hardware takes care of the mirroring and rebuilding when dead card removed+replaced Be sure to use cards from different batches or even makes so they don't all fail at the same time.
- Koshkin 2y ago
  Wouldn't a USB hub and, say, BTRFS do the job?
  - doubled112 2y ago
    It would work. I RAIDed a bunch of cheap USB 2.0 flash drives on a hub with MDRAID as a learning tool back in the day. It was horrendously unreliable. USB wasn’t a good choice for storage back then, and I’m convinced the hub had issues. This would work much better now. I did, however, get to watch the blinkin lights, learn how to recover from failures, and discover quite a few gotchas.
    - aaronmdjones 2y ago
      MDRAID is good for availability and fault tolerance but no good for integrity. For example, in a RAID-1, if one of the drives has a silently corrupted block, MDRAID will happily return that if you're unlucky enough for it to decide to use that drive to satisfy that read request. If you have error detection at a higher level, you might start pulling all but one drive from the array at a time and re-issuing the read request until it gives you bad data again (then you know which drive is bad). If you have an 8-drive RAID-6 and one of the data blocks in a stripe is corrupt, again, it will happily return that (it won't even read the parity blocks, because every drive is present). Again you would have to pull one drive at a time and re-issue the read request until you get back good data, assuming you have a way to know that (e.g. a Zip archive with a CRC32). If you're still getting bad data, you didn't pull the bad drive; re-add it and pull the next one. This would happen when you pull the drive with the corrupted block, because then it would calculate what that block was supposed to contain based on the parity in that stripe. Most distros have something akin to a monthly scrub job, where MDRAID will confirm mirrors against each other and parity against data. Unfortunately this will only tell you when they don't agree; it's still your responsibility to identify which drive is at fault and correct it (by pulling the corrupted drive from the array, nuking the MDRAID metadata on it, and re-adding it, simulating a drive replacement and thus array rebuild). Worse still, the RAID-5 and RAID-6 levels in MDRAID have a "repair" action in addition to the "check" action that detects the above. This doesn't do what you think it does; instead it just iterates every stripe in the array and recalculates the parity blocks based on the data blocks, writing new parity back. Thus you lose the above option of pulling 1 drive at a time because now your parity is for corrupted data. You need a filesystem that detects and repairs corruption. Btrfs and ZFS both do this, but Btrfs' multi-device RAID support is (still) explicitly marked experimental and for throwaway data only. In ZFS, you can either do mirroring (equivalent to an MDRAID RAID-1), a RAID-Z (equivalent to an MDRAID RAID-5 in practice but not implementation), a RAID-Z2 (RAID-6), a RAID-Z3 (no MDRAID equivalent), set the "copies" property which writes the same data multiple times (e.g. creating a 1 GiB file with copies=2 uses 2 GiB of filesystem free space while still reporting that the file is 1 GiB in size), or some combination thereof. ZFS checksums everything (data and metadata blocks), and every read request (for metadata or data) is confirmed against the checksum when it is performed. If it does detect checksum mismatch and it has options for recovery (RAID-Z parity or another drive in a mirror or an extra filesystem-level copy created by the "copies" property being greater than 1), it will automatically correct this corruption and then return good data. If it doesn't, it will NOT return corrupted data; it will return -EIO. Better still, checksums are also metadata, so they are also replicated if you have any of the above topologies (except copies=). This protects against corruption that destroys a checksum (which would ordinarily prevent a read) rather than destroying (meta)data. A ZFS scrub will similarly detect all instances of checksum and (meta)data mismatch and automatically correct any corruption. Better still, a ZFS scrub only needs to operate on filesystem-level allocated space, not every stripe or block. tl;dr: Don't use MDRAID on questionable storage media. It won't go well.
error9397 2y ago
I'm a bit disappointed Industrial XI Ultra cards weren't evaluated.
TacticalCoder 2y ago
[dead]
bityard 2y ago
Every time Raspberry Pi comes up as a topic on HN, there are scores of comments decrying the futility and frustration of unreliable SD cards. The thing with SD cards is that there are wide variances in quality, even among the top name-brand manufacturers. You can get the normal crappy cards that everyone buys which are likely to last not much more than 2-3 years under very occasional use all the way up to industrial cards with 10 year warranties and reliability matching the best enterprise SSDs. And everything in between. The point is: If you buy the cheapest thing on Amazon, you're going to get exactly what you paid for. For my part, I have been using "High Endurance" cards in my Pis for several years as normal OS and data disks. They are not actually much more expensive than "normal" cards. I have not had any of them fail yet.
- spintin 2y ago
  [dead]
- NikolaNovak 2y ago
  I don't disagree with you, but the other perennial unanswered question on HN is: how do I ensure I'm paying extra for actual quality and not just sucker tax? Memory cards and ssds are famously obtuse like this. There are the branded versions (custom made for your Switch, for your Xbox,etc) which are exactly the same as their cheaper counterparts. Sandisk itself recently started a "creator" line of cards and ssds which are, again, exactly the same but more expensive. Samsung had Evo Plus which was different than Evo+ but same as Evo Select. Go figure! Sometimes looking at specs helps but usually not - I.e. The "up to" speed marks are famously irrelevant for real time usage, and brand name companies will offer any number of cards with same specs and tags (u2/v3/class10/mark5/whatever) at varying price points. And then there's the WD Red saga where drives branded "NAS" were completely inappropriate for NAS usage. I ran a photography business a while back and always bought extreme pro because I just couldn't risk it, but honestly, I felt like a bit of a sucker. It's hard to know when you're actually buying real quality and endurance increase.
  - ploxiln 2y ago
    Looking for the most "extreme" high-quality micro-sd cards on Amazon, and also on B&H to avoid the chance of counterfeits, I'm struck by how they are all so damn cheap. Whether SanDisk Ultra, Extreme, Extreme Plus, Extreme Pro, or Max Endurance, the 64GB card is less than $20. (I've seen reports of Sandisk Extreme micro-sd cards dying, at close to the "typical raspi" rate ... I haven't seen that yet for Max Endurance, and that does sound like what I want, so I got one and I'm trying it. But it bothered me, ironically, that it's pretty much the same price.) Meanwhile, if you go to a professional electronics component vendor like Mouser or Newark, a SwissBit "pseudo-SLC" micro-sd card is over $50 for 32 GB. I'm pretty sure that is real industrial usage quality, and will have similar longevity to a "real SSD" of similar size. (I also got one of these, it works well so far, but that's not a conclusive test yet :) I guess I could instead get the 512 GB "Sandisk Extreme" whatever, and the additional size will probably give similar longevity? It just seems silly, and a bit less of a sure thing, than the priced-as-expected industrial thing. It is kinda strange/annoying that in the consumer side of the market, they keep adding superlatives, but it seems like the same cheap flash for the same cheap price? The real stuff surely costs more than $16, but it's not on the consumer market, at all. And why doesn't the Raspi foundation/company sell some of the high-priced industrial stuff, just to make it an obvious option, even if not for most people? If you're just messing around and learning, use the cheap one, but if you want a raspi to do real work for more than 4 months straight, you're gonna want one of these real industrial SD cards. I guess most people who care about storage use m.2 adapters, which you might want for the performance, or the better pricing at higher capacity. But if you don't need the performance or capacity, or don't have room for the m.2 in your application, and just need the damn thing to not break ... the real "extreme" does exist, it is the SwissBit.
spintin 2y ago
[dead]
wvh 2y ago
There's a lack of transparency on what you're actually getting with storage these days. Akin to the actual flash architecture, I've been trying to find information on the level of OPAL/Sed support in SSDs and many manufacturers don't even mention that information anymore, nor do help desk people have any useful information. The whole flash industry has a shady feel to it when it comes to pricing, capabilities and reliability, not to mention the sheer amount of counterfeit productions flooding the market. That's a bit bizarre, data security is pretty paramount, this is not an industry that should be clouded in a shroud of secrecy and shadiness.
- bayindirh 2y ago
  From what I understand, Flash manufacturing and management during operation are black arts at this point and nobody wants to give the smallest clue for anything related to Flash management and specifications. For example, Crucial (Micron) doesn't give any TBW for their external SSDs, giving indications that they're using mixed-binning as long as the speed specs are satisfied. Same for lower level Kingston (NV series) SSDs. At least they openly say that the drives can wildly differ from batch to batch (and oh boy, they do). As the industry is pinched by customers for more capacity, QLC becomes the norm, TBW numbers are continuously hidden, and you're only left with the internal endurance counters, if you can access them. Controllers are in the same boat. Wear leveling, write amplification, flash monitoring are all left to whims of the controller designers, resulting in strange situations. Off-shift/dark production is another problem, but it's not new by any means. BunnyStudios has a great write up which has been linked a couple of times.
  - userbinator 2y ago
    The datasheets for the early flash memories (SLC) were easily available and loudly proclaimed 100K or even 1M cycles of endurance and 10-20 years of retention. Now they're very difficult to obtain and are rated for only a few hundred cycles and a few months of retention. No wonder they're secretive about it.
    - bayindirh 2y ago

274 points by userbinator 2y ago

Previously, Bunnie Huang on the problems of counterfeit sd cards https://www.bunniestudios.com/blog/on-microsd-problems/ https://www.bunniestudios.com/blog/on-microsd-problems/ https://www.bunniestudios.com/blog/2013/on-hacking-microsd-cards/ https://www.bunniestudios.com/blog/2013/on-hacking-microsd-c...

everybodyknows 2y ago
2010, and 2013.

mdp2021 2y ago

Related: a few years ago had to select microSD models for reliability (longevity): turned out that because of scarcity of available technical details*, the best indicator seemed to be the length of the warranty (5yrs, 10yrs, lifetime...). Do you have better advice? * cpr the article: > [manufacturer] doesn’t publish any detailed specifications about the cards’ internal workings [...] I shouldn’t have to go this far in hardware reverse-engineering to just ask a simple question of what Flash SanDisk used in their high-endurance card -- About details in the article, instead: it states that 3D NAND should be more reliable (3D has less interference than planar) - but actually I read that 3D NAND can be more fragile physically, and that the bits-per-cell value remains crucial (the less the better)...

deleted 2y ago
[deleted]
muragekibicho 2y ago
For maximum clarity, is there a positive or negative relationship between warranty length and longevity?
- Uvix 2y ago
  Positive. A longer warranty means they don’t think it will fail during that period (i.e. higher longevity).
- ptero 2y ago
  Positive and strongly correlated, as long as the part is not a fake from a fly-by-night shop. I personally buy SD cards and USB memory sticks from bhphotovideo, whose customers (pro and enthusiast photographers), care greatly about not losing their images, so B&H watches their suppliers closely. My 2c.
megous 2y ago
That can just as well be marketing or a way to pad the price. 64 GiB microSD still costs ~$14 locally, but the same thing can be had for $4 from China without any warranty. They stick endless limited warranty on it and increase the price so much that it's economical for them to just replace a few cards that people will actually bother to return, and still make profit. And warranty will tell you nothing about quality. In fact on many products you can explicitly choose to pay more for longer warranty.
- deleted 2y ago
  [deleted]

dekoruotas 2y ago

Very interesting. This longevity aspect frequently comes up in the Wii U modding community. It is tempting to plug in a simple USB stick to store the game files, but, because of the intense read/write nature, it is deemed to be prone to failure. Recently use of high-endurance SD cards has grown, but some say it is still not as safe as an external HDD. It would be interesting to hear thoughts from someone more experienced about the safest storage option as the last thing you want is for your save files to get corrupted.

Neywiny 2y ago
Safe is a multifaceted term. Essentially these storage media (I'm more experienced with solid state but HDDs may be included) the probability the data you wrote is the data you read is a function of how many program/erase cycles, how long ago that was, and naturally the part's specifics. For example, a lot of NOR flash is rated to 10 years at up to 100k cycles. But devices > 10 years old rarely make the news for their flash being dead. On the other hand, I believe there was a Tesla fiasco where their logs were wearing out the flash prematurely. There's usually trends to look for in regards to that third factor. The lower the # of bits per cell, the higher probability the voltage level is still working the right range. Which is why so much flash is still SLC or pSLC capable. Usually this is more industrial. Then you have entirely different technologies altogether. NVRAM/FRAM/MRAM are various terms for extremely high (or infinite) read/write technologies while still being non-volatile (keeps its data with power off). I don't know how much of a drop in replacement those are. I think LTT had one of those on a flash drive a while back https://youtu.be/oJ5fFph0AEM https://youtu.be/oJ5fFph0AEM, but it's so low capacity it'll probably be useless. It may be possible to hack something up with a MR5A16A. It's a whole 4 MB but has unlimited endurance and over 20 years of endurance. It looks like it has more of an SRAM interface than NAND, but should be capable of saturating a USB high speed link. The drive would likely cost $75? TBH if there was a market it may be a fun project. If you sacrifice some endurance you can go up to 1Gb per device which might be interesting. But the cost scales.
- Scoundreller 2y ago
  > But devices > 10 years old rarely make the news for their flash being dead. Accelerated stability testing is fraught with potential issues, and any output is intentionally conservative. An issue with estimating lifespan on new products is that they'll expose them to more extreme conditions, but those more extreme conditions may trigger (exponentially faster) higher order reactions that are relative non-issues at regular conditions. Then you have things like activation energy requirements for a reaction that just might not be met at regular conditions, but happen at higher temperatures. And an IC is quite the soup of molecules in varying combinations unlike a straightforward solution.

PaulKeeble 2y ago

Its really hard to compare SD cards and especially durability because we get no information as to what they are doing differently. You can get a better idea of the performance characteristics than the broad categories (A1 or A2, largely useless) with a review on storagereview but they don't have anything further or a way to compare durability. It matters less in cameras but for Single board computers or dashcam uses it would be nice to have a better idea of the practical durability and the usage pattern that would preserve it.

thrdbndndn 2y ago

I don’t quite understand the sarcastic, if not slightly passive-aggressive, tone in this article. This kind of attitude seems quite common in hacking and reverse engineering write-ups. In my opinion, manufacturers aren’t obligated to disclose every technical detail beyond what a typical spec sheet would cover, such as the specific type of flash used. It’s incredibly impressive that someone as talented as the author can figure it out independently, but I don’t see why there’s a need for the constant tone of frustration or an “us vs. the company” mindset throughout the article.

foxhop 2y ago
When i buy 512G microsd it becomes my property. If i need to repair or replace my property, it should be disclosed what technology i bought, so that i or my repairman could understand my property enough to decide a proper path forward depending on my data & other environmental devices. The author needed to reverse engineer what could have been on the spec sheet...
bean-weevil 2y ago
As someone who's been in a position like this before, I suspect the author is angry at the manufacturers and feels they're not being honest by not providing proper specs. It's hard to spec out a project when no one will tell you exactly what their parts can do. It's like if you were trying to buy a light truck for work and intsead of telling you how much weight it could pull or how often maintenance would be needed, the manufacturer refused to say anything more than "it can tow a speedboat" and "it requires infrequent maintenance."
hilbert42 2y ago
The reason why we know how these media storage devices work is that our valuable data is being stored on them! We need to know how fragile that storage is and the only way we can do that is to have all the engineering data/test info. SSD and MicroSD/thumb drive and even HD manufacturers have a damn hide by being so secretive about their devices—of course it's never the manufactures who suffer the burden of data loss, it's the customer. What's desperately needed are open-source manufacturers who will publish the necessary data. This problem isn't new, sleazebag Kodak knowingly released shoddy unstable color stock in the 1950s and decades later precious family and wedding photos had faded to nothing. Let that be a lesson, this solid-state storage shit hasn't been around long enough yet to know whether we'll be seeing a repeat of that Kodak fuckup.

jqpabc123 2y ago

My experience: 1) All microSD cards have a high failure rate. 2) Cheap cards sourced directly from China are usually fake and will fail a format test. I tried multiple times and could never find a reliable source. 3) MS Bitlocker is a security measure that also seems to serve as a reliability test. Cards with physical memory reliability issues seem more prone to write failures with Bitlocker. It's better to know about this up front. 4) If your data is really important, always make a backup copy of any card. 5) Physically, microSD cards are fairly durable and even water resistant thanks to being encased in epoxy. I have one on my watch strap and take it wherever I go, even the shower and pool. Just make sure it is dry before plugging it in. https://www.thingiverse.com/thing:6784665 https://www.thingiverse.com/thing:6784665

matsemann 2y ago
Ref 2: if you buy from Amazon, you never know if you get a genuine card. Even if you shop from an official seller, as inventory is commingled so you might as well get a fake one from another seller with the same SKU. Big problem in action cam related circles, where lots of people have broken recordings due to fake cards.
- everybodyknows 2y ago
  Are you saying that the counterfeiting of name brand packaging is so skillful that even careful inspection by us the consumers cannot reasonably hope to detect it?
  - AlexandrB 2y ago
    Unless you have the real and the fake side by side it can be really hard.
    - dboreham 2y ago
      Even then it's hard because the legit vendors change their packaging frequently.
  - 2y ago

casenmgreen 2y ago

Very good work.

takeda-kun 2y ago

MLC is the opposite of "high-endurance".

jsheard 2y ago
It's high endurance compared to TLC and QLC, which make up >99.9% of flash sold nowadays. Sure SLC is even better but it's all relative.

_benj 2y ago

Something I found interesting when learning about flash was how it is manufactured and how much reliable single layer cells are (SLC) There’s certainly a very visible increase in price and decrease in capacity but it’s certainly interesting when you get sd cards with a proper datasheet vs. customer level devices https://www.digikey.com/en/products/filter/memory-cards/501?s=N4IgjCBcoKwOwAYqgMZQC4CcCuBTANCAPZQDaIALAgMwzVwgC6hADulCAMpYCWAdgHMQAX0IBaAEzIQaSFjyESkcjCbDRIAGzSeAEw5iwSVu0ggQhdAE8WuDiOFA https://www.digikey.com/en/products/filter/memory-cards/501?...

error9397 2y ago
These don't use SLC, they use TLC. SLC is hard to find in SD cards and SSDs due to cost.
- l11r 2y ago
  In the provided link price seems to be reasonable for SLC, no? 15 bucks for 512 MB (not GB). Also datashits clearly say it's SLC and not TLC: https://static6.arrow.com/aropdfconversion/55056b1a2a966f599450d16486375ea493d946b4/atp_microsd_datasheet.pdf https://static6.arrow.com/aropdfconversion/55056b1a2a966f599...
msarnoff 2y ago
Most eMMC chips (basically the chip version of an SD card) can be configured to work in pseudo-SLC (pSLC) mode. This halves the capacity but improves the write endurance by several times.
- anthk 2y ago
  Guess what the PocketCHIP guys did in a upgrade. Exactly the oppsite. Yes, ubifs died every time.
  - Dylan16807 2y ago
    They switched it from SLC to MLC? I wouldn't expect 2 bits per cell to cause problems... how bad was the flash? Unless you're saying the switch corrupted the existing data, in which case wow what a screwup.

las_balas_tres 2y ago

We have hundreds of raspberry pi s out in the field that experience sdcard failures far too often even though the filesystems are setup as readonly using a ram based overlay filesystem. I suspect something happens during reboot of these pi's and that the SoC generates spurious signals on the bus lines causing havoc with some of the sdcards. It doesn't seem to happen with high end cards or the same setup but with a compute module (which we now use exclusively) with an on board emmc chip.

dezgeg 2y ago
Are there other SBCs / SoMs that are as widely used with SD cards though? That's certainly a source of bias.
- imtringued 2y ago
  99% of devices that use SD cards are battery powered. It's an issue with the power supply.
OptionOfT 2y ago
This is interesting, I got gifted an one of those hydroponic systems. The thing was 2 years old. Runs on a Raspberry Zero. The issue? SD card corrupt. I got an image from someone else and fixed it with that. What's worse is that these things are connected to the internet (with VNC installed!), and they don't do updates... The system is awesome, but I VERY quickly moved it to a separate VLAN.
lysace 2y ago
These haven't failed for me: https://www.digikey.com/en/product-highlight/a/atp/advancedmlc-amlc-microsd-microsdhc-cards https://www.digikey.com/en/product-highlight/a/atp/advancedm... I'm cynically guessing it's down to a relatively simple workaround in the SD card firmware that fixes this very common corruption on power loss while writing. Or maybe that combined with a 0.1 cent capacitor.
lxgr 2y ago
Writes aren't the only things that can degrade data stored on NAND flash. Not sure how SD cards are mitigating read disturb errors, for example.

ck2 2y ago

When are we going to get tiny RAID10 microSDXC USB3 devices? Just plug in four microsd cards and the hardware takes care of the mirroring and rebuilding when dead card removed+replaced Be sure to use cards from different batches or even makes so they don't all fail at the same time.

Koshkin 2y ago
Wouldn't a USB hub and, say, BTRFS do the job?
- doubled112 2y ago
  It would work. I RAIDed a bunch of cheap USB 2.0 flash drives on a hub with MDRAID as a learning tool back in the day. It was horrendously unreliable. USB wasn’t a good choice for storage back then, and I’m convinced the hub had issues. This would work much better now. I did, however, get to watch the blinkin lights, learn how to recover from failures, and discover quite a few gotchas.
  - aaronmdjones 2y ago
    MDRAID is good for availability and fault tolerance but no good for integrity. For example, in a RAID-1, if one of the drives has a silently corrupted block, MDRAID will happily return that if you're unlucky enough for it to decide to use that drive to satisfy that read request. If you have error detection at a higher level, you might start pulling all but one drive from the array at a time and re-issuing the read request until it gives you bad data again (then you know which drive is bad). If you have an 8-drive RAID-6 and one of the data blocks in a stripe is corrupt, again, it will happily return that (it won't even read the parity blocks, because every drive is present). Again you would have to pull one drive at a time and re-issue the read request until you get back good data, assuming you have a way to know that (e.g. a Zip archive with a CRC32). If you're still getting bad data, you didn't pull the bad drive; re-add it and pull the next one. This would happen when you pull the drive with the corrupted block, because then it would calculate what that block was supposed to contain based on the parity in that stripe. Most distros have something akin to a monthly scrub job, where MDRAID will confirm mirrors against each other and parity against data. Unfortunately this will only tell you when they don't agree; it's still your responsibility to identify which drive is at fault and correct it (by pulling the corrupted drive from the array, nuking the MDRAID metadata on it, and re-adding it, simulating a drive replacement and thus array rebuild). Worse still, the RAID-5 and RAID-6 levels in MDRAID have a "repair" action in addition to the "check" action that detects the above. This doesn't do what you think it does; instead it just iterates every stripe in the array and recalculates the parity blocks based on the data blocks, writing new parity back. Thus you lose the above option of pulling 1 drive at a time because now your parity is for corrupted data. You need a filesystem that detects and repairs corruption. Btrfs and ZFS both do this, but Btrfs' multi-device RAID support is (still) explicitly marked experimental and for throwaway data only. In ZFS, you can either do mirroring (equivalent to an MDRAID RAID-1), a RAID-Z (equivalent to an MDRAID RAID-5 in practice but not implementation), a RAID-Z2 (RAID-6), a RAID-Z3 (no MDRAID equivalent), set the "copies" property which writes the same data multiple times (e.g. creating a 1 GiB file with copies=2 uses 2 GiB of filesystem free space while still reporting that the file is 1 GiB in size), or some combination thereof. ZFS checksums everything (data and metadata blocks), and every read request (for metadata or data) is confirmed against the checksum when it is performed. If it does detect checksum mismatch and it has options for recovery (RAID-Z parity or another drive in a mirror or an extra filesystem-level copy created by the "copies" property being greater than 1), it will automatically correct this corruption and then return good data. If it doesn't, it will NOT return corrupted data; it will return -EIO. Better still, checksums are also metadata, so they are also replicated if you have any of the above topologies (except copies=). This protects against corruption that destroys a checksum (which would ordinarily prevent a read) rather than destroying (meta)data. A ZFS scrub will similarly detect all instances of checksum and (meta)data mismatch and automatically correct any corruption. Better still, a ZFS scrub only needs to operate on filesystem-level allocated space, not every stripe or block. tl;dr: Don't use MDRAID on questionable storage media. It won't go well.

error9397 2y ago

I'm a bit disappointed Industrial XI Ultra cards weren't evaluated.

TacticalCoder 2y ago

[dead]

bityard 2y ago

Every time Raspberry Pi comes up as a topic on HN, there are scores of comments decrying the futility and frustration of unreliable SD cards. The thing with SD cards is that there are wide variances in quality, even among the top name-brand manufacturers. You can get the normal crappy cards that everyone buys which are likely to last not much more than 2-3 years under very occasional use all the way up to industrial cards with 10 year warranties and reliability matching the best enterprise SSDs. And everything in between. The point is: If you buy the cheapest thing on Amazon, you're going to get exactly what you paid for. For my part, I have been using "High Endurance" cards in my Pis for several years as normal OS and data disks. They are not actually much more expensive than "normal" cards. I have not had any of them fail yet.

spintin 2y ago
[dead]
NikolaNovak 2y ago
I don't disagree with you, but the other perennial unanswered question on HN is: how do I ensure I'm paying extra for actual quality and not just sucker tax? Memory cards and ssds are famously obtuse like this. There are the branded versions (custom made for your Switch, for your Xbox,etc) which are exactly the same as their cheaper counterparts. Sandisk itself recently started a "creator" line of cards and ssds which are, again, exactly the same but more expensive. Samsung had Evo Plus which was different than Evo+ but same as Evo Select. Go figure! Sometimes looking at specs helps but usually not - I.e. The "up to" speed marks are famously irrelevant for real time usage, and brand name companies will offer any number of cards with same specs and tags (u2/v3/class10/mark5/whatever) at varying price points. And then there's the WD Red saga where drives branded "NAS" were completely inappropriate for NAS usage. I ran a photography business a while back and always bought extreme pro because I just couldn't risk it, but honestly, I felt like a bit of a sucker. It's hard to know when you're actually buying real quality and endurance increase.
- ploxiln 2y ago
  Looking for the most "extreme" high-quality micro-sd cards on Amazon, and also on B&H to avoid the chance of counterfeits, I'm struck by how they are all so damn cheap. Whether SanDisk Ultra, Extreme, Extreme Plus, Extreme Pro, or Max Endurance, the 64GB card is less than $20. (I've seen reports of Sandisk Extreme micro-sd cards dying, at close to the "typical raspi" rate ... I haven't seen that yet for Max Endurance, and that does sound like what I want, so I got one and I'm trying it. But it bothered me, ironically, that it's pretty much the same price.) Meanwhile, if you go to a professional electronics component vendor like Mouser or Newark, a SwissBit "pseudo-SLC" micro-sd card is over $50 for 32 GB. I'm pretty sure that is real industrial usage quality, and will have similar longevity to a "real SSD" of similar size. (I also got one of these, it works well so far, but that's not a conclusive test yet :) I guess I could instead get the 512 GB "Sandisk Extreme" whatever, and the additional size will probably give similar longevity? It just seems silly, and a bit less of a sure thing, than the priced-as-expected industrial thing. It is kinda strange/annoying that in the consumer side of the market, they keep adding superlatives, but it seems like the same cheap flash for the same cheap price? The real stuff surely costs more than $16, but it's not on the consumer market, at all. And why doesn't the Raspi foundation/company sell some of the high-priced industrial stuff, just to make it an obvious option, even if not for most people? If you're just messing around and learning, use the cheap one, but if you want a raspi to do real work for more than 4 months straight, you're gonna want one of these real industrial SD cards. I guess most people who care about storage use m.2 adapters, which you might want for the performance, or the better pricing at higher capacity. But if you don't need the performance or capacity, or don't have room for the m.2 in your application, and just need the damn thing to not break ... the real "extreme" does exist, it is the SwissBit.

spintin 2y ago

[dead]

wvh 2y ago

There's a lack of transparency on what you're actually getting with storage these days. Akin to the actual flash architecture, I've been trying to find information on the level of OPAL/Sed support in SSDs and many manufacturers don't even mention that information anymore, nor do help desk people have any useful information. The whole flash industry has a shady feel to it when it comes to pricing, capabilities and reliability, not to mention the sheer amount of counterfeit productions flooding the market. That's a bit bizarre, data security is pretty paramount, this is not an industry that should be clouded in a shroud of secrecy and shadiness.

bayindirh 2y ago
From what I understand, Flash manufacturing and management during operation are black arts at this point and nobody wants to give the smallest clue for anything related to Flash management and specifications. For example, Crucial (Micron) doesn't give any TBW for their external SSDs, giving indications that they're using mixed-binning as long as the speed specs are satisfied. Same for lower level Kingston (NV series) SSDs. At least they openly say that the drives can wildly differ from batch to batch (and oh boy, they do). As the industry is pinched by customers for more capacity, QLC becomes the norm, TBW numbers are continuously hidden, and you're only left with the internal endurance counters, if you can access them. Controllers are in the same boat. Wear leveling, write amplification, flash monitoring are all left to whims of the controller designers, resulting in strange situations. Off-shift/dark production is another problem, but it's not new by any means. BunnyStudios has a great write up which has been linked a couple of times.
- userbinator 2y ago
  The datasheets for the early flash memories (SLC) were easily available and loudly proclaimed 100K or even 1M cycles of endurance and 10-20 years of retention. Now they're very difficult to obtain and are rated for only a few hundred cycles and a few months of retention. No wonder they're secretive about it.
  - bayindirh 2y ago