8 ms·

The case of the missing WAV audio files on the FAT32 SD Card

156 points by shanselman 7y ago

solstice 7y ago
Wow, what a ride. 7-zip is awesome. For anything involving (suspected) data loss or "data disappearance", I usually try Testdisk/photorec from https://www.cgsecurity.org/ https://www.cgsecurity.org/
- skunkworker 7y ago
  I've used 7-zip to extract data from vmware VMDK files. Nothing else has come close to opening obscure files for me.
rahimnathwani 7y ago
This advice should go without saying: "Re-format the random SD cards you get from Amazon specifically on the device you're gonna use them." Do this also for portable hard drives.
- james-skemp 7y ago
  Since it comes up so often now, reformat the drive and then do what to verify that it's a legit drive, and not one that advertises as being larger than it actually is? Put a bunch of large media files on it and verify that they still play properly?
  - Polylactic_acid 7y ago
    I always use this tool https://github.com/AltraMayor/f3 https://github.com/AltraMayor/f3 It fills the entire device with data and then tries to read it all back. It can tell you how many bytes were successfully read, how many were corrupted and how many were written over by other writes. Even on cards I know are real I still run the test because I have had a card that had a few bytes that got corrupted which caused loads of issues with my rpi.
    - rahimnathwani 7y ago
      I usually use f3probe, which IIRC uses some heuristics and trades of accuracy for speed, i.e. it doesn't write to the entire drive. The docs are lower down the page from the main f3 utils: https://fight-flash-fraud.readthedocs.io/en/latest/usage.html#f3probe-the-fastest-drive-test https://fight-flash-fraud.readthedocs.io/en/latest/usage.htm...
      - Polylactic_acid 7y ago
        The test time can be pretty insane on the high capacity cards. I remember it took about 4 hours to test a 256gb card but I find its worth it to check the card just once since it saves a lot of pain later when you find one tiny part of the card is failing.
  - solstice 7y ago
    I used h2testw on Windows to do this. It writes a 1-GB files with a special pattern on the disk until it is full (or however many GB you specified before starting it). It then tries to read them all back to see if they have been written correctly.
- userbinator 7y ago
  Any storage device I buy gets at least a full pass of read/write testing, any errors means it's being returned as not fit for purpose. SSDs and HDDs get 8 full passes.
  - floatingatoll 7y ago
    Why 8, and not 4 or 7 or 16?
  - zimpenfish 7y ago
    To be fair, he did test the storage device: "I did a local recording right there and played it back. Sounds good. I played it back locally on the Zoom and I could hear the recording from the Zoom's local speaker"
    - wtallis 7y ago
      That test doesn't really give you any more information than noticing that you didn't get a "card not found" error. A full-capacity write and read verification is a test that's actually informed by the possible failure modes that wouldn't be immediately apparent from ordinary usage. If you're worried enough to take the deliberate step of manually testing your storage device, you should at least make sure it's a useful test that tells you something you don't already know.
  - Hello71 7y ago
    with a QLC drive, like the 660p, 8 cycles is 4% of your total drive life...
heywire 7y ago
Great read! I agree, 7-zip is amazing. You can open all kinds of files for exploration using it.
mercora 7y ago
its a really nice read but i am somewhat sad they neither know what actually happened nor why it happened at all. I hope this gets some attention from someone here that is skilled enough with this kind of things and willing to figure this out as it kinda bugs me now...
rdp36 7y ago
lesson: use linux end to end (unless you work for Microsoft). :)
- Arnavion 7y ago
  The author eventually recovered their files using 7-zip, which is a Windows program.
  - ptspts 7y ago
    p7zip, the portable version of 7-Zip works on Linux, macOS and other Unix systems as well. It also has a Debian package. Data recovery from that FAT32 filesystem would have worked identically in p7zip (and 7-Zip).
    - Arnavion 7y ago
      I know that p7zip exists. Nevertheless, the author used the Windows version and it worked for them, so I'm not sure why the Linux implementation is relevant to this conversation.
  - progre 7y ago
    It's been available on all linux distros I have tried.
- zimpenfish 7y ago
  He tried reading the SD image on Linux and that didn't work either. "We can see them, but can't get at them with the vfat file system driver on Linux or with Windows."
Fr0styMatt88 7y ago
That was an awesome article! The kind of rabbit hole I’d gladly go down :) SD cards and USB sticks are just.... weird. I haven’t delved too deeply but maybe someone here knows.... why is this? At this point I almost think of them as ‘not disks’. The particular weirdness I’m thinking of is I’ve seen more than one drive just seem to die, just because you try and repartition them in Windows. Not even filling the drives up. Is it the case that USB drive and SD card firmware is just crap? Like it makes assumptions about the device other than ‘this is just a set of blocks on flash memory’ and actually needs things on the disk at a file system level to be a certain way for it to work properly? I’m really curious.
- naikrovek 7y ago
  I've seen that happen on Mac, but not Windows, unless it's a counterfeit card. If you "dd" to a mounted disk on Mac, you are going to have a bad time. Also, people use "dd" a lot when all they need is "cat".
  - genpfault 7y ago
    >people use "dd" a lot when all they need is "cat". Well, GNU dd has nice things like 'status=progress' and options for direct-I/O :)
    - gbin 7y ago
      Or the awesome progress viewer (pv command). This is cat with some feedback. I use it a lot to image stuff when the media is readable, if you have bad sectors you need some like ddrescue to image it.
- ObsoleteNerd 7y ago
  > The particular weirdness I’m thinking of is I’ve seen more than one drives just seem to die, just because you try and repartition them in Windows. Not even filling the drives up. I’ve absolutely had this issue. Lots. I can’t figure out why either. I have about half dozen Micro SD cards that now say they’re not formatted, and error when trying to format them in windows. All of them “broke” during a simple repartition in Disk Manager. I’ve had minor success with some of them using fdisk to “repair” them but others just fail there too.
amluto 7y ago
> I didn't want to take any chances to I picked up a 5 pack of 32GIG high quality SD Cards. [link to Amazon] The lesson: never ever buy SD cards or USB sticks from Amazon. Buy them from a reputable company.
- knaik94 7y ago
  There was nothing wrong with the SD cards physically, they weren't formatted properly for the sound recording device. There was zero data loss, it was a data access issue. An initial format would have helped avoid the problem. I'm not sure the lesson you state is the lesson the author was trying to share.
  - zenexer 7y ago
    There’s a fairly good chance that the physical cards themselves were fake. That happens a lot on Amazon. Given that cards purchased from reputable manufacturers tend to be formatted without issue, it’s reasonable to guess that the real problem here is that those are counterfeit cards.
    - londons_explore 7y ago
      Agreed. This was probably someone trying to tweak the fat32 header for better fake-ability, for example putting the sector maps elsewhere.
  - speedgoose 7y ago
    It could be for a example a 4gb SD card formatted as 32gb. Which is fine if you only use 300mb like he did.
  - deleted 7y ago
    [deleted]
- xeeeeeeeeeeenu 7y ago
  Even if those cards aren't fake, they definitely aren't "high quality". Memory cards that don't use SLC NANDs (maaaybe pSLC too) shouldn't be trusted for anything serious.
userbinator 7y ago
Among one of the reasons the FAT family of filesystems is in widespread use is its simplicity, which makes data recovery easier. Writing a FAT driver/reader is a common assignment for systems programming courses, and I've done it myself too. I took a quick look at the truncated image he provided, and it looks like the fields in the MBR and boot sector are OK; but things start getting weird after that. The root is at cluster 2 where it normally should be (byte offset C00000 in the image), but its second entry is that all-spaces directory which claims to start in cluster 4, and furthermore its creation date is recent (2020-03-12). Cluster 3 (C08000) is a directory which points to ZOOM0002.hprj, ZOOM0002_Tr1.WAV, and ZOOM0002_Tr2.WAV, its "." is correctly pointing to itself, but its parent according to ".." is at cluster 4. It is directory "ZOOM0002". Cluster 4 (C10000) is a directory which contains 3 directories ZOOM0001 through 3, its "." is correctly pointing to itself, and its parent is pointing to the root (indicated with a cluster of 0). This is, according to the root, the "all spaces" directory. Cluster 5 (C18000) is ZOOM0001 directory, and its "." and ".." are correct. Cluster 6 (C20000) is a file, ZOOM0001.hprj Cluster 7 (C28000) is a file, ZOOM0001_LR.WAV In other words, everything looks OK except for that all-spaces directory in the root. According to page 25 of https://www.zoom-na.com/sites/default/files/products/downloads/pdfs/H6-Manual_0_1.pdf https://www.zoom-na.com/sites/default/files/products/downloa... that should have been named something like FOLDER01 through FOLDER10, so fixing that should make it valid again. It's hard to say what happened here without knowing the state of the filesystem before the Zoom started writing to it, but the fact that the creation date/time of that nameless directory (2020-03-12 12:29) matches exactly with that of the first file it wrote (ZOOM0001.hprj/ZOOM0001_LR.WAV) strongly suggests that the card did not already have a corrupted filesystem beforehand, and the Zoom somehow wrote a blank name directory where it should've written FOLDER01. A search for "FOLDER" in the image yields no results either, so it's not like it wrote them somewhere else (unless it did so beyond the first 500MB of the card.) I've seen embedded devices become confused and write corrupted filesystems a few times before, but they were far more egregious than this; e.g. ignoring/assuming certain fields' values would result in filesystem structures being wholesale shifted by some offset, etc. This is an unusual case because nothing about the filesystem stood out as being "obviously suspicious/non-standard", and everything except for the name was fine --- hence why 7-zip could still operate on it; it didn't care about the name. The value of 1458 reserved sectors (729KB!) initially stood out, but upon further thought, that may simply be to align the first cluster with the eraseblocks of the flash, as if the Zoom really was to ignore it, the filesystem structures would've been far more mangled.
Neil44 7y ago
Why not try a ready rolled file recovery utility first? Why grind out this reinventing of the wheel?
- qayxc 7y ago
  Because he's a curious mind and wanted to figure out what was happening here. Admittedly not the most pragmatic approach, but definitely the more entertaining and informative one :)
  - shanselman 7y ago
    Exactly! Thanks ;)
hag 7y ago
I'm wondering if `binwalk`'ing the initial dd-image would have worked. That's the first thing I would try.
- blattimwind 7y ago
  Patch dosfsck to just dump every file it sees.
- akx 7y ago
  It absolutely doesn't, in this case. $ binwalk hanselman.img DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 13026788 0xC6C5E4 MySQL ISAM index file Version 6 13064186 0xC757FA MySQL ISAM index file Version 2 15513282 0xECB6C2 YAFFS filesystem, little endian 18368322 0x1184742 YAFFS filesystem, little endian 42678040 0x28B3718 MySQL ISAM compressed data file Version 6 59068786 0x3855172 YAFFS filesystem, little endian 60315328 0x39856C0 YAFFS filesystem, little endian
sbradford26 7y ago
7-Zip is pretty awesome. At one point I had a file that simply wouldn't delete in Windows. Came across a random thing online that said to try and delete it from 7-Zip. Simple as that 7-Zip was able to delete the file while everything else couldn't.
- qayxc 7y ago
  That's a common issue with Windows. The internal file API is much more powerful than the user-accessible front-end programs, which can be very frustrating at times. I had similar issues with corrupted file names created by some application bug. There was no way to delete these invalid file entries using OS-provided tools.
  - sbradford26 7y ago
    That clears up some of the mystery. I was wondering how 7-Zip could do it but not explorer or command prompt. But it makes lot of sense that 7-Zip is just fully utilizing the internal file API.
- speeder 7y ago
  Another weird thing 7-Zip can do: extract files really, really, really correctly. Often I download some japanese games (not pirated stuff, I mean freeware indie games, often being fan-made, like a fan game of some anime or something), and windows Zip somehow extracts them wrong, sometimes files are missing, or the game runs but with corrupted textures and so on. If I extract same games using 7-zip then they work fine. If there are files that use non-latin characters then 7-zip is mandatory, windows just fails to extract those files properly.
lostmsu 7y ago
He did not try chkdsk, the default tool to fix file systems on Windows?

156 points by shanselman 7y ago

Wow, what a ride. 7-zip is awesome. For anything involving (suspected) data loss or "data disappearance", I usually try Testdisk/photorec from https://www.cgsecurity.org/ https://www.cgsecurity.org/

skunkworker 7y ago
I've used 7-zip to extract data from vmware VMDK files. Nothing else has come close to opening obscure files for me.

rahimnathwani 7y ago

This advice should go without saying: "Re-format the random SD cards you get from Amazon specifically on the device you're gonna use them." Do this also for portable hard drives.

james-skemp 7y ago
Since it comes up so often now, reformat the drive and then do what to verify that it's a legit drive, and not one that advertises as being larger than it actually is? Put a bunch of large media files on it and verify that they still play properly?
- Polylactic_acid 7y ago
  I always use this tool https://github.com/AltraMayor/f3 https://github.com/AltraMayor/f3 It fills the entire device with data and then tries to read it all back. It can tell you how many bytes were successfully read, how many were corrupted and how many were written over by other writes. Even on cards I know are real I still run the test because I have had a card that had a few bytes that got corrupted which caused loads of issues with my rpi.
  - rahimnathwani 7y ago
    I usually use f3probe, which IIRC uses some heuristics and trades of accuracy for speed, i.e. it doesn't write to the entire drive. The docs are lower down the page from the main f3 utils: https://fight-flash-fraud.readthedocs.io/en/latest/usage.html#f3probe-the-fastest-drive-test https://fight-flash-fraud.readthedocs.io/en/latest/usage.htm...
    - Polylactic_acid 7y ago
      The test time can be pretty insane on the high capacity cards. I remember it took about 4 hours to test a 256gb card but I find its worth it to check the card just once since it saves a lot of pain later when you find one tiny part of the card is failing.
- solstice 7y ago
  I used h2testw on Windows to do this. It writes a 1-GB files with a special pattern on the disk until it is full (or however many GB you specified before starting it). It then tries to read them all back to see if they have been written correctly.
userbinator 7y ago
Any storage device I buy gets at least a full pass of read/write testing, any errors means it's being returned as not fit for purpose. SSDs and HDDs get 8 full passes.
- floatingatoll 7y ago
  Why 8, and not 4 or 7 or 16?
- zimpenfish 7y ago
  To be fair, he did test the storage device: "I did a local recording right there and played it back. Sounds good. I played it back locally on the Zoom and I could hear the recording from the Zoom's local speaker"
  - wtallis 7y ago
    That test doesn't really give you any more information than noticing that you didn't get a "card not found" error. A full-capacity write and read verification is a test that's actually informed by the possible failure modes that wouldn't be immediately apparent from ordinary usage. If you're worried enough to take the deliberate step of manually testing your storage device, you should at least make sure it's a useful test that tells you something you don't already know.
- Hello71 7y ago
  with a QLC drive, like the 660p, 8 cycles is 4% of your total drive life...

heywire 7y ago

Great read! I agree, 7-zip is amazing. You can open all kinds of files for exploration using it.

mercora 7y ago

its a really nice read but i am somewhat sad they neither know what actually happened nor why it happened at all. I hope this gets some attention from someone here that is skilled enough with this kind of things and willing to figure this out as it kinda bugs me now...

rdp36 7y ago

lesson: use linux end to end (unless you work for Microsoft). :)

Arnavion 7y ago
The author eventually recovered their files using 7-zip, which is a Windows program.
- ptspts 7y ago
  p7zip, the portable version of 7-Zip works on Linux, macOS and other Unix systems as well. It also has a Debian package. Data recovery from that FAT32 filesystem would have worked identically in p7zip (and 7-Zip).
  - Arnavion 7y ago
    I know that p7zip exists. Nevertheless, the author used the Windows version and it worked for them, so I'm not sure why the Linux implementation is relevant to this conversation.
- progre 7y ago
  It's been available on all linux distros I have tried.
zimpenfish 7y ago
He tried reading the SD image on Linux and that didn't work either. "We can see them, but can't get at them with the vfat file system driver on Linux or with Windows."

Fr0styMatt88 7y ago

That was an awesome article! The kind of rabbit hole I’d gladly go down :) SD cards and USB sticks are just.... weird. I haven’t delved too deeply but maybe someone here knows.... why is this? At this point I almost think of them as ‘not disks’. The particular weirdness I’m thinking of is I’ve seen more than one drive just seem to die, just because you try and repartition them in Windows. Not even filling the drives up. Is it the case that USB drive and SD card firmware is just crap? Like it makes assumptions about the device other than ‘this is just a set of blocks on flash memory’ and actually needs things on the disk at a file system level to be a certain way for it to work properly? I’m really curious.

naikrovek 7y ago
I've seen that happen on Mac, but not Windows, unless it's a counterfeit card. If you "dd" to a mounted disk on Mac, you are going to have a bad time. Also, people use "dd" a lot when all they need is "cat".
- genpfault 7y ago
  >people use "dd" a lot when all they need is "cat". Well, GNU dd has nice things like 'status=progress' and options for direct-I/O :)
  - gbin 7y ago
    Or the awesome progress viewer (pv command). This is cat with some feedback. I use it a lot to image stuff when the media is readable, if you have bad sectors you need some like ddrescue to image it.
ObsoleteNerd 7y ago
> The particular weirdness I’m thinking of is I’ve seen more than one drives just seem to die, just because you try and repartition them in Windows. Not even filling the drives up. I’ve absolutely had this issue. Lots. I can’t figure out why either. I have about half dozen Micro SD cards that now say they’re not formatted, and error when trying to format them in windows. All of them “broke” during a simple repartition in Disk Manager. I’ve had minor success with some of them using fdisk to “repair” them but others just fail there too.

amluto 7y ago

> I didn't want to take any chances to I picked up a 5 pack of 32GIG high quality SD Cards. [link to Amazon] The lesson: never ever buy SD cards or USB sticks from Amazon. Buy them from a reputable company.

knaik94 7y ago
There was nothing wrong with the SD cards physically, they weren't formatted properly for the sound recording device. There was zero data loss, it was a data access issue. An initial format would have helped avoid the problem. I'm not sure the lesson you state is the lesson the author was trying to share.
- zenexer 7y ago
  There’s a fairly good chance that the physical cards themselves were fake. That happens a lot on Amazon. Given that cards purchased from reputable manufacturers tend to be formatted without issue, it’s reasonable to guess that the real problem here is that those are counterfeit cards.
  - londons_explore 7y ago
    Agreed. This was probably someone trying to tweak the fat32 header for better fake-ability, for example putting the sector maps elsewhere.
- speedgoose 7y ago
  It could be for a example a 4gb SD card formatted as 32gb. Which is fine if you only use 300mb like he did.
- deleted 7y ago
  [deleted]
xeeeeeeeeeeenu 7y ago
Even if those cards aren't fake, they definitely aren't "high quality". Memory cards that don't use SLC NANDs (maaaybe pSLC too) shouldn't be trusted for anything serious.

userbinator 7y ago

Among one of the reasons the FAT family of filesystems is in widespread use is its simplicity, which makes data recovery easier. Writing a FAT driver/reader is a common assignment for systems programming courses, and I've done it myself too. I took a quick look at the truncated image he provided, and it looks like the fields in the MBR and boot sector are OK; but things start getting weird after that. The root is at cluster 2 where it normally should be (byte offset C00000 in the image), but its second entry is that all-spaces directory which claims to start in cluster 4, and furthermore its creation date is recent (2020-03-12). Cluster 3 (C08000) is a directory which points to ZOOM0002.hprj, ZOOM0002_Tr1.WAV, and ZOOM0002_Tr2.WAV, its "." is correctly pointing to itself, but its parent according to ".." is at cluster 4. It is directory "ZOOM0002". Cluster 4 (C10000) is a directory which contains 3 directories ZOOM0001 through 3, its "." is correctly pointing to itself, and its parent is pointing to the root (indicated with a cluster of 0). This is, according to the root, the "all spaces" directory. Cluster 5 (C18000) is ZOOM0001 directory, and its "." and ".." are correct. Cluster 6 (C20000) is a file, ZOOM0001.hprj Cluster 7 (C28000) is a file, ZOOM0001_LR.WAV In other words, everything looks OK except for that all-spaces directory in the root. According to page 25 of https://www.zoom-na.com/sites/default/files/products/downloads/pdfs/H6-Manual_0_1.pdf https://www.zoom-na.com/sites/default/files/products/downloa... that should have been named something like FOLDER01 through FOLDER10, so fixing that should make it valid again. It's hard to say what happened here without knowing the state of the filesystem before the Zoom started writing to it, but the fact that the creation date/time of that nameless directory (2020-03-12 12:29) matches exactly with that of the first file it wrote (ZOOM0001.hprj/ZOOM0001_LR.WAV) strongly suggests that the card did not already have a corrupted filesystem beforehand, and the Zoom somehow wrote a blank name directory where it should've written FOLDER01. A search for "FOLDER" in the image yields no results either, so it's not like it wrote them somewhere else (unless it did so beyond the first 500MB of the card.) I've seen embedded devices become confused and write corrupted filesystems a few times before, but they were far more egregious than this; e.g. ignoring/assuming certain fields' values would result in filesystem structures being wholesale shifted by some offset, etc. This is an unusual case because nothing about the filesystem stood out as being "obviously suspicious/non-standard", and everything except for the name was fine --- hence why 7-zip could still operate on it; it didn't care about the name. The value of 1458 reserved sectors (729KB!) initially stood out, but upon further thought, that may simply be to align the first cluster with the eraseblocks of the flash, as if the Zoom really was to ignore it, the filesystem structures would've been far more mangled.

Neil44 7y ago

Why not try a ready rolled file recovery utility first? Why grind out this reinventing of the wheel?

qayxc 7y ago
Because he's a curious mind and wanted to figure out what was happening here. Admittedly not the most pragmatic approach, but definitely the more entertaining and informative one :)
- shanselman 7y ago
  Exactly! Thanks ;)

hag 7y ago

I'm wondering if `binwalk`'ing the initial dd-image would have worked. That's the first thing I would try.

blattimwind 7y ago
Patch dosfsck to just dump every file it sees.
akx 7y ago
It absolutely doesn't, in this case. $ binwalk hanselman.img DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 13026788 0xC6C5E4 MySQL ISAM index file Version 6 13064186 0xC757FA MySQL ISAM index file Version 2 15513282 0xECB6C2 YAFFS filesystem, little endian 18368322 0x1184742 YAFFS filesystem, little endian 42678040 0x28B3718 MySQL ISAM compressed data file Version 6 59068786 0x3855172 YAFFS filesystem, little endian 60315328 0x39856C0 YAFFS filesystem, little endian

sbradford26 7y ago

7-Zip is pretty awesome. At one point I had a file that simply wouldn't delete in Windows. Came across a random thing online that said to try and delete it from 7-Zip. Simple as that 7-Zip was able to delete the file while everything else couldn't.

qayxc 7y ago
That's a common issue with Windows. The internal file API is much more powerful than the user-accessible front-end programs, which can be very frustrating at times. I had similar issues with corrupted file names created by some application bug. There was no way to delete these invalid file entries using OS-provided tools.
- sbradford26 7y ago
  That clears up some of the mystery. I was wondering how 7-Zip could do it but not explorer or command prompt. But it makes lot of sense that 7-Zip is just fully utilizing the internal file API.
speeder 7y ago
Another weird thing 7-Zip can do: extract files really, really, really correctly. Often I download some japanese games (not pirated stuff, I mean freeware indie games, often being fan-made, like a fan game of some anime or something), and windows Zip somehow extracts them wrong, sometimes files are missing, or the game runs but with corrupted textures and so on. If I extract same games using 7-zip then they work fine. If there are files that use non-latin characters then 7-zip is mandatory, windows just fails to extract those files properly.

lostmsu 7y ago

He did not try chkdsk, the default tool to fix file systems on Windows?