5 ms·

I've ran into situations where this is pertinent information to know w/ a music pet-project. I ended up jamming all my .mp3s into the DB though, because not onl

by bch 12y ago

I've ran into situations where this is pertinent information to know w/ a music pet-project. I ended up jamming all my .mp3s into the DB though, because not only is it storage and access in this case, but the collection (in a DB) is in a defacto archive; all I do is copy the (admittedly large) DB where I need it, and I've got my data and metadata. If I were doing it again, I'd rethink this design, which isn't to say I'd abandon it, but I'd reconsider it; it's unclear whether the all-in-one solution is worth the bulk, though it might still be.

bane 12y ago
I'd always be concerned of some minor level of corruption to the big file, something that would only ruin a song or two, could render your entire dB useless.
- bch 12y ago
  True -- that could be considered a point for "files to FS, metadata to DB".
- icefox 12y ago
  So the natural question is why don't "file systems" (aka big database) become useless with some minor level or corruption?
  - hobs 12y ago
    I was about to comment that a database will only be corrupt if it is corrupted in the same way a file system would be corrupted, or its not a great database file format. I work with SQL Server and while database corruption can/does happen, I often can get away with a row level restore from a backup, no problem. I still dont like storing big ass files in a database though, I dont love the additional level of abstraction when you generally dont need to have a high transactional/relational system.
    - emn13 12y ago
      I think you've got that backwards: if you have a database for something, is it worth the additional levels of abstraction to also store some data in files? If your data is fairly small, and perf not the biggest issue (which is pretty common), you might as well go for the simplest solution, which is going to be purely DB or purely FS, not some mixture of the two.
      - hobs 12y ago
        Agreed, the overhead of maintaining two solutions can be a huge burden (especially when troubleshooting something going wrong via the synthesis of two solutions), but I definitely called out "big ass files" because the difference between some text data and some smaller blob is almost nothing in terms of performance.
- pmoriarty 12y ago
  This is a good point. Not just for DB's but for any large file. It's not uncommon to lose an inode or two on certain filesystems during OS crashes. When one of those inodes is of a large, valuable file that you don't have a backup of, you've got problems. Of course, the best solution is to have recent backups. Unfortunately, the more data you have, the harder, more time consuming, and more expensive your backups will get.
icefox 12y ago
Were you worried that the OS makes it really easy for a user would muck with the filesystem and not update the applications metadata? On the easy of copying one file was it that coping all of the files was too error prone?
heavenlyhash 12y ago
I have the opposite reaction to this situation: I avoid products that use DBs for smallish datasets because the DB blob doesn't dedup well, and that impacts me negatively for maintaining file-system level backups as well as for efficient sync between hosts. Sure, I could write a custom script that dumps the DB to a format that deltas better, but I'd really rather my filesystem level dedup and sync tools just work and not have to do something special-case for every application I use. (I'm currently going through this exact process with my media library -- it's a dozen megabyte database. It's so small loading the entire thing into RAM is easier than not; indexing it is beyond unnecessary, etc. And yet because it doesn't diff, every time I sync it between laptop and desktop, I'm irritated to discover I'm waiting for the entire thing to cross the network every single time.)
- XorNot 12y ago
  Agree with this. It's the current thing I am hating about every single music player application. There just isn't that much data to manage - why do they all insist on a binary blob for playlists and organization?
  - cube00 12y ago
    Managing a playlist (sorting and searching multiple fields) is easier with some kind of DB layer. Usually those blobs are either MS Jet DB (MS Access) or SQLLite and can be opened for inspection/export. Unless of course of the blob is in a proprietary binary format in which case it's all about vendor lock in.
    - XorNot 12y ago
      Some kind yes, but I'd prefer the index be transitive and the data confirmed against what's actually on the disk. Not to mention, those proprietary formats usually fail to sync to player devices automagically.
  - frik 12y ago
    WinAmp, FooBar2000 (Plugins) and Windows Media Player (Win7) store the music metadata in the mp3 files itself (IDv2 format). Using the Windows search advanced syntax I can find all songs that I rated with >=4 stars. One can also add various metadata columns to Explorer details view (like album name, stars-rating, etc.). As playlist format use M3U or its unicode sibling - a format that every music player supports; avoid WPL or other lesser known/supported formats. It would be great if more music players like car entertainment systems, iTunes, iOS Music would show the stats rating that is stored in the mp3 files itself (metadata). Monolithic databased based music management systems may be convenient for some users. I personally prefer to copy the mp3 files (with embedded metadata) to my other devices. My wish list: More intelligent playlist playback that e.g. recognises song skipping as down-voting and manual replaying a song as up-voting and plays similar songs based on the music fingerprint like iTunes Genius but as completely offline feature. Casual people nowadays listen music on streaming platforms like Youtube, Spotify or old-school radio stations. So it's unfortunate but we saw no new exiting new features in music players for some years.

by bch 12y ago

bane 12y ago

I'd always be concerned of some minor level of corruption to the big file, something that would only ruin a song or two, could render your entire dB useless.

bch 12y ago
True -- that could be considered a point for "files to FS, metadata to DB".
icefox 12y ago
So the natural question is why don't "file systems" (aka big database) become useless with some minor level or corruption?
- hobs 12y ago
  I was about to comment that a database will only be corrupt if it is corrupted in the same way a file system would be corrupted, or its not a great database file format. I work with SQL Server and while database corruption can/does happen, I often can get away with a row level restore from a backup, no problem. I still dont like storing big ass files in a database though, I dont love the additional level of abstraction when you generally dont need to have a high transactional/relational system.
  - emn13 12y ago
    I think you've got that backwards: if you have a database for something, is it worth the additional levels of abstraction to also store some data in files? If your data is fairly small, and perf not the biggest issue (which is pretty common), you might as well go for the simplest solution, which is going to be purely DB or purely FS, not some mixture of the two.
    - hobs 12y ago
      Agreed, the overhead of maintaining two solutions can be a huge burden (especially when troubleshooting something going wrong via the synthesis of two solutions), but I definitely called out "big ass files" because the difference between some text data and some smaller blob is almost nothing in terms of performance.
pmoriarty 12y ago
This is a good point. Not just for DB's but for any large file. It's not uncommon to lose an inode or two on certain filesystems during OS crashes. When one of those inodes is of a large, valuable file that you don't have a backup of, you've got problems. Of course, the best solution is to have recent backups. Unfortunately, the more data you have, the harder, more time consuming, and more expensive your backups will get.

icefox 12y ago

Were you worried that the OS makes it really easy for a user would muck with the filesystem and not update the applications metadata? On the easy of copying one file was it that coping all of the files was too error prone?

heavenlyhash 12y ago

I have the opposite reaction to this situation: I avoid products that use DBs for smallish datasets because the DB blob doesn't dedup well, and that impacts me negatively for maintaining file-system level backups as well as for efficient sync between hosts. Sure, I could write a custom script that dumps the DB to a format that deltas better, but I'd really rather my filesystem level dedup and sync tools just work and not have to do something special-case for every application I use. (I'm currently going through this exact process with my media library -- it's a dozen megabyte database. It's so small loading the entire thing into RAM is easier than not; indexing it is beyond unnecessary, etc. And yet because it doesn't diff, every time I sync it between laptop and desktop, I'm irritated to discover I'm waiting for the entire thing to cross the network every single time.)

XorNot 12y ago
Agree with this. It's the current thing I am hating about every single music player application. There just isn't that much data to manage - why do they all insist on a binary blob for playlists and organization?
- cube00 12y ago
  Managing a playlist (sorting and searching multiple fields) is easier with some kind of DB layer. Usually those blobs are either MS Jet DB (MS Access) or SQLLite and can be opened for inspection/export. Unless of course of the blob is in a proprietary binary format in which case it's all about vendor lock in.
  - XorNot 12y ago
    Some kind yes, but I'd prefer the index be transitive and the data confirmed against what's actually on the disk. Not to mention, those proprietary formats usually fail to sync to player devices automagically.
- frik 12y ago
  WinAmp, FooBar2000 (Plugins) and Windows Media Player (Win7) store the music metadata in the mp3 files itself (IDv2 format). Using the Windows search advanced syntax I can find all songs that I rated with >=4 stars. One can also add various metadata columns to Explorer details view (like album name, stars-rating, etc.). As playlist format use M3U or its unicode sibling - a format that every music player supports; avoid WPL or other lesser known/supported formats. It would be great if more music players like car entertainment systems, iTunes, iOS Music would show the stats rating that is stored in the mp3 files itself (metadata). Monolithic databased based music management systems may be convenient for some users. I personally prefer to copy the mp3 files (with embedded metadata) to my other devices. My wish list: More intelligent playlist playback that e.g. recognises song skipping as down-voting and manual replaying a song as up-voting and plays similar songs based on the music fingerprint like iTunes Genius but as completely offline feature. Casual people nowadays listen music on streaming platforms like Youtube, Spotify or old-school radio stations. So it's unfortunate but we saw no new exiting new features in music players for some years.