11 ms·
Show HN: Tired of the non-portability of my playlists, I wrote my own format
- qntmfred 10y agoMy own format = https://xkcd.com/927/ https://xkcd.com/927/
- rafaelgarrido 10y agooh man... 10 minutes late!
- bb88 10y agoI don't see why this is being downvoted. It's fairly apropos to the subject. And it was the first thing I thought of too when I read the title.
- alnitak 10y agoÀ format that is independent of a specific platform or player has its own merits, besides the scratch your itch part. With a good plugin support an ecosystem to allow interoperability of playlists becomes possible.
- pwdisswordfish 10y agoBecause posting Xkcd comics with no further elaboration is a lazy method of grabbing upvotes from the node.js circlejerk^W^W 'hacker' tribe. If that comic was indeed the first thing you thought of, why should you want to have people post a link here? You already know about it, and lots of other people too: I bet a majority of commenters here, because Xkcd isn't at all obscure among all sorts of nerds. If the post contained some further discussion of how the situation described in the comic applies to circumstances at hand, then it would have some value; but then, it wouldn't need to link to Xkcd anyway. As it is, the post is just noise. Besides, that comic is itself an example of lazy writing. I'm pretty sure Munroe wrote it as bait for 'obligatory Xkcd' posters like your parent commenter. Seriously, that meme needs to die. Xkcd has become the new Monty Python: over-referenced to the point of killing all the humour in it.
- StavrosK 10y agoI don't plan on creating a standard that will end all others, though. I just want to move from Google Play to my local library.
- pavel_lishin 10y agoThe introduction sounds much more ambitious than your comment: > The Universal Playlist Format (UPF) is a data format for music playlists. It aims to be more flexible and complete than existing formats. A Universal Playlist (UPL) will allow better description of the tracks it contains, as well as enable added functionality in music players. I am really curious about moving from Google Play to a local library, though - can you say more about that? GP has been losing tracks, slowly but steadily, or mislabeling them horribly.
- StavrosK 10y ago> The introduction sounds much more ambitious than your comment It aims to be more flexible, but it doesn't specifically aim to replace them. I don't mind players using M3U, PLS, or whatever else, I just want the extra features that UPL would provide. I guess it's a fine distinction. > I am really curious about moving from Google Play to a local library, though - can you say more about that? Well, if Google Play allowed me to export my playlists as UPL, my local foobar2000 could easily find the tracks in my local library (I already have them all) and correlate which tracks on Google Play are which tracks locally. As it stands, I have to go into my Google Play playlist, find the thousands of songs in my playlists one by one locally, and add them to my local playlists as I find them.
- Jaruzel 10y agoI think we all thought of this XKCD. However, it has been said, that there are no new stories; they've all be written, and that everything worth inventing has already been invented. If we actually believed that, there'd be no new books, and stuff like graphene, and CRISPR wouldn't exist. I say, let everyone re-invent the wheel, one day someone might actually get anti-grav ones working - You never know.
- a2tech 10y agoI'd love something like this, but I don't see any upside for iTunes or Spotify to support it. Anything that makes it easier for a customer to switch to a competitor is something they're going to resist.
- snissn 10y agoHa that reminds me to cancel my Spotify, thank you!
- StavrosK 10y agoThey have plenty of upside to allow import through this format, as you can just upload your Winamp (yes, I'm old) playlist to Spotify and immediately migrate to their service. I'm hoping there'll be enough user demand that they'll choose not to appear evil and support export as well.
- supermdguy 10y agoThis looks interesting. Do you have any plans to create converters to convert to UPL from other formats?
- StavrosK 10y agoYes, right now I'm collecting ideas (there have been some very good ones in this thread), and I plan to write a few small converter programs to convert from M3U/PLS, as well as some plugins (I'm thinking about beets, currently). Any help would be appreciated!
- enimodas 10y agowhy does the example have md5 and sha1?
- StavrosK 10y agoGood catch, I had them there initially but decided to play it safe and avoid the broken hashes, so I removed them from the list but forgot the example, thank you.
- discreditable 10y agoWhat's wrong with xspf? Couldn't you just extend that with more attributes (which legacy apps could ignore)? https://en.m.wikipedia.org/wiki/XML_Shareable_Playlist_Format https://en.m.wikipedia.org/wiki/XML_Shareable_Playlist_Forma...
- StavrosK 10y agoNothing's wrong per se (and it has roughly the same goals as mine), it's just that it's pretty much the XML equivalent of M3U, and doesn't have any provisions for stronger identification (it relies on titles), at least as far as I know. I would really like the MBID to be the main way of identifying songs throughout the industry, and possibly the AcoustID fingerprint, as that's more specific. I think it would be fantastic if each song had its own UUID that computers could refer to, and this spec tries to incorporate that. Since I don't really gain anything from reusing its format (most players won't support the strong IDs), I thought I might as well create a new format that was better suited to this task.
- lucasgonze 10y agoXSPF is a format to enable sharing, AKA universality. It does this by defining a list of metadata fields to be used for resolving each track in the local context of the listener. The UPF "ids" field maps to the XSPF "identifier" field: http://xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.1.2 http://xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.1.2 XSPF is not M3U in any way. From the spec: http://xspf.org/xspf-v1.html#rfc.section.3.4 http://xspf.org/xspf-v1.html#rfc.section.3.4 3.4 Content resolver On a surface level you can use XSPF like any other playlist format. Drop a bunch of filenames into an XSPF document, prepend "file://" to each, and you're ready to go. Under the surface there is much more. The guiding design principle was to separate the functionality of a catalog of files from the functionality of a list of songs. Most software music players on the PC have some sort of cache for file information. This cache stores a list, or catalog, of available files and metadata from ID3 tags and other sources. XSPF is not a catalog format. XSPF exists only to say which songs to play. Almost everything in XSPF is for the purpose of answering the question which resource, rather than the question what is this resource. If XSPF is not a catalog format, what is it? XSPF is an intermediate format. We expected a new kind of software called a content resolver to do the job of converting XSPF to a plain old list of files or URIs. A content resolver would be smart enough to keep your playlists from breaking when you move your media from /oggs to /music/ogg. It would be able to figure out that a playlist entry by the artist "Hank Williams" with the title "Your Cheating Heart" could be satisfied by the file /vorbis/hankwilliams/yourcheatingheart.ogg. It might even know how to query the iTunes music store or another online provider to locate and download a missing song. The content resolver maintains the catalog of your songs in whatever format it prefers. It might use a flatfile, a file in the Berkeley DB format, or a SQL database. It might use only ID3 metadata, but it might also know how to query MusicBrainz or another metadata service. All XSPF user agents are content resolvers, in that they have complete leeway to turn the contents of a track element into a specific set of bytes. 3.5 Fuzzy names Any given track can be identified in a number of ways. We provided means for absolute identifiers like URIs, filesystem paths and secure hashes, but also for query-based identifiers — free text fields like artist and work title and numeric fields for song length, all of which together should be enough for a good content resolver to turn into files.
- hawski 10y agoAre relative paths supported? Probably in this day and age it would have better chance at adoption if it would be a JSON file. It's just network effects.
- deleted 10y ago[deleted]
- StavrosK 10y agoNot yet, but great idea, I've added it, thank you! YAML is a superset of JSON, so the file can look like JSON if you want, although that's not mandatory. I don't think there are many languages that don't support YAML these days, and one of my goals was to make the format not-entirely-unreadable by humans.
- masklinn 10y ago> I don't think there are many languages that don't support YAML these days YAML is a big and complex format though, libyaml's header is bigger than the entirety of libtoml. I think hand-editable playlists are a good idea, just not a fan of yaml.
- StavrosK 10y agoHm, yeah, that's true, and I see your point. Right now I think the benefits outweigh the costs, but I'll definitely think about this (for embedded players, etc).
- TazeTSchnitzel 10y agoAnother consideration is that yaml's sheer complexity creates more possibilities for interoperability issues. No two parsers for JSON behave identically, let alone for something as large as yaml.
- StavrosK 10y ago
- chadlavi 10y agonot very constructive suggestion: an h2 with a recursion joke, something like "YAML As Music List"
- ianwalter 10y agoThis is such a good idea. If nothing else, third party tools that people make to transfer playlists between platforms could standardize on this and, in doing so, would reduce duplicated efforts. After trying (and being dissatisfied with) EVERY music service, I'm currently trying to consolidate back to Apple Music. Manually. It's awful.
- StavrosK 10y agoSince this is coming along rather okay, I think I will write a converter this weekend. I aim to have it accept an M3U/PLS playlist and the corresponding files and write out a UPL. The opposite step would be a bit harder, I'd have to index all music files in a library and convert the UPL to PLS, which might be too much work for a simple tool, but I'll see if it can be done quickly and dirtily. That would be a nice idea, `upl2pls -i favorites.upl -o favorites.pls` and your favorite player is ready to play the files.
- rocky1138 10y agoI've never had a problem with my Winamp playlists going from one PC to another. I just double click them and the music plays.
- partycoder 10y agoXKCD predicted this. https://xkcd.com/927/ https://xkcd.com/927/
- cmpb 10y agoI'm a little confused why '3.3.1. entries' is a sub-bullet of '3.3. id'. Perhaps it's just a typo?
- StavrosK 10y agoYes, the entry names are automatically done by a hacky module I wrote, I'll fix it, thank you.
- woodruffw 10y agoThis is great, thanks for creating and sharing it. I've been working on my own music player of sorts [1] that uses YAML to store playlists, and I'm going to look into switching to this. [1]: https://github.com/muzak-project/muzak https://github.com/muzak-project/muzak
- StavrosK 10y agoOh, fantastic! Please let me know if you need help with anything or have any feedback. An issue in the project repo is great, or just email me directly (email in profile). Thanks!
- _pmf_ 10y agoDirectories of numbered symlinks, my friends.
- autocracy 10y agoRemoval performance of large directories is very bad. Reading large directories can result in lots of seeks. Directories are not suited to this at scale, and symlink directories definitely aren't portable.
- autocracy 10y agoA side note: at least in python, I benchmarked MsgPack as 3 times faster than JSON, and a whopping 850 times faster to read than YAML. It seems unlikely that people will ever be editing this file by hand, so for unstructured data, especially where playlists can get very large, I suggest MsgPack over YAML. BUT... you have a defined schema, so it's probably to your advantage to use a storage format with a defined schema: ProtoBuf or Thrift. That would mean somebody trying to use your code would already have generated objects in their language. As for the hashing algorithm, this is a good use for MD5 -- cheap and fast. You're unlikely to be concerned about somebody actively trying to generate the same checksum for two music files. For non-security (integrity verification) purposes, MD5 is still very appropriate.
- StavrosK 10y agoThank you, I agree. If I'm going to use JSON, I might as well use ProtoBuf. About MD5, I was worried about a case where a service that serves user-submitted files would be exploited by MD5 collisions, leading users to open files that might exploit decoder bugs to execute code. Far-fetched, I know, but the tradeoff didn't seem worth it. I'm not married to that decision, though.
- nbadg 10y agoThe question of hash usage made me think of an alternative approach -- what about some kind of audio perceptual hash? P-hash has support for audio hashes [1] (at least it claims to, but I've never used it). The metadata is useful, sure, but coupling it with the playlist seems like a bit of a strange design choice, if it could be avoided. In my mind, an ideal world would have two databases (or equivalent): one for metadata -> perceptual hash, and one for playlist -> List[perceptual hashes]. The downside of course is this requires pre-calculation of the p-hash for every track to use. But I can't think of a music application that doesn't require some kind of "library loading" step, so perhaps this could be accomplished then? Of course none of this mitigates your concern with decoder bugs resulting in RCE, but I think that's probably best handled elsewhere (for example, sandboxed upload validation in your hypothetical user-uploaded-files service). [1] http://www.phash.org/ http://www.phash.org/
- jstimpfle 10y agoIf I understand correctly, you want to make playlists portable by associating each item with enough metadata to allow identification even when some metadata, such as filepath, changes. Playlist and metadata are separate things, and coupling them is not good from an architectural perspective (redundancy/denormalization). The two can still be related, even in a single file: relational database. (plug: I'm working on http://jstimpfle.de/projects/wsl/main.html http://jstimpfle.de/projects/wsl/main.html) Sketch to make more clear what I mean: { "metadata": { "buss_und_reu": { "filepath": "Bach/Matthäuspassion/Buss_und_Reu.mp3" "composer": "Johann Sebastian Bach" }, "highway_to_hell": { "filepath": "ACDC/Highway_to_Hell.mp3", "artist": "ACDC", "sha512": ... } }, "playlists": { "mypl": [ "buss_und_reu", "highway_to_hell" ], "mypl2": [ "buss_und_reu" ] } } This is more normalized, less redundant. Of course other schemes are thinkable, for example storing each playlist in its own file, but this way metadata and playlists will more quickly get out of sync. The library I'm working on allows automated conversions of such hierarchical data to/from relational databases, so data can be viewed, and even stored, as JSON, while integrity is checked based on an relational schema - something like % TABLE Track trackid % TABLE Filepath trackid filepath % TABLE SHA512 trackid sha512 % TABLE Title trackid title % TABLE Composer trackid composer % [Declare also domains, keys, and foreign keys here...]
- StavrosK 10y agoHmm, yes, this is clearly a better approach, architecturally. However, I don't know whether it's worth the tradeoffs, given that it will make the file harder to parse, and playlists tend to have each song only once, so the added complexity seems to come at no benefit... It's definitely a better approach for a library, though, and it's how one would model the data in a database as well.
- TTPrograms 10y agoI think a few people have open-sourced various audio fingerprinting algorithms. Perhaps we should standardize around one and enable real UUID for audio files? That seems to be the missing piece for a lot of the novel functionality you're looking to hit. Otherwise you're stuck doing fuzzy pattern matching on metadata, which seems tricky especially considering possible variation in live recording etc.
- StavrosK 10y agoI don't think the two are different goals. I would very much like standardization around the MBID and AcoustID fingerprint, and my playlist format is just a playlist that uses that. There's no way to use those with a PLS file, for example, even if they were standard and ubiquitous.
- zeveb 10y agoI think rather than having a single URI attribute and other identifiers, you ought to allow multiple URIs, and make all other identifiers just be URIs (since, after all, a URI is a Uniform Resource Identifier, and music files are resources). You can represent hashes using the named-information URI scheme (https://tools.ietf.org/html/rfc6920 https://tools.ietf.org/html/rfc6920), e.g. ni:///sha-256;u88lYWn4xAlto-6Bs79KHHYDAu28US71ui5Be6C-ZVw. Your filepath could be file:///Anciients/Following%20the%20Voice.mp3; your mbtrackid could be musicbrainz:b00a2b97-53f1-485a-9121-1fe76b55e651 (since I don't think an authority makes sense in this case). This would also permit multiple entries for certain types of identifier, which doesn't make so much sense for hashes (although it's possible for the same recording to have two different MP3 encodings, so … maybe it's not crazy), but would be nice when e.g. a song is found in multiple places in the filesystem, or can be retrieved from multiple locations.
- lucasgonze 10y agoThat is exactly the approach taken in XSPF: http://www.xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.1.2 http://www.xspf.org/xspf-v1.html#rfc.section.4.1.1.2.14.1.1.... Notice that the cardinality is "zero or more."
- StavrosK 10y agoThis isn't too different from the current format, except the URI method is pushing the type into the URI, rather than into the name of the key. I do wonder about what you said, though: Would it be useful to be able to specify two different file paths or URIs for the same item? I'm leaning towards no, because, when clicking on an entry, you want it to start playing, so the choice is made when adding the file, rather than when playing it. The problem with multiple file paths is that both have the same weight, so how do you choose? I guess you'd choose the same way you choose between the formats already, based on whatever is available/found.
- zeveb 10y ago> This isn't too different from the current format, except the URI method is pushing the type into the URI, rather than into the name of the key. Yup, which I think is good, since URI schemes already exist for many identifiers, and more can be added. > Would it be useful to be able to specify two different file paths or URIs for the same item? I think so. Maybe one is one removable medium and the other is on another. > The problem with multiple file paths is that both have the same weight, so how do you choose? If they have equal weight, does it matter which you choose?
- djrogers 10y agoIs the lack of portability really due to the unavailability of a (or yet another) standard format? Seems it's far more related to the music player decisions about what it's playlist format should be...
- StavrosK 10y agoHow so? How will my music player know that "Dark Tranquillity - The Gallery" from Google Music is actually thegallery.mp3 on my local drive?
- geocar 10y agoDo you know about magnet uri[1]? magnet:?dn=Anciients%2FFollowing%20the%20Voice&xt.1=urn:sha1: e577cce68a69735acccd5d8603b3e663f6aa5bc9&xs=nfs://example.com/music/ftv.mp3&x.dur=408.764081632 [1]: https://en.wikipedia.org/wiki/Magnet_URI_scheme https://en.wikipedia.org/wiki/Magnet_URI_scheme
- StavrosK 10y agoI do, but it doesn't seem directly applicable to this case. It's similar, but it's better suited to torrents and things like that.
- geocar 10y agoProviding enough metadata to recognize the file different ways, is exactly what you're doing. Magnet already does this. > it's better suited to torrents and things like that. I don't understand what you mean by this.
- StavrosK 10y agoIt's more suited to applications that require a URI for a single file. I don't see how it can be applied here, or what benefit it has over a simple list.
- geocar 10y ago> It's more suited to applications that require a URI for a single file. Why exactly is it more suited? > I don't see how it can be applied here, or what benefit it has over a simple list. Winamp supported playlists containing http urls. If it supported magnet urls you would have every feature of your proposal and tooling to build other applications on. You could build this in about an hour and see a portable durable playlist. * https://github.com/captbaritone/winamp2-js https://github.com/captbaritone/winamp2-js * https://github.com/feross/webtorrent https://github.com/feross/webtorrent
- rsync 10y agoYou know what always, always works ? Copying the actual songs into a directory that becomes "the playlist". It's very inefficient and inelegant but after the 100th car stereo I encountered that couldn't make sense of the most basic m3u file, I just gave up ... my "playlists" are just more copies of the actual songs. Problem solved.
- choward 10y agoYou could combine both solutions and process a playlist file to put copies in a directory whenever you need to actually use the playlist.
- ibizaman 10y agoThat's where fuse really shines. For example (note I never used it) https://github.com/rtokarev/m3ufs2 https://github.com/rtokarev/m3ufs2
- StavrosK 10y agoI did that too, and then uploaded those files on Google Music, and now I want to get them back (with the songs I've added since), and... oops.
- ibizaman 10y agoYou can avoid wasting disk space by using symlinks/hardlinks. And I must agree, a filesystem interface is always neat.
- lutusp 10y agoTitle: "Tired of the non-portability of my playlists, I wrote my own format." Subtitle: "The Universal Playlist Format" Analysis: https://xkcd.com/927/ https://xkcd.com/927/
- skookumchuck 10y agoThis is not in the least bit surprising. I download financial information from my accounts in the form of .csv files, but have never found any accounting software that will import those files reliably. I finally had to write my own csv parser and then tweak it for each wretched csv data provider to get things to work right. For example, download credit card transactions as csv. Import into Quicken. Quicken decides that skookumchuck is the payee on all the transactions. Arggh. This is 2017 fer cryin out loud.
- StavrosK 10y agoI'm currently writing a small converter utility to convert PLS/M3U/etc playlists to UPL and back, and to serve as a reference implementation/proof of concept. If you want to follow development, feel free to star this repo: https://gitlab.com/universal-playlist/pls2upl https://gitlab.com/universal-playlist/pls2upl You can also star the repo of the spec website itself: https://gitlab.com/universal-playlist/universal-playlist-spec https://gitlab.com/universal-playlist/universal-playlist-spe...
- adamdecaf 10y agoI was starting to write one too! I've got a couple of commits I can share if you're interested.
- StavrosK 10y agoOh, definitely, what's it like?
- adamdecaf 10y agohttps://github.com/adamdecaf/univplayfmt https://github.com/adamdecaf/univplayfmt I was planning on making a cli around some parsing logic and convert/export from various services (wherever I can get an API from).
- StavrosK 10y agoOh, nice! I... uh... changed the spec to JSON from YAML in the mean time, sorry :/ Here's the Python converter I'm playing with: https://gitlab.com/universal-playlist/pls2upl/ https://gitlab.com/universal-playlist/pls2upl/
- joshka 10y agoOne of the key things I've looked for in a playlist format for many years is the ability to handle files that contain multiple tracks. My main example of this is continuous DJ mixes that are a single file. You might also see this in live recordings of performances where the interaction with the crowd between each song. These single file media items generally have 1 or more Album Artists (e.g. the DJ) with each track in the performance blending into the next. Each track may have an associated Artist and Track name. Conventionally the best approach to solving these is to use a Cue sheet and a plugin for whatever media player that can load these simultaneously. Alternatively (though rarely) a container format that supports chapters / sub streams (e.g. mka) can be used. I don't see a secondary (CUE sheet) format as being the right solution to this problem. To me a playlist and a list of tracks played in a performance are kind of the same thing and should be handled by the playlist. My proposed extension to the UPL format is to allow each playlist entry to specify a part of the identified file. Now there's two possible ways of doing this: 1. Sub items under the entry (e.g. chapters) 2. Each entry specifies a start and end time (or duration) You mention at [1] that this format is about display, I feel this fits within that idea by being something that is otherwise difficult to display using any other available info (ID3 tags etc.) [1]: https://gitlab.com/universal-playlist/universal-playlist-spec/issues/1 https://gitlab.com/universal-playlist/universal-playlist-spe...
- jason_s 10y ago+1, I was going to suggest the same thing. Or at least the ability to note multiple files which are supposed to be chained together in order. It pisses me off when Shuffle doesn't handle these properly: - Pink Floyd, Brain Damage followed by Eclipse - Queen, We Will Rock You followed by We Are the Champions - Yes, Long Distance Runaround followed by The Fish (Schindleria Praematurus) - the different movements of Beethoven's Fifth Symphony
- StavrosK 10y agoVery good idea, thanks! I went with #2, as I feel that it's more generic and easier for players to implement.
- MobesMobes 10y agoIt's not trying to solve the exact same problem, but m-Tags is also using metatags stored in a seperate file. http://www.m-tags.org/ http://www.m-tags.org/