7 ms·
Sqlite is a great bit of technology but sometimes I read articles like this and think, maybe they should have used postgres. I you don’t specifically need the “
by mangecoeur 11mo ago
Sqlite is a great bit of technology but sometimes I read articles like this and think, maybe they should have used postgres. I you don’t specifically need the “one file portability” aspect of sqlite, or its not embedded (in which case you shouldn’t have concurrency issues), Postgres is easy to get running and solves these problems.
- eduction 11mo ago100%. I specifically clicked for the “why you should care” and was disappointed I could not find it. I certainly don’t mind if someone is pushing the limits of what SQLite is designed for but personally I’d just rather invest the (rather small) overhead of setting up a db server if I need a lot of concurrency.
- abound 11mo agoJellyfin is a self-hostable media server. If they "used Postgres", that means anyone who runs it needs Postgres. I think SQLite is the better choice for this kind of application, if one is going to choose a single database instead of some pluggable layer
- morshu9001 11mo agoExactly, there are use cases where SQLite makes sense but you also want to make it faster. I really don't get why there isn't a more portable Postgres.
- zie 11mo agoThere is, you can even run PG under wasm if you are desperate. :) SQLite is probably the better option here and in most places where you want portability though.
- deleted 11mo ago[deleted]
- tombert 11mo agoI share my Jellyfin with about a dozen people, and it's not weird to have several people streaming at the same time. I have a two gigabit connection so bandwidth isn't generally an issue, but I've had issues when three people all streaming a VC-1 encoded video to H264 in software. This is something that I think I could fairly easily ameliorate if I could simply load-balance the application server by user, but historically (with Emby), I've not been able to do that due to SQLite locking not allowing me to run multiple instances pointing to the same config instance. There's almost certainly ways to do this correctly with SQLite but if they allowed for using almost literally any other database this would be a total non-issue. ETA: For clarification if anyone is reading this, all this media LEGALLY OBTAINED with PERMISSION FROM THE COPYRIGHT HOLDER(S).
- reddalo 11mo agoYeah, I'm sure those twelve people love watching your vacation clips all the time ;)
- apitman 11mo agoWhy not encode to H264 or another codec more widely supported by clients? Storage is cheap.
- MayeulC 11mo ago> I've had issues when three people all streaming a VC-1 encoded video to H264 in software. I don't quite get the "in software" part. I assume you mean that the video needs to be transcoded to h.264 on your server for their client to play it. The way I mostly solved this is to ask people to install and use the native app (jellyfin-media-player or Android app) whenever possible, as it is compatible with more codecs. You can also configure HW acceleration for transcoding, a decent GPU should have no trouble encoding a few h.264 streams in real time. And lastly, you can play with distributed versions of ffmpeg, since Jellyfin calls ffmpeg. There are multiple options, such as https://hub.docker.com/r/bitwrk/jellyfin-rffmpeg https://hub.docker.com/r/bitwrk/jellyfin-rffmpeg (I never used it myself, though).
- tombert 11mo ago
- reddalo 11mo agoThey're actually planning on migrating to Postgres in a future release: >[...] it also opens up new possibilities - not officially yet, but soon - for running Jellyfin backed by "real" database systems like PostgreSQL, providing new options for redundancy, load-balancing, and easier maintenance and administration. The future looks very bright! https://jellyfin.org/posts/jellyfin-release-10.11.0/ https://jellyfin.org/posts/jellyfin-release-10.11.0/
- apitman 11mo agoI hope they keep sqlite as a first class citizen.
- npodbielski 11mo agoWhat is the problem to bundle postgress db engine in the docker server? If you want to install it from package, they can have postgress dB as an option with the warning somewhere that it is 'recomended'. I am sure that if you are able to slefhost stuff you are able to install postgress too.
- apitman 11mo agoJellyfin is one of the very few selfhosted apps that can be run as a simple GUI app on Windows. As an advocate for making selfhosting accessible to less technical people, I'm glad they're using sqlite and also that they don't require docker.
- xorcist 11mo agoA database is never hard to install, but it can be tricky to operate. You have to at least have at least a slight idea about the specifics, from different types of vacuum to how it behaves in low memory conditions. The idea that docker has something to do this is a misdirection at best. And if you think sqlite has many knobs and special modes, wait until you hear about Postgres.
- npodbielski 11mo ago> And if you think sqlite has many knobs and special modes, wait until you hear about Postgres. And why do you think I think that?
- bambax 11mo agoJellyfin is a media server app that gets installed on a great variety of platforms and while it would certainly be possible to add a postgres server to the install, the choice of sqlite is more than justified here IMHO.
- throwaway894345 11mo agoAs a user of Jellyfin, I’m very sad that it doesn’t just use Postgres. I basically have to run an NFS system just for Jellyfin so that its data can be available to it no matter which node it gets scheduled on and also that there are never multiple instances running at the same time, even during deployments (e.g., I need to take care that deployments completely stop the first Jellyfin instance before starting the subsequent instance). There are so many unnecessary single points of failure, and Postgres would make a pretty big one go away (never mind addressing the parallelism problems that plague the developers). Jellyfin is by far the least reliable application I run, but it also seems to be best in class.
- KingMob 11mo agoI gave up on Jellyfin after media library updates kept hanging on certain video files, and switched to the original Emby it was forked from (iiuc). Emby has a scarily-ancient install process, but it's been working just fine with less hassle.
- ants_everywhere 11mo agoI have the same experience. SQLite has been a source of most Jellyfin problems, and Jellyfin has more problems than the rest of the ~ 150 containers I run regularly. A stateless design where a stateless jellyfin server talks to a postgres database would be simpler and more robust.
- throwaway894345 11mo agoYeah, honestly I’m kind of thinking about a media server architecture that has a stateless media server that vends links to pre-transcoded media in object storage (which video players would source from), since pretty much anything can handle mp4/h264/acc video. Maybe in the future I could add on some on-the-fly transcoding (which would happen on a dedicated cluster, reading and writing to object storage), but that seems like a pretty big undertaking.
- apitman 11mo agoYou're from the current generation of selfhosters, which culturally is very similar to kit car builders. The next generation of selfhosters/indiehosters just want a car to get from point A to point B. Sqlite is better for those people.
- thayne 11mo agoUsing postgres would make it significantly more complicated for Jellyfin users to install and set up Jellyfin. And then users would need to worry about migrating the databases when PostgreSQL has a major version upgrade. An embedded database like sqlite is a much better fit for something like Jellyfin.
- throwaway894345 11mo agoAs a Jellyfin user, this hasn’t been my experience. I needed to do a fair bit of work to make sure Jellyfin could access its database no matter which node it was scheduled onto and that no more than one instance ever accessed the database at the same time. Jellyfin by far required more work to setup maintainably than any of the other applications I run, and it is also easily the least reliable application. This isn’t all down to SQLite, but it’s all down to a similar set of assumptions (exactly one application instance interacting with state over a filesystem interface).
- thayne 11mo agoIs running multiple nodes a typical way to run Jellyfin through? I would expect that most Jellyfin users only run a single instance at a time.
- throwaway894345 11mo agoYes, but you have to go out of your way when writing software to make it so the software can only run on one node at a time. Or rather, well-architected software should require minimal, isolated edits to run in a distributed configuration (for example, replacing SQLite with a distributed SQLite).
- thayne 11mo agoThat's just not true. Distributed software is much more complicated and difficult than non-distributed software. Distributed systems have many failure modes that you don't have to worry about in non-distributed systems. Now maybe you could have an abstraction layer over your storage layer that supports multiple data stores, including a distributed one. But that comes with tradeoffs, like being limited to the least common denominator of features of the data stores, and having to implement the abstraction layer for multiple data stores.
- amaccuish 11mo agoTheir whole recent rewrite of the DB code (to Entity Framework) is to allow the user choice of DB in future.
- petters 11mo agoJellyfin is mostly for a single household, right? Sqlite should be much more than sufficient for Jellyfin (if used correctly). Unfortunately, reading this article you get the impression that they are not using it optimally
- nick_ 11mo agoAgreed. How can a media file sharing app possibly saturate Sqlite's write limit? I would use an app-level global lock on all writes to Sqlite.
- npodbielski 11mo agoProbably during scanning libraries? They read hundreds of files and for each of them look for metadata in the internet like discogs and similar. So sure if implemented as async in c# you could run into this issue.
- nick_ 11mo agoAre you hinting at the lack of an `AsyncLock` in .NET?
- npodbielski 11mo agoI did not really inspect their code in this detail. I just did it once few years ago when I was exploring integrating with my oidc provider that use. At that time auth code looked not great with static class handling it it somehow. Anyway I meant that if you scanning a lot of files and do some kind of operations for each of them, and obviously you need to save the result. Depending on how you do it. You could ram sqlite DB with thousands of connection. Which is not great way to do it but it is possible.
- o11c 11mo agoEven with postgres, you don't have to use the system instance; there's nothing stopping you from running the server as a child process. You probably need to support this for your testsuite anyway.
- hamandcheese 11mo agoMaybe in theory. In practice, most people who need Postgres for their test suite will boot an instance in a docker container in CI, and maybe just assume a system version is available for local dev.
- zeroq 11mo agoSqlite has so many small benefits for tiny projects it can't be easily replaced. It's like saying "oh, you want to visit Austrian country side next month and you're asking for advice for best tent? How about you build a cabin instead?".
- heavyset_go 11mo agoI run Jellyfin in a multi-arch cluster because I hate myself, and this would force me to think about where Jellyfin/Postgres is deployed because Postgres databases aren't portable. I already had to do that for my authoritative PG deployment, and my media manager shouldn't require a full RDBMS. Using SQLite for Jellyfin has made running it wherever really, really easy, same thing with doing backups and lazy black box debugging.
- andersmurphy 11mo agoSqlite is fine you need to read the extensive documentation though to get the most out of it. It also has terrible defaults. I think the author od this article missed sqlite_busy. Once you do have it set up correctly, are handling a single writer at the application level and have litestream set up your off to the races assuming your app can scale on a single box (it most likely can).