4 ms·

> The idea is that a more complicated DBMS should be faster, especially for huge music libraries. How "huge" are they talking about? I built a tool that import

> The idea is that a more complicated DBMS should be faster, especially for huge music libraries. How "huge" are they talking about? I built a tool that imports Apache logs into sqlite for quick analysis and that easily handles several million records.

loeg 9y ago
Just a ballpark: my beets DB is 15 MB for ~1000 albums (well, folders). This post[0] claims 10M albums ever released. So maybe 150 GB upper bound? Assuming my db has been vacuumed recently (~~probably not true~~ Edit: nope, still 15M after vacuum) and ignoring that indexes will scale slightly non-linearly. [0]: https://www.quora.com/How-many-music-albums-are-available-in-the-world https://www.quora.com/How-many-music-albums-are-available-in...
- laumars 9y ago
  I think 10m is far too low to be the total number of albums. But it would be impossible to actually calculate that figure given the number of producers that have released content outside the scope of any particular authority (eg NIN has released stuff to download only from their site. Aphex Twin uploaded a load of stuff to SoundCloud. Etc). But we digress. My music collection is massive. It's got 30 years of singles and albums in there. And several years of DJing too. Plus a massive amount of DJ sets (like several hundred gig of DJ sets alone). Yet XBMC / Kodi handles it fine "despite" being sqlite backed. So does Subsonic and that just uses some Java equivalent. And as I've said already, I can throw several million records of Apache logs into sqlite without any issue. The performance of sqlite is actually really good. It's super fast. Which is why the author can make those boasts on its website. Plus the compactness of the db (ie one file produced) and the ease to create a db makes it a perfect choice for desktop and mobile applications. I think some people read that WordPress runs on MySQL or spend 10 minutes building their first CMS in PostgreSQL and then think the world and their mother should be using this miraculous new RDBMS they've just discovered themselves. (I'm probably being too harsh there)
  - loeg 9y ago
    Yeah. To be clear, I think even a 150GB sqlite database would perform just fine for beets' purposes.
    - snuxoll 9y ago
      Even as a heavy PostgreSQL fan and promoter I don't see why you would choose anything BUT sqlite for single-user applications unless you needed something that only other databases provide (richer procedural language support or other extensions that the fairly basic but still adequate featureset SQLite provides) - size of the dataset is only one factor to consider when choosing the storage layer for your application, and there's a lot of valid solutions for handling 1TB or less of data easily.
      - ptman 9y ago
        Real typing? https://firebirdsql.org/ https://firebirdsql.org/
octorian 9y ago
For single-threaded access, its possible that a more complicated DBMS will actually be slower than SQLite. (Or similar in performance.) Of course once your application needs to do multithreaded DB access (especially if writes are involved), then you're far better off ditching SQLite. Then again, this is probably not a very common use case for a typical desktop application.

loeg 9y ago

Just a ballpark: my beets DB is 15 MB for ~1000 albums (well, folders). This post[0] claims 10M albums ever released. So maybe 150 GB upper bound? Assuming my db has been vacuumed recently (~~probably not true~~ Edit: nope, still 15M after vacuum) and ignoring that indexes will scale slightly non-linearly. [0]: https://www.quora.com/How-many-music-albums-are-available-in-the-world https://www.quora.com/How-many-music-albums-are-available-in...

laumars 9y ago
I think 10m is far too low to be the total number of albums. But it would be impossible to actually calculate that figure given the number of producers that have released content outside the scope of any particular authority (eg NIN has released stuff to download only from their site. Aphex Twin uploaded a load of stuff to SoundCloud. Etc). But we digress. My music collection is massive. It's got 30 years of singles and albums in there. And several years of DJing too. Plus a massive amount of DJ sets (like several hundred gig of DJ sets alone). Yet XBMC / Kodi handles it fine "despite" being sqlite backed. So does Subsonic and that just uses some Java equivalent. And as I've said already, I can throw several million records of Apache logs into sqlite without any issue. The performance of sqlite is actually really good. It's super fast. Which is why the author can make those boasts on its website. Plus the compactness of the db (ie one file produced) and the ease to create a db makes it a perfect choice for desktop and mobile applications. I think some people read that WordPress runs on MySQL or spend 10 minutes building their first CMS in PostgreSQL and then think the world and their mother should be using this miraculous new RDBMS they've just discovered themselves. (I'm probably being too harsh there)
- loeg 9y ago
  Yeah. To be clear, I think even a 150GB sqlite database would perform just fine for beets' purposes.
  - snuxoll 9y ago
    Even as a heavy PostgreSQL fan and promoter I don't see why you would choose anything BUT sqlite for single-user applications unless you needed something that only other databases provide (richer procedural language support or other extensions that the fairly basic but still adequate featureset SQLite provides) - size of the dataset is only one factor to consider when choosing the storage layer for your application, and there's a lot of valid solutions for handling 1TB or less of data easily.
    - ptman 9y ago
      Real typing? https://firebirdsql.org/ https://firebirdsql.org/

octorian 9y ago

For single-threaded access, its possible that a more complicated DBMS will actually be slower than SQLite. (Or similar in performance.) Of course once your application needs to do multithreaded DB access (especially if writes are involved), then you're far better off ditching SQLite. Then again, this is probably not a very common use case for a typical desktop application.