5 ms·

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you

by pilif 7y ago

«just write your own server on top of serverless embedded SQLite to get to acceptable performance without needing a client/server database» (Honestly, once you are at the point where concurrency causes performance issues with SQLite, you are better off moving to databases designed to handle concurrency rather than trying to cobble together your own workaround - you have reached the point where the drawback of SQLite‘s architecture outweigh its advantages and the advantages of other databases architectures outweigh their drawbacks)

pdimitar 7y ago
Sure, I don't disagree. I've ditched sqlite for Postgres many times. It's just that for 99% of the projects I ever worked on, writes are like 100x less than reads so wrapping the writes in a queue of sorts has been quite okay and performant. It can and it has been coming to a point when it's easier to use a full-blown database server, too. I was simply pointing out that for a lot of classic workflows wrapping/centralising writes works quite fine.
logicallee 7y ago
As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes. so if everyone's saying this, is there such a standard dummy program?
- aikah 7y ago
  > As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes. And use what protocol for inter-process communication with the daemon managing SQLite? At the end of the day you just created the equivalent of a server anyway...
  - logicallee 7y ago
    Any protocol. People are saying this is a common solution. So is there a standard dummy tool like that?
    - aikah 7y ago
      > So is there a standard dummy tool like that? My point is that by that time SQlite doesn't fit "Serveless" as defined by the linked article anymore.
      - logicallee 7y ago
        True
- pdimitar 7y ago
  There shouldn’t be should a such standard program. It’s up to you to implement it in-process so you reap all the benefits of sqlite’s speed and lack of dedicated server process. In the case of Erlang/Elixir where I most actively work in the last few years, it’s really easy to centralise write access (dedicated actor receiving messages). With other languages it should also be fairly easy.
- tracker1 7y ago
  Given the extensions that people compile into SQLite, I don't think one can even say that... there's mention of ActorDB in another thread that seems to cover this... Wrapping a typical API in either GRPC or even over HTTP isn't too hard.. you take in a parameterized query with parameters using library X and return JSON stringified to that library... you may want more specific interfaces, or even GraphQL over the database for that matter... it really depends on your needs.
wpietri 7y ago
Depends on what you're up to, I suppose. Having a particular chunk of data wrapped in a microservice has other benefits. If that's the direction one is headed anyhow, then wrapping SQLite + single access can be more tractable than allowing multiple access that you'll eventually have to rip out again. I know it works for plenty of people, but for me "database as integration layer" has always been an antipattern.
tracker1 7y ago
Depends on what you're doing... If you're working in descrete projects that need to be run, then archived.. using an RDBMS vs a service over SQLite is an actual consideration... if you have a remote API interface that talks to different SQLite db files on a per project basis, then backup and archival become trivial matters... if you're using a classic RDBMS then it can become much more complicated. Aside, adjusting schema over time also becomes easier as archived projects don't need to be updated, they just continue to exist with the older schema.

by pilif 7y ago

pdimitar 7y ago

Sure, I don't disagree. I've ditched sqlite for Postgres many times. It's just that for 99% of the projects I ever worked on, writes are like 100x less than reads so wrapping the writes in a queue of sorts has been quite okay and performant. It can and it has been coming to a point when it's easier to use a full-blown database server, too. I was simply pointing out that for a lot of classic workflows wrapping/centralising writes works quite fine.

logicallee 7y ago

As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes. so if everyone's saying this, is there such a standard dummy program?

aikah 7y ago
> As you quoted, people in this discussion are saying that writing a dummy program that has its own sqlite and does nothing but pass messages to it that it receives from all other processes on the system that need to talk to that sqlite file results in much better performance than accessing that file "directly" from the separate processes. And use what protocol for inter-process communication with the daemon managing SQLite? At the end of the day you just created the equivalent of a server anyway...
- logicallee 7y ago
  Any protocol. People are saying this is a common solution. So is there a standard dummy tool like that?
  - aikah 7y ago
    > So is there a standard dummy tool like that? My point is that by that time SQlite doesn't fit "Serveless" as defined by the linked article anymore.
    - logicallee 7y ago
      True
pdimitar 7y ago
There shouldn’t be should a such standard program. It’s up to you to implement it in-process so you reap all the benefits of sqlite’s speed and lack of dedicated server process. In the case of Erlang/Elixir where I most actively work in the last few years, it’s really easy to centralise write access (dedicated actor receiving messages). With other languages it should also be fairly easy.
tracker1 7y ago
Given the extensions that people compile into SQLite, I don't think one can even say that... there's mention of ActorDB in another thread that seems to cover this... Wrapping a typical API in either GRPC or even over HTTP isn't too hard.. you take in a parameterized query with parameters using library X and return JSON stringified to that library... you may want more specific interfaces, or even GraphQL over the database for that matter... it really depends on your needs.

wpietri 7y ago

Depends on what you're up to, I suppose. Having a particular chunk of data wrapped in a microservice has other benefits. If that's the direction one is headed anyhow, then wrapping SQLite + single access can be more tractable than allowing multiple access that you'll eventually have to rip out again. I know it works for plenty of people, but for me "database as integration layer" has always been an antipattern.

tracker1 7y ago

Depends on what you're doing... If you're working in descrete projects that need to be run, then archived.. using an RDBMS vs a service over SQLite is an actual consideration... if you have a remote API interface that talks to different SQLite db files on a per project basis, then backup and archival become trivial matters... if you're using a classic RDBMS then it can become much more complicated. Aside, adjusting schema over time also becomes easier as archived projects don't need to be updated, they just continue to exist with the older schema.