7 ms·

I Migrated from a Postgres Cluster to Distributed SQLite with LiteFS

129 points by Fudgel 4y ago

Vt71fcAqt7 4y ago
Off topic I guess, but I've been thinking about a collaborative spreadsheet that uses something like Sqlite over LiteFS. Where each spreadsheet is just a sqlite file. Not sure what the architecture would look like though. Ideally the main server would just keep the files in sync, perhaps with some way to update each user without having to send the whole file. This has the advantage of programabilty and a familiar UI. All at just the cost of hosting the file and a webserver.
- endisneigh 4y ago
  you will find that maintaining schema consistency during migrations will be more trouble than its worth
  - Vt71fcAqt7 4y ago
    Hey thanks for your reply. Can you expound on what the issue would be? What do you mean by migrations?
    - endisneigh 4y ago
      Let’s say you have a hundred databases and you want to do a schema migration. You could naively do all of them at once, but what if they take a long time, or if one fails? What then? So you implement some thing to do rollbacks and retries. Now you’ve created a complicated monstrosity because you will have users dealing with potentially different databases with different schemas. You were better off with a single database with a sharding key.
      - Vt71fcAqt7 4y ago
        Hmmm thanks for the reply. Would that be true even if the database is just a single table each? In my mind I shouldn't need to do any schema migrations because I'm really just a file server and a UI. But I guess with adding more tables (akin to sheets in excel world) I would need to keep that in mind. I may try this when I get some free time, thanks for the feedback!
        DAlperin 4y ago
        The problem appears less when creating new tables, creating tables is relatively safe. But let's say in your one table per db you keep track of a few things (columns). For example you have a column for the name of the sheet and another column for the time it was created and some other miscellaneous columns. Later on once you have a hundred users or so you get a persistent feature request: keep track of the time it was last updated. You write up a simple migration, it looks something like ALTER TABLE your_table ADD updated_at TEXT; This is also relatively harmless, but now you have to figure out how to run this migration across your 100 databases, and deal with the case where maybe some fail, what do you do then? Rollback everywhere? But what if some users have already have rows that use the new column? Do you delete their data? Etc. Now consider a more complicated migration, maybe it renames a column, drops a column, adds a default value, drops an index, etc. You can see where problems might come from. That said I do really like your idea, I've gotten into sqlite a lot lately (I work on the database team at Fly.io), I'm just responding to you questions specifically about migrations.
        Vt71fcAqt7 4y ago
        Thanks for the detailed answer. I was thinking of a more hands-of approach in terms of what I do vs what the user does. So in terms of this feature request, it would actually be implemented client side. That is, I would update the client to ask the user if they want this new feature, then use IF EXISTS to see if the table already exists, and if it does, tell the user to fix it themselves. Or could we keep all the metadata in a seperate table and JOIN it on the ID? In any event, I'm literally just a reliable file server and a UI in this equation (plus backups). For context, this would really be aimed at developers and admins to manage data for their clients or co-workers. Stuff that currently is an excel sheet, but needs to leave excel world to do some mundane task, such as filling a pdf and uplading it somewhere. You could do this is word mailmerge + excel, but now you (the admin) are in charge of performing this mundane task every week. And belive me this kind of thing is very common. In fact, the one special thing I may do is somehow write the data into an excel file and somehow allow them to sync to each other (not the logic part from either side though). Not even sure if I would try to sell or just make it for free. It wouldn't be something streamlined like airtable, although if it got interest maybe I would take your approach. If you like my idea btw, feel free to use it! I have nothing but respect for fly.io, I used it breifly for a side project and it was realy simple to set up. You saying you like my idea actually gives me confidence in trying it out, thank you!
- chillfox 4y ago
  LiteFS is meant to get write forwarding in an upcoming release, so you could use maybe use that to sync, assuming the writes are less than about 10/s.
LAC-Tech 4y ago
I have a couple of questions... - So LiteFS turns sqlite into a client server database. But SQLite being embedded is what gives it its low latency in the first place. So, I kind of have to ask - what's point? Just prefering the way sqlite does SQL over postgres? - What language is that database model under "Evaluating the existing data"?
- ergonaught 4y ago
  The Litestream and LiteFS sections near the beginning of the article cover your first question. And I guess the second one is covered by the "generator" entry, but: https://www.prisma.io/docs/concepts/components/prisma-client https://www.prisma.io/docs/concepts/components/prisma-client
  - sureglymop 4y ago
    Do you like Prisma? I think it's cool but it sucks that their prisma schema doesn't support constraints and more complex things. It is very basic and for any real world use migrations need to be used. And I've had issues using it with SQLite. It doesn't support case insensitive filtering when using SQLite (I know, neither does SQLite). But as a result, i have to build a custom query which is pretty much what i wanted to avoid by using Prisma.
    - davnicwil 4y ago
      I really like Prisma, and talk about it to anyone who'll listen, but it is still very new and as such I think doesn't have a lot of features figured out yet and some really surprising things do come up. One I discovered just the other day [0] was that if you pass an undefined value into a field in a where clause of findFirst, it's treated the same as if you didn't pass anything at all. If you do this with an id, for example, this can result in just the first record in the table being returned, because it's as though you have no where clause at all. Naturally this can lead to some surprising bugs if undefined values crop up at runtime, even potentially some really bad security bugs (imagine if this happens on the account table for instance). I don't mean this as a criticism, I think it is just a normal part of being new and exciting software, maturing over time. But it's always a tradeoff to be aware of. [0] https://github.com/prisma/prisma/issues/5149#issuecomment-1368247621 https://github.com/prisma/prisma/issues/5149#issuecomment-13...
js-celebrity 4y ago
Site takes several seconds to load. Author completely overhauls an application just over one year old which was marketed at the time as a prescription for modern web development. Migration plagued by errors and implemented with obvious inefficiencies in data handling. Feature parity with old application not yet reached. Handling analytics with a relational db.
- _a_a_a_ 4y ago
  Site loads instantly for me with JS disabled but after that I've never seen anything like it. It's all narrow columns overlapping.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- bicijay 4y ago
  Whats the problem with handling analytics data with a relational db?
  - mr90210 4y ago
    Go read about OLAP vs OLTP, those keywords will lead to the answer you are seeking.
    - Tostino 4y ago
      It's more about transactional vs analytical systems. You want those decoupled. You can absolutely still use a relational DB for your analytics database, it can even be column oriented and vectorized if you want.
- mr90210 4y ago
  Kent is a great Developer specialized in Frontend, but his whole Full-Stack experiment is rather overengineered.
danpalmer 4y ago
Interesting article covering the nitty gritty details of production deployments. I can’t help but think something is wrong with anonymous users though? Loading the site was noticeably slow in MEL despite a SYD deployment. Took ~8s to load the page on a fast home fibre connection, abnormally slow for local sites, typical for US-only deployments. I wonder if anonymous users are unnecessarily being given a session for tracking purposes, and that is causing the slowness. Cycling session IDs for authenticated users makes sense for the UX benefit of not having to log in again, but that speed issue is terrible for low intent anonymous users landing on the site for the first time.
pcthrowaway 4y ago
Good writeup on using LiteStream at this early stage, though I wouldn't say it's production-ready based on a skim, though it might be a good fit for read-heavy apps which don't have a lot of data. One thing that's not clear to me is whether Litestream+SQLite works in-memory. If Litestream wrote to disk on the primary node, but kept the full DB in-memory for facilitating reads, I can definitely see how it would be a lot faster than Postgres (as the author claims). Though I'd also like to know if any RDBMSs have been designed for that use case, as I'd imagine traditional DB optimizations have been made for high incidence of disk access that would no longer hold true if the entire DB was in-memory
- nickcox 4y ago
  Even reading from disk ought to be way faster than the typical Postgres use case of going over the network, shouldn't it?
  - chillfox 4y ago
    It's faster even if the PostgreSQL instance is on the same box. If PostgreSQL is on a different box, then it's a lot faster.
  - pcthrowaway 4y ago
    I would expect Postgres to be faster for complex operations over large datasets (of course with a helping of "It depends!"), but very interested in knowing if this is generally untrue.
    - mrkurt 4y ago
      This is generally untrue, depending on what you mean by "faster". The way most web apps use DBs, sqlite is incredibly fast. Because most web apps make multiple queries, often sequentially, and they're very sensitive to DB latency. The network overhead adds up: https://twitter.com/benbjohnson/status/1514969870529560580 https://twitter.com/benbjohnson/status/1514969870529560580
- mhio 4y ago
sakopov 4y ago
Great writeup. Thanks for covering DB writes. I was curious how that worked with SQLite and LiteFS.
rish1_2 4y ago
Does this mean an entire copy of the DB is stored in-app? Isn't that a bad thing for large sites?
- chillfox 4y ago
  Yep, one full copy of the DB on each app server. So yes, it's bad if your data is 100s of GB, but if you only have in the low 10s GB of data or less then it's fine. It all depends upon how many server you need to run your app, If you only have one or two server then even very large databases of 100s of GB could be just fine.
  - mrkurt 4y ago
    It's often better than you'd expect, because the alternative is usually in memory caching. Disk is cheap, RAM is less cheap.
  - kgeist 4y ago
    Interestingly, our SRE people specifically have a rule "never share the DB and the app on the same server", because apparently a high load on the DB can decrease the overall performance of the server. It also allows them to freely add new DB servers (sharding) without touching the app server
    - chillfox 4y ago
      That makes sense with client/server databases and if you are trying to separately manage database resources from app resources. When you use SQLite you have basically decided that the database and the app is one integrated unit that you will manage as a whole.
      - deleted 4y ago
        [deleted]
klabb3 4y ago
I get that relational dbs like both Postgres and SQLite are extremely battletested and reliable, incredible pieces of tech, with tons of features. But what I’m reading here is a minefield of footguns, for fairly trivial logic, so I’m wondering if these tools were really designed for these use cases. Now, I don’t know if every server-side app NEEDS to run in 10 regions, auto-refresh session cookies, notify when someone starts typing a comment, report metrics every time the user moves the mouse pointer (or whatever else the kids do for fun these days). But let’s assume for a moment that's the common case. My understanding is that these relational dbs require a lot of config, awareness of the difficult distributed issues, and other layers such as caching, batching and so on, all of which is incredibly error-prone. In fact, the most important parts of the ground laws of data manipulation isn’t even part of a schema! I could worry less about enums and nulls, in a world with orders of magnitude more dangerous creatures: losing writes because I wrote to a replica, reading or writing inconsistent state, duplicate writes etc etc. It feels like we have a lot of structure around the easy stuff, but no structure around the hard parts. As a mere a dabbler with databases, can someone help calm my nerves?
- paulryanrogers 4y ago
  Core issue is usually multiple writable nodes, therefore conflicts and acting on outdated data. Then there are isolation level issues. More distributed solutions usually opt for eventual consistency. Enforcing a single writable nodes / primary can go a long way. Of course you've got a lot more options if your data naturally partitions, with few or no relationships among them.
- benbjohnson 4y ago
  LiteFS author here. I agree with you that most applications don't need to have all the distributed bells and whistles. Most projects that I run start out with just SQLite on a single node with Litestream and/or a cron backup. However, the number of applications that need to reduce latency in multiple regions is growing and the options tend to be pretty complicated or expensive. LiteFS aims to be a low-cost, self-hosted alternative to those options. > I could worry less about enums and nulls, in a world with orders of magnitude more dangerous creatures: losing writes because I wrote to a replica, reading or writing inconsistent state, duplicate writes etc etc LiteFS uses a primary/replica setup (as do many distributed databases) where the primary can perform writes and replicas are read-only. You won't be able to lose writes written to a replica because LiteFS doesn't allow that. As for inconsistent state, there is a transaction ID that lets you track your replication position[1]. You can implement strict serializability across your cluster using that. Then for duplicate writes, all writes go to the primary so it has the same serializable guarantees as regular SQLite. I hope that explanation helps. Let me know if you have any other questions. [1]: https://fly.io/docs/litefs/position/ https://fly.io/docs/litefs/position/
jacob019 4y ago
Seems there is a bit of vendor lock-in here with flyio, the way he's throwing custom http error codes for their infra. Can I just run LiteFS on a few virtual machines and forward writes back to master? With LiteFS, what happens if master goes down? Can a new one be elected?
- mildbyte 4y ago
  You can indeed run LiteFS by yourself, without Consul, as a sidecar / wrapper around your application. We do it in our project and have a Docker Compose example at [0]. In this case, you specify a specific known leader node. We haven't tried getting it running independently with Consul to do leader election / failover. [0] https://github.com/splitgraph/seafowl/blob/main/examples/litefs/configs/seafowl-writer/litefs.yml#L4 https://github.com/splitgraph/seafowl/blob/main/examples/lit...
- benbjohnson 4y ago
  > Seems there is a bit of vendor lock-in here with flyio, the way he's throwing custom http error codes for their infra. Can I just run LiteFS on a few virtual machines and forward writes back to master? LiteFS author here. You can run LiteFS anywhere -- even on a laptop with a few instances. There's no special Fly.io lock-in. The "fly-replay" is some syntactic sugar for our proxy that makes it easy to redirect requests but you could do the same with regular HTTP proxying. > With LiteFS, what happens if master goes down? Can a new one be elected? Yes, if you use Consul-based leasing then a new primary will come up automatically. Consul sessions have a minimum TTL of 10 seconds so you could lose write availability for up to that amount of time in the event of unexpectedly losing the primary. If you cleanly shutdown the primary then it will release its session and another node will pick up the lease right away. The other option is static leasing where a single node will always be primary. This is a simpler option but you do lose write availability if your primary node goes down.

129 points by Fudgel 4y ago

Off topic I guess, but I've been thinking about a collaborative spreadsheet that uses something like Sqlite over LiteFS. Where each spreadsheet is just a sqlite file. Not sure what the architecture would look like though. Ideally the main server would just keep the files in sync, perhaps with some way to update each user without having to send the whole file. This has the advantage of programabilty and a familiar UI. All at just the cost of hosting the file and a webserver.

endisneigh 4y ago
you will find that maintaining schema consistency during migrations will be more trouble than its worth
- Vt71fcAqt7 4y ago
  Hey thanks for your reply. Can you expound on what the issue would be? What do you mean by migrations?
  - endisneigh 4y ago
    Let’s say you have a hundred databases and you want to do a schema migration. You could naively do all of them at once, but what if they take a long time, or if one fails? What then? So you implement some thing to do rollbacks and retries. Now you’ve created a complicated monstrosity because you will have users dealing with potentially different databases with different schemas. You were better off with a single database with a sharding key.
    - Vt71fcAqt7 4y ago
      Hmmm thanks for the reply. Would that be true even if the database is just a single table each? In my mind I shouldn't need to do any schema migrations because I'm really just a file server and a UI. But I guess with adding more tables (akin to sheets in excel world) I would need to keep that in mind. I may try this when I get some free time, thanks for the feedback!
      - DAlperin 4y ago
        The problem appears less when creating new tables, creating tables is relatively safe. But let's say in your one table per db you keep track of a few things (columns). For example you have a column for the name of the sheet and another column for the time it was created and some other miscellaneous columns. Later on once you have a hundred users or so you get a persistent feature request: keep track of the time it was last updated. You write up a simple migration, it looks something like ALTER TABLE your_table ADD updated_at TEXT; This is also relatively harmless, but now you have to figure out how to run this migration across your 100 databases, and deal with the case where maybe some fail, what do you do then? Rollback everywhere? But what if some users have already have rows that use the new column? Do you delete their data? Etc. Now consider a more complicated migration, maybe it renames a column, drops a column, adds a default value, drops an index, etc. You can see where problems might come from. That said I do really like your idea, I've gotten into sqlite a lot lately (I work on the database team at Fly.io), I'm just responding to you questions specifically about migrations.
        Vt71fcAqt7 4y ago
        Thanks for the detailed answer. I was thinking of a more hands-of approach in terms of what I do vs what the user does. So in terms of this feature request, it would actually be implemented client side. That is, I would update the client to ask the user if they want this new feature, then use IF EXISTS to see if the table already exists, and if it does, tell the user to fix it themselves. Or could we keep all the metadata in a seperate table and JOIN it on the ID? In any event, I'm literally just a reliable file server and a UI in this equation (plus backups). For context, this would really be aimed at developers and admins to manage data for their clients or co-workers. Stuff that currently is an excel sheet, but needs to leave excel world to do some mundane task, such as filling a pdf and uplading it somewhere. You could do this is word mailmerge + excel, but now you (the admin) are in charge of performing this mundane task every week. And belive me this kind of thing is very common. In fact, the one special thing I may do is somehow write the data into an excel file and somehow allow them to sync to each other (not the logic part from either side though). Not even sure if I would try to sell or just make it for free. It wouldn't be something streamlined like airtable, although if it got interest maybe I would take your approach. If you like my idea btw, feel free to use it! I have nothing but respect for fly.io, I used it breifly for a side project and it was realy simple to set up. You saying you like my idea actually gives me confidence in trying it out, thank you!
chillfox 4y ago
LiteFS is meant to get write forwarding in an upcoming release, so you could use maybe use that to sync, assuming the writes are less than about 10/s.

LAC-Tech 4y ago

I have a couple of questions... - So LiteFS turns sqlite into a client server database. But SQLite being embedded is what gives it its low latency in the first place. So, I kind of have to ask - what's point? Just prefering the way sqlite does SQL over postgres? - What language is that database model under "Evaluating the existing data"?

ergonaught 4y ago
The Litestream and LiteFS sections near the beginning of the article cover your first question. And I guess the second one is covered by the "generator" entry, but: https://www.prisma.io/docs/concepts/components/prisma-client https://www.prisma.io/docs/concepts/components/prisma-client
- sureglymop 4y ago
  Do you like Prisma? I think it's cool but it sucks that their prisma schema doesn't support constraints and more complex things. It is very basic and for any real world use migrations need to be used. And I've had issues using it with SQLite. It doesn't support case insensitive filtering when using SQLite (I know, neither does SQLite). But as a result, i have to build a custom query which is pretty much what i wanted to avoid by using Prisma.
  - davnicwil 4y ago
    I really like Prisma, and talk about it to anyone who'll listen, but it is still very new and as such I think doesn't have a lot of features figured out yet and some really surprising things do come up. One I discovered just the other day [0] was that if you pass an undefined value into a field in a where clause of findFirst, it's treated the same as if you didn't pass anything at all. If you do this with an id, for example, this can result in just the first record in the table being returned, because it's as though you have no where clause at all. Naturally this can lead to some surprising bugs if undefined values crop up at runtime, even potentially some really bad security bugs (imagine if this happens on the account table for instance). I don't mean this as a criticism, I think it is just a normal part of being new and exciting software, maturing over time. But it's always a tradeoff to be aware of. [0] https://github.com/prisma/prisma/issues/5149#issuecomment-1368247621 https://github.com/prisma/prisma/issues/5149#issuecomment-13...

js-celebrity 4y ago

Site takes several seconds to load. Author completely overhauls an application just over one year old which was marketed at the time as a prescription for modern web development. Migration plagued by errors and implemented with obvious inefficiencies in data handling. Feature parity with old application not yet reached. Handling analytics with a relational db.

_a_a_a_ 4y ago
Site loads instantly for me with JS disabled but after that I've never seen anything like it. It's all narrow columns overlapping.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
bicijay 4y ago
Whats the problem with handling analytics data with a relational db?
- mr90210 4y ago
  Go read about OLAP vs OLTP, those keywords will lead to the answer you are seeking.
  - Tostino 4y ago
    It's more about transactional vs analytical systems. You want those decoupled. You can absolutely still use a relational DB for your analytics database, it can even be column oriented and vectorized if you want.
mr90210 4y ago
Kent is a great Developer specialized in Frontend, but his whole Full-Stack experiment is rather overengineered.

danpalmer 4y ago

Interesting article covering the nitty gritty details of production deployments. I can’t help but think something is wrong with anonymous users though? Loading the site was noticeably slow in MEL despite a SYD deployment. Took ~8s to load the page on a fast home fibre connection, abnormally slow for local sites, typical for US-only deployments. I wonder if anonymous users are unnecessarily being given a session for tracking purposes, and that is causing the slowness. Cycling session IDs for authenticated users makes sense for the UX benefit of not having to log in again, but that speed issue is terrible for low intent anonymous users landing on the site for the first time.

pcthrowaway 4y ago

Good writeup on using LiteStream at this early stage, though I wouldn't say it's production-ready based on a skim, though it might be a good fit for read-heavy apps which don't have a lot of data. One thing that's not clear to me is whether Litestream+SQLite works in-memory. If Litestream wrote to disk on the primary node, but kept the full DB in-memory for facilitating reads, I can definitely see how it would be a lot faster than Postgres (as the author claims). Though I'd also like to know if any RDBMSs have been designed for that use case, as I'd imagine traditional DB optimizations have been made for high incidence of disk access that would no longer hold true if the entire DB was in-memory

nickcox 4y ago
Even reading from disk ought to be way faster than the typical Postgres use case of going over the network, shouldn't it?
- chillfox 4y ago
  It's faster even if the PostgreSQL instance is on the same box. If PostgreSQL is on a different box, then it's a lot faster.
- pcthrowaway 4y ago
  I would expect Postgres to be faster for complex operations over large datasets (of course with a helping of "It depends!"), but very interested in knowing if this is generally untrue.
  - mrkurt 4y ago
    This is generally untrue, depending on what you mean by "faster". The way most web apps use DBs, sqlite is incredibly fast. Because most web apps make multiple queries, often sequentially, and they're very sensitive to DB latency. The network overhead adds up: https://twitter.com/benbjohnson/status/1514969870529560580 https://twitter.com/benbjohnson/status/1514969870529560580
mhio 4y ago

sakopov 4y ago

Great writeup. Thanks for covering DB writes. I was curious how that worked with SQLite and LiteFS.

rish1_2 4y ago

Does this mean an entire copy of the DB is stored in-app? Isn't that a bad thing for large sites?

chillfox 4y ago
Yep, one full copy of the DB on each app server. So yes, it's bad if your data is 100s of GB, but if you only have in the low 10s GB of data or less then it's fine. It all depends upon how many server you need to run your app, If you only have one or two server then even very large databases of 100s of GB could be just fine.
- mrkurt 4y ago
  It's often better than you'd expect, because the alternative is usually in memory caching. Disk is cheap, RAM is less cheap.
- kgeist 4y ago
  Interestingly, our SRE people specifically have a rule "never share the DB and the app on the same server", because apparently a high load on the DB can decrease the overall performance of the server. It also allows them to freely add new DB servers (sharding) without touching the app server
  - chillfox 4y ago
    That makes sense with client/server databases and if you are trying to separately manage database resources from app resources. When you use SQLite you have basically decided that the database and the app is one integrated unit that you will manage as a whole.
    - deleted 4y ago
      [deleted]

klabb3 4y ago

I get that relational dbs like both Postgres and SQLite are extremely battletested and reliable, incredible pieces of tech, with tons of features. But what I’m reading here is a minefield of footguns, for fairly trivial logic, so I’m wondering if these tools were really designed for these use cases. Now, I don’t know if every server-side app NEEDS to run in 10 regions, auto-refresh session cookies, notify when someone starts typing a comment, report metrics every time the user moves the mouse pointer (or whatever else the kids do for fun these days). But let’s assume for a moment that's the common case. My understanding is that these relational dbs require a lot of config, awareness of the difficult distributed issues, and other layers such as caching, batching and so on, all of which is incredibly error-prone. In fact, the most important parts of the ground laws of data manipulation isn’t even part of a schema! I could worry less about enums and nulls, in a world with orders of magnitude more dangerous creatures: losing writes because I wrote to a replica, reading or writing inconsistent state, duplicate writes etc etc. It feels like we have a lot of structure around the easy stuff, but no structure around the hard parts. As a mere a dabbler with databases, can someone help calm my nerves?

paulryanrogers 4y ago
Core issue is usually multiple writable nodes, therefore conflicts and acting on outdated data. Then there are isolation level issues. More distributed solutions usually opt for eventual consistency. Enforcing a single writable nodes / primary can go a long way. Of course you've got a lot more options if your data naturally partitions, with few or no relationships among them.
benbjohnson 4y ago
LiteFS author here. I agree with you that most applications don't need to have all the distributed bells and whistles. Most projects that I run start out with just SQLite on a single node with Litestream and/or a cron backup. However, the number of applications that need to reduce latency in multiple regions is growing and the options tend to be pretty complicated or expensive. LiteFS aims to be a low-cost, self-hosted alternative to those options. > I could worry less about enums and nulls, in a world with orders of magnitude more dangerous creatures: losing writes because I wrote to a replica, reading or writing inconsistent state, duplicate writes etc etc LiteFS uses a primary/replica setup (as do many distributed databases) where the primary can perform writes and replicas are read-only. You won't be able to lose writes written to a replica because LiteFS doesn't allow that. As for inconsistent state, there is a transaction ID that lets you track your replication position[1]. You can implement strict serializability across your cluster using that. Then for duplicate writes, all writes go to the primary so it has the same serializable guarantees as regular SQLite. I hope that explanation helps. Let me know if you have any other questions. [1]: https://fly.io/docs/litefs/position/ https://fly.io/docs/litefs/position/

jacob019 4y ago

Seems there is a bit of vendor lock-in here with flyio, the way he's throwing custom http error codes for their infra. Can I just run LiteFS on a few virtual machines and forward writes back to master? With LiteFS, what happens if master goes down? Can a new one be elected?

mildbyte 4y ago
You can indeed run LiteFS by yourself, without Consul, as a sidecar / wrapper around your application. We do it in our project and have a Docker Compose example at [0]. In this case, you specify a specific known leader node. We haven't tried getting it running independently with Consul to do leader election / failover. [0] https://github.com/splitgraph/seafowl/blob/main/examples/litefs/configs/seafowl-writer/litefs.yml#L4 https://github.com/splitgraph/seafowl/blob/main/examples/lit...
benbjohnson 4y ago
> Seems there is a bit of vendor lock-in here with flyio, the way he's throwing custom http error codes for their infra. Can I just run LiteFS on a few virtual machines and forward writes back to master? LiteFS author here. You can run LiteFS anywhere -- even on a laptop with a few instances. There's no special Fly.io lock-in. The "fly-replay" is some syntactic sugar for our proxy that makes it easy to redirect requests but you could do the same with regular HTTP proxying. > With LiteFS, what happens if master goes down? Can a new one be elected? Yes, if you use Consul-based leasing then a new primary will come up automatically. Consul sessions have a minimum TTL of 10 seconds so you could lose write availability for up to that amount of time in the event of unexpectedly losing the primary. If you cleanly shutdown the primary then it will release its session and another node will pick up the lease right away. The other option is static leasing where a single node will always be primary. This is a simpler option but you do lose write availability if your primary node goes down.