4 ms·
I wonder if this would make S3 a host for sqlite!
by hendry 6y ago
I wonder if this would make S3 a host for sqlite!
- foxhop 6y agoYup, I think it would.
- sudeepj 6y agoI have started working on this actually (the vfs being developed in Rust). The design was complex but now this simplifies a lot. The vfs will also have snapshotting capabilities i.e. you can have multiple versions of sqlite db.
- js4ever 6y agoHow are you handling latency (up to 800ms on S3) or the fact that you have to write the whole file each time?
- sudeepj 6y ago> you have to write the whole file each time Will not write the whole file. Instead each page of the database (default = 4KB) will be separate put. The use case is: one-writer multiple readers. The readers will read an old snapshot of the database (from s3) and writer will write to the active one. This way a lot of readers can open the DB in read-only mode. As for writer, each page will be a separate put. The writes will be appended to a log(s) locally and the page will be synced in the background. This helps to exploit lot of parallel PUTs to S3. Once the sync happens the disk space for that page will be reclaimed. For the writer: v = open_new_version() //inserts + updates //commit close_version(v) For readers: open_version_for_read(v) //select
- ComodoHacker 6y agoObviously, a sane implementation would divide db file into blocks and write them into separate objects. Not as small as traditional FS blocks, 1-4 Mb, perhaps.
- daviesliu 6y agoYes, JuiceFS splits files into 4MB blocks and put them into S3.
- speleding 6y agoYou could shard your SQLlite files to only contain the data for a single user, if your application allows you to do that. Would make it a bit painful to run migrations, but other than that you've now got an infinitely scaling database with 9 nines uptime for no money at all.