10 ms·

Kind of cool to see work on this. I do hope that the final db file result is still binary compatible with SQLite 3 in whatever direction Turso moves towards th

Kind of cool to see work on this. I do hope that the final db file result is still binary compatible with SQLite 3 in whatever direction Turso moves towards though... Rust or not. I've been advocating with several projects over recent years to get SQLite3 as an archive/export/interchange format for data. Need to archive 2019 data from the database, dump it into a SQLite db with roughly the same schema... Need to pass multiple CSVs worth of data dumps, use a SQLite file instead. As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection.

simonw 1y ago
SQLite against S3 can work with some clever tricks. The neatest version of that I've seen is still this WebAssembly one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... I also got sqlite-s3vfs working from Python a few months ago: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ Both of these are very much read-only mechanisms though.
- glommer 1y ago
  simon, you will be pleased to know that the python package for turso is in good shape! (and if you do find any issues please scream at us)
huntaub 1y ago
> As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection. You could achieve this today using one of the many adapters that turn S3 into a file system, without needing to wait for any SQLite buy in.
ncruces 1y ago
SQLite directly against S3 is workable if you mean querying a read-only database. For example, from Go, you could use my driver, and point it to a database file stored in S3 using this: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readervfs#example-package-Http https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readerv... For read-write it's a terrible idea. Object storage assumes objects are immutable. There may be some support for appends, but modifying the middle of an object in place involves copying the entire thing. What is on the verge of becoming viable is to use Litestream to do asynchronous replication to S3, and have read replicas that stream the data directly from S3. But what's stored in S3 isn't a database file, but a format created for the purpose called LTX.
- jtarchie 1y ago
  ncruces helped me with some code I made for VFS. It uses [zstd seekable](https://github.com/jtarchie/sqlitezstd https://github.com/jtarchie/sqlitezstd) for reading a file. I thought it would be really well-suited for S3. - Support for HTTP range queries - "Fast" read times - No disk required I was wrong. It turns out that for specific SQL queries, it might be fine, but not fast. For queries that do aggregations, like `COUNT`, sqlite loads the whole database anyway.

simonw 1y ago

SQLite against S3 can work with some clever tricks. The neatest version of that I've seen is still this WebAssembly one: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database... I also got sqlite-s3vfs working from Python a few months ago: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ Both of these are very much read-only mechanisms though.

glommer 1y ago
simon, you will be pleased to know that the python package for turso is in good shape! (and if you do find any issues please scream at us)

huntaub 1y ago

> As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection. You could achieve this today using one of the many adapters that turn S3 into a file system, without needing to wait for any SQLite buy in.

ncruces 1y ago

SQLite directly against S3 is workable if you mean querying a read-only database. For example, from Go, you could use my driver, and point it to a database file stored in S3 using this: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readervfs#example-package-Http https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readerv... For read-write it's a terrible idea. Object storage assumes objects are immutable. There may be some support for appends, but modifying the middle of an object in place involves copying the entire thing. What is on the verge of becoming viable is to use Litestream to do asynchronous replication to S3, and have read replicas that stream the data directly from S3. But what's stored in S3 isn't a database file, but a format created for the purpose called LTX.

jtarchie 1y ago
ncruces helped me with some code I made for VFS. It uses [zstd seekable](https://github.com/jtarchie/sqlitezstd https://github.com/jtarchie/sqlitezstd) for reading a file. I thought it would be really well-suited for S3. - Support for HTTP range queries - "Fast" read times - No disk required I was wrong. It turns out that for specific SQL queries, it might be fine, but not fast. For queries that do aggregations, like `COUNT`, sqlite loads the whole database anyway.