4 ms·

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream an

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream and scripts for too long. I've been bugging cloudflare to do this with D1+tunnels since D1 was announced and they constantly seemed confused what I was even talking about.

jgrahamc 3y ago
Email me and tell me jgc@cloudflare.com
benbjohnson 3y ago
Let me know how it works for you if you get a chance to try it out. We learned a lot since making Litestream so hopefully it's a big step up! The underlying format of how we're storing snapshots and incremental backups should make restores A LOT faster. Litestream has to replay all WAL writes since the last snapshot whereas LiteFS is able to perform a lot of deduplication and use multi-level compaction to create a minimal set of restore data.
bjyule 3y ago
What do you have in mind with respect to D1+Tunnels? We're launching PITR support for the new experimental backend within the next 1-2 weeks. Stay tuned. (I am the eng director for Cloudflare Workers)
- deleted 3y ago
  [deleted]
- Spunkie 3y ago
  I wanted this when D1 was first teased, and before I figured out that D1 is a database product that happens to be built on SQLite rather than SQLite being the star of the D1 show. Nonetheless, what I want is to be able to point a CF tunnel at a local SQLite file and instantly be able to leverage all the benefits of D1. I should be able to... 1. Configure a CF tunnel through the config file or the dash, pointing it to a local path of an existing SQLite file. 1a. On startup, the SQLite file should create and link a new D1 db. 2. Configure a CF tunnel through the config file or the dash, pointing it to an existing D1 db. 2a. On startup, the local sqlite db should be either overwritten or created. 3. Create read replicates through CF tunnels on another server or the local computer/wrangler for development. 4. Have automatic global read replicas on the edge, which is inherent to D1 no? 5. Have automatic/rolling backups and export and import those backups automatically with S3/R2. 6. Do a PITR through the dash or CF tunnel CLI. 6a. PITR should be possible to either the existing live DB or restoring it into a separate new D1 copy. 7. Leverage my SQlite files in workers, another automatic bonus of connecting it to D1. There are likely plenty of more cool things that could be done if D1 could be exposed as a normal SQLite file. Ultimately, CF tunnels are just in the mix because it seems like the obvious choice for plumbing SQLite files into and out of the CF network/edge.

jgrahamc 3y ago

Email me and tell me jgc@cloudflare.com

benbjohnson 3y ago

Let me know how it works for you if you get a chance to try it out. We learned a lot since making Litestream so hopefully it's a big step up! The underlying format of how we're storing snapshots and incremental backups should make restores A LOT faster. Litestream has to replay all WAL writes since the last snapshot whereas LiteFS is able to perform a lot of deduplication and use multi-level compaction to create a minimal set of restore data.

bjyule 3y ago

What do you have in mind with respect to D1+Tunnels? We're launching PITR support for the new experimental backend within the next 1-2 weeks. Stay tuned. (I am the eng director for Cloudflare Workers)

deleted 3y ago
[deleted]
Spunkie 3y ago
I wanted this when D1 was first teased, and before I figured out that D1 is a database product that happens to be built on SQLite rather than SQLite being the star of the D1 show. Nonetheless, what I want is to be able to point a CF tunnel at a local SQLite file and instantly be able to leverage all the benefits of D1. I should be able to... 1. Configure a CF tunnel through the config file or the dash, pointing it to a local path of an existing SQLite file. 1a. On startup, the SQLite file should create and link a new D1 db. 2. Configure a CF tunnel through the config file or the dash, pointing it to an existing D1 db. 2a. On startup, the local sqlite db should be either overwritten or created. 3. Create read replicates through CF tunnels on another server or the local computer/wrangler for development. 4. Have automatic global read replicas on the edge, which is inherent to D1 no? 5. Have automatic/rolling backups and export and import those backups automatically with S3/R2. 6. Do a PITR through the dash or CF tunnel CLI. 6a. PITR should be possible to either the existing live DB or restoring it into a separate new D1 copy. 7. Leverage my SQlite files in workers, another automatic bonus of connecting it to D1. There are likely plenty of more cool things that could be done if D1 could be exposed as a normal SQLite file. Ultimately, CF tunnels are just in the mix because it seems like the obvious choice for plumbing SQLite files into and out of the CF network/edge.