9 ms·

Store SQLite in Cloudflare Durable Objects

244 points by jgrahamc 5y ago

samwillis 5y ago
This is super exciting! I see there is concern about Asyncify. I believe in the browser there is a way around this using Atomics and a shared buffer (Absurd SQL[0] uses it), but suspect this isn't possible in CloudFlare Workers as it is working across the worker/main thread boundary? Is it possible to launch another worker from a CloudFlare worker, in the same process, like a web worker? I am somewhat hoping that CloudFlare have something up their sleeve to make these sorts of projects easer, maybe even some sort of distributed block store. 0: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql
- kentonv 5y ago
  Indeed Cloudflare Workers does not and likely will not support threads with shared memory, due to Spectre concerns. Asyncify is the way to go here.
  - samwillis 5y ago
    Thats a pity, is that due to the architecture of sharing a V8 runtime with other customers workers? How is this different from V8 in Chrome and shared buffers?
    - kentonv 5y ago
      Chrome is all-in on strict process isolation as their Spectre defense. Given the web platform that they need to support, they don't really have another choice -- precise timers and threads, as well as many other ad hoc timing mechanisms, were already part of the platform before Spectre hit. Using process for everything is pretty costly, though, to the point that they have to make some compromises which are open to attack: https://www.spookjs.com/ https://www.spookjs.com/ In Cloudflare Workers, process isolation for every worker isn't practical -- if we had to do that then we'd only be able to offer real "edge compute" to a small number of big enterprises with deep pockets, rather than at prices affordable to everyone. Our fundamental difficulty is that we need a single machine to be able to support thousands of running applications where each app may only get a small trickle of traffic, so that we can deploy every application to every one of our edge locations. It's like if you always had 10,000 tabs open in Chrome. Instead our strategy is to stack a lot of different mitigations that make attacks slower, until the point where they are too slow to be remotely useful[0]. One part of the strategy is a novel defense we designed with researchers at TU Graz called Dynamic Process Isolation[1], in which we use hardware performance counters to detect execution patterns indicative of an attack and move those workers into private processes. For that strategy to work, though, we first need to slow down attacks enough to give ourselves a chance to detect the patterns -- and that requires disallowing precise timers or anything that could be used as a precise timer, such as multiple threads with shared memory. Luckily, we were thinking about timing attacks from the very start of the project (even before Spectre was known), so we were able to avoid ever putting these into the platform in the first place. In general I think threads are not as important on servers because distributing work across machines is more powerful anyway. We're trying to create a platform where that is really easy. [0] https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... [1] https://blog.cloudflare.com/spectre-research-with-tu-graz/ https://blog.cloudflare.com/spectre-research-with-tu-graz/
      - samwillis 5y ago
        Thanks for the detailed explanation, makes complete sense. I suppose that means that for projects such as python-wasm [0] that are porting other language runtimes to WASM they will have to go the Asyncify route for CF Workers with all the overhead that entails? From the looks of it they don't think asyncifying Python will be possible in the near to medium term [1]. 0: https://github.com/ethanhs/python-wasm https://github.com/ethanhs/python-wasm 1: https://github.com/ethanhs/python-wasm/issues/12 https://github.com/ethanhs/python-wasm/issues/12
        kentonv 5y ago
        If this becomes a big enough problem, we might be able to build a work-around into the Cloudflare Workers Runtime itself so that it can support synchronous waits. It'll take some hacks but I think it's doable, without actually supporting threads.
        ec109685 5y ago
        Essentially doing what Asyncify is doing as part of the platform?
        kentonv 5y ago
        I'd probably use fibers, like node-fibers used to do. The problem is V8 stopped supporting fibers a few versions ago. We'd have to hack support back in. That's something we probably could get done but I'm not exactly excited about it. :)
      - meekins 5y ago
        > we'd only be able to offer real "edge compute" to a small number of big enterprises with deep pockets, rather than at prices affordable to everyone What a nice way to formulate the tradeoffs between cost and security. When Workers came out with huge headlines about performance and cost I was very disappointed to find out there was no special technical wizardry behind that but just conscious trade-offs in disregard to customer data security. Workers simply skip all the sandboxing steps other providers take to implement a multi-tenant application runtime in a secure manner. So far the mitigations in place seem to make the possibility of Cloudflare customers getting bitten by a V8 vulnerability unlikely instead of impossible. Sure, the platform has interesting ideas and I'm looking forward to try it out as a full-stack serverless platform. I just cannot foresee running anything serious on it before they come up with a more convincing security story.
quadrature 5y ago
A long these lines if you're looking for a way to sync your SQLite DB to an S3 like object store there is https://litestream.io/ https://litestream.io/. It creates a snapshot of your DB on s3 and then handles checkpointing of the WAL.
- Glench 5y ago
  Yes, love Litestream! It felt like the missing piece for replacing Postgres or MySQL with SQLite.
- WrtCdEvrydy 5y ago
  That is something I was looking for.
- wirelesspotat 5y ago
  Does anyone know if it's possible to use Litestream with SQLite running on Cloudflare Durable Objects? I suppose you'd have to compile Litestream to WASM or have Litestream running on a different machine and reading SQLite from the Durable Object?
  - benbjohnson 5y ago
    Litestream author here. I haven’t looked at Durable Objects yet so I don’t think Litestream would currently work with it. I may add support in the future but I’m adding separate serverless functionality to be released later this year. That’ll allow Litestream to run on Cloudflare Workers or GCR or other ephemeral platforms like those.
  - Glench 5y ago
    Haha check the original post, my friend
- deleted 5y ago
  [deleted]
Glench 5y ago
Whenever SQLite comes up, always have to link to this post about using SQLite in production: https://blog.wesleyac.com/posts/consider-sqlite https://blog.wesleyac.com/posts/consider-sqlite :) I'm running https://extensionpay.com https://extensionpay.com off SQLite and a $5/month DigitalOcean box and it's serving around 3 million requests a month with absolutely no issues and seriously low CPU load. I'm kind of astonished, frankly.
- nicoburns 5y ago
  > 3 million requests a month with absolutely no issues and seriously low CPU load That's just under 1 request per second, so I can't say I'm surprised. Even if traffic is spiky that would leave you with a lot of headroom!
  - Glench 5y ago
    Yeah, thanks for putting that in perspective! ExtensionPay is still relatively new, too, so I'm glad there's plenty of headroom :)
  - ryanianian 5y ago
    A more useful metric for the parent comment to include would be the peak throughput in ops/time versus CPU or IO load.
- rexreed 5y ago
  Do you have any tips load-balancing or horizontal scaling of SQLite if you have to have a 2nd server for whatever reason?
  - Glench 5y ago
    I don't. My current understanding is that even though there are sharding / replication projects out there for SQLite, it's usually a better idea to just go with a typical database server when you need many machines. But a lot of sites never get to that point, especially given how fast servers and SSDs are these days, so SQLite seems like it can get you further than you might realize.
psanford 5y ago
I built something very similar for using DynamoDB as a VFS backing store for sqlite: https://github.com/psanford/donutdb https://github.com/psanford/donutdb
eropple 5y ago
I read through this entire post going "oh, this is cursed", slowly growing in volume the entire time. As is proper for something this clever. I've been doing a bit of prodding at Durable Objects lately, in part inspired by the same note that the OP quoted about implementing datastores atop the framework, and this is the sort of environment that I could see being really, really useful at scale in the future. I'm anxious about using it without a public spec and some portability, but the concept is just tremendously cool.
- juancampa 5y ago
  What do you mean by "cursed"?
  - k__ 5y ago
    My guess in this context: "A hacky solution that probably is unstable"
  - eropple 5y ago
    Substitute "this is a terrible idea and I love it" if you want. ;)
- techgnosis 5y ago
  https://www.urbandictionary.com/define.php?term=Cursed https://www.urbandictionary.com/define.php?term=Cursed Second definition Similar to "janky"
hinkley 5y ago
I really think I want a “JSON” format for SQLite that is a strict subset of the SQLite file syntax that can be generated fairly trivially by other applications and read by SQLite. There are apps that produce JSON that would be better served by something I can run search queries upon and jq is nice but it’s not SQLite. Is there a protobuf implementation of the SQLite file format out there somewhere?
- ignoramous 5y ago
  > There are apps that produce JSON that would be better served by something I can run search queries upon... https://duckdb.org/ https://duckdb.org/ ?
- rileymat2 5y ago
  I am having trouble with the mapping? I presume this touches on the orm impedance problem? Unless sqlite dictates the json format. But then it probably won’t be the json you like to deal with?
- pphysch 5y ago
  PostgreSQL has excellent JSON support today, including advanced queries, indexing, constraints. There are many cheap/free managed offerings that could rival SQLite in operational simplicity. If you want a SQL interface to JSON with room to grow, that's your best bet IMO.
liveoneggs 5y ago
WebSql could have been..
vdm 5y ago
sqlite vfs on browser indexeddb: https://jlongster.com/future-sql-web https://jlongster.com/future-sql-web
mathgladiator 5y ago
I actually think this is onto something that I'm finding in a different way. Instead of a massive database, what if we had a key-value store mapping keys to tiny databases. This is, to some degree, what I'm building over at http://www.adama-lang.org/ http://www.adama-lang.org/ without a full SQL engine. Each document has tables, and the tables can be indexed. I have yet to find a usecase (in my domain) which requires joins. HOWEVER, I've had a ton of fun building it and I'm getting ready to start making games. I do believe it would be amazing to have a key-logger service where a reducer like sqlite/adama could come into collapse the log into a single file. The closest I see is from the Boki paper ( https://www.cs.utexas.edu/~zjia/boki-sosp21.pdf https://www.cs.utexas.edu/~zjia/boki-sosp21.pdf ) which was presented at SOSP21.
dantodor 5y ago
I see the bundle size after `npm install -S @rkusa/wasm-sqlite` it's a little bit above 4MB. To my current knowledge, limit is 1MB for workers. I asked one of the companies I work with and that uses CF extensively a few months ago for a limit increase, it took them like almost one month and gave us 2MB. So my question is: what is the limit increase one should ask? What was your experience with that and what I need to ask for if I want to run this? Other than that, great job, and thanks a lot for sharing with the community. Until now I was trying to get DuckDB to run in a durable object (they also have a WASM bundle), now I have an excellent alternative. Thanks again!

244 points by jgrahamc 5y ago

This is super exciting! I see there is concern about Asyncify. I believe in the browser there is a way around this using Atomics and a shared buffer (Absurd SQL[0] uses it), but suspect this isn't possible in CloudFlare Workers as it is working across the worker/main thread boundary? Is it possible to launch another worker from a CloudFlare worker, in the same process, like a web worker? I am somewhat hoping that CloudFlare have something up their sleeve to make these sorts of projects easer, maybe even some sort of distributed block store. 0: https://github.com/jlongster/absurd-sql https://github.com/jlongster/absurd-sql

kentonv 5y ago
Indeed Cloudflare Workers does not and likely will not support threads with shared memory, due to Spectre concerns. Asyncify is the way to go here.
- samwillis 5y ago
  Thats a pity, is that due to the architecture of sharing a V8 runtime with other customers workers? How is this different from V8 in Chrome and shared buffers?
  - kentonv 5y ago
    Chrome is all-in on strict process isolation as their Spectre defense. Given the web platform that they need to support, they don't really have another choice -- precise timers and threads, as well as many other ad hoc timing mechanisms, were already part of the platform before Spectre hit. Using process for everything is pretty costly, though, to the point that they have to make some compromises which are open to attack: https://www.spookjs.com/ https://www.spookjs.com/ In Cloudflare Workers, process isolation for every worker isn't practical -- if we had to do that then we'd only be able to offer real "edge compute" to a small number of big enterprises with deep pockets, rather than at prices affordable to everyone. Our fundamental difficulty is that we need a single machine to be able to support thousands of running applications where each app may only get a small trickle of traffic, so that we can deploy every application to every one of our edge locations. It's like if you always had 10,000 tabs open in Chrome. Instead our strategy is to stack a lot of different mitigations that make attacks slower, until the point where they are too slow to be remotely useful[0]. One part of the strategy is a novel defense we designed with researchers at TU Graz called Dynamic Process Isolation[1], in which we use hardware performance counters to detect execution patterns indicative of an attack and move those workers into private processes. For that strategy to work, though, we first need to slow down attacks enough to give ourselves a chance to detect the patterns -- and that requires disallowing precise timers or anything that could be used as a precise timer, such as multiple threads with shared memory. Luckily, we were thinking about timing attacks from the very start of the project (even before Spectre was known), so we were able to avoid ever putting these into the platform in the first place. In general I think threads are not as important on servers because distributing work across machines is more powerful anyway. We're trying to create a platform where that is really easy. [0] https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec... [1] https://blog.cloudflare.com/spectre-research-with-tu-graz/ https://blog.cloudflare.com/spectre-research-with-tu-graz/
    - samwillis 5y ago
      Thanks for the detailed explanation, makes complete sense. I suppose that means that for projects such as python-wasm [0] that are porting other language runtimes to WASM they will have to go the Asyncify route for CF Workers with all the overhead that entails? From the looks of it they don't think asyncifying Python will be possible in the near to medium term [1]. 0: https://github.com/ethanhs/python-wasm https://github.com/ethanhs/python-wasm 1: https://github.com/ethanhs/python-wasm/issues/12 https://github.com/ethanhs/python-wasm/issues/12
      - kentonv 5y ago
        If this becomes a big enough problem, we might be able to build a work-around into the Cloudflare Workers Runtime itself so that it can support synchronous waits. It'll take some hacks but I think it's doable, without actually supporting threads.
        ec109685 5y ago
        Essentially doing what Asyncify is doing as part of the platform?
        kentonv 5y ago
        I'd probably use fibers, like node-fibers used to do. The problem is V8 stopped supporting fibers a few versions ago. We'd have to hack support back in. That's something we probably could get done but I'm not exactly excited about it. :)
    - meekins 5y ago
      > we'd only be able to offer real "edge compute" to a small number of big enterprises with deep pockets, rather than at prices affordable to everyone What a nice way to formulate the tradeoffs between cost and security. When Workers came out with huge headlines about performance and cost I was very disappointed to find out there was no special technical wizardry behind that but just conscious trade-offs in disregard to customer data security. Workers simply skip all the sandboxing steps other providers take to implement a multi-tenant application runtime in a secure manner. So far the mitigations in place seem to make the possibility of Cloudflare customers getting bitten by a V8 vulnerability unlikely instead of impossible. Sure, the platform has interesting ideas and I'm looking forward to try it out as a full-stack serverless platform. I just cannot foresee running anything serious on it before they come up with a more convincing security story.

quadrature 5y ago

A long these lines if you're looking for a way to sync your SQLite DB to an S3 like object store there is https://litestream.io/ https://litestream.io/. It creates a snapshot of your DB on s3 and then handles checkpointing of the WAL.

Glench 5y ago
Yes, love Litestream! It felt like the missing piece for replacing Postgres or MySQL with SQLite.
WrtCdEvrydy 5y ago
That is something I was looking for.
wirelesspotat 5y ago
Does anyone know if it's possible to use Litestream with SQLite running on Cloudflare Durable Objects? I suppose you'd have to compile Litestream to WASM or have Litestream running on a different machine and reading SQLite from the Durable Object?
- benbjohnson 5y ago
  Litestream author here. I haven’t looked at Durable Objects yet so I don’t think Litestream would currently work with it. I may add support in the future but I’m adding separate serverless functionality to be released later this year. That’ll allow Litestream to run on Cloudflare Workers or GCR or other ephemeral platforms like those.
- Glench 5y ago
  Haha check the original post, my friend
deleted 5y ago
[deleted]

Glench 5y ago

Whenever SQLite comes up, always have to link to this post about using SQLite in production: https://blog.wesleyac.com/posts/consider-sqlite https://blog.wesleyac.com/posts/consider-sqlite :) I'm running https://extensionpay.com https://extensionpay.com off SQLite and a $5/month DigitalOcean box and it's serving around 3 million requests a month with absolutely no issues and seriously low CPU load. I'm kind of astonished, frankly.

nicoburns 5y ago
> 3 million requests a month with absolutely no issues and seriously low CPU load That's just under 1 request per second, so I can't say I'm surprised. Even if traffic is spiky that would leave you with a lot of headroom!
- Glench 5y ago
  Yeah, thanks for putting that in perspective! ExtensionPay is still relatively new, too, so I'm glad there's plenty of headroom :)
- ryanianian 5y ago
  A more useful metric for the parent comment to include would be the peak throughput in ops/time versus CPU or IO load.
rexreed 5y ago
Do you have any tips load-balancing or horizontal scaling of SQLite if you have to have a 2nd server for whatever reason?
- Glench 5y ago
  I don't. My current understanding is that even though there are sharding / replication projects out there for SQLite, it's usually a better idea to just go with a typical database server when you need many machines. But a lot of sites never get to that point, especially given how fast servers and SSDs are these days, so SQLite seems like it can get you further than you might realize.

psanford 5y ago

I built something very similar for using DynamoDB as a VFS backing store for sqlite: https://github.com/psanford/donutdb https://github.com/psanford/donutdb

eropple 5y ago

I read through this entire post going "oh, this is cursed", slowly growing in volume the entire time. As is proper for something this clever. I've been doing a bit of prodding at Durable Objects lately, in part inspired by the same note that the OP quoted about implementing datastores atop the framework, and this is the sort of environment that I could see being really, really useful at scale in the future. I'm anxious about using it without a public spec and some portability, but the concept is just tremendously cool.

juancampa 5y ago
What do you mean by "cursed"?
- k__ 5y ago
  My guess in this context: "A hacky solution that probably is unstable"
- eropple 5y ago
  Substitute "this is a terrible idea and I love it" if you want. ;)
techgnosis 5y ago
https://www.urbandictionary.com/define.php?term=Cursed https://www.urbandictionary.com/define.php?term=Cursed Second definition Similar to "janky"

hinkley 5y ago

I really think I want a “JSON” format for SQLite that is a strict subset of the SQLite file syntax that can be generated fairly trivially by other applications and read by SQLite. There are apps that produce JSON that would be better served by something I can run search queries upon and jq is nice but it’s not SQLite. Is there a protobuf implementation of the SQLite file format out there somewhere?

ignoramous 5y ago
> There are apps that produce JSON that would be better served by something I can run search queries upon... https://duckdb.org/ https://duckdb.org/ ?
rileymat2 5y ago
I am having trouble with the mapping? I presume this touches on the orm impedance problem? Unless sqlite dictates the json format. But then it probably won’t be the json you like to deal with?
pphysch 5y ago
PostgreSQL has excellent JSON support today, including advanced queries, indexing, constraints. There are many cheap/free managed offerings that could rival SQLite in operational simplicity. If you want a SQL interface to JSON with room to grow, that's your best bet IMO.

liveoneggs 5y ago

WebSql could have been..

vdm 5y ago

sqlite vfs on browser indexeddb: https://jlongster.com/future-sql-web https://jlongster.com/future-sql-web

mathgladiator 5y ago

I actually think this is onto something that I'm finding in a different way. Instead of a massive database, what if we had a key-value store mapping keys to tiny databases. This is, to some degree, what I'm building over at http://www.adama-lang.org/ http://www.adama-lang.org/ without a full SQL engine. Each document has tables, and the tables can be indexed. I have yet to find a usecase (in my domain) which requires joins. HOWEVER, I've had a ton of fun building it and I'm getting ready to start making games. I do believe it would be amazing to have a key-logger service where a reducer like sqlite/adama could come into collapse the log into a single file. The closest I see is from the Boki paper ( https://www.cs.utexas.edu/~zjia/boki-sosp21.pdf https://www.cs.utexas.edu/~zjia/boki-sosp21.pdf ) which was presented at SOSP21.

dantodor 5y ago

I see the bundle size after `npm install -S @rkusa/wasm-sqlite` it's a little bit above 4MB. To my current knowledge, limit is 1MB for workers. I asked one of the companies I work with and that uses CF extensively a few months ago for a limit increase, it took them like almost one month and gave us 2MB. So my question is: what is the limit increase one should ask? What was your experience with that and what I need to ask for if I want to run this? Other than that, great job, and thanks a lot for sharing with the community. Until now I was trying to get DuckDB to run in a durable object (they also have a WASM bundle), now I have an excellent alternative. Thanks again!