5 ms·

How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.

by haolez 3y ago

Xeoncross 3y ago
The linked https://blog.cloudflare.com/d1-turning-it-up-to-11/ https://blog.cloudflare.com/d1-turning-it-up-to-11/ page shows pricing for databases over 5GB
tiernano 3y ago
the docs say the Alpha is limited to 100Mb, and, given the size of SQLite DBs i have seen, 100Mb is quite large, is it not?
- electroly 3y ago
  SQLite is readily capable of vastly bigger databases than that. Whatever limitation there is here isn't from SQLite.
- remram 3y ago
  The SQLite website recommends using a client/server DBMS "when the size of the content looks like it might creep into the terabyte range" https://sqlite.org/whentouse.html https://sqlite.org/whentouse.html
ignoramous 3y ago
afaik, that's how it is envisioned to be used. Many databases rather one big monolith. The era of microdbs is here. Who's not going to bite? See also: https://archive.is/e7u9x https://archive.is/e7u9x / https://www.the-paper-trail.org/post/2020-04-06-physalia https://www.the-paper-trail.org/post/2020-04-06-physalia
kentonv 3y ago
The maximum size will increase to at least 1GB in the near future, enabled by our recent rewrite of the underlying storage layer. Perhaps we can push it further, even, we'll see. But yes, I think the next step is then some sort of sharding. Sharding by user would be an obvious approach for many apps. I think we should build a framework to help manage this, so apps would only need to provide some callbacks e.g. to compute shard key for a particular query. Alternatively, apps that want full control will be able to use Durable Objects directly. We'll soon (in a few months, probably?) enable the new storage engine for all Durable Objects which means every object will have a private SQLite database. (I'm the tech lead for Workers in general, and currently focused on this project in particular.)
- no_wizard 3y ago
  do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems. Much like R2 has an S3 compat API as well as a way to retrieve files outside of workers (IIRC) will this be true of all forms of storage on the platform? Sometimes it'd be nice to prime the cache with KV for example, without having to invoke a worker
  - elithrar 3y ago
    > do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems. For D1, yes — on the roadmap is a native HTTP API. KV has a HTTP API as well, so you can write directly.
- tebbers 3y ago
  Awesome, thanks for the info! We are building more and more features on Cloudflare and absolutely love the whole experience, faultless.
fauigerzigerk 3y ago
>Should I create one database per user, for example? You can't. D1 won't let you create databases at runtime. At least that was the case last time I tried it, and it's what killed it for me.
- ibejoeb 3y ago
  Not really sure how you distinguish between runtime and something else. It seems to me that you could provision a "shard" as simply as executing in a worker: wrangler d1 create customer28 --experimental-backend wrangler d1 execute customer28 --file=schema.sql
  - fauigerzigerk 3y ago
    What I mean is that workers cannot create databases. You have to create each database using wrangler (or the UI) and then edit the configuration file to add a binding. The binding becomes a variable in the JavaScript environment, which means you have to change the source code every time a new database is created. Therefore, creating a database per user as part of the user signup flow is not possible.
    - ibejoeb 3y ago
      Ah, I see. I thought it would be possible to dynamically set a secret which would effectively be a database uri. But upon further investigation, it does indeed need a redeploy. I suppose you could make something to write the bindings to the toml and trigger a redeploy, but that's definitely not pretty.

How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.

by haolez 3y ago

How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.

Xeoncross 3y ago

The linked https://blog.cloudflare.com/d1-turning-it-up-to-11/ https://blog.cloudflare.com/d1-turning-it-up-to-11/ page shows pricing for databases over 5GB

tiernano 3y ago

the docs say the Alpha is limited to 100Mb, and, given the size of SQLite DBs i have seen, 100Mb is quite large, is it not?

electroly 3y ago
SQLite is readily capable of vastly bigger databases than that. Whatever limitation there is here isn't from SQLite.
remram 3y ago
The SQLite website recommends using a client/server DBMS "when the size of the content looks like it might creep into the terabyte range" https://sqlite.org/whentouse.html https://sqlite.org/whentouse.html

ignoramous 3y ago

afaik, that's how it is envisioned to be used. Many databases rather one big monolith. The era of microdbs is here. Who's not going to bite? See also: https://archive.is/e7u9x https://archive.is/e7u9x / https://www.the-paper-trail.org/post/2020-04-06-physalia https://www.the-paper-trail.org/post/2020-04-06-physalia

kentonv 3y ago

The maximum size will increase to at least 1GB in the near future, enabled by our recent rewrite of the underlying storage layer. Perhaps we can push it further, even, we'll see. But yes, I think the next step is then some sort of sharding. Sharding by user would be an obvious approach for many apps. I think we should build a framework to help manage this, so apps would only need to provide some callbacks e.g. to compute shard key for a particular query. Alternatively, apps that want full control will be able to use Durable Objects directly. We'll soon (in a few months, probably?) enable the new storage engine for all Durable Objects which means every object will have a private SQLite database. (I'm the tech lead for Workers in general, and currently focused on this project in particular.)

no_wizard 3y ago
do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems. Much like R2 has an S3 compat API as well as a way to retrieve files outside of workers (IIRC) will this be true of all forms of storage on the platform? Sometimes it'd be nice to prime the cache with KV for example, without having to invoke a worker
- elithrar 3y ago
  > do you think you will have any API driven access to the data held in these services? IE, I don't per se want to have to use a worker to arbitrage between the Cloudflare platform and HTTP access to the data within Cloudflare systems. For D1, yes — on the roadmap is a native HTTP API. KV has a HTTP API as well, so you can write directly.
tebbers 3y ago
Awesome, thanks for the info! We are building more and more features on Cloudflare and absolutely love the whole experience, faultless.

fauigerzigerk 3y ago

>Should I create one database per user, for example? You can't. D1 won't let you create databases at runtime. At least that was the case last time I tried it, and it's what killed it for me.

ibejoeb 3y ago
Not really sure how you distinguish between runtime and something else. It seems to me that you could provision a "shard" as simply as executing in a worker: wrangler d1 create customer28 --experimental-backend wrangler d1 execute customer28 --file=schema.sql
- fauigerzigerk 3y ago
  What I mean is that workers cannot create databases. You have to create each database using wrangler (or the UI) and then edit the configuration file to add a binding. The binding becomes a variable in the JavaScript environment, which means you have to change the source code every time a new database is created. Therefore, creating a database per user as part of the user signup flow is not possible.
  - ibejoeb 3y ago
    Ah, I see. I thought it would be possible to dynamically set a secret which would effectively be a database uri. But upon further investigation, it does indeed need a redeploy. I suppose you could make something to write the bindings to the toml and trigger a redeploy, but that's definitely not pretty.