19 ms·

D1: Our SQL database

592 points by elithrar 4y ago

rmbyrro 4y ago
I'm buying Cloudflare stocks right now. In 2-3 years from now, these services will be so mature and strong they will be crushing the cloud market. They're turning dreams into reality, one after another.
- endisneigh 4y ago
  Cloud business is driven by enterprise generally. Would enterprise be using SQLite?
  - Quarrelsome 4y ago
    they should be using SQLite more often than they are.
    - endisneigh 4y ago
      Why? What use cases are better with SQLite vs Postgres, MySQL, etc?
      - bmon 4y ago
        If you consider cost... I would imagine a fair few. From the article: > We will ensure that D1 costs less and performs better than comparable centralized solutions.
      - sophacles 4y ago
        Some "pros" that many find appealing: * Copying the database around its a file copy in sqlite. Each database is it's own single file. (there's also WAL stuff that you get control of) * No extra service to deploy, manage, and/or optimize. I don't fully agree with the following, but I had a colleague who used to say "If you don't have multiple app servers writing to the db, postgres is a waste of effort". * embedded means way lower data latency - if the dataset is in the fs cache even lower, no waiting on network transactions. I've frequently chosen it over PG in cases where I needed basic relational data operations. In one case we ingested a large dataset (a few gbs of measurements) once an hour. Then we did some initial analytics on those measurements and threw the results in the same db file. After that step was done, the data was read only for several other systems and we just copied the db file to each of the systems that needed the data on-demand. A couple of the systems did additional analytics and effectively imported the db file to a different db (one was PG another was a graph db - neo4j). A couple of the systems just used the db file directly. It worked our really well.
      - Quarrelsome 4y ago
        less things to go wrong. This gives you benefits all the way up the dev stack. Changing an integration test from needing its own db server installed to just a couple of files on disk is a big difference in complexity. You can probably run that test with just a local disk almost infinitely entirely deterministically, conversely as soon as you go onto the network all bets are off. Granted, if you already have the tooling its less of a big deal but if you don't then you need the tooling, e.g. your build and test machines need to have access to some sql installation somewhere and that the process of maintaining that can be a flaky one.
samwillis 4y ago
This is really interesting, it's (basing it on SQLite) exactly what I was expecting CloudFlare to do for their first DB. Its perfect for content type sites that want search and querying. Anyone from CF here, is it using Litestream (https://litestream.io https://litestream.io) for its replication or have you built your own replication system? I assume this first version is somewhat limited on write performance having a single "main" instance and SQLite laking concurrent writes? It seems to me that using SQLite sessions[0] would be a good way to build an eventually consistent replication system for SQLite, would be perfect for an edge first sql database, maybe D2? 0: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html
- jgrahamc 4y ago
  1. No, it's not built on Litestream. Operating a massive network and shuttling data around is kind of our thing. 2. We are going all in on databases and D2 sounds like a cool name for something...
  - xafke 4y ago
    R2, D2. I see what you did there!
rvz 4y ago
Now is this a Cloudflare ($NET) buy signal? I think you know the answer. Maybe they will announce a Hashicorp competitor in their next reveal. Who knows.
frogger8 4y ago
Not a expert on DOM or JavaScript so be kind ;) One thing I hope to see in the future is a better product filtering experience. When I worked on a jquery product filter I realized the DOM bloat was the main problem. I wonder if D1 can help devs build instant product filtering pages that don’t require the reload like microcenter or Newegg does. IE https://www.newegg.com/p/pl?d=hdmi+cable&N=-1&SortType=8 https://www.newegg.com/p/pl?d=hdmi+cable&N=-1&SortType=8
- mbreese 4y ago
  At any sufficient scale, it is difficult to do filtering on the client. Yes, it can be done, but with 10,000+ potential records, you don’t want to ship that to the client for each query. (Note: I’m thinking Newegg scale for “hdmi cable” here. There are certainly situations where you can ship the entire database to the client for filtering.) It’s not DOM bloat… it’s too many records. If you’re building a DOM node for each record, that’s bloat, but you still have the problem even if the results are stored in a JSON object and dynamically queried on the client side. So, for each new filter or new query you need to hit the server anyway. If that’s an asynchronous query that returns a json blob or a full refresh, IMHO, it doesn’t really matter that much. Either way, you’re rebuilding a large portion of the DOM with the new results. The only thing that skews things in favor of an async call is if the rest of the page is so heavyweight that reloading the page takes a significant amount of time. This is probably what you’re taking about. Having a SQLite db close to your worker node really isn’t going to affect this problem all that much.
- Cthulhu_ 4y ago
  It's probably better - especially for more advanced search engines - to have an elasticsearch instance or whichever is the more recent example handle product search and filtering like that.
kurinikku 4y ago
wow SQLite getting a lot of love these days https://tailscale.com/blog/database-for-2022 https://tailscale.com/blog/database-for-2022 https://fly.io/blog/all-in-on-sqlite-litestream https://fly.io/blog/all-in-on-sqlite-litestream https://blog.cloudflare.com/introducing-d1 https://blog.cloudflare.com/introducing-d1
- alberth 4y ago
  SQLite was originally great for desktop applications. Problem is, there's still a huge market for these apps but everything has moved to the web (no one is making desktop apps anymore). So having a full-blown RDMS is overkill for these kind of app, and now SQLite is starting to fill these web app needs. @sqlite - if you are reading this, any word on merging WAL2 and BEGIN CONCURRENT into main? There clearly is a new class of needs to do so in this world that has completely moved over to web app development (which introduces concurrency problems never experienced on desktop). Any thoughts of focusing more on these web related needs for SQLite (or maybe even fork your own code base to have a more enhanced SQLite version targeted at web needs)?
- jgrahamc 4y ago
  SQLite has been cool forever. It was the underlying data store for my machine learning email filter POPFile 20 years ago! https://en.wikipedia.org/wiki/POPFile https://en.wikipedia.org/wiki/POPFile https://getpopfile.org/browser/trunk/engine/POPFile/Database.pm https://getpopfile.org/browser/trunk/engine/POPFile/Database...
  - runlevel1 4y ago
    It's high-quality software too. It's well-commented and exceptionally well tested.[^1][^2] > As of version 3.33.0 (2020-08-14), the SQLite library consists of approximately 143.4 KSLOC of C code. ... By comparison, the project has 640 times as much test code and test scripts - 91911.0 KSLOC. I don't usually place much stock in those sort of counts, but 640x is notable. It makes sense considering the wide variety of use-cases, from embedded devices to edge computing and everything in between. [1]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html [2]: https://sqlite.org/src/dir?ci=trunk https://sqlite.org/src/dir?ci=trunk
mwcampbell 4y ago
Any current or planned support for existing ORMs, such as Prisma or TypeOrm? Also, I wonder how hard it will be to migrate existing PostgreSQL databases and SQL statements. Of course, I understand if Cloudflare is focused on greenfield applications.
- jgrahamc 4y ago
  We are definitely interested in ORMs. Want to make it easy to use. I hope someone creates the next Rails using Workers. And having other models on top of our SQL offerings will be important. Get in contact and let us know what you'd like.
  - eatonphil 4y ago
    Will not any existing ORM that supports SQLite support D1? I looked in the post for details on how it extends SQLite (is the query language different or extended, semantics very different, etc.) but didn't notice anything.
    - jgrahamc 4y ago
      They should.
    - mwcampbell 4y ago
      I think the main issue will be with ORMs that are tightly coupled to a specific SQLite driver, such as Prisma.
  - joshstrange 4y ago
    > I hope someone creates the next Rails using Workers I too am eagerly waiting for a good serverless nodejs framework that is "batteries included". I've deployed on Lambda using the "Serverless Framework" but once your app grows to a certain size everything starts to fall apart and you lose some of the magic. Unfortunately, most of the things that advertise themselves as serverless/lambda/worker nodejs frameworks are monoliths and/or an existing monolith framework that "supports" lambda (with a billion asterisks after that). There is absolutely nothing wrong with monolith frameworks, I love them, but just not for lambda, I want to deploy a single endpoint as a single function (or as a cron, or queue listener, etc), not all of my code for every function (you hit size limits quick with this method). I want express/nestjs/etc-type routes that I define with code or annotations that result in /only/ that function (endpoint) being bundled up and deployed. I ended up rolling my own "framework" on top of Serverless Framework (uses serverless.ts config file that scans my directories for a special file that defines the routes defined in that directory) but Serverless Framework is pretty shaky ground. Their documentation is a mess, Serverless Components appears dead, and they seem to be busy with their own "cloud" so I don't know how much longer I can keep building on top of them. When it works it's like magic but there are a ton of walls you run headfirst into: Cloud formation entity limits, package size limits, typescript/bundling support, clear disregard for medium/large projects ("Just use multiple services", this leads to a terrible dev experience), and long deploy times. I wish CF Workers had been out when I first started building my current project, I might have gone in that direction instead, I still might.
alberth 4y ago
First, super excited by having Cloudflare offer a RDMS (can SQLite be called that?) This enables entirely new classes of applications where everything can now be hosted by Cloudflare. Questions: a. To help with concurrent writes, will Cloudflare be using WAL2 and BEGIN CONCURRENT branches of SQLite? b. How is Cloudflare replicating the data cross region? Will it be Litestream.io behind the scenes? c. Will our Worker code need to be written differently to ensure only a single-writer is writing to SQLite database? d. How does data persistency and database file size get factored in? I have to imagine their is a limit to how much storage can be used, whether or not that storage is local to the Worker machine, and if its persistent.
jgrahamc 4y ago
BTW R2 is open beta now: https://blog.cloudflare.com/r2-open-beta/ https://blog.cloudflare.com/r2-open-beta/
- mariushn 4y ago
  R2 is 3x more expensive than B2 (storage) https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html Am I missing something? Is there no bandwidth cost at all?
  - messe 4y ago
    Yep, you're not charged for egress.
    - einichi 4y ago
      B2 to Cloudflare also does not incur egress fees: https://www.backblaze.com/blog/backblaze-and-cloudflare-partner-to-provide-free-data-transfer/ https://www.backblaze.com/blog/backblaze-and-cloudflare-part... Backblaze B2 customers will be able to download data stored in B2 to Cloudflare for zero transfer fees. This happens automatically once Cloudflare is configured to distribute your B2 files.
      - kjksf 4y ago
        I did Backblaze via Cloudflare setup. I really don't care about the cost of storage. In my case it's the bandwidth costs that were killing me. If it was available at the time, I would use R2 if only for simplicity. If I was using Cloudflare Workers it would be another reason to use R2: I assume that it's easier to use and faster to use than any other storage system, since it's on the same network and written by the same people. Also, exposing Backblaze via Cloudflare has it's issues. I ran into Cloudflare caching 404 responses from Backblaze and Backblaze being slow to make write visible. So I would write into Backblaze and tried to access that key via Cloudflare proxy. While the write was acknowledged to my client it wasn't yet visible via http endpoint so Cloudflare would cache 404 response. I would have to clear the cache to fix and then I've added 5 min delay "just in case" to work around this.
greenie_beans 4y ago
dang i was hoping for postgres so i can use postgis edit: maybe one day! this looks cool regardless
- edvinbesic 4y ago
  I'm right there with you. I wonder if this is an SQLite compatible API on top of their own solution, or if it's using actual SQLite under the hood with custom replication. If the latter, and anyone from CloudFlare is here, is there any chance to have SpatiaLite enabled? https://www.gaia-gis.it/fossil/libspatialite/index https://www.gaia-gis.it/fossil/libspatialite/index
  - durkie 4y ago
    Seconding a vote for Spatialite support! I came here just to make that same request.
- yawaramin 4y ago
  No need to call dang!
  - greenie_beans 4y ago
    lol i'm from a place where "dang" is a natural part of our vocab
ranguna 4y ago
This looks amazing! I see cloudflare people are on this post, any chance to compar D1 vs postgres in terms of DB features? Insert ... Returning Stored procedures and triggers Etc etc Would be really helpful to get a comparison like cockroachDB did here https://www.cockroachlabs.com/docs/stable/postgresql-compatibility.html https://www.cockroachlabs.com/docs/stable/postgresql-compati... Or even better, a general sql compatibility matrix like this https://www.cockroachlabs.com/docs/stable/sql-feature-support.html https://www.cockroachlabs.com/docs/stable/sql-feature-suppor... Kudos to the cloudflare team!
- the_duke 4y ago
  Well, it's sqlite... so presumably you will get most of the capabilities sqlite has. RETURNING is covered. Stored procedures are indirectly there by running your own code "next to the database", as mentioned in the post. Which is arguably much nicer than having to use some database specific language, given that you can run WASM on workers.
  - tyingq 4y ago
    There is a layer on top of Sqlite here, so I imagine it's something less than all the capabilities sqlite has, at least initially. Plus the upsides and downsides from their approach to have a master and read replicas.
    - ranguna 4y ago
      Yes was thinking the same. Nice to see some people here actually understood the question, thank you.
  - ranguna 4y ago
    > Stored procedures are indirectly there by running your own code "next to the database", "indirectly" is a keyword here, because running code when data is modified potentially won't replace triggers since they'll probably execute outside the running transaction.
- Cthulhu_ 4y ago
onphonenow 4y ago
Our first database … I like it. I wonder what’s next
endisneigh 4y ago
Have any of the problems that led people to use Postgres instead of SQLite actually been solved? Are we doomed to repeat the same mistakes? Also, any plans to support PATCH x-update-range so SQLite can be used entirely in the browser via SQLite.js? Can someone enlighten me with the types of use cases this would be better for vs say Postgres?
- jve 4y ago
  What problems? Both are for different use cases albeit overlapping.
  - endisneigh 4y ago
    Concurrent writes, for one.
- nindalf 4y ago
  Which problems were you thinking of? Cloudflare and fly.io both promise hassle free read replicas and backup. They will both offer only a single node capable of writes, because that’s how SQLite rolls. This is a pretty good fit for a read heavy load that requires SQL and very low latency.
  - endisneigh 4y ago
    I guess I’m not understanding what the benefit is vs hosted Postgres. Also low latency and setup can be equally trivial - see supabase for example.
    - simonw 4y ago
      Biggest benefit over hosted PostgreSQL is that you get SELECT queries that are measured in microseconds, because SQLite avoids needing network overhead per query. https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html
      - 4y ago
ryanto 4y ago
This is so cool! From the blog post it says read-only replicas are created close to users and kept up to date with the latest data. - How should I think about this in terms of CAP? If there's a write and I query a replica what happens? - How are writes handled? Do they go to a single location or are they handled by various locations? I'm excited to try this. It's so cool to see databases being distributed "on CDNs" for lack of a better term.
- leonidasv 4y ago
  I think they're replicated asynchronously, so reading directly from the replica may return old data. That's why they've added the ability to deploy special workers that "live" closer to the primary: > Embedded compute > But we're going further. With D1, it will be possible to define a chunk of your Worker code that runs directly next to the database, giving you total control and maximum performance — each request first hits your Worker near your users, but depending on the operation, can hand off to another Worker deployed alongside a replica or your primary D1 instance to complete its work.
philholden 4y ago
Glad to hear was considering moving to Deno Deploy + Supabase because KV was not good for relationships.
hn_ei_ser_23 4y ago
First, I'm very excited. Sure, SQLite has some limitations compared to Postgres, esp. regarding the type system and concurrency. But we get ACID compliance and SQL. But it is really hard getting some useful information from this article. I can't even tell if it is not there or just buried in all this marketing hot air. So, what is it really? Is there one Write-Master that is asynchronously replicated to all other locations? Will writes be forwarded to this master and then replicated back? I'm very curious about how it performs in real life. Especially considering the locking behavior (SQLite has always the isolation level 'serializable' iirc). The more you put in a transaction or the longer you have to wait for another process to finish their writes, the more likely you have to deal with stale data. But overall I'm very excited. Also by the fly.io announcement, of course. Lots of innovation and competition. Good times for customers.
- tyingq 4y ago
  >So, what is it really? Is there one Write-Master that is asynchronously replicated to all other locations? Will writes be forwarded to this master and then replicated back? Not a lot of detail, but that is mentioned: "But we're going further. With D1, it will be possible to define a chunk of your Worker code that runs directly next to the database, giving you total control and maximum performance—each request first hits your Worker near your users, but depending on the operation, can hand off to another Worker deployed alongside a replica or your primary D1 instance to complete its work."
_kyran 4y ago
So can we assume that D2 will be postgres/mysql ?
- eatonphil 4y ago
  It sounds like you're making a simile but I don't understand it. The article did literally state D1 is based on sqlite.
  - _kyran 4y ago
    The opening paragraph reads "Today, we're excited to announce D1, our first SQL database." read: first and well R2 and D2 would make for a great naming scheme.
    - eatonphil 4y ago
      Ah sorry I missed you saying d2 not D1.
the_duke 4y ago
All this recent hype around sqlite... sqlite is a great embedded database and thanks to use by browsers and on mobile the most used database in the world by orders of magnitude. But it also comes with lots of limitations. * there is no type safety, unless you run with the new strict mode, which comes with some significant drawbacks (eg limited to the handful of primitive types) * very narrow set of column types and overall functionality in general * the big one for me: limited migration support, requiring quite a lot of ceremony for common tasks (eg rewriting a whole table and swapping it out) These approaches (like fly.io s) with read replication also (apparently?) seem to throw away read after write consistency. Which might be fine for certain use cases and even desirable for resilience, but can impact application design quite a lot. With sqlite you have do to a lot more in your own code because the database gives you fewer tools. Which is usually fine because most usage is "single writer, single or a few local readers". Moving that to a distributed setting with multiple deployed versions of code is not without difficulty. This seems to be mitigated/solved here though by the ability to run worker code "next to the database". I'm somewhat surprised they went this route. It probably makes sense given the constraints of Cloudflares architecture and the complexity of running a more advanced globally distributed database. On the upside: hopefully this usage in domains that are somewhat unusual can lead to funding for more upstream sqlite features.
- vlovich123 4y ago
  D1 does not throw away consistency. It’s built on top of Durable Objects which is globally strongly consistent.
  - mwcampbell 4y ago
    Interesting that D1 is built on top of Durable Objects. Does this mean that it would be practical for a single worker to access multiple D1 databases, so it could use, for example, a separate database for each tenant in a B2B SaaS application? Edit: And could each database be in a different primary region?
    - a-robinson 4y ago
losvedir 4y ago
Wow, this looks potentially very interesting. Since this is sort of fresh in my mind from the recent Fly post about it: * How exactly is the read replication implemented? Is it using litestream behind the scenes to stream the WAL somewhere? How do the readers keep up? Last I saw you just had to poll it, but that could be computationally expensive depending on the size of the data (since I thought you had to download the whole DB), and could potentially introduce a bit of latency in propagation. Any idea what the metrics are for latency in propagation? * How are writes handled? Does it do the Fly thing about sending all requests to one worker? I don't quite know what a "worker" is but I'm assuming it's kind of like a Lambda? If you have it replicated around the world, is that one worker all running the same code, and Cloudflare somehow manages the SQL replicating and write forwarding? Or would those all be separate workers?
polskibus 4y ago
Is this going to be open sourced? Seems to be building on the shoulder of a particular giant that could use a bit wider ecosystem.
robertlagrant 4y ago
This looks awesome. I was thinking about creating a custom version of this to live behind a CF Worker. Much better to have an official version!
tyingq 4y ago
"With D1, it will be possible to define a chunk of your Worker code that runs directly next to the database...each request first hits your Worker near your users, but depending on the operation, can hand off to another Worker deployed alongside a replica or your primary D1 instance to complete its work." That's interesting to me. It opens the door for Cloudflare to offer something more like a "normal" serverless offering. One that can run containers, or least natively run Python/Golang/Java/etc, like AWS Lambda does. And with this ecosystem described above that can conditionally route between the lighter edge Workers and the heavier central serverless functions. To me, that's the tipping point where they start to threaten larger portions of AWS.
fzaninotto 4y ago
Love the Northwind Traders reference! However, for a demo, I suggest a slightly larger and more complex data set, [data-generator-retail](https://www.npmjs.com/package/data-generator-retail https://www.npmjs.com/package/data-generator-retail). The demo is also a bit buggy: orders are duplicated as many times as there are products, but clicking on the various lines of the same order leads to the same record, where the user can only see the first product... I also think the demo would have more impact if it wasn't read-only (although I understand that this could lead to broken pages if visitors mess up with the data). Anyway, kudos to the CloudFlare team!
- naiv 4y ago
  I was thinking the same. The dataset is way too small.
- celso 4y ago
  Fixed the orders table. Good catch.
infogulch 4y ago
Very cool! Glad to see all the love for SQLite recently. One thing I've noticed that many commenters miss about read-replicated SQLite is assuming that the only valid model is having one, giant, centralized database with all the data. Lets be honest with ourselves, the vast majority of applications hold personal or B2B data and don't need centralized transactions, and at scale will use multi-tenant primary keys or manual sharding anyways. For private data, a single SQLite database per user / business will easily satisfy the write load of all but the most gigantic corporations. With this model you have unbounded compute scaling for new users because they very likely don't need online transactions across multiple databases at once. Some questions: Will D1 be able to deliver this design of having many thousands of separate databases for a single application? Will this be problematic from a cost perspective? > since we're building on the redundant storage of Durable Objects, your database can physically move locations as needed Will D1 be able to easily migrate the "primary" at will? CockroachDB described this as "follow the sun" primary.
- unraveller 4y ago
  I guess the first answer is: similar to Durable Object limits (unlimited databases / 50 GB total) since they alluded to those abilities more so than a simple file stored on R2 (only for backups).
whitepaint 4y ago
Will they seriously challenge Azure, AWS and GCP eventually? Cloudflare is very innovative and what they are doing is really exciting.
- 015a 4y ago
  The unique thing about Cloudflare's product offerings is how global-first they are; traditional cloud providers (AWS to DigitalOcean) have a very region-oriented domain model, with select christened services allowed or architected to be global (ex: AWS Cloudfront, IAM, Route53, that's about it there). That's their disaster/failure model; but all it really does is force cross-regional architecture onto the customer. Most customers don't bother. In comparison, everything at CF is global. And its not just "global" from an AWS perspective of "we've got 14 regions and your stuff runs in all of them"; its global from 300+ points-of-presence, within 50ms of like 98% of all humans. CDN for compute, databases, etc. CF has a way to go in DevEx on many of their products. For example; Workers, being based on V8 Isolates, is a pain to use even compared to e.g. Lambda. It's a battle of figuring out what's possible and what isn't within the runtime. But I'm sure it'll be improved!
estensen 4y ago
Too bad you probably can't use this to store data about EU citizens. Phone numbers like they show in the demo are considered PII, right?
- methyl 4y ago
  why?
lucasyvas 4y ago
To the person from Cloudflare I complained to in last year's thread about putting your money where your mouth is on serverless databases: You weren't lying, and this is super cool - the SQLite hype train also seems to be in full force.
- throwaway894345 4y ago
  It's interesting to see a relatively old technology get hyped.
- jgrahamc 4y ago
  :-)
pier25 4y ago
So where are the databases running? In the same regions as workers? Is the data replicated to all regions?
oxff 4y ago
Its a bold strategy, Cotton, sounding a bit like they want to compete with AWS.
dinkleberg 4y ago
This is convenient, I’ve been building an app which is using SQLite but am wanting to deploy it to Cloudflare pages. I expected I was going to have to switch to a hosted Postgres instance somewhere, but this could be perfect.
xwdv 4y ago
With this we can probably switch our infrastructure off AWS and entirely onto Cloudflare.
ralusek 4y ago
Unless I missed it by skimming, where are the deets? Is this strongly or eventually consistent? What are max table sizes, and do they become partitioned? Are there cross partition joins?
lucasyvas 4y ago
The API for this is currently the only thing I wish I could grok a bit better. It seems like it would be hard to make it work with existing libraries that can access SQLite, which is kind of a shame. I'm thinking of sqlx in Rust (or any other language binding / ORM for that matter), which has compile time schema safety. This is a nice capability, and because this interface seems non-standard (possibly for good reason), I guess we are being asked to give some of those things up. I am getting a bit ahead of myself on the Rust part (presumably that will eventually be supported as part of workers-rs), but I think the feelings still stand if you consider the JS ecosystem. Edit: I may actually be wrong, but presumably the entire surface isn't covered because there's no file opening, etc.
- mritchie712 4y ago
  There might be a `env.DB.url` (e.g. the jdbc URL) which you could pass into an existing library.
  - lucasyvas 4y ago
    Interesting thought! Would love to see more details.
  - yencabulator 4y ago
    I'm kinda willing to make a bet that this rides on top of what looks like HTTP to the Javascript engine. That's how their worker-to-worker and worker-to-durable-object protocols are. (It's not really HTTP as in it might never cross a TCP socket, just get shuffled from one V8 isolate to another, but it looks like a `fetch` call to the Javascript.) It's also worth remembering that SQLite itself has no wire protocol, it's a library. And there is no such thing as a "SQL wire protocol". It sure isn't gonna be Postgres wire protocol either. From the article: > D1’s API includes batching: anywhere you can send a single SQL statement you can also provide an array of them, meaning you only need a single HTTP round-trip to perform multiple operations. This is perfect for transactions that need to execute and commit atomically:
slashdev 4y ago
For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters. I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates. I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the master? Ship the SQL statements that did the write and reapply them? Lots of performance and other tradeoffs here. What's the latency like? This likely doesn't run in every edge location. Does the database ship out on the first request. Get cached with an expiry? Does the request itself move to the database instead of running at the edge - like maybe this runs on a select subset of locations? So many questions, but no details yet.
- dragonwriter 4y ago
  > I'm guessing this is a single master database with multiple read replicas. That means it's not consistent Single master with read replicas is fully consistent if commits don't return until propagated to and acknowledged by replicas (the expense here being commit latency.)
  - otoolep 4y ago
    You've basically described rqlite [1], which uses Raft to coordinate the changes to the Leader, and then across some number of Followers. The write won't be acked until a quorum has persisted the change, and committed to the underlying SQLite database. Disclaimer: I am the creator of rqlite. [1] https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite
    - otoolep 4y ago
      rqlite also supports read-only nodes, so in theory you can have more nodes at the edge, just like D1 -- but these nodes won't participate in the distributed consensus process. Those nodes will keep up-to-date with changes, even catching up in the event of a temporary disconnection.
  - eloff 4y ago
jcuenod 4y ago
So I assume we'll see a nice big donation to the sqlite coffers, then?
didip 4y ago
All these hype around SQLite recently and I am still confused. * How do you replicate it consistently? * Who has the master privilege (or masters if sharded)? What's the failover story? I am guessing a blob store is involved, but I have gaps in my understanding here.
- discodave 4y ago
  SQLite has a write ahead log (journal) mode. If you write that log to some store that is already replicated (S3, CloudFlare Durable Objects, Kafka?) then the concept of a 'master' is less important.
jpcapdevila 4y ago
If SQLite gets you excited, I'm building a firebase alternative based on sqlite. I'm betting hard on sqlite so this get's me super excited!! https://javascriptdb.com https://javascriptdb.com CF people around, I would love to chat, if anyone is interested please reach out at: jp@javascriptdb.com I'll be applying to this beta for sure!
- js4ever 4y ago
  Super interesting! I really like the idea. I'll join the beta, email sent :)
  - jpcapdevila 4y ago
    Any feedback on what do you find interesting would be awesome :) thanks!!
SheinhardtWigCo 4y ago
Big fan of Cloudflare but I wish they would stick to descriptive product names. Good: Workers, KV, Durable Objects, Cron Triggers Bad: Spectrum, Zaraz, R2, D1
- alberth 4y ago
  Naming is hard. > Zaraz That's the name of the company they acquired. Though, I do agree that more descriptive naming is nice. E.g. Zaraz = SafeXXS D1 = LDS (light database system) R2 = ObjectStore Spectrum = Reverse Proxy
ctur 4y ago
- systemvoltage 4y ago
  No one really means anything ill and this political correctness madness needs to stop.
  - nwsm 4y ago
    - manigandham 4y ago
      Because it's not painful to others and intent always matters. These words are everywhere in the language; you're not really changing anything with these antics other than derailing the subject to appease those who assume offense on behalf of an imagined group of people that can't distinguish context.
      - sorenbs 4y ago
        Most of us moved on to better terminology 4+ years ago. The only ones derailing conversations are grumps like yourself who refuse to get with the program. Why is this so important to you?
        manigandham 4y ago
        Who are you and what's this "program" you deem to impose on others? No thanks, I'll stick with the actual majority that have mastered using relevant language and rational context in discussions without being slaves to performative social constructs. Instead of assuming what's actually important to me, perhaps some introspection of why you immediately think of slavery in a computing context would be more helpful.
    - systemvoltage 4y ago
      People want to change Chess to red/blue. I'll continue to play with white and black pieces. This is a slippery slope of destroying the society by being hyper sensitive about things that no one really means. Absolutely hate this and fills me with disgust that people obssess over this kind of petty things. Life is beautiful. Enjoy it. Be kind to others that have zero intention of offending you.
aeyes 4y ago
What write throughput and latency can we expect from this database? Are there any limitations, for example on the number of tables or size of the database?
ngrilly 4y ago
Not clear from reading the post if the SQLite C library is embedded and linked in the Worker runtime (which would mean no network roundtrip) or if each query or batch of queries is converted to a network request to a server embedding the SQLite C library. That's important to understand because that's one of the key advantages of SQLite compared to the usual client-server architecture of databases like PostgreSQL or MySQL: https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html
benjiweber 4y ago
I was expecting this to be using https://en.wikipedia.org/wiki/D_(data_language_specification) https://en.wikipedia.org/wiki/D_(data_language_specification... given the name.
deanc 4y ago
Any word on pricing =)?
irq-1 4y ago
Best Effort Writes[1] are an opportunity here. Non-transactional, write to the local replica (ensure foreign keys, constrains, valid data, etc...) and then try to write to the main write-enabled DB. Caching should work without changes since the local replica is updated. This could be cheaper (send binary diffs) and more resilient to brief network issues. The key is to let the user decide what really needs ACID and what doesn't. If someone wants to make the next Facebook or Reddit they'll need huge write throughput and if some votes or updates are lost, that may be a good trade-off. [1] You could add a BEW file (like WAL file) to sqlite for Best Effort Writes.
jzer0cool 4y ago
How does this work when developing locally. Is it SQLite for local development?

592 points by elithrar 4y ago

I'm buying Cloudflare stocks right now. In 2-3 years from now, these services will be so mature and strong they will be crushing the cloud market. They're turning dreams into reality, one after another.

endisneigh 4y ago
Cloud business is driven by enterprise generally. Would enterprise be using SQLite?
- Quarrelsome 4y ago
  they should be using SQLite more often than they are.
  - endisneigh 4y ago
    Why? What use cases are better with SQLite vs Postgres, MySQL, etc?
    - bmon 4y ago
      If you consider cost... I would imagine a fair few. From the article: > We will ensure that D1 costs less and performs better than comparable centralized solutions.
    - sophacles 4y ago
      Some "pros" that many find appealing: * Copying the database around its a file copy in sqlite. Each database is it's own single file. (there's also WAL stuff that you get control of) * No extra service to deploy, manage, and/or optimize. I don't fully agree with the following, but I had a colleague who used to say "If you don't have multiple app servers writing to the db, postgres is a waste of effort". * embedded means way lower data latency - if the dataset is in the fs cache even lower, no waiting on network transactions. I've frequently chosen it over PG in cases where I needed basic relational data operations. In one case we ingested a large dataset (a few gbs of measurements) once an hour. Then we did some initial analytics on those measurements and threw the results in the same db file. After that step was done, the data was read only for several other systems and we just copied the db file to each of the systems that needed the data on-demand. A couple of the systems did additional analytics and effectively imported the db file to a different db (one was PG another was a graph db - neo4j). A couple of the systems just used the db file directly. It worked our really well.
    - Quarrelsome 4y ago
      less things to go wrong. This gives you benefits all the way up the dev stack. Changing an integration test from needing its own db server installed to just a couple of files on disk is a big difference in complexity. You can probably run that test with just a local disk almost infinitely entirely deterministically, conversely as soon as you go onto the network all bets are off. Granted, if you already have the tooling its less of a big deal but if you don't then you need the tooling, e.g. your build and test machines need to have access to some sql installation somewhere and that the process of maintaining that can be a flaky one.

samwillis 4y ago

This is really interesting, it's (basing it on SQLite) exactly what I was expecting CloudFlare to do for their first DB. Its perfect for content type sites that want search and querying. Anyone from CF here, is it using Litestream (https://litestream.io https://litestream.io) for its replication or have you built your own replication system? I assume this first version is somewhat limited on write performance having a single "main" instance and SQLite laking concurrent writes? It seems to me that using SQLite sessions[0] would be a good way to build an eventually consistent replication system for SQLite, would be perfect for an edge first sql database, maybe D2? 0: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html

jgrahamc 4y ago
1. No, it's not built on Litestream. Operating a massive network and shuttling data around is kind of our thing. 2. We are going all in on databases and D2 sounds like a cool name for something...
- xafke 4y ago
  R2, D2. I see what you did there!

rvz 4y ago

Now is this a Cloudflare ($NET) buy signal? I think you know the answer. Maybe they will announce a Hashicorp competitor in their next reveal. Who knows.

frogger8 4y ago

Not a expert on DOM or JavaScript so be kind ;) One thing I hope to see in the future is a better product filtering experience. When I worked on a jquery product filter I realized the DOM bloat was the main problem. I wonder if D1 can help devs build instant product filtering pages that don’t require the reload like microcenter or Newegg does. IE https://www.newegg.com/p/pl?d=hdmi+cable&N=-1&SortType=8 https://www.newegg.com/p/pl?d=hdmi+cable&N=-1&SortType=8

mbreese 4y ago
At any sufficient scale, it is difficult to do filtering on the client. Yes, it can be done, but with 10,000+ potential records, you don’t want to ship that to the client for each query. (Note: I’m thinking Newegg scale for “hdmi cable” here. There are certainly situations where you can ship the entire database to the client for filtering.) It’s not DOM bloat… it’s too many records. If you’re building a DOM node for each record, that’s bloat, but you still have the problem even if the results are stored in a JSON object and dynamically queried on the client side. So, for each new filter or new query you need to hit the server anyway. If that’s an asynchronous query that returns a json blob or a full refresh, IMHO, it doesn’t really matter that much. Either way, you’re rebuilding a large portion of the DOM with the new results. The only thing that skews things in favor of an async call is if the rest of the page is so heavyweight that reloading the page takes a significant amount of time. This is probably what you’re taking about. Having a SQLite db close to your worker node really isn’t going to affect this problem all that much.
Cthulhu_ 4y ago
It's probably better - especially for more advanced search engines - to have an elasticsearch instance or whichever is the more recent example handle product search and filtering like that.

kurinikku 4y ago

wow SQLite getting a lot of love these days https://tailscale.com/blog/database-for-2022 https://tailscale.com/blog/database-for-2022 https://fly.io/blog/all-in-on-sqlite-litestream https://fly.io/blog/all-in-on-sqlite-litestream https://blog.cloudflare.com/introducing-d1 https://blog.cloudflare.com/introducing-d1

alberth 4y ago
SQLite was originally great for desktop applications. Problem is, there's still a huge market for these apps but everything has moved to the web (no one is making desktop apps anymore). So having a full-blown RDMS is overkill for these kind of app, and now SQLite is starting to fill these web app needs. @sqlite - if you are reading this, any word on merging WAL2 and BEGIN CONCURRENT into main? There clearly is a new class of needs to do so in this world that has completely moved over to web app development (which introduces concurrency problems never experienced on desktop). Any thoughts of focusing more on these web related needs for SQLite (or maybe even fork your own code base to have a more enhanced SQLite version targeted at web needs)?
jgrahamc 4y ago
SQLite has been cool forever. It was the underlying data store for my machine learning email filter POPFile 20 years ago! https://en.wikipedia.org/wiki/POPFile https://en.wikipedia.org/wiki/POPFile https://getpopfile.org/browser/trunk/engine/POPFile/Database.pm https://getpopfile.org/browser/trunk/engine/POPFile/Database...
- runlevel1 4y ago
  It's high-quality software too. It's well-commented and exceptionally well tested.[^1][^2] > As of version 3.33.0 (2020-08-14), the SQLite library consists of approximately 143.4 KSLOC of C code. ... By comparison, the project has 640 times as much test code and test scripts - 91911.0 KSLOC. I don't usually place much stock in those sort of counts, but 640x is notable. It makes sense considering the wide variety of use-cases, from embedded devices to edge computing and everything in between. [1]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html [2]: https://sqlite.org/src/dir?ci=trunk https://sqlite.org/src/dir?ci=trunk

mwcampbell 4y ago

Any current or planned support for existing ORMs, such as Prisma or TypeOrm? Also, I wonder how hard it will be to migrate existing PostgreSQL databases and SQL statements. Of course, I understand if Cloudflare is focused on greenfield applications.

jgrahamc 4y ago
We are definitely interested in ORMs. Want to make it easy to use. I hope someone creates the next Rails using Workers. And having other models on top of our SQL offerings will be important. Get in contact and let us know what you'd like.
- eatonphil 4y ago
  Will not any existing ORM that supports SQLite support D1? I looked in the post for details on how it extends SQLite (is the query language different or extended, semantics very different, etc.) but didn't notice anything.
  - jgrahamc 4y ago
    They should.
  - mwcampbell 4y ago
    I think the main issue will be with ORMs that are tightly coupled to a specific SQLite driver, such as Prisma.
- joshstrange 4y ago
  > I hope someone creates the next Rails using Workers I too am eagerly waiting for a good serverless nodejs framework that is "batteries included". I've deployed on Lambda using the "Serverless Framework" but once your app grows to a certain size everything starts to fall apart and you lose some of the magic. Unfortunately, most of the things that advertise themselves as serverless/lambda/worker nodejs frameworks are monoliths and/or an existing monolith framework that "supports" lambda (with a billion asterisks after that). There is absolutely nothing wrong with monolith frameworks, I love them, but just not for lambda, I want to deploy a single endpoint as a single function (or as a cron, or queue listener, etc), not all of my code for every function (you hit size limits quick with this method). I want express/nestjs/etc-type routes that I define with code or annotations that result in /only/ that function (endpoint) being bundled up and deployed. I ended up rolling my own "framework" on top of Serverless Framework (uses serverless.ts config file that scans my directories for a special file that defines the routes defined in that directory) but Serverless Framework is pretty shaky ground. Their documentation is a mess, Serverless Components appears dead, and they seem to be busy with their own "cloud" so I don't know how much longer I can keep building on top of them. When it works it's like magic but there are a ton of walls you run headfirst into: Cloud formation entity limits, package size limits, typescript/bundling support, clear disregard for medium/large projects ("Just use multiple services", this leads to a terrible dev experience), and long deploy times. I wish CF Workers had been out when I first started building my current project, I might have gone in that direction instead, I still might.

alberth 4y ago

First, super excited by having Cloudflare offer a RDMS (can SQLite be called that?) This enables entirely new classes of applications where everything can now be hosted by Cloudflare. Questions: a. To help with concurrent writes, will Cloudflare be using WAL2 and BEGIN CONCURRENT branches of SQLite? b. How is Cloudflare replicating the data cross region? Will it be Litestream.io behind the scenes? c. Will our Worker code need to be written differently to ensure only a single-writer is writing to SQLite database? d. How does data persistency and database file size get factored in? I have to imagine their is a limit to how much storage can be used, whether or not that storage is local to the Worker machine, and if its persistent.

jgrahamc 4y ago

BTW R2 is open beta now: https://blog.cloudflare.com/r2-open-beta/ https://blog.cloudflare.com/r2-open-beta/

mariushn 4y ago
R2 is 3x more expensive than B2 (storage) https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html Am I missing something? Is there no bandwidth cost at all?
- messe 4y ago
  Yep, you're not charged for egress.
  - einichi 4y ago
    B2 to Cloudflare also does not incur egress fees: https://www.backblaze.com/blog/backblaze-and-cloudflare-partner-to-provide-free-data-transfer/ https://www.backblaze.com/blog/backblaze-and-cloudflare-part... Backblaze B2 customers will be able to download data stored in B2 to Cloudflare for zero transfer fees. This happens automatically once Cloudflare is configured to distribute your B2 files.
    - kjksf 4y ago
      I did Backblaze via Cloudflare setup. I really don't care about the cost of storage. In my case it's the bandwidth costs that were killing me. If it was available at the time, I would use R2 if only for simplicity. If I was using Cloudflare Workers it would be another reason to use R2: I assume that it's easier to use and faster to use than any other storage system, since it's on the same network and written by the same people. Also, exposing Backblaze via Cloudflare has it's issues. I ran into Cloudflare caching 404 responses from Backblaze and Backblaze being slow to make write visible. So I would write into Backblaze and tried to access that key via Cloudflare proxy. While the write was acknowledged to my client it wasn't yet visible via http endpoint so Cloudflare would cache 404 response. I would have to clear the cache to fix and then I've added 5 min delay "just in case" to work around this.

greenie_beans 4y ago

dang i was hoping for postgres so i can use postgis edit: maybe one day! this looks cool regardless

edvinbesic 4y ago
I'm right there with you. I wonder if this is an SQLite compatible API on top of their own solution, or if it's using actual SQLite under the hood with custom replication. If the latter, and anyone from CloudFlare is here, is there any chance to have SpatiaLite enabled? https://www.gaia-gis.it/fossil/libspatialite/index https://www.gaia-gis.it/fossil/libspatialite/index
- durkie 4y ago
  Seconding a vote for Spatialite support! I came here just to make that same request.
yawaramin 4y ago
No need to call dang!
- greenie_beans 4y ago
  lol i'm from a place where "dang" is a natural part of our vocab

ranguna 4y ago

This looks amazing! I see cloudflare people are on this post, any chance to compar D1 vs postgres in terms of DB features? Insert ... Returning Stored procedures and triggers Etc etc Would be really helpful to get a comparison like cockroachDB did here https://www.cockroachlabs.com/docs/stable/postgresql-compatibility.html https://www.cockroachlabs.com/docs/stable/postgresql-compati... Or even better, a general sql compatibility matrix like this https://www.cockroachlabs.com/docs/stable/sql-feature-support.html https://www.cockroachlabs.com/docs/stable/sql-feature-suppor... Kudos to the cloudflare team!

the_duke 4y ago
Well, it's sqlite... so presumably you will get most of the capabilities sqlite has. RETURNING is covered. Stored procedures are indirectly there by running your own code "next to the database", as mentioned in the post. Which is arguably much nicer than having to use some database specific language, given that you can run WASM on workers.
- tyingq 4y ago
  There is a layer on top of Sqlite here, so I imagine it's something less than all the capabilities sqlite has, at least initially. Plus the upsides and downsides from their approach to have a master and read replicas.
  - ranguna 4y ago
    Yes was thinking the same. Nice to see some people here actually understood the question, thank you.
- ranguna 4y ago
  > Stored procedures are indirectly there by running your own code "next to the database", "indirectly" is a keyword here, because running code when data is modified potentially won't replace triggers since they'll probably execute outside the running transaction.
Cthulhu_ 4y ago

onphonenow 4y ago

Our first database … I like it. I wonder what’s next

endisneigh 4y ago

Have any of the problems that led people to use Postgres instead of SQLite actually been solved? Are we doomed to repeat the same mistakes? Also, any plans to support PATCH x-update-range so SQLite can be used entirely in the browser via SQLite.js? Can someone enlighten me with the types of use cases this would be better for vs say Postgres?

jve 4y ago
What problems? Both are for different use cases albeit overlapping.
- endisneigh 4y ago
  Concurrent writes, for one.
nindalf 4y ago
Which problems were you thinking of? Cloudflare and fly.io both promise hassle free read replicas and backup. They will both offer only a single node capable of writes, because that’s how SQLite rolls. This is a pretty good fit for a read heavy load that requires SQL and very low latency.
- endisneigh 4y ago
  I guess I’m not understanding what the benefit is vs hosted Postgres. Also low latency and setup can be equally trivial - see supabase for example.
  - simonw 4y ago
    Biggest benefit over hosted PostgreSQL is that you get SELECT queries that are measured in microseconds, because SQLite avoids needing network overhead per query. https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html
    - 4y ago

ryanto 4y ago

This is so cool! From the blog post it says read-only replicas are created close to users and kept up to date with the latest data. - How should I think about this in terms of CAP? If there's a write and I query a replica what happens? - How are writes handled? Do they go to a single location or are they handled by various locations? I'm excited to try this. It's so cool to see databases being distributed "on CDNs" for lack of a better term.

leonidasv 4y ago
I think they're replicated asynchronously, so reading directly from the replica may return old data. That's why they've added the ability to deploy special workers that "live" closer to the primary: > Embedded compute > But we're going further. With D1, it will be possible to define a chunk of your Worker code that runs directly next to the database, giving you total control and maximum performance — each request first hits your Worker near your users, but depending on the operation, can hand off to another Worker deployed alongside a replica or your primary D1 instance to complete its work.

philholden 4y ago

Glad to hear was considering moving to Deno Deploy + Supabase because KV was not good for relationships.

hn_ei_ser_23 4y ago

First, I'm very excited. Sure, SQLite has some limitations compared to Postgres, esp. regarding the type system and concurrency. But we get ACID compliance and SQL. But it is really hard getting some useful information from this article. I can't even tell if it is not there or just buried in all this marketing hot air. So, what is it really? Is there one Write-Master that is asynchronously replicated to all other locations? Will writes be forwarded to this master and then replicated back? I'm very curious about how it performs in real life. Especially considering the locking behavior (SQLite has always the isolation level 'serializable' iirc). The more you put in a transaction or the longer you have to wait for another process to finish their writes, the more likely you have to deal with stale data. But overall I'm very excited. Also by the fly.io announcement, of course. Lots of innovation and competition. Good times for customers.

tyingq 4y ago
>So, what is it really? Is there one Write-Master that is asynchronously replicated to all other locations? Will writes be forwarded to this master and then replicated back? Not a lot of detail, but that is mentioned: "But we're going further. With D1, it will be possible to define a chunk of your Worker code that runs directly next to the database, giving you total control and maximum performance—each request first hits your Worker near your users, but depending on the operation, can hand off to another Worker deployed alongside a replica or your primary D1 instance to complete its work."

_kyran 4y ago

So can we assume that D2 will be postgres/mysql ?

eatonphil 4y ago
It sounds like you're making a simile but I don't understand it. The article did literally state D1 is based on sqlite.
- _kyran 4y ago
  The opening paragraph reads "Today, we're excited to announce D1, our first SQL database." read: first and well R2 and D2 would make for a great naming scheme.
  - eatonphil 4y ago
    Ah sorry I missed you saying d2 not D1.

the_duke 4y ago

All this recent hype around sqlite... sqlite is a great embedded database and thanks to use by browsers and on mobile the most used database in the world by orders of magnitude. But it also comes with lots of limitations. * there is no type safety, unless you run with the new strict mode, which comes with some significant drawbacks (eg limited to the handful of primitive types) * very narrow set of column types and overall functionality in general * the big one for me: limited migration support, requiring quite a lot of ceremony for common tasks (eg rewriting a whole table and swapping it out) These approaches (like fly.io s) with read replication also (apparently?) seem to throw away read after write consistency. Which might be fine for certain use cases and even desirable for resilience, but can impact application design quite a lot. With sqlite you have do to a lot more in your own code because the database gives you fewer tools. Which is usually fine because most usage is "single writer, single or a few local readers". Moving that to a distributed setting with multiple deployed versions of code is not without difficulty. This seems to be mitigated/solved here though by the ability to run worker code "next to the database". I'm somewhat surprised they went this route. It probably makes sense given the constraints of Cloudflares architecture and the complexity of running a more advanced globally distributed database. On the upside: hopefully this usage in domains that are somewhat unusual can lead to funding for more upstream sqlite features.

vlovich123 4y ago
D1 does not throw away consistency. It’s built on top of Durable Objects which is globally strongly consistent.
- mwcampbell 4y ago
  Interesting that D1 is built on top of Durable Objects. Does this mean that it would be practical for a single worker to access multiple D1 databases, so it could use, for example, a separate database for each tenant in a B2B SaaS application? Edit: And could each database be in a different primary region?
  - a-robinson 4y ago

losvedir 4y ago

Wow, this looks potentially very interesting. Since this is sort of fresh in my mind from the recent Fly post about it: * How exactly is the read replication implemented? Is it using litestream behind the scenes to stream the WAL somewhere? How do the readers keep up? Last I saw you just had to poll it, but that could be computationally expensive depending on the size of the data (since I thought you had to download the whole DB), and could potentially introduce a bit of latency in propagation. Any idea what the metrics are for latency in propagation? * How are writes handled? Does it do the Fly thing about sending all requests to one worker? I don't quite know what a "worker" is but I'm assuming it's kind of like a Lambda? If you have it replicated around the world, is that one worker all running the same code, and Cloudflare somehow manages the SQL replicating and write forwarding? Or would those all be separate workers?

polskibus 4y ago

Is this going to be open sourced? Seems to be building on the shoulder of a particular giant that could use a bit wider ecosystem.

robertlagrant 4y ago

This looks awesome. I was thinking about creating a custom version of this to live behind a CF Worker. Much better to have an official version!

tyingq 4y ago

"With D1, it will be possible to define a chunk of your Worker code that runs directly next to the database...each request first hits your Worker near your users, but depending on the operation, can hand off to another Worker deployed alongside a replica or your primary D1 instance to complete its work." That's interesting to me. It opens the door for Cloudflare to offer something more like a "normal" serverless offering. One that can run containers, or least natively run Python/Golang/Java/etc, like AWS Lambda does. And with this ecosystem described above that can conditionally route between the lighter edge Workers and the heavier central serverless functions. To me, that's the tipping point where they start to threaten larger portions of AWS.

fzaninotto 4y ago

Love the Northwind Traders reference! However, for a demo, I suggest a slightly larger and more complex data set, [data-generator-retail](https://www.npmjs.com/package/data-generator-retail https://www.npmjs.com/package/data-generator-retail). The demo is also a bit buggy: orders are duplicated as many times as there are products, but clicking on the various lines of the same order leads to the same record, where the user can only see the first product... I also think the demo would have more impact if it wasn't read-only (although I understand that this could lead to broken pages if visitors mess up with the data). Anyway, kudos to the CloudFlare team!

naiv 4y ago
I was thinking the same. The dataset is way too small.
celso 4y ago
Fixed the orders table. Good catch.

infogulch 4y ago

Very cool! Glad to see all the love for SQLite recently. One thing I've noticed that many commenters miss about read-replicated SQLite is assuming that the only valid model is having one, giant, centralized database with all the data. Lets be honest with ourselves, the vast majority of applications hold personal or B2B data and don't need centralized transactions, and at scale will use multi-tenant primary keys or manual sharding anyways. For private data, a single SQLite database per user / business will easily satisfy the write load of all but the most gigantic corporations. With this model you have unbounded compute scaling for new users because they very likely don't need online transactions across multiple databases at once. Some questions: Will D1 be able to deliver this design of having many thousands of separate databases for a single application? Will this be problematic from a cost perspective? > since we're building on the redundant storage of Durable Objects, your database can physically move locations as needed Will D1 be able to easily migrate the "primary" at will? CockroachDB described this as "follow the sun" primary.

unraveller 4y ago
I guess the first answer is: similar to Durable Object limits (unlimited databases / 50 GB total) since they alluded to those abilities more so than a simple file stored on R2 (only for backups).

whitepaint 4y ago

Will they seriously challenge Azure, AWS and GCP eventually? Cloudflare is very innovative and what they are doing is really exciting.

015a 4y ago
The unique thing about Cloudflare's product offerings is how global-first they are; traditional cloud providers (AWS to DigitalOcean) have a very region-oriented domain model, with select christened services allowed or architected to be global (ex: AWS Cloudfront, IAM, Route53, that's about it there). That's their disaster/failure model; but all it really does is force cross-regional architecture onto the customer. Most customers don't bother. In comparison, everything at CF is global. And its not just "global" from an AWS perspective of "we've got 14 regions and your stuff runs in all of them"; its global from 300+ points-of-presence, within 50ms of like 98% of all humans. CDN for compute, databases, etc. CF has a way to go in DevEx on many of their products. For example; Workers, being based on V8 Isolates, is a pain to use even compared to e.g. Lambda. It's a battle of figuring out what's possible and what isn't within the runtime. But I'm sure it'll be improved!

estensen 4y ago

Too bad you probably can't use this to store data about EU citizens. Phone numbers like they show in the demo are considered PII, right?

methyl 4y ago
why?

lucasyvas 4y ago

To the person from Cloudflare I complained to in last year's thread about putting your money where your mouth is on serverless databases: You weren't lying, and this is super cool - the SQLite hype train also seems to be in full force.

throwaway894345 4y ago
It's interesting to see a relatively old technology get hyped.
jgrahamc 4y ago
:-)

pier25 4y ago

So where are the databases running? In the same regions as workers? Is the data replicated to all regions?

oxff 4y ago

Its a bold strategy, Cotton, sounding a bit like they want to compete with AWS.

dinkleberg 4y ago

This is convenient, I’ve been building an app which is using SQLite but am wanting to deploy it to Cloudflare pages. I expected I was going to have to switch to a hosted Postgres instance somewhere, but this could be perfect.

xwdv 4y ago

With this we can probably switch our infrastructure off AWS and entirely onto Cloudflare.

ralusek 4y ago

Unless I missed it by skimming, where are the deets? Is this strongly or eventually consistent? What are max table sizes, and do they become partitioned? Are there cross partition joins?

lucasyvas 4y ago

The API for this is currently the only thing I wish I could grok a bit better. It seems like it would be hard to make it work with existing libraries that can access SQLite, which is kind of a shame. I'm thinking of sqlx in Rust (or any other language binding / ORM for that matter), which has compile time schema safety. This is a nice capability, and because this interface seems non-standard (possibly for good reason), I guess we are being asked to give some of those things up. I am getting a bit ahead of myself on the Rust part (presumably that will eventually be supported as part of workers-rs), but I think the feelings still stand if you consider the JS ecosystem. Edit: I may actually be wrong, but presumably the entire surface isn't covered because there's no file opening, etc.

mritchie712 4y ago
There might be a `env.DB.url` (e.g. the jdbc URL) which you could pass into an existing library.
- lucasyvas 4y ago
  Interesting thought! Would love to see more details.
- yencabulator 4y ago
  I'm kinda willing to make a bet that this rides on top of what looks like HTTP to the Javascript engine. That's how their worker-to-worker and worker-to-durable-object protocols are. (It's not really HTTP as in it might never cross a TCP socket, just get shuffled from one V8 isolate to another, but it looks like a `fetch` call to the Javascript.) It's also worth remembering that SQLite itself has no wire protocol, it's a library. And there is no such thing as a "SQL wire protocol". It sure isn't gonna be Postgres wire protocol either. From the article: > D1’s API includes batching: anywhere you can send a single SQL statement you can also provide an array of them, meaning you only need a single HTTP round-trip to perform multiple operations. This is perfect for transactions that need to execute and commit atomically:

slashdev 4y ago

For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters. I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates. I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the master? Ship the SQL statements that did the write and reapply them? Lots of performance and other tradeoffs here. What's the latency like? This likely doesn't run in every edge location. Does the database ship out on the first request. Get cached with an expiry? Does the request itself move to the database instead of running at the edge - like maybe this runs on a select subset of locations? So many questions, but no details yet.

dragonwriter 4y ago
> I'm guessing this is a single master database with multiple read replicas. That means it's not consistent Single master with read replicas is fully consistent if commits don't return until propagated to and acknowledged by replicas (the expense here being commit latency.)
- otoolep 4y ago
  You've basically described rqlite [1], which uses Raft to coordinate the changes to the Leader, and then across some number of Followers. The write won't be acked until a quorum has persisted the change, and committed to the underlying SQLite database. Disclaimer: I am the creator of rqlite. [1] https://github.com/rqlite/rqlite https://github.com/rqlite/rqlite
  - otoolep 4y ago
    rqlite also supports read-only nodes, so in theory you can have more nodes at the edge, just like D1 -- but these nodes won't participate in the distributed consensus process. Those nodes will keep up-to-date with changes, even catching up in the event of a temporary disconnection.
- eloff 4y ago

jcuenod 4y ago

So I assume we'll see a nice big donation to the sqlite coffers, then?

didip 4y ago

All these hype around SQLite recently and I am still confused. * How do you replicate it consistently? * Who has the master privilege (or masters if sharded)? What's the failover story? I am guessing a blob store is involved, but I have gaps in my understanding here.

discodave 4y ago
SQLite has a write ahead log (journal) mode. If you write that log to some store that is already replicated (S3, CloudFlare Durable Objects, Kafka?) then the concept of a 'master' is less important.

jpcapdevila 4y ago

If SQLite gets you excited, I'm building a firebase alternative based on sqlite. I'm betting hard on sqlite so this get's me super excited!! https://javascriptdb.com https://javascriptdb.com CF people around, I would love to chat, if anyone is interested please reach out at: jp@javascriptdb.com I'll be applying to this beta for sure!

js4ever 4y ago
Super interesting! I really like the idea. I'll join the beta, email sent :)
- jpcapdevila 4y ago
  Any feedback on what do you find interesting would be awesome :) thanks!!

SheinhardtWigCo 4y ago

Big fan of Cloudflare but I wish they would stick to descriptive product names. Good: Workers, KV, Durable Objects, Cron Triggers Bad: Spectrum, Zaraz, R2, D1

alberth 4y ago
Naming is hard. > Zaraz That's the name of the company they acquired. Though, I do agree that more descriptive naming is nice. E.g. Zaraz = SafeXXS D1 = LDS (light database system) R2 = ObjectStore Spectrum = Reverse Proxy

ctur 4y ago

systemvoltage 4y ago
No one really means anything ill and this political correctness madness needs to stop.
- nwsm 4y ago
  - manigandham 4y ago
    Because it's not painful to others and intent always matters. These words are everywhere in the language; you're not really changing anything with these antics other than derailing the subject to appease those who assume offense on behalf of an imagined group of people that can't distinguish context.
    - sorenbs 4y ago
      Most of us moved on to better terminology 4+ years ago. The only ones derailing conversations are grumps like yourself who refuse to get with the program. Why is this so important to you?
      - manigandham 4y ago
        Who are you and what's this "program" you deem to impose on others? No thanks, I'll stick with the actual majority that have mastered using relevant language and rational context in discussions without being slaves to performative social constructs. Instead of assuming what's actually important to me, perhaps some introspection of why you immediately think of slavery in a computing context would be more helpful.
  - systemvoltage 4y ago
    People want to change Chess to red/blue. I'll continue to play with white and black pieces. This is a slippery slope of destroying the society by being hyper sensitive about things that no one really means. Absolutely hate this and fills me with disgust that people obssess over this kind of petty things. Life is beautiful. Enjoy it. Be kind to others that have zero intention of offending you.

aeyes 4y ago

What write throughput and latency can we expect from this database? Are there any limitations, for example on the number of tables or size of the database?

ngrilly 4y ago

Not clear from reading the post if the SQLite C library is embedded and linked in the Worker runtime (which would mean no network roundtrip) or if each query or batch of queries is converted to a network request to a server embedding the SQLite C library. That's important to understand because that's one of the key advantages of SQLite compared to the usual client-server architecture of databases like PostgreSQL or MySQL: https://www.sqlite.org/np1queryprob.html https://www.sqlite.org/np1queryprob.html

benjiweber 4y ago

I was expecting this to be using https://en.wikipedia.org/wiki/D_(data_language_specification) https://en.wikipedia.org/wiki/D_(data_language_specification... given the name.

deanc 4y ago

Any word on pricing =)?

irq-1 4y ago

Best Effort Writes[1] are an opportunity here. Non-transactional, write to the local replica (ensure foreign keys, constrains, valid data, etc...) and then try to write to the main write-enabled DB. Caching should work without changes since the local replica is updated. This could be cheaper (send binary diffs) and more resilient to brief network issues. The key is to let the user decide what really needs ACID and what doesn't. If someone wants to make the next Facebook or Reddit they'll need huge write throughput and if some votes or updates are lost, that may be a good trade-off. [1] You could add a BEW file (like WAL file) to sqlite for Best Effort Writes.

jzer0cool 4y ago

How does this work when developing locally. Is it SQLite for local development?