9 ms·
D1: Improvements to performance and scalability
- ericstegemann 3y agoEek 0.75 per GB of storage! AWS is 0.115 One of our DBs has 2.3TB in it.
- yashap 3y agoYeah, I’d imagine this will be cheaper than AWS RDS for low storage use cases (lack of fixed monthly compute costs wins out), but more expensive for high storage. Like quite cheap for a 10-100 GB DB, quite expensive for a 1-10 TB GB. Though they do say: > when we enable global read replication, you won’t have to pay extra for it, nor will replication multiply your storage consumption With AWS you’ll have at least one read replica for failover, so $0.23/GB. And if you really want global read replicas, with AWS you might end up with something like a primary in North America and read replicas in South America, Europe and Asia. That would work out to $0.46/GB, so closes the gap a bit.
- re-thc 3y agoDepends if you do more compute than store data. I’ve been in a lot of analytical workloads running different types of reports besides your normal CRUD.
- jFriedensreich 3y agoat least at the moment D1 is for a completely different use case, d1 has currently a limit of 100mb and will now be increased to 1gb.
- Akkuma 3y agoThe costs as a whole has me worried. I'm not sure it'll be better than just using https://turso.tech/pricing https://turso.tech/pricing as that is already a free tier of 8GB and it might cost less overall the paid tier.
- re-thc 3y agoIt's 8gb in total including replicas. 1 database with 2 replicas (up to 3 free) would be <3gb in reality. It'll most likely cost more overall.
- re-thc 3y agoBetter than Planetscale at 2.5 per GB and you have to pay for replicas.
- ccorda 3y agoKenton Varda (tech lead) has some more notes in this twitter thread: https://twitter.com/KentonVarda/status/1659551757796515846 https://twitter.com/KentonVarda/status/1659551757796515846
- aranke 3y agoCurious if you prototyped DuckDB before deciding to invest further into SQLite. DuckDB works great as an in-memory database (it's also the default mode).
- rektide 3y agoI expect most workloads are more OLTP/transactional than OLAP/analytical.
- elithrar 3y agoCorrect. DuckDB is really interesting technology, but it's not a direct successor to SQLite for transactional workloads. It's also very new: there's a LOT of new code in DuckDB on top of the (heavily fuzzed) SQLite parts. (I use it personally, but it's not the same thing as what we're building with D1)
- piperswe 3y agoAlso, for OLAP workloads there's Workers Analytics Engine. Analytics Engine is arbitrarily scalable as opposed to a D1-like solution - I can almost guarantee I've inserted more data into its internal variant (which is built on extremely similar underlying infrastructure) than just about any customer would think of doing, and it handles it like a champ. Basically, we already have a product for doing DuckDB-like things, and the D1 architecture isn't great for high volume OLAP workloads. Disclaimer: I'm an engineer at Cloudflare, but not on a developer platform team. I'm not speaking for our developer platform strategy or anything, I'm just commenting on how it looks from where I sit.
- kentonv 3y agoWe didn't, no. I'm sure there's a lot of really cool local-first databases out there, but SQLite has the benefit of being incredibly widely battle-tested, with literally billions of installations worldwide. It has received thorough security research and fuzzing (it's part of Chrome's attack surface after all). And there's tons of resources online to help people understand how to use it. Although I'm sure there are alternatives that serve certain use cases better it's hard to imagine anything coming close for ours. That said, the storage engine we've built is not that heavily dependent on SQLite specifically. Any database that uses a write-ahead log like SQLite does should be possible to adapt to it in the future. So maybe we'll eventually open it up to a variety of choices, or even let you bring your own as a Wasm module.
- pier25 3y agoHas the DX of Workers improved? I think Deno is lightyears ahead.
- KRAKRISMOTT 3y agoStill no compatibility with standard ORMs like Prisma either. And the team has been aware of the issue for years now https://github.com/cloudflare/workers-sdk/issues/2701 https://github.com/cloudflare/workers-sdk/issues/2701 https://news.ycombinator.com/item?id=31341513 https://news.ycombinator.com/item?id=31341513
- kentonv 3y agoSorry that D1 has been slow out the gate. Now that we've solved the basic technical issues we can really focus on improving DX.
- Akkuma 3y agoYou can use drizzle-orm with Cloudflare and that is fully compatible.
- rozenmd 3y agoDrizzle is amazing.
- Akkuma 3y agoThis is my first time using it and I've been very pleased with it so far. It keeps it simple, has solid typing & schema building, and reminds me of LINQ. I'm also a thin models kind of person and the fact this is just an object without needing to build ORM classes is even better.
- radiofmd 3y ago> standard ORMs like Prisma Prisma isn't standard and too slow anyway.
- davnicwil 3y agoSo this looks like basically a distributed SQLite with read replication for free across cloudflare's edge. Is that right?
- m3kw9 3y agoSo serverless meaning you don’t manage the db servers, so they may or may not put your stuff on the edge, cloudflare takes care of maybe load balancing too?
- jgrahamc 3y agoYes, Cloudflare handles load balancing and neat tricks like measuring the latency between your users, your code and databases etc. and moving your code around the network to make it run fast: https://blog.cloudflare.com/announcing-workers-smart-placement/ https://blog.cloudflare.com/announcing-workers-smart-placeme... The vision of the Supercloud is that you give us your code and we'll figure out where and how to execute it: https://blog.cloudflare.com/welcome-to-the-supercloud-and-developer-week-2022/ https://blog.cloudflare.com/welcome-to-the-supercloud-and-de...
- threatofrain 3y agoIMO serverless is more about the pricing model more than whether a service is managed. Both Amazon RDS and Aurora are managed services, but Aurora is the one with the serverless pricing model.
- kneebonian 3y agoSo I hate cloudflare with a burning passion, at some point cloudflare decided me home IP was bad and has started flagging it all over the place, which leads to vast swathes of the internet now being inaccessible to me. There is danger in centralized systems.
- kentonv 3y agoHave you tried using Privacy Pass? It is a browser extension which helps prove that your traffic is legitimate which should reduce the incidence of challenges from Cloudflare, even if your IP has bad rep. It uses advanced cryptography to do this without creating any new tracking vectors. (Disclosure/disclaimer: I work for Cloudflare but in a different department; I'm not an expert on Privacy Pass.)
- simiones 3y ago> It uses advanced cryptography to do this without creating any new tracking vectors. Sounds like something which is designed to create hard to detect tracking vectors.
- kentonv 3y agoNote that Privacy Pass is not a Cloudflare product, it's an open protocol which Cloudflare supports. https://datatracker.ietf.org/doc/charter-ietf-privacypass/ https://datatracker.ietf.org/doc/charter-ietf-privacypass/
- ignoramous 3y agoBlog: https://blog.cloudflare.com/privacy-pass-the-math/ https://blog.cloudflare.com/privacy-pass-the-math/ (older) Design: https://privacypass.github.io/protocol/ https://privacypass.github.io/protocol/ PATs: https://blog.cloudflare.com/eliminating-captchas-on-iphones-and-macs-using-new-standard/ https://blog.cloudflare.com/eliminating-captchas-on-iphones-...
- 3y ago
- nemothekid 3y agoThis is Cloudflare's flavor of Fly.io?
- pier25 3y agoFly doesn't really offer distributed data. Edit: It does!
- avinassh 3y agois this not same - https://fly.io/docs/litefs/ https://fly.io/docs/litefs/ ? They mention it as 'LiteFS - Distributed SQLite'
- pier25 3y agoAh you're right! I thought this was a wrapper for Litestream but apparently it's a parallel project by the same author who Fly hired. https://litestream.io/ https://litestream.io/ https://github.com/benbjohnson/litestream/pull/411 https://github.com/benbjohnson/litestream/pull/411
- re-thc 3y agoFly requires you to commit compute and Fly's compute gets expensive real fast.
- pottertheotter 3y agoSo when would someone use something like this? I learn better by example if anyone has any. And for reference, here’s the original D1 announcement with some additional info https://blog.cloudflare.com/introducing-d1/ https://blog.cloudflare.com/introducing-d1/
- leetrout 3y agoI use D1 through microfeed https://www.microfeed.org/ https://www.microfeed.org/
- kentonv 3y agoYou'd use it whenever you're building an application on Cloudflare Workers and you need to store data. D1 provides you with a SQL database. It's based on SQLite, so it's designed for relatively small datasets but can serve them very quickly from the edge. Note there are several alternatives here, too. Workers Durable Objects[0] provide a lower-level primitive for building advanced distributed systems. But D1 is easier to use for typical use cases. For blob storage you might use R2[1]. And for large databases Workers can easily integrate with several serverless database providers.[2] [0] https://blog.cloudflare.com/introducing-workers-durable-objects/ https://blog.cloudflare.com/introducing-workers-durable-obje... [1] https://www.cloudflare.com/products/r2/ https://www.cloudflare.com/products/r2/ [2] https://blog.cloudflare.com/announcing-database-integrations/ https://blog.cloudflare.com/announcing-database-integrations...
- Akkuma 3y agoIs it a non-goal to be long term usable for larger databases? That would force the usage of something like turso your closest direct comparison as a possible migration strategy or relying on "Smart Placement" (which from my point of view reduces the benefit of global edge) for other serverless non-global dbs.
- kentonv 3y agoPersonally, I'm a firm believer that most "web app" use cases are better served by many small databases (e.g. per-user or per-document) rather than a single monolithic databases. This is especially true when serving users all around the world -- per-user databases can be located near each user (both for speed and to comply with data locality laws). What I'd like to enable here is a progression where you start out prototyping your app with a single D1 database, which is easy to use and reasonably fast. Then as you grow we provide tools to let you transition to many D1 databases sharded in a way that makes sense (e.g. per-user). Apps that want even more control can move to using full-on Durable Objects (which will soon support a SQLite database per-object). That said, there are certainly many use cases out there where simple monoliths make sense, especially non-interactive data crunching. I'm not sure yet if D1 will ever be the right choice for those, but the Workers platform aims to provide many options.
- Mystery-Machine 3y ago"up to 37x faster" is NOT before: 37.81ms, now: 1.82ms You can't just round down 1.82ms to 1ms.
- elithrar 3y agoFixing this. We ran a few benchmarks (and some were much faster than 37x), but this was a more typical case. Not our goal to inflate numbers.
- marcopicentini 3y agoWhy and when should I use a serverless Postgresql instead of a postresql hosted on a server?
- rozenmd 3y agothis isn't postgres
- trollied 3y agoInteresting, Time Travel works the same as Oracle Flashback. Hope there aren't any patents to trip over.
- threatofrain 3y agoI enjoy Cloudflare and have been using workers + D1 for a few months, but warn that Cloudflare's definition of beta is far more beta than what other companies mean. The API surface may change repeatedly without corresponding documentation and their Discord can be a bit sparse on help.
- ignoramous 3y ago> ...warn that Cloudflare's definition of beta is far more beta than what other companies mean D1 was in open alpha prior to today, iirc.
- fire 3y agohmm, is this roughly equivalent to Neon[1], but sqlite based rather than postgres? 1: https://neon.tech https://neon.tech
- nikita 3y ago(neon CEO) This is true. Neon however offers bottomless storage and D1 is 100Mb currently going to 1Gb.
- radiofmd 3y agoNah, D1 is in a different league.[1] One key feature of D1 is having free read replicas around the globe. That's CF's DNA and I expect this to happen this or latest Q1/24, and again, for free (so, without any surcharge). Re storage: You get ofc more than 1GB on the paid plan. Neon doesn't have read replicas and even if they already work on it, I wouldn't expect it before 2025 if at all and never at CF's pricing (Neon still charges for egress). [1] I would compare D1 rather to Turso or LiteFS from Fly or PlanetScale with many read-replicas
- colesantiago 3y agoCan I connect directly to a D1 database with a SQLite URL? That would be awesome.
- radiofmd 3y agoI am pretty excited about D1 and in general a CF fanboy, so congrats to this huge milestione. But, we need more--following is just a wish list: - Development since start took time (with a good reason), hope we see soon read replication and higher pace - So, some roadmap _with_ ETAs would be great, e.g., will replication come this or next year? - I don't care about read performance because I know that sqlite paired with CF's infrastructur will perform but I would love to have more infos on write performance; this is really sqlite's weakest point and I haven't seen any implementation which could compete anywhere with other dbs; sqlite slows down very quickly when doing a couple of small writes/sec Then, I'd like to see the close competition here on HN, so db-providers with many read-replicas (CEO's and/or devs from Turso and Fly/LiteFS), commenting on D1 and how they compare against and what they plan to compete. This is a too exciting space and time to be laissez-faire.
- benbjohnson 3y agoLiteFS author here. I’d love to compare and contrast D1 with how LiteFS works but there’s not enough technical details at the moment. I’m looking forward to future technical blog posts from Cloudflare or perhaps they could open source their implementation. There are a lot of ways to approach SQLite replication and one isn’t better than another necessarily. They simply have different trade-offs.