5 ms·

Show HN: Show HN: Dzero – The fastest globally distributed SQLite database

24 points by yoeven 2y ago

At JigsawStack.com, we are using Redis by Upstash to store and validate API keys among other datasets for speedy retrieval. As we scaled to more Asian markets and added replicated nodes, the cost started to shoot up while performance got a lot worse. I discovered hosting SQLite on Fly which was crazy fast & cheap with the full power of a relational database but difficult to use and scale without writing tons of code & devops. Then I found Turso as a managed service which is pretty cool but again got really expensive and had to manage a bunch of replications manually and pay per replications (also not a big fan of the UI, lol). Eventually, I found Cloudflare D1, which was amazing both cost at scale and performance with Cloudflare edge infra but it stops there, it's very hard to work with D1 out of the box and to scale it for an actual production platform with tons of limits and without getting locked into the Cloudflare ecosystem. So I built Dzero on top of D1 as a fully managed service that automatically scales globally with over 320 edge locations at no additional cost and can easily be integrated into any platform or framework without getting locked into any ecosystem! The goal of the dashboard was to be like a Supabase for SQLite, meaning that everything should be easily manageable from the dashboard, from creating tables to editing data! Right now we are in early access and would love for you to sign up and give it a try! Feedback is always appreciated! We have big plans for Dzero, check out some of our plans on the site!

deleted 2y ago
[deleted]
simlevesque 2y ago
Just a heads up, nothing loads on the website with WebGL disabled, instead of just the 3d globe thing being missing.
- yoeven 2y ago
  Thanks! will take a look :)
hobofan 2y ago
So you say that it's built on top of D1, but are claiming that you are working around its limitations. What does it actually look like in practice? From everything that I can see, the whole offering looks very identical to Cloudflare D1 + a SQL editor UI (which is a really negligable value add). The copy that mentions the "free global replication" reads like that is a unique feature, while that's already a builtin feature of D1. This feels like the Vercel serverless fiasko from a month ago where they tried to boast better serverless performance characteristics than AWS Lambda, while building on top of that.
- yoeven 2y ago
  Few ways we are taking it a step further, one, ofc is an actual well built dashboard with table editors, which makes a huge difference to many devs trying to manage their DB with all the tools they need in a UI. Which in comparison Cloudflare UI doesn't do much beyond creating the database for you and the table editors are horrible to use/buggy. With CF you would need work with D1 within a worker function and manage the balancing of that worker with hundreds micro DBs when you scale which gets very complex. Something we solved to an extent but still more things to build to fully achieve that! Dzero allows you to use SQLite without the lock into the ecosystem of CF, so locally as well without the worker CLI and a bunch of workaround to run your project locally. While D1 offers cached read data to some of their Edge locations, we took it a step further by cloning SQLite files and actually storing it in S3/R2 instances around the globe with read consistency. This allow for better performance gains and less cold starts of D1 and workers! Lastly, working around the limits. The limits of 10GB per DB is a hard limit due to the infra, however account limits has "unlimited" scale which CF can unlock at a certain enterprise level. Our goal is to build a form of Horizontal/Vertical sharding similar to planetscale with multiple D1 instances to scale your data with 10GB per DB.
bahador 2y ago
How do you get past the d1 storage limits[0]? Maximum database size 10 GB (Workers Paid) Maximum storage per account 250 GB (Workers Paid) - [0] https://developers.cloudflare.com/d1/platform/limits/ https://developers.cloudflare.com/d1/platform/limits/
- yoeven 2y ago
  Not yet fully, but the goal is to eventually introduce Horizontal/Vertical sharding with multiple DBs that work together
MuffinFlavored 2y ago
> So I built Dzero on top of D1 as a fully managed service that automatically scales globally with over 320 edge locations at no additional cost and can easily be integrated into any platform or framework without getting locked into any ecosystem! A managed service layer on top of a managed service layer Sounds expensive? Looks super well done though. I would just be concerned... what's the funnel for people who find Cloudflare? Ok, what's the funnel for people who use D1? Ok, what's the funnel for people who know about Cloudflare/D1 and use it, and then go look for an extra layer on top/competitor I guess you could hope on people never having to go down that path (how many people already integrated against D1 would eject and switch to this versus how many just find this originally)...
- yoeven 2y ago
  Different levels of managed haha, I think D1 provides a great infra layer for SQLite on the Edge, however it stops there. It's missing the dev tools required to manage a production database for a bigger team. We're finding ways to keep cost low, with other methods of smart caching at the CDN level, S3 storage of SQLite files, etc. One thing I found personally and have seen others face is that D1 is great on the technical side of things but just bad on the usability side which prevents many from trying out D1 or even actually implementing it for prod! Having the right UI and well built dev tooling around database is as important as the database itself. we see this with companies like supabase, people go there for the tooling built around Postgres and not just Postgres.
  - MuffinFlavored 2y ago
    > It's missing the dev tools required to manage a production database for a bigger team. What's stopping Cloudflare from implementing this themselves?
    - yoeven 2y ago
      Nothing honestly, the same way nothing stops AWS/Google from implementing the whole Supabase platform themselves or Planetscale themselves. The thing is as a more focused startup, we are able to move faster and hopefully better than some of the bigger giants that have hundreds of different verticals to focus on. D1 is a great start and we love it, we just way to take it all the way!
alberth 2y ago
Suggestion: make it more clear what value-add is being done beyond what D1 gets you. Because I don’t know if this is intended but the vibe I get is that D1 does all the db stuff and Dzero is just a web view into your data itself. —- Slight OT: I appreciate the information density of the website.
- yoeven 2y ago
  Thanks for the feedback, will make the messaging clearer! D1 does the base infra stuff of running SQLite really well. We do the web dashboard stuff and also the API layer making it way more easier to interact with the database. We do some infra stuff beyond D1 like duplicating SQLite nodes across the globe! Our goal is to make this even better with more feature allowing for better scalability
wg0 2y ago
Great packaging - simple interface. This is built on top of Cloudflare D1 as it clearly states but there are few things that I don't like about Cloudflare D1: You cannot download your SQLite database as is. All you can do is dump an SQL file and when your database sizes are clocking at few GBs (let's say 5), that option is probably not going to work because the query/API also has a time out limit of 30 seconds and that data dumped as SQL (text) takes up way more space than it is in its binary form on disk. This essentially means that you can kiss your data a goodbye and it is locked in there forever with no hope of getting it back ever as a whole. For smaller databases, to do lists, note taking, personal expense tracker etc - probably you can live with those limitations.
- jgrahamc 2y ago
  I'll ask the team to add this as a feature.
- yoeven 2y ago
  Ohh try out our dump API, it lets you pick specific tables with schema only or data only, etc. Right now for 10GB dbs, it should be ok but again haven't tested for that specific use case. The goal is eventually to provide a way for resumable downloads so you can download chunks at a time even for backups. We'll be launching a backup dashboard soon to download your db backups!
dang 2y ago
For a Show HN you need to have more than a signup. Please see https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html.
- yoeven 2y ago
  You can use the platform, signup and I can approve your request. I just want to filter out bad actors
thdxr 2y ago
i'm confused - how are you getting the database replicated to 320 edge locations? afaik d1 is still working on read replicas
- yoeven 2y ago
  We take it a step further since sqlite is a file based system we can duplicate and manage consistency across storage instances using R2 and S3 instances

24 points by yoeven 2y ago

deleted 2y ago

[deleted]

simlevesque 2y ago

Just a heads up, nothing loads on the website with WebGL disabled, instead of just the 3d globe thing being missing.

yoeven 2y ago
Thanks! will take a look :)

hobofan 2y ago

So you say that it's built on top of D1, but are claiming that you are working around its limitations. What does it actually look like in practice? From everything that I can see, the whole offering looks very identical to Cloudflare D1 + a SQL editor UI (which is a really negligable value add). The copy that mentions the "free global replication" reads like that is a unique feature, while that's already a builtin feature of D1. This feels like the Vercel serverless fiasko from a month ago where they tried to boast better serverless performance characteristics than AWS Lambda, while building on top of that.

yoeven 2y ago
Few ways we are taking it a step further, one, ofc is an actual well built dashboard with table editors, which makes a huge difference to many devs trying to manage their DB with all the tools they need in a UI. Which in comparison Cloudflare UI doesn't do much beyond creating the database for you and the table editors are horrible to use/buggy. With CF you would need work with D1 within a worker function and manage the balancing of that worker with hundreds micro DBs when you scale which gets very complex. Something we solved to an extent but still more things to build to fully achieve that! Dzero allows you to use SQLite without the lock into the ecosystem of CF, so locally as well without the worker CLI and a bunch of workaround to run your project locally. While D1 offers cached read data to some of their Edge locations, we took it a step further by cloning SQLite files and actually storing it in S3/R2 instances around the globe with read consistency. This allow for better performance gains and less cold starts of D1 and workers! Lastly, working around the limits. The limits of 10GB per DB is a hard limit due to the infra, however account limits has "unlimited" scale which CF can unlock at a certain enterprise level. Our goal is to build a form of Horizontal/Vertical sharding similar to planetscale with multiple D1 instances to scale your data with 10GB per DB.

bahador 2y ago

How do you get past the d1 storage limits[0]? Maximum database size 10 GB (Workers Paid) Maximum storage per account 250 GB (Workers Paid) - [0] https://developers.cloudflare.com/d1/platform/limits/ https://developers.cloudflare.com/d1/platform/limits/

yoeven 2y ago
Not yet fully, but the goal is to eventually introduce Horizontal/Vertical sharding with multiple DBs that work together

MuffinFlavored 2y ago

> So I built Dzero on top of D1 as a fully managed service that automatically scales globally with over 320 edge locations at no additional cost and can easily be integrated into any platform or framework without getting locked into any ecosystem! A managed service layer on top of a managed service layer Sounds expensive? Looks super well done though. I would just be concerned... what's the funnel for people who find Cloudflare? Ok, what's the funnel for people who use D1? Ok, what's the funnel for people who know about Cloudflare/D1 and use it, and then go look for an extra layer on top/competitor I guess you could hope on people never having to go down that path (how many people already integrated against D1 would eject and switch to this versus how many just find this originally)...

yoeven 2y ago
Different levels of managed haha, I think D1 provides a great infra layer for SQLite on the Edge, however it stops there. It's missing the dev tools required to manage a production database for a bigger team. We're finding ways to keep cost low, with other methods of smart caching at the CDN level, S3 storage of SQLite files, etc. One thing I found personally and have seen others face is that D1 is great on the technical side of things but just bad on the usability side which prevents many from trying out D1 or even actually implementing it for prod! Having the right UI and well built dev tooling around database is as important as the database itself. we see this with companies like supabase, people go there for the tooling built around Postgres and not just Postgres.
- MuffinFlavored 2y ago
  > It's missing the dev tools required to manage a production database for a bigger team. What's stopping Cloudflare from implementing this themselves?
  - yoeven 2y ago
    Nothing honestly, the same way nothing stops AWS/Google from implementing the whole Supabase platform themselves or Planetscale themselves. The thing is as a more focused startup, we are able to move faster and hopefully better than some of the bigger giants that have hundreds of different verticals to focus on. D1 is a great start and we love it, we just way to take it all the way!

alberth 2y ago

Suggestion: make it more clear what value-add is being done beyond what D1 gets you. Because I don’t know if this is intended but the vibe I get is that D1 does all the db stuff and Dzero is just a web view into your data itself. —- Slight OT: I appreciate the information density of the website.

yoeven 2y ago
Thanks for the feedback, will make the messaging clearer! D1 does the base infra stuff of running SQLite really well. We do the web dashboard stuff and also the API layer making it way more easier to interact with the database. We do some infra stuff beyond D1 like duplicating SQLite nodes across the globe! Our goal is to make this even better with more feature allowing for better scalability

wg0 2y ago

Great packaging - simple interface. This is built on top of Cloudflare D1 as it clearly states but there are few things that I don't like about Cloudflare D1: You cannot download your SQLite database as is. All you can do is dump an SQL file and when your database sizes are clocking at few GBs (let's say 5), that option is probably not going to work because the query/API also has a time out limit of 30 seconds and that data dumped as SQL (text) takes up way more space than it is in its binary form on disk. This essentially means that you can kiss your data a goodbye and it is locked in there forever with no hope of getting it back ever as a whole. For smaller databases, to do lists, note taking, personal expense tracker etc - probably you can live with those limitations.

jgrahamc 2y ago
I'll ask the team to add this as a feature.
yoeven 2y ago
Ohh try out our dump API, it lets you pick specific tables with schema only or data only, etc. Right now for 10GB dbs, it should be ok but again haven't tested for that specific use case. The goal is eventually to provide a way for resumable downloads so you can download chunks at a time even for backups. We'll be launching a backup dashboard soon to download your db backups!

dang 2y ago

For a Show HN you need to have more than a signup. Please see https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html.

yoeven 2y ago
You can use the platform, signup and I can approve your request. I just want to filter out bad actors

thdxr 2y ago

i'm confused - how are you getting the database replicated to 320 edge locations? afaik d1 is still working on read replicas

yoeven 2y ago
We take it a step further since sqlite is a file based system we can duplicate and manage consistency across storage instances using R2 and S3 instances