9 ms·
Deno KV internals: building a database for the modern web
- jaxelr 2y ago(2023)
- hpeter 2y agoI think Deno KV is a great tool with a nice API, it's great to see how it works. Really well designed. I used it a couple of times locally with Sqlite for CLI apps, if you want to do some data manipulation stuff with TS from the CLI and need a db, don't look further. I also used it in Production with FoundationDB on Deno Deploy. It does not replace your postgres/mysql database but a different beast entirely for many reasons. One is pricing. You pay per read and write. An issue I had is that it's hard to migrate from using KV with Deno Deploy. You can migrate to a local SQLite backed instance but will need to develop your own solutions on how you migrate and it will cost more the larger your database gets because you pay for reads. I do think it's great, but I would recommend using Deno Deploy only if your reads and writes produce value that offset the costs, else you can find yourself in the issue of needing to migrate. For example, use it for features you offer to authenticated users, but don't use it for things available in the open, else you open up yourself to high fees from DDOS.
- skybrian 2y agoFor reads, maybe in-memory caching would help with DDOS?
- mbb70 2y agoDeno charges for "inbound HTTP requests", so the DDOS can just query uuids till your checks start to bounce
- nbbaier 2y ago> I used it a couple of times locally with Sqlite for CLI apps, if you want to do some data manipulation stuff with TS from the CLI and need a db, don't look further. Are any of the these tools open source? Would love to look at what you're doing
- hpeter 2y agoI often build project specific tooling, to generate code or just transform some data, but maybe someday I create a FOSS version I can share. Stuff like parse some XML or JSON and output Go structs and functions and Typescript types and functions, HTML,React Components, SQL Tables, Stored functions, Pl/PGSQL. Mostly to avoid writing boilerplate when I use the same data structures in the database, middleware, client. For simple CRUD apps, it works well. I use local KV to track changes so I don't have to rerun things I don't need. But Deno is great for CLI reporting tools or Scheduled tasks, fetch and aggregate data. I think of Deno as a little swiss army knife. It's a tool that got everything built in. I use the Repl a lot, just for a specific task, get it done then move on.
- thedarkdev 2y agoYou can now back it up to Google Cloud Storage and Amazon S3. KV is still in beta, its only going to get better.
- fidotron 2y agoI happened to use foundationdb for the leaderboards for an experimental game recently, for which fdb was the main experiment. The big difference was I used the “record layer”, however, naively believing this meant only needing to understand the record layer . . . No. In foundationdb it is vital to understand all the layers up to and including the one you are using, partly so you can make sense of the documentation, but also because the problems you run into are so tied to the semantics of the lower layers. That said, the great thing about fdb is deploying it is so easy. It is like a secret weapon hiding in plain sight. The game https://luduxia.com/showdown/ https://luduxia.com/showdown/
- dzonga 2y agobeautiful game!! all on the web :clap
- fidotron 2y agoThanks! I also added them to my other game that's been here before: https://www.luduxia.com/whichwayround/ https://www.luduxia.com/whichwayround/ The funny thing is I modeled the service for leaderboards on a commercial product an old boss of mine wrote v1 of in PHP/MySQL over 20 years ago. https://web.archive.org/web/20040806114005/http://www.macrospace.com/services/connectx_overview.shtml https://web.archive.org/web/20040806114005/http://www.macros... Games people end up with things like massively sharded mysql and replication fun. One of the nice potential things with fdb is keeping transactions within user scope, and not having to deal with the sharding/replication yourself, you just have to arrange the keyspace correctly instead. I have worked on games where people sorely underestimated the complexities of db replication and literally burned tens of millions in recovering from the mistake.
- the_gipsy 2y ago"Deno KV" looks uncomfortably shoehorned into Deno. Some people are of course gonna use it if it's built in, but it doesn't look something that a good language/platform would ship with, if it wasn't for the money.
- satvikpendem 2y agoSeems like these companies want to be like Vercel or Supabase, a full backend as a service where you just write code and deploy and the backend handles everything for you.
- BoorishBears 2y agoDoesn't just seem like it, follow Guillermo Rauch's investments. The same game plan every. single. time. Get into the JS developers' zeitgeist, wrap an existing cloud service with a new frontend, sell it at a markup. Bonus points if the services can sell for each other.
- satvikpendem 2y agoTrue, however the DX is outstanding for Vercel, and the price is pretty decent enough, even though I know they all just wrap AWS. If I don't have to deal with AWS, I would (and do) pay extra to not do so.
- BoorishBears 2y agoVercel is not a problem. Pumping millions of dollars into the JS ecosystem through sponsorships and events to define the development landscape in terms of what helps your bottom line... that's bad. React has RSC because Vercel wanted to fight Shopify. NextAuth was practically bought out by Clerk to serve as sales funnel. <img> tags are marked as "not good" by a leading React framework because the hosting provider behind it wants you to pay them for image optimization. What Rauch is doing is the developer equivalent of private equity squeezing, and what's insane is how well it's working.
- mcpar-land 2y agopaid SaaS features appearing in my language's runtime is... odd. I remember on launch, Deno specifically did not add backwards compatibility for installing Node.js packages from npm, by design. It was supposed to be a "greenfield" approach to serverside javascript/typescript... buuut then they folded and added npm dependency mgmt to their tooling. Some of the decisions in Deno feel like the "grow and find ways to monetize" strategy of your average vc-funded tech startup, but instead of being a SaaS it's your whole runtime.
- pjmlp 2y agoOne of the reasons .NET contracts were never widely adopted was that the infrastructure to make them useful was only available on VS Enterprise. Since most companies go with VS Professional, zero adoption. Same applies to unit testing code coverage on VS.
- skybrian 2y agoThe built-in Sqlite-based implementation is free and fully functional. It should be useful for local apps.
- cryptonector 2y agoSQLite3-based implementation of... what? Oh. Deno KV uses SQLite3 under the covers. That's... funny.
- threecheese 2y agoOutsider perspective; wrapping SQLite to provide a simple k/v JavaScript object persistence that supports all the strangeness of their types feels valuable, no? Irrespective of the economics of Deno itself.
- cryptonector 2y agoSQLite3 seems like overkill for a K/V store, and in particular it seems like a pessimization for a K/V store. Besides, if you're going to layer a network service on top of SQLite3 then things like rqlite and similar start to look very good, though, admittedly a K/V store is much simpler.
- cpursley 2y agoHalf of HN posts are about poor implementations of what's already a first class citizen in the Erlang runtime / Elixir ecosystem.
- rgrieselhuber 2y agoFor non-Erlang people, what’s the reference?
- EarthLaunch 2y agoSort of. I was using Erlang in 2007, well before switching to Node/JS. Erlang/Elixir has its downsides, which is why I switched. The entire industry has been twisting itself in knots trying to solve ops problems that Erlang/OTP solved in software long ago. I'm trying Deno Deploy though, because it seems like an attempt to combine those benefits with the JS ecosystem. That has advantages in: Language usability, frontend/isomorphic, libraries, and serverless. So far it feels like the future. Something like this will be, though I'm almost expecting a new language for it.
- ianbutler 2y agoWe actually implemented FoundationDB for our Python focused cloud platform [0] recently too. We found it to be super easy to set up and scale and it. Setting up FDB is literally install a package and make sure a file gets to all nodes in a cluster and you’re good. It also integrated really well with our function runtime. You get a lot of distributed goodies for free in FDB with little effort and they “stack” their primitives very well too. As an example, there’s a built in multi-tenancy layer that is just using key prefixes under the hood, but it’s built in natively and can be accessed by the higher level apis. It’s interesting that Deno went with a full separate transaction layer per region on top of a global cluster instead of doing regional clusters and making one region the primary writer and then doing request coalescing. [0] https://www.bismuthos.com https://www.bismuthos.com
- jauntywundrkind 2y agoI'm used to KV systems being really simple shallow systems. WASI-keyvalue for example is quite simple, stopping at providing an increment capability. https://github.com/WebAssembly/wasi-keyvalue/blob/main/wit/atomic.wit https://github.com/WebAssembly/wasi-keyvalue/blob/main/wit/a... I knew Deno KV was built on FoundationDB, expected this would be a neat but simple systems architecture rundown. But... Turns out Deni really went super deep in building a KV, by adding atomics! > To maximize performance and concurrency for atomic operations, we built the Deno KV Transaction Layer that manages the global order of atomic operations. For each transaction that is received by this Transaction Layer: I thought it was particularly creative how a .get returns not just the value, but some reference to what the get was. So when the atomic change comes, they can check the get. This was a neat way to let some sets use previous data safely, I thought. Does anyone else have any other examples of thicker kv APIs? It feels like we're nearing a cap'n'proto level of promise pipelining, as we extend kv this way; I though Denos dodge was extremely expertly picked to limit how complex references to existing operations would need to be.
- skybrian 2y agoDeno KV has a maximum record size of 64Kib [1] which is pretty severe. Compare with 25Mib for Cloudflare Worker KV [2]. It can be worked around, but it will make your design more complex if you have potentially large textareas, let alone images. Hopefully it will be raised by the time Deno KV gets out of beta. [1] https://deno.land/api@v1.43.2?s=Deno.Kv https://deno.land/api@v1.43.2?s=Deno.Kv [2] https://developers.cloudflare.com/kv/platform/limits/ https://developers.cloudflare.com/kv/platform/limits/
- pier25 2y agoI big issue with Cloudflare KV (and Workers too) is that values do not persist at the edge. They are copied on demand much like a CDN and then expired after some time. This works great when you have a ton of traffic but sucks when you want consistent good performance with erratic traffic. I wish CF had a premium tier where both Workers and KV would persist the the edge.
- kentonv 2y agoDurable Objects persist at the edge.
- pier25 2y agoso when you make a change it's replicated to all regions?
- sorenbs 2y agoNo. A DO only exists in a single edge location. That's the entire point of Durable Objects. D1 will replicate soon, but doesn't do so yet. My understanding is that it will replicate to ~5 locations and keep these locations fresh.
- pier25 2y agoIt's weird because all their marketing says it's distributed but it's not really
- nimchimpsky 2y ago[dead]
- giancarlostoro 2y agoI have to wonder if it might have been easier to make something like localStorage, but call it serverStorage and add that into deno, then anyone could use both interchangeably. localStorage is well known and super simple to use.
- holoduke 2y agoYes of course. 40 xhr requests to load a single page. Developers with no clue on the impact on what they are doing. Sorry a bit pessimistic here. Just had to deal with some serious similar issues in a team.
- louismerlin 2y agoI'd be curious to know if the SQLite version would be suitable in production for a small to medium-sized project. Did anybody try it out?
- cryptonector 2y ago> Web apps very often require some persistent application state. Setting up a database involves numerous configuration steps and the subsequent integration of an ORM or other systems. So this is a NoSQL argument. > Atomic transactions: a database that provides atomic operations to ensure data integrity and enable complex application logic Ok, so let's see if they mention "ACID". Nope. They do mention atomicity and durability, but they don't mention consistency and isolation. So this is about NoSQL and NoACID, but it's not really stated this way. K/V stores are great for many things. And not having a query language is great!! right up until you need one, then you get to be sad (I should know since I've had to write a query engine due to use of a K/V store). [No]ACID is where NoSQL gets interesting, IMO. But NoACID doesn't mean you can't have a query language. Anyways, if you're going to implement yet another K/V store you really should justify it in relation to other K/V stores that exist, not as "using RDBMSes is ETOOHARD".
- dangoodmanUT 2y agoThis feels like an unnecessary layer on top? Increments can just be a function that uses an existing txn to read, incr, write. It feels like they implemented transactions on top of transactions, adding complexity? Maybe I didn't read it right though
- losfair 2y ago(I work on Deno KV) A read-modify-write retry loop causes a high number of commit conflicts, e.g. for atomic increments on integers. Getting higher than 1/RTT per-key throughput requires the "backend" to understand the semantics of the operations - apply a function to the current value, instead of just checking whether timestamp(value) < timestamp(txn.start) and aborting commit if not.
- ec109685 2y agoWhy not funnel all writes to a single region and leverage simpler transaction semantics.
- losfair 2y agoDistributed consensus + fsync() determines the lower bound on commit latency. We have to wait for the data to be durable on a quorum of transaction logs before returning success for a transaction. That's usually 5-10ms, even within a single region.
- spirobelv2 2y agojust use bun and sqlite on your own server. got burned by deno deploy and will never use deno again. serverless is useless and slow.
- tqkxzugoaupvwqr 2y agoCould you please elaborate on what happened?
- spirobelv2 2y agoyes. took part in a hackathon and decided to make my own webframework while doing it. It was supposed to be compatible with deno and bun and I chose deno deploy to deploy my stuff. But 17 hours before the deadline this happened: https://twitter.com/spirobel/status/1786665928954085562 https://twitter.com/spirobel/status/1786665928954085562 deno deploy just showed a completely useless error message and stopped working. I installed caddy on a vps and got it to work in 5 minutes. I ripped out the deno compatibility and switchded to bun sqlite instead of turso and the website got even faster. That is because the server has the data in the sqlite db. No need to roundtrip to turso. btw. the framework is open source now: https://github.com/spirobel/mininext https://github.com/spirobel/mininext It is like a non bloated version of nextjs. My intention is to displace php and wordpress. It is possible to get started with just html and css knowledge, but you have the whole javascript ecosystem at your fingertips. It has no external dependencies, just 3 modestly sized files you can understand in an afternoon and it will reload your browser and rebuild when you save your code.
- chrstphrknwtn 2y agoI've been messing around with Bun, esbuild and TSX templates for generating static sites. Bun feels like the future.
- spirobelv2 2y agoI agree. It is so much fun. Difference between deno and bun is night and day. They really shouldnt be put in the same bucket.
- gr4vityWall 2y agoI read this article when it came out last year, and I still wonder if this could be implemented in a cross-runtime way.
- thedarkdev 2y agohttps://jsr.io/@cross/kv https://jsr.io/@cross/kv
- gr4vityWall 2y agoThanks for the link. They seem to build some cool libraries.
- fulafel 2y agoThe atomics system use cases with the check() calls sound error prone, do you just get wrong answers if you omit a field that's involved? Could they be automatically done for involved keys? (Or for preserving explicitness in API usage, error-signaled if missing)
- Dave_Rosenthal 2y agoAs a FoundationDB partisan, it's been great to see more and more people starting to build stuff with it. Someone recently told me that they think of FDB as "assembly language for databases". I liked that. I think it faithfully captures both the good and bad parts of FoundationDB: 1) You are starting with a blank canvas 2) You will be dealing with low-level primitives 3) You have complete freedom in what you build 4) It's fast if you use it right For each one of these items you could look at it and say "that's awesome" or "yikes." You know which group you are in! I hope more people in the 'awesome' group realize that using FoundationDB can transform the development of a new distributed data store from a multi-year project into a few weeks of hacking. This is because FoundationDB solves the scalability, fault tolerance, and ACID parts for free--the least creative and most time-consuming aspects.
- btown 2y agoDo you feel like the tutorials on https://apple.github.io/foundationdb/tutorials.html#fundamentals-class-scheduling https://apple.github.io/foundationdb/tutorials.html#fundamen... are a good way for someone to get an impression of what the experience of working with FoundationDB feels like in practice? Are there any go-to walkthroughs that you'd recommend?
- Dave_Rosenthal 2y agoThat class scheduling one is a good place to grok the basics. Just extending the kind of approach introduced there will get you decently far. The best resource after the tutorials is: https://github.com/FoundationDB/awesome-foundationdb https://github.com/FoundationDB/awesome-foundationdb Unfortunately, I think a lot of the advanced 'tricks of the trade' (alternatives to use cases for long-running transactions, when exactly to dispatch to a cloud object store, how to migrate whatever schemas you end up creating, etc.) that all the big serious users of FDB are doing are not as well covered.
- btown 2y agoI for one would be an avid audience if you ever wanted to blog about any of that!