10 ms·

Building a KV store into the language is kind of nuts. I love how it abstracts away the local SQLite and deployed FoundationDB behind one interface. Testing wou

Building a KV store into the language is kind of nuts. I love how it abstracts away the local SQLite and deployed FoundationDB behind one interface. Testing would be super easy, as there is no question of "do I spin up an entire db instance or mock the db interface?" as SQLite is relatively lightweight and the burden for keeping both cases consistent falls to the language instead. That being said, was wondering whether you can take advantage of this without using deno deploy, or if you'd be locked in by using deno kv. Also, wondering how many bugs will only show up when deployed due to the different backbends.

samwillis 3y ago
There is a good little overview of Deno KV here (by @simonw): https://til.simonwillison.net/deno/deno-kv https://til.simonwillison.net/deno/deno-kv So it's backed by SQLite on the OS local version. I'm intrigued if there will be a way to swap out the backend for other cloud providers.
wokwokwok 3y ago
I dunno. Is having a database that happens to be a product you sell as part of your runtime good, or are you creating some mixed incentives here? Are you a database vendor now? If not, don’t build a database. If this is a mature product that someone else is looking after, and it’s good and free and we’ll maintained like redis or SQLite, sure. …but it seems like building cloud databases is not the core competency of a group building a javascript runtime. Once money is involved, it’ll either be a distraction from their core mission, or become their core mission. A KV db you can use for quick hacks? Sure. Awesome! A scalable production database you’re selling to people? O_o why are you doing that?
- frou_dh 3y ago
  They need this because Deno Deploy (their Edge platform) is not a normal/traditional deployment target. If Deno were nothing but "Deno CLI" (the equivalent of the 'node' executable, that runs as a typical server process on a Unix box) then you would be right. But there are two separate Deno runtimes (CLI and Deploy).
- csiegert 3y ago
  Deno Inc. has two core products: The free Deno runtime and the for-profit Deno Deploy, a Deno hosting service with 35 locations around the world. The question that often popped up was where to store data. Deno Inc. provided several guides to connect to different cloud services. But they want the friction reduced to a simple `await Deno.openKv()`. Deno Inc. has enough expertise in running a global service that two other companies rely on their work to offer edge functions to their customers (Netlify and Supabase). Adding a database to the service makes sense. And to be clear, they don’t develop a brand new database. They build atop of SQLite and FoundationDB.
  - wokwokwok 3y ago
    > The question that often popped up was where to store data. Deno Inc. provided several guides to connect to different cloud services. Sure. > But they want the friction reduced to a simple `await Deno.openKv()`. Do “they”? If so, who’s using it to solve that problem? …because it seems the big uses of deno deploy are not using it, fine with that and it’s pretty unclear who the “they” is in this circumstance. Still, if it’s a thin layer over foundation db or some other established database product and this is just part of the lock-in for their cloud offering, fair enough. It’s not like others (eg firebase) don’t do the same thing. The messaging “we’re building a database” and “we’re offering a hosted database service based on existing mature reliable technology” are different things though. The latter all cloud vendors do. The former is ridiculous, and it really really wasn’t clear that wasn’t what was happening.
    - electroly 3y ago
      "They" in OP's post is referring to Deno Land Inc.
- suction 3y ago
  [dead]
- jpalomaki 3y ago
  I think this kind of "batteries included" approach is interesting. My feeling is that in many cases we spend way too much time when building our platforms from various bits and pieces. Some apps and businesses would get huge productivity boost by using more opinionated platform. Not saying this is for everything and everyone, but there's so much competition on this space that I can understand why somebody would like to try something different.
ohgodplsno 3y ago
Building a KV store into the runtime, that has more features when using deno deploy is just that: vendor lock-in. It's just one more way to force you into paying for the service when you have locked yourself in and you can't use an alternative store. Deno is not and has never been an "alternative runtime" to node.
- olric 3y ago
  [dead]
- frou_dh 3y ago
  If they provide a compelling product offering, and the people understand what's available where, and choose to use it, then I guess that can be called vendor lock-in. But it's hardly nefarious.
  - ohgodplsno 3y ago
    It is nefarious when you build up your entire product for years, spamming it everywhere as "node but better", yell "it's so open source too!" on every roof, only to sneakily slide in lock-in features, it's worse than being nefarious. You most likely had this plan in mind all along, and actively lied to get there. Watch as the same happens with Bun.
    - frou_dh 3y ago
      What's stopping you sticking with Deno CLI, using it for free forever, and completely ignoring anything related to Deno Deploy? In this relationship of 3 years and counting, they are the positive contributor because they give you something useful for free and you presumably haven't given them anything.
    - iudqnolq 3y ago
      In this specific case I think it's super reasonable. The KV store has two backbends: local sqlite and distributed deno cloud. If you choose the deno cloud backend you're signing up for a closed source cloud service from the start. There's no trickery.
potamic 3y ago
It's a lock-in strategy. There is zero reason for a language runtime to be opinionated about the database. All the good programming advice will tell you such a coupling is a bad idea. What if you want to move parts of your application to a different language in the future, do you need to redo the entire database? That is just nuts! Database providers, cloud application platforms should be agnostic of the language their users are using. There are many such decisions deno is making where they want to be the end-to-end service. That is not a good sign. It creates perverse incentives for the company to devise lock-in mechanisms at each stage and discourage compatibility with the rest of the programming ecosystem, perhaps even undermine the rest of the ecosystem. I would stay away for this reason.
- deleted 3y ago
  [deleted]
- olric 3y ago
  [dead]
- p-e-w 3y ago
  > There is zero reason for a language runtime to be opinionated about the database. The API is extremely simple. It's little more than a hash map. That's hardly opinionated. > All the good programming advice will tell you such a coupling is a bad idea. That advice is outdated. Virtually any modern application will want a database, and this API can serve as a foundation for it – a foundation that conveniently already has many of the features (esp. global replication and consistency) that you will want and that are incredibly difficult to get right if you build them yourself. Think of this as the application's "file system". PaaS today usually doesn't provide direct disk access, so this is the low-level abstraction for persistent storage that is available. An abstraction that, in many cases, will in fact be all your application ever needs.
  - jabradoodle 3y ago
    I'm not familiar with the offering but find your comment confusing. It seems to suggest there isn't lock in because it's just a hashmap, but also that it is globally replicated and consistent.
thepratt 3y ago
To me this is quite similar to the Erlang VM's ETS in-memory store. There's a lot of usefulness in ETS itself, especially how it distributes inside clusters. Having an easy to use, efficient, and optimized store as part of the language can be a good enabler. Only time will tell if it is/isn't.
scotty79 3y ago
I think what's nuts is that databases data structures are not built into the language/stdlibs. I can make an array but if I want to add an index so I can access it fast then I have to jump through some inordinate hoops. I can make an in memory hashmap, but if I want to persist it and keep accessing it fast directly from persistent storage suddenly things get unnecessarily complicated. I think fast searchable file backed data structures should be primitives of any computer language and runtime.
kybernetikos 3y ago
I think almost all languages should have a database built in. Almost all programs need a way of storing and querying data but the drama required to hook up to an external db is excessive for little programs. Basic dB functionality should be as available in language standard libraries as file access.
- dgellow 3y ago
  What do you mean by drama to hook up external db? It’s almost always just a one liner: let conn = db.newConn(host, port, …).
  - qbasic_forever 3y ago
    One line _after_ you figure out the myriad of database libraries available, half of which are abandoned, a quarter which are little toys, and if you're lucky one that has a core dev who is being paid real money to support and maintain the library.
    - dgellow 3y ago
      Ah yes, that's fair. Finding the good library to rely on can be really time consuming.
flohofwoe 3y ago
It's not really built into the language though? It's just a function on the global Deno object, similar to how there's a global 'document' object in browsers (and that's not 'built into the language' either).
- hu3 3y ago
  Backed into the runtime is probably what your parent meant.
zja 3y ago
Deno.dev (deploy) uses a different, proprietary runtime than Deno. I believe Ry mentioned at a recent talk that on deno.dev, the kv will be synced across all instances. Obviously this can’t be the case for stock Deno.
- komon 3y ago
  KV storage is SQLite, and SQLite replication and syncing using an out-of-process daemon is becoming a solved problem. For that application at least no changes to the runtime would be necessary
theGnuMe 3y ago
The Mumps language did this in the 60s.
plumeria 3y ago
> Building a KV store into the language is kind of nuts. Isn't it part of the standard library rather than the language? Anyways, Erlang has mnesia [0] and ETS [1]. ETS is practically a KV store, which can also be persisted to disk. [0] https://www.erlang.org/doc/man/mnesia.html https://www.erlang.org/doc/man/mnesia.html [1] https://www.erlang.org/doc/man/ets.html https://www.erlang.org/doc/man/ets.html
adamredwoods 3y ago
Why not just serialize objects to disk? I don't see how this is different from a javascript object, other than being disc-based. Unless I'm missing a detail? Also, local KV obviously won't work for compute lambdas, as the KV isn't shared across instances.

samwillis 3y ago

There is a good little overview of Deno KV here (by @simonw): https://til.simonwillison.net/deno/deno-kv https://til.simonwillison.net/deno/deno-kv So it's backed by SQLite on the OS local version. I'm intrigued if there will be a way to swap out the backend for other cloud providers.

wokwokwok 3y ago

I dunno. Is having a database that happens to be a product you sell as part of your runtime good, or are you creating some mixed incentives here? Are you a database vendor now? If not, don’t build a database. If this is a mature product that someone else is looking after, and it’s good and free and we’ll maintained like redis or SQLite, sure. …but it seems like building cloud databases is not the core competency of a group building a javascript runtime. Once money is involved, it’ll either be a distraction from their core mission, or become their core mission. A KV db you can use for quick hacks? Sure. Awesome! A scalable production database you’re selling to people? O_o why are you doing that?

frou_dh 3y ago
They need this because Deno Deploy (their Edge platform) is not a normal/traditional deployment target. If Deno were nothing but "Deno CLI" (the equivalent of the 'node' executable, that runs as a typical server process on a Unix box) then you would be right. But there are two separate Deno runtimes (CLI and Deploy).
csiegert 3y ago
Deno Inc. has two core products: The free Deno runtime and the for-profit Deno Deploy, a Deno hosting service with 35 locations around the world. The question that often popped up was where to store data. Deno Inc. provided several guides to connect to different cloud services. But they want the friction reduced to a simple `await Deno.openKv()`. Deno Inc. has enough expertise in running a global service that two other companies rely on their work to offer edge functions to their customers (Netlify and Supabase). Adding a database to the service makes sense. And to be clear, they don’t develop a brand new database. They build atop of SQLite and FoundationDB.
- wokwokwok 3y ago
  > The question that often popped up was where to store data. Deno Inc. provided several guides to connect to different cloud services. Sure. > But they want the friction reduced to a simple `await Deno.openKv()`. Do “they”? If so, who’s using it to solve that problem? …because it seems the big uses of deno deploy are not using it, fine with that and it’s pretty unclear who the “they” is in this circumstance. Still, if it’s a thin layer over foundation db or some other established database product and this is just part of the lock-in for their cloud offering, fair enough. It’s not like others (eg firebase) don’t do the same thing. The messaging “we’re building a database” and “we’re offering a hosted database service based on existing mature reliable technology” are different things though. The latter all cloud vendors do. The former is ridiculous, and it really really wasn’t clear that wasn’t what was happening.
  - electroly 3y ago
    "They" in OP's post is referring to Deno Land Inc.
suction 3y ago
[dead]
jpalomaki 3y ago
I think this kind of "batteries included" approach is interesting. My feeling is that in many cases we spend way too much time when building our platforms from various bits and pieces. Some apps and businesses would get huge productivity boost by using more opinionated platform. Not saying this is for everything and everyone, but there's so much competition on this space that I can understand why somebody would like to try something different.

ohgodplsno 3y ago

Building a KV store into the runtime, that has more features when using deno deploy is just that: vendor lock-in. It's just one more way to force you into paying for the service when you have locked yourself in and you can't use an alternative store. Deno is not and has never been an "alternative runtime" to node.

olric 3y ago
[dead]
frou_dh 3y ago
If they provide a compelling product offering, and the people understand what's available where, and choose to use it, then I guess that can be called vendor lock-in. But it's hardly nefarious.
- ohgodplsno 3y ago
  It is nefarious when you build up your entire product for years, spamming it everywhere as "node but better", yell "it's so open source too!" on every roof, only to sneakily slide in lock-in features, it's worse than being nefarious. You most likely had this plan in mind all along, and actively lied to get there. Watch as the same happens with Bun.
  - frou_dh 3y ago
    What's stopping you sticking with Deno CLI, using it for free forever, and completely ignoring anything related to Deno Deploy? In this relationship of 3 years and counting, they are the positive contributor because they give you something useful for free and you presumably haven't given them anything.
  - iudqnolq 3y ago
    In this specific case I think it's super reasonable. The KV store has two backbends: local sqlite and distributed deno cloud. If you choose the deno cloud backend you're signing up for a closed source cloud service from the start. There's no trickery.

potamic 3y ago

It's a lock-in strategy. There is zero reason for a language runtime to be opinionated about the database. All the good programming advice will tell you such a coupling is a bad idea. What if you want to move parts of your application to a different language in the future, do you need to redo the entire database? That is just nuts! Database providers, cloud application platforms should be agnostic of the language their users are using. There are many such decisions deno is making where they want to be the end-to-end service. That is not a good sign. It creates perverse incentives for the company to devise lock-in mechanisms at each stage and discourage compatibility with the rest of the programming ecosystem, perhaps even undermine the rest of the ecosystem. I would stay away for this reason.

deleted 3y ago
[deleted]
olric 3y ago
[dead]
p-e-w 3y ago
> There is zero reason for a language runtime to be opinionated about the database. The API is extremely simple. It's little more than a hash map. That's hardly opinionated. > All the good programming advice will tell you such a coupling is a bad idea. That advice is outdated. Virtually any modern application will want a database, and this API can serve as a foundation for it – a foundation that conveniently already has many of the features (esp. global replication and consistency) that you will want and that are incredibly difficult to get right if you build them yourself. Think of this as the application's "file system". PaaS today usually doesn't provide direct disk access, so this is the low-level abstraction for persistent storage that is available. An abstraction that, in many cases, will in fact be all your application ever needs.
- jabradoodle 3y ago
  I'm not familiar with the offering but find your comment confusing. It seems to suggest there isn't lock in because it's just a hashmap, but also that it is globally replicated and consistent.

thepratt 3y ago

To me this is quite similar to the Erlang VM's ETS in-memory store. There's a lot of usefulness in ETS itself, especially how it distributes inside clusters. Having an easy to use, efficient, and optimized store as part of the language can be a good enabler. Only time will tell if it is/isn't.

scotty79 3y ago

I think what's nuts is that databases data structures are not built into the language/stdlibs. I can make an array but if I want to add an index so I can access it fast then I have to jump through some inordinate hoops. I can make an in memory hashmap, but if I want to persist it and keep accessing it fast directly from persistent storage suddenly things get unnecessarily complicated. I think fast searchable file backed data structures should be primitives of any computer language and runtime.

kybernetikos 3y ago

I think almost all languages should have a database built in. Almost all programs need a way of storing and querying data but the drama required to hook up to an external db is excessive for little programs. Basic dB functionality should be as available in language standard libraries as file access.

dgellow 3y ago
What do you mean by drama to hook up external db? It’s almost always just a one liner: let conn = db.newConn(host, port, …).
- qbasic_forever 3y ago
  One line _after_ you figure out the myriad of database libraries available, half of which are abandoned, a quarter which are little toys, and if you're lucky one that has a core dev who is being paid real money to support and maintain the library.
  - dgellow 3y ago
    Ah yes, that's fair. Finding the good library to rely on can be really time consuming.

flohofwoe 3y ago

It's not really built into the language though? It's just a function on the global Deno object, similar to how there's a global 'document' object in browsers (and that's not 'built into the language' either).

hu3 3y ago
Backed into the runtime is probably what your parent meant.

zja 3y ago

Deno.dev (deploy) uses a different, proprietary runtime than Deno. I believe Ry mentioned at a recent talk that on deno.dev, the kv will be synced across all instances. Obviously this can’t be the case for stock Deno.

komon 3y ago
KV storage is SQLite, and SQLite replication and syncing using an out-of-process daemon is becoming a solved problem. For that application at least no changes to the runtime would be necessary

theGnuMe 3y ago

The Mumps language did this in the 60s.

plumeria 3y ago

> Building a KV store into the language is kind of nuts. Isn't it part of the standard library rather than the language? Anyways, Erlang has mnesia [0] and ETS [1]. ETS is practically a KV store, which can also be persisted to disk. [0] https://www.erlang.org/doc/man/mnesia.html https://www.erlang.org/doc/man/mnesia.html [1] https://www.erlang.org/doc/man/ets.html https://www.erlang.org/doc/man/ets.html

adamredwoods 3y ago

Why not just serialize objects to disk? I don't see how this is different from a javascript object, other than being disc-based. Unless I'm missing a detail? Also, local KV obviously won't work for compute lambdas, as the KV isn't shared across instances.