12 ms·

Deno Queues

351 points by 0xedb 3y ago

nmca 3y ago
I find deno to be very exciting. Viable business model, great ergonomics, glorious lack of bullshit configuration busywork.
- paxys 3y ago
  I have tried to be excited about Deno, but just don't see it. What are they doing that hasn't already been done a hundred times in the past? Deno itself is essentially Node.js + automatic TS compilation (which would otherwise have been 1 extra config file). When the project launched they made a big deal about escaping from the messy NPM ecosystem, but then ended up having to turn around and add support for it, taking that differentiator away as well. Deno Deploy is the same as dozens of other Lambda-like services out there. Their KV store and now Queues is equivalent to Redis/LevelDB/RocksDB/Dynamo/SQS or countless similar options. I have yet to come across a single feature that actually sets Deno apart from the rest. I could maybe see the appeal if you prefer to have a single company in charge of your language runtime + compute hosting + data storage, but I'd personally want to avoid that, especially when the company doesn't have a track record in it.
  - skybrian 3y ago
    I'm not that familiar with these alternatives. What's a good service provider you recommend for a persistent KV store?
    - paxys 3y ago
      - Redis - ScyllaDB - Riak - Couchbase - Cloudflare KV - Many different AWS offerings (DynamoDB, ElastiCache) - Similar alternatives from Google & Azure if you are in those ecosystems Of course there's nothing unique about a KV store. You can use literally any SQL or noSQL service out there, or set up your own in any way you want. They also all have the advantage that you will get bindings for every language, not just Deno.
  - herpdyderp 3y ago
    Deno is compatible with web code to a much deeper level than Node.js is. As the Deno website puts it: "Built with web standard APIs". This might not make a difference to most. Personally though, as someone who strives really hard to share code between frontend and backend, I'm excited for it. This is pretty much the only reason I see Deno eventually winning out over Node.js (in the long run). I don't have the time to try to port any of my meaningful projects to it right now, but I look forward to the day when I can do so.
    - paxys 3y ago
      The single piece of extra browser compatibility I have found in Deno is that it supports the fetch API, and Node 18+ has that as well. Beyond that there is really no difference between the two.
      - domh 3y ago
        Whilst I mostly agree, and Node has arguably been forced down the browser-compatible route (for stuff like fetch() and web streams). There is a bunch of other stuff that isn't in Node, or it has it's own APIs for... alert/confirm for getting user input, Web Crypto, URLPattern etc. You can see them all here: https://docs.deno.com/runtime/manual/runtime/web_platform_apis https://docs.deno.com/runtime/manual/runtime/web_platform_ap...
  - iobdas 3y ago
    I think you're underestimating just how much tooling Deno has built-in. It's not just automatic TS compilation, but formatting, linting, testing, and benchmarking at the bare minimum. I use those for almost every project. Each of those would probably be at least another configuration file. About Deno Deploy, I completely disagree with the analysis that it's just like any other lambda service. It's not just about easy deploys, the service itself is simply better. There are no cold starts and they don't just do that by keeping a vm up all the time. It's really magical. You should give it a second look.
    - mcintyre1994 3y ago
      > There are no cold starts and they don't just do that by keeping a vm up all the time. It's really magical. Do you know how they're doing that, technically? Is it running on Cloudflare under the hood, or if not does it have a similar set of tradeoffs and benefits vs Lambda? Or is it a third completely different way with different tradeoffs/benefits?
  - jallmann 3y ago
    I use Deno because I couldn't be bothered to figure out the Node ecosystem and how to actually wire together the Typescript compiler, package json, builders and bundlers, etc. I just do not care and the mess of tooling is frankly exhausting to wade through. Deno Just Works for me. `deno run file.ts` and you are good to go. Not to say it's been 100% smooth sailing; I have hit a few rough patches with deployment into my own infrastructure (some within Deno and its dependency management, some within libs like sqlite3) but overall I am happy to not deal with nodejs.
    - meowtimemania 3y ago
      Even after you figure out typescript with node, then you need to figure out sorucemaps otherwise your stack traces won’t be very useful
  - meowtimemania 3y ago
    With all those alternatives you need to spend time installing libraries, then for each environment you need to create the db, and setup environment variables. You’ll also probably want to figure out a local environment. With deno all that setup/configuration is handled by deno. There’s obvious trade offs but I think what deno is doing is pretty cool.
- hobofan 3y ago
  What makes you believe that Deno has a viable business model?
catlover76 3y ago
I don't want my programming language to "implement" task queues and charge me for it. What is this edit: people seem to be getting hung up on the "runtime" v "programming language" distinction. I'm not sure why--it's weird to me that this is "part of the language + runtime" at all. Clearly, reasonable people can disagree about that, but hopefully on points more substantive than pedantic If tomorrow, someone started a for-profit company making a faster Python runtime and introduced features like this, it would be weird, and it would feel as pointless to me as Deno is
- snapetom 3y ago
  It's an ad.
- vorticalbox 3y ago
  You're getting charged for the replication of deno kv across all regions when you use deno.deploy. KV is free to use locally as I assume this will be too.
  - shafyy 3y ago
    Sure, but genuine question: How easy it is to run my own infrastructure for Deno Queue, and, crucially, how can I know that it will still be as easy and transparent in 5 years from now when the VCs want to see some $$$. That the company developing Deno and the services that monetize Deno is the same is a major conflict of interest. And if history is of any guide, in the end economic incentives always win, no matter how good-hearted people are.
    - LostLocalMan 3y ago
      If Deno doesn't get adequate funding, the company developing Deno will not exist 5 years from now.
    - skybrian 3y ago
      When imagining future scenarios, maybe don't get too fixated on one? Another possibility is that competitive service providers will implement the same services. (It didn't really happen with App Engine, but this seems like a cleaner API?)
nmca 3y ago
(on the off chance the devs read this) I can see pain in the future for the currently excellent KV ergonomics around access control --- is the solution just "implement it in userland and don't write bugs" or is there anything planned?
- kwhinnery 3y ago
  User-level access control (like ACLs for authenticated users) or something else? Feel free to share longer form thoughts here and I can make sure we keep this concern in mind.
  - nmca 3y ago
    systems level - so if, for example, I can have two services or serverless functions, one of which can access the world and the other of which cannot see, for example, the pii tables. (I'm not a production customer, and I haven't thought through how this ought to work in deno-deploy-land, I've just seen a lot of painful ACL set ups. But I also like knowing that lambda function foo written by the intern can't read every DB table)
lopatin 3y ago
> Leveraging public cloud infrastructure has traditionally demanded sifting through layers of boilerplate code and intricate configurations, often monopolizing a significant chunk of the developer’s time and energy. I don't buy this line of reasoning. At the end of the day we are building infrastructure that needs to be reliable. Spending 30 minutes to set up an SQS queue (proven technology) doesn't sound as bad as putting all my trust in a toy queue that Deno built "on top of SQLite / FoundationDB". Is the 30 minute setup cost and added "developer experience" really worth the risk? How often are you setting up queues anyway.
- falcolas 3y ago
  > How often are you setting up queues anyway. Very infrequently. Even more infrequently as we move to IAC and use terraform or even cloudformation.
- asim 3y ago
  Many people don't have experience with AWS or SQS and even those that do will still need to think about it. Deno is taking a batteries included approach which means the Dev can just import something and go. If someone provided equivalent libraries for the cloud then I'm sure people would adopt them. In the case of Deno I'm sure they've identified a common problem many Devs in their ecosystem have and are tackling it by bundling in a good solution.
  - leptons 3y ago
    If Deno is only going after amateur devs by implementing simplistic features like this, then it will remain an amateur product. AWS/SQS may seem complex to some, but they are very powerful. Maybe Deno is doing what Apple did in the 80's by putting free Apple computers in classrooms - try to get new users hooked into their system and then they'll use it for life. Anyone doing anything serious with cloud computing is already using AWS, or should be. I would rather put in the time to learn AWS (which is entirely free in many small use cases), so I won't build stuff in a toy system and then later realize I need something better.
    - 3y ago
lucasyvas 3y ago
I actually do quite like the overall idea of having a unified API at the language level that a (configurable) runtime can provide implementations for. Sure, in some cases an API might not be easy to abstract over due to underlying concerns, but in a lot of cases it does work. It does show a level of understanding by the language creator that they know how the language can and probably should be used - I like how KV defaults to SQLite locally and takes on new meaning in a hosted environment. I haven't seen, but as long as you can actually override the behavior to use your own KV/queue technology (so long as it satisfies the interface) I see absolutely no issue with this. There is a huge benefit where the code always looks the same for common things we all need to do. I wouldn't mind seeing other languages take an overall stab at this so there is some choice other than JavaScript.
sebringj 3y ago
I'm currently using Deno deploy and found it to be fantastically performant and dumb simple for my lone wolf project. I'm experienced in AWS development in larger teams and it is nice to see a move away from complexity for a change where you can easily set something up without having to think about setting it up at all. The DNS stuff was just dead simple as well and automatic ssl certs was super nice. I have 0 complaints for what this is trying to be and am excited for the road map.
tmikaeld 3y ago
Why not instructions on how to use it with FoundationDB so that we can host it on our company infrastructure?
- meowtimemania 3y ago
  I think you can add your own implementation to Deno.openKv so that you can use your own companies infra.
  - tmikaeld 3y ago
    That requires knowledge of rust and how deno work internally, it would probably be far from production ready…
- baq 3y ago
  How do you think they make money?
  - tmikaeld 3y ago
    It's MIT licensed, if anyone wanted to do that they'd just patch deno with it anyway.
zoogeny 3y ago
I think what Deno is aiming for here is actually very forward thinking. I'm using Go for the first time this year and some aspects of the language are very C like. Of the list of things that isn't C like (e.g. GC allowing one to return what look like stack allocated pointers from functions) there is the obvious inclusion of things like `map[string]string`. I bring this up because it struck me that inventing a language in 2023+, it would seem almost insane not to have built in syntax for the language to handle map types. And so it seems logical that a web-server focused eco-system would start to garner libraries and even language syntax (maybe one day) for the kind of primitives we frequently use on web-servers. I mean, I can't even recall the last time I worked on a distributed web server that didn't have a KV store as a cache, or a locking mechanism, or even an ad-hoc queue. A distributed system without a KV store feels vaguely isomorphic to a computer language without a map type. It is such a Swiss army knife kind of technology. One potential issue is that Deno is going alone down this path. Currently, I don't feel confident that the new features they are adding will be available on competing platforms. Even if they open source the API (it is on a `Deno` namespace), I'm not sure it will just work on AWS if I wanted to switch out FoundationDB for e.g. Redis. For that reason, I feel I want to avoid Deno. Even if the syntax and exposed features are really attractive, I'm worried about becoming locked in and then having to do a lot of surgery to code to make it deployable on multiple cloud infrastructures. That is a requirement from a lot of clients. E.g. maybe I want to sell to Oracle or Salesforce one day but they mandate I have to run my systems on-prem. I'm now on the hook trying to figure out how to adapt whatever available KV store they have to the features I'm using from the `Deno` package. It is a double edged sword. Maybe they will succeed in pushing forward this vision to a broader audience. For now I'll probably remain cautious.
- paxys 3y ago
  I disagree. Tying a language runtime to a specific KV interface which is tied to a specific hosted service is the opposite of forward thinking. In fact the tech industry has made a lot of progress away from vendor locked-in stacks, and this just reminds me of those. What is really the difference between Deno.KV being shipped as part of the runtime vs adding an `import KV` statement at the top of the file (which could come from Deno or wherever else you want)? And what are the chances that the Deno team is going to ship bindings for any language besides their own? If Google started adding Google Cloud specific primitives natively to Go would you call that forward thinking as well?
abdellah123 3y ago
I'm excited about the recent Jupyter support and this queues.. Super cool stuff. But I won't write servers in Deno and put them to production in my own servers. (I don't like serverless) You see, Deno(and Bun)'s business model is built around serverless (Deno deploy). And Deno deploy is ANOTHER runtime. It's clear with KV and queues... Locally and if you self host, you have a barely working version backed by another tech... This means you're self hosting a version of Deno that's different from the majority of the Deno customer base. How many people will use Deno to run servers on their own hardware? Since that number is low, then I'd rather use Node.
trescenzi 3y ago
I wonder how this differs from BroadcastChannel[1] which Deno already implements and DenoDeploy seems to support. [1]: https://docs.deno.com/deploy/api/runtime-broadcast-channel https://docs.deno.com/deploy/api/runtime-broadcast-channel
- msoad 3y ago
  persistance?
  - trescenzi 3y ago
    Yea I think that's the answer. The queues are deeply integrated into a datastore so you have a trace of what happened and you can do it all atomically.
EGreg 3y ago
Isn't deno running locally no your self-hosted instance? How are they charging? Enqueuing a Message: Each enqueue action translates into a KV write operation. Receiving a Message: Every received message entails a KV write, and a single request charge. I would be really curious to know. Are they only talking about Deno Deploy in the cloud, with FoundationDB? Is this "open core"?
brundolf 3y ago
> Leveraging public cloud infrastructure has traditionally demanded sifting through layers of boilerplate code and intricate configurations, often monopolizing a significant chunk of the developer’s time and energy. Our goal is to distill these intricacies into user-friendly primitives, enabling developers to design, refine, and launch their projects with unmatched speed. I really like this
math_dandy 3y ago
Under the hood, does DenoKV implement an abstraction that specializes locally to SQLite and to Deno Deploy in the cloud? Or are DenoKV local and DenoKV cloud two separate products with a common public API?
- vorticalbox 3y ago
  Code was the api is the same. When you use it locally the database is only on your device however all deno processes can access the same dB so you can use it to pass data around just like you can for localStorage that deno also supports. When you use it in the cloud (deno deploy) then saves are replicated across regions. Deploy is server less. KV is simply a wrapper around sqlite with that you get atomic transactions that you wouldn't with local storage
- dkn 3y ago
  When deployed to Deno Deploy, KV is a wrapper around FoundationDB, selected for its use in the distributed/edge environments that Deno operates in. When used locally, it's a wrapper around SQLite. The API is standardized for both.
ranting-moth 3y ago
Pricing? I thought Deno was some sort of node.js replacement. What am I missing and can I use this either locally and/or self hosted without paying for it?
- rahkiin 3y ago
  The article says development (locally) it uses SQLite. On deployment to Deno it is part of their cloud offering
- simonw 3y ago
  This is the same trick as the rest of Deno KV: the open source version uses SQLite, but when you deploy to their cloud product you get Foundation DB (proprietary) instead: > Since Queues are built on Deno KV, it uses SQLite when running locally and FoundationDB when running on Deno Deploy for maximum availability and throughput. I wrote a bit more about this pattern here: https://til.simonwillison.net/deno/deno-kv https://til.simonwillison.net/deno/deno-kv
  - tra3 3y ago
    Simon, I love reading your TILs. What's your process? Is there a TIL on writing TILs? :)
    - simonw 3y ago
      Partly it's about building habits around them. I watch out for any time I have to spend half an hour or more figuring something out - that's generally a sign that it's worth tidying up my notes into a TIL, since the internet is clearly missing that piece of information! I keep very detailed notes on everything I'm doing already - either as a VS Code scratch document or Apple Notes or often a GitHub Issues thread. A lot of my TILs start by me pasting those notes into a Markdown file and tidying them up.
  - hueho 3y ago
    Just as an side note, not relevant to anything in particular, Foundation DB itself is open-source (https://www.foundationdb.org/ https://www.foundationdb.org/), but the integration layer used by Deno to make it the backend for Deno KV is not. Although, from reading the Foundation DB docs and checking the Deno KV API, I honestly suspect it is a thin layer. Self-hosting FDB is somewhat inscrutable though, so their value add is in not having to handle infrastructure while being backed by FDB.
ankit70 3y ago
I never liked Agenda (Persisted delayed task queue using MongoDB). This example[1] using Deno Queues & KV is easy enough to be a replacement for Agenda.
simonw 3y ago
I dug into the internals of the local, SQLite version of this just now and wrote up some notes here: https://til.simonwillison.net/deno/deno-kv#user-content-deno-queues-27th-september-2023 https://til.simonwillison.net/deno/deno-kv#user-content-deno... The most interesting detail is probably the schema they're using for that: CREATE TABLE queue ( ts integer not null, id text not null, data blob not null, backoff_schedule text not null, keys_if_undelivered blob not null, primary key (ts, id) ); CREATE TABLE queue_running( deadline integer not null, id text not null, data blob not null, backoff_schedule text not null, keys_if_undelivered blob not null, primary key (deadline, id) ); CREATE INDEX kv_expiration_ms_idx on kv (expiration_ms);
- deleted 3y ago
  [deleted]
- londons_explore 3y ago
  Is it not normally bad practice to have 2 tables with basically identical fields and move rows between them? Isn't that exactly what indexes were designed for?
  - postalrat 3y ago
    If one table had millions of rows and the other only a few dozen and you are querying the few dozen often then it could make sense to have two tables. Different database treat indexes with sparse data differently.
    - simonw 3y ago
      Yeah, that seems like the reason for this design decision here. The queue_running table sits empty the majority of the time, very occasionally gaining a row or two just while they are being processed.
    - giovannibonetti 3y ago
      Partial indexes solve exactly this problem. You add a WHERE condition that restricts the index to a subset of the whole table.
mindcrash 3y ago
I don't know if Ryan and the rest of the Deno team are lurking here, but given "Anchored on the robust capabilities of FoundationDB" what I think would be absolutely fricking cool is to get full access to the underlying FoundationDB engine, which would surface the "model any data model you want the way you want it" capabilities of FoundationDB directly to Deno developers. In case anyone wonders what this would make possible out of the box: https://apple.github.io/foundationdb/design-recipes.html https://apple.github.io/foundationdb/design-recipes.html
rubenfiszel 3y ago
This is amazing syntax. If you are looking for a self-hostable version of this, that can run deno but also python, go, and bash, with primitives similar to airflow and more (retries, cache, suspend, approval steps), check out windmill: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill
- nesarkvechnep 3y ago
  Amazing syntax? Isn’t this standard JS chaining?
  - rubenfiszel 3y ago
    Exactly! It looks and feel like normal JS and hides all the complexity behind it which is the print of great syntax.
    - dsQTbR7Y5mRHnZv 3y ago
      Perhaps "API design" would be more apt than syntax here
welder 3y ago
Very timely... I'm almost finished porting WakaQ (my custom background task queue to replace Celery) from Python into TypeScript to power background tasks for a new Next.js website. I'm using T3 not Deno so I wouldn't have used Deno Queues, but it's a core part of every web app so makes sense they would build this into their stack. Some questions: * Looks very powerful, but at the core is it just one single queue? Does Deno.openKv() create a new queue or re-use the same queue every time it's called? * Usually I need multiple queues of different priority, so when bottlenecks happen the highest priority jobs run first. * Maybe they guarantee infinite worker capacity so you don't need queue priorities? No bottlenecks if you can pay for the jobs you enqueue?
simonw 3y ago
Unless I'm missing something, it looks like each Deno.openKv() instance only gets a single queue. For the local version you could get multiple queues by calling Deno.openKv("db-2.db") with different SQLite file paths each time, but that feels like a lot of overhead for a pretty common need. I guess this is a Deno architectural style thing - maybe when you build complex apps on Deno it's expected that you'll have a microservice style architecture where lots of different scripts work together, each of them with their own KV store and hence their own queue?
- gabeidx 3y ago
  You are right, there's only a single queue at the moment. One of the core devs has confirmed this on their discord: https://discord.com/channels/684898665143206084/1156714282533265459/1156750131014668348 https://discord.com/channels/684898665143206084/115671428253... Quoting here: > Correct. Currently a single queue is supported. You could multiplex multiple types of messages on the single queue though.
pier25 3y ago
Deno is amazing for backend. Is there any full stack solution other than Fresh?
dackerlunghack 3y ago
With grand promises rode the ploy Venture capital, ahoy! We make a lovable product to capture audience Vendor lock them at the earliest convenience Hey look at our awesome developer experience Add ANSI and emojis and comments saying it’s genius Some next level tech, plebs just don’t get it Next round of funding, we’ve already spent it I took the big dough, got puppets and framed it R-O-I is bad though but layoffs gon’ fix it Never mind the drama *cough* are ya gon’ install it Run it, ship it, kill yourself with working on it Entangle your business with it Tell the client to deal with it Bring it to the big wig Get ’em to sign off on it Crown your own sh*t Dogfood is tasty, ain’t it
- meowtimemania 3y ago
  chat gpt?
  - dackerlunghack 3y ago
    Human brains proudly present Real poetry? Grouchy resent! My rhyme is not gradient descent After all, it is barely nascent
pests 3y ago
I feel its odd for a compiler/language toolchain to have a cloud offering. I understand the motivations but 20% of the article was about their Deno Deploy product and how to calculate API costs... I understand its only when using their cloud and there is a local implementation, but I just couldn't imagine using a feature in LLVM, for example, where if you were to upload the binary to a specific cloud it would use a different implmentation that costs behind the scenes. It just feels odd to me. I guess it's kind of cool. Maybe different clouds can implement the underlying KV and then this becomes cross-cloud with different underlying benefits without changing the code. But I'm not entirely sure how I feel about this.
- wmf 3y ago
  These features pay for Deno. If they don't exist, Deno doesn't exist.
  - pests 3y ago
    I agree projects need funding and developers need to be paid... Deno was released 4 years before Deno Deploy and existed just fine.
- billywhizz 3y ago
  there is also a b2b game in play here by deno. i'd imagine a FaaS JS/TS only platform powered by Deno runtime is attractive to a lot of already existing web services who want to expose the ability to write webhooks/event-based functions in their service offerings? https://deno.com/blog/netlify-edge-functions-on-deno-deploy https://deno.com/blog/netlify-edge-functions-on-deno-deploy
londons_explore 3y ago
> at least once semantics. > user code. A past life has taught me that users will never properly understand at-least-once semantics. Anytime you redeliver messages, you will get a flurry of user complaints and breakage. Either you do the impossible and invent a way to do exactly-once semantics. Or you should always redeliver 0.1% of all messages, just so that users don't come to depend on messages being delivered once.
- chewbacha 3y ago
  Early SQS basically did this. I'm not sure if this was intentional or not but it quickly taught me to respect redelivery.
- plaguuuuuu 3y ago
  If messages need to be idempotent, I definitely recommend creating integration tests to ensure this from the sending side :-)
endisneigh 3y ago
Can we deploy both deno kv and queues on our own foundation db cluster?
spion 3y ago
I like how updating the data and enqueueing a message can be part of a single transaction. That's indeed pretty powerful. A bit of an aside, but not sure we want at-most-once delivery for email.
- crabmusket 3y ago
  Didn't the article say at-LEAST-once?
  - spion 3y ago
    Yeah it did. My bad writing "at most". But the problem is with "at least once" (email sending cannot be made idempotent AFAIK)
hintymad 3y ago
What happens to the mantra that "database as a queue is harmful"[1]? Of course Amazon teams internally use DynamoDB for all kinds of queues, which implies that we get to do a lot of things easily if we get a super robust storage solution. So, I guess the question is really whether FoundationDB can be used as a backend for queues. [1] https://www.google.com/search?q=Use+database+as+a+queue+is+harmful https://www.google.com/search?q=Use+database+as+a+queue+is+h...
robertwt7 3y ago
How is this different than using RabbitMQ? Can anyone tell me what’s the benefit of choosing this one over some other message queues or even kafka?
- ollien 3y ago
  Well, a couple things. For one, this is just a simple queue with no routing capabilities. It's a much simpler tool. For two, and more importantly, it's built into your application and introduces no additional operational dependency. You could probably achieve a pretty similar API with a custom AMQP library, though!
Etheryte 3y ago
On the topic of Deno's cloud deployment and Deno KV, does it offer a reasonable user story for caching? Or is that something you're expected to handle yourself (for now)?

351 points by 0xedb 3y ago

nmca 3y ago

I find deno to be very exciting. Viable business model, great ergonomics, glorious lack of bullshit configuration busywork.

paxys 3y ago
I have tried to be excited about Deno, but just don't see it. What are they doing that hasn't already been done a hundred times in the past? Deno itself is essentially Node.js + automatic TS compilation (which would otherwise have been 1 extra config file). When the project launched they made a big deal about escaping from the messy NPM ecosystem, but then ended up having to turn around and add support for it, taking that differentiator away as well. Deno Deploy is the same as dozens of other Lambda-like services out there. Their KV store and now Queues is equivalent to Redis/LevelDB/RocksDB/Dynamo/SQS or countless similar options. I have yet to come across a single feature that actually sets Deno apart from the rest. I could maybe see the appeal if you prefer to have a single company in charge of your language runtime + compute hosting + data storage, but I'd personally want to avoid that, especially when the company doesn't have a track record in it.
- skybrian 3y ago
  I'm not that familiar with these alternatives. What's a good service provider you recommend for a persistent KV store?
  - paxys 3y ago
    - Redis - ScyllaDB - Riak - Couchbase - Cloudflare KV - Many different AWS offerings (DynamoDB, ElastiCache) - Similar alternatives from Google & Azure if you are in those ecosystems Of course there's nothing unique about a KV store. You can use literally any SQL or noSQL service out there, or set up your own in any way you want. They also all have the advantage that you will get bindings for every language, not just Deno.
- herpdyderp 3y ago
  Deno is compatible with web code to a much deeper level than Node.js is. As the Deno website puts it: "Built with web standard APIs". This might not make a difference to most. Personally though, as someone who strives really hard to share code between frontend and backend, I'm excited for it. This is pretty much the only reason I see Deno eventually winning out over Node.js (in the long run). I don't have the time to try to port any of my meaningful projects to it right now, but I look forward to the day when I can do so.
  - paxys 3y ago
    The single piece of extra browser compatibility I have found in Deno is that it supports the fetch API, and Node 18+ has that as well. Beyond that there is really no difference between the two.
    - domh 3y ago
      Whilst I mostly agree, and Node has arguably been forced down the browser-compatible route (for stuff like fetch() and web streams). There is a bunch of other stuff that isn't in Node, or it has it's own APIs for... alert/confirm for getting user input, Web Crypto, URLPattern etc. You can see them all here: https://docs.deno.com/runtime/manual/runtime/web_platform_apis https://docs.deno.com/runtime/manual/runtime/web_platform_ap...
- iobdas 3y ago
  I think you're underestimating just how much tooling Deno has built-in. It's not just automatic TS compilation, but formatting, linting, testing, and benchmarking at the bare minimum. I use those for almost every project. Each of those would probably be at least another configuration file. About Deno Deploy, I completely disagree with the analysis that it's just like any other lambda service. It's not just about easy deploys, the service itself is simply better. There are no cold starts and they don't just do that by keeping a vm up all the time. It's really magical. You should give it a second look.
  - mcintyre1994 3y ago
    > There are no cold starts and they don't just do that by keeping a vm up all the time. It's really magical. Do you know how they're doing that, technically? Is it running on Cloudflare under the hood, or if not does it have a similar set of tradeoffs and benefits vs Lambda? Or is it a third completely different way with different tradeoffs/benefits?
- jallmann 3y ago
  I use Deno because I couldn't be bothered to figure out the Node ecosystem and how to actually wire together the Typescript compiler, package json, builders and bundlers, etc. I just do not care and the mess of tooling is frankly exhausting to wade through. Deno Just Works for me. `deno run file.ts` and you are good to go. Not to say it's been 100% smooth sailing; I have hit a few rough patches with deployment into my own infrastructure (some within Deno and its dependency management, some within libs like sqlite3) but overall I am happy to not deal with nodejs.
  - meowtimemania 3y ago
    Even after you figure out typescript with node, then you need to figure out sorucemaps otherwise your stack traces won’t be very useful
- meowtimemania 3y ago
  With all those alternatives you need to spend time installing libraries, then for each environment you need to create the db, and setup environment variables. You’ll also probably want to figure out a local environment. With deno all that setup/configuration is handled by deno. There’s obvious trade offs but I think what deno is doing is pretty cool.
hobofan 3y ago
What makes you believe that Deno has a viable business model?

catlover76 3y ago

I don't want my programming language to "implement" task queues and charge me for it. What is this edit: people seem to be getting hung up on the "runtime" v "programming language" distinction. I'm not sure why--it's weird to me that this is "part of the language + runtime" at all. Clearly, reasonable people can disagree about that, but hopefully on points more substantive than pedantic If tomorrow, someone started a for-profit company making a faster Python runtime and introduced features like this, it would be weird, and it would feel as pointless to me as Deno is

snapetom 3y ago
It's an ad.
vorticalbox 3y ago
You're getting charged for the replication of deno kv across all regions when you use deno.deploy. KV is free to use locally as I assume this will be too.
- shafyy 3y ago
  Sure, but genuine question: How easy it is to run my own infrastructure for Deno Queue, and, crucially, how can I know that it will still be as easy and transparent in 5 years from now when the VCs want to see some $$$. That the company developing Deno and the services that monetize Deno is the same is a major conflict of interest. And if history is of any guide, in the end economic incentives always win, no matter how good-hearted people are.
  - LostLocalMan 3y ago
    If Deno doesn't get adequate funding, the company developing Deno will not exist 5 years from now.
  - skybrian 3y ago
    When imagining future scenarios, maybe don't get too fixated on one? Another possibility is that competitive service providers will implement the same services. (It didn't really happen with App Engine, but this seems like a cleaner API?)

nmca 3y ago

(on the off chance the devs read this) I can see pain in the future for the currently excellent KV ergonomics around access control --- is the solution just "implement it in userland and don't write bugs" or is there anything planned?

kwhinnery 3y ago
User-level access control (like ACLs for authenticated users) or something else? Feel free to share longer form thoughts here and I can make sure we keep this concern in mind.
- nmca 3y ago
  systems level - so if, for example, I can have two services or serverless functions, one of which can access the world and the other of which cannot see, for example, the pii tables. (I'm not a production customer, and I haven't thought through how this ought to work in deno-deploy-land, I've just seen a lot of painful ACL set ups. But I also like knowing that lambda function foo written by the intern can't read every DB table)

lopatin 3y ago

> Leveraging public cloud infrastructure has traditionally demanded sifting through layers of boilerplate code and intricate configurations, often monopolizing a significant chunk of the developer’s time and energy. I don't buy this line of reasoning. At the end of the day we are building infrastructure that needs to be reliable. Spending 30 minutes to set up an SQS queue (proven technology) doesn't sound as bad as putting all my trust in a toy queue that Deno built "on top of SQLite / FoundationDB". Is the 30 minute setup cost and added "developer experience" really worth the risk? How often are you setting up queues anyway.

falcolas 3y ago
> How often are you setting up queues anyway. Very infrequently. Even more infrequently as we move to IAC and use terraform or even cloudformation.
asim 3y ago
Many people don't have experience with AWS or SQS and even those that do will still need to think about it. Deno is taking a batteries included approach which means the Dev can just import something and go. If someone provided equivalent libraries for the cloud then I'm sure people would adopt them. In the case of Deno I'm sure they've identified a common problem many Devs in their ecosystem have and are tackling it by bundling in a good solution.
- leptons 3y ago
  If Deno is only going after amateur devs by implementing simplistic features like this, then it will remain an amateur product. AWS/SQS may seem complex to some, but they are very powerful. Maybe Deno is doing what Apple did in the 80's by putting free Apple computers in classrooms - try to get new users hooked into their system and then they'll use it for life. Anyone doing anything serious with cloud computing is already using AWS, or should be. I would rather put in the time to learn AWS (which is entirely free in many small use cases), so I won't build stuff in a toy system and then later realize I need something better.
  - 3y ago

lucasyvas 3y ago

I actually do quite like the overall idea of having a unified API at the language level that a (configurable) runtime can provide implementations for. Sure, in some cases an API might not be easy to abstract over due to underlying concerns, but in a lot of cases it does work. It does show a level of understanding by the language creator that they know how the language can and probably should be used - I like how KV defaults to SQLite locally and takes on new meaning in a hosted environment. I haven't seen, but as long as you can actually override the behavior to use your own KV/queue technology (so long as it satisfies the interface) I see absolutely no issue with this. There is a huge benefit where the code always looks the same for common things we all need to do. I wouldn't mind seeing other languages take an overall stab at this so there is some choice other than JavaScript.

sebringj 3y ago

I'm currently using Deno deploy and found it to be fantastically performant and dumb simple for my lone wolf project. I'm experienced in AWS development in larger teams and it is nice to see a move away from complexity for a change where you can easily set something up without having to think about setting it up at all. The DNS stuff was just dead simple as well and automatic ssl certs was super nice. I have 0 complaints for what this is trying to be and am excited for the road map.

tmikaeld 3y ago

Why not instructions on how to use it with FoundationDB so that we can host it on our company infrastructure?

meowtimemania 3y ago
I think you can add your own implementation to Deno.openKv so that you can use your own companies infra.
- tmikaeld 3y ago
  That requires knowledge of rust and how deno work internally, it would probably be far from production ready…
baq 3y ago
How do you think they make money?
- tmikaeld 3y ago
  It's MIT licensed, if anyone wanted to do that they'd just patch deno with it anyway.

zoogeny 3y ago

I think what Deno is aiming for here is actually very forward thinking. I'm using Go for the first time this year and some aspects of the language are very C like. Of the list of things that isn't C like (e.g. GC allowing one to return what look like stack allocated pointers from functions) there is the obvious inclusion of things like `map[string]string`. I bring this up because it struck me that inventing a language in 2023+, it would seem almost insane not to have built in syntax for the language to handle map types. And so it seems logical that a web-server focused eco-system would start to garner libraries and even language syntax (maybe one day) for the kind of primitives we frequently use on web-servers. I mean, I can't even recall the last time I worked on a distributed web server that didn't have a KV store as a cache, or a locking mechanism, or even an ad-hoc queue. A distributed system without a KV store feels vaguely isomorphic to a computer language without a map type. It is such a Swiss army knife kind of technology. One potential issue is that Deno is going alone down this path. Currently, I don't feel confident that the new features they are adding will be available on competing platforms. Even if they open source the API (it is on a `Deno` namespace), I'm not sure it will just work on AWS if I wanted to switch out FoundationDB for e.g. Redis. For that reason, I feel I want to avoid Deno. Even if the syntax and exposed features are really attractive, I'm worried about becoming locked in and then having to do a lot of surgery to code to make it deployable on multiple cloud infrastructures. That is a requirement from a lot of clients. E.g. maybe I want to sell to Oracle or Salesforce one day but they mandate I have to run my systems on-prem. I'm now on the hook trying to figure out how to adapt whatever available KV store they have to the features I'm using from the `Deno` package. It is a double edged sword. Maybe they will succeed in pushing forward this vision to a broader audience. For now I'll probably remain cautious.

paxys 3y ago
I disagree. Tying a language runtime to a specific KV interface which is tied to a specific hosted service is the opposite of forward thinking. In fact the tech industry has made a lot of progress away from vendor locked-in stacks, and this just reminds me of those. What is really the difference between Deno.KV being shipped as part of the runtime vs adding an `import KV` statement at the top of the file (which could come from Deno or wherever else you want)? And what are the chances that the Deno team is going to ship bindings for any language besides their own? If Google started adding Google Cloud specific primitives natively to Go would you call that forward thinking as well?

abdellah123 3y ago

I'm excited about the recent Jupyter support and this queues.. Super cool stuff. But I won't write servers in Deno and put them to production in my own servers. (I don't like serverless) You see, Deno(and Bun)'s business model is built around serverless (Deno deploy). And Deno deploy is ANOTHER runtime. It's clear with KV and queues... Locally and if you self host, you have a barely working version backed by another tech... This means you're self hosting a version of Deno that's different from the majority of the Deno customer base. How many people will use Deno to run servers on their own hardware? Since that number is low, then I'd rather use Node.

trescenzi 3y ago

I wonder how this differs from BroadcastChannel[1] which Deno already implements and DenoDeploy seems to support. [1]: https://docs.deno.com/deploy/api/runtime-broadcast-channel https://docs.deno.com/deploy/api/runtime-broadcast-channel

msoad 3y ago
persistance?
- trescenzi 3y ago
  Yea I think that's the answer. The queues are deeply integrated into a datastore so you have a trace of what happened and you can do it all atomically.

EGreg 3y ago

Isn't deno running locally no your self-hosted instance? How are they charging? Enqueuing a Message: Each enqueue action translates into a KV write operation. Receiving a Message: Every received message entails a KV write, and a single request charge. I would be really curious to know. Are they only talking about Deno Deploy in the cloud, with FoundationDB? Is this "open core"?

brundolf 3y ago

> Leveraging public cloud infrastructure has traditionally demanded sifting through layers of boilerplate code and intricate configurations, often monopolizing a significant chunk of the developer’s time and energy. Our goal is to distill these intricacies into user-friendly primitives, enabling developers to design, refine, and launch their projects with unmatched speed. I really like this

math_dandy 3y ago

Under the hood, does DenoKV implement an abstraction that specializes locally to SQLite and to Deno Deploy in the cloud? Or are DenoKV local and DenoKV cloud two separate products with a common public API?

vorticalbox 3y ago
Code was the api is the same. When you use it locally the database is only on your device however all deno processes can access the same dB so you can use it to pass data around just like you can for localStorage that deno also supports. When you use it in the cloud (deno deploy) then saves are replicated across regions. Deploy is server less. KV is simply a wrapper around sqlite with that you get atomic transactions that you wouldn't with local storage
dkn 3y ago
When deployed to Deno Deploy, KV is a wrapper around FoundationDB, selected for its use in the distributed/edge environments that Deno operates in. When used locally, it's a wrapper around SQLite. The API is standardized for both.

ranting-moth 3y ago

Pricing? I thought Deno was some sort of node.js replacement. What am I missing and can I use this either locally and/or self hosted without paying for it?

rahkiin 3y ago
The article says development (locally) it uses SQLite. On deployment to Deno it is part of their cloud offering
simonw 3y ago
This is the same trick as the rest of Deno KV: the open source version uses SQLite, but when you deploy to their cloud product you get Foundation DB (proprietary) instead: > Since Queues are built on Deno KV, it uses SQLite when running locally and FoundationDB when running on Deno Deploy for maximum availability and throughput. I wrote a bit more about this pattern here: https://til.simonwillison.net/deno/deno-kv https://til.simonwillison.net/deno/deno-kv
- tra3 3y ago
  Simon, I love reading your TILs. What's your process? Is there a TIL on writing TILs? :)
  - simonw 3y ago
    Partly it's about building habits around them. I watch out for any time I have to spend half an hour or more figuring something out - that's generally a sign that it's worth tidying up my notes into a TIL, since the internet is clearly missing that piece of information! I keep very detailed notes on everything I'm doing already - either as a VS Code scratch document or Apple Notes or often a GitHub Issues thread. A lot of my TILs start by me pasting those notes into a Markdown file and tidying them up.
- hueho 3y ago
  Just as an side note, not relevant to anything in particular, Foundation DB itself is open-source (https://www.foundationdb.org/ https://www.foundationdb.org/), but the integration layer used by Deno to make it the backend for Deno KV is not. Although, from reading the Foundation DB docs and checking the Deno KV API, I honestly suspect it is a thin layer. Self-hosting FDB is somewhat inscrutable though, so their value add is in not having to handle infrastructure while being backed by FDB.

ankit70 3y ago

I never liked Agenda (Persisted delayed task queue using MongoDB). This example[1] using Deno Queues & KV is easy enough to be a replacement for Agenda.

simonw 3y ago

I dug into the internals of the local, SQLite version of this just now and wrote up some notes here: https://til.simonwillison.net/deno/deno-kv#user-content-deno-queues-27th-september-2023 https://til.simonwillison.net/deno/deno-kv#user-content-deno... The most interesting detail is probably the schema they're using for that: CREATE TABLE queue ( ts integer not null, id text not null, data blob not null, backoff_schedule text not null, keys_if_undelivered blob not null, primary key (ts, id) ); CREATE TABLE queue_running( deadline integer not null, id text not null, data blob not null, backoff_schedule text not null, keys_if_undelivered blob not null, primary key (deadline, id) ); CREATE INDEX kv_expiration_ms_idx on kv (expiration_ms);

deleted 3y ago
[deleted]
londons_explore 3y ago
Is it not normally bad practice to have 2 tables with basically identical fields and move rows between them? Isn't that exactly what indexes were designed for?
- postalrat 3y ago
  If one table had millions of rows and the other only a few dozen and you are querying the few dozen often then it could make sense to have two tables. Different database treat indexes with sparse data differently.
  - simonw 3y ago
    Yeah, that seems like the reason for this design decision here. The queue_running table sits empty the majority of the time, very occasionally gaining a row or two just while they are being processed.
  - giovannibonetti 3y ago
    Partial indexes solve exactly this problem. You add a WHERE condition that restricts the index to a subset of the whole table.

mindcrash 3y ago

I don't know if Ryan and the rest of the Deno team are lurking here, but given "Anchored on the robust capabilities of FoundationDB" what I think would be absolutely fricking cool is to get full access to the underlying FoundationDB engine, which would surface the "model any data model you want the way you want it" capabilities of FoundationDB directly to Deno developers. In case anyone wonders what this would make possible out of the box: https://apple.github.io/foundationdb/design-recipes.html https://apple.github.io/foundationdb/design-recipes.html

rubenfiszel 3y ago

This is amazing syntax. If you are looking for a self-hostable version of this, that can run deno but also python, go, and bash, with primitives similar to airflow and more (retries, cache, suspend, approval steps), check out windmill: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill

nesarkvechnep 3y ago
Amazing syntax? Isn’t this standard JS chaining?
- rubenfiszel 3y ago
  Exactly! It looks and feel like normal JS and hides all the complexity behind it which is the print of great syntax.
  - dsQTbR7Y5mRHnZv 3y ago
    Perhaps "API design" would be more apt than syntax here

welder 3y ago

Very timely... I'm almost finished porting WakaQ (my custom background task queue to replace Celery) from Python into TypeScript to power background tasks for a new Next.js website. I'm using T3 not Deno so I wouldn't have used Deno Queues, but it's a core part of every web app so makes sense they would build this into their stack. Some questions: * Looks very powerful, but at the core is it just one single queue? Does Deno.openKv() create a new queue or re-use the same queue every time it's called? * Usually I need multiple queues of different priority, so when bottlenecks happen the highest priority jobs run first. * Maybe they guarantee infinite worker capacity so you don't need queue priorities? No bottlenecks if you can pay for the jobs you enqueue?

simonw 3y ago

Unless I'm missing something, it looks like each Deno.openKv() instance only gets a single queue. For the local version you could get multiple queues by calling Deno.openKv("db-2.db") with different SQLite file paths each time, but that feels like a lot of overhead for a pretty common need. I guess this is a Deno architectural style thing - maybe when you build complex apps on Deno it's expected that you'll have a microservice style architecture where lots of different scripts work together, each of them with their own KV store and hence their own queue?

gabeidx 3y ago
You are right, there's only a single queue at the moment. One of the core devs has confirmed this on their discord: https://discord.com/channels/684898665143206084/1156714282533265459/1156750131014668348 https://discord.com/channels/684898665143206084/115671428253... Quoting here: > Correct. Currently a single queue is supported. You could multiplex multiple types of messages on the single queue though.

pier25 3y ago

Deno is amazing for backend. Is there any full stack solution other than Fresh?

dackerlunghack 3y ago

With grand promises rode the ploy Venture capital, ahoy! We make a lovable product to capture audience Vendor lock them at the earliest convenience Hey look at our awesome developer experience Add ANSI and emojis and comments saying it’s genius Some next level tech, plebs just don’t get it Next round of funding, we’ve already spent it I took the big dough, got puppets and framed it R-O-I is bad though but layoffs gon’ fix it Never mind the drama *cough* are ya gon’ install it Run it, ship it, kill yourself with working on it Entangle your business with it Tell the client to deal with it Bring it to the big wig Get ’em to sign off on it Crown your own sh*t Dogfood is tasty, ain’t it

meowtimemania 3y ago
chat gpt?
- dackerlunghack 3y ago
  Human brains proudly present Real poetry? Grouchy resent! My rhyme is not gradient descent After all, it is barely nascent

pests 3y ago

I feel its odd for a compiler/language toolchain to have a cloud offering. I understand the motivations but 20% of the article was about their Deno Deploy product and how to calculate API costs... I understand its only when using their cloud and there is a local implementation, but I just couldn't imagine using a feature in LLVM, for example, where if you were to upload the binary to a specific cloud it would use a different implmentation that costs behind the scenes. It just feels odd to me. I guess it's kind of cool. Maybe different clouds can implement the underlying KV and then this becomes cross-cloud with different underlying benefits without changing the code. But I'm not entirely sure how I feel about this.

wmf 3y ago
These features pay for Deno. If they don't exist, Deno doesn't exist.
- pests 3y ago
  I agree projects need funding and developers need to be paid... Deno was released 4 years before Deno Deploy and existed just fine.
billywhizz 3y ago
there is also a b2b game in play here by deno. i'd imagine a FaaS JS/TS only platform powered by Deno runtime is attractive to a lot of already existing web services who want to expose the ability to write webhooks/event-based functions in their service offerings? https://deno.com/blog/netlify-edge-functions-on-deno-deploy https://deno.com/blog/netlify-edge-functions-on-deno-deploy

londons_explore 3y ago

> at least once semantics. > user code. A past life has taught me that users will never properly understand at-least-once semantics. Anytime you redeliver messages, you will get a flurry of user complaints and breakage. Either you do the impossible and invent a way to do exactly-once semantics. Or you should always redeliver 0.1% of all messages, just so that users don't come to depend on messages being delivered once.

chewbacha 3y ago
Early SQS basically did this. I'm not sure if this was intentional or not but it quickly taught me to respect redelivery.
plaguuuuuu 3y ago
If messages need to be idempotent, I definitely recommend creating integration tests to ensure this from the sending side :-)

endisneigh 3y ago

Can we deploy both deno kv and queues on our own foundation db cluster?

spion 3y ago

I like how updating the data and enqueueing a message can be part of a single transaction. That's indeed pretty powerful. A bit of an aside, but not sure we want at-most-once delivery for email.

crabmusket 3y ago
Didn't the article say at-LEAST-once?
- spion 3y ago
  Yeah it did. My bad writing "at most". But the problem is with "at least once" (email sending cannot be made idempotent AFAIK)

hintymad 3y ago

What happens to the mantra that "database as a queue is harmful"[1]? Of course Amazon teams internally use DynamoDB for all kinds of queues, which implies that we get to do a lot of things easily if we get a super robust storage solution. So, I guess the question is really whether FoundationDB can be used as a backend for queues. [1] https://www.google.com/search?q=Use+database+as+a+queue+is+harmful https://www.google.com/search?q=Use+database+as+a+queue+is+h...

robertwt7 3y ago

How is this different than using RabbitMQ? Can anyone tell me what’s the benefit of choosing this one over some other message queues or even kafka?

ollien 3y ago
Well, a couple things. For one, this is just a simple queue with no routing capabilities. It's a much simpler tool. For two, and more importantly, it's built into your application and introduces no additional operational dependency. You could probably achieve a pretty similar API with a custom AMQP library, though!

Etheryte 3y ago

On the topic of Deno's cloud deployment and Deno KV, does it offer a reasonable user story for caching? Or is that something you're expected to handle yourself (for now)?