4 ms·

Not directly related, I've been really tempted by all these exciting new database offerings.. PlanetScale, CockroachDB, FaunaDB.. but there don't seem to be any

by johtso 4y ago

Not directly related, I've been really tempted by all these exciting new database offerings.. PlanetScale, CockroachDB, FaunaDB.. but there don't seem to be any solid (node.js) background job queue libraries that support them. I really want to take advantage of these low dev-ops high availability offerings, but I don't want to have to tack Redis onto my stack just so I can use BullMQ, with data persistence and robustness as a bit of an afterthought. And I don't want to roll my own because it's not a trivial thing to solve, especially with stall detection etc. I'd love to just have one robust data-store. Am I missing something?

jai_ 4y ago
Assuming your async job workload isn't massive, you could use job systems that have database backed queues?
- johtso 4y ago
  That's exactly what I'm looking for, I'm just not seeing the libraries.
lewisl9029 4y ago
Yeah... I wish Temporal had FaunaDB support. Seems like a great fit for globally distributed jobs.
- johtso 4y ago
  Temporal sounds _very_ interesting, hadn't heard of it before.. Edit: It sounds like temporal supports CockroachDB
  - lewisl9029 4y ago
    It could work in theory, since it's supposed to be compatible with Postgres, but last time I checked they don't test against it, and it's not anywhere close to 100% identical (https://www.cockroachlabs.com/docs/v22.1/postgresql-compatibility.html https://www.cockroachlabs.com/docs/v22.1/postgresql-compatib...), so I'm personally not too comfortable relying on it for production workloads. Things might have changed since though, I'd try asking folks in the forums to hear their experiences with it before diving in. Would be awesome if this setup worked well though! I personally prefer Fauna from an operational perspective since it's a 0-maintenance SaaS that scales pricing with usage with a generous free tier, but it's probably unlikely to be supported anytime soon.
- deleted 4y ago
  [deleted]
14u2c 4y ago
Any type of job queue that can use MySQL for scheduling state support should work just fine with PlanetScale.
derekperkins 4y ago
Vitess, the underlying technology for Planetscale, does natively support messaging. It's pretty awesome because you can rely on native MySQL transactions to atomically: - Normal database work - Ack source message - Add N messages to N destination queues Our whole stack is built on it and we process ~1B messages / day. https://vitess.io/docs/13.0/reference/features/messaging/ https://vitess.io/docs/13.0/reference/features/messaging/ Hopefully Planetscale sees fit to expose that functionality in the future, and to be honest, since all the interaction with it is over the MySQL protocol, it's very possible that it would just work today.

by johtso 4y ago

jai_ 4y ago

Assuming your async job workload isn't massive, you could use job systems that have database backed queues?

johtso 4y ago
That's exactly what I'm looking for, I'm just not seeing the libraries.

lewisl9029 4y ago

Yeah... I wish Temporal had FaunaDB support. Seems like a great fit for globally distributed jobs.

johtso 4y ago
Temporal sounds _very_ interesting, hadn't heard of it before.. Edit: It sounds like temporal supports CockroachDB
- lewisl9029 4y ago
  It could work in theory, since it's supposed to be compatible with Postgres, but last time I checked they don't test against it, and it's not anywhere close to 100% identical (https://www.cockroachlabs.com/docs/v22.1/postgresql-compatibility.html https://www.cockroachlabs.com/docs/v22.1/postgresql-compatib...), so I'm personally not too comfortable relying on it for production workloads. Things might have changed since though, I'd try asking folks in the forums to hear their experiences with it before diving in. Would be awesome if this setup worked well though! I personally prefer Fauna from an operational perspective since it's a 0-maintenance SaaS that scales pricing with usage with a generous free tier, but it's probably unlikely to be supported anytime soon.
deleted 4y ago
[deleted]

14u2c 4y ago

Any type of job queue that can use MySQL for scheduling state support should work just fine with PlanetScale.

derekperkins 4y ago

Vitess, the underlying technology for Planetscale, does natively support messaging. It's pretty awesome because you can rely on native MySQL transactions to atomically: - Normal database work - Ack source message - Add N messages to N destination queues Our whole stack is built on it and we process ~1B messages / day. https://vitess.io/docs/13.0/reference/features/messaging/ https://vitess.io/docs/13.0/reference/features/messaging/ Hopefully Planetscale sees fit to expose that functionality in the future, and to be honest, since all the interaction with it is over the MySQL protocol, it's very possible that it would just work today.