5 ms·

This is more an advertisement for a type of database than a statement that they are unnecessary. From what I can tell in the article it seems their differentia

This is more an advertisement for a type of database than a statement that they are unnecessary. From what I can tell in the article it seems their differentiator is Event Sourcing and having arbitrary complex index builders on top of the events. It seems similar to EventStoreDB[1]. I have always been interested by the concept of an event sourcing database with projections and I want to build one eventually so it is interesting to see how they have approached the problem. Also they mention on their site: > Rama is programmed entirely with a Java API – no custom languages or DSLs. It makes sense why they have gone this route if they want a "Turing-complete dataflow API" but this can be a major barrier to adoption. This is a big challenge with implementing these databases in my opinion because you want to allow for any logic to build out your indexes/projections/views but then you are stuck between a new complicated DSL or using a particular language. 1: https://developers.eventstore.com/server/v23.10/#getting-started https://developers.eventstore.com/server/v23.10/#getting-sta...

cryptonector 3y ago
The API could be LinkQ-like, so it'd still be SQL-ish, just in a query builder sort of way.
vintermann 3y ago
Well, it is an advertisement, but much of what it says is undoubtedly true. It's true that databases are effectively global mutable state in practice, with all the headaches that gives, and that event sourcing and materialized views is a damn appealing solution to it. However, the jump from that to saying you have it is a bit like the jump from talking about the messiah and saying that he's sitting in the next room just now. We investigated event sourcing and materialized views as the backbone of a (new) big business application five years ago. The problem was that there were so few products back then and they were so untested (unless my mind is playing tricks on me, we did look at EventStoreDB, as well as Kafka streams), and despite being open source they were also usually heavily tied to one commercial provider. No one had anything like experience with it. I couldn't convince my colleagues to go for this, much less my bosses. Hell, I could barely convince myself. Philosophically, I believe it's sound, but how much has changed in those years?
foofie 3y ago
> This is more an advertisement for a type of database than a statement that they are unnecessary. I stopped reading the article once it was evident that the author was not making any case regarding databases being unnecessary, and instead was putting up a series of strawmen that sounded like desperate attempts to reject all alternatives to the thing they were selling. The author's attempt to talk about data structures was particularly baffling, knowing that pretty standard RDBMS such as PostgreSQL already support JSON data types. Are we supposed to forget about that?
nathanmarz 3y ago
True, Rama does have a learning curve. It's easier to explain why its indexing capabilities are so powerful, as data structures vs. data models is fairly relatable to most devs. But I actually consider its dataflow API to be the bigger breakthrough. Our Twitter-scale Mastodon implementation and examples in rama-demo-gallery are demonstrations of this. Also, Rama has a first-class Clojure API (I should probably update the website). https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery
- morsecodist 3y ago
  Cool on the clojure API. Functional programing really fits with this sort of thing. Does your implementation limit you to the JVM?
  - nathanmarz 3y ago
    Yes, it's JVM-only for the foreseeable future.
    - worik 3y ago
      That is sad Why take that path? I get your point about global mutable state, but I do not see how you have done anything more than shuffle the complexity around Being dependant on software like the JVM makes me very reluctant to investigate further. The JVM is not involved in anything I use, I do not think i want it.
pdonis 3y ago
> This is more an advertisement for a type of database than a statement that they are unnecessary. That was my reaction as well. The article's claimed argument against databases is that they are global mutable state, which is supposed to be bad. But none of what the article advocates for gets rid of global mutable state (which of course you can't do in any real world application, because the real world itself is global mutable state and anything that deals with the real world has to deal with that). It's just hawking a particular way of storing global mutable state.
- adammarples 3y ago
  No, they store data in 'depots', which are append only logs, like kafka. There is no mutable state.
  - pdonis 3y ago
    > There is no mutable state. Yes, there is. Appending a new log entry to the depot mutates the depot. It doesn't mutate earlier records in the depot, but the depot is the entire record set, not just individual records. The record set is different after the append than before it.
- a-saleh 3y ago
  T.b.h. when looking at things, global mutable state might not be bad, if it is principled? I.e. I did like how Clojure had it's atomically updating refs. And you realisticaly won't get more principled than with traditional RDBS, with it's ability to transactions, rollbacks, e.t.c. Only alternative that I have seen proposed to using DBAS while not relaxing the guaranties, is to double down on event-sourcing and not having mutable-state anymore. Everything could be just a fold over all of the incomming events, right? But I don't think I have seen non-toy example of that anywhere I worked at.
  - pdonis 3y ago
    > double down on event-sourcing and not having mutable-state anymore. Event sourcing doesn't get rid of mutable state. The event store gets mutated every time a new event comes in.
zubairq 3y ago
Agreed that this article is an advert for a certain database type, but the advert for red planet labs worked. I certainly read it! :)
- andreareina 3y ago
  It also turned me off from them so there's that.
  - wavemode 3y ago
    Sure but, were you seriously considering them beforehand?
  - zubairq 3y ago
    Good point. I don’t use red planet labs product myself either, but in a world where it is hard to get people to even consider your product the article did work
mike_hearn 3y ago
A Java API doesn't mean you can only access it from Java. Such APIs can also be accessed from most languages. Obviously bytecode languages like Clojure, Kotlin, Scala, Groovy etc. But also via Truffle languages like JS, Python, Ruby. All without bindings. You can also do C++ and Rust, but that does require a bit of binding work. So it's not really limiting you to a single language, or these days, even a single runtime.
- vidarh 3y ago
  Assuming you're willing to run a JVM and possibly specific language implementations. That's fine for a lot of people, I'm sure, but it's not something I'd be willing to do for what is fairly little benefit over "just" whatever solution you're already familiar with to use as the depot and the "P-stores" with workers in between.
vidarh 3y ago
Yeah, the Java dependency makes me reject it instantly - the only advantage they potentially could have over "just" using any database with ability to query remote databases is not having to roll your own, but the barrier to rolling my own is lower for me than introducing a Java dependency. (A replicated set of databases for the depots; a set of workers creating the "p-stores" in ... another set of databases, and you've replicated the architecture; heck you can even run this entirely off postgres w/replication + triggers to handle partitioning and materialization with no external workers if you like) EDIT: Also their API fairly unambiguously is effectively a DSL, just one without it's own separate syntax and parser allowing access from outside the JVM.
- lambda_garden 3y ago
  > the barrier to rolling my own is lower for me than introducing a Java dependency. Really? I never had problems using some JVM base image and deploying via a Docker image, which is what I would be doing anyway.
  - worik 3y ago
    So now you are dependant on JVM and Docker
  - vidarh 3y ago
    I've had enough problems with JVM deployments over the years to just not want the hassle in order to use something which adds marginal value. If you don't, then by all means, use it - I'm not suggesting it's bad for everyone. If it provided some massive major advantage over the large number of other options, I'd consider it too, but I don't see anything that indicates that's the case here.
    - adra 3y ago
      Do you reject Kafka because it runs on the JVM? This is a ludicrous position to take. You may not want to fund development of projects when you're forced to train devs to work them, but to reject using technologies based on the programming language / framework is just fundamentally flawed. Don't use Python tools because I hate python. Don't use kubernetes because it's golang don't use AWS because it has a bunch of java services. S3? I'm not entirely sure but I wouldn't be surprised if it was java edge service based as well.

cryptonector 3y ago

The API could be LinkQ-like, so it'd still be SQL-ish, just in a query builder sort of way.

vintermann 3y ago

Well, it is an advertisement, but much of what it says is undoubtedly true. It's true that databases are effectively global mutable state in practice, with all the headaches that gives, and that event sourcing and materialized views is a damn appealing solution to it. However, the jump from that to saying you have it is a bit like the jump from talking about the messiah and saying that he's sitting in the next room just now. We investigated event sourcing and materialized views as the backbone of a (new) big business application five years ago. The problem was that there were so few products back then and they were so untested (unless my mind is playing tricks on me, we did look at EventStoreDB, as well as Kafka streams), and despite being open source they were also usually heavily tied to one commercial provider. No one had anything like experience with it. I couldn't convince my colleagues to go for this, much less my bosses. Hell, I could barely convince myself. Philosophically, I believe it's sound, but how much has changed in those years?

foofie 3y ago

> This is more an advertisement for a type of database than a statement that they are unnecessary. I stopped reading the article once it was evident that the author was not making any case regarding databases being unnecessary, and instead was putting up a series of strawmen that sounded like desperate attempts to reject all alternatives to the thing they were selling. The author's attempt to talk about data structures was particularly baffling, knowing that pretty standard RDBMS such as PostgreSQL already support JSON data types. Are we supposed to forget about that?

nathanmarz 3y ago

True, Rama does have a learning curve. It's easier to explain why its indexing capabilities are so powerful, as data structures vs. data models is fairly relatable to most devs. But I actually consider its dataflow API to be the bigger breakthrough. Our Twitter-scale Mastodon implementation and examples in rama-demo-gallery are demonstrations of this. Also, Rama has a first-class Clojure API (I should probably update the website). https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/twitter-scale-mastodon https://github.com/redplanetlabs/rama-demo-gallery https://github.com/redplanetlabs/rama-demo-gallery

morsecodist 3y ago
Cool on the clojure API. Functional programing really fits with this sort of thing. Does your implementation limit you to the JVM?
- nathanmarz 3y ago
  Yes, it's JVM-only for the foreseeable future.
  - worik 3y ago
    That is sad Why take that path? I get your point about global mutable state, but I do not see how you have done anything more than shuffle the complexity around Being dependant on software like the JVM makes me very reluctant to investigate further. The JVM is not involved in anything I use, I do not think i want it.

pdonis 3y ago

> This is more an advertisement for a type of database than a statement that they are unnecessary. That was my reaction as well. The article's claimed argument against databases is that they are global mutable state, which is supposed to be bad. But none of what the article advocates for gets rid of global mutable state (which of course you can't do in any real world application, because the real world itself is global mutable state and anything that deals with the real world has to deal with that). It's just hawking a particular way of storing global mutable state.

adammarples 3y ago
No, they store data in 'depots', which are append only logs, like kafka. There is no mutable state.
- pdonis 3y ago
  > There is no mutable state. Yes, there is. Appending a new log entry to the depot mutates the depot. It doesn't mutate earlier records in the depot, but the depot is the entire record set, not just individual records. The record set is different after the append than before it.
a-saleh 3y ago
T.b.h. when looking at things, global mutable state might not be bad, if it is principled? I.e. I did like how Clojure had it's atomically updating refs. And you realisticaly won't get more principled than with traditional RDBS, with it's ability to transactions, rollbacks, e.t.c. Only alternative that I have seen proposed to using DBAS while not relaxing the guaranties, is to double down on event-sourcing and not having mutable-state anymore. Everything could be just a fold over all of the incomming events, right? But I don't think I have seen non-toy example of that anywhere I worked at.
- pdonis 3y ago
  > double down on event-sourcing and not having mutable-state anymore. Event sourcing doesn't get rid of mutable state. The event store gets mutated every time a new event comes in.

zubairq 3y ago

Agreed that this article is an advert for a certain database type, but the advert for red planet labs worked. I certainly read it! :)

andreareina 3y ago
It also turned me off from them so there's that.
- wavemode 3y ago
  Sure but, were you seriously considering them beforehand?
- zubairq 3y ago
  Good point. I don’t use red planet labs product myself either, but in a world where it is hard to get people to even consider your product the article did work

mike_hearn 3y ago

A Java API doesn't mean you can only access it from Java. Such APIs can also be accessed from most languages. Obviously bytecode languages like Clojure, Kotlin, Scala, Groovy etc. But also via Truffle languages like JS, Python, Ruby. All without bindings. You can also do C++ and Rust, but that does require a bit of binding work. So it's not really limiting you to a single language, or these days, even a single runtime.

vidarh 3y ago
Assuming you're willing to run a JVM and possibly specific language implementations. That's fine for a lot of people, I'm sure, but it's not something I'd be willing to do for what is fairly little benefit over "just" whatever solution you're already familiar with to use as the depot and the "P-stores" with workers in between.

vidarh 3y ago

Yeah, the Java dependency makes me reject it instantly - the only advantage they potentially could have over "just" using any database with ability to query remote databases is not having to roll your own, but the barrier to rolling my own is lower for me than introducing a Java dependency. (A replicated set of databases for the depots; a set of workers creating the "p-stores" in ... another set of databases, and you've replicated the architecture; heck you can even run this entirely off postgres w/replication + triggers to handle partitioning and materialization with no external workers if you like) EDIT: Also their API fairly unambiguously is effectively a DSL, just one without it's own separate syntax and parser allowing access from outside the JVM.

lambda_garden 3y ago
> the barrier to rolling my own is lower for me than introducing a Java dependency. Really? I never had problems using some JVM base image and deploying via a Docker image, which is what I would be doing anyway.
- worik 3y ago
  So now you are dependant on JVM and Docker
- vidarh 3y ago
  I've had enough problems with JVM deployments over the years to just not want the hassle in order to use something which adds marginal value. If you don't, then by all means, use it - I'm not suggesting it's bad for everyone. If it provided some massive major advantage over the large number of other options, I'd consider it too, but I don't see anything that indicates that's the case here.
  - adra 3y ago
    Do you reject Kafka because it runs on the JVM? This is a ludicrous position to take. You may not want to fund development of projects when you're forced to train devs to work them, but to reject using technologies based on the programming language / framework is just fundamentally flawed. Don't use Python tools because I hate python. Don't use kubernetes because it's golang don't use AWS because it has a bunch of java services. S3? I'm not entirely sure but I wouldn't be surprised if it was java edge service based as well.