8 ms·

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tas

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.

sdwr 3y ago
This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
- endisneigh 3y ago
  Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.
leonidasv 3y ago
IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?
- andrenotgiant 3y ago
  incremental view maintenance is the database equivalent of: "recompute only the changed data by checking the relationship between fields," Oracle has decent support for incrementally updated materialized views, redshift has some too. Materialize.com is an entire snowflake-like platform built around incrementally maintained materialized views.
- dustingetz 3y ago
  isn't Materialized performing symbolic differentiation of SQL queries?
sixo 3y ago
The difference is that the materialized-view logic lives naturally in the application code; there's no step where they go out of the DB to do computations and then reinsert. Once SQL materialized views aren't enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB/Elasticsearch/etc. Very common architecture. (Writ small, could also use a queue processor like RabbitMQ.) Their approach is to instead--apparently--make all of these first-class elements of the same ecosystem, not by "putting it all in the database", but by putting the database into the application code. Which seems wild, but colocates data, transformation, and view. Seems like it would open up a lot of cans of worms, but if you solve those, sounds great.
- cormacrelf 3y ago
  You can do all of this with https://materialize.com https://materialize.com, and you don’t need to write it in Java. Just connect it to a Postgres instance and start creating materialised views using SQL. These views then auto update. So much so, that you can create a view for the top 10 of something, and let it sit there as the list updates. Otherwise just use normal select statements from your views using any Postgres client.
nathanmarz 3y ago
We are very open in the post that the core concepts are not new: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems. Indexes as arbitrary data structures that you shape to perfectly meet your use cases, a powerful computation API that's like a "distributed programming language", and everything being so integrated make a world of difference. I understand the desire to see all the code, and that's coming in two weeks. That said, the code in the post isn't trivial as it's showing almost the complete implementations of two major parts of Mastodon: the social graph and timeline fanout. Next week you'll be able to play with Rama when we release a build of it, and the documentation will help with that.
- Aeolun 3y ago
  > But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity Every time I hear this the reality turns out to be that building anything with this tech is like building something on top of SAP. But I’m also just allergic to any post that says ‘look how amazing’ in general, so I’m a bit prejudiced.
  - Aeolun 3y ago
    After reading through the post a bit more, I’m inclined to believe it’s not hot air, but I think most of the innovation where is in the management layer, not the ease of application development. Just looking at the first example tells me that there’s a million ways someone that doesn’t know what they’re doing can mess this up. If the author of the platform implements some service on their own platform it’s always going to seem simple.
throw10920 3y ago
> I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. How about you go and implement a Mastodon server to their level of feature parity, and tell us how much effort and how many lines of code it takes? I really don't appreciate this kind of fluffy, insubstantial, overly dismissive non-content on HN.

sdwr 3y ago

This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.

endisneigh 3y ago
Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.

leonidasv 3y ago

IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?

andrenotgiant 3y ago
incremental view maintenance is the database equivalent of: "recompute only the changed data by checking the relationship between fields," Oracle has decent support for incrementally updated materialized views, redshift has some too. Materialize.com is an entire snowflake-like platform built around incrementally maintained materialized views.
dustingetz 3y ago
isn't Materialized performing symbolic differentiation of SQL queries?

sixo 3y ago

The difference is that the materialized-view logic lives naturally in the application code; there's no step where they go out of the DB to do computations and then reinsert. Once SQL materialized views aren't enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB/Elasticsearch/etc. Very common architecture. (Writ small, could also use a queue processor like RabbitMQ.) Their approach is to instead--apparently--make all of these first-class elements of the same ecosystem, not by "putting it all in the database", but by putting the database into the application code. Which seems wild, but colocates data, transformation, and view. Seems like it would open up a lot of cans of worms, but if you solve those, sounds great.

cormacrelf 3y ago
You can do all of this with https://materialize.com https://materialize.com, and you don’t need to write it in Java. Just connect it to a Postgres instance and start creating materialised views using SQL. These views then auto update. So much so, that you can create a view for the top 10 of something, and let it sit there as the list updates. Otherwise just use normal select statements from your views using any Postgres client.

nathanmarz 3y ago

We are very open in the post that the core concepts are not new: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems. Indexes as arbitrary data structures that you shape to perfectly meet your use cases, a powerful computation API that's like a "distributed programming language", and everything being so integrated make a world of difference. I understand the desire to see all the code, and that's coming in two weeks. That said, the code in the post isn't trivial as it's showing almost the complete implementations of two major parts of Mastodon: the social graph and timeline fanout. Next week you'll be able to play with Rama when we release a build of it, and the documentation will help with that.

Aeolun 3y ago
> But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity Every time I hear this the reality turns out to be that building anything with this tech is like building something on top of SAP. But I’m also just allergic to any post that says ‘look how amazing’ in general, so I’m a bit prejudiced.
- Aeolun 3y ago
  After reading through the post a bit more, I’m inclined to believe it’s not hot air, but I think most of the innovation where is in the management layer, not the ease of application development. Just looking at the first example tells me that there’s a million ways someone that doesn’t know what they’re doing can mess this up. If the author of the platform implements some service on their own platform it’s always going to seem simple.

throw10920 3y ago

> I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. How about you go and implement a Mastodon server to their level of feature parity, and tell us how much effort and how many lines of code it takes? I really don't appreciate this kind of fluffy, insubstantial, overly dismissive non-content on HN.