5 ms·

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-st

by sdwr 3y ago

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales terribly with # of accounts. - Normal approach. Create a "followed by" table, update it whenever an account follows / unfollows / is deleted / is blocked. Normal sounds good, but add 10x features, or 1000x users, and it gets trickier. You need to make a new table for each feature, and add conditions to the update calls, and they start overlapping... Or you have to split the database up so it scales, but then you have to pay attention to consistency, and watch which order stuff gets updated in. Their solution is separating the "true" data tables from the "view" tables, formally defining the relationship between the two, and creating the "view" tables magically behind the scenes.

endisneigh 3y ago
I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.
- sdwr 3y ago
  This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
  - endisneigh 3y ago
    Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.
- leonidasv 3y ago
  IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?
  - andrenotgiant 3y ago
    incremental view maintenance is the database equivalent of: "recompute only the changed data by checking the relationship between fields," Oracle has decent support for incrementally updated materialized views, redshift has some too. Materialize.com is an entire snowflake-like platform built around incrementally maintained materialized views.
  - dustingetz 3y ago
    isn't Materialized performing symbolic differentiation of SQL queries?
- sixo 3y ago
  The difference is that the materialized-view logic lives naturally in the application code; there's no step where they go out of the DB to do computations and then reinsert. Once SQL materialized views aren't enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB/Elasticsearch/etc. Very common architecture. (Writ small, could also use a queue processor like RabbitMQ.) Their approach is to instead--apparently--make all of these first-class elements of the same ecosystem, not by "putting it all in the database", but by putting the database into the application code. Which seems wild, but colocates data, transformation, and view. Seems like it would open up a lot of cans of worms, but if you solve those, sounds great.
  - cormacrelf 3y ago
    You can do all of this with https://materialize.com https://materialize.com, and you don’t need to write it in Java. Just connect it to a Postgres instance and start creating materialised views using SQL. These views then auto update. So much so, that you can create a view for the top 10 of something, and let it sit there as the list updates. Otherwise just use normal select statements from your views using any Postgres client.
- nathanmarz 3y ago
  We are very open in the post that the core concepts are not new: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems. Indexes as arbitrary data structures that you shape to perfectly meet your use cases, a powerful computation API that's like a "distributed programming language", and everything being so integrated make a world of difference. I understand the desire to see all the code, and that's coming in two weeks. That said, the code in the post isn't trivial as it's showing almost the complete implementations of two major parts of Mastodon: the social graph and timeline fanout. Next week you'll be able to play with Rama when we release a build of it, and the documentation will help with that.
  - Aeolun 3y ago
    > But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity Every time I hear this the reality turns out to be that building anything with this tech is like building something on top of SAP. But I’m also just allergic to any post that says ‘look how amazing’ in general, so I’m a bit prejudiced.
    - Aeolun 3y ago
      After reading through the post a bit more, I’m inclined to believe it’s not hot air, but I think most of the innovation where is in the management layer, not the ease of application development. Just looking at the first example tells me that there’s a million ways someone that doesn’t know what they’re doing can mess this up. If the author of the platform implements some service on their own platform it’s always going to seem simple.
- throw10920 3y ago
  > I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. How about you go and implement a Mastodon server to their level of feature parity, and tell us how much effort and how many lines of code it takes? I really don't appreciate this kind of fluffy, insubstantial, overly dismissive non-content on HN.
ethbr1 3y ago
So... at a high level, early React for data? In other words, letting a framework manage update dependency graph tracking, and then cascading updates through its graph in an optimized manner to enhance performance? Obviously, with tons of implementation difficulties and details, and not actual graph structures, but as a top level analogy.
- ricardobeat 3y ago
  Not at all, especially because React doesn’t do much dependency tracking on its own and is built for predictable UI updates and not performance. To be honest any parallel with frontend here is meaningless, reactivity and all the concepts at play have existed long before JS and browsers came along, it’s easier to explain from first principles.
  - Aeolun 3y ago
    I think that’s probably not the case for many new developers that don’t have any exposure to anything not React. Of course ‘react for data’ is entirely misleading, but it may give a decent idea if you don’t have an hour to spend on an explanation.
    - throwuxiytayq 3y ago
      In other words, it’s the Dark Souls of application backends, but entirely different.
      - Aeolun 3y ago
        Only if your expectation is to be constantly frustrated, and eventually die, after which you have to do it all again.
mananaysiempre 3y ago
I’m getting Noria[1] / Materialize / Readyset vibes from this, or perhaps even Samsa[2] ones. (Incidentally, I’d appreciate it if anyone could elaborate on the differences between the two.) Explicit inspiration? Parallel evolution? [1] https://news.ycombinator.com/item?id=29615085 https://news.ycombinator.com/item?id=29615085 [2] https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html https://martin.kleppmann.com/2015/03/04/turning-the-database...
rickyhacks 3y ago
This is a problem that is very similar to the one solved by Differential Data Flow and implemented by https://github.com/MaterializeInc/materialize https://github.com/MaterializeInc/materialize Have you considered using that?

by sdwr 3y ago

endisneigh 3y ago

I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.

sdwr 3y ago
This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
- endisneigh 3y ago
  Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.
leonidasv 3y ago
IIUIC, the most significant difference from a materialized view is that the Rama infrastructure recompute only the changed data by checking the relationship between fields, while a traditional materialized view recomputes the whole table?
- andrenotgiant 3y ago
  incremental view maintenance is the database equivalent of: "recompute only the changed data by checking the relationship between fields," Oracle has decent support for incrementally updated materialized views, redshift has some too. Materialize.com is an entire snowflake-like platform built around incrementally maintained materialized views.
- dustingetz 3y ago
  isn't Materialized performing symbolic differentiation of SQL queries?
sixo 3y ago
The difference is that the materialized-view logic lives naturally in the application code; there's no step where they go out of the DB to do computations and then reinsert. Once SQL materialized views aren't enough, you might do this by replicating your database into Kafka, implementing logic in Flink or something, and reinserting into the same DB/Elasticsearch/etc. Very common architecture. (Writ small, could also use a queue processor like RabbitMQ.) Their approach is to instead--apparently--make all of these first-class elements of the same ecosystem, not by "putting it all in the database", but by putting the database into the application code. Which seems wild, but colocates data, transformation, and view. Seems like it would open up a lot of cans of worms, but if you solve those, sounds great.
- cormacrelf 3y ago
  You can do all of this with https://materialize.com https://materialize.com, and you don’t need to write it in Java. Just connect it to a Postgres instance and start creating materialised views using SQL. These views then auto update. So much so, that you can create a view for the top 10 of something, and let it sit there as the list updates. Otherwise just use normal select statements from your views using any Postgres client.
nathanmarz 3y ago
We are very open in the post that the core concepts are not new: Individually, none of these concepts are new. I’m sure you’ve seen them all before. You may be tempted to dismiss Rama’s programming model as just a combination of event sourcing and materialized views. But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity that characterize and overwhelm existing systems. Indexes as arbitrary data structures that you shape to perfectly meet your use cases, a powerful computation API that's like a "distributed programming language", and everything being so integrated make a world of difference. I understand the desire to see all the code, and that's coming in two weeks. That said, the code in the post isn't trivial as it's showing almost the complete implementations of two major parts of Mastodon: the social graph and timeline fanout. Next week you'll be able to play with Rama when we release a build of it, and the documentation will help with that.
- Aeolun 3y ago
  > But what Rama does is integrate and generalize these concepts to such an extent that you can build entire backends end-to-end without any of the impedance mismatches or complexity Every time I hear this the reality turns out to be that building anything with this tech is like building something on top of SAP. But I’m also just allergic to any post that says ‘look how amazing’ in general, so I’m a bit prejudiced.
  - Aeolun 3y ago
    After reading through the post a bit more, I’m inclined to believe it’s not hot air, but I think most of the innovation where is in the management layer, not the ease of application development. Just looking at the first example tells me that there’s a million ways someone that doesn’t know what they’re doing can mess this up. If the author of the platform implements some service on their own platform it’s always going to seem simple.
throw10920 3y ago
> I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. How about you go and implement a Mastodon server to their level of feature parity, and tell us how much effort and how many lines of code it takes? I really don't appreciate this kind of fluffy, insubstantial, overly dismissive non-content on HN.

ethbr1 3y ago

So... at a high level, early React for data? In other words, letting a framework manage update dependency graph tracking, and then cascading updates through its graph in an optimized manner to enhance performance? Obviously, with tons of implementation difficulties and details, and not actual graph structures, but as a top level analogy.

ricardobeat 3y ago
Not at all, especially because React doesn’t do much dependency tracking on its own and is built for predictable UI updates and not performance. To be honest any parallel with frontend here is meaningless, reactivity and all the concepts at play have existed long before JS and browsers came along, it’s easier to explain from first principles.
- Aeolun 3y ago
  I think that’s probably not the case for many new developers that don’t have any exposure to anything not React. Of course ‘react for data’ is entirely misleading, but it may give a decent idea if you don’t have an hour to spend on an explanation.
  - throwuxiytayq 3y ago
    In other words, it’s the Dark Souls of application backends, but entirely different.
    - Aeolun 3y ago
      Only if your expectation is to be constantly frustrated, and eventually die, after which you have to do it all again.

mananaysiempre 3y ago

I’m getting Noria[1] / Materialize / Readyset vibes from this, or perhaps even Samsa[2] ones. (Incidentally, I’d appreciate it if anyone could elaborate on the differences between the two.) Explicit inspiration? Parallel evolution? [1] https://news.ycombinator.com/item?id=29615085 https://news.ycombinator.com/item?id=29615085 [2] https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html https://martin.kleppmann.com/2015/03/04/turning-the-database...

rickyhacks 3y ago

This is a problem that is very similar to the one solved by Differential Data Flow and implemented by https://github.com/MaterializeInc/materialize https://github.com/MaterializeInc/materialize Have you considered using that?