6 ms·

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we

I do C++ backend work in a non-web industry and this entire post is Greek to me. Even though this is targeted at developers, you need a better pitch. I get "we did this 100x faster" but the obvious followup question is "how" but then the answer seems to be a ton of flow diagrams with way too many nodes that tell me approximately nothing and some handwaving about something called P-States that are basically defined to be entirely nebulous because they are any kind of data structure. I'm not saying there's nothing here, but I am adjacent to your core audience and I have no idea whether there is after reading your post. I think you are strongly assuming a shared basis where everybody has worked on the same kind of large scale web app before; I would find it much more useful to have an overview of, "This what you would usually do, here are the problems with it, here is what we do instead" with side by side code comparison of Rama vs what a newbie is likely to hack together with single instance postgres.

ldayley 3y ago
Nathan Marz created Apache Storm, coauthored the book "Big Data", and founded an early real-time infrastructure team at Twitter. It's likely the 'curse of knowledge' of working on this specific problem for so long is responsible for the unique and/or unfamiliar style of communication here. EDIT: Specifics
HaZeust 3y ago
... Maybe the post isn't targeted to your audience at all? How is "C++" and "non-web work" adjacent to web work with web launguage audiences?
- slim 3y ago
  he's a developer and curious about the subject. Since it's a blog post, not a scientific paper, the fact that he did not understand could be a communication failure. I think he's being helpful
- rollcat 3y ago
  OP did not specify what their industry actually is. I've been doing "web work" for 17 years and I'm sharing their concern: where's the TL;DR for this? If this somehow can make me 100x as productive, how about starting with a "hello world" example that shows me how is it different from pip install django, etc?
sdwr 3y ago
In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales terribly with # of accounts. - Normal approach. Create a "followed by" table, update it whenever an account follows / unfollows / is deleted / is blocked. Normal sounds good, but add 10x features, or 1000x users, and it gets trickier. You need to make a new table for each feature, and add conditions to the update calls, and they start overlapping... Or you have to split the database up so it scales, but then you have to pay attention to consistency, and watch which order stuff gets updated in. Their solution is separating the "true" data tables from the "view" tables, formally defining the relationship between the two, and creating the "view" tables magically behind the scenes.
- endisneigh 3y ago
  I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.
  - sdwr 3y ago
    This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
    - endisneigh 3y ago
      Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.
falsandtru 3y ago
> Whereas Twitter stores home timelines in a dedicated in-memory database, in Rama they’re stored in-memory in the same processes executing the ETL for timeline fanout. So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than operating a separate in-memory database. The timelines themselves are stored like this: > To minimize memory usage and GC pressure, we use a ring buffer and Java primitives to represent each home timeline. The buffer contains pairs of author ID and status ID. The author ID is stored along with the status ID since it is static information that will never change, and materializing it means that information doesn’t need to be looked up at query time. The home timeline stores the most recent 600 statuses, so the buffer size is 1,200 to accommodate each author ID and status ID pair. The size is fixed since storing full timelines would require a prohibitive amount of memory (the number of statuses times the average number of followers). > Each user utilizes about 10kb of memory to represent their home timeline. For a Twitter-scale deployment of 500M users, that requires about 4.7TB of memory total around the cluster, which is easily achievable. Isn't this where the most difficult(expensive) part is and Rama has little to do with it? It appears that the other parts also do not have to be Rama.
- nathanmarz 3y ago
  We're storing those in-memory within the Rama modules materializing the home timelines. And the query topologies that refresh home timelines for lost partitions is colocated with that. This is dramatically simpler than operating a separate in-memory database, and Rama has everything to do with that.
  - falsandtru 3y ago
    It appears simpler and better without Rama. > So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than operating a separate in-memory database.
rasz 3y ago
Here in video form: Microservices https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ
jwmoz 3y ago
Agreed, just reading through half of it I have no idea what Rama is.

ldayley 3y ago

Nathan Marz created Apache Storm, coauthored the book "Big Data", and founded an early real-time infrastructure team at Twitter. It's likely the 'curse of knowledge' of working on this specific problem for so long is responsible for the unique and/or unfamiliar style of communication here. EDIT: Specifics

HaZeust 3y ago

... Maybe the post isn't targeted to your audience at all? How is "C++" and "non-web work" adjacent to web work with web launguage audiences?

slim 3y ago
he's a developer and curious about the subject. Since it's a blog post, not a scientific paper, the fact that he did not understand could be a communication failure. I think he's being helpful
rollcat 3y ago
OP did not specify what their industry actually is. I've been doing "web work" for 17 years and I'm sharing their concern: where's the TL;DR for this? If this somehow can make me 100x as productive, how about starting with a "hello world" example that shows me how is it different from pip install django, etc?

sdwr 3y ago

In a typical architecture, the DB stores data, and the backend calls the DB to make updates and compile views. Here, the "views" are defined formally (the P-states), and incrementally, automatically updated when the underlying data changes. Example problem: Get a list of accounts that follow account 1306 "Classic architecture": - Naive approach. Search through all accounts follow lists for "1306". Super slow, scales terribly with # of accounts. - Normal approach. Create a "followed by" table, update it whenever an account follows / unfollows / is deleted / is blocked. Normal sounds good, but add 10x features, or 1000x users, and it gets trickier. You need to make a new table for each feature, and add conditions to the update calls, and they start overlapping... Or you have to split the database up so it scales, but then you have to pay attention to consistency, and watch which order stuff gets updated in. Their solution is separating the "true" data tables from the "view" tables, formally defining the relationship between the two, and creating the "view" tables magically behind the scenes.

endisneigh 3y ago
I read their post and honestly it’s not really that much different than just materialized views in a regular database plus async jobs to do the long running tasks. It’s a ridiculous amount of fluff to describe that. Not to mention it’s proprietary and only supports the JVM and doesn’t integrate with the tons of tooling designed about RDBMS unless you stream everything to them, defeating the purpose. What really irks me is that they go on and on bragging about the low LoC count and literally show nothing complete. They should’ve held on this post and released it simultaneously with the code.
- sdwr 3y ago
  This is all armchair for me, but I think they have containers and sharding built in as well, which is the other half of the puzzle when it comes to scaling.
  - endisneigh 3y ago
    Yes, but there are plenty of NewSQL that support views and offer all of that too. Yugabyte, Cockroach, TiDB and that’s just off the top of my head and open source. If we count proprietary then you have Fauna, Cloud Spanner and more I’m sure.

falsandtru 3y ago

> Whereas Twitter stores home timelines in a dedicated in-memory database, in Rama they’re stored in-memory in the same processes executing the ETL for timeline fanout. So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than operating a separate in-memory database. The timelines themselves are stored like this: > To minimize memory usage and GC pressure, we use a ring buffer and Java primitives to represent each home timeline. The buffer contains pairs of author ID and status ID. The author ID is stored along with the status ID since it is static information that will never change, and materializing it means that information doesn’t need to be looked up at query time. The home timeline stores the most recent 600 statuses, so the buffer size is 1,200 to accommodate each author ID and status ID pair. The size is fixed since storing full timelines would require a prohibitive amount of memory (the number of statuses times the average number of followers). > Each user utilizes about 10kb of memory to represent their home timeline. For a Twitter-scale deployment of 500M users, that requires about 4.7TB of memory total around the cluster, which is easily achievable. Isn't this where the most difficult(expensive) part is and Rama has little to do with it? It appears that the other parts also do not have to be Rama.

nathanmarz 3y ago
We're storing those in-memory within the Rama modules materializing the home timelines. And the query topologies that refresh home timelines for lost partitions is colocated with that. This is dramatically simpler than operating a separate in-memory database, and Rama has everything to do with that.
- falsandtru 3y ago
  It appears simpler and better without Rama. > So instead of having to do network operations, serialization, and deserialization, the reads and writes to home timelines in our implementation are literally just in-memory operations on a hash map. This is dramatically simpler and more efficient than operating a separate in-memory database.

rasz 3y ago

Here in video form: Microservices https://www.youtube.com/watch?v=y8OnoxKotPQ https://www.youtube.com/watch?v=y8OnoxKotPQ

jwmoz 3y ago

Agreed, just reading through half of it I have no idea what Rama is.