7 ms·

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against,

by Pxtl 3y ago

I've seen many people describe frameworks like this - you know, first you have the slow back-end event-driven master database that you don't query live against, then you've got eventual-consistency flows against the various data-warehouses and data-stores and partitioned sharded databases in useful query-friendly layouts that you actually read live from... and I never see it clearly explained: how do you read a change back to the user literally just after they made the change? How do you say "other views eventual-consistency is fine but for this view of this bit of info we need it updated now". This write-up is very detailed but I couldn't find that explanation.

chubot 3y ago
Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" ... is less than satisfying to me I FREQUENTLY have the problem where I reload the page and Reddit shows me stale data. It's SUPER buggy. --- Anyway I definitely look forward to hearing people try this and what their longer term impressions are ! I basically want to know what the tradeoffs are -- it sounds good, but there are always tradeoffs So is the tradeoff "eventual consistency" ? What are the other tradeoffs?
- chubot 3y ago
  Hilariously, I went to edit the above comment, and HN was overloaded. Then it served me three or four 500's, AND it served me stale data in between I was pissed off that I would have to type my comment again, but actually it did save it, and refreshing worked. From what I understand Hacker News is architected more in-memory, on one big box ... Perhaps similar to the event sourcing model (not knocking hacker news -- it's generally a very fast site, MUCH better than Reddit. Just that scaling beyond a single machine is difficult and full of gotchas )
  - kaba0 3y ago
    StackOverflow used to run on a single (very beefy) machine also for a long time — databases make efficient use of vertical scaling, horizontal scaling is much harder. Of course, specialist systems can often do much better.
- nathanmarz 3y ago
  When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding. The way applications are built, and have been built since before I was born, is by combining together potentially dozens of narrow tools together: databases, computation systems, caches, monitoring tools, etc. There has never been a cohesive model capable of expressing arbitrary backends end-to-end, and every application built has to be twisted to fit onto the existing narrow pieces. Rama is a lot more than just "event sourcing" and "materialized views". Those are two concepts at its foundation, but the real breakthrough is being that cohesive model capable of expressing diverse backends in their entirety. It took me more than five years of dedicated research to discover this model, and it was extremely difficult.
  - chubot 3y ago
    Yes, I 100% agree with you. I would like something like this to succeed, and agree the problem is real. But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs (side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had a Rails app, a database, and RAID disks, and grew it from there. We had BigTable and perf reviews :-P ) Eventual consistency is probably one? Can I specify that comment editing is correct and ACID, while likes/upvotes are eventually consistent? (No is a fine answer, these problems are hard) I read through much of the doc, and don't see a mention of the word "consistency" at all, which seems like an oversight for something that is unifying what would be in a database with computation.
    - nathanmarz 3y ago
      Rama is a much broader platform than a database, so the consistency semantics you get depend on how you use it. When using Rama, you're not mutating indexes directly like you do with a database, but adding source data that then gets materialized into any number of indexes. You get read-after-write consistency for any PStates in a streaming ETL colocated with the depot you appended to. This is if you do the depot append with "full acking", which coordinates its response with the completion of colocated streaming ETLs. If you append at a lower level of acking, then you get eventual consistency on those PStates at the benefit of lower latency appends. Microbatching is always eventually consistent as processing is asynchronous and uncoordinated to depot appends. Microbatching is higher thorughput than streaming and has simpler fault-tolerance semantics. You'll be able to read a lot more about this when we release the docs next week.
lossolo 3y ago
You have the option to track the latest update time and, during the minute immediately following this update, direct all reads to come from the leader. Additionally, you could oversee the replication lag among followers and block queries on any follower that lags more than a minute behind the leader. For the client, it's feasible to retain the timestamp of its most recent write. In this way, the system can ensure that the replica responsible for any reads related to that user incorporates updates at minimum up to that recorded timestamp. If a replica isn't adequately current, the read can either be managed by another replica or the query can wait until the replica catches up. The timestamp might take the form of a logical timestamp, signifying the order of writes (e.g., log sequence number), or it could be based on the actual system clock, where synchronized clocks become vital. When your replicas are spread across multiple datacenters—whether for user proximity or enhanced availability—there's an added layer of complexity. Requests requiring the leader's involvement must be directed to the datacenter housing the leader.
throw14082020 3y ago
It's called "read-after-write consistency". Write-through cache is one way, or just use a strongly consistent database, haha. https://avikdas.com/2020/04/13/scalability-concepts-read-after-write-consistency.html https://avikdas.com/2020/04/13/scalability-concepts-read-aft... https://en.wikipedia.org/wiki/Cache_%28computing%29#Writing_policies https://en.wikipedia.org/wiki/Cache_%28computing%29#Writing_... https://cloud.google.com/blog/products/databases/why-you-should-pick-strong-consistency-whenever-possible https://cloud.google.com/blog/products/databases/why-you-sho...
Groxx 3y ago
One strategy (somewhat common in lambda architectures) is to query both the long-term store and the in-flight operations, and blend the results. The in-flight stuff is both small and already in memory so it's pretty often trivially fast, even if blending the data is relatively complex. That does limit you to operations/queries you can describe in this dual format, but pretty often that's fine. Or if you can relax read-after-write you can just ignore the in-flight stuff and read from the main store and then there are no (added) limitations.
jokethrowaway 3y ago
You can hack it and optimistically render the data you know about because your client created it - on the frontend, at no additional cost.
- hot_gril 3y ago
  This is usually what I do. Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. One famous example of this going to far: Mac Mail app used to play a whoosh sound when your email is actually sent. They changed it to whoosh instantly no matter what. Given how often an email might fail to send or get delayed, this meant an actually useful indication of "great, your thing was sent, you can close your laptop now" was rendered useless.
  - ceejayoz 3y ago
    > Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. HN does this, and on slow days, about half of my upvotes don't go through.
    - wizofaus 3y ago
      You actually check your list of upvoted comments?
      - ceejayoz 3y ago
        No, I just notice it when I come back to the thread later in the day and a bunch of comments I know I upvoted are back to normal.
    - hot_gril 3y ago
      Messaging apps often have a checkmark to indicate the message actually went to the server, and maybe another checkmark to indicate it was received on the other end. Maybe HN needs an icon indicating that your vote went through.
deleted 3y ago
[deleted]
sixo 3y ago
I imagine you get some UUID back from your write, and effectively "block" until you see it committed to the event stream. The intent of such a system is certainly for the read-after-write latency to be not much longer than a traditional RDBMS. (This is roughly what the RDBMS is doing under the hood anyway.) Probably you can isolate latency-critical paths so they don't get stuck behind big stream processing jobs. The advantage of the overall architecture is that nearly all application functionality (for something like a social network) can tolerate much higher latency than an RDBMS, so you really want to have architectural building blocks that let you actually use this headroom.
jedberg 3y ago
The short answer is write-through cache. You write the update directly to the cache closest to the user and into the eventually consistent queue. We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back then it was the server cache.
- jitl 3y ago
  Rama should bundle a write-through cache! Another in-memory JVM cluster thingamabob (Apache Ignite) used to propose write-through caching as it's primary selling point: https://ignite.apache.org/use-cases/in-memory-cache.html#:~:text=Read%2DThrough%20%2F%20Write%2DThrough%20Caching&text=When%20Apache%20Ignite%20is%20deployed,with%20the%20in%2Dmemory%20data https://ignite.apache.org/use-cases/in-memory-cache.html#:~:.... Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.
  - nathanmarz 3y ago
    Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.
    - jitl 3y ago
      is there a way to guarantee reading your own writes from a client perspective?
      - nathanmarz 3y ago
        Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend. Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.

by Pxtl 3y ago

chubot 3y ago

Yeah definitely, these ideas always sound very appealing to me, in theory -- I almost wonder why nobody has built it before e.g. they mention "event sourcing" and "materialized views" in the post -- sounds good But I thought I heard from a few people who were like "we ripped event sourcing" out of our codebase and so forth And yeah your question is an obvious good one, and the Reddit answer of "write through cache" ... is less than satisfying to me I FREQUENTLY have the problem where I reload the page and Reddit shows me stale data. It's SUPER buggy. --- Anyway I definitely look forward to hearing people try this and what their longer term impressions are ! I basically want to know what the tradeoffs are -- it sounds good, but there are always tradeoffs So is the tradeoff "eventual consistency" ? What are the other tradeoffs?

chubot 3y ago
Hilariously, I went to edit the above comment, and HN was overloaded. Then it served me three or four 500's, AND it served me stale data in between I was pissed off that I would have to type my comment again, but actually it did save it, and refreshing worked. From what I understand Hacker News is architected more in-memory, on one big box ... Perhaps similar to the event sourcing model (not knocking hacker news -- it's generally a very fast site, MUCH better than Reddit. Just that scaling beyond a single machine is difficult and full of gotchas )
- kaba0 3y ago
  StackOverflow used to run on a single (very beefy) machine also for a long time — databases make efficient use of vertical scaling, horizontal scaling is much harder. Of course, specialist systems can often do much better.
nathanmarz 3y ago
When you step back and consider the incredible amount of manpower and resources that have been put into these applications, it's amazing how buggy these applications are. To put it simply, they're buggy because the underlying infrastructure and techniques used to build them are so complex that the implementation is beyond the realm of human understanding. The way applications are built, and have been built since before I was born, is by combining together potentially dozens of narrow tools together: databases, computation systems, caches, monitoring tools, etc. There has never been a cohesive model capable of expressing arbitrary backends end-to-end, and every application built has to be twisted to fit onto the existing narrow pieces. Rama is a lot more than just "event sourcing" and "materialized views". Those are two concepts at its foundation, but the real breakthrough is being that cohesive model capable of expressing diverse backends in their entirety. It took me more than five years of dedicated research to discover this model, and it was extremely difficult.
- chubot 3y ago
  Yes, I 100% agree with you. I would like something like this to succeed, and agree the problem is real. But what are the tradeoffs? There's nothing that comes with 100x benefit with no tradeoffs (side note: I worked on Google Code for a short while in 2008, concurrent with Github's founding ... I think Github moved a lot faster in a large part because they weren't dealing with distributed systems at first -- they had a Rails app, a database, and RAID disks, and grew it from there. We had BigTable and perf reviews :-P ) Eventual consistency is probably one? Can I specify that comment editing is correct and ACID, while likes/upvotes are eventually consistent? (No is a fine answer, these problems are hard) I read through much of the doc, and don't see a mention of the word "consistency" at all, which seems like an oversight for something that is unifying what would be in a database with computation.
  - nathanmarz 3y ago
    Rama is a much broader platform than a database, so the consistency semantics you get depend on how you use it. When using Rama, you're not mutating indexes directly like you do with a database, but adding source data that then gets materialized into any number of indexes. You get read-after-write consistency for any PStates in a streaming ETL colocated with the depot you appended to. This is if you do the depot append with "full acking", which coordinates its response with the completion of colocated streaming ETLs. If you append at a lower level of acking, then you get eventual consistency on those PStates at the benefit of lower latency appends. Microbatching is always eventually consistent as processing is asynchronous and uncoordinated to depot appends. Microbatching is higher thorughput than streaming and has simpler fault-tolerance semantics. You'll be able to read a lot more about this when we release the docs next week.

lossolo 3y ago

You have the option to track the latest update time and, during the minute immediately following this update, direct all reads to come from the leader. Additionally, you could oversee the replication lag among followers and block queries on any follower that lags more than a minute behind the leader. For the client, it's feasible to retain the timestamp of its most recent write. In this way, the system can ensure that the replica responsible for any reads related to that user incorporates updates at minimum up to that recorded timestamp. If a replica isn't adequately current, the read can either be managed by another replica or the query can wait until the replica catches up. The timestamp might take the form of a logical timestamp, signifying the order of writes (e.g., log sequence number), or it could be based on the actual system clock, where synchronized clocks become vital. When your replicas are spread across multiple datacenters—whether for user proximity or enhanced availability—there's an added layer of complexity. Requests requiring the leader's involvement must be directed to the datacenter housing the leader.

throw14082020 3y ago

It's called "read-after-write consistency". Write-through cache is one way, or just use a strongly consistent database, haha. https://avikdas.com/2020/04/13/scalability-concepts-read-after-write-consistency.html https://avikdas.com/2020/04/13/scalability-concepts-read-aft... https://en.wikipedia.org/wiki/Cache_%28computing%29#Writing_policies https://en.wikipedia.org/wiki/Cache_%28computing%29#Writing_... https://cloud.google.com/blog/products/databases/why-you-should-pick-strong-consistency-whenever-possible https://cloud.google.com/blog/products/databases/why-you-sho...

Groxx 3y ago

One strategy (somewhat common in lambda architectures) is to query both the long-term store and the in-flight operations, and blend the results. The in-flight stuff is both small and already in memory so it's pretty often trivially fast, even if blending the data is relatively complex. That does limit you to operations/queries you can describe in this dual format, but pretty often that's fine. Or if you can relax read-after-write you can just ignore the in-flight stuff and read from the main store and then there are no (added) limitations.

jokethrowaway 3y ago

You can hack it and optimistically render the data you know about because your client created it - on the frontend, at no additional cost.

hot_gril 3y ago
This is usually what I do. Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. One famous example of this going to far: Mac Mail app used to play a whoosh sound when your email is actually sent. They changed it to whoosh instantly no matter what. Given how often an email might fail to send or get delayed, this meant an actually useful indication of "great, your thing was sent, you can close your laptop now" was rendered useless.
- ceejayoz 3y ago
  > Don't even want to wait for an HTTP roundtrip for some of these, e.g. "liking" a post should fill in the heart icon or whatever instantly. HN does this, and on slow days, about half of my upvotes don't go through.
  - wizofaus 3y ago
    You actually check your list of upvoted comments?
    - ceejayoz 3y ago
      No, I just notice it when I come back to the thread later in the day and a bunch of comments I know I upvoted are back to normal.
  - hot_gril 3y ago
    Messaging apps often have a checkmark to indicate the message actually went to the server, and maybe another checkmark to indicate it was received on the other end. Maybe HN needs an icon indicating that your vote went through.

deleted 3y ago

[deleted]

sixo 3y ago

I imagine you get some UUID back from your write, and effectively "block" until you see it committed to the event stream. The intent of such a system is certainly for the read-after-write latency to be not much longer than a traditional RDBMS. (This is roughly what the RDBMS is doing under the hood anyway.) Probably you can isolate latency-critical paths so they don't get stuck behind big stream processing jobs. The advantage of the overall architecture is that nearly all application functionality (for something like a social network) can tolerate much higher latency than an RDBMS, so you really want to have architectural building blocks that let you actually use this headroom.

jedberg 3y ago

The short answer is write-through cache. You write the update directly to the cache closest to the user and into the eventually consistent queue. We did this at reddit. When you make a comment the HTML is rendered and put straight into the cache, and the raw text is put into the queue to go into the database. Same with votes. I suspect they do this client side now, which is now the closest cache to the user, but back then it was the server cache.

jitl 3y ago
Rama should bundle a write-through cache! Another in-memory JVM cluster thingamabob (Apache Ignite) used to propose write-through caching as it's primary selling point: https://ignite.apache.org/use-cases/in-memory-cache.html#:~:text=Read%2DThrough%20%2F%20Write%2DThrough%20Caching&text=When%20Apache%20Ignite%20is%20deployed,with%20the%20in%2Dmemory%20data https://ignite.apache.org/use-cases/in-memory-cache.html#:~:.... Or, maybe their pitch is that the streaming bits are so fast, you can just await the downstream commit of some write to a depot and it'll be as fast as a normal SQL UPDATE.
- nathanmarz 3y ago
  Rama is extremely fast, as you can see for yourself by playing with our Mastodon instance.
  - jitl 3y ago
    is there a way to guarantee reading your own writes from a client perspective?
    - nathanmarz 3y ago
      Yes. Depot appends by default don't return success until colocated streaming topologies have completed processing the data. So this is one way to coordinate the frontend with changes on the backend. Within an ETL, when the computations you do on PStates are colocated with them, you always read your own writes.