8 ms·

Digging down a couple layers of links from this, the underlying paper, "Partial State in Dataflow-Based Materialized Views" https://jon.thesquareplanet.com/pape

by sedev 5y ago

Digging down a couple layers of links from this, the underlying paper, "Partial State in Dataflow-Based Materialized Views" https://jon.thesquareplanet.com/papers/phd-thesis.pdf https://jon.thesquareplanet.com/papers/phd-thesis.pdf is pretty intriguing. It sounds like a potential free lunch in specific performance areas, which means it also sounds too good to be true, but if it turns out to be a metaphorical 90%-off lunch that's still very promising.

Jonhoo 5y ago
Oh hey, that's my thesis! Happy to answer any questions you may have about it :) There's also the OSDI'18 paper here which may be of interest: https://jon.tsp.io/papers/osdi18-noria.pdf https://jon.tsp.io/papers/osdi18-noria.pdf
- BenoitP 5y ago
  Big fan here! I've been following the space since a bit of time, and I must say it's exciting. To me this is the future of apps where the Truth lives server-side, and everything reacts from there; With partial state evaluation lowering resource consumption to a minimum. Kafka Streams and Apache Flink seem to be focused on real-time analytics, and I wish they'd get there to stimulate the space. Are you affiliated with ReadySet?
  - educaysean 5y ago
    According to the linked article, Jon appears to be one of the co-founders
  - Jonhoo 5y ago
    I'm pretty excited about it too! I remember when I initially started the research I was amazed that this didn't already exist. Some context: https://twitter.com/jonhoo/status/1511401461669720068 https://twitter.com/jonhoo/status/1511401461669720068 Basically, I co-founded the company around the time I graduated, but had had my fill of database research after six years of PhD. So I joined AWS to work on Rust while Alana (the CEO) took on leading ReadySet.
- adamgordonbell 5y ago
  This is super exciting. Ever since I talked to you about Noria I've been telling people about this concept. I'm excited to see a production ready implementation of it.
- ignoramous 5y ago
  I haven't read much about Noria other than this readme [0], but would like to know if you are familiar enough to contrast Materialize [1] with it in terms of perf, overhead, approach, and fundamental (design) principles? [0] https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria [1] https://github.com/MaterializeInc/materialize https://github.com/MaterializeInc/materialize
  - benesch 5y ago
    Back when we announced Materialize we got the same question in reverse! You can read my response from a few years back here: https://news.ycombinator.com/item?id=22362301 https://news.ycombinator.com/item?id=22362301 Unfortunately I'm not privy to whatever improvements ReadySet has made in the past two years, so I can't comment on differences between ReadySet and Materialize. Perhaps Jon can, though!
    - ignoramous 4y ago
      Insightful. Thanks!
glittershark 5y ago
the remaining 10% of the 90%-off free lunch is pretty much just eventual consistency - it can occasionally be the case that you write something to the DB, and an immediate subsequent write doesn't see it. That said, there are escape-hatches there (we'll proxy queries that happen inside of a transaction to the upstream mysql/postgresql database, and there's an experimental implementation of opt-in Read-Your-Writes consistency), and I'd wager that the vast majority of "traditional" web applications can tolerate slightly stale reads. Our official docs also have an aptly-titled: "what's the catch?" section: https://docs.readyset.io/concepts/overview#whats-the-catch https://docs.readyset.io/concepts/overview#whats-the-catch
skyde 5y ago
this is similar to an asynchronously replicated database replica. Which mean read might return stale data because of the replication lag. It will also increase the load on the server you read the replication log from. But the primary database could dump the transaction log to (S3/kafka) and have ReadySet instance read it from there instead of directly from Primary database. So for a read mostly website (hackernews/reddit) this is indeed a free lunch.

by sedev 5y ago

Jonhoo 5y ago

Oh hey, that's my thesis! Happy to answer any questions you may have about it :) There's also the OSDI'18 paper here which may be of interest: https://jon.tsp.io/papers/osdi18-noria.pdf https://jon.tsp.io/papers/osdi18-noria.pdf

BenoitP 5y ago
Big fan here! I've been following the space since a bit of time, and I must say it's exciting. To me this is the future of apps where the Truth lives server-side, and everything reacts from there; With partial state evaluation lowering resource consumption to a minimum. Kafka Streams and Apache Flink seem to be focused on real-time analytics, and I wish they'd get there to stimulate the space. Are you affiliated with ReadySet?
- educaysean 5y ago
  According to the linked article, Jon appears to be one of the co-founders
- Jonhoo 5y ago
  I'm pretty excited about it too! I remember when I initially started the research I was amazed that this didn't already exist. Some context: https://twitter.com/jonhoo/status/1511401461669720068 https://twitter.com/jonhoo/status/1511401461669720068 Basically, I co-founded the company around the time I graduated, but had had my fill of database research after six years of PhD. So I joined AWS to work on Rust while Alana (the CEO) took on leading ReadySet.
adamgordonbell 5y ago
This is super exciting. Ever since I talked to you about Noria I've been telling people about this concept. I'm excited to see a production ready implementation of it.
ignoramous 5y ago
I haven't read much about Noria other than this readme [0], but would like to know if you are familiar enough to contrast Materialize [1] with it in terms of perf, overhead, approach, and fundamental (design) principles? [0] https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria [1] https://github.com/MaterializeInc/materialize https://github.com/MaterializeInc/materialize
- benesch 5y ago
  Back when we announced Materialize we got the same question in reverse! You can read my response from a few years back here: https://news.ycombinator.com/item?id=22362301 https://news.ycombinator.com/item?id=22362301 Unfortunately I'm not privy to whatever improvements ReadySet has made in the past two years, so I can't comment on differences between ReadySet and Materialize. Perhaps Jon can, though!
  - ignoramous 4y ago
    Insightful. Thanks!

glittershark 5y ago

the remaining 10% of the 90%-off free lunch is pretty much just eventual consistency - it can occasionally be the case that you write something to the DB, and an immediate subsequent write doesn't see it. That said, there are escape-hatches there (we'll proxy queries that happen inside of a transaction to the upstream mysql/postgresql database, and there's an experimental implementation of opt-in Read-Your-Writes consistency), and I'd wager that the vast majority of "traditional" web applications can tolerate slightly stale reads. Our official docs also have an aptly-titled: "what's the catch?" section: https://docs.readyset.io/concepts/overview#whats-the-catch https://docs.readyset.io/concepts/overview#whats-the-catch

skyde 5y ago

this is similar to an asynchronously replicated database replica. Which mean read might return stale data because of the replication lag. It will also increase the load on the server you read the replication log from. But the primary database could dump the transaction log to (S3/kafka) and have ReadySet instance read it from there instead of directly from Primary database. So for a read mostly website (hackernews/reddit) this is indeed a free lunch.