6 ms·

Noria: Dynamic, partially-stateful data-flow for high-perf web applications

61 points by mlerner 5y ago

maxthegeek1 5y ago
I could see this being very useful. The standard cache + db architecture has some sharp edges that end up biting people all of the time e.g. the thundering herd problem.
- rubiquity 5y ago
  Not only that but your database and app are working under the assumption of Read Committed or better and now by introducing a cache that is out of band from your database transactions you’re introducing a whole slew of consistency bugs (that often go undetected).
  - lf-non 5y ago
    I may be missing the point here, but isn't it common to address this through a combination of timestamped keys[1] and interleaving database and redis transactions (start redis transaction, start db transaction, update db and add cache operations to current trx, and finally commit db trx and if it succeeds commit the redis trx, otherwise rollback the redis trx.) [1] https://blogcabin.37signals.com/posts/3113-how-key-based-cache-expiration-works https://blogcabin.37signals.com/posts/3113-how-key-based-cac...
    - Orou 5y ago
      Is the interleaving of redis and db transactions atomic?
      - lf-non 5y ago
        Not technically, but you are guaranteed that if the db transaction fails, we don't end up with invalid data in redis. The other side is not true though, but that will just cause a cache miss if the keys are timestamped.
Orou 5y ago
The author of Noria, Jon Gjengset, gave a great talk about the system: https://www.youtube.com/watch?v=s19G6n0UjsM https://www.youtube.com/watch?v=s19G6n0UjsM The speeds achieved are impressive but you do not get consistency.
awinter-py 5y ago
+1 for anything that schematizes caching semantics I've gotten this wrong so many times, and because it only bites you in read-after-write scenarios, it can be difficult to catch with types + tests feels like there are a bunch of patterns in client-server protocols that should be 'as declarative as the schema', but aren't currently. caching, sync, field transformation, batching rules, multi-RPC flows like oauth
fiatjaf 5y ago
Where is the link to the code and installation steps?
- DylanSp 5y ago
  Some googling turned up https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria.
vlmutolo 5y ago
Noria is great research. Though, for anyone wanting to play with it for new projects, be aware that Jon (the author) has said that it's basically research-quality software, and the point was more to be a proof of value rather than to build a new production database.
mrtweetyhack 5y ago

61 points by mlerner 5y ago

I could see this being very useful. The standard cache + db architecture has some sharp edges that end up biting people all of the time e.g. the thundering herd problem.

rubiquity 5y ago
Not only that but your database and app are working under the assumption of Read Committed or better and now by introducing a cache that is out of band from your database transactions you’re introducing a whole slew of consistency bugs (that often go undetected).
- lf-non 5y ago
  I may be missing the point here, but isn't it common to address this through a combination of timestamped keys[1] and interleaving database and redis transactions (start redis transaction, start db transaction, update db and add cache operations to current trx, and finally commit db trx and if it succeeds commit the redis trx, otherwise rollback the redis trx.) [1] https://blogcabin.37signals.com/posts/3113-how-key-based-cache-expiration-works https://blogcabin.37signals.com/posts/3113-how-key-based-cac...
  - Orou 5y ago
    Is the interleaving of redis and db transactions atomic?
    - lf-non 5y ago
      Not technically, but you are guaranteed that if the db transaction fails, we don't end up with invalid data in redis. The other side is not true though, but that will just cause a cache miss if the keys are timestamped.

Orou 5y ago

The author of Noria, Jon Gjengset, gave a great talk about the system: https://www.youtube.com/watch?v=s19G6n0UjsM https://www.youtube.com/watch?v=s19G6n0UjsM The speeds achieved are impressive but you do not get consistency.

awinter-py 5y ago

+1 for anything that schematizes caching semantics I've gotten this wrong so many times, and because it only bites you in read-after-write scenarios, it can be difficult to catch with types + tests feels like there are a bunch of patterns in client-server protocols that should be 'as declarative as the schema', but aren't currently. caching, sync, field transformation, batching rules, multi-RPC flows like oauth

fiatjaf 5y ago

Where is the link to the code and installation steps?

DylanSp 5y ago
Some googling turned up https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria.

vlmutolo 5y ago

Noria is great research. Though, for anyone wanting to play with it for new projects, be aware that Jon (the author) has said that it's basically research-quality software, and the point was more to be a proof of value rather than to build a new production database.

mrtweetyhack 5y ago