3 ms·

Good write up, Nathan. Nonetheless, you started by saying you were going to challenge the basic assumptions of how data systems should be built, but the resulti

by rxin 15y ago

Good write up, Nathan. Nonetheless, you started by saying you were going to challenge the basic assumptions of how data systems should be built, but the resulting system architecture is in fact very similar to a lot of data streaming systems that the academia and industry have proposed and built for a number of years. An example of such system is Google's Percolator [1]. In essence, you separate the system into two parts, a read-only part and a real-time part. CAP explains fundamental trade-offs in OLTP type of queries, whereas this system design is catered towards streaming, analytical queries. These are two very different contexts. [1] http://research.google.com/pubs/pub36726.html http://research.google.com/pubs/pub36726.html

nathanmarz 15y ago
I was under the impression that Percolator was a purely incremental system. I'll have to give the paper another read to see if they're using a hybrid batch/realtime approach. The system I described isn't limited to any particular "type" of query. It's a way of supporting arbitrary queries on arbitrary data, which encapsulates everything. Most people assume mutable state and incremental updates when building data systems. This is in fact the approach that almost all databases take. These are the assumptions I challenged.
- VladRussian 15y ago
  >Most people assume mutable state and incremental updates when building data systems. This is in fact the approach that almost all databases take. These are the assumptions I challenged. it isn't assumptions. It is requirements for some systems. You drop requirements - you get a breathing room and can build a different system.
  - nathanmarz 15y ago
    Mutability and incremental updates are never requirements. Requirements have to do with performance and resource usage: X amount of space, Y amount of memory, Z amount of latency, etc. Mutability and incremental updates are approaches. I showed a different approach with immutability at the core which leads to a system with some amazing properties.
    - eru 15y ago
      I hope that examples like this and git make more people see the value of immutability. In the context of functional programming the same concept is called purity.

by rxin 15y ago

nathanmarz 15y ago

I was under the impression that Percolator was a purely incremental system. I'll have to give the paper another read to see if they're using a hybrid batch/realtime approach. The system I described isn't limited to any particular "type" of query. It's a way of supporting arbitrary queries on arbitrary data, which encapsulates everything. Most people assume mutable state and incremental updates when building data systems. This is in fact the approach that almost all databases take. These are the assumptions I challenged.

VladRussian 15y ago
>Most people assume mutable state and incremental updates when building data systems. This is in fact the approach that almost all databases take. These are the assumptions I challenged. it isn't assumptions. It is requirements for some systems. You drop requirements - you get a breathing room and can build a different system.
- nathanmarz 15y ago
  Mutability and incremental updates are never requirements. Requirements have to do with performance and resource usage: X amount of space, Y amount of memory, Z amount of latency, etc. Mutability and incremental updates are approaches. I showed a different approach with immutability at the core which leads to a system with some amazing properties.
  - eru 15y ago
    I hope that examples like this and git make more people see the value of immutability. In the context of functional programming the same concept is called purity.