5 ms·

Fun read. I am a bit curious on not mentioning OLAP, at all? From my experience, the vast majority of complications in systems is people not realizing they ar

by taeric 2y ago

Fun read. I am a bit curious on not mentioning OLAP, at all? From my experience, the vast majority of complications in systems is people not realizing they are asking an OLAP question while wanting parts of OLTP semantics. I'm also curious how much of this extra complication comes from having a central database that tries to be a source of truth for everything? As an easy example, inventory systems that try and have a cloud source of truth have obvious problems when on the ground inventory issues crop up. That is, a lot of the complication is between the distributed nature of the application and database, sure. But, another large source is the non-distributed abstraction that a central database can represent not centralized activity.

honestSysAdmin 2y ago
> From my experience, the vast majority of complications in systems is people not realizing they are asking an OLAP question while wanting parts of OLTP semantics. If you could elaborate on this further, I and others are probably very interested in reading more about it.
- juancn 2y ago
  The main data access difference between OLAP systems and OLTP systems is how many records on average do you need to access: - OLAP: most queries need most records (aggregations span large swaths of data) - OLTP: most queries access just a few records Also, in OLAP, in many cases, you can live with a single-updater model without much trouble, where OLTP, the strength is to have many concurrent updaters (but mostly non-overlapping).
  - hobs 2y ago
    OLAP - Most queries need an aggregate of records. Generally you do NOT need most records, but simply the records grouped by dimensions per interval (for almost all olap reporting). You do not change the data, you observe it. If you change it, you are not dealing with OLAP data. OLTP - You are dealing with the ins and outs of people using stuff to do things. You buy something, you check out something, you some way perturb the state of things. This should not require large amount of row lookups in 99.9% of cases.
  - greggyb 2y ago
    - OLAP: read-mostly, table-scan heavy, many queries run ad-hoc by users - OLTP: write-mostly, index-seek heavy, ~all queries pre-defined up front
  - yoda97 2y ago
    So the first focuses on analytics and reporting, the second on transactions and performance. They are not meant to replace each other, they serve different purposes. Some teams may need both.
- taeric 2y ago
  As the sibling says, it is easy to think about in terms of what you are doing with the data. Reporting on how many transactions of a type have happened over a duration? Probably OLAP. Really, if the word "report" shows up, at all, probably OLAP. Executing parts of a workflow for an active transaction? OLTP. Briefly looking, I can't find the books that I thought covered a lot of this in a good way. Will keep looking, apologies.
  - datadrivenangel 2y ago
    Designing Data Intensive Applications is a very good book in this space.
    - bobnamob 2y ago
      Designing Data Intensive Applications by Martin Kleppmann is required reading if you're in this space. I can't recommend it highly enough. [1] https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 https://www.amazon.com/Designing-Data-Intensive-Applications...
      - taeric 2y ago
        Indeed, I upvoted that response to me, but I should have said as much. This is the book I couldn't remember the name of for the life of me. Really good book.
- guru4consulting 2y ago
  Apart from the above differences, another important difference is that OLAP is often columnar based, as opposed to the typical OLTP being row-based. So, OLAP queries use different kinds of index. Snowflake has introduced Hybrid tables where the same data is stored and indexed twice, once in OLAP columnar type and the other in OLTP style row index.
datadrivenangel 2y ago
I agree that analytical queries cause a significant portion of the complication in transactional systems. However, at global/twitter scale, just basic CRUD operations become pretty hard for OLTP, so scalable OLTP in the cloud is a pretty big deal, especially when you get fun things like phantom read/writes as writes may take time to get to distributed read replicas, so a user may send a tweet and then not see it on their profile for a while.
- taeric 2y ago
  I mean, sure? I hesitate to take the global/twitter scale debates seriously, unfortunately. They are dominated by straw men arguments that you can get sub 50ms latency on every action for all viewers. :(
  - tharkun__ 2y ago
    You can't and it shouldn't matter and I don't care if everyone sees my upvotes on Reddit right away / at the same time as others or if I see theirs or when exactly my comments show up. But global/twitter/reddit etc. scale does matter in that I don't really remember a PHPBB or slashdot or HN swallowing my comments, ever. Even if you get an error you can always just go back in the browser and your input box is there with your comment. But reddit has been effing atrocious over the last few months. You get "Internal Server Error" when trying to do any voting or commenting a lot and it won't go away until you refresh the page like 17 times (until presumably you round robin get to some BE server that actually works. It's also been swallowing comments, where about every third or fourth comment it accepts the comment instead of throwing an Internal Server Error at you (which would be preferable) and instead it "accepts" it but it will never show up. Ever. Only chance is to copy every comment before submitting, in case you have to re-submit it from scratch.
    - taeric 2y ago
      Exactly on that first statement. I think I agree across the board. HN and Stack Overflow are both amusing examples of much simpler architectures that kind of get to the point, though? Such that I'm not clear if we are on the same page, there. (And I, sadly, have zero experience with Reddit.)
mamcx 2y ago
> I'm also curious how much of this extra complication comes from having a central database that tries to be a source of truth for everything? You can reverse this as 'how much extra complication comes from having many disparate databases, so now you push all ACID complications at the company level instead of the software(rdbms)' It is a tricky problem! Having a single source of truth is a tremendous simplification, especially for the people. It should be ideal that the other databases are pure materializations on top of it, but then not matter what you have the need to input more stuff that is not part of the central one, and now you have both problems.
- taeric 2y ago
  A single source of truth for facts is fine. A single source of truth that has to be reflected by external reality? Less fine. At that point, you pretty much have to have reconciliation processes to account for when they don't match. Oddly, I'd argue that the better way to view this is that the central database is the materialization of the data sources scattered around where work actually happens. As such, you have to have constructs that account for provenance and timing introduced there that aren't as necessary at the edges.
- nyrikki 2y ago
  Single source of truth is different in distributed systems, well the costs are different. Using the language of 'software architecture: thee hard parts' this forces your entire system into a single 'architectural quanta', basically it is a monolith. There are situations where monoliths are acceptable or even the least worst option. The 'Fallacies of distributed computing' covers most of these. It can work better if you use a hierarchical model, but OLTP is almost exclusively the relational model. ACID transactions become pretty brittle, software becomes set in stone etc... There are use cases, but the value proposition is typically much weaker

by taeric 2y ago

honestSysAdmin 2y ago

> From my experience, the vast majority of complications in systems is people not realizing they are asking an OLAP question while wanting parts of OLTP semantics. If you could elaborate on this further, I and others are probably very interested in reading more about it.

juancn 2y ago
The main data access difference between OLAP systems and OLTP systems is how many records on average do you need to access: - OLAP: most queries need most records (aggregations span large swaths of data) - OLTP: most queries access just a few records Also, in OLAP, in many cases, you can live with a single-updater model without much trouble, where OLTP, the strength is to have many concurrent updaters (but mostly non-overlapping).
- hobs 2y ago
  OLAP - Most queries need an aggregate of records. Generally you do NOT need most records, but simply the records grouped by dimensions per interval (for almost all olap reporting). You do not change the data, you observe it. If you change it, you are not dealing with OLAP data. OLTP - You are dealing with the ins and outs of people using stuff to do things. You buy something, you check out something, you some way perturb the state of things. This should not require large amount of row lookups in 99.9% of cases.
- greggyb 2y ago
  - OLAP: read-mostly, table-scan heavy, many queries run ad-hoc by users - OLTP: write-mostly, index-seek heavy, ~all queries pre-defined up front
- yoda97 2y ago
  So the first focuses on analytics and reporting, the second on transactions and performance. They are not meant to replace each other, they serve different purposes. Some teams may need both.
taeric 2y ago
As the sibling says, it is easy to think about in terms of what you are doing with the data. Reporting on how many transactions of a type have happened over a duration? Probably OLAP. Really, if the word "report" shows up, at all, probably OLAP. Executing parts of a workflow for an active transaction? OLTP. Briefly looking, I can't find the books that I thought covered a lot of this in a good way. Will keep looking, apologies.
- datadrivenangel 2y ago
  Designing Data Intensive Applications is a very good book in this space.
  - bobnamob 2y ago
    Designing Data Intensive Applications by Martin Kleppmann is required reading if you're in this space. I can't recommend it highly enough. [1] https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 https://www.amazon.com/Designing-Data-Intensive-Applications...
    - taeric 2y ago
      Indeed, I upvoted that response to me, but I should have said as much. This is the book I couldn't remember the name of for the life of me. Really good book.
guru4consulting 2y ago
Apart from the above differences, another important difference is that OLAP is often columnar based, as opposed to the typical OLTP being row-based. So, OLAP queries use different kinds of index. Snowflake has introduced Hybrid tables where the same data is stored and indexed twice, once in OLAP columnar type and the other in OLTP style row index.

datadrivenangel 2y ago

I agree that analytical queries cause a significant portion of the complication in transactional systems. However, at global/twitter scale, just basic CRUD operations become pretty hard for OLTP, so scalable OLTP in the cloud is a pretty big deal, especially when you get fun things like phantom read/writes as writes may take time to get to distributed read replicas, so a user may send a tweet and then not see it on their profile for a while.

taeric 2y ago
I mean, sure? I hesitate to take the global/twitter scale debates seriously, unfortunately. They are dominated by straw men arguments that you can get sub 50ms latency on every action for all viewers. :(
- tharkun__ 2y ago
  You can't and it shouldn't matter and I don't care if everyone sees my upvotes on Reddit right away / at the same time as others or if I see theirs or when exactly my comments show up. But global/twitter/reddit etc. scale does matter in that I don't really remember a PHPBB or slashdot or HN swallowing my comments, ever. Even if you get an error you can always just go back in the browser and your input box is there with your comment. But reddit has been effing atrocious over the last few months. You get "Internal Server Error" when trying to do any voting or commenting a lot and it won't go away until you refresh the page like 17 times (until presumably you round robin get to some BE server that actually works. It's also been swallowing comments, where about every third or fourth comment it accepts the comment instead of throwing an Internal Server Error at you (which would be preferable) and instead it "accepts" it but it will never show up. Ever. Only chance is to copy every comment before submitting, in case you have to re-submit it from scratch.
  - taeric 2y ago
    Exactly on that first statement. I think I agree across the board. HN and Stack Overflow are both amusing examples of much simpler architectures that kind of get to the point, though? Such that I'm not clear if we are on the same page, there. (And I, sadly, have zero experience with Reddit.)

mamcx 2y ago

> I'm also curious how much of this extra complication comes from having a central database that tries to be a source of truth for everything? You can reverse this as 'how much extra complication comes from having many disparate databases, so now you push all ACID complications at the company level instead of the software(rdbms)' It is a tricky problem! Having a single source of truth is a tremendous simplification, especially for the people. It should be ideal that the other databases are pure materializations on top of it, but then not matter what you have the need to input more stuff that is not part of the central one, and now you have both problems.

taeric 2y ago
A single source of truth for facts is fine. A single source of truth that has to be reflected by external reality? Less fine. At that point, you pretty much have to have reconciliation processes to account for when they don't match. Oddly, I'd argue that the better way to view this is that the central database is the materialization of the data sources scattered around where work actually happens. As such, you have to have constructs that account for provenance and timing introduced there that aren't as necessary at the edges.
nyrikki 2y ago
Single source of truth is different in distributed systems, well the costs are different. Using the language of 'software architecture: thee hard parts' this forces your entire system into a single 'architectural quanta', basically it is a monolith. There are situations where monoliths are acceptable or even the least worst option. The 'Fallacies of distributed computing' covers most of these. It can work better if you use a hierarchical model, but OLTP is almost exclusively the relational model. ACID transactions become pretty brittle, software becomes set in stone etc... There are use cases, but the value proposition is typically much weaker