6 ms·

> There are many reasons why an organization would need to build a distributed system, but here are two examples: > - The demands of a consumer Web site/API or

by apeace 9y ago

> There are many reasons why an organization would need to build a distributed system, but here are two examples: > - The demands of a consumer Web site/API or multitenant enterprise application simply exceed the computing capacity of any one machine. > - An enterprise moves an existing application, such as a three-tier system, onto a cloud service provider in order to save on hardware/data-center costs. When you've exhausted the capacity of a single machine, typically you don't jump straight to a distributed system. You can scale out horizontally with a stateless application layer, as long as the data storage on the backend can handle all the load. You can also scale database reads horizontally, using read replicas. This horizontal scale-out is not a distributed system, since consensus ("source of truth") still lives in one machine. So I think a better phrasing of #1 would be "When your write patterns exceed the computing capacity of any one machine".

saryant 9y ago
That's... still a distributed system with all the joys of eventual consistency.
- jdc 9y ago
  However, I do think the general consensus is that stateless/shared-nothing app servers and following something like the 12 Factor App[1] pattern makes things much easier. 1. https://12factor.net https://12factor.net
  - saryant 9y ago
    Yes, it does. But just because your storage layer is abstracted behind an API doesn't mean eventual consistency isn't your concern. Take S3 for example. LIST calls lag deletes but exists() doesn't, so exists() could return 404 for a document that's still included in the result of LIST. Plus, all exists() calls go through a caching load-balancer, so you could receive different answers in two subsequent exists() calls. That's just a small subset. There's a good comment here[1] on S3's consistency semantics. Point is, 12FA makes things easier but any time you have more than one computer involved (i.e, always), consistency becomes complicated. [1] https://issues.apache.org/jira/browse/FLINK-5706 https://issues.apache.org/jira/browse/FLINK-5706
    - zip1234 9y ago
      Seems like when cache needs to get involved, things can get complicated very quickly.
      - dmoy 9y ago
        One of the two hard problems...
    - hamandcheese 9y ago
      I don't really see how your example applies to the point the root comment made. S3 surely _is_ a distributed system, but he is specifically talking about data residing on _one_ machine (notwithstanding the bits about read only followers).
- hueving 9y ago
  Only if the db replicas allow stale reads.
  - saryant 9y ago
    A replica always lags master by some degree. It may be small—until things go sideways—but it's still another form of eventual consistency.
    - peterwwillis 9y ago
      You're saying ACID databases are eventually consistent?
      - saryant 9y ago
        I'm saying read replicas are not instantly in sync with their masters. Updates are asynchronously copied to the replica.
        peterwwillis 9y ago
        Or, you know, use synchronous replication, or semi-synchronous, or a different asynchronous model that isn't weakly consistent such as distributed transaction logs, write-read consensus, complicated locking mechanisms, etc
        saryant 9y ago
        All of these solutions are still battling the fact that distributed systems are hard and there's a tradeoff to every decision. If you use synchronous replication, you now require at least two systems to commit a write, so you've increased the chance that your commit will fail. Maybe that's the right choice—but it's a tradeoff and the decision ultimately rests in your business requirements. My point is simply that there is no magic bullet in ensuring that all relevant systems have the same answer to the same question at the same time.
- di4na 9y ago
  Especially on the load balancer side...
mhluongo 9y ago
AFAIK that's not the definition of the rest of us are using for a distributed system. If anything, considering the fat client trend we're in, I'd go the other direction and claim a single web server backend with multiple clients (browsers) is a distributed system.
- deleted 9y ago
  [deleted]
SFJulie 9y ago
When you have digged a tad in the hell of the clocks/timer of a x86, it is very easy to saturate your capacity of having reliable timestamping Modern computers are not only CPU bound, they are also interrupt bound, and sadly very limited in "available reliable timestamp" capacity. And controlling which clock you use is hellish. And if the availability of a monotonic raw clock was not enough, I feel bad that in 2017 we still rely on 8254. Documentation for having a reliable clock on every OSes is close to 0. Without reliable clocks, without reliable synchronisation, we are anyway unable to provide "good" distributed systems at low costs (yes GPS is a solution that can be easily hijacked). x86 has won because of legacy supports, but the legacy support means we are stuck with outdated components
adrianN 9y ago
Yes, computers today are fast and "distributed" computing has massive overheads. With some engineering effort you can save many machines. I like this paper about this topic a lot: http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf
stcredzero 9y ago
This horizontal scale-out is not a distributed system, since consensus ("source of truth") still lives in one machine. I think this is a key idea, though my experience with it only comes from writing game servers. Things are much simpler when there is a single "source of truth" per user. In my system, there is an "active" state maintained in an "instance" where the user is rapidly updated in memory, and a "persisted" state stored to the database when a user joins an instance. This keeps the frequency of database writes down. (And enables many users to be able to interact with each other in realtime.)
flukus 9y ago
There's also old fashioned performance improvements. Most enterprise software has plenty of room for improvement there. I've noticed a correlation between people that push for distributed systems and ones that don't know how things like indexes and transactions work or the need to avoid n+1 queries. I'm sure some apps need to be distributed systems, but I bet it's a tiny minority.
- LoSboccacc 9y ago
  there's also availability, especially for enterprises. so redundant frontend and master/slave db at least. may or may be not distributed, according to one's personal definition of distributed.
- UK-AL 9y ago
  People who actually know how to build distributed systems would understand transactions and indexes.
- apeace 9y ago
  Yes, that's exactly my point. This article makes it sound like you need a distributed system at almost any level of scale. Yet the reality is, you only need one of you are doing tons of writes. Most systems are read-heavy, which is why most people scale reads horizontally instead of using a distributed system.
  - mazerackham 9y ago
    More or less agree with your point, just wanted to point out that scaling reads horizontally usually means adding a distributed quality to your system, and not understanding how that impacts the system can have bad side effects
    - flukus 9y ago
      Where do you draw the line of where a horizontally scalable system begins? Scaling reads might only involve a caching server, but most of us probably wouldn't consider an app server with a cache to be distributed.

by apeace 9y ago

saryant 9y ago

That's... still a distributed system with all the joys of eventual consistency.

jdc 9y ago
However, I do think the general consensus is that stateless/shared-nothing app servers and following something like the 12 Factor App[1] pattern makes things much easier. 1. https://12factor.net https://12factor.net
- saryant 9y ago
  Yes, it does. But just because your storage layer is abstracted behind an API doesn't mean eventual consistency isn't your concern. Take S3 for example. LIST calls lag deletes but exists() doesn't, so exists() could return 404 for a document that's still included in the result of LIST. Plus, all exists() calls go through a caching load-balancer, so you could receive different answers in two subsequent exists() calls. That's just a small subset. There's a good comment here[1] on S3's consistency semantics. Point is, 12FA makes things easier but any time you have more than one computer involved (i.e, always), consistency becomes complicated. [1] https://issues.apache.org/jira/browse/FLINK-5706 https://issues.apache.org/jira/browse/FLINK-5706
  - zip1234 9y ago
    Seems like when cache needs to get involved, things can get complicated very quickly.
    - dmoy 9y ago
      One of the two hard problems...
  - hamandcheese 9y ago
    I don't really see how your example applies to the point the root comment made. S3 surely _is_ a distributed system, but he is specifically talking about data residing on _one_ machine (notwithstanding the bits about read only followers).
hueving 9y ago
Only if the db replicas allow stale reads.
- saryant 9y ago
  A replica always lags master by some degree. It may be small—until things go sideways—but it's still another form of eventual consistency.
  - peterwwillis 9y ago
    You're saying ACID databases are eventually consistent?
    - saryant 9y ago
      I'm saying read replicas are not instantly in sync with their masters. Updates are asynchronously copied to the replica.
      - peterwwillis 9y ago
        Or, you know, use synchronous replication, or semi-synchronous, or a different asynchronous model that isn't weakly consistent such as distributed transaction logs, write-read consensus, complicated locking mechanisms, etc
        saryant 9y ago
        All of these solutions are still battling the fact that distributed systems are hard and there's a tradeoff to every decision. If you use synchronous replication, you now require at least two systems to commit a write, so you've increased the chance that your commit will fail. Maybe that's the right choice—but it's a tradeoff and the decision ultimately rests in your business requirements. My point is simply that there is no magic bullet in ensuring that all relevant systems have the same answer to the same question at the same time.
di4na 9y ago
Especially on the load balancer side...

mhluongo 9y ago

AFAIK that's not the definition of the rest of us are using for a distributed system. If anything, considering the fat client trend we're in, I'd go the other direction and claim a single web server backend with multiple clients (browsers) is a distributed system.

deleted 9y ago
[deleted]

SFJulie 9y ago

When you have digged a tad in the hell of the clocks/timer of a x86, it is very easy to saturate your capacity of having reliable timestamping Modern computers are not only CPU bound, they are also interrupt bound, and sadly very limited in "available reliable timestamp" capacity. And controlling which clock you use is hellish. And if the availability of a monotonic raw clock was not enough, I feel bad that in 2017 we still rely on 8254. Documentation for having a reliable clock on every OSes is close to 0. Without reliable clocks, without reliable synchronisation, we are anyway unable to provide "good" distributed systems at low costs (yes GPS is a solution that can be easily hijacked). x86 has won because of legacy supports, but the legacy support means we are stuck with outdated components

adrianN 9y ago

Yes, computers today are fast and "distributed" computing has massive overheads. With some engineering effort you can save many machines. I like this paper about this topic a lot: http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf

stcredzero 9y ago

This horizontal scale-out is not a distributed system, since consensus ("source of truth") still lives in one machine. I think this is a key idea, though my experience with it only comes from writing game servers. Things are much simpler when there is a single "source of truth" per user. In my system, there is an "active" state maintained in an "instance" where the user is rapidly updated in memory, and a "persisted" state stored to the database when a user joins an instance. This keeps the frequency of database writes down. (And enables many users to be able to interact with each other in realtime.)

flukus 9y ago

There's also old fashioned performance improvements. Most enterprise software has plenty of room for improvement there. I've noticed a correlation between people that push for distributed systems and ones that don't know how things like indexes and transactions work or the need to avoid n+1 queries. I'm sure some apps need to be distributed systems, but I bet it's a tiny minority.

LoSboccacc 9y ago
there's also availability, especially for enterprises. so redundant frontend and master/slave db at least. may or may be not distributed, according to one's personal definition of distributed.
UK-AL 9y ago
People who actually know how to build distributed systems would understand transactions and indexes.
apeace 9y ago
Yes, that's exactly my point. This article makes it sound like you need a distributed system at almost any level of scale. Yet the reality is, you only need one of you are doing tons of writes. Most systems are read-heavy, which is why most people scale reads horizontally instead of using a distributed system.
- mazerackham 9y ago
  More or less agree with your point, just wanted to point out that scaling reads horizontally usually means adding a distributed quality to your system, and not understanding how that impacts the system can have bad side effects
  - flukus 9y ago
    Where do you draw the line of where a horizontally scalable system begins? Scaling reads might only involve a caching server, but most of us probably wouldn't consider an app server with a cache to be distributed.