5 ms·

Whilst not making light of the complexity of the field. Distributed systems, it honestly is just a toy for the tech monopolies. Nobody I've ever heard of in a

by urthor 5y ago

Whilst not making light of the complexity of the field. Distributed systems, it honestly is just a toy for the tech monopolies. Nobody I've ever heard of in a company with less than a billion dollars of revenue is bothering. Who ever thought "let's build a software program which has major issues dealing with clock errors!" My day job doing distributed compute at a "main street" company is/was completely eaten by "why the hell would you even, just throw Postgres on a 64 core 2TB RAM machine with replication." Why indeed.

hughrr 5y ago
That’s ok until you reach the logical IO and compute capacity of off the shelf PC hardware. Also 20 years of insane coupling and complexity makes it pretty difficult to move away from that architecture which is where most companies seem to end up with from experience. Also it’s a complete fallacy that “if we’re successful or grow large enough, we’ll just rewrite it”. And then there’s the logical problems of managing complexity, distributing work to development teams and even things as determining change impact that become terrible difficult. I say we should design with distribution in mind but deploy without it. Assuming you can scale up forever is an expensive and stupid mistake. Incidentally it turns out that even adding any deferred processing (in our case SQS+workers) on simple LOB systems can have nasty problems like clock skew and QoS to consider which are very distributed pains :)
- legulere 5y ago
  If you have good enough engineers and manpower for distributed systems, you can also optimize your systems enough to fit again, if you are not a billion dollar company. Distributed systems are exciting and interesting, but currently not worth their investment for 99% of the cases. Things might change though in some decades.
  - hughrr 5y ago
    Been there done that. That gave us two years and cost a year of developer time.
  - urthor 5y ago
    Generally speaking, the second you're running tricks to "jam" something that's overly large into a single machine, yeah you should just give up. The issue becomes when you reach the scale of compute needs such that you cannot "jam" all your requirements into a single leader multiple node cluster. The amount of data you'll need for THAT, is becoming unimaginable. 64 core 128 thread leader nodes have outrageous potential to scale. That said, that doesn't quite apply to my Postgres example (barring some nifty DBA tricks and third party extensions). But you can appreciate how much data you're going to need to blow past what 128 threads can do for you.
    - hughrr 5y ago
      Actually 20 years of shitty code built on assumptions like this can break an AWS 96 core instance running SQL server. That’s my life. It does however shift 10k shitty queries per second which is impressive.
- urthor 5y ago
  > 20 years of insane coupling and complexity makes it pretty difficult to move away from that architecture which is where most companies seem to end up with from experience. Pretty much. I don't want to generalise for legacy context however. > I say we should design with distribution in mind but deploy without it. Oh yes, absolutely. You can pretty reasonably predict when your compute needs will scale up such that you genuinely need to make the shift, and plan for it with a modular architecture. > Assuming you can scale up forever is an expensive and stupid mistake. See here's the thing. I think it has been, in the past. I'm definitely sure that there will be applications in the future which will scale past single leader, multiple nodes. What I've seen in the last few years was we just didn't ever have those demands (except for deep learning). I wasn't working for small business, at a genuine mega-corp. The total production data inflows and egresses of my mega corp, never peaked past I think 200MB/s in any year (for business critical systems. There was some user facing video that was jettison-able). The daily peak was far lower than that. All production compute needs, bandwidth and compute, were DWARFED by employees on Zoom and Youtube. The sum total of all proprietary OLTP data, across the company? And we had roughly 14 different legacy proprietary business systems from acquisitions. We had COBAL, we had DB2, we had it all. The architect of our consolidation had it at less than 5TB (excluding photographs, backups and duplication etc). 40+ years of business critical OLTP data. And all the analytics was done on trailing views of the replicas. Given the direction of Moore's law, I can safely say that for my former multi-billion dollar employer, Moore's law is going to outpace any of our business requirements. (Except for photographs. But all our deep learning was being done by a spin out).
  - hughrr 5y ago
    One size doesn’t fit all. Wait until you’re bought by another multi billion dollar employer and expected to scale out to their workload…
    - urthor 5y ago
      Oh definitely. I'm not saying it doesn't happen.
antoinealb 5y ago
But in a way you are doing distributed systems; that's what the "with replication" part of your architecture is saying. Now you can choose to never ever test that your replication setup is working, but I think that "distributed systems testing is for big tech" is bad advice. Something that would work pretty well in your setup: take down the Postgres leader, and see if your website is still up. That's already Chaos Engineering.
- yakshaving_jgt 5y ago
  In a way, yes. But that’s not what most people are referring to when they talk about “distributed systems”, so I think you’ve constructed a straw man.
  - eklavya 5y ago
    What do those people mean by "distributed systems"? Are they running a single node server with web app and database on the same node? Is there a load balancer/cache/replica/cluster? If it's 1+ machine, it is distributed. In your opinion what would qualify as a "distributed system"?
    - urthor 5y ago
      Distributed system probably isn't precise enough in of itself. But I was very much thinking of multiple leader distributed systems. Anecdotal. But dealing with clock errors and multi-leader issues are of far greater complexity than caching data. Replicas, load balancers, and all the vagaries of single leader systems are not absurdly difficult technical problems. They're problems that can be solved by a team with a single senior engineer who knows what they're doing.
    - yakshaving_jgt 5y ago
      I'm not taking a position on what would qualify as a distributed system, and neither was that the point of my comment. I believe that when most people talk about "distributed systems", they're talking about some form of microservices architecture, where components of a single system are arbitrarily separated by network requests for nebulous reasons.
deleted 5y ago
[deleted]
dikei 5y ago
Distributed systems are not limited to distributed computing: once you start breaking your monolith application into multiple services, you have yourself a distributed system.
- cle 5y ago
  Distributed systems are really about persistent state. As soon as you have/need that, you run into these problems. If you need to deploy while maintaining state (like DB contents), or fail gracefully without losing your entire DB, then you have entered the realm of distributed systems. These properties are present even in most "monoliths".
- cmeiklejohn 5y ago
  A bit of a shameless plug here, but I'm a Ph.D. student at Carnegie Mellon University (Ph.D. in Software Engineering) and my thesis is on the resilience of microservice applications at scale. We've published a little bit on this topic, with some new work under submission currently, but I've put together a website that talks about a.) why doing this work as a student is very difficult due to the lack of open-source applications in this style and b.) proposed a new technique for addressing these problems. It's called Filibuster: https://filibuster.cloud/index.html https://filibuster.cloud/index.html Example of testing a small Java application with Filibuster: https://www.youtube.com/watch?v=BhZLHpxQ7mI https://www.youtube.com/watch?v=BhZLHpxQ7mI
  - nijave 5y ago
    I think there's a fair bit of microservices in the CNCF space. Cortex is implemented as micro services (and Kubernetes largely is, too)
  - deleted 5y ago
    [deleted]
cgio 5y ago
Plenty non tech monopolies out there with more than 1bn revenue. Also distribution is not just relevant on the basis of data volume but also e.g. to accommodate variety of use cases. Don’t get me wrong, Postgres is a great tool but the moment you do replication, the only reason you potentially don’t have to test your distributed system is because other people did it for you, not because it’s not relevant.
tzone 5y ago
High availability + uptime is now necessary for any business no matter how small. If you need to run 24/7 available high uptime services you can’t avoid distributed systems one way or another. (Distributed systems vary by complexity of course but anything that has some failover type of mechanism is going to be a distributed system ) Also, industry is now transitioning where even small business might need geo distribution , not just for compute but also for data. That is going to make distributed systems even more complex even for the small guys. Cloud services have made a lot of this stuff easier for average Joe but you can’t really avoid dealing with at least some fundamental concepts of distributed systems
- adevx 5y ago
  Not sure I agree. My Hetzner vps has less downtime than some AWS regions. To be honest, I have yet to experience any downtime from them over multiple years. There is a good chance you introduce more potential downtime by blowing up the complexity with a distributed system.
zozbot234 5y ago
> Distributed systems, it honestly is just a toy for the tech monopolies. Not so, decentralized systems are also inherently distributed. The whole point of both the old "Web 3.0" and the newer "Web3" is to make "distributed" thinking a part of the web, beyond the now trivial "linking to a document-like resource on a different host" scenario that defines hypertext/the "Web of documents". (Of course one may then retort, as Jack Dorsey apparently has, that "Web3" itself is merely "a VCs' plaything". One has to be aware that these developments are somewhat speculative and pretty far from a real consensus.)
- deleted 5y ago
  [deleted]

by urthor 5y ago

hughrr 5y ago

That’s ok until you reach the logical IO and compute capacity of off the shelf PC hardware. Also 20 years of insane coupling and complexity makes it pretty difficult to move away from that architecture which is where most companies seem to end up with from experience. Also it’s a complete fallacy that “if we’re successful or grow large enough, we’ll just rewrite it”. And then there’s the logical problems of managing complexity, distributing work to development teams and even things as determining change impact that become terrible difficult. I say we should design with distribution in mind but deploy without it. Assuming you can scale up forever is an expensive and stupid mistake. Incidentally it turns out that even adding any deferred processing (in our case SQS+workers) on simple LOB systems can have nasty problems like clock skew and QoS to consider which are very distributed pains :)

legulere 5y ago
If you have good enough engineers and manpower for distributed systems, you can also optimize your systems enough to fit again, if you are not a billion dollar company. Distributed systems are exciting and interesting, but currently not worth their investment for 99% of the cases. Things might change though in some decades.
- hughrr 5y ago
  Been there done that. That gave us two years and cost a year of developer time.
- urthor 5y ago
  Generally speaking, the second you're running tricks to "jam" something that's overly large into a single machine, yeah you should just give up. The issue becomes when you reach the scale of compute needs such that you cannot "jam" all your requirements into a single leader multiple node cluster. The amount of data you'll need for THAT, is becoming unimaginable. 64 core 128 thread leader nodes have outrageous potential to scale. That said, that doesn't quite apply to my Postgres example (barring some nifty DBA tricks and third party extensions). But you can appreciate how much data you're going to need to blow past what 128 threads can do for you.
  - hughrr 5y ago
    Actually 20 years of shitty code built on assumptions like this can break an AWS 96 core instance running SQL server. That’s my life. It does however shift 10k shitty queries per second which is impressive.
urthor 5y ago
> 20 years of insane coupling and complexity makes it pretty difficult to move away from that architecture which is where most companies seem to end up with from experience. Pretty much. I don't want to generalise for legacy context however. > I say we should design with distribution in mind but deploy without it. Oh yes, absolutely. You can pretty reasonably predict when your compute needs will scale up such that you genuinely need to make the shift, and plan for it with a modular architecture. > Assuming you can scale up forever is an expensive and stupid mistake. See here's the thing. I think it has been, in the past. I'm definitely sure that there will be applications in the future which will scale past single leader, multiple nodes. What I've seen in the last few years was we just didn't ever have those demands (except for deep learning). I wasn't working for small business, at a genuine mega-corp. The total production data inflows and egresses of my mega corp, never peaked past I think 200MB/s in any year (for business critical systems. There was some user facing video that was jettison-able). The daily peak was far lower than that. All production compute needs, bandwidth and compute, were DWARFED by employees on Zoom and Youtube. The sum total of all proprietary OLTP data, across the company? And we had roughly 14 different legacy proprietary business systems from acquisitions. We had COBAL, we had DB2, we had it all. The architect of our consolidation had it at less than 5TB (excluding photographs, backups and duplication etc). 40+ years of business critical OLTP data. And all the analytics was done on trailing views of the replicas. Given the direction of Moore's law, I can safely say that for my former multi-billion dollar employer, Moore's law is going to outpace any of our business requirements. (Except for photographs. But all our deep learning was being done by a spin out).
- hughrr 5y ago
  One size doesn’t fit all. Wait until you’re bought by another multi billion dollar employer and expected to scale out to their workload…
  - urthor 5y ago
    Oh definitely. I'm not saying it doesn't happen.

antoinealb 5y ago

But in a way you are doing distributed systems; that's what the "with replication" part of your architecture is saying. Now you can choose to never ever test that your replication setup is working, but I think that "distributed systems testing is for big tech" is bad advice. Something that would work pretty well in your setup: take down the Postgres leader, and see if your website is still up. That's already Chaos Engineering.

yakshaving_jgt 5y ago
In a way, yes. But that’s not what most people are referring to when they talk about “distributed systems”, so I think you’ve constructed a straw man.
- eklavya 5y ago
  What do those people mean by "distributed systems"? Are they running a single node server with web app and database on the same node? Is there a load balancer/cache/replica/cluster? If it's 1+ machine, it is distributed. In your opinion what would qualify as a "distributed system"?
  - urthor 5y ago
    Distributed system probably isn't precise enough in of itself. But I was very much thinking of multiple leader distributed systems. Anecdotal. But dealing with clock errors and multi-leader issues are of far greater complexity than caching data. Replicas, load balancers, and all the vagaries of single leader systems are not absurdly difficult technical problems. They're problems that can be solved by a team with a single senior engineer who knows what they're doing.
  - yakshaving_jgt 5y ago
    I'm not taking a position on what would qualify as a distributed system, and neither was that the point of my comment. I believe that when most people talk about "distributed systems", they're talking about some form of microservices architecture, where components of a single system are arbitrarily separated by network requests for nebulous reasons.

deleted 5y ago

[deleted]

dikei 5y ago

Distributed systems are not limited to distributed computing: once you start breaking your monolith application into multiple services, you have yourself a distributed system.

cle 5y ago
Distributed systems are really about persistent state. As soon as you have/need that, you run into these problems. If you need to deploy while maintaining state (like DB contents), or fail gracefully without losing your entire DB, then you have entered the realm of distributed systems. These properties are present even in most "monoliths".
cmeiklejohn 5y ago
A bit of a shameless plug here, but I'm a Ph.D. student at Carnegie Mellon University (Ph.D. in Software Engineering) and my thesis is on the resilience of microservice applications at scale. We've published a little bit on this topic, with some new work under submission currently, but I've put together a website that talks about a.) why doing this work as a student is very difficult due to the lack of open-source applications in this style and b.) proposed a new technique for addressing these problems. It's called Filibuster: https://filibuster.cloud/index.html https://filibuster.cloud/index.html Example of testing a small Java application with Filibuster: https://www.youtube.com/watch?v=BhZLHpxQ7mI https://www.youtube.com/watch?v=BhZLHpxQ7mI
- nijave 5y ago
  I think there's a fair bit of microservices in the CNCF space. Cortex is implemented as micro services (and Kubernetes largely is, too)
- deleted 5y ago
  [deleted]

cgio 5y ago

Plenty non tech monopolies out there with more than 1bn revenue. Also distribution is not just relevant on the basis of data volume but also e.g. to accommodate variety of use cases. Don’t get me wrong, Postgres is a great tool but the moment you do replication, the only reason you potentially don’t have to test your distributed system is because other people did it for you, not because it’s not relevant.

tzone 5y ago

High availability + uptime is now necessary for any business no matter how small. If you need to run 24/7 available high uptime services you can’t avoid distributed systems one way or another. (Distributed systems vary by complexity of course but anything that has some failover type of mechanism is going to be a distributed system ) Also, industry is now transitioning where even small business might need geo distribution , not just for compute but also for data. That is going to make distributed systems even more complex even for the small guys. Cloud services have made a lot of this stuff easier for average Joe but you can’t really avoid dealing with at least some fundamental concepts of distributed systems

adevx 5y ago
Not sure I agree. My Hetzner vps has less downtime than some AWS regions. To be honest, I have yet to experience any downtime from them over multiple years. There is a good chance you introduce more potential downtime by blowing up the complexity with a distributed system.

zozbot234 5y ago

> Distributed systems, it honestly is just a toy for the tech monopolies. Not so, decentralized systems are also inherently distributed. The whole point of both the old "Web 3.0" and the newer "Web3" is to make "distributed" thinking a part of the web, beyond the now trivial "linking to a document-like resource on a different host" scenario that defines hypertext/the "Web of documents". (Of course one may then retort, as Jack Dorsey apparently has, that "Web3" itself is merely "a VCs' plaything". One has to be aware that these developments are somewhat speculative and pretty far from a real consensus.)

deleted 5y ago
[deleted]