3 ms·

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting poin

Hacker News is much easier to read when you realize that 95% of people have never worked on a "high" (maybe we could say >1B requests per day as a starting point) scale distributed service and think it's trivial to run one with more than 2 nines. You see comments all the time here mentioning that their own desktop at home is achieving more than that which belies deep misunderstanding of how systems are measured. Or that unofficial github status page repeatedly posted here that counts all github services together into one number.

ocdtrekkie 10d ago
> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's easier to operate reliably.
- toomuchtodo 10d ago
  Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale). "Simplicity is the ultimate sophistication." -- Da Vinci
  - a_conservative 10d ago
    Hidden in this discussion around self-hosting reliability are other options as well. Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted. Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.
    - toomuchtodo 10d ago
      That was my intent, pull the thread.
- jedberg 10d ago
  Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service? I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.
  - toomuchtodo 10d ago
    In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for the use case and target outcome. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%22is%20down%22&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
    - jedberg 10d ago
      Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply? My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse. That’s why these systems are built the way they are. And generally four nines is considered the gold standard these days. I can tell you for sure that both Netflix and Ebay would lose money anytime they drop below four nines because I have at some point been responsible for both. You’re correct that Reddit has a lot more leeway and outage time before they start losing money but not that much leeway.
      - toomuchtodo 10d ago
        I've contributed to building out data centers, as well as managed colos for others, primarily in downtown Chicago at Level3 and at 350 E Cermak. I am familiar with architecture required for reliability and diversity, from power and fiber in all the way up the stack to the Kubernetes cluster and software defined networking. If you participate in the capital markets, your data traverses systems I've participated in designing and implementing. There is a time and place for complexity (in this context, large/global distributed systems), but too often, complexity exists where it need not (imho).
leptons 10d ago
While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per day service. Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.
- pixl97 10d ago
  >Salesforce is not simple. It's wildly, overly complex. It turns out business environments are wildly overly complex.
  - lifeisstillgood 10d ago
    I think it’s like advertising - 50% of my code is wildly over complicated - I just don’t know which 50% But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner. So you could not actually make 80% of them happy - they would ever buy it.
    - DANmode 10d ago
      > I just don’t know which 50% Yes, you largely do - they’re the commits that get rushed to, and through. This take that showstopping technical debt is unavoidable is very new, and will age like milk.
      - fragmede 10d ago
        > showstopping technical debt is unavoidable is very new No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.
deleted 10d ago
[deleted]
torginus 10d ago
Well ackchually.. I get that large scale systems pose their own challenges on their own, but it also matters what's the smallest isolable unit. What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. In contrast, something like a bank or social media isn't really reducible - every user needs to be able to interact with every other user in a consistent manner. So running a midsize bank's backend which processes 10m transactions per day, might be as if not more complex (all consistent, repeatable, and must never fail), that having a product which is a 10-10k org's IT infra replicated a thousand times. And yes, lots of people have worked at banks and other fintech companies of this scale, including me. I am not an expert, as I never worked on the 'core' systems but I know folks who did, and everyone told me there's an arcane database monolith that sits at the heart of these, very expensive and exotic big box SW & HW (at least for us unwashed rubes used to EC2 instances)
- r3trohack3r 10d ago
  > What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. This is only true if you exclude problems like “finding a CDN node from the device,” “managing congestion,” etc. as part of the problem statement
- supriyo-biswas 10d ago
  > What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. They do though! They mostly try to avoid it since hitting the network to serve any kind of latency would unacceptably increase latency, but you wildly underestimated the amount of complexity there is to running a CDN.
  - torginus 10d ago
    I probably underestimated the complexity and I didn't mean to knock on CDNs - I just wanted to say that not all distributed systems have equal complexity, and some require essentially almost serializable transactions, while others are fine with small channels of eventual consistency
tedd4u 8d ago
The other thing people miss is change velocity. NTP server doesn't change much. Salesforce teams are under enormous pressure to constantly deliver new features and ship changes every day. 99.95+% with constant change, millions of customers, 1e10-11 requests a day is challenging.

ocdtrekkie 10d ago

> which belies deep misunderstanding I think you are missing the point. When I state my Exchange server is more reliable than Exchange Online, I don't think I'm a better engineer. I recognize Microsoft has harder problems to solve than I do. I think building overengineered, oversized SaaS environments is introducing extreme risk. It's an inherent flaw of the current approach. Smaller is, in fact, better, because it's easier to operate reliably.

toomuchtodo 10d ago
Indeed, the scale Anon1096 refers to wrt distributed systems is anti pattern. It is designed to vacuum up revenue and create enterprise value with scale, not to create resiliency for customers (although resiliency might be a byproduct of a well architected and operated distributed system at scale). "Simplicity is the ultimate sophistication." -- Da Vinci
- a_conservative 10d ago
  Hidden in this discussion around self-hosting reliability are other options as well. Depending on your time and appetite for tinkering with all of this, it's not hard to imagine a home setup that fails over to a cheap Hetzner or DO VM. A manual failover at the DNS level isn't overly complex, and could be scripted. Keeping a database in sync between home and the instance might be simple or more complex depending on needs, but would it really be that hard to have Claude help you setup a replicating Postgres server? If your database (or data files) are 1 gigabyte and don't update that often... maybe just rsync it every night or something There's a thread you and others are pulling on here, and we need to pull it. Hosting doesn't have to be the domain of the big vendors anymore.
  - toomuchtodo 10d ago
    That was my intent, pull the thread.
jedberg 10d ago
Is it? When your internet is out for five days because your ISP takes a few days to get to you, do you acknowledge that you're now at 98.5% availability for the year, far worse than any SaaS email service? I think people forget that those large environments are there for a reason. To make sure the service stays up in the face of problems outside your own control.
- toomuchtodo 10d ago
  In my entire adult lifetime (mid 40s), my ISP has never been out for five days. Compare to Github, Microsoft, Salesforce, and AWS outages that are always occurring in some fashion. Reddit is down constantly in various ways and still continues to operate as a business, public no less, so I disagree about the need to chase five nines and broadly speaking, large distributed systems that are potentially unnecessary for the use case and target outcome. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=%22is%20down%22&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
  - jedberg 10d ago
    Consider yourself lucky that you’ve never been the victim of a fiber cut. But what about if the power to your house goes out? Or what if your server blows the power supply? My entire point is that you have no redundancy in your system and you also aren’t big enough to have any pull with the vendors who can fix these types of outages so you’re basically at the mercy of your providers with no recourse. That’s why these systems are built the way they are. And generally four nines is considered the gold standard these days. I can tell you for sure that both Netflix and Ebay would lose money anytime they drop below four nines because I have at some point been responsible for both. You’re correct that Reddit has a lot more leeway and outage time before they start losing money but not that much leeway.
    - toomuchtodo 10d ago
      I've contributed to building out data centers, as well as managed colos for others, primarily in downtown Chicago at Level3 and at 350 E Cermak. I am familiar with architecture required for reliability and diversity, from power and fiber in all the way up the stack to the Kubernetes cluster and software defined networking. If you participate in the capital markets, your data traverses systems I've participated in designing and implementing. There is a time and place for complexity (in this context, large/global distributed systems), but too often, complexity exists where it need not (imho).

leptons 10d ago

While not a home-run server, the NTP system is a distributed service that receives 100 billion to trillions of requests per day, and it's running pretty smoothly - it's never gone down completely since it started in 1985. It's also very simple. The reason it has so many 9's uptime is because it is simple. Given a low amount of complexity, it's not unreasonable to think that an individual could run a >1B requests per day service. Salesforce is not simple. It's wildly, overly complex. It's amazing it has any 9's at all and not 8's or 7's. Salesforce offers three 9's, which allows for 43 minutes downtime per month. The current outage is at 8 hours (and counting) so Salesforce is now at 98.9% uptime for the month - there's an "8" in there now. Not good, but considering the complexity of Salesforce, it's still kind of amazing.

pixl97 10d ago
>Salesforce is not simple. It's wildly, overly complex. It turns out business environments are wildly overly complex.
- lifeisstillgood 10d ago
  I think it’s like advertising - 50% of my code is wildly over complicated - I just don’t know which 50% But the GP is essentially correct - there is a 2% of salesforce that could be built run and keep 80% of salesforce users happy. Except that you could not charge enough to be able to advertise on F1 cars and take SVPs out to dinner. So you could not actually make 80% of them happy - they would ever buy it.
  - DANmode 10d ago
    > I just don’t know which 50% Yes, you largely do - they’re the commits that get rushed to, and through. This take that showstopping technical debt is unavoidable is very new, and will age like milk.
    - fragmede 10d ago
      > showstopping technical debt is unavoidable is very new No it's not. The push and pull between shipping and paying down technical debt is as old as there's been software to sell. Sales has been selling features that don't exist quite yet ever since they've been talking to customers, and engineering has been pushing back on implementing them yesterday since there's been features to implement. Showstopping technical debt is merely a side effect of who wins that argument in a given org.

deleted 10d ago

[deleted]

torginus 10d ago

Well ackchually.. I get that large scale systems pose their own challenges on their own, but it also matters what's the smallest isolable unit. What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. In contrast, something like a bank or social media isn't really reducible - every user needs to be able to interact with every other user in a consistent manner. So running a midsize bank's backend which processes 10m transactions per day, might be as if not more complex (all consistent, repeatable, and must never fail), that having a product which is a 10-10k org's IT infra replicated a thousand times. And yes, lots of people have worked at banks and other fintech companies of this scale, including me. I am not an expert, as I never worked on the 'core' systems but I know folks who did, and everyone told me there's an arcane database monolith that sits at the heart of these, very expensive and exotic big box SW & HW (at least for us unwashed rubes used to EC2 instances)

r3trohack3r 10d ago
> What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. This is only true if you exclude problems like “finding a CDN node from the device,” “managing congestion,” etc. as part of the problem statement
supriyo-biswas 10d ago
> What I mean by this is a CDN consists of nodes that are horizontally replicable and don't really talk to each other, and thus are easy to run even at scale. They do though! They mostly try to avoid it since hitting the network to serve any kind of latency would unacceptably increase latency, but you wildly underestimated the amount of complexity there is to running a CDN.
- torginus 10d ago
  I probably underestimated the complexity and I didn't mean to knock on CDNs - I just wanted to say that not all distributed systems have equal complexity, and some require essentially almost serializable transactions, while others are fine with small channels of eventual consistency

tedd4u 8d ago

The other thing people miss is change velocity. NTP server doesn't change much. Salesforce teams are under enormous pressure to constantly deliver new features and ship changes every day. 99.95+% with constant change, millions of customers, 1e10-11 requests a day is challenging.