5 ms·

A single bare metal server is more reliable than most people think it is. Complexity adds a lot of overhead and layer after layer that could possibly fail.

by nik736 4y ago

jazzyjackson 4y ago
Obligatory Lamport quote: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."
srer 4y ago
A single server is much faster than most people think, too! In the microservice or serverless arrangements I've seen, data is scattered across the cloud. It's common for the dominant factor in performance to be data locality, most times this talk about data locality is about avoiding trips to RAM, or worse disk. But in our "modern" distributed cloud things, finding a bit of data frequently involves a trip over the network. In the monolith world what was once invoking a method on an account object, has become making a HTTP POST to the accounts microservice. What might have been a microseconds operation in the single server world, might become hundreds of milliseconds in the distributed cloud world. While you can't horizontally scale a single server, your 1000x head start in performance might delay scaling issues for a very long time. A most excellent paper related to this topic that I think should be mandatory reading before allowing anyone an AWS account is http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf :)
- bayindirh 4y ago
  When you give half the effort to set things up properly, a single server can handle a lot of load and traffic, and get a lot of things done. If you know some details of the services you're going to host on that hardware, the things you can do while saving a lot of resources is considered as black magic by many people who only deploys microservices to K8S systems. ...and you don't need VMs, containers, K8S and anything.
  - GGfpc 4y ago
    What are those details?
    - bayindirh 4y ago
      It generally boils down to three things: 1) How much resource a service need to run well, 2) How much the service wants to consume if left unchecked, 3) How performant you want your service to be. After understanding these parameters, you can limit the resources of your application by running it under a cgroup. Doing this won't allow a service to surpass the limits you've put onto it, and cgroup will pressure your service when it nears its resource limits. Also, sharing resources is good. Instead of having 10 web server containers, you can host all of them under a single one with virtual hosts, most of the time. This allows good resource sharing and doing more with less processes. On the extreme case, I'm running a home server (DNS server, torrent client, synching client, a small HTTP server and an FTP server with some other services) under a 512MB OrangePi Zero. The guy works well, and never locks up. It has plenty of free RAM, and none of the services are choking.
      - codethief 4y ago
        I agree but at the same time: Inter-process communication is also faster when a process is allowed to write to or read from another process's memory. Doesn't make it a good idea, though.
        bayindirh 4y ago
        The way I deploy them doesn't mean "compromise one, compromise all". For example, I generally leave SELinux intact. So they're properly isolated.
- moralestapia 4y ago
  Yeah, nowadays you could just scale-up and still have a lot of leeway. A high-end server (w/ redundancy maybe) is more than enough for 95% of all common startup use cases. Distributed computing only makes sense when you're starting to deal with millions of daily users.
  - efortis 4y ago
    Yes. HN works well with one dual socket server (2 x 4) A new server could have 4 × 64. Also, to distribute the load you can use an ‘A’ DNS record per server: https://blog.uidrafter.com/freebsd-jails-network-setup https://blog.uidrafter.com/freebsd-jails-network-setup
- awoimbee 4y ago
  To be fair you can run a microservices stack on a single server and it will be very fast, especially if you use gRPC instead of HTTP
  - postalrat 4y ago
    Would grpc make a huge difference if requests took a second each?
    - dopp0 4y ago
      gRPC being an RPC, it avoids the overload of HTTP by miles, achieving low latency as well.
iso1631 4y ago
I think the main risk of the HN architecture is that (I believe) it's a single datacentre in a single location. Hopefully they have offsite backups. The other risk I guess would be that all the NS are AWS/route53 severs, so if that went down they'd be up for about a minutes (looks like the DNS TTL is 2 minutes). You could host your own NS servers in two different locations on two different providers, you could have a third hot spare server ready to go too, that would allow the service to survive an earthquake flattening San Diego (off site server ready to go), and cope with the loss of AWS DNS. Whether the cost/benefit ratio is there is another matter. I think the serving side of route53 is fairly reliable though (even when the interface to update records fails on a frequent basis), and the cost of being down isn't particularly terrible.
- barelysapient 4y ago
  > I think the main risk of the HN architecture is that (I believe) it's a single datacentre in a single location. Exactly, Amazon US EAST makes the same point several times a year! :)
  - systemvoltage 4y ago
    We should ask this question to ourselves: As a user, if I have to manage the risk of US-East-1 going down, may be we should question this entire Cloud thing. It was marketed as "100% bulletproof" in 2005 when we discarded on-prem.
    - iso1631 4y ago
      To be fair to amazon I don't think they've had an outage that affected different data centres at the same time, but yes I would far rather have my main and backup on two providers in two appropriately diverse locations.
      - dopp0 4y ago
        but never forget that AWS uses us-east-1 to host core services that, when they go down, they take almost the whole thing down, no matter which region.
tomsmeding 4y ago
> A single bare metal server is more reliable than most people think it is. Well, two, presumably. Otherwise you can't reboot without taking the website offline.
- stavros 4y ago
  Sometimes that's fine, we can go without HN for five minutes once every few months.
  - theandrewbailey 4y ago
    Few months? I notice HN down for a few minutes about once a week.
- NorwegianDude 4y ago
  If you want it to be online, why turn it off or reboot in the first place? You can run updates without rebooting, at least on Linux.
  - tomsmeding 4y ago
    Can't replace the kernel though, and updating the web server application itself is going to be hard too. Unless it's some Erlang like hotpluggable server I guess.
    - fragmede 4y ago
      > Can't replace the kernel though Yes you can, since 2008. https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice

A single bare metal server is more reliable than most people think it is. Complexity adds a lot of overhead and layer after layer that could possibly fail.

by nik736 4y ago

A single bare metal server is more reliable than most people think it is. Complexity adds a lot of overhead and layer after layer that could possibly fail.

jazzyjackson 4y ago

Obligatory Lamport quote: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."

srer 4y ago

A single server is much faster than most people think, too! In the microservice or serverless arrangements I've seen, data is scattered across the cloud. It's common for the dominant factor in performance to be data locality, most times this talk about data locality is about avoiding trips to RAM, or worse disk. But in our "modern" distributed cloud things, finding a bit of data frequently involves a trip over the network. In the monolith world what was once invoking a method on an account object, has become making a HTTP POST to the accounts microservice. What might have been a microseconds operation in the single server world, might become hundreds of milliseconds in the distributed cloud world. While you can't horizontally scale a single server, your 1000x head start in performance might delay scaling issues for a very long time. A most excellent paper related to this topic that I think should be mandatory reading before allowing anyone an AWS account is http://www.frankmcsherry.org/assets/COST.pdf http://www.frankmcsherry.org/assets/COST.pdf :)

bayindirh 4y ago
When you give half the effort to set things up properly, a single server can handle a lot of load and traffic, and get a lot of things done. If you know some details of the services you're going to host on that hardware, the things you can do while saving a lot of resources is considered as black magic by many people who only deploys microservices to K8S systems. ...and you don't need VMs, containers, K8S and anything.
- GGfpc 4y ago
  What are those details?
  - bayindirh 4y ago
    It generally boils down to three things: 1) How much resource a service need to run well, 2) How much the service wants to consume if left unchecked, 3) How performant you want your service to be. After understanding these parameters, you can limit the resources of your application by running it under a cgroup. Doing this won't allow a service to surpass the limits you've put onto it, and cgroup will pressure your service when it nears its resource limits. Also, sharing resources is good. Instead of having 10 web server containers, you can host all of them under a single one with virtual hosts, most of the time. This allows good resource sharing and doing more with less processes. On the extreme case, I'm running a home server (DNS server, torrent client, synching client, a small HTTP server and an FTP server with some other services) under a 512MB OrangePi Zero. The guy works well, and never locks up. It has plenty of free RAM, and none of the services are choking.
    - codethief 4y ago
      I agree but at the same time: Inter-process communication is also faster when a process is allowed to write to or read from another process's memory. Doesn't make it a good idea, though.
      - bayindirh 4y ago
        The way I deploy them doesn't mean "compromise one, compromise all". For example, I generally leave SELinux intact. So they're properly isolated.
moralestapia 4y ago
Yeah, nowadays you could just scale-up and still have a lot of leeway. A high-end server (w/ redundancy maybe) is more than enough for 95% of all common startup use cases. Distributed computing only makes sense when you're starting to deal with millions of daily users.
- efortis 4y ago
  Yes. HN works well with one dual socket server (2 x 4) A new server could have 4 × 64. Also, to distribute the load you can use an ‘A’ DNS record per server: https://blog.uidrafter.com/freebsd-jails-network-setup https://blog.uidrafter.com/freebsd-jails-network-setup
awoimbee 4y ago
To be fair you can run a microservices stack on a single server and it will be very fast, especially if you use gRPC instead of HTTP
- postalrat 4y ago
  Would grpc make a huge difference if requests took a second each?
  - dopp0 4y ago
    gRPC being an RPC, it avoids the overload of HTTP by miles, achieving low latency as well.

iso1631 4y ago

I think the main risk of the HN architecture is that (I believe) it's a single datacentre in a single location. Hopefully they have offsite backups. The other risk I guess would be that all the NS are AWS/route53 severs, so if that went down they'd be up for about a minutes (looks like the DNS TTL is 2 minutes). You could host your own NS servers in two different locations on two different providers, you could have a third hot spare server ready to go too, that would allow the service to survive an earthquake flattening San Diego (off site server ready to go), and cope with the loss of AWS DNS. Whether the cost/benefit ratio is there is another matter. I think the serving side of route53 is fairly reliable though (even when the interface to update records fails on a frequent basis), and the cost of being down isn't particularly terrible.

barelysapient 4y ago
> I think the main risk of the HN architecture is that (I believe) it's a single datacentre in a single location. Exactly, Amazon US EAST makes the same point several times a year! :)
- systemvoltage 4y ago
  We should ask this question to ourselves: As a user, if I have to manage the risk of US-East-1 going down, may be we should question this entire Cloud thing. It was marketed as "100% bulletproof" in 2005 when we discarded on-prem.
  - iso1631 4y ago
    To be fair to amazon I don't think they've had an outage that affected different data centres at the same time, but yes I would far rather have my main and backup on two providers in two appropriately diverse locations.
    - dopp0 4y ago
      but never forget that AWS uses us-east-1 to host core services that, when they go down, they take almost the whole thing down, no matter which region.

tomsmeding 4y ago

> A single bare metal server is more reliable than most people think it is. Well, two, presumably. Otherwise you can't reboot without taking the website offline.

stavros 4y ago
Sometimes that's fine, we can go without HN for five minutes once every few months.
- theandrewbailey 4y ago
  Few months? I notice HN down for a few minutes about once a week.
NorwegianDude 4y ago
If you want it to be online, why turn it off or reboot in the first place? You can run updates without rebooting, at least on Linux.
- tomsmeding 4y ago
  Can't replace the kernel though, and updating the web server application itself is going to be hard too. Unless it's some Erlang like hotpluggable server I guess.
  - fragmede 4y ago
    > Can't replace the kernel though Yes you can, since 2008. https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice