4 ms·

I disagree strongly with this. Before all these cloud providers websites were not noticeably less reliable than they are now. I was around back when Apache and

by slackingoff2017 9y ago

I disagree strongly with this. Before all these cloud providers websites were not noticeably less reliable than they are now. I was around back when Apache and CGI was all there was, even then uptime was so good that it was rare to hit a website that was down. There's a lot of koolaid being thrown around by the companies with cloud to sell. Unfortunately these also happen to be the big "market leaders" so it's hard to deny what they're saying and get taken seriously. It's like six sigma, agile, or stack ranking. The big guys are doing it so it must be the right thing to do... Right? Until everyone realized it's mostly a ploy to make money selling books and conference tickets, or in this case rent out a bunch of excess capacity for huge profit. I disagree with reliability as well. Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. With the cloud you have to worry a lot more about fault tolerance whereas with dedicated equipment simple offline backups are often enough to meet any reasonable SLA.

mbesto 9y ago
> Before all these cloud providers websites were not noticeably less reliable than they are now. Before all of these providers existed: 1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructure that exists now can properly support it...at scale. 2) Websites that had large volumes of traffic had top tier expensive admins to maintain them (surprise surprise, Amazon did and turned it into a service) > Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. 3) What's your definition of "most"? Sources for those numbers please?
- samstave 9y ago
  Yeah, well my BBS runs just fine over 2400 baud and 640kb ram. Ill see your ass in the Pit.
  - Clubber 9y ago
    Call waiting will take you down every time! :)
- slackingoff2017 9y ago
  As we've gotten more device computers have gotten more powerful. Server software has gotten better. HTTP has Keepalive and multiplexing now. Encryption and networking are offloaded to hardware and we have a lot more cores. I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets. Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly. I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds. Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.
  - garyclarke27 9y ago
    Thanks for the info very useful, I didn't realise how unreliable AWS is for database servers. You're correct random unannounced shutdowns of db servers is just not acceptable for critical data. I will be launching a new business based on Postgres soon and the thought of this is terrifying. I'm not keen on the RDS type services or CoLo so this is an unexpected problem I need to overcome. Do you know whether VPS provider such as Digital Ocean or CloudSigma would be more reliable?
    - slackingoff2017 9y ago
jcastro 9y ago
I think you're being a bit rose-colored glasses here if you're implying that apache/cgi sites had some sort of incredible uptime record back in the time when "the slashdot effect" was still a thing. > Most of my bare metal and colocated machines have uptime of many years. Nobody cares about host uptime, that just means you're not applying security updates. You're lamenting that you have to worry about fault tolerance in the cloud when the entire point is to have throw away instances!
- scott_karana 9y ago
  > Nobody cares about host uptime, that just means you're not applying security updates. Um... Only kernel updates used to require a reboot, and production security-critical ones are fairly rare. Solutions like ksplice, kexec, kpatch, or kgraft have existed since about 2011, and now the Linux kernel has first-class support for "no reboot" updates.
Spooky23 9y ago
You're comparing the simplist possible scenario with a platform that can replace a large data center. How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs? The answer historically is that you would buy a bunch of stuff from Cisco, pay through the nose for TAC, and hire a team of network engineers who may or may not be idiots. One place I worked at lived with a 30 day eta in firewall changes. Your server that has a long uptime is a risk in a large org. It's obviously not patched, nobody knows how to configure it again. Different customers have different needs.
- slackingoff2017 9y ago
  You don't need anything but regular PC's these days... 2 front end IPVS servers running MAC layer direct return. Those can handle close to 40 gigabits each. If that's not fast enough hack something together with DPDK or netmap. Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right. Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works. Redundant transit handled by datacenter level multihoming. Extra redundancy using dual data centers if you want with DNS round robin. Buy dedicated lines to avoid bandwidth costs. You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP. Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation. Now everybody can have a great setup as long as they pay out the nose. Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching. You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years

mbesto 9y ago

> Before all these cloud providers websites were not noticeably less reliable than they are now. Before all of these providers existed: 1) The volume of internet traffic that exists today didn't exist then. Mobile devices didn't exist. Mobile devices that aren't always connected to the internet and consume hours of our day didn't exist. Large downloads (1GB) didn't exist. They couldn't exist because the infrastructure that exists now can properly support it...at scale. 2) Websites that had large volumes of traffic had top tier expensive admins to maintain them (surprise surprise, Amazon did and turned it into a service) > Most of my bare metal and colocated machines have uptime of many years. Most AWS VM's die after a year or two. 3) What's your definition of "most"? Sources for those numbers please?

samstave 9y ago
Yeah, well my BBS runs just fine over 2400 baud and 640kb ram. Ill see your ass in the Pit.
- Clubber 9y ago
  Call waiting will take you down every time! :)
slackingoff2017 9y ago
As we've gotten more device computers have gotten more powerful. Server software has gotten better. HTTP has Keepalive and multiplexing now. Encryption and networking are offloaded to hardware and we have a lot more cores. I would guess a single server can handle thousands of times as many users as it could handle years ago. HAProxy, Netty, Nginx, and others can handle over a million (simple) HTTP requests per second. That's more requests than Google.com gets. Most as in I've been watching over 100 AWS VM's and maybe 30 on Azure for years and they die or crash far more often than than the VM's hosted here, at colo, or our old bare metal machines. It's anecdotal but it seems like AWS doesn't really care about warning you before shutting off your machine. Azure is slightly better but still goes down regularly. I know everyone says "it's okay! Just make your servers fault tolerant!". Well that works great for load balancers and frontend, but doesn't work at all for SQL databases. ACID compliant transactions require a single source of truth and a true multi master SQL database is impossible. Failover yes, but you always risk losing data in the switchover unless you use two phase commit which actually makes your multimaster database slower than a single system. In practice the failover almost always causes some data loss and log conflicts you have to diddle with later. And God help you if the replica falls behind more than a couple seconds. Anyways, for SQL databases system reliability is as essential as ever and it's a lot easier to get high SLA numbers when you control the hardware and the power switch. The closest you can get to the Holy Grail is running KVM VMs locally and doing live machine migrations when hardware starts to fail, but even that won't keep your database running if something really bad happens.
- garyclarke27 9y ago
  Thanks for the info very useful, I didn't realise how unreliable AWS is for database servers. You're correct random unannounced shutdowns of db servers is just not acceptable for critical data. I will be launching a new business based on Postgres soon and the thought of this is terrifying. I'm not keen on the RDS type services or CoLo so this is an unexpected problem I need to overcome. Do you know whether VPS provider such as Digital Ocean or CloudSigma would be more reliable?
  - slackingoff2017 9y ago

jcastro 9y ago

I think you're being a bit rose-colored glasses here if you're implying that apache/cgi sites had some sort of incredible uptime record back in the time when "the slashdot effect" was still a thing. > Most of my bare metal and colocated machines have uptime of many years. Nobody cares about host uptime, that just means you're not applying security updates. You're lamenting that you have to worry about fault tolerance in the cloud when the entire point is to have throw away instances!

scott_karana 9y ago
> Nobody cares about host uptime, that just means you're not applying security updates. Um... Only kernel updates used to require a reboot, and production security-critical ones are fairly rare. Solutions like ksplice, kexec, kpatch, or kgraft have existed since about 2011, and now the Linux kernel has first-class support for "no reboot" updates.

Spooky23 9y ago

You're comparing the simplist possible scenario with a platform that can replace a large data center. How do you handle segmentation of workload among thousands of servers? Provide firewall services? Meter bandwidth? Provide redundancy to protect against failure of a switch or any other single network component? Provide redundant transit via multiple ISPs? The answer historically is that you would buy a bunch of stuff from Cisco, pay through the nose for TAC, and hire a team of network engineers who may or may not be idiots. One place I worked at lived with a 30 day eta in firewall changes. Your server that has a long uptime is a risk in a large org. It's obviously not patched, nobody knows how to configure it again. Different customers have different needs.

slackingoff2017 9y ago
You don't need anything but regular PC's these days... 2 front end IPVS servers running MAC layer direct return. Those can handle close to 40 gigabits each. If that's not fast enough hack something together with DPDK or netmap. Firewall is on the front end load balancers. Linux is one of the best firewalls you can get if you configure it right. Redundant L2 switches past that running RIPV2 or OSPF for routing. I've found that crappy consumer quality switches usually work fine, as long as you wire them in parallel. You can do dumb things like wire them quad redundant and it just works. Redundant transit handled by datacenter level multihoming. Extra redundancy using dual data centers if you want with DNS round robin. Buy dedicated lines to avoid bandwidth costs. You can rent a cage and do all this with used last gen dell workstations running Ubuntu. Get some last gen fibre NIC's off eBay too. Total cost of maybe 2k in hardware to run top 500 site levels of traffic...as long as you're not using something piss slow like PHP. Historically how shitty your setup is depended on how good your network guys are, the cloud just took that out of the equation. Now everybody can have a great setup as long as they pay out the nose. Edit: Also you can patch Linux without reboots, even the kernel. I don't remember the last time I had to reboot from patching. You do need to reboot for major version upgrades but if you stick with LTS versions you're usually good for 5-10 years