9 ms·

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scamperi

I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the good ol’ days (perhaps with some “member berries” https://youtu.be/mPs8-ZBSjok https://youtu.be/mPs8-ZBSjok), or you’ve just straight up forgotten what it takes to build and run your own infra from the ground up. And to those who happen across this comment before posting your own: please contribute something to the discussion beyond the above. Thanks.

stjohnswarts 5y ago
There is nothing wrong with pointing out the warts. I wonder if you've ever been at the mercy of google/aws and their lack of customer service if you aren't representing the business $$ of a fortune 500 company?
cosmotic 5y ago
I don't think anyone is saying they could do better than AWS at making their own infrastructure. I think the complaint is that it's a single point of failure for a huge swath of the internet.
technobabbler 5y ago
Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.
- christophilus 5y ago
  I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.
- londons_explore 5y ago
  I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend. Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.
  - technobabbler 5y ago
    If you're going closer-to-the-metal cloud, then yeah. I think that's part of the allure of newer, more abstracted cloud hosts like Vercel, Netlify, Gatsby, etc. They take care of all the infrastructure for ya, and if something breaks, you just twiddle your thumbs while they twiddle the dials. Of course this sort of laziness is not a valid approach if you're a huge enterprise or running some live-saving project that needs seventy nines of uptime, but for non-critical websites, it's a great sigh of relief...
- ksec 5y ago
  >making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.
  - tuldia 5y ago
    > You can now simply point to AWS is down Heck, if that was even possible... "everything is green" dashboards... :-)
    - technobabbler 5y ago
      Obviously, the answer is to pay for a cloud dashboard-of-dashboards that show orange lights when AWS's is falsely green. "We need this new fancy dashboard to monitor our other dashboards in one place, more accurately!"
  - SahAssar 5y ago
    Do people not take responsibility for their choice of vendors? If boeing crashed a bunch of planes and said "Not our fault, our subcontractor made shitty software" would we accept it? If a startup leaked all their customers data would we accept "Blame this other company, we just gave them all our data for analysis" as an excuse? If you accept that then it means that we need to investigate and scrutinize the whole supply chain of any product before trusting it and it would also mean that any company switching even the smallest part of their supply chain would require us to reevaluate their product. When we buy things (either products, services or software) we consider the company selling it to be responsible for the whole and deferring responsibility to a vendor is basically just saying that you were either promising more than you knew could deliver or that you did not do a good job picking vendors.
    - greedo 5y ago
      It used to be "No one got fired for buying IBM." Now it's AWS. Many CTOs are very risk averse, and having someone to blame is incredibly valuable to them. My company has a large multi-region set of datacenters, but we're using AWS for a lot of new projects, despite being able to deliver similar services with better performance and cost, on-prem. Some of the reasons are no one wants to be seen using old tech; stuck being the last COBOL dev. So there's a big incentive for developers and their managers to be using the latest tech; it looks good on the devs resume and grants them job security in the field, and it makes their managers look like they're sharp and on top of new technology. Also, recruiting people familiar with AWS services is often easier than finding someone versed in on-prem tech. And for companies in less than desirable locations, this helps remove a staffing issue since it can all be done remotely. And when something goes wrong on-prem, the CTO is expected by the rest of the C-suite to fix the issue. When it's a reputable 3rd party, he can absolve both himself and his team of responsibility.
gerbilly 5y ago
Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead. Some of us old timers argued against this only to be shouted down by people who don't even understand TCP/IP. The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design. Today we'd just cruft together an MVP, solve only the interesting parts (or more likely the easy parts) and then move on, letting dashboards which lie to cover it up. Many of us have been taking shots at those 'big boys' since the start of this trend. Now that because of recent events and we have a chance to be heard you're telling us to be quiet. Why?What are you afraid of?
- gtirloni 5y ago
  Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.
  - ok123456 5y ago
    It doesn't sound like the "cloud" is working for anyone right now innit?
    - gtirloni 5y ago
      You're just trolling. "The cloud" doesn't exist as a single entity. I'm sure people in other various AWS regions, all GCP, all Azure, all DigitalOcean is working just fine.
  - gerbilly 5y ago
ignoramous 5y ago
> oh boy! Time to rethink this whole cloud thing! The problem isn't the cloud, the problem is all of the web properties going down all at once, exacerbated by the fact that AWS charges a premium for any viable cross-region / multi-cloud architecture (given its relatively high egress fees). This is discounting the inter-dependence among AWS services themselves, some of which aren't multi-region (afaik).
dimmke 5y ago
Thank you for saying this.
deleted 5y ago
[deleted]
- reidfromhome 5y ago
  There never was an outage. It was just the marketing home page for aws.amazon.com that was down for about 30 minutes and everyone had a knee jerk reaction to “haha AWS down went down again!!” Everything that mattered works fine.
RIMR 5y ago
I pray everyone in this thread contributes to the conversation more than you have with whatever this whiney nonsense is.
deleted 5y ago
[deleted]
PragmaticPulp 5y ago
> I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Engineers like to build stuff. Historically, running your own infrastructure and configuring servers from the ground up were the building blocks of the internet. These new cloud services took away those building blocks and they had the nerve to charge for it. In the real world, there are more interesting problems to solve than setting up and maintaining your own servers so it isn't really a complaint. It's actually more fun to get the hosting stuff out of the way and focus on the problem at hand. HN comments are basically notorious for exaggerating the downsides of hosted services while overplaying the ease and benefits of DIY. A lot of the comments here are similar to the famous Dropbox comment thread where users couldn't understand why anyone would want to use Dropbox when they could simply set up a complicated self-hosted rsync contraption to sync their own files.
emptybottle 5y ago
It's not "weasels" scampering out of the wood work. This is a community of highly technical users expressing legitimate frustration and concern that the biggest player in cloud hosting is a) having a service impacting event, and b) the status page says everything is up That's why people talk about the beneits of alternate hosting arrangements. If the status page said what was happening, there would be no need to ask "is AWS down?" the status would be clear.
unethical_ban 5y ago
You're correct that the knee jerk reactions are a bit over the top, but what is so interesting is how little the benefits of cloud are discussed here on HN. People who haven't used cloud native services think it's just EC2 and someone else's computer. But the idea of compute and data all being controlled through a single set of APIs, controlled by the same security syntaxes, is pretty incredible. Of course I'm a fan of the idea of on-prem alternatives, but the unified architecture is nice.
- gerbilly 5y ago
  > controlled by the same company FTFY
doublerabbit 5y ago
Serverless is just buzzword for a container on a virtual machine on a server. The Cloud is just a buzzword for a company with a data centre selling virtual servers from other big servers. > You’re either so new to this stuff to have no experience to remember the days before cloud I do, and it was much more peaceful. Everything you can do on "in the cloud" you can do in colocation. Which in the long-run is cheaper, more secure* and it's yours! Including the data. There are caveats, network, component failure. Investment in to these and you can have a pretty king setup. The cloud enabled magnitude of email spam, brute-forcing, botnets, security vulnerabilities and much more. Operators are lazy don't want to combat it. Providers are bias, then again you can say that about any business. People flock to the cloud like it's the greatest thing, when all your buying in to is a expensive price-plan for a company who will happily knock you off their service if you somehow brush up the wrong way and then charge you for a closed account. I do laugh when something goes wrong for FANG. Partly, I'm cynical and want to see the world burn, but it's also these companies exploit their userbase, their staff and the environments resources. When Facebook locked themselves out of their own offices due to the BGP issue, now that's funny. My colocation costs are: $5000 covers for a three-year 1Gbit 2u server racking space in two different DCs. Where I have full-control, as many services as I desire, allowed to host what I desire and where the internet space is actually mine. It may be a small cube of internet but I know it's my network, my traffic. Cloud is whitewash for me, I won't buy in to it. It has it purposes and if your happy with it, fine. But for me Colo for life.
- gerbilly 5y ago
  Preach it brother! Also, you should consider renting out space on your cube. You could even design APIs around the internet protocols so they can create their own DNS records, mailboxes. Wait...
hintymad 5y ago
I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up. Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud. Now, how many people think they can do better than Uber?
- greedo 5y ago
  Most of those issues sound like management/process issues, not technical issues. Our staff can't provision AWS instances without extensive review...
dudeinjapan 5y ago
Disagree on two counts. 1) Like any utility, the cloud is predicated on the assumption that it never goes down. If AWS itself goes down, it causes knock-on effects more most mid-sized (i.e. 1 region, multi-AZ) services on their platform, so we can't have it. 2) If we're going to have to accept downtime (1) then yes, it IS time we rethink the cloud--either how we can achieve no downtime, or how we achieve fault-tolerance in some other way.

stjohnswarts 5y ago

There is nothing wrong with pointing out the warts. I wonder if you've ever been at the mercy of google/aws and their lack of customer service if you aren't representing the business $$ of a fortune 500 company?

cosmotic 5y ago

I don't think anyone is saying they could do better than AWS at making their own infrastructure. I think the complaint is that it's a single point of failure for a huge swath of the internet.

technobabbler 5y ago

Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem. Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

christophilus 5y ago
I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.
londons_explore 5y ago
I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend. Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.
- technobabbler 5y ago
  If you're going closer-to-the-metal cloud, then yeah. I think that's part of the allure of newer, more abstracted cloud hosts like Vercel, Netlify, Gatsby, etc. They take care of all the infrastructure for ya, and if something breaks, you just twiddle your thumbs while they twiddle the dials. Of course this sort of laziness is not a valid approach if you're a huge enterprise or running some live-saving project that needs seventy nines of uptime, but for non-critical websites, it's a great sigh of relief...
ksec 5y ago
>making downtime somebody else's problem. It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.
- tuldia 5y ago
  > You can now simply point to AWS is down Heck, if that was even possible... "everything is green" dashboards... :-)
  - technobabbler 5y ago
    Obviously, the answer is to pay for a cloud dashboard-of-dashboards that show orange lights when AWS's is falsely green. "We need this new fancy dashboard to monitor our other dashboards in one place, more accurately!"
- SahAssar 5y ago
  Do people not take responsibility for their choice of vendors? If boeing crashed a bunch of planes and said "Not our fault, our subcontractor made shitty software" would we accept it? If a startup leaked all their customers data would we accept "Blame this other company, we just gave them all our data for analysis" as an excuse? If you accept that then it means that we need to investigate and scrutinize the whole supply chain of any product before trusting it and it would also mean that any company switching even the smallest part of their supply chain would require us to reevaluate their product. When we buy things (either products, services or software) we consider the company selling it to be responsible for the whole and deferring responsibility to a vendor is basically just saying that you were either promising more than you knew could deliver or that you did not do a good job picking vendors.
  - greedo 5y ago
    It used to be "No one got fired for buying IBM." Now it's AWS. Many CTOs are very risk averse, and having someone to blame is incredibly valuable to them. My company has a large multi-region set of datacenters, but we're using AWS for a lot of new projects, despite being able to deliver similar services with better performance and cost, on-prem. Some of the reasons are no one wants to be seen using old tech; stuck being the last COBOL dev. So there's a big incentive for developers and their managers to be using the latest tech; it looks good on the devs resume and grants them job security in the field, and it makes their managers look like they're sharp and on top of new technology. Also, recruiting people familiar with AWS services is often easier than finding someone versed in on-prem tech. And for companies in less than desirable locations, this helps remove a staffing issue since it can all be done remotely. And when something goes wrong on-prem, the CTO is expected by the rest of the C-suite to fix the issue. When it's a reputable 3rd party, he can absolve both himself and his team of responsibility.

gerbilly 5y ago

Well at least you said please, however: I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud. The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead. Some of us old timers argued against this only to be shouted down by people who don't even understand TCP/IP. The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design. Today we'd just cruft together an MVP, solve only the interesting parts (or more likely the easy parts) and then move on, letting dashboards which lie to cover it up. Many of us have been taking shots at those 'big boys' since the start of this trend. Now that because of recent events and we have a chance to be heard you're telling us to be quiet. Why?What are you afraid of?

gtirloni 5y ago
Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.
- ok123456 5y ago
  It doesn't sound like the "cloud" is working for anyone right now innit?
  - gtirloni 5y ago
    You're just trolling. "The cloud" doesn't exist as a single entity. I'm sure people in other various AWS regions, all GCP, all Azure, all DigitalOcean is working just fine.
- gerbilly 5y ago

ignoramous 5y ago

> oh boy! Time to rethink this whole cloud thing! The problem isn't the cloud, the problem is all of the web properties going down all at once, exacerbated by the fact that AWS charges a premium for any viable cross-region / multi-cloud architecture (given its relatively high egress fees). This is discounting the inter-dependence among AWS services themselves, some of which aren't multi-region (afaik).

dimmke 5y ago

Thank you for saying this.

deleted 5y ago

[deleted]

reidfromhome 5y ago
There never was an outage. It was just the marketing home page for aws.amazon.com that was down for about 30 minutes and everyone had a knee jerk reaction to “haha AWS down went down again!!” Everything that mattered works fine.

RIMR 5y ago

I pray everyone in this thread contributes to the conversation more than you have with whatever this whiney nonsense is.

deleted 5y ago

[deleted]

PragmaticPulp 5y ago

> I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work. Engineers like to build stuff. Historically, running your own infrastructure and configuring servers from the ground up were the building blocks of the internet. These new cloud services took away those building blocks and they had the nerve to charge for it. In the real world, there are more interesting problems to solve than setting up and maintaining your own servers so it isn't really a complaint. It's actually more fun to get the hosting stuff out of the way and focus on the problem at hand. HN comments are basically notorious for exaggerating the downsides of hosted services while overplaying the ease and benefits of DIY. A lot of the comments here are similar to the famous Dropbox comment thread where users couldn't understand why anyone would want to use Dropbox when they could simply set up a complicated self-hosted rsync contraption to sync their own files.

emptybottle 5y ago

It's not "weasels" scampering out of the wood work. This is a community of highly technical users expressing legitimate frustration and concern that the biggest player in cloud hosting is a) having a service impacting event, and b) the status page says everything is up That's why people talk about the beneits of alternate hosting arrangements. If the status page said what was happening, there would be no need to ask "is AWS down?" the status would be clear.

unethical_ban 5y ago

You're correct that the knee jerk reactions are a bit over the top, but what is so interesting is how little the benefits of cloud are discussed here on HN. People who haven't used cloud native services think it's just EC2 and someone else's computer. But the idea of compute and data all being controlled through a single set of APIs, controlled by the same security syntaxes, is pretty incredible. Of course I'm a fan of the idea of on-prem alternatives, but the unified architecture is nice.

gerbilly 5y ago
> controlled by the same company FTFY

doublerabbit 5y ago

Serverless is just buzzword for a container on a virtual machine on a server. The Cloud is just a buzzword for a company with a data centre selling virtual servers from other big servers. > You’re either so new to this stuff to have no experience to remember the days before cloud I do, and it was much more peaceful. Everything you can do on "in the cloud" you can do in colocation. Which in the long-run is cheaper, more secure* and it's yours! Including the data. There are caveats, network, component failure. Investment in to these and you can have a pretty king setup. The cloud enabled magnitude of email spam, brute-forcing, botnets, security vulnerabilities and much more. Operators are lazy don't want to combat it. Providers are bias, then again you can say that about any business. People flock to the cloud like it's the greatest thing, when all your buying in to is a expensive price-plan for a company who will happily knock you off their service if you somehow brush up the wrong way and then charge you for a closed account. I do laugh when something goes wrong for FANG. Partly, I'm cynical and want to see the world burn, but it's also these companies exploit their userbase, their staff and the environments resources. When Facebook locked themselves out of their own offices due to the BGP issue, now that's funny. My colocation costs are: $5000 covers for a three-year 1Gbit 2u server racking space in two different DCs. Where I have full-control, as many services as I desire, allowed to host what I desire and where the internet space is actually mine. It may be a small cube of internet but I know it's my network, my traffic. Cloud is whitewash for me, I won't buy in to it. It has it purposes and if your happy with it, fine. But for me Colo for life.

gerbilly 5y ago
Preach it brother! Also, you should consider renting out space on your cube. You could even design APIs around the internet protocols so they can create their own DNS records, mailboxes. Wait...

hintymad 5y ago

I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up. Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud. Now, how many people think they can do better than Uber?

greedo 5y ago
Most of those issues sound like management/process issues, not technical issues. Our staff can't provision AWS instances without extensive review...

dudeinjapan 5y ago

Disagree on two counts. 1) Like any utility, the cloud is predicated on the assumption that it never goes down. If AWS itself goes down, it causes knock-on effects more most mid-sized (i.e. 1 region, multi-AZ) services on their platform, so we can't have it. 2) If we're going to have to accept downtime (1) then yes, it IS time we rethink the cloud--either how we can achieve no downtime, or how we achieve fault-tolerance in some other way.