4 ms·

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar

• WJW 6y ago
  The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.
  • rorykoehler 6y ago
    AWS is pretty good whenever we've needed them. Google? Probably not.
    • jjoonathan 6y ago
      Google swears gcloud support is good, of course, but I've never actually used it -- but then again, I've never actually needed it. Meanwhile, I need AWS support constantly because their entire platform is a gigantic social experiment in minimum viable products. How crusty are people willing to tolerate? Evidently: very, very crusty.
      • batter 6y ago
        Vice versa for us. Google always has troubles and outages and AWS worked fine. Not sure why.
      • rorykoehler 6y ago
        We don't use most of the platform but the stuff we do use is rock solid (VPC, EC2, RDS, Route53, S3 etc). I do sometimes wonder what the point is for half the stuff they release. What I do know about GCP is they had a production bug in their tooling that was breaking everything for literally 100s of customers and they never even bothered replying to the bug report on their support forums. That experience along with the general modus operandi of shutting things down that don't further their surveillance capitalism business model means I won't be trying them again.
        • deleted 6y ago
          [deleted]
        • bmurphy1976 6y ago
          > I do sometimes wonder what the point is for half the stuff they release. I'm convinced they mvp every possible idea because it makes their platform more sticky. The more services you use, the harder it is to leave them for something better. The problem with that is you get 100 half dead zombie services and it's really hard to know which services are actually supported and which aren't. It's the Amazon equivalent of Google creating 10 different messenger applications only Amazon never kills an old service, they just let it rot forever.
          • freehunter 6y ago
            It almost seems to me (as an outsider) that Amazon has a policy that any tool they build internally or for a specific customer also has to be offered to the public on AWS. Like their managed satellite service or any number of things that make you say “I’ll bet some critical AWS thing was the first customer of this hyper-specific product”.
            • tidepod12 6y ago
              AFAIK this is pretty spot on. I don't know if it's "policy" or what, but based on talking with AWS account managers, AWS does not often (or ever) create custom AWS services/products for individual customers. They do however let customers submit feature requests, and then let customers lean heavily on AWS to implement those feature requests in the public AWS services. I think this is what leads to a lot of MVP-type services. A large customer clamors for some individual niche feature, AWS implements a MVP version of it, and then the team moves on to the next feature request (which might be for a completely different service, leaving the MVP in a perpetual MVP state).
    • dijit 6y ago
      I launched a large AAA game backed by a hybrid (GCE+Physical) infrastructure and I would swear by their support. Yes, we paid for enterprise support, but it's especially good. They even contributed code to third-party open source projects to solve one of our bugs. I hear often about google's support being terrible, but the enterprise support on google cloud is definitely an exception. F.D: Aside from giving a talk at google stockholm once, I am not affiliated in any way.
  • sudhirj 6y ago
    This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.
    • bigiain 6y ago
      Slack's problem though had nothing to do with racks going down. I'm not convinced Amazon's team is immune from the sort of complex failure mode described here. 'll bet there's people with equivalent sorts of stories about where edge cases in service interactions (either their own set of Lambda services or the AWS ones behind them, or more likely both) lead to a similar unexpected failure cascade.
      • jjoonathan 6y ago
        > I'm not convinced Amazon's team is immune from the sort of complex failure mode You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Like, hourly cron job doubling / halving within narrow safety rails primitive, where it's not merely possible to find a load that trips it up, it's all but inevitable. This varies by service, but they always project an image of their infrastructure being rather smart, and in the cases where I've been able to make an informed guess about what's actually going on, it's usually wildly inconsistent with the marketing. They don't warn you about the stinkers and even on services with good autoscaling and no true incompatibility between AWS's hidden choices and your needs, your scaling journey will involve periodic downtime as you trip over hidden built in limits and have to beg support to raise them. Sometimes you get curiously high resistance to this, leading to the impression that these aren't so much "safety rails" as hardcoded choices. Oh, and just last week we managed to completely wedge a service. The combination of a low limit on in-flight processes, two hung processes, immutability on running processes, and delete functionality being predicated on proper termination led to a situation where an AWS service became completely unusable for days while we begged support to log in and clear the hung process. Naturally, this isn't going to count as downtime on any reliability charts, even though it's a known problem and definitely looked a lot like downtime on our end. We're a small (<10) team with modest needs. AWS lets us do some crazy awesome things, but it really bugs me how reliably they over-promise and under-deliver.
• foota 6y ago
  What? You can totally run disturbed SQL without data loss. Did you mean to imply something in particular?
  • bigiain 6y ago
    CAP theorem strongly suggests if you optimise against data loss you have to trade off something else which is probably important too...
  • rutthenut 6y ago
    'Disturbed' SQL confused me, had to look up the thread to see Distributed SQL. And yes, I agree, you certainly can run distributed without data loss, depending on the platform design and your 'transaction level'.
• mcbain 6y ago
  Could you describe what you mean by a distributed SQL database? I don’t think RDS would generally fit that concept as I understand it. Aurora’s data store possibly but you choose to use that specifically.
• aflag 6y ago
  Though they do have good availability, they are certainly not infallible and have been down for hours or even days. And, when that happens, all you can do is pray they get to it soon. You have no control over when they will make riskier changes or how fast they will be able to respond. If they fail to respect the SLA, the maximum they are going to do is giving your money back for your services. Obviously, there are good aspects of outsourcing devops/admin work. It's a tradeoff, as most things. If you are a struggling startup it's difficult to justify the cost of owning a bunch of hardware and hiring expensive infra people to manage it. However, facebook is probably better off owning their own infraestructure.
  • saurik 6y ago
    FWIW, that time AWS had a massive massive failure it was actually really convenient being on AWS as so much of the Internet was down at the same time that pretty much no one was upset at me for my one service also being offline.
    • aflag 6y ago
      I think it depends on your competitors and what your software does. If your competitors are still running their services but not you, you're bound to lose some customers. Also, regardless of that, you can easily be losing money while the service is down. For instance, if you have a food delivery app and it's down, while your competitor is up, people will just use the other app and you're going to lose money while AWS is down. Even worse, they may actually end up liking the competitor's app better, and you can even lose customers that way. However, if you're looking from the point of view of an employee trying to justify to your boss why the service is down. I'm sure they'll be more understanding that's not your fault in this case. It's a risk they decided to take.
  • Nextgrid 6y ago
    This is an underrated point. The problem with AWS and those massive services is that they are "alive" and continuously evolving. In most cases it's fine and you don't notice, but when it goes wrong it can affect you and you have no control over when that is likely to happen. In contrast with in-house infrastructure, you can make your stack as simple or as complex as you'd like depending on your needs (a lot of projects can get away with a handful of physical machines all configured manually, no Terraform/Kubernetes/etc) and you control when you make drastic changes that risk breaking things so you can plan them during a time when downtime would be the least damaging to your business.