6 ms·

I worry that moving to a larger company will just hasten the consolidation of hosting to a few players.

• deleted 11y ago
  [deleted]
  • nothrabannosir 11y ago
    He(/she?) never said he wouldn't do it, just that it worried him. Back seat doesn't mean thrown out of the car. People need to keep bringing this up, lest we completely forget about it and wake up when it's too late.
• swiley 11y ago
  The best thing for reliability is to use multiple companies. AWS has proved that multiple times.
  • yeukhon 11y ago
    Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to use AWS, please build on multiple regions.
    • mwcampbell 11y ago
      I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.
      • yeukhon 11y ago
        Human capital cost. Also, infrastructure is expensive by nature. We managed to run four at once with a team of 12 people. As I said before, vendor X ~= vendor Y in some way, so your deal and the service you get are not the same, so there is cost associated with that also. There is a reason to pick a good strong vendor and stick with it. Big names end up building their own data center (they can probably capx it for tax purpose). Small to medium usually don't have time to deal with multiple vendors. Try to be vendor agnostic is really great but again, even in the world of open stack, you either manage all of your open stack installation yourself (which is expensive), or you end up one vendor running one version behind, or not offering 100% feature. I think it is more worthwhile to first complete automation and self-healing in one vendor, before thinking leaping to another one. It took so many engineers at Netflix to build a resilient AWS environments. VMs go down all the time, more often than AWS pushing some bad commits and cause half of their native services go down. There is so much more to engineer in other areas and those are more likely to put you out of service first, so unless you want to all your team dealing with fires every day.... we were putting band-aids together most of the time.
        • Dylan16807 11y ago
          12 people whose sole, full-time job is to install the same OS on multiple vendors and make the networks look the same? I'm having trouble imagining what problems have to be constantly dealt with such that multiple millions of dollars have to go into abstracting the multiple vendors.
          • yeukhon 11y ago
            I can go on and give you the full story, but here is the gist, consolidation is the first step of real resiliency. I don't know what you do in your day job, but you weren't in my position and let me tell you, it was a nightmare to pull off in that situation. I am happy we are dealing with AWS. We can mirror tools to actually make two VPCs alike. Human cost is not cheap, people have to work over time in order put out fire, people rarely worked eight hours a day, we were all consultants.
    • mark-ruwt 11y ago
      I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), because they're all running the same distro. A whole data center can go down, and there's no reason for me to get out of bed.
      • mwcampbell 11y ago
        Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.
        • mark-ruwt 11y ago
          No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from scratch, sending status emails throughout the build process.
          • mwcampbell 11y ago
            I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.
            • mark-ruwt 11y ago
              Ah, I get you. There's an HAProxy at the front of each data center "pod". If all servers within the pod go down, the proxy routes traffic to servers in another pod. If HAProxy itself goes down, a Route 53 health check just takes it out of the A Record rotation.
    • TheSwordsman 11y ago
      We operate on multiple regions at PagerDuty. It's slightly painful, but I've nowhere near as bad an experience as you claim. We've done it using Chef to do the heavy lifting. It's worked pretty well. It takes a bit of work and determination, but it comes together in the end. One problem with building on AWS is their lack of network diversity. If there is a network cut, causing congested links, they will do little to try and alleviate congestion to save on cost. Issues in Oregon have caused network degradation between WEST-1 and WEST-2 that last for days with no improvement.
  • albertoleal 11y ago
    How does one set up to use multiple vendors?
    • ryanlol 11y ago
      You buy servers from multiple different companies, you install software on them.
      • albertoleal 11y ago
        Anyone can do that. I was actually wondering if there are software to automate this.
        • ryanlol 11y ago
          Openstack might allow you to do this. (I haven't used it myself.)
        • ultramancool 11y ago
          Ansible is probably the best bet - once you have SSH it'll pretty much work with anything and it also has support for creation of servers on several cloud providers and stuff. We use it for automatic configuration of everything from colocated hardware to $5/mo VMs on DigitalOcean and other low-cost virtual server providers and it works great.
        • rubiquity 11y ago
          You could use something like Terraform to help automate this. It's like CloudFormation but also supports GCE, Azure, DigitalOcean, etc. 0 - https://terraform.io/ https://terraform.io/
    • mcone 11y ago
      You might consider taking a look at http://www.cloud66.com/ http://www.cloud66.com/
      • doublerebel 11y ago
• beachstartup 11y ago
  this is going to happen no matter what - the rate of consolidation in my opinion is actually accelerating. despite what anyone says, the vast majority of people choose their hosting provider in order to "not get fired", i.e. they are extremely risk-averse, to the point of irrationality, even when presented with massive savings, better service, better underlying hardware and network, and more transparency. most of the time they are spending someone else's money, also. you should see some of the deals that have come across my desk. the 'big names' can literally charge 2-5x more than a competitive quote and get away with it, oftentimes with worse deliverables (i.e. long stretches of downtime that somehow get a pass from their customers). at the end of the day it's a server sitting in a rack in a datacenter, connected to ethernet. beyond a certain level of quality (tier 3 dc, server class hardware, an enterprise quality network) it's really all the same. people should choose their hosting provider on 1. quality of implementation 2. price and 3. whether or not the provider actually gives a shit about you and your account, but they usually just go with the name, like many other markets.
  • yeukhon 11y ago
    > most of the time they are spending someone else's money, also. you should see some of the deals that have come across my desk. the 'big names' can literally charge 2-5x more than a competitive quote and get away with it, oftentimes with worse deliverables (i.e. long stretches of downtime that somehow get a pass from their customers). Even AWS can charge you big time. As I am getting more and more familiar with AWS every day, the #1 thing on my list going forward is to to sit down with your TAM and organize architecture review. There are services on AWS lacking completeness and can bite you in the end if you go straight with it without knowing what you are getting into. Disclaimer: I like AWS pretty well.