5 ms·

Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network set

by yeukhon 11y ago

Disagree. It is hard to maintain codebase and consistent infrastructure setup for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network setup and configuration are also nightmare. Speaking from experience dealing with four cloud vendors at once. It sounds great from a textbook perspective, but unless you are ready to spend millions every year to fight fire, please don't do that. If you were to use AWS, please build on multiple regions.

mwcampbell 11y ago
I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.
- yeukhon 11y ago
  Human capital cost. Also, infrastructure is expensive by nature. We managed to run four at once with a team of 12 people. As I said before, vendor X ~= vendor Y in some way, so your deal and the service you get are not the same, so there is cost associated with that also. There is a reason to pick a good strong vendor and stick with it. Big names end up building their own data center (they can probably capx it for tax purpose). Small to medium usually don't have time to deal with multiple vendors. Try to be vendor agnostic is really great but again, even in the world of open stack, you either manage all of your open stack installation yourself (which is expensive), or you end up one vendor running one version behind, or not offering 100% feature. I think it is more worthwhile to first complete automation and self-healing in one vendor, before thinking leaping to another one. It took so many engineers at Netflix to build a resilient AWS environments. VMs go down all the time, more often than AWS pushing some bad commits and cause half of their native services go down. There is so much more to engineer in other areas and those are more likely to put you out of service first, so unless you want to all your team dealing with fires every day.... we were putting band-aids together most of the time.
  - Dylan16807 11y ago
    12 people whose sole, full-time job is to install the same OS on multiple vendors and make the networks look the same? I'm having trouble imagining what problems have to be constantly dealt with such that multiple millions of dollars have to go into abstracting the multiple vendors.
    - yeukhon 11y ago
      I can go on and give you the full story, but here is the gist, consolidation is the first step of real resiliency. I don't know what you do in your day job, but you weren't in my position and let me tell you, it was a nightmare to pull off in that situation. I am happy we are dealing with AWS. We can mirror tools to actually make two VPCs alike. Human cost is not cheap, people have to work over time in order put out fire, people rarely worked eight hours a day, we were all consultants.
      - Dylan16807 11y ago
        I know it's extremely hard to make a resilient system, but I don't understand how using multiple vendors makes it more complicated. Just on AWS you already have computers in different datacenters. How much harm can it really cause that datacenter 5 has 20% more ram on the nearly-identical machines, and you have to connect the network differently? If you're spending millions of dollars on making systems redundant, I completely understand. But that is not the same as spending millions of dollars to allow datacenters 4-7 to be on other vendors. Maybe if you name some problems specific to multi-vendor support that took several man-months to fix I might comprehend. Just a sentence or two description. Thanks if you do, sorry for being confused if you don't.
        yeukhon 11y ago
        I want to say I know the pain of vendor lock-in. We use an open source PaaS solution and it works well for the most part but the solution doesn't really scale the way I want it to be, and doesn't provide all the utility I consider to be crucial for a successful PaaS. Whether moving away from it has always been an interesting discussion at work. So I and my collagues are careful now with what to include in our technology stack pofotilo. I can't just have someone randomly decide to add a Redis to the dependency when the purpose is just a key-value store for some trivial task. Anyway, the problem started with bad management to be honest, which I don't think my case is too rare to hear. When the first project rolled in, we ran PaaS on a single provider, but more projects joined, it was time to choose another vendor because of better equipment and better deals. Yet, none of them really provide good resiliency. With the third vendor probably around 12-18 physical servers were purchased and managed by the vendor. It was openstack. That version of OpenStack at the time was not compatible with AWS, which later becomes our 4th vendor, and runs our dev environment. You want to run additional performance testing, well, we can't get the same elasticity as AWS because we only have so many physical servers to provision VMs, but that's where our real data lives. So we had to do our QA on AWS and doing the data copy can take a whole business day. The script we wrote for Amazon doesn't work on OpenStack. Security, network, I/O metrics are not consistent across vendors. That adds complexity to code. I know exactly how to abstract things, but just trust me :-) it really makes code hard to maintain, and really painful to integrate with multiple vendors. As a matter of fact, I don't like working with AWS API (with boto) myself because API response formats are inconsistent! Finally, we got rid of one of them, then two and finally we are on the final stage of consolidating everything on AWS and focus on infrastructure automation and lower the number of chores. Like many projects out there, things usually start out real nice, but then once you get too busy fighting fire here and there, you will accumulate some debts. If one is not careful, the debt can backfire and we had our lessons. There is no resiliency in most of the other vendors because they require you to purchase more servers and they themselves have hard time to go true elasticity. AWS, at least, for the most part, doesn't run out of instance availability that often (it happened a few times to our EMR processing). I do have some issues with AWS myself, but so far, AWS seems to be the only true cloud provider you can hang on to for several years. Before building a grand multi-vendor infrastructure, build on a single-vendor well, then decide on the next step.
- scurvy 11y ago
  Do you run Postgresql, MySQL, or something else? What led you to your choice? Because one is inherently better than the others for what you want to do? That's why you stick with a single cloud provider. You want the vendor lock-in. You go whole hog into using everything that they provide. Once you start mixing and matching the best of breed across cloud providers, you've lost. Once you start coding to the least common denominator amongst cloud providers, you've lost. To really succeed, you must choose the best cloud offering and then use everything that they offer. Otherwise, why are you running stuff in the cloud anyway? Also, steer clear of "cloud neutral" services and products that will magically move data and services across cloud providers. Interop is the last thing on any proprietary vendor's mind. An example. You have a pair of border edge routers. Do you buy 2 Junipers, 2 Ciscos, or 1 Juniper and 1 Cisco for fear of a bad vendor bug taking out all the routers? I'll tell you which one I would not choose. The .com TLD nameserver requirements used to mandate dual vendor setups. They sure did learn their lesson.
  - antsar 11y ago
    > They sure did learn their lesson. Is there any relevant reading material? I would have assumed diversification to be a good idea for such critical infra.
    - scurvy 11y ago
      I don't know if there is any reading material. I heard the story from someone who was on the front lines in the outage, directly dealing with the situation. Vendor diversification is a bad thing for critical infrastructure when interoperation is required. "It's a Cisco problem!" says the Juniper rep. "It's a Juniper problem!" says the Cisco rep. You're stuck in the middle. It's terrible. You can only hold one vendor's feet to the fire and they won't care at all if you're in a heterogeneous environment. Remember all the middleware products and companies from the late 90's? Neither do I.
  - X-Istence 11y ago
    Comcast as a company purchases from two vendors to avoid lock-in and to get the best prices. Juniper and Cisco, although last I heard Arista was also starting to be used. Dual-sourcing makes sense in a lot of cases.
mark-ruwt 11y ago
I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), because they're all running the same distro. A whole data center can go down, and there's no reason for me to get out of bed.
- mwcampbell 11y ago
  Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.
  - mark-ruwt 11y ago
    No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from scratch, sending status emails throughout the build process.
    - mwcampbell 11y ago
      I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.
      - mark-ruwt 11y ago
        Ah, I get you. There's an HAProxy at the front of each data center "pod". If all servers within the pod go down, the proxy routes traffic to servers in another pod. If HAProxy itself goes down, a Route 53 health check just takes it out of the A Record rotation.
TheSwordsman 11y ago
We operate on multiple regions at PagerDuty. It's slightly painful, but I've nowhere near as bad an experience as you claim. We've done it using Chef to do the heavy lifting. It's worked pretty well. It takes a bit of work and determination, but it comes together in the end. One problem with building on AWS is their lack of network diversity. If there is a network cut, causing congested links, they will do little to try and alleviate congestion to save on cost. Issues in Oregon have caused network degradation between WEST-1 and WEST-2 that last for days with no improvement.

by yeukhon 11y ago

mwcampbell 11y ago

I have a hard time imagining how supporting multiple providers would cost millions of dollars per year. I think it's a worthwhile way to make your software and infrastructure more resilient. And it protects you from vendor lock-in. As long as you keep things simple and don't use features that are only implemented by one provider, VMs are basically interchangeable.

yeukhon 11y ago
Human capital cost. Also, infrastructure is expensive by nature. We managed to run four at once with a team of 12 people. As I said before, vendor X ~= vendor Y in some way, so your deal and the service you get are not the same, so there is cost associated with that also. There is a reason to pick a good strong vendor and stick with it. Big names end up building their own data center (they can probably capx it for tax purpose). Small to medium usually don't have time to deal with multiple vendors. Try to be vendor agnostic is really great but again, even in the world of open stack, you either manage all of your open stack installation yourself (which is expensive), or you end up one vendor running one version behind, or not offering 100% feature. I think it is more worthwhile to first complete automation and self-healing in one vendor, before thinking leaping to another one. It took so many engineers at Netflix to build a resilient AWS environments. VMs go down all the time, more often than AWS pushing some bad commits and cause half of their native services go down. There is so much more to engineer in other areas and those are more likely to put you out of service first, so unless you want to all your team dealing with fires every day.... we were putting band-aids together most of the time.
- Dylan16807 11y ago
  12 people whose sole, full-time job is to install the same OS on multiple vendors and make the networks look the same? I'm having trouble imagining what problems have to be constantly dealt with such that multiple millions of dollars have to go into abstracting the multiple vendors.
  - yeukhon 11y ago
    I can go on and give you the full story, but here is the gist, consolidation is the first step of real resiliency. I don't know what you do in your day job, but you weren't in my position and let me tell you, it was a nightmare to pull off in that situation. I am happy we are dealing with AWS. We can mirror tools to actually make two VPCs alike. Human cost is not cheap, people have to work over time in order put out fire, people rarely worked eight hours a day, we were all consultants.
    - Dylan16807 11y ago
      I know it's extremely hard to make a resilient system, but I don't understand how using multiple vendors makes it more complicated. Just on AWS you already have computers in different datacenters. How much harm can it really cause that datacenter 5 has 20% more ram on the nearly-identical machines, and you have to connect the network differently? If you're spending millions of dollars on making systems redundant, I completely understand. But that is not the same as spending millions of dollars to allow datacenters 4-7 to be on other vendors. Maybe if you name some problems specific to multi-vendor support that took several man-months to fix I might comprehend. Just a sentence or two description. Thanks if you do, sorry for being confused if you don't.
      - yeukhon 11y ago
        I want to say I know the pain of vendor lock-in. We use an open source PaaS solution and it works well for the most part but the solution doesn't really scale the way I want it to be, and doesn't provide all the utility I consider to be crucial for a successful PaaS. Whether moving away from it has always been an interesting discussion at work. So I and my collagues are careful now with what to include in our technology stack pofotilo. I can't just have someone randomly decide to add a Redis to the dependency when the purpose is just a key-value store for some trivial task. Anyway, the problem started with bad management to be honest, which I don't think my case is too rare to hear. When the first project rolled in, we ran PaaS on a single provider, but more projects joined, it was time to choose another vendor because of better equipment and better deals. Yet, none of them really provide good resiliency. With the third vendor probably around 12-18 physical servers were purchased and managed by the vendor. It was openstack. That version of OpenStack at the time was not compatible with AWS, which later becomes our 4th vendor, and runs our dev environment. You want to run additional performance testing, well, we can't get the same elasticity as AWS because we only have so many physical servers to provision VMs, but that's where our real data lives. So we had to do our QA on AWS and doing the data copy can take a whole business day. The script we wrote for Amazon doesn't work on OpenStack. Security, network, I/O metrics are not consistent across vendors. That adds complexity to code. I know exactly how to abstract things, but just trust me :-) it really makes code hard to maintain, and really painful to integrate with multiple vendors. As a matter of fact, I don't like working with AWS API (with boto) myself because API response formats are inconsistent! Finally, we got rid of one of them, then two and finally we are on the final stage of consolidating everything on AWS and focus on infrastructure automation and lower the number of chores. Like many projects out there, things usually start out real nice, but then once you get too busy fighting fire here and there, you will accumulate some debts. If one is not careful, the debt can backfire and we had our lessons. There is no resiliency in most of the other vendors because they require you to purchase more servers and they themselves have hard time to go true elasticity. AWS, at least, for the most part, doesn't run out of instance availability that often (it happened a few times to our EMR processing). I do have some issues with AWS myself, but so far, AWS seems to be the only true cloud provider you can hang on to for several years. Before building a grand multi-vendor infrastructure, build on a single-vendor well, then decide on the next step.
scurvy 11y ago
Do you run Postgresql, MySQL, or something else? What led you to your choice? Because one is inherently better than the others for what you want to do? That's why you stick with a single cloud provider. You want the vendor lock-in. You go whole hog into using everything that they provide. Once you start mixing and matching the best of breed across cloud providers, you've lost. Once you start coding to the least common denominator amongst cloud providers, you've lost. To really succeed, you must choose the best cloud offering and then use everything that they offer. Otherwise, why are you running stuff in the cloud anyway? Also, steer clear of "cloud neutral" services and products that will magically move data and services across cloud providers. Interop is the last thing on any proprietary vendor's mind. An example. You have a pair of border edge routers. Do you buy 2 Junipers, 2 Ciscos, or 1 Juniper and 1 Cisco for fear of a bad vendor bug taking out all the routers? I'll tell you which one I would not choose. The .com TLD nameserver requirements used to mandate dual vendor setups. They sure did learn their lesson.
- antsar 11y ago
  > They sure did learn their lesson. Is there any relevant reading material? I would have assumed diversification to be a good idea for such critical infra.
  - scurvy 11y ago
    I don't know if there is any reading material. I heard the story from someone who was on the front lines in the outage, directly dealing with the situation. Vendor diversification is a bad thing for critical infrastructure when interoperation is required. "It's a Cisco problem!" says the Juniper rep. "It's a Juniper problem!" says the Cisco rep. You're stuck in the middle. It's terrible. You can only hold one vendor's feet to the fire and they won't care at all if you're in a heterogeneous environment. Remember all the middleware products and companies from the late 90's? Neither do I.
- X-Istence 11y ago
  Comcast as a company purchases from two vendors to avoid lock-in and to get the best prices. Juniper and Cisco, although last I heard Arista was also starting to be used. Dual-sourcing makes sense in a lot of cases.

mark-ruwt 11y ago

I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), because they're all running the same distro. A whole data center can go down, and there's no reason for me to get out of bed.

mwcampbell 11y ago
Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.
- mark-ruwt 11y ago
  No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from scratch, sending status emails throughout the build process.
  - mwcampbell 11y ago
    I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.
    - mark-ruwt 11y ago
      Ah, I get you. There's an HAProxy at the front of each data center "pod". If all servers within the pod go down, the proxy routes traffic to servers in another pod. If HAProxy itself goes down, a Route 53 health check just takes it out of the A Record rotation.

TheSwordsman 11y ago

We operate on multiple regions at PagerDuty. It's slightly painful, but I've nowhere near as bad an experience as you claim. We've done it using Chef to do the heavy lifting. It's worked pretty well. It takes a bit of work and determination, but it comes together in the end. One problem with building on AWS is their lack of network diversity. If there is a network cut, causing congested links, they will do little to try and alleviate congestion to save on cost. Issues in Oregon have caused network degradation between WEST-1 and WEST-2 that last for days with no improvement.