5 ms·

I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It to

I think you're looking through the wrong end of the telescope, here. We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and Vultr. It took me two weeks to craft a ~400 line script that abstracted the server creation APIs for each. Once spun up, they're each bootstrapped with a script that builds each server from scratch identically regardless of the provider (with a couple one-offs for Vultr), because they're all running the same distro. A whole data center can go down, and there's no reason for me to get out of bed.

mwcampbell 11y ago
Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.
- mark-ruwt 11y ago
  No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from scratch, sending status emails throughout the build process.
  - mwcampbell 11y ago
    I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.
    - mark-ruwt 11y ago
      Ah, I get you. There's an HAProxy at the front of each data center "pod". If all servers within the pod go down, the proxy routes traffic to servers in another pod. If HAProxy itself goes down, a Route 53 health check just takes it out of the A Record rotation.
yeukhon 11y ago
Maybe, maybe not. I still believe in single-vendor approach, perphas because in my view I am either going for AWS or GCE. There is so much to with than just be able to spin up an VM and then run Ansible/Chef/Puppet on it. Heck I can write all of that in Fabric. There is no direct connect on Digital Ocean. I am not sure how you set up VPN with Digital Ocean or Linode. We use cloudformation on AWS, and I am pretty sure there is no such thing on Linode or Digital Ocean. Exception and response codes different across providers. Able to reproduce an environment from scratch is important to us, and of course, we try to do that in stages. I own a DO box myself, and that box turns out to be really slow in the NY region (where I live), maybe I am just an lucky bastard. But to be honest, did you really build your entire infrastructure in three vendors to begin with? What are your reasons to really build on Linode, Digital Ocean and Vultr? How do you copy your data across environments? Are you splitting dev/qa/ci/sandbox/stage/prod?
- mark-ruwt 11y ago
  No, not at all! http://areyouwatchingthis.com http://areyouwatchingthis.com is almost 10 years old at this point, and the architecture in its current form wasn't solidified until last year. As a one-man shop, redundancy and failover are my best friends.

mwcampbell 11y ago

Have you considered selling that system in some form? Presumably it has to be tailored to the specific software stack, particularly the database(s) and other persistent storage. But maybe you could write an ebook on how to set up that kind of system, with examples from your scripts, and sell that.

mark-ruwt 11y ago
No, no thoughts to sell it. I'd open-source it before I'd do that. It's not complicated when you break it up into two steps. Step 1: An if/else-heavy script that will take in a few parameters (for us it's city, a server type, and a numeral for naming) and build a clean server with all of the needed keys populated. Step 2: A "yum install"-heavy script passed into the clean server, that builds everything needed from scratch, sending status emails throughout the build process.
- mwcampbell 11y ago
  I figured the tricky part would be automatic failover to a different data center when one goes offline. At least, I inferred that you had implemented that, from the part about you not having to get out of bed if a data center goes offline.
  - mark-ruwt 11y ago
    Ah, I get you. There's an HAProxy at the front of each data center "pod". If all servers within the pod go down, the proxy routes traffic to servers in another pod. If HAProxy itself goes down, a Route 53 health check just takes it out of the A Record rotation.

yeukhon 11y ago

Maybe, maybe not. I still believe in single-vendor approach, perphas because in my view I am either going for AWS or GCE. There is so much to with than just be able to spin up an VM and then run Ansible/Chef/Puppet on it. Heck I can write all of that in Fabric. There is no direct connect on Digital Ocean. I am not sure how you set up VPN with Digital Ocean or Linode. We use cloudformation on AWS, and I am pretty sure there is no such thing on Linode or Digital Ocean. Exception and response codes different across providers. Able to reproduce an environment from scratch is important to us, and of course, we try to do that in stages. I own a DO box myself, and that box turns out to be really slow in the NY region (where I live), maybe I am just an lucky bastard. But to be honest, did you really build your entire infrastructure in three vendors to begin with? What are your reasons to really build on Linode, Digital Ocean and Vultr? How do you copy your data across environments? Are you splitting dev/qa/ci/sandbox/stage/prod?

mark-ruwt 11y ago
No, not at all! http://areyouwatchingthis.com http://areyouwatchingthis.com is almost 10 years old at this point, and the architecture in its current form wasn't solidified until last year. As a one-man shop, redundancy and failover are my best friends.