4 ms·

What would you consider to be cheaper than AWS?

mark-ruwt 10y ago
We were exclusively on AWS, but now are distributed across Linode, DigitalOcean, and Vultr. We're saving a lot, and getting better performance.
- beat 10y ago
  Can you share a little what your stack looks like? And how are you doing infrastructure automation to build and reproduce it? I'm really interested in where you found ways to cut costs, and what kind of effort was involved.
  - mark-ruwt 10y ago
    Sure, think of it this way: If you had a perfectly clean install of your distro of choice, could you write a shell script that could build your server from scratch? If you can answer Yes to that (and you should), then you can build an if/then-heavy shell script that works with each of the APIs to create a perfectly clean install of your favorite distro. One script to create the clean slate machine. One script to build what you need. We have 9 "pods" around the globe, each with API servers (Java/Tomcat/Apache), static servers (Varnish), a MySQL slave, and an HAProxy Maître D'. With close to 60 servers, our monthly bills are less than $1,000, and we haven't had downtime in years. Spinning up a new server is just: sh build.sh atl api 1. Feel free to ping me if you want any more details: mark@areyouwatchingthis.com.
    - mrsirduke 10y ago
      How do you tie things together and handle cross cloud failover?
      - mark-ruwt 10y ago
        I use AWS Route 53 DNS with Health Checks, and put one "A" record in for each "pod". If an entire pod somehow disappears, Route 53 will take it out of the rotation.
  - beachstartup 10y ago
    trust me, no matter how you slice it, it's a shitload of effort. that is exactly how AWS makes a ton of money. they make hard things easy (and expensive). do you want the pain, or do you want the money, that's the basic proposition here. when you start looking at 25, 50, 150k/month of savings by doing stuff yourself, the choice becomes much clearer. in many cases i've seen, you could theoretically hire an entire team to take care of the stuff that AWS does for you, and still come out ahead.
  - jen20 10y ago
    Terraform is not AWS-specific - there are 30-odd providers covering most major cloud services and many SaaS systems also. It can be valuable for multi-cloud orchestration!
beat 10y ago
If all you want is stuff running on the equivalent of EC2 instances, you can use DigitalOcean or some other competitor. However, AWS offers a degree of depth and sophistication that just isn't available from the competition. Trying to do something that is as security-clean as this VPC using DO or Linode or something, from scratch, sounds like weeks of hell to me. "Not Invented Here" is a big problem in this industry in general. We tend to be too comfortable cobbling a solution together from stone knives and bearskins, rather than using someone else's solution (and paying for it). If you are running a business, though, you shouldn't be building things that aren't what you sell, unless you really cannot otherwise buy them.
- ethbro 10y ago
  > If you are running a business, though, you shouldn't be building things that aren't what you sell [or that you can't build more efficiently], unless you really cannot otherwise buy them. I'd add the above. Otherwise AWS would have never been built.
  - beat 10y ago
    Indeed. Remember, AWS started as Amazon providing infrastructure as a service to internal projects, so individual teams and projects wouldn't have to go buy a bunch of hardware and ops staff in order to build products. Amazon itself is the biggest customer AWS has. Thanks to economies of scale, it's worthwhile for Amazon to develop new features as full-scale products, which leads to truly amazing things like Lambda. No one in their right mind would develop Lambda in-house, but for AWS, it makes a ton of sense.
    - tootie 10y ago
      I think Netflix is their biggest customer.
  - beat 10y ago
    I'd add, though, that "efficiently" isn't just whether you can build cheaper than buying. It's if the cost of building the functionality is cheaper than the profit/growth you can generate with a sellable product built using the same amount of developer effort. That's a very, very different (and probably more expensive) proposition. That's why executives should make the decisions, not engineers! Back in the dot-com days, I worked on a project to build some functionality in-house that we could easily have bought off the shelf. The engineers argued that we'd save the company a million dollars. But frankly, we just wanted to do it because it was badass. And it turned out our solution would actually have cost us more per-system than the commercial solution we sneered at (hardware costs, not just development cost). Six man-months of engineering when everyone knew we were racing the clock before the money ran out? Absolute stupidity. If I were CEO/CTO and caught wind of such a project, I'd tell people that if they lifted a finger on it, they'd be fired. But that's a very different perspective than I had back then. Risking the very existence of what could have been a very big company in order to someday save a million dollars? Feh. (Of course, no one stopped us, because the CTO was just head nerd, and the money execs were busy fundraising rather than supervising)
Thaxll 10y ago
baremetal!
- bespoke_engnr 10y ago
  I'm assuming that this was originally sarcastic, but bare metal can actually turn out really well in some situations. If you're big enough to save just $50k/month by moving from AWS to physical machines, that's several System Engineers you can hire to do your day-to-day ops work and maintenance on your Ansible/whatever scripts. The thing that's more difficult to do on physical hardware, obviously, is scaling down every day if your peak load is some insane multiple of your base load for a 24h cycle. That's where AWS makes a lot of sense. I don't think there's an obvious 'best' answer -- it depends on what you need. It just bugs me that the current state of Ops/Infrastructure sometimes looks so much like the worst parts of the Javascript ecosystem.
deleted 10y ago
[deleted]

We were exclusively on AWS, but now are distributed across Linode, DigitalOcean, and Vultr. We're saving a lot, and getting better performance.

beat 10y ago
Can you share a little what your stack looks like? And how are you doing infrastructure automation to build and reproduce it? I'm really interested in where you found ways to cut costs, and what kind of effort was involved.
- mark-ruwt 10y ago
  Sure, think of it this way: If you had a perfectly clean install of your distro of choice, could you write a shell script that could build your server from scratch? If you can answer Yes to that (and you should), then you can build an if/then-heavy shell script that works with each of the APIs to create a perfectly clean install of your favorite distro. One script to create the clean slate machine. One script to build what you need. We have 9 "pods" around the globe, each with API servers (Java/Tomcat/Apache), static servers (Varnish), a MySQL slave, and an HAProxy Maître D'. With close to 60 servers, our monthly bills are less than $1,000, and we haven't had downtime in years. Spinning up a new server is just: sh build.sh atl api 1. Feel free to ping me if you want any more details: mark@areyouwatchingthis.com.
  - mrsirduke 10y ago
    How do you tie things together and handle cross cloud failover?
    - mark-ruwt 10y ago
      I use AWS Route 53 DNS with Health Checks, and put one "A" record in for each "pod". If an entire pod somehow disappears, Route 53 will take it out of the rotation.
- beachstartup 10y ago
  trust me, no matter how you slice it, it's a shitload of effort. that is exactly how AWS makes a ton of money. they make hard things easy (and expensive). do you want the pain, or do you want the money, that's the basic proposition here. when you start looking at 25, 50, 150k/month of savings by doing stuff yourself, the choice becomes much clearer. in many cases i've seen, you could theoretically hire an entire team to take care of the stuff that AWS does for you, and still come out ahead.
- jen20 10y ago
  Terraform is not AWS-specific - there are 30-odd providers covering most major cloud services and many SaaS systems also. It can be valuable for multi-cloud orchestration!

beat 10y ago

If all you want is stuff running on the equivalent of EC2 instances, you can use DigitalOcean or some other competitor. However, AWS offers a degree of depth and sophistication that just isn't available from the competition. Trying to do something that is as security-clean as this VPC using DO or Linode or something, from scratch, sounds like weeks of hell to me. "Not Invented Here" is a big problem in this industry in general. We tend to be too comfortable cobbling a solution together from stone knives and bearskins, rather than using someone else's solution (and paying for it). If you are running a business, though, you shouldn't be building things that aren't what you sell, unless you really cannot otherwise buy them.

ethbro 10y ago
> If you are running a business, though, you shouldn't be building things that aren't what you sell [or that you can't build more efficiently], unless you really cannot otherwise buy them. I'd add the above. Otherwise AWS would have never been built.
- beat 10y ago
  Indeed. Remember, AWS started as Amazon providing infrastructure as a service to internal projects, so individual teams and projects wouldn't have to go buy a bunch of hardware and ops staff in order to build products. Amazon itself is the biggest customer AWS has. Thanks to economies of scale, it's worthwhile for Amazon to develop new features as full-scale products, which leads to truly amazing things like Lambda. No one in their right mind would develop Lambda in-house, but for AWS, it makes a ton of sense.
  - tootie 10y ago
    I think Netflix is their biggest customer.
- beat 10y ago
  I'd add, though, that "efficiently" isn't just whether you can build cheaper than buying. It's if the cost of building the functionality is cheaper than the profit/growth you can generate with a sellable product built using the same amount of developer effort. That's a very, very different (and probably more expensive) proposition. That's why executives should make the decisions, not engineers! Back in the dot-com days, I worked on a project to build some functionality in-house that we could easily have bought off the shelf. The engineers argued that we'd save the company a million dollars. But frankly, we just wanted to do it because it was badass. And it turned out our solution would actually have cost us more per-system than the commercial solution we sneered at (hardware costs, not just development cost). Six man-months of engineering when everyone knew we were racing the clock before the money ran out? Absolute stupidity. If I were CEO/CTO and caught wind of such a project, I'd tell people that if they lifted a finger on it, they'd be fired. But that's a very different perspective than I had back then. Risking the very existence of what could have been a very big company in order to someday save a million dollars? Feh. (Of course, no one stopped us, because the CTO was just head nerd, and the money execs were busy fundraising rather than supervising)

Thaxll 10y ago

baremetal!

bespoke_engnr 10y ago
I'm assuming that this was originally sarcastic, but bare metal can actually turn out really well in some situations. If you're big enough to save just $50k/month by moving from AWS to physical machines, that's several System Engineers you can hire to do your day-to-day ops work and maintenance on your Ansible/whatever scripts. The thing that's more difficult to do on physical hardware, obviously, is scaling down every day if your peak load is some insane multiple of your base load for a 24h cycle. That's where AWS makes a lot of sense. I don't think there's an obvious 'best' answer -- it depends on what you need. It just bugs me that the current state of Ops/Infrastructure sometimes looks so much like the worst parts of the Javascript ecosystem.

deleted 10y ago

[deleted]