4 ms·

>There are certain benefits of cloud or not, but this kind of thinking confuses me. Why stop there? Because using EC2/GCE infrastructure is an incredible waste

>There are certain benefits of cloud or not, but this kind of thinking confuses me. Why stop there? Because using EC2/GCE infrastructure is an incredible waste of money if you have any reasonable-sized base-load and bandwidth utilization.

MichaelGG 9y ago
As I said, there are benefits to not using cloud, sure. But don't pitch it as because anyone that loves programming should build their own servers.
- valuearb 9y ago
  Etsy has pay and benefits far outstripping the competition. They should have the best engineers and developers. So why then can't they build and maintain the best infrastructure? Why does it suck so?
- snuxoll 9y ago
  Build their own? Probably not, unless they are doing it at massive scale and the cost of replacing parts beats out buying a server with a 3/5/7 year warranty. If you have static usage, or at least a base load owning your own hardware to support it does make sense though, colo space is cheap, the biggest disadvantage from a financial perspective to owning your own gear is you have a depreciating CapEx instead of a 100% deductible OpEx.
  - valuearb 9y ago
    Real investors care about cash flow, which means the difference between purchasing/depreciating and leasing/renting(OpEx) is negligible. Whats far more important is the business benefits of the best approach for your business.
  - sciurus 9y ago
    I don't know what the current state of things is, but 5 years ago they were doing a mix of buying off the shelf and building their own supermicro-based systems. https://codeascraft.com/2012/08/31/what-hardware-powers-etsy-com/ https://codeascraft.com/2012/08/31/what-hardware-powers-etsy...
djsumdog 9y ago
Most of the larger shops I've worked at have started moving away from hosted providers to save money. When you're starting out, hosted solutions will save you a lot of time. Running your own servers is a high cost and you need a dedicated team. But once you're pulling in that revenue and can afford it, running your own can save you some money and add a lot of flexibility. If you're starting out, I highly recommend not tightly coupling yourself to a single service (using AMIs or any really specific AWS stuff) but write your provisioning scripts in such a way as you can plug in different services or deploy locally. Things like Terraform + a provisioner (Ansible/Puppet/etc) can make it easy to move your system to another provider or host your own and more easily measure costs.
- jamiesonbecker 9y ago
  Agreed. Or stay flexible and negotiate between clouds. Even Netflix's legendary AWS relationship doesn't stop them from building their own custom hardware (Open Connect) and building what is essentially their own custom CDN. People used to ask us when we started Userify (SSH Key management, https://userify.com https://userify.com)... what happens when AWS clones you? Well.. that was five years ago, and they haven't cloned us yet. But even if they ever did, they have a vested interest in ensuring lock-in to AWS, so they will always choose to make things work with things that only they offer (such as IAM) instead of making it cross-cloud. That makes things a lot more challenging for large enterprises, especially when you're talking about users that may have to log in to servers hosted in multiple clouds. Having multiple sources of truth doesn't work at scale. Most large organizations are at least on AWS, GCP, and/or Azure already, so I don't really think Userify's SSH key management (or any other cross-cloud tools like ansible/pupet/terraform/etc) are going away anytime soon.
- giobox 9y ago
  I completely see how it's theoretically desirable to not be tied to a particular cloud/VPS provider, but I'm not entirely convinced Terraform alone is the answer. In my experience, Terraform does not let you declare VPS resources in a "provider neutral" way - you would still have to completely rewrite your Terraform files if you say, switch from AWS to Azure. Then there's the wider issue that whole concepts in one VPS provider that you used in Terraform don't necessarily exist in another, or require a completely different approach. A great example is just look at the documentation for Terraform's DigitalOcean Support, vs their AWS support. The AWS documentation is many times larger! Once your stack reaches a certain degree of complexity this is a non-trivial task. I should make clear this isn't a knock against Terraform - it's a great tool, but it isn't going to magically make it easy to leave AWS for lots of people. Amazon aren't stupid, and to really realise the full value/potential of their cloud offering you kind of have to design for the proprietary features of their stack, which of course is the whole point - driving you to be locked in as much as they can. Treat them as just a bunch of generic compute boxes in the cloud and you will end up driving your costs up, not to mention often markedly increasing the complexity of your deployment.

MichaelGG 9y ago

As I said, there are benefits to not using cloud, sure. But don't pitch it as because anyone that loves programming should build their own servers.

valuearb 9y ago
Etsy has pay and benefits far outstripping the competition. They should have the best engineers and developers. So why then can't they build and maintain the best infrastructure? Why does it suck so?
snuxoll 9y ago
Build their own? Probably not, unless they are doing it at massive scale and the cost of replacing parts beats out buying a server with a 3/5/7 year warranty. If you have static usage, or at least a base load owning your own hardware to support it does make sense though, colo space is cheap, the biggest disadvantage from a financial perspective to owning your own gear is you have a depreciating CapEx instead of a 100% deductible OpEx.
- valuearb 9y ago
  Real investors care about cash flow, which means the difference between purchasing/depreciating and leasing/renting(OpEx) is negligible. Whats far more important is the business benefits of the best approach for your business.
- sciurus 9y ago
  I don't know what the current state of things is, but 5 years ago they were doing a mix of buying off the shelf and building their own supermicro-based systems. https://codeascraft.com/2012/08/31/what-hardware-powers-etsy-com/ https://codeascraft.com/2012/08/31/what-hardware-powers-etsy...

djsumdog 9y ago

Most of the larger shops I've worked at have started moving away from hosted providers to save money. When you're starting out, hosted solutions will save you a lot of time. Running your own servers is a high cost and you need a dedicated team. But once you're pulling in that revenue and can afford it, running your own can save you some money and add a lot of flexibility. If you're starting out, I highly recommend not tightly coupling yourself to a single service (using AMIs or any really specific AWS stuff) but write your provisioning scripts in such a way as you can plug in different services or deploy locally. Things like Terraform + a provisioner (Ansible/Puppet/etc) can make it easy to move your system to another provider or host your own and more easily measure costs.

jamiesonbecker 9y ago
Agreed. Or stay flexible and negotiate between clouds. Even Netflix's legendary AWS relationship doesn't stop them from building their own custom hardware (Open Connect) and building what is essentially their own custom CDN. People used to ask us when we started Userify (SSH Key management, https://userify.com https://userify.com)... what happens when AWS clones you? Well.. that was five years ago, and they haven't cloned us yet. But even if they ever did, they have a vested interest in ensuring lock-in to AWS, so they will always choose to make things work with things that only they offer (such as IAM) instead of making it cross-cloud. That makes things a lot more challenging for large enterprises, especially when you're talking about users that may have to log in to servers hosted in multiple clouds. Having multiple sources of truth doesn't work at scale. Most large organizations are at least on AWS, GCP, and/or Azure already, so I don't really think Userify's SSH key management (or any other cross-cloud tools like ansible/pupet/terraform/etc) are going away anytime soon.
giobox 9y ago
I completely see how it's theoretically desirable to not be tied to a particular cloud/VPS provider, but I'm not entirely convinced Terraform alone is the answer. In my experience, Terraform does not let you declare VPS resources in a "provider neutral" way - you would still have to completely rewrite your Terraform files if you say, switch from AWS to Azure. Then there's the wider issue that whole concepts in one VPS provider that you used in Terraform don't necessarily exist in another, or require a completely different approach. A great example is just look at the documentation for Terraform's DigitalOcean Support, vs their AWS support. The AWS documentation is many times larger! Once your stack reaches a certain degree of complexity this is a non-trivial task. I should make clear this isn't a knock against Terraform - it's a great tool, but it isn't going to magically make it easy to leave AWS for lots of people. Amazon aren't stupid, and to really realise the full value/potential of their cloud offering you kind of have to design for the proprietary features of their stack, which of course is the whole point - driving you to be locked in as much as they can. Treat them as just a bunch of generic compute boxes in the cloud and you will end up driving your costs up, not to mention often markedly increasing the complexity of your deployment.