4 ms·

You just end up wasting your time maintain relationships with hardware vendors, troubleshooting low level issue, upgrading firmware. It wastes a lot of people's

You just end up wasting your time maintain relationships with hardware vendors, troubleshooting low level issue, upgrading firmware. It wastes a lot of people's time in the company, not just the team, and you're building one off tooling to work with some vendor's half broken management API. No thanks. :)

skuhn 10y ago
Sure, that happens. If you can save millions of dollars a month and move your business from non-viable to viable, I think it's a good tradeoff. That's what I do for a living, so I see value in it. Not everyone should have to deal with it though. Also, AWS is not problem free. When low level issues (like a firmware problem) strike, which they sometimes will, you often cannot identify or fix the issue yourself. AWS instances still run on that same shitty hardware, and while your odds are better that Amazon figured out most of the issues before you encountered them, your odds are also worse that they tested your particular special use case. So you can wind up playing the 'debug by support ticket' game. Check everything on your end. Check your deployment times for correlations to the problem. Finally give up and reach out to AWS support. Insist that the issue isn't on your end. Find out that the issue can't be resolved until their deployment next week. It's all understandable given the scope of what Amazon is doing, but they will never be as incentivized as you are to solve your problem.
- xorgar831 10y ago
  I don't think non-viable to viable is exactly what's going on however, at least in most cases. I think the costs are not being fully understood, the impact to the mindshare of the team, the quality of service and redundancy, the additional headcounts and quirky management code that someone's going to be toiling away on when a cloud provider can handle it at a much lower cost. Yes, you still have to open tickets, and yes, they still have problems that suck up a lot of time, however they spend the brunt of the time burning folks out fixing those issues, not you. If you talk to folks working with physical systems, that consumes their mindshare to the point that they're not focusing on their core product anymore.
  - skuhn 10y ago
    > If you talk to folks working with physical systems, that consumes their mindshare to the point that they're not focusing on their core product anymore. All I can say is, it isn't always the case and it doesn't have to be with the right organizational and design decisions.
- coredog64 10y ago
  That story isn't much different from my interactions with the on premise infrastructure team.
kuschku 10y ago
Which is why often the best tradeoff can be in colocating in existing datacenters, and just putting a few dozen of your own racks in. Or in renting dedicated servers. I’ve done some checks for the same performance between colocating, dedicated, VPS, and stuff like container engine + cloud databases, and with each level of abstraction the cost per month usually goes up by an order of magnitude.
- skuhn 10y ago
  Absolutely, there's a whole spectrum of options. "Building a datacenter" doesn't have to mean finding an empty field next to a river. Get a dedicated or shared cage, rent servers, do what fits your needs. It makes sense that each layer of abstraction costs dramatically more, because these services aren't vertically integrated -- they operate as value adds on top of other services. Heroku runs on AWS who buys server hardware and network hardware from vendors and puts it in racks that sit in buildings operated by datacenter providers who supply power from the electric company. So when you run something at the abstraction layer of a Heroku dyno, you're contributing to the profit margins of at least: 1. The actual service you use (Heroku, and Salesforce) 2. Their instance provider (Amazon) 3a. (maybe) Their server hardware manufacturer (HP / Dell / Supermicro) 3b. (maybe) Their network hardware manufacturer (Cisco / Juniper / Arista) 4a. Their server ODM (Quanta / et. al.) 5a. Their server components manufacturer (Intel / Hynix / Samsung / HGST / Mellanox / Adaptec / Avago / etc.) 5b. Their network ODM or ASIC designer (Avago) 6. (maybe) Their hosting provider (Peer1 or someone) 7. (maybe) Their retail datacenter provider (Equinix / CoreSite / etc.) 8. Their wholesale datacenter provider (DuPont Fabros / Digital Realty / etc.) 9. Their suppliers and contractors (electrical contractors / Square D / security guards / 10. The electricity and the taxes paid to the municipality Each of these companies intends to make a profit on the services they provide to the company up the food chain, and that all gets bundled into what you pay. Sure there are economies of scale here that you can't get, and smart companies are trying to eliminate some of these layers -- but that doesn't necessarily mean you are paying less. The margins increase as you go up the stack. Electric companies make peanuts for margins (8-10%) compared to a company like Mellanox that makes $1200 network cards (20+%). But who cares about selling you a $1200 network card when you can charge people $100/mo to use a dyno?
- 10y ago

skuhn 10y ago

Sure, that happens. If you can save millions of dollars a month and move your business from non-viable to viable, I think it's a good tradeoff. That's what I do for a living, so I see value in it. Not everyone should have to deal with it though. Also, AWS is not problem free. When low level issues (like a firmware problem) strike, which they sometimes will, you often cannot identify or fix the issue yourself. AWS instances still run on that same shitty hardware, and while your odds are better that Amazon figured out most of the issues before you encountered them, your odds are also worse that they tested your particular special use case. So you can wind up playing the 'debug by support ticket' game. Check everything on your end. Check your deployment times for correlations to the problem. Finally give up and reach out to AWS support. Insist that the issue isn't on your end. Find out that the issue can't be resolved until their deployment next week. It's all understandable given the scope of what Amazon is doing, but they will never be as incentivized as you are to solve your problem.

xorgar831 10y ago
I don't think non-viable to viable is exactly what's going on however, at least in most cases. I think the costs are not being fully understood, the impact to the mindshare of the team, the quality of service and redundancy, the additional headcounts and quirky management code that someone's going to be toiling away on when a cloud provider can handle it at a much lower cost. Yes, you still have to open tickets, and yes, they still have problems that suck up a lot of time, however they spend the brunt of the time burning folks out fixing those issues, not you. If you talk to folks working with physical systems, that consumes their mindshare to the point that they're not focusing on their core product anymore.
- skuhn 10y ago
  > If you talk to folks working with physical systems, that consumes their mindshare to the point that they're not focusing on their core product anymore. All I can say is, it isn't always the case and it doesn't have to be with the right organizational and design decisions.
coredog64 10y ago
That story isn't much different from my interactions with the on premise infrastructure team.

kuschku 10y ago

Which is why often the best tradeoff can be in colocating in existing datacenters, and just putting a few dozen of your own racks in. Or in renting dedicated servers. I’ve done some checks for the same performance between colocating, dedicated, VPS, and stuff like container engine + cloud databases, and with each level of abstraction the cost per month usually goes up by an order of magnitude.

skuhn 10y ago
Absolutely, there's a whole spectrum of options. "Building a datacenter" doesn't have to mean finding an empty field next to a river. Get a dedicated or shared cage, rent servers, do what fits your needs. It makes sense that each layer of abstraction costs dramatically more, because these services aren't vertically integrated -- they operate as value adds on top of other services. Heroku runs on AWS who buys server hardware and network hardware from vendors and puts it in racks that sit in buildings operated by datacenter providers who supply power from the electric company. So when you run something at the abstraction layer of a Heroku dyno, you're contributing to the profit margins of at least: 1. The actual service you use (Heroku, and Salesforce) 2. Their instance provider (Amazon) 3a. (maybe) Their server hardware manufacturer (HP / Dell / Supermicro) 3b. (maybe) Their network hardware manufacturer (Cisco / Juniper / Arista) 4a. Their server ODM (Quanta / et. al.) 5a. Their server components manufacturer (Intel / Hynix / Samsung / HGST / Mellanox / Adaptec / Avago / etc.) 5b. Their network ODM or ASIC designer (Avago) 6. (maybe) Their hosting provider (Peer1 or someone) 7. (maybe) Their retail datacenter provider (Equinix / CoreSite / etc.) 8. Their wholesale datacenter provider (DuPont Fabros / Digital Realty / etc.) 9. Their suppliers and contractors (electrical contractors / Square D / security guards / 10. The electricity and the taxes paid to the municipality Each of these companies intends to make a profit on the services they provide to the company up the food chain, and that all gets bundled into what you pay. Sure there are economies of scale here that you can't get, and smart companies are trying to eliminate some of these layers -- but that doesn't necessarily mean you are paying less. The margins increase as you go up the stack. Electric companies make peanuts for margins (8-10%) compared to a company like Mellanox that makes $1200 network cards (20+%). But who cares about selling you a $1200 network card when you can charge people $100/mo to use a dyno?
10y ago