5 ms·

I work in an ecommerce agency environment and it can be quite frustrating. Every client, no matter how small, has been sold the spiel and had some guy come thro

by ehnto 6y ago

I work in an ecommerce agency environment and it can be quite frustrating. Every client, no matter how small, has been sold the spiel and had some guy come through and build a boutique cloud infrastructure, CI pipeline and development VM. As if traffic will increase 1000% by next week with no prior warning. It just doesn't happen like that. I spend half my time debugging Hyperscale Bullshit™ that the client never needed and that failed the moment the devops guy left. If you want this stuff, you need a long term tech team, and a devops member on hand. Every time you hand your project around this stuff costs you hours. You also need to tell your devs to use the devops guy like tech support. They built this rickety tower of obtuse config files, make the devops guy fix it and don't let your devs spin their wheels for 4 hours before their assets compile on the Super Cool Distributed Docker VM microservice stack of teetering jenga blocks.

michaelbuckbee 6y ago
I did work for a YC startup that replaced my ~$100/mo Heroku app with a $5,000/mo AWS stack that nobody in the place knew how to manage. Their site was so ridiculously low traffic that I did some napkin math and figured out they were paying around $0.10 per web request.
- jrochkind1 6y ago
  Huh, making a manual AWS stack 50 times more expensive than a heroku setup seems like a real trick to me! I'm used to heroku being more expensive than running it yourself -- as I would expect, since you're paying them for a lot of management.
  - user5994461 6y ago
    Heroku is both incredibly cheap and incredibly expensive. It's only $7/month to deploy a web application. More if you want some of the paid features and a database instance. It all works out of the box with instant deployment out of the box, it's fantastic. Then it suddenly goes $50 more per gigabyte of RAM, which makes it massively expensive for any serious workload. It's crazy hot much they can try to charge, makes AWS looks like a bunch of clowns in comparison.
    - ksec 6y ago
      Well Heroku is running on top of AWS. So you are basically paying an extra premium on top of the premium from AWS. Although whether it is worth it depends on your view. ( I keep thinking SalesForces is just not a good fit for Heroku, they should have sold it to MS or Google )
    - jrochkind1 6y ago
      If it saves you a FTE or two from managing your own infrastructure, there is a lot of headroom before it's a losing proposition. Which is what I think the OP misses discussing in as much detail as they could with AWS -- are there ways AWS is saving some customers developer/ops staff time over the cheaper alternatives? Cause that is often more expensive than the resources themselves. Could just be "because that's what we're familiar with" (and that's not illegitimate as cost savings), but could be actually better dashboard UIs, APIs, integrations, whatever. [I am currently investigating heroku for our low-traffic tiny-team app, after we had our sysadmin/devops position eliminated. The performance characteristics are surprising me negatively (I can't figure out how anyone gets by with a `standard` instead of `performance` dyno, even for a small low-traffic app; currently writing up my findings for public consumption), but the value of the management they are doing so we don't have to is meeting and exceeding my expectations. (We currently manage our own AWS resources directly, so I know what it is we can't do sustainably anymore with the eliminated position, and the value we're getting from not having to do it).]
      - vidarh 6y ago
        My experience - from doing devops consulting and moving clients off AWS every chance I get - is that it's bad business for devops consultants to move clients off AWS in terms of short term billable hours, because clients spent more money on me when they're on AWS. If I was after maximising billable hours in the short term, then I'd recommend AWS all the time... As such a lot of devops consultants certainly have all the wrong incentives to recommend it, and these days most of them also lack experience of how to price out alternatives. E.g. a typical beginners mistake is to think you'd price out a one-to-one match of servers between AWS and an alternative, but one of the benefits of picking other options is that you're able to look at your app and design a setup that fits your needs better. Network latency matters. Ability to add enough RAM to fit your database working set in RAM if at all possible matters. And so on. With AWS this is often possible but often at the cost of scaling out other things too that you don't need. And most developers won't do it if you don't give them budget responsibility and make them justify the cost and then cut their budget. Development teams used to AWS tends to spin up instance after instance instead of actually measuring and figuring out why they're running into limits to keep costs down. Giving dev teams control over infra without having someone with extensive operations experience is an absolute disaster if you want to manage costs. I work for a VC now. When I evaluate the tech teams of people who apply to us, it's perfectly fine if they use AWS, but it's a massive red flag to me if they don't understand the costs and the costs of alternatives. Usually the ones who do know they're paying for speed and convenience, and have thoughts on how to cut costs by moving parts or all of their services off AWS as they scale, or they have a rationale for why their hosting is never going to be a big part of their overall cost base. The only case where AWS is cost effective is if you get big enough to negotiate really hefty discounts. It's possible - I've heard examples. But if you're paying AWS list prices, chances are sooner or later you'll come across a competitor that isn't.
- KingOfCoders 6y ago
  Many (startup) SaaS companies (except metrics collectors e.g) have very low traffic from logged in users. When I moved from an ecommerce company with 1000 logins/sec to a SaaS company I could not believe the low traffic :-)
- NikolaeVarius 6y ago
  Thats just incompetence.
- fourseventy 6y ago
  I've seen this exact same thing happen to a company I used to work for. They had an enterprise B2B app that had extremely low traffic. The Heroku app worked fine but the CTO spent a year building a super complicated AWS stack as an exercise in resume building. After the AWS stack launch filed miserably and we had to switch back to Heroku the CTO quit and still has "migrated company to infrastructure as code AWS stack" on their Linkedin profile.
hbogert 6y ago
You make it sound as if devops is the initiator of the complexity problems. This might well be in your case. My experience is the other way around. I'm forced to think about stuff like Kubernetes because the devs are popping "micro-services" for almost anything and the answer to trying to keep things manageable is, hopefully, K8s. Of course they then send funny memes and argue that K8s is overkill. Yet they have no idea how much a devops guy has to do to actually ramp up one of their multi-gigabyte container. The whole micro-services mentality is proving to be backwards in my environment. It's seen as the answer to everything. "Oh we have a behemoth, let's just not use this part of the application and reimplement it somewhere else." In essence that's great. However, the grunt of the work is making tests to capture old behaviour/semantics; and who are the (indirect) clients of this piece of code. Ignoring a piece of code to death and reimplementing it somewhere else behind a socket is only part of the solution. The hyperscalars are benefiting greatly of this mindset imo.
- ehnto 6y ago
  Oh yeah, my point of view is entirely based on my experience with small to medium sized companies, and there especially developers can be the instigators in sudden complexity multipliers. I tried my best not to rag on devops as I am always incredibly impressed by these systems and what can be achieved and automated. It absolutely has it's place. It's just that all the technology that big companies use become trendy and then end up used at small companies too. Small companies don't realise that the tools aren't solving the problems they have, and they don't understand the vendor lock ins and dependencies that they've introduced into their platform for no real gain. This happens in all kinds of ways, not just devops. Your example of developers using the microservice pattern is exactly that. When I see a developer recommend microservices my mind flashes forward to all of the extra complexity that entails and how we've just turned a small monolith on one server into 6 pieces of software across distributed servers. Great for big companies, too much complexity for small companies.
devonkim 6y ago
As an ops and infrastructure engineer I cringe every time people talk about trying to automate all these processes early in the life of an organization or if there’s zero way for the company to have explosive growth. Not every company is going to need to rapidly scale up and down and I rarely see cloud saving money unless one’s infrastructure is pretty garbage already. Cloud lift and shifts are like bad rewrites inheriting all the architectural problems and increasing opex for the sake of a “cloud” stamp aka cloud washing. But I do recommend companies have a means of deploying stuff to AWS and doing basic security there for clients that really require AWS. Having a VPC ready to go costs nothing operational and is good for at least having some semblance of collective skill with a major public cloud provider. Invariably though when I see undisciplined developers take on colo style hosting systems become brittle and unable to accept changes and with any success it becomes a Herculean task to deploy new features without putting something important at risk. This results in more time spent on systems archaeology for new engineers eventually than creating new features or improving maintainability. In 2020 I’d recommend developing as much as possible on PaaS type platforms like Heroku, a managed K8S, or even App Engine to avoid more bike shedding that really doesn’t buy a company anything materially advantageous early on like deciding which kind of EC2 instance to standardize on. Until engineers know what to optimize and concentrate upon in terms of process and there’s perhaps tens of thousands in monthly infrastructure costs most skilled (read: expensive) ops engineers won’t really be of much value except offloading work that developers shouldn’t have been thinking about in the first place.

by ehnto 6y ago

michaelbuckbee 6y ago

I did work for a YC startup that replaced my ~$100/mo Heroku app with a $5,000/mo AWS stack that nobody in the place knew how to manage. Their site was so ridiculously low traffic that I did some napkin math and figured out they were paying around $0.10 per web request.

jrochkind1 6y ago
Huh, making a manual AWS stack 50 times more expensive than a heroku setup seems like a real trick to me! I'm used to heroku being more expensive than running it yourself -- as I would expect, since you're paying them for a lot of management.
- user5994461 6y ago
  Heroku is both incredibly cheap and incredibly expensive. It's only $7/month to deploy a web application. More if you want some of the paid features and a database instance. It all works out of the box with instant deployment out of the box, it's fantastic. Then it suddenly goes $50 more per gigabyte of RAM, which makes it massively expensive for any serious workload. It's crazy hot much they can try to charge, makes AWS looks like a bunch of clowns in comparison.
  - ksec 6y ago
    Well Heroku is running on top of AWS. So you are basically paying an extra premium on top of the premium from AWS. Although whether it is worth it depends on your view. ( I keep thinking SalesForces is just not a good fit for Heroku, they should have sold it to MS or Google )
  - jrochkind1 6y ago
    If it saves you a FTE or two from managing your own infrastructure, there is a lot of headroom before it's a losing proposition. Which is what I think the OP misses discussing in as much detail as they could with AWS -- are there ways AWS is saving some customers developer/ops staff time over the cheaper alternatives? Cause that is often more expensive than the resources themselves. Could just be "because that's what we're familiar with" (and that's not illegitimate as cost savings), but could be actually better dashboard UIs, APIs, integrations, whatever. [I am currently investigating heroku for our low-traffic tiny-team app, after we had our sysadmin/devops position eliminated. The performance characteristics are surprising me negatively (I can't figure out how anyone gets by with a `standard` instead of `performance` dyno, even for a small low-traffic app; currently writing up my findings for public consumption), but the value of the management they are doing so we don't have to is meeting and exceeding my expectations. (We currently manage our own AWS resources directly, so I know what it is we can't do sustainably anymore with the eliminated position, and the value we're getting from not having to do it).]
    - vidarh 6y ago
      My experience - from doing devops consulting and moving clients off AWS every chance I get - is that it's bad business for devops consultants to move clients off AWS in terms of short term billable hours, because clients spent more money on me when they're on AWS. If I was after maximising billable hours in the short term, then I'd recommend AWS all the time... As such a lot of devops consultants certainly have all the wrong incentives to recommend it, and these days most of them also lack experience of how to price out alternatives. E.g. a typical beginners mistake is to think you'd price out a one-to-one match of servers between AWS and an alternative, but one of the benefits of picking other options is that you're able to look at your app and design a setup that fits your needs better. Network latency matters. Ability to add enough RAM to fit your database working set in RAM if at all possible matters. And so on. With AWS this is often possible but often at the cost of scaling out other things too that you don't need. And most developers won't do it if you don't give them budget responsibility and make them justify the cost and then cut their budget. Development teams used to AWS tends to spin up instance after instance instead of actually measuring and figuring out why they're running into limits to keep costs down. Giving dev teams control over infra without having someone with extensive operations experience is an absolute disaster if you want to manage costs. I work for a VC now. When I evaluate the tech teams of people who apply to us, it's perfectly fine if they use AWS, but it's a massive red flag to me if they don't understand the costs and the costs of alternatives. Usually the ones who do know they're paying for speed and convenience, and have thoughts on how to cut costs by moving parts or all of their services off AWS as they scale, or they have a rationale for why their hosting is never going to be a big part of their overall cost base. The only case where AWS is cost effective is if you get big enough to negotiate really hefty discounts. It's possible - I've heard examples. But if you're paying AWS list prices, chances are sooner or later you'll come across a competitor that isn't.
KingOfCoders 6y ago
Many (startup) SaaS companies (except metrics collectors e.g) have very low traffic from logged in users. When I moved from an ecommerce company with 1000 logins/sec to a SaaS company I could not believe the low traffic :-)
NikolaeVarius 6y ago
Thats just incompetence.
fourseventy 6y ago
I've seen this exact same thing happen to a company I used to work for. They had an enterprise B2B app that had extremely low traffic. The Heroku app worked fine but the CTO spent a year building a super complicated AWS stack as an exercise in resume building. After the AWS stack launch filed miserably and we had to switch back to Heroku the CTO quit and still has "migrated company to infrastructure as code AWS stack" on their Linkedin profile.

hbogert 6y ago

You make it sound as if devops is the initiator of the complexity problems. This might well be in your case. My experience is the other way around. I'm forced to think about stuff like Kubernetes because the devs are popping "micro-services" for almost anything and the answer to trying to keep things manageable is, hopefully, K8s. Of course they then send funny memes and argue that K8s is overkill. Yet they have no idea how much a devops guy has to do to actually ramp up one of their multi-gigabyte container. The whole micro-services mentality is proving to be backwards in my environment. It's seen as the answer to everything. "Oh we have a behemoth, let's just not use this part of the application and reimplement it somewhere else." In essence that's great. However, the grunt of the work is making tests to capture old behaviour/semantics; and who are the (indirect) clients of this piece of code. Ignoring a piece of code to death and reimplementing it somewhere else behind a socket is only part of the solution. The hyperscalars are benefiting greatly of this mindset imo.

ehnto 6y ago
Oh yeah, my point of view is entirely based on my experience with small to medium sized companies, and there especially developers can be the instigators in sudden complexity multipliers. I tried my best not to rag on devops as I am always incredibly impressed by these systems and what can be achieved and automated. It absolutely has it's place. It's just that all the technology that big companies use become trendy and then end up used at small companies too. Small companies don't realise that the tools aren't solving the problems they have, and they don't understand the vendor lock ins and dependencies that they've introduced into their platform for no real gain. This happens in all kinds of ways, not just devops. Your example of developers using the microservice pattern is exactly that. When I see a developer recommend microservices my mind flashes forward to all of the extra complexity that entails and how we've just turned a small monolith on one server into 6 pieces of software across distributed servers. Great for big companies, too much complexity for small companies.

devonkim 6y ago

As an ops and infrastructure engineer I cringe every time people talk about trying to automate all these processes early in the life of an organization or if there’s zero way for the company to have explosive growth. Not every company is going to need to rapidly scale up and down and I rarely see cloud saving money unless one’s infrastructure is pretty garbage already. Cloud lift and shifts are like bad rewrites inheriting all the architectural problems and increasing opex for the sake of a “cloud” stamp aka cloud washing. But I do recommend companies have a means of deploying stuff to AWS and doing basic security there for clients that really require AWS. Having a VPC ready to go costs nothing operational and is good for at least having some semblance of collective skill with a major public cloud provider. Invariably though when I see undisciplined developers take on colo style hosting systems become brittle and unable to accept changes and with any success it becomes a Herculean task to deploy new features without putting something important at risk. This results in more time spent on systems archaeology for new engineers eventually than creating new features or improving maintainability. In 2020 I’d recommend developing as much as possible on PaaS type platforms like Heroku, a managed K8S, or even App Engine to avoid more bike shedding that really doesn’t buy a company anything materially advantageous early on like deciding which kind of EC2 instance to standardize on. Until engineers know what to optimize and concentrate upon in terms of process and there’s perhaps tens of thousands in monthly infrastructure costs most skilled (read: expensive) ops engineers won’t really be of much value except offloading work that developers shouldn’t have been thinking about in the first place.