6 ms·

I have worked at a startup that was paying > 20k per month for AWS. I am certain the whole system could run on one dedicated blade. There was just this cultur

by Tistel 8y ago

I have worked at a startup that was paying > 20k per month for AWS. I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. I was like: "if cost is a big issue, then lets spin up one beefy machine and create users for each dev", because thats what everyone using unix did back in the day (it works fine). The reply was "thats not possible" so every dev got their own dedicated instance that was 99% idle. We are paying >3k per month to aws and the business team is complaining about waste. sigh. old guy rant over.

sigfubar 8y ago
I've worked for a certain IoT startup that at one point paid $850k/month (eight hundred and fifty thousand USD per month) for AWS usage. After about a year of full time work by a team of SREs and programmers, the bill was made to go down by half.
- h1d 8y ago
  Any breakdowns on cost?
mancerayder 8y ago
I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. Wait, you're saying there's a Bachelor of Science DevOps degree?
- toomuchtodo 8y ago
  Documentation is no replacement for experience. EDIT: I agree that OPs choice of words makes the idea expressed ambiguous.
  - mancerayder 8y ago
    I just don't understand what a "self-taught" DevOps person is -- as opposed to a someone-taught? Self-taught usually means "not educated formally." All DevOps / SysAdmins / etc. are "self-taught." So it's unclear what the OP meant.
- Tistel 8y ago
  They are very young, nice kids, who have started from a pure aws standpoint. They started with AWS tutorials. These tutorials did not teach the full unix environment. The gaps in their knowledge is like the grande canyon. If they just read the O'reilly book on linux: http://shop.oreilly.com/product/9780596154493.do http://shop.oreilly.com/product/9780596154493.do. they would be miles ahead. But, their only interest is docker/kubernetes. Those technologies are great, but, they simply have massive gaps in their knowledge of the fundementals. They don't seem to like reading technical books from cover to cover. So they just read tutorials far enough to get the system started. Then they cut/paste from stackoverflow when problems arise. If that does not work, they blame the tools ("couched is too slow!"). Again, its not their fault, just the new dev culture. I freely admit I am an old fogey.
  - mancerayder 8y ago
    We agree. I interviewed at (a fast growing real estate startup) and as an old guy, I highlighted my vast enterprise Linux engineering experience. The response from management was, oh, that's nice but doesn't matter here. I can teach people the command line in one day. This is a big AWS and exclusively Linux shop. What do you say to that?
    - heydonovan 8y ago
      From my experience, the hardcore Unix folks should stick to jobs that highlights that type of knowledge. If you join an AWS shop, you're never going to dig into low level things like the proc filesystem, and the arcane knowledge of knowing every single tool just isn't used. I've gone from remoting into machines on the daily, to not using SSH for almost a year now. Just depends on the job I suppose. Went from having knowledge of all the different signals, and getting really good at tcpdump/gdb, to just writing Terraform/CloudFormation scripts, and nuking an instance from orbit if there were any issues with it. Such a different workflow.
      - mancerayder 8y ago
        Yes, BUT: Grey-haired Linux folks who've hammered on complex, changing, sometimes esoteric systems in different companies with unique challenges COMBINED with having to have learned a programming language, to view these people as obsolete is an ultimate folly of youth here in IT. To suggest that it's all different now, your Shell/Perl/Webserver/database/caching/networking experience somehow has nothing on Python/Terraform/AWS/Docker really shows inexperience of hiring managers. I learned both, and I can tell you AWS+someyaml+Python are MUCH easier than any of the stuff I listed above. Yet, there's a new generation of hiring managers that believes otherwise. Today at work I had to explain absolutely basic DNS to someone. I dunno, call me skeptical about this new paradigm business. The paradigms look awfully familiar.
thruhiker 8y ago
Did you weigh these costs against the value of the developers not stepping on eachother's toes using the same instance?Separate users with sudo privileges provides zero isolation. I'm unsure what use case was, but if it was a testing or staging environment dedicated to each developer, having a separate right-sized instance for each is quite sane and reasonable. EC2 instances such as the T2/T3 series are cheap enough that there's normally not a need to be so miserly in my experience.
- CharlesColeman 8y ago
  >> I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. > Did you weigh these costs against the value of the developers not stepping on each other's toes using the same instance? Separate users with sudo privileges provides zero isolation. Each developer's machine is a separate instance. If the whole system could run on "one dedicated blade," then the developers could easily run a local instance on their own machines (either directly or in a VM) without stepping on anyone's toes. Using AWS for local development seems wasteful and over-complicated to me. The only good reason I can see for doing it is if you're really vendor-locked into AWS-proprietary services, and it's impractical to run your code anywhere except on AWS (but that's arguably a mistake in itself).
  - deathanatos 8y ago
    > Each developer's machine is a separate instance. If the whole system could run on "one dedicated blade," then the developers could easily run a local instance on their own machines (either directly or in a VM) without stepping on anyone's toes. What the parent you're responding to likely meant was that, by running everything on one system ("one dedicated blade") you are causing all of the things on that system to share certain things. The set of system-installed packages; if nginx needs to be installed and running, is its configuration now shared between all of the services on that machine; and even the very OS itself (e.g., if it needs to go down for updates, or it needs to do, say a dist-upgrade, which means that all of the associates services on that instance need to dist-upgrade together, and one laggard can thus hold everyone back.) Even in the case of developers using an instance on AWS, the same applies, really: a dev, legitimately developing some component, could have cross-talk to others, in all of the same ways listed above. I do agree that giving devs dedicated instances might not be appropriate in many instances, and it perhaps wasn't appropriate in yours. But there's two points here: whether the devs needed those instances, or could develop on local laptops, and whether putting all instances or users on a single instance works well. Even for development, given that most corps seem to love giving devs that code primarily for Linux servers OS X laptops, there might be advantages to allowing them to develop on the actual target hardware. > There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. Docker here should make it rather trivial to combine everything onto a single instance, since that's what isolation gets you: a well defined unit that doesn't have endless unknown dependencies on the host system.
eftpotrm 8y ago
Knew a place that turned off the developers' test environment because it was costing 10x that per month. Which was as popular and successful as you can imagine. I'm.... skeptical about the benefits of the cloud architecture they actually had, given those costs.
- twh270 8y ago
  Our org just implemented this. A script turns off all services in lower environments (dev, qa) at night; another one turns 'em on in the morning. Of course if you need to, you can run your own script to turn on your services "if you need to". This is being sold as a "step in the right direction", where "right direction" is anything that reduces cloud costs. (Payroll costs are apparently invisible, so the time people waste dealing with this baloney doesn't count. Not to mention the annoyance.) I'm tempted to point out that they could save a lot more money by merging microservices into macroservices....
UncleMeat 8y ago
That's like half of one engineer. Less if you include all of the overhead of having an employee. That can easily be the right choice.
nunez 8y ago
it can but if that blade or the enclosure dies, the system is sol...also, procurement can be a pain

by Tistel 8y ago

sigfubar 8y ago

I've worked for a certain IoT startup that at one point paid $850k/month (eight hundred and fifty thousand USD per month) for AWS usage. After about a year of full time work by a team of SREs and programmers, the bill was made to go down by half.

h1d 8y ago
Any breakdowns on cost?

mancerayder 8y ago

I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. Wait, you're saying there's a Bachelor of Science DevOps degree?

toomuchtodo 8y ago
Documentation is no replacement for experience. EDIT: I agree that OPs choice of words makes the idea expressed ambiguous.
- mancerayder 8y ago
  I just don't understand what a "self-taught" DevOps person is -- as opposed to a someone-taught? Self-taught usually means "not educated formally." All DevOps / SysAdmins / etc. are "self-taught." So it's unclear what the OP meant.
Tistel 8y ago
They are very young, nice kids, who have started from a pure aws standpoint. They started with AWS tutorials. These tutorials did not teach the full unix environment. The gaps in their knowledge is like the grande canyon. If they just read the O'reilly book on linux: http://shop.oreilly.com/product/9780596154493.do http://shop.oreilly.com/product/9780596154493.do. they would be miles ahead. But, their only interest is docker/kubernetes. Those technologies are great, but, they simply have massive gaps in their knowledge of the fundementals. They don't seem to like reading technical books from cover to cover. So they just read tutorials far enough to get the system started. Then they cut/paste from stackoverflow when problems arise. If that does not work, they blame the tools ("couched is too slow!"). Again, its not their fault, just the new dev culture. I freely admit I am an old fogey.
- mancerayder 8y ago
  We agree. I interviewed at (a fast growing real estate startup) and as an old guy, I highlighted my vast enterprise Linux engineering experience. The response from management was, oh, that's nice but doesn't matter here. I can teach people the command line in one day. This is a big AWS and exclusively Linux shop. What do you say to that?
  - heydonovan 8y ago
    From my experience, the hardcore Unix folks should stick to jobs that highlights that type of knowledge. If you join an AWS shop, you're never going to dig into low level things like the proc filesystem, and the arcane knowledge of knowing every single tool just isn't used. I've gone from remoting into machines on the daily, to not using SSH for almost a year now. Just depends on the job I suppose. Went from having knowledge of all the different signals, and getting really good at tcpdump/gdb, to just writing Terraform/CloudFormation scripts, and nuking an instance from orbit if there were any issues with it. Such a different workflow.
    - mancerayder 8y ago
      Yes, BUT: Grey-haired Linux folks who've hammered on complex, changing, sometimes esoteric systems in different companies with unique challenges COMBINED with having to have learned a programming language, to view these people as obsolete is an ultimate folly of youth here in IT. To suggest that it's all different now, your Shell/Perl/Webserver/database/caching/networking experience somehow has nothing on Python/Terraform/AWS/Docker really shows inexperience of hiring managers. I learned both, and I can tell you AWS+someyaml+Python are MUCH easier than any of the stuff I listed above. Yet, there's a new generation of hiring managers that believes otherwise. Today at work I had to explain absolutely basic DNS to someone. I dunno, call me skeptical about this new paradigm business. The paradigms look awfully familiar.

thruhiker 8y ago

Did you weigh these costs against the value of the developers not stepping on eachother's toes using the same instance?Separate users with sudo privileges provides zero isolation. I'm unsure what use case was, but if it was a testing or staging environment dedicated to each developer, having a separate right-sized instance for each is quite sane and reasonable. EC2 instances such as the T2/T3 series are cheap enough that there's normally not a need to be so miserly in my experience.

CharlesColeman 8y ago
>> I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. > Did you weigh these costs against the value of the developers not stepping on each other's toes using the same instance? Separate users with sudo privileges provides zero isolation. Each developer's machine is a separate instance. If the whole system could run on "one dedicated blade," then the developers could easily run a local instance on their own machines (either directly or in a VM) without stepping on anyone's toes. Using AWS for local development seems wasteful and over-complicated to me. The only good reason I can see for doing it is if you're really vendor-locked into AWS-proprietary services, and it's impractical to run your code anywhere except on AWS (but that's arguably a mistake in itself).
- deathanatos 8y ago
  > Each developer's machine is a separate instance. If the whole system could run on "one dedicated blade," then the developers could easily run a local instance on their own machines (either directly or in a VM) without stepping on anyone's toes. What the parent you're responding to likely meant was that, by running everything on one system ("one dedicated blade") you are causing all of the things on that system to share certain things. The set of system-installed packages; if nginx needs to be installed and running, is its configuration now shared between all of the services on that machine; and even the very OS itself (e.g., if it needs to go down for updates, or it needs to do, say a dist-upgrade, which means that all of the associates services on that instance need to dist-upgrade together, and one laggard can thus hold everyone back.) Even in the case of developers using an instance on AWS, the same applies, really: a dev, legitimately developing some component, could have cross-talk to others, in all of the same ways listed above. I do agree that giving devs dedicated instances might not be appropriate in many instances, and it perhaps wasn't appropriate in yours. But there's two points here: whether the devs needed those instances, or could develop on local laptops, and whether putting all instances or users on a single instance works well. Even for development, given that most corps seem to love giving devs that code primarily for Linux servers OS X laptops, there might be advantages to allowing them to develop on the actual target hardware. > There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. Docker here should make it rather trivial to combine everything onto a single instance, since that's what isolation gets you: a well defined unit that doesn't have endless unknown dependencies on the host system.

eftpotrm 8y ago

Knew a place that turned off the developers' test environment because it was costing 10x that per month. Which was as popular and successful as you can imagine. I'm.... skeptical about the benefits of the cloud architecture they actually had, given those costs.

twh270 8y ago
Our org just implemented this. A script turns off all services in lower environments (dev, qa) at night; another one turns 'em on in the morning. Of course if you need to, you can run your own script to turn on your services "if you need to". This is being sold as a "step in the right direction", where "right direction" is anything that reduces cloud costs. (Payroll costs are apparently invisible, so the time people waste dealing with this baloney doesn't count. Not to mention the annoyance.) I'm tempted to point out that they could save a lot more money by merging microservices into macroservices....

UncleMeat 8y ago

That's like half of one engineer. Less if you include all of the overhead of having an employee. That can easily be the right choice.

nunez 8y ago

it can but if that blade or the enclosure dies, the system is sol...also, procurement can be a pain