9 ms·

When I joined a particular startup 4 years ago I was surprised that they owned their own servers mounted in racks in a rented colo. I was all, how retro! But th

When I joined a particular startup 4 years ago I was surprised that they owned their own servers mounted in racks in a rented colo. I was all, how retro! But then the head of IT showed me the cost savings over AWS and how he orchestrated it all using VMWare images, and I saw he had a point. He saved more than enough money to pay for his full time salary to keep it all going, even dealing with hardware maintenance. What was unique was that the company had a predictable load that didn't change much. If we had needed to change capacity level drastically over a regular cycle then AWS would have started to make sense.

lykr0n 8y ago
This. If you can predict your workload with some accuracy, and have someone who knows how to design systems- you can save a lot of money. AWS' value is that "we can do everything except write your code for you" which you end up paying a lot for.
- chomp 8y ago
  Even if you can't predict the workload too well, you can go hybrid cloud - colo for the baseline and then AWS for the unpredictable stuff. ManageIQ is great for this.
- ekovarski 8y ago
  ... isn't that the hardest part, predicting workload, for many companies. Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. If you can predict workload then you could also reduce your cloud bills, in theory, by moving spinning up/down instances. Many are going the cloud route because the cost of entry is much easier to start developing but eventually, yes, the cost skyrockets and you do end up paying for putting little forethought into the real workload requirements.
  - toomuchtodo 8y ago
    > Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. One senior sysadmin/infrastructure engineer with on-prem experience is enough to scale up your colo presence. You can either assemble your own servers or custom order and have them shipped directly to the colo. Remote console and power management (PDUs) is a solved problem. Out of band management, also solved. Virtualization, solved. Push your VM snapshots, content, artifacts, and database backups into your object store of choice for offsite backups. Ideally, your colo facility is within driving distance of your office if you absolutely need physical access. Otherwise, you pay someone at the colo a few hundred bucks a month for remote hands services. EDIT: k8s arguably makes this easier than in the past. You want to run on your own cluster? You can. You need to burst your containers to spot instances in someone else's cloud? You can.
    - user5994461 8y ago
      All the problems you consider solved are only solved if you have the tooling from AWS/Google or half solved if you have VMWare. With physical hardware, the sysadmin will spend a lot of his time fighting the hardware supplier and the colo to get hardware in place and running, then reinvent the tooling. Any trip to the facility is half a day lost.
      - toomuchtodo 8y ago
        Many organizations run their own hardware successfully without the tooling I mention. There are problems, just different problems from cloud with significant cost savings realized. Why would you be fighting your hardware supplier or your colo? I have never had a hardware vendor "fight me", and only once had challenging negotiations with a colo provider. Have you never lost half a day of getting work done because Amazon's EC2 API was failing to launch new instances? I have! [1] To simply hand wave away hosting your own hardware as infeasible and more costly (especially when it's been proven to be more costly) is unproductive at best. [1] https://hn.algolia.com/?query=AWS%20outage&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=AWS%20outage&sort=byPopularity... (HN search of "AWS Outage")
  - deleted 8y ago
    [deleted]
  - drieddust 8y ago
    Servers are cheap enough so buying a few and keeping on standby is no brainer and they are good for 5 years generally.
- weinzierl 8y ago
  If you have someone who knows how to design systems and your workload is constant most of the time with occasioal spikes you could run your base load on your own machines and offload the spikes to AWS.
  - lykr0n 8y ago
    Yep. There is a 99.99% chance one of the following will happen: - Your Colo has direct connect to AWS, so you can deploy EC2 under times of load - Your Colo offers some sort of virtualized infrastructure that you could wire into your network and offload under times of load - Your Colo has a virtual server provider running that you can offload to in times of load. - Is close enough to one of the big 3 that a Site to Site will work fine
Tistel 8y ago
I have worked at a startup that was paying > 20k per month for AWS. I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. I was like: "if cost is a big issue, then lets spin up one beefy machine and create users for each dev", because thats what everyone using unix did back in the day (it works fine). The reply was "thats not possible" so every dev got their own dedicated instance that was 99% idle. We are paying >3k per month to aws and the business team is complaining about waste. sigh. old guy rant over.
- sigfubar 8y ago
  I've worked for a certain IoT startup that at one point paid $850k/month (eight hundred and fifty thousand USD per month) for AWS usage. After about a year of full time work by a team of SREs and programmers, the bill was made to go down by half.
  - h1d 8y ago
    Any breakdowns on cost?
- mancerayder 8y ago
  I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. Wait, you're saying there's a Bachelor of Science DevOps degree?
  - toomuchtodo 8y ago
    Documentation is no replacement for experience. EDIT: I agree that OPs choice of words makes the idea expressed ambiguous.
    - mancerayder 8y ago
      I just don't understand what a "self-taught" DevOps person is -- as opposed to a someone-taught? Self-taught usually means "not educated formally." All DevOps / SysAdmins / etc. are "self-taught." So it's unclear what the OP meant.
drieddust 8y ago
I agree completely. I consult with multiple customers and they all want cloud. It's "Cloud First" and any discussion to the contrary results in lost business. If you do that you are considered a Dinosaur. On the issue of predictable load, most businesses not only have predictable load but a lot of capacity is just allocated and never used. VMware serves such scenarios even better by going for heavy over allocation. Everyone will demand 10 servers with 8vCPU, 64 GB of RAM without actually using even 10% of the allocated capacity. But hey Cloud is cool so I have to be on it.
- ses1984 8y ago
  You can be cloud first and still not squander money on idle resources...
  - drieddust 8y ago
    That requires culture change and it's hard for big boys with checkbooks.
- scarface74 8y ago
  If all you’re doing with AWS is a bunch of EC2 instances - the equivalent of a bunch of VMs - yes you will spend more and not get any operational experiences. That’s like the old school Netops people who got one AWS certificate, did a lift and shift of a bunch of VMs and then proclaimed their client has “moved to the cloud”. The client still has the same amount of staff, same processes, etc. Of course it isn’t going to save any money.
  - drieddust 8y ago
    That's the reality of it. Any proper move requires longer term planning (> 2 years) and currently CIO themselves don't usually last 2 years. Any project however complex needs to be executed in a quarter which means "Lift and Shift". Sad reality.
user5994461 8y ago
You were lucky to have a guy with VMWare experience. Most startups who avoid the cloud go bare metal and that's completely unmanageable. Basically a pair of high memory servers running every services with no isolation whatsoever. By the way, Vmware is not exactly cheap either. I recall when I was selling VMWare licenses 10 years ago, it was starting around $5000 per server.
dba7dba 8y ago
I think VMware has expensive licensing. At least for me when I was just trying to set up home linux lab environment. And than I discovered Xen Project. I played around with that, and than AWS/DO came along.
- fosco 8y ago
  I've been tinkering with bhyve lately at home. there are definitely a few kinks to work out but I am _very_ happy with it and hope the community around it grows.

lykr0n 8y ago

This. If you can predict your workload with some accuracy, and have someone who knows how to design systems- you can save a lot of money. AWS' value is that "we can do everything except write your code for you" which you end up paying a lot for.

chomp 8y ago
Even if you can't predict the workload too well, you can go hybrid cloud - colo for the baseline and then AWS for the unpredictable stuff. ManageIQ is great for this.
ekovarski 8y ago
... isn't that the hardest part, predicting workload, for many companies. Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. If you can predict workload then you could also reduce your cloud bills, in theory, by moving spinning up/down instances. Many are going the cloud route because the cost of entry is much easier to start developing but eventually, yes, the cost skyrockets and you do end up paying for putting little forethought into the real workload requirements.
- toomuchtodo 8y ago
  > Scaling up within a colo can be troublesome due to procuring hw, having remote hands rack it, setup, etc. One senior sysadmin/infrastructure engineer with on-prem experience is enough to scale up your colo presence. You can either assemble your own servers or custom order and have them shipped directly to the colo. Remote console and power management (PDUs) is a solved problem. Out of band management, also solved. Virtualization, solved. Push your VM snapshots, content, artifacts, and database backups into your object store of choice for offsite backups. Ideally, your colo facility is within driving distance of your office if you absolutely need physical access. Otherwise, you pay someone at the colo a few hundred bucks a month for remote hands services. EDIT: k8s arguably makes this easier than in the past. You want to run on your own cluster? You can. You need to burst your containers to spot instances in someone else's cloud? You can.
  - user5994461 8y ago
    All the problems you consider solved are only solved if you have the tooling from AWS/Google or half solved if you have VMWare. With physical hardware, the sysadmin will spend a lot of his time fighting the hardware supplier and the colo to get hardware in place and running, then reinvent the tooling. Any trip to the facility is half a day lost.
    - toomuchtodo 8y ago
      Many organizations run their own hardware successfully without the tooling I mention. There are problems, just different problems from cloud with significant cost savings realized. Why would you be fighting your hardware supplier or your colo? I have never had a hardware vendor "fight me", and only once had challenging negotiations with a colo provider. Have you never lost half a day of getting work done because Amazon's EC2 API was failing to launch new instances? I have! [1] To simply hand wave away hosting your own hardware as infeasible and more costly (especially when it's been proven to be more costly) is unproductive at best. [1] https://hn.algolia.com/?query=AWS%20outage&sort=byPopularity&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=AWS%20outage&sort=byPopularity... (HN search of "AWS Outage")
- deleted 8y ago
  [deleted]
- drieddust 8y ago
  Servers are cheap enough so buying a few and keeping on standby is no brainer and they are good for 5 years generally.
weinzierl 8y ago
If you have someone who knows how to design systems and your workload is constant most of the time with occasioal spikes you could run your base load on your own machines and offload the spikes to AWS.
- lykr0n 8y ago
  Yep. There is a 99.99% chance one of the following will happen: - Your Colo has direct connect to AWS, so you can deploy EC2 under times of load - Your Colo offers some sort of virtualized infrastructure that you could wire into your network and offload under times of load - Your Colo has a virtual server provider running that you can offload to in times of load. - Is close enough to one of the big 3 that a Site to Site will work fine

Tistel 8y ago

I have worked at a startup that was paying > 20k per month for AWS. I am certain the whole system could run on one dedicated blade. There was just this culture (made easy with docker containers) of "spin up another box" for every silly little tool. I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. I was like: "if cost is a big issue, then lets spin up one beefy machine and create users for each dev", because thats what everyone using unix did back in the day (it works fine). The reply was "thats not possible" so every dev got their own dedicated instance that was 99% idle. We are paying >3k per month to aws and the business team is complaining about waste. sigh. old guy rant over.

sigfubar 8y ago
I've worked for a certain IoT startup that at one point paid $850k/month (eight hundred and fifty thousand USD per month) for AWS usage. After about a year of full time work by a team of SREs and programmers, the bill was made to go down by half.
- h1d 8y ago
  Any breakdowns on cost?
mancerayder 8y ago
I have met self taught dev ops people who did not understand that you could create more than one user on a aws ubuntu machine. Wait, you're saying there's a Bachelor of Science DevOps degree?
- toomuchtodo 8y ago
  Documentation is no replacement for experience. EDIT: I agree that OPs choice of words makes the idea expressed ambiguous.
  - mancerayder 8y ago
    I just don't understand what a "self-taught" DevOps person is -- as opposed to a someone-taught? Self-taught usually means "not educated formally." All DevOps / SysAdmins / etc. are "self-taught." So it's unclear what the OP meant.

drieddust 8y ago

I agree completely. I consult with multiple customers and they all want cloud. It's "Cloud First" and any discussion to the contrary results in lost business. If you do that you are considered a Dinosaur. On the issue of predictable load, most businesses not only have predictable load but a lot of capacity is just allocated and never used. VMware serves such scenarios even better by going for heavy over allocation. Everyone will demand 10 servers with 8vCPU, 64 GB of RAM without actually using even 10% of the allocated capacity. But hey Cloud is cool so I have to be on it.

ses1984 8y ago
You can be cloud first and still not squander money on idle resources...
- drieddust 8y ago
  That requires culture change and it's hard for big boys with checkbooks.
scarface74 8y ago
If all you’re doing with AWS is a bunch of EC2 instances - the equivalent of a bunch of VMs - yes you will spend more and not get any operational experiences. That’s like the old school Netops people who got one AWS certificate, did a lift and shift of a bunch of VMs and then proclaimed their client has “moved to the cloud”. The client still has the same amount of staff, same processes, etc. Of course it isn’t going to save any money.
- drieddust 8y ago
  That's the reality of it. Any proper move requires longer term planning (> 2 years) and currently CIO themselves don't usually last 2 years. Any project however complex needs to be executed in a quarter which means "Lift and Shift". Sad reality.

user5994461 8y ago

You were lucky to have a guy with VMWare experience. Most startups who avoid the cloud go bare metal and that's completely unmanageable. Basically a pair of high memory servers running every services with no isolation whatsoever. By the way, Vmware is not exactly cheap either. I recall when I was selling VMWare licenses 10 years ago, it was starting around $5000 per server.

dba7dba 8y ago

I think VMware has expensive licensing. At least for me when I was just trying to set up home linux lab environment. And than I discovered Xen Project. I played around with that, and than AWS/DO came along.

fosco 8y ago
I've been tinkering with bhyve lately at home. there are definitely a few kinks to work out but I am _very_ happy with it and hope the community around it grows.