9 ms·

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house

by mabbo 4y ago

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-house. At the right time, you make the right transitions.

lbriner 4y ago
Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electrician to fix some power issue? How much to setup and run a large air-con system for the data centre. Maybe not much in the US where aircon is common but much more expensive in Europe. The fact they could afford to do this over such a short time period speaks to having a decent amount of cash on-hand.
- hamandcheese 4y ago
  > At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. Co-locating has no capital investment other than hardware, and is pretty cheap. A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year. Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital. All this to say is that I don’t think capital is as big a factor as you might think.
  - boltzmann-brain 4y ago
    In my experience doing detailed projections of exactly something like this - with racks full of GPU compute power - the infrastructure has paid itself after one quarter, maybe two, depending on what volume you're at. There very rarely is a reason to use GPU compute on the cloud - and advantages start as quickly as with just one single GPU.
  - Corrado 4y ago
    I feel that a lot of posts like this might be under-representing the true costs of running your own hardware. I was only tangentially associated with a large-ish operation and I can tell you that there are loads of things that take a lot of time but are often over looked. Things like detecting and replacing bad hardware. HDs don't last forever and when they go bad it's not fun; especially if you have to source some specific model for your 7 year old server platform. Understanding your licensing and warranties is another huge cost that people don't take into account. We used to spend hours and hours figuring out if we could replace systems and what it would cost us. Finally, you have to dispose of all that hardware when it gets too old or out of warranty. If you've never had to do that you probably have no idea how hard it is to do it correctly and so that it satisfies your SOC2 auditor. All of these things (plus more) really add up. It's not just purchasing the components and installing them in racks. The management is probably even more expensive than the hardware. And all of these problems go away with a cloud provider.
    - Melatonic 4y ago
      Your VAR should be doing a lot of that work for you - sure you need to understand licensing but if you have a good rep they should be providing you with up to date info on all of this. HDD replacement is trivial and some COLOs can even do it for you - and any good vendor will have a thing setup where your appliance automatically notifies them and a new drive is overnighted to your COLO. Sure - maybe a few times a year you drive out to swap a drive. If you do not have good monitoring setup then that is entirely fixable - there are many stellar solutions these days out there. Hardware is easier than ever.
    - hamandcheese 4y ago
      Right, I agree that the cost might be underestimated, but I was specifically talking about the capital expenditure required to “eject” from the cloud, not the ongoing operating costs.
- jjav 4y ago
  > fix some power issue > large air-con system You wouldn't usually jump from AWS to buying up real estate to build your own physical data center. A sensible first step is to rent a rack at a colocation facility. They handle power, cooling, redundancy, physical access for you.
  - carlivar 4y ago
    I wonder if there's a whole new generation of engineers that don't realize this? After all, how would they know about co-location if only ever using the cloud? We already interview ops and network slanted candidates that don't know what ARP is (we've now added this as an early quick screen).
MajimasEyepatch 4y ago
Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they have to hire ten additional ops engineers to run it at a cost of $1 million per year, then they've actually gone and burned $200k on something that provides no additional value to the business or their customers.
- Dma54rhs 4y ago
  AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.
  - boltzmann-brain 4y ago
    Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.
    - monkpit 4y ago
      Brazil?
      - maxfurman 4y ago
        This is a reference to the film Brazil[0], which centers on a labyrinthine bureaucracy [0] https://www.imdb.com/title/tt0088846/ https://www.imdb.com/title/tt0088846/
        dalmo3 4y ago
        > labyrinthine bureaucracy This is a reference to the country Brazil [0], which centers on... [0] https://www.bbc.com/news/business-18020623 https://www.bbc.com/news/business-18020623
systemvoltage 4y ago
The initial bit is important though. It creates a circular dependency. If you start out without AWS, your entire company's software from how you build monoliths/microservices/queues changes. Look at Stack Overflow's architecture which stands apart because it was never designed to work in cloud from the beginning: https://stackexchange.com/performance https://stackexchange.com/performance I'd argue that 90% of the SaaS doesn't have SO's scale. The whole thing would work just fine on a couple of FreeBSD servers running postgres and un-dockerized monolith. Half a rack at most with redundancy and replication. But, if you've built your whole company around proprietary lamda functions and a vast range of AWS offerings, you're setting up yourself to never get out of the mess.
- jacobsenscott 4y ago
  Yes - but the FOMO industry is too strong.
- tekknik 4y ago
  > I'd argue that 90% of the SaaS doesn't have SO's scale. While Stack Overflow is popular in the dev world, not so much outside of it. The show they’re only serving 300 req/s. 19 servers to power the entire stack as well. I wouldn’t call this efficient.
- mr_toad 4y ago
  > The whole thing would work just fine on a couple of FreeBSD servers And you need dev servers. And a way to keep them in sync with production because you can’t just create a branch and provision a bare metal server on the fly. And you need a deployment process to get code from dev through test, UAT and production, and this process is lengthy and fraught because you can’t just fire up a new instance and switch the elastic IP, you actually have to deploy code on the physical machine and make sure there aren’t any configuration differences between your test and prod environments that’ll cause ‘it worked before’ problems. Developer productivity tanks, people are afraid to make changes, any pretence of agile/devops goes out the window and eventually everyone gets sick of the crusty old server and decides a complete rewrite is needed.
vlunkr 4y ago
Also your company needs to be mature enough to know really well what your hardware requirements are. For a growing company, it's really great to switch your RDS instance to a bigger type because your database load has tripled in a few months and you didn't know that was coming.
outworlder 4y ago
Thank you! At work, people keep complaining about our costs and coming up with spreadsheets showing how much money we would save with our own hardware. They never add the engineering costs. When they do, they forget to include the ongoing maintenance. Or the new SMEs that need to be hired (and on call). Or even the opportunity cost of doing a multi-year migration to arrive at the exact spot they already are today. All that money, and noone is looking into optimizing our systems to shrink the bill...
i_have_an_idea 4y ago
Anecdotal, but for one of my projects, Google Cloud / Compute Engine VMs cost around ~$5k a month all in. The exact same setup, when we moved it to LiquidWeb, cost us $2k. Don't underestimate the savings that can be made from switching from a big-name cloud provider to a more old school hosting provider.
hintymad 4y ago
People don't consider productivity? Maybe things have gotten a lot better in the industry now. Otherwise, to rehash an older comment on HN: I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up and what the funk is eBPF? Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.
- humanwhosits 4y ago
  and they had trouble moving workloads to the cloud because bringing up new capacity was a giant set of circular microservice dependencies
- Melatonic 4y ago
  That also just sounds like Uber had hired crap talent....
  - GauntletWizard 4y ago
    And crap management to lead them. Uber hired plenty of people smart enough to do better, but let the crap take the reins and management failed to lead on anything.
deleted 4y ago
[deleted]
xani_ 4y ago
80k ? Honestly probably does. Our ops team of 3 spends maybe 10% of the time on the managing of few hardware racks we have in our local colocation. There are even months where nothing at hardware/hypervisor level is touched
- oxfordmale 4y ago
  3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you. You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.
  - xani_ 4y ago
    > 3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you. which is less than 80k, yes. > You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance. Please stop assuming stupid shit, ask for clarification if you're not sure what I included in it. Racking 2 racks took us 2 days, one on site, on on configuration. Patching is also included in that 10%. We track that time. If you're patching manually you already fucked up, cloud or no cloud. Only time spent actually patching is "difficult" customers that need paperwork for taking down even one server of HA pair. And it's utterly ignorant to assume cloud somehow saves you on this point, you just have more time used dealing on security stuff on dev side instead of ops side. Hell, I dare to say average VM with auto-updates will be more secure than average container deployment. Or maybe that's our devs that didn't get kicked enough to set their container pipeline correctly...
    - oxfordmale 4y ago
      Cloud is generally more expensive by about 20-30%, I however, I stand by my point you are underestimating your costs. What about hardware deprecation? It is something you generally don't have to consider in the cloud.
senectus1 4y ago
this point is even more poignant when taking into account scalability and elasticity. Our usage of Azure/O365/AWS saved us a lot of time staff and money during our make it or break it growth period. now we're a lot bigger and more stable we're having to reconsider allowing MS/Amazon bending us over the barrel quite so much.
chii 4y ago
> This company saved $800k/year. Perfect time to go in-house with this solution. $800k/yr is like the cost of 2-3 engineers. Even if they're capable of doing all of the work that used to have been done by aws, you don't have any room to expand without having to put up high capital costs, and certainly not on a dime. It sounds amazing now, but wait till the future comes and you can no longer run your own data center as cheaply as amazon.
hisorange 4y ago
Hey there, (Zsolt Varga here) Yep, we should have pointed out that this has to be done in at the right time. Generally this article was about how we saved $1M expense and we was able to share this saving with our customers. Not an attack on AWS, I still use their platform for my other projects. But to be accurate, this migration happened last year, and now we already doubled our usage and this brings us to $2M / year saving now. I 100% agree with the comments below, use AWS while you are defining your product and growing, Todd (the founder) did this right. Spending days and nights on managing your own infrastructure or paying a devops employee to do it, in the early phase is just pure waste, and takes away focus. But we moved away from AWS at the right time. Also, yes we had to do this to be competitive, as prerendering is a resource heavy service, and if your site is like ProductBoard (awesome tool) which will not consume terabits of traffic or use petabytes of ram, then you can stay on AWS forever and enjoy the benefit of not caring. And what we lost? Minor inconvenience at this point, I cannot just pull up a new database in 10 minutes, but we don't really do that anymore, most of our current projects requires months of research, planning, and delivery. With those time frames we can notify our devops and get everything in place way before we need it. So, keep on using AWS (it rocks!), just be sure to pay attention to the bill, and don't underestimate the cost of the traffic. Have a nice one

by mabbo 4y ago

lbriner 4y ago

Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electrician to fix some power issue? How much to setup and run a large air-con system for the data centre. Maybe not much in the US where aircon is common but much more expensive in Europe. The fact they could afford to do this over such a short time period speaks to having a decent amount of cash on-hand.

hamandcheese 4y ago
> At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. Co-locating has no capital investment other than hardware, and is pretty cheap. A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year. Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital. All this to say is that I don’t think capital is as big a factor as you might think.
- boltzmann-brain 4y ago
  In my experience doing detailed projections of exactly something like this - with racks full of GPU compute power - the infrastructure has paid itself after one quarter, maybe two, depending on what volume you're at. There very rarely is a reason to use GPU compute on the cloud - and advantages start as quickly as with just one single GPU.
- Corrado 4y ago
  I feel that a lot of posts like this might be under-representing the true costs of running your own hardware. I was only tangentially associated with a large-ish operation and I can tell you that there are loads of things that take a lot of time but are often over looked. Things like detecting and replacing bad hardware. HDs don't last forever and when they go bad it's not fun; especially if you have to source some specific model for your 7 year old server platform. Understanding your licensing and warranties is another huge cost that people don't take into account. We used to spend hours and hours figuring out if we could replace systems and what it would cost us. Finally, you have to dispose of all that hardware when it gets too old or out of warranty. If you've never had to do that you probably have no idea how hard it is to do it correctly and so that it satisfies your SOC2 auditor. All of these things (plus more) really add up. It's not just purchasing the components and installing them in racks. The management is probably even more expensive than the hardware. And all of these problems go away with a cloud provider.
  - Melatonic 4y ago
    Your VAR should be doing a lot of that work for you - sure you need to understand licensing but if you have a good rep they should be providing you with up to date info on all of this. HDD replacement is trivial and some COLOs can even do it for you - and any good vendor will have a thing setup where your appliance automatically notifies them and a new drive is overnighted to your COLO. Sure - maybe a few times a year you drive out to swap a drive. If you do not have good monitoring setup then that is entirely fixable - there are many stellar solutions these days out there. Hardware is easier than ever.
  - hamandcheese 4y ago
    Right, I agree that the cost might be underestimated, but I was specifically talking about the capital expenditure required to “eject” from the cloud, not the ongoing operating costs.
jjav 4y ago
> fix some power issue > large air-con system You wouldn't usually jump from AWS to buying up real estate to build your own physical data center. A sensible first step is to rent a rack at a colocation facility. They handle power, cooling, redundancy, physical access for you.
- carlivar 4y ago
  I wonder if there's a whole new generation of engineers that don't realize this? After all, how would they know about co-location if only ever using the cloud? We already interview ops and network slanted candidates that don't know what ARP is (we've now added this as an early quick screen).

MajimasEyepatch 4y ago

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they have to hire ten additional ops engineers to run it at a cost of $1 million per year, then they've actually gone and burned $200k on something that provides no additional value to the business or their customers.

Dma54rhs 4y ago
AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.
- boltzmann-brain 4y ago
  Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.
  - monkpit 4y ago
    Brazil?
    - maxfurman 4y ago
      This is a reference to the film Brazil[0], which centers on a labyrinthine bureaucracy [0] https://www.imdb.com/title/tt0088846/ https://www.imdb.com/title/tt0088846/
      - dalmo3 4y ago
        > labyrinthine bureaucracy This is a reference to the country Brazil [0], which centers on... [0] https://www.bbc.com/news/business-18020623 https://www.bbc.com/news/business-18020623

systemvoltage 4y ago

The initial bit is important though. It creates a circular dependency. If you start out without AWS, your entire company's software from how you build monoliths/microservices/queues changes. Look at Stack Overflow's architecture which stands apart because it was never designed to work in cloud from the beginning: https://stackexchange.com/performance https://stackexchange.com/performance I'd argue that 90% of the SaaS doesn't have SO's scale. The whole thing would work just fine on a couple of FreeBSD servers running postgres and un-dockerized monolith. Half a rack at most with redundancy and replication. But, if you've built your whole company around proprietary lamda functions and a vast range of AWS offerings, you're setting up yourself to never get out of the mess.

jacobsenscott 4y ago
Yes - but the FOMO industry is too strong.
tekknik 4y ago
> I'd argue that 90% of the SaaS doesn't have SO's scale. While Stack Overflow is popular in the dev world, not so much outside of it. The show they’re only serving 300 req/s. 19 servers to power the entire stack as well. I wouldn’t call this efficient.
mr_toad 4y ago
> The whole thing would work just fine on a couple of FreeBSD servers And you need dev servers. And a way to keep them in sync with production because you can’t just create a branch and provision a bare metal server on the fly. And you need a deployment process to get code from dev through test, UAT and production, and this process is lengthy and fraught because you can’t just fire up a new instance and switch the elastic IP, you actually have to deploy code on the physical machine and make sure there aren’t any configuration differences between your test and prod environments that’ll cause ‘it worked before’ problems. Developer productivity tanks, people are afraid to make changes, any pretence of agile/devops goes out the window and eventually everyone gets sick of the crusty old server and decides a complete rewrite is needed.

vlunkr 4y ago

Also your company needs to be mature enough to know really well what your hardware requirements are. For a growing company, it's really great to switch your RDS instance to a bigger type because your database load has tripled in a few months and you didn't know that was coming.

outworlder 4y ago

Thank you! At work, people keep complaining about our costs and coming up with spreadsheets showing how much money we would save with our own hardware. They never add the engineering costs. When they do, they forget to include the ongoing maintenance. Or the new SMEs that need to be hired (and on call). Or even the opportunity cost of doing a multi-year migration to arrive at the exact spot they already are today. All that money, and noone is looking into optimizing our systems to shrink the bill...

i_have_an_idea 4y ago

Anecdotal, but for one of my projects, Google Cloud / Compute Engine VMs cost around ~$5k a month all in. The exact same setup, when we moved it to LiquidWeb, cost us $2k. Don't underestimate the savings that can be made from switching from a big-name cloud provider to a more old school hosting provider.

hintymad 4y ago

People don't consider productivity? Maybe things have gotten a lot better in the industry now. Otherwise, to rehash an older comment on HN: I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up and what the funk is eBPF? Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.

humanwhosits 4y ago
and they had trouble moving workloads to the cloud because bringing up new capacity was a giant set of circular microservice dependencies
Melatonic 4y ago
That also just sounds like Uber had hired crap talent....
- GauntletWizard 4y ago
  And crap management to lead them. Uber hired plenty of people smart enough to do better, but let the crap take the reins and management failed to lead on anything.

deleted 4y ago

[deleted]

xani_ 4y ago

80k ? Honestly probably does. Our ops team of 3 spends maybe 10% of the time on the managing of few hardware racks we have in our local colocation. There are even months where nothing at hardware/hypervisor level is touched

oxfordmale 4y ago
3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you. You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.
- xani_ 4y ago
  > 3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you. which is less than 80k, yes. > You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance. Please stop assuming stupid shit, ask for clarification if you're not sure what I included in it. Racking 2 racks took us 2 days, one on site, on on configuration. Patching is also included in that 10%. We track that time. If you're patching manually you already fucked up, cloud or no cloud. Only time spent actually patching is "difficult" customers that need paperwork for taking down even one server of HA pair. And it's utterly ignorant to assume cloud somehow saves you on this point, you just have more time used dealing on security stuff on dev side instead of ops side. Hell, I dare to say average VM with auto-updates will be more secure than average container deployment. Or maybe that's our devs that didn't get kicked enough to set their container pipeline correctly...
  - oxfordmale 4y ago
    Cloud is generally more expensive by about 20-30%, I however, I stand by my point you are underestimating your costs. What about hardware deprecation? It is something you generally don't have to consider in the cloud.

senectus1 4y ago

this point is even more poignant when taking into account scalability and elasticity. Our usage of Azure/O365/AWS saved us a lot of time staff and money during our make it or break it growth period. now we're a lot bigger and more stable we're having to reconsider allowing MS/Amazon bending us over the barrel quite so much.

chii 4y ago

> This company saved $800k/year. Perfect time to go in-house with this solution. $800k/yr is like the cost of 2-3 engineers. Even if they're capable of doing all of the work that used to have been done by aws, you don't have any room to expand without having to put up high capital costs, and certainly not on a dime. It sounds amazing now, but wait till the future comes and you can no longer run your own data center as cheaply as amazon.

hisorange 4y ago

Hey there, (Zsolt Varga here) Yep, we should have pointed out that this has to be done in at the right time. Generally this article was about how we saved $1M expense and we was able to share this saving with our customers. Not an attack on AWS, I still use their platform for my other projects. But to be accurate, this migration happened last year, and now we already doubled our usage and this brings us to $2M / year saving now. I 100% agree with the comments below, use AWS while you are defining your product and growing, Todd (the founder) did this right. Spending days and nights on managing your own infrastructure or paying a devops employee to do it, in the early phase is just pure waste, and takes away focus. But we moved away from AWS at the right time. Also, yes we had to do this to be competitive, as prerendering is a resource heavy service, and if your site is like ProductBoard (awesome tool) which will not consume terabits of traffic or use petabytes of ram, then you can stay on AWS forever and enjoy the benefit of not caring. And what we lost? Minor inconvenience at this point, I cannot just pull up a new database in 10 minutes, but we don't really do that anymore, most of our current projects requires months of research, planning, and delivery. With those time frames we can notify our devops and get everything in place way before we need it. So, keep on using AWS (it rocks!), just be sure to pay attention to the bill, and don't underestimate the cost of the traffic. Have a nice one