8 ms·

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$

by warent 5y ago

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is better than you might imagine. Being in full control--the master of your own destiny--is so liberating and empowering. It feels the difference between constantly ordering Lyft/Uber/riding with friends, vs. owning your own car. Not to mention, again, my hardware resources are so much better. This one server can run multiple profitable SaaS apps / businesses and still have room for experimental projects and market tests. Couldn't be happier with my decision to get off the cloud.

deleted 5y ago
[deleted]
baybal2 5y ago
> it pays for itself in less than a year. https://news.ycombinator.com/item?id=13198157 https://news.ycombinator.com/item?id=13198157 On one meeting we had a typical discussion with ops guys: - "why wouldn't we optimise our hardware utilisation by doing things a, b, and c." - "hardware is crap cheap these days. If you need more capacity, just throw more servers at that" - "is $24k a month in new servers crap cheap by your measure?" - "comparatively to the amount of how much money these servers will make the same month, it is crap cheap. It is just a little less than an annual cost of mid-tier software dev in Russian office. We account only 12% increase in our revenue due to algorithmic improvements and almost 80 to more traffic we handle. A new server pays back the same month, and you and other devs pay off only in 2 years"
- jiggawatts 5y ago
  I’ve found this to be an unsuccessful approach in practice. Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify. Customers leave in disgust because the site is slow. No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere. Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing. I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content. Invariably, doing the right thing from the beginning would have been cheaper. Listen to Casey explain it: https://youtu.be/pgoetgxecw8 https://youtu.be/pgoetgxecw8 You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer. Learn your craft properly so you can do more with less — including less developer time!
rank0 5y ago
Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.
- qmarchi 5y ago
  Hey, you actually have a few options, notably, doing nothing! Comcast doesn't actually change your public IP address between DHCP renewals and thus it's effectively static. The only time that it'll change is when the modem is powered off for an amount of time, or the upstream DOCSIS concentrator is powered off for maintenance or otherwise.
  - orf 5y ago
    So: arbitrarily and without warning.
    - sokoloff 5y ago
      Sure; however, I’ve had the same IP on Comcast residential for a little over a decade. There are many endeavors that could fade that low a frequency of IP changing.
  - mwcampbell 5y ago
    I would be more worried about violating the ISP's terms of service. Running a business based on that seems pretty precarious.
    - Nextgrid 5y ago
      What’s the worst that can realistically happen?
      - awslattery 5y ago
        If you don't have another provider option, you stand to lose Internet as a utility for your residence, which many would consider a pretty poor situation.
pmarreck 5y ago
Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your version upgrade strategy? What's your OS upgrade strategy? Failover? IPv6? VPN access? DMZ? Basically, I think cloud provides a loooooot of details that you have to now take on yourself if you self-host (at least if you want to do it "legitimately and professionally" as a reliable service). It's not clearly a win-win. That all said, I recently canceled my cloud9 dev account at amazon because the resources I needed were getting too expensive, and am self-hosting my new dev env in a VM and accessing it from anywhere via Tailscale, so that's been nice.
- oceanplexian 5y ago
  > Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your version upgrade strategy? What's your OS upgrade strategy? Failover? IPv6? VPN access? DMZ? So yes, for those of us who have done Systems Administration as a lifestyle/career, yeah you do all of those things and it's part of the fun. I started doing OS upgrades, monitoring, firewalls, and home backups of my own Linux Servers some time in High School. Over-utilization of bandwidth isn't really a "problem" unless you're doing something weird like streaming video, a 1Gbps circuit can support thousands upon thousands of requests per second.
aledalgrande 5y ago
I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets flooded or your machine blows up? What would you do in case a malicious process takes over the machine? These are some things that are managed in a cloud environment. I understand investing in a datacenter rack where you own your hardware, if you have the skills, but running it in a home office cannot support a successful business nowadays IMO.
- TacticalCoder 5y ago
  > How do you handle traffic spikes, especially from the networking point of view? I don't know about GP but managing your own server doesn't mean you cannot use a CDN with your webapp.
  - aledalgrande 5y ago
    A CDN wouldn't be enough if the traffic spike involves writes.
    - giantrobot 5y ago
      Oh no a website went down! Oh, wait, that's not an emergency. Where did the idea that every site and service needs five nines availability come from? A side project goes down or is read only for a few hours. Who gives a shit? It's not an insulin pump or nuclear control rods. It's ok for people to be mildly inconvenienced for a short period of time.
      - kafkaIncarnate 5y ago
        "Always-on culture" I guess people who use Twitter don't like seeing bad things written about them on Twitter. Though the solution is to disown Twitter and all who use it.
deepspace 5y ago
> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has a limited lifespan, so the $3000 was never a one-time investment. I think there is often still a case to be made for self-hosting but the numbers are not as rosy as they seem at first glance.
- dijit 5y ago
  You need to replicate in Cloud too, most people tend not to because they think the cloud is magic, but it's computers and computers can fail- even if they're someone else's. Also "if some component fails or the app is critical" has a lot of nuance, I agree with your sentiment but you should know: 1) Component failures in hardware are much rarer than you think 2) Component failures in hardware can be mitigated (dead ram, dead PSU, dead hard disk, even dead CPUs in some cases: all mitigated) The only true failure of a machine is an unmitigated failure due to not configuring memory mirroring or something' or a motherboard failure (which is extremely uncommon) 3) The next step after "single server" isn't "build a datacenter", it's buying a couple more servers and renting half a rack from your local datacenter, they'll have redundant power, redundant cooling and redundant networking. They'll even help you get set up if it's 2-3 machines with their own hardware techs. I do this last one at a larger scale in Bahnhof. also, $3000 will get you about 3-5 years out of hardware, at which point, yeah, you should think about upgrading, if for no other reason than it's going to be slower.
  - Lamad123 5y ago
    I don't know what they call that logical fallacy cloud fanatics use when they say "if blah blah just make build your own datacenter".
- StillBored 5y ago
  So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of it gets retired because its functionally obsolete, not because a power supply/whatever fails. Which brings up the point, that the vast number of failures with server grade hardware are on replaceable components like power supplies, disks, SFP's, etc. Three or four years out those parts are available on the secondary markets frequently for pocket change.
0xbadcafebee 5y ago
Personally I wouldn't want to become an auto mechanic just to drive my own car for my business, but you do you. (Makes sense if you have a fleet of vehicles, but for one car?)
ozim 5y ago
When I read small/medium business administrators horror stories on reddit and other forums I think IaaS is a god send. Especially when someone else is making money decisions. Of course in horror stories admins have to run their servers in broom closets or in sheds because business owner is too cheap to get a proper space for something that whole company is running.

by warent 5y ago

deleted 5y ago

[deleted]

baybal2 5y ago

> it pays for itself in less than a year. https://news.ycombinator.com/item?id=13198157 https://news.ycombinator.com/item?id=13198157 On one meeting we had a typical discussion with ops guys: - "why wouldn't we optimise our hardware utilisation by doing things a, b, and c." - "hardware is crap cheap these days. If you need more capacity, just throw more servers at that" - "is $24k a month in new servers crap cheap by your measure?" - "comparatively to the amount of how much money these servers will make the same month, it is crap cheap. It is just a little less than an annual cost of mid-tier software dev in Russian office. We account only 12% increase in our revenue due to algorithmic improvements and almost 80 to more traffic we handle. A new server pays back the same month, and you and other devs pay off only in 2 years"

jiggawatts 5y ago
I’ve found this to be an unsuccessful approach in practice. Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify. Customers leave in disgust because the site is slow. No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere. Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing. I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content. Invariably, doing the right thing from the beginning would have been cheaper. Listen to Casey explain it: https://youtu.be/pgoetgxecw8 https://youtu.be/pgoetgxecw8 You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer. Learn your craft properly so you can do more with less — including less developer time!

rank0 5y ago

Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

qmarchi 5y ago
Hey, you actually have a few options, notably, doing nothing! Comcast doesn't actually change your public IP address between DHCP renewals and thus it's effectively static. The only time that it'll change is when the modem is powered off for an amount of time, or the upstream DOCSIS concentrator is powered off for maintenance or otherwise.
- orf 5y ago
  So: arbitrarily and without warning.
  - sokoloff 5y ago
    Sure; however, I’ve had the same IP on Comcast residential for a little over a decade. There are many endeavors that could fade that low a frequency of IP changing.
- mwcampbell 5y ago
  I would be more worried about violating the ISP's terms of service. Running a business based on that seems pretty precarious.
  - Nextgrid 5y ago
    What’s the worst that can realistically happen?
    - awslattery 5y ago
      If you don't have another provider option, you stand to lose Internet as a utility for your residence, which many would consider a pretty poor situation.

pmarreck 5y ago

Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your version upgrade strategy? What's your OS upgrade strategy? Failover? IPv6? VPN access? DMZ? Basically, I think cloud provides a loooooot of details that you have to now take on yourself if you self-host (at least if you want to do it "legitimately and professionally" as a reliable service). It's not clearly a win-win. That all said, I recently canceled my cloud9 dev account at amazon because the resources I needed were getting too expensive, and am self-hosting my new dev env in a VM and accessing it from anywhere via Tailscale, so that's been nice.

oceanplexian 5y ago
> Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your version upgrade strategy? What's your OS upgrade strategy? Failover? IPv6? VPN access? DMZ? So yes, for those of us who have done Systems Administration as a lifestyle/career, yeah you do all of those things and it's part of the fun. I started doing OS upgrades, monitoring, firewalls, and home backups of my own Linux Servers some time in High School. Over-utilization of bandwidth isn't really a "problem" unless you're doing something weird like streaming video, a 1Gbps circuit can support thousands upon thousands of requests per second.

aledalgrande 5y ago

I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets flooded or your machine blows up? What would you do in case a malicious process takes over the machine? These are some things that are managed in a cloud environment. I understand investing in a datacenter rack where you own your hardware, if you have the skills, but running it in a home office cannot support a successful business nowadays IMO.

TacticalCoder 5y ago
> How do you handle traffic spikes, especially from the networking point of view? I don't know about GP but managing your own server doesn't mean you cannot use a CDN with your webapp.
- aledalgrande 5y ago
  A CDN wouldn't be enough if the traffic spike involves writes.
  - giantrobot 5y ago
    Oh no a website went down! Oh, wait, that's not an emergency. Where did the idea that every site and service needs five nines availability come from? A side project goes down or is read only for a few hours. Who gives a shit? It's not an insulin pump or nuclear control rods. It's ok for people to be mildly inconvenienced for a short period of time.
    - kafkaIncarnate 5y ago
      "Always-on culture" I guess people who use Twitter don't like seeing bad things written about them on Twitter. Though the solution is to disown Twitter and all who use it.

deepspace 5y ago

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has a limited lifespan, so the $3000 was never a one-time investment. I think there is often still a case to be made for self-hosting but the numbers are not as rosy as they seem at first glance.

dijit 5y ago
You need to replicate in Cloud too, most people tend not to because they think the cloud is magic, but it's computers and computers can fail- even if they're someone else's. Also "if some component fails or the app is critical" has a lot of nuance, I agree with your sentiment but you should know: 1) Component failures in hardware are much rarer than you think 2) Component failures in hardware can be mitigated (dead ram, dead PSU, dead hard disk, even dead CPUs in some cases: all mitigated) The only true failure of a machine is an unmitigated failure due to not configuring memory mirroring or something' or a motherboard failure (which is extremely uncommon) 3) The next step after "single server" isn't "build a datacenter", it's buying a couple more servers and renting half a rack from your local datacenter, they'll have redundant power, redundant cooling and redundant networking. They'll even help you get set up if it's 2-3 machines with their own hardware techs. I do this last one at a larger scale in Bahnhof. also, $3000 will get you about 3-5 years out of hardware, at which point, yeah, you should think about upgrading, if for no other reason than it's going to be slower.
- Lamad123 5y ago
  I don't know what they call that logical fallacy cloud fanatics use when they say "if blah blah just make build your own datacenter".
StillBored 5y ago
So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of it gets retired because its functionally obsolete, not because a power supply/whatever fails. Which brings up the point, that the vast number of failures with server grade hardware are on replaceable components like power supplies, disks, SFP's, etc. Three or four years out those parts are available on the secondary markets frequently for pocket change.

0xbadcafebee 5y ago

Personally I wouldn't want to become an auto mechanic just to drive my own car for my business, but you do you. (Makes sense if you have a fleet of vehicles, but for one car?)

ozim 5y ago

When I read small/medium business administrators horror stories on reddit and other forums I think IaaS is a god send. Especially when someone else is making money decisions. Of course in horror stories admins have to run their servers in broom closets or in sheds because business owner is too cheap to get a proper space for something that whole company is running.