10 ms·

It always makes my head dizzle when I read and hear how much people manage to bloat their stack because they think they need to "scale". I have web application

by TekMol 6y ago

It always makes my head dizzle when I read and hear how much people manage to bloat their stack because they think they need to "scale". I have web applications running that reliably serve 50k users per day and cost me $10/month, running on a single, cheap VPS.

tsumnia 6y ago
Not to sound mean, but if there aren't posts/blogs/whatever explicitly telling people how to minimize costs... then they'll continue to follow the ones that make them pay $100+.
- rapsey 6y ago
  Or people could do some quick research what options there are. Not everything needs to be spoon fed.
- fock 6y ago
  basically most people don't do that for saving costs, but just because it makes sense to them. And there's lots of tutorials explaining how to run your own webapp.
- TekMol 6y ago
  Ok, here is a quick tutorial: Step 1: Copy Peter Levels stack: https://www.nocsdegree.com/pieter-levels-learn-coding/ https://www.nocsdegree.com/pieter-levels-learn-coding/ If he serves hundreds of thousands of monthly users and makes a million a month with a single VPS, you certainly won't have scaling issues when you start out or just have tens of thousands of users. Step 2: If you really scale beyond that, resist the urge to bloat your stack. Think long and hard about every piece you add to it. Really understand each piece you add to it. Don't fall into the trap of paid services. Don't fall into the trap of "best practices".
  - chillfox 6y ago
    I don’t know how many times I have seen “best practices” used as excuse to avoid thinking. Usually by people who don’t even have the problem that the supposed “best practices” is meant to solve.
  - young_unixer 6y ago
    > PHP (He doesn't use any frameworks like Laravel) I wouldn't want to read (or write) that code, but thanks for the article.
    - paul_f 6y ago
      Well, aren't you special. How about staying on topic and not dropping in this, what, developer virtue signaling?
      - afarrell 6y ago
        > staying on topic A major subtopic of this thread is "different situations call for different setups." Saying "I won't work with PHP" is like saying "Gluten will make me shit my brains out." It is not virtue signalling.
  - tsumnia 6y ago
    While this article has steps in the right direction, it's a lot of "use this thing". I operate a simple Flask/PostgreSQL webapp for CS Education through the luxury of my university. If/When I graduate from my program, what are the aspects I can do to minimize my costs for hosting the app? Of course I'll look for best approaches but why does that need to be forbidden knowledge known to a select few? Its an aside, but the mentality of "let them figure it out" is a major issue in education. Foundational knowledge should be easy to acquire so I can worry about higher level thinking issues. Literally spending hours trying to figure out how to set things up through hours of Googling doesn't really help that, nor does it promote the "figuring it out" people think it does - its just stumbling upon the right set of commands that let me move past this particular hurdle. From the devops perspective, what about telling someone how to set up their own server to do/minimize X is so taxing?
    - richiecute 6y ago
      I run a cheap website and here is what I do - cloudflare free tier for caching, DNS, page rules, etc - run everything on one VPS(digital ocean, linode, etc) pick cheapest that has specs you need - any non-trivial storage (media, big files) move to Backblaze B2 it's cheap (you can use free tier cloudflare workers to redirect to B2 for free bandwidth due to Bandwitdth Alliance) - free static page from Netifly (I can redirect to this with cloudflare in case my VPS falls over or something to provide info/links) - If I want to look at logs or something I rsync it my local machine (if I cared I could set up a process to push logs/backup etc to private B2 bucket) You may not need exact same setup, I am optimizing for caching and cheap storage because my site stores/serves lots of media files.
    - yabai_yatsu 6y ago
      >Literally spending hours trying to figure out how to set things up through hours of Googling doesn't really help that, nor does it promote the "figuring it out" people think it does - its just stumbling upon the right set of commands that let me move past this particular hurdle. That's basically all of software development for your entire career. Never not had a day or a week not like that.
      - 6y ago
  - renewiltord 6y ago
    Haha, the man is clearly a beast of a product designer and executor. I don't think I would succeed with the same tools. Honestly, spending $200/mo is insignificant to me. And I'm pretty happy to answer the question of "Why can this guy build this thing on a single VPS and you can't?" with "Well, because he's better than me". I can be up in 15 mins on Heroku with a Rails+React web app. And in an hour have a thing. Or for a static no-login thing, faster with Netlify. But it doesn't matter. Because I never made a product as nice as the one he made. If the outcome is I'm -$200/mo that is irrelevant to me. If the outcome is I'm +$50k/mo that is very relevant. So I'm going to optimize for how I can do the latter.
  - srtjstjsj 6y ago
    If I have a "million a month" business (which is coming from rich recruitijg fees on a trivial website, not amazing tech), no way am I sweating $171/mo in server bills.
- aahortwwy 6y ago
  Don't do blue-green deploys when you have no revenue. Downtime costs you nothing and at 3,400 visitors a day you'll be lucky to drop a single request while deploying. Pre-compute and cache to reduce your need for beefy servers. When you can run something on your machine or in the cloud, choose your machine. EDIT: It's less about saving money and more about not spending it. EDIT2: Forgot to mention: use SQLite.
  - chillfox 6y ago
    SQLite Really is a bit of a hidden gem.
    - rapsey 6y ago
      Not really hidden when it is one of the most widely deployed pieces of software on the planet.
      - bigiain 6y ago
        Sure, but as an underlying dependency for iOS/Android, it's pretty well hidden from casual users...
        krtkush 6y ago
        That's true for even a casual website user. They have no idea or care about the underlying database. Every app developer is aware about SQLite.
      - chillfox 6y ago
        I guess not in absolute terms, but it’s mostly overlooked for web apps.
  - stevekemp 6y ago
    Caching is, in general, a good thing. But it's worth thinking about how you do it. I recently sped up a large system by dropping the use of redis entirely - because it was being badly used. During a single request there might be 50+ cache-lookups, each taking a round-trip to a remote redis server to fetch a single key at a time. Batching those up to a set/hash would have been more efficient, but the codebase had evolved in such a way as to make that difficult. Instead of making 50+ redis fetches it turned out that just fetching all the stuff from the database was faster. (There will be refactoring to batch up the key fetches, but for the moment there was a measurable increase in performance under current loads just by removing redis.)
threeseed 6y ago
It always amazes me how people are so free to judge other's situations. It's great that your web app only costs $10/month but others may have web apps that are more computationally intensive e.g. video processing or ML inference etc or simply can't join everything they need at runtime. And it's great that you're willing to deny those 50k users a day access to your service when that cheap VPS inevitably falls over. But others may be monetising that traffic and will want a HA solution so their revenue isn't impacted. All of those add complexity and cost to an architecture.
- TekMol 6y ago
  The article does not mention video processing or ML. It describes their setup for what they describe as a generic web app.
  - FrozenSynapse 6y ago
    If you want to provide uninterrupted service to your clients you’ll have to spend some $. You want to have redundancy, machines hosted in different different locations, backup prod servers, monitoring, analysis tools. Even if it is for 1k monthly users, if you want reliability - it will increase the costs.
    - TekMol 6y ago
      I beg to differ. In my experience, the complicated setups that are justified by the argument of "reliability" have more downtime then a single VPS. The reason is probably that there are more moving parts and more has to be maintained / can go wrong. These days, a single VPS in the right datacenter has excellent uptime.
      - tristanperry 6y ago
        Agreed. I'll probably be downvoted, but these setups strike me as people who prefer to drink the cool aid than be pragmatic and only use what they need. I've also had very high reliability rates with a single VPS. They've actually given me less downtime than AWS services at times.
caspii 6y ago
I'v added this paragraph to the article: the servers are oversized for the load we're currently seeing. The reason for that is that we tried to solve a production issue by increasing the server specs. It didn't solve the problem, and now we can't down-size the servers without re-provisioning them ️.
- mkesper 6y ago
  But as they are redundant this should be a no-brainer?
- pastage 6y ago
  I have found the same with digital ocean it is hard to down size but that is only true because I have so much unicorn data, and a move has become very painfull, so I'm stuck paying for 2x the storage cost. You should really learn to add new servers by provisioning them and remove the old servers. Your app seems a perfect fit for it for moving between servers.
- IanCal 6y ago
  What's the problem with reprovisioning? Don't you have two that you're switching between?
bigiain 6y ago
I get what you're saying, but I think you're being a bit harsh. In my mind, choosing a dual redundant prod platform so you can do blue/green deployments is _totally_ unnecessary for a 55k MAU site generating no revenue. Same with cloud hosted metabase. You could run that on your own hardware - a spare laptop or probably even a raspberry pi for effectively nothing. On the other hand, a hobby project/side gig where you can demonstrate real world experience in those two things could _easily_ pay off in the first week of a new job it helps you land. If it's _just_ that extra $100/month they're spending there, it seems difficult to justify. If it's commercial experience doing that which is helping him try and land a job paying 20k or more extra per year? That's totally money well spent, in my opinion...
- TekMol 6y ago
  If we were on a site called "CV News" that teaches people how to get jobs in dull cooperations, I might agree with you. On the other hand, I would not frequent such a site. So we would never have had this discussion.
  - bigiain 6y ago
    Yeah fair cop...
  - iakov 6y ago
    You can apply the skills that you learn in any job, not just the dull ones. The skills that you learn by implementing this kind of setup are valuable to lots of interesting jobs I'd say.
- maxk42 6y ago
  Given that he's complaining about the price I don't think the wasteful costs here are justified. I'll also note the article was just edited and the costs are now up to $171 / month. This app could easily be run on the AWS free tier, although even in the paid tier he could probably be managing a lot more than this workload for under $40 / month. (That's for two servers - which as has been pointed-out is wholly unnecessary for an app of this size.) The price he's paying is presently listed at $95.
mjburgess 6y ago
The additional expense in OP had nothing to do with scale; it arises from: redundancy, analytics, off-the-shelf integrations. It is a common misunderstanding amongst software engineers that infrastructure serves "performance". Most of the complexity comes from redundancy and analytics (, realtime especially).
- lomereiter 6y ago
  Right, but how much do these nice-to-haves add to the revenue? Is it truly worth it to bake them all in from the very beginning? E.g. five minutes of downtime in morning hours will affect only a handful of users, and if so, why bother with blue-green deployments other than out of professional interest?
smt88 6y ago
Dev time is more expensive than server time. If my dev uses a slow stack or doesn't optimize v1, maybe I spent $100/mo instead of $10/mo. Hell, let's say I'm spending $500/mo. My mixed rate is ~$75/hr for dev time, so a week of dev time is equivalent to 6 months of hosting. If optimizing (or using a difficult stack) takes one dev an extra week, then I'd better save $3,000 of hosting.
young_unixer 6y ago
What's your stack?
Andrew_nenakhov 6y ago
This. Before opening a link, I thought to myself, "$5/month ?", because that's what our Django websites with such load cost.
creshal 6y ago
4+k daily visitors on one of the biggest fan sites for a popular mobile game, and I'm still running it with sqlite as DB backend and no caching on a shared VPS.
bdcravens 6y ago
User count matters less than the amount of data and the amount of processing power required per user. Web servers are extremely efficient, so if most of that occurs within the context of requests, it's easy.
CryoLogic 6y ago
Depends on the type of web app. https://anim8.io https://anim8.io - a site for hobby animators could never run on a $10VPS. The bandwidth usage is just way too high and video compression requires GPU compute.
- dylz 6y ago
  The videos on that site aren't hosted inhouse or first-party. Definitely doable.

by TekMol 6y ago

tsumnia 6y ago

Not to sound mean, but if there aren't posts/blogs/whatever explicitly telling people how to minimize costs... then they'll continue to follow the ones that make them pay $100+.

rapsey 6y ago
Or people could do some quick research what options there are. Not everything needs to be spoon fed.
fock 6y ago
basically most people don't do that for saving costs, but just because it makes sense to them. And there's lots of tutorials explaining how to run your own webapp.
TekMol 6y ago
Ok, here is a quick tutorial: Step 1: Copy Peter Levels stack: https://www.nocsdegree.com/pieter-levels-learn-coding/ https://www.nocsdegree.com/pieter-levels-learn-coding/ If he serves hundreds of thousands of monthly users and makes a million a month with a single VPS, you certainly won't have scaling issues when you start out or just have tens of thousands of users. Step 2: If you really scale beyond that, resist the urge to bloat your stack. Think long and hard about every piece you add to it. Really understand each piece you add to it. Don't fall into the trap of paid services. Don't fall into the trap of "best practices".
- chillfox 6y ago
  I don’t know how many times I have seen “best practices” used as excuse to avoid thinking. Usually by people who don’t even have the problem that the supposed “best practices” is meant to solve.
- young_unixer 6y ago
  > PHP (He doesn't use any frameworks like Laravel) I wouldn't want to read (or write) that code, but thanks for the article.
  - paul_f 6y ago
    Well, aren't you special. How about staying on topic and not dropping in this, what, developer virtue signaling?
    - afarrell 6y ago
      > staying on topic A major subtopic of this thread is "different situations call for different setups." Saying "I won't work with PHP" is like saying "Gluten will make me shit my brains out." It is not virtue signalling.
- tsumnia 6y ago
  While this article has steps in the right direction, it's a lot of "use this thing". I operate a simple Flask/PostgreSQL webapp for CS Education through the luxury of my university. If/When I graduate from my program, what are the aspects I can do to minimize my costs for hosting the app? Of course I'll look for best approaches but why does that need to be forbidden knowledge known to a select few? Its an aside, but the mentality of "let them figure it out" is a major issue in education. Foundational knowledge should be easy to acquire so I can worry about higher level thinking issues. Literally spending hours trying to figure out how to set things up through hours of Googling doesn't really help that, nor does it promote the "figuring it out" people think it does - its just stumbling upon the right set of commands that let me move past this particular hurdle. From the devops perspective, what about telling someone how to set up their own server to do/minimize X is so taxing?
  - richiecute 6y ago
    I run a cheap website and here is what I do - cloudflare free tier for caching, DNS, page rules, etc - run everything on one VPS(digital ocean, linode, etc) pick cheapest that has specs you need - any non-trivial storage (media, big files) move to Backblaze B2 it's cheap (you can use free tier cloudflare workers to redirect to B2 for free bandwidth due to Bandwitdth Alliance) - free static page from Netifly (I can redirect to this with cloudflare in case my VPS falls over or something to provide info/links) - If I want to look at logs or something I rsync it my local machine (if I cared I could set up a process to push logs/backup etc to private B2 bucket) You may not need exact same setup, I am optimizing for caching and cheap storage because my site stores/serves lots of media files.
  - yabai_yatsu 6y ago
    >Literally spending hours trying to figure out how to set things up through hours of Googling doesn't really help that, nor does it promote the "figuring it out" people think it does - its just stumbling upon the right set of commands that let me move past this particular hurdle. That's basically all of software development for your entire career. Never not had a day or a week not like that.
    - 6y ago
- renewiltord 6y ago
  Haha, the man is clearly a beast of a product designer and executor. I don't think I would succeed with the same tools. Honestly, spending $200/mo is insignificant to me. And I'm pretty happy to answer the question of "Why can this guy build this thing on a single VPS and you can't?" with "Well, because he's better than me". I can be up in 15 mins on Heroku with a Rails+React web app. And in an hour have a thing. Or for a static no-login thing, faster with Netlify. But it doesn't matter. Because I never made a product as nice as the one he made. If the outcome is I'm -$200/mo that is irrelevant to me. If the outcome is I'm +$50k/mo that is very relevant. So I'm going to optimize for how I can do the latter.
- srtjstjsj 6y ago
  If I have a "million a month" business (which is coming from rich recruitijg fees on a trivial website, not amazing tech), no way am I sweating $171/mo in server bills.
aahortwwy 6y ago
Don't do blue-green deploys when you have no revenue. Downtime costs you nothing and at 3,400 visitors a day you'll be lucky to drop a single request while deploying. Pre-compute and cache to reduce your need for beefy servers. When you can run something on your machine or in the cloud, choose your machine. EDIT: It's less about saving money and more about not spending it. EDIT2: Forgot to mention: use SQLite.
- chillfox 6y ago
  SQLite Really is a bit of a hidden gem.
  - rapsey 6y ago
    Not really hidden when it is one of the most widely deployed pieces of software on the planet.
    - bigiain 6y ago
      Sure, but as an underlying dependency for iOS/Android, it's pretty well hidden from casual users...
      - krtkush 6y ago
        That's true for even a casual website user. They have no idea or care about the underlying database. Every app developer is aware about SQLite.
    - chillfox 6y ago
      I guess not in absolute terms, but it’s mostly overlooked for web apps.
- stevekemp 6y ago
  Caching is, in general, a good thing. But it's worth thinking about how you do it. I recently sped up a large system by dropping the use of redis entirely - because it was being badly used. During a single request there might be 50+ cache-lookups, each taking a round-trip to a remote redis server to fetch a single key at a time. Batching those up to a set/hash would have been more efficient, but the codebase had evolved in such a way as to make that difficult. Instead of making 50+ redis fetches it turned out that just fetching all the stuff from the database was faster. (There will be refactoring to batch up the key fetches, but for the moment there was a measurable increase in performance under current loads just by removing redis.)

threeseed 6y ago

It always amazes me how people are so free to judge other's situations. It's great that your web app only costs $10/month but others may have web apps that are more computationally intensive e.g. video processing or ML inference etc or simply can't join everything they need at runtime. And it's great that you're willing to deny those 50k users a day access to your service when that cheap VPS inevitably falls over. But others may be monetising that traffic and will want a HA solution so their revenue isn't impacted. All of those add complexity and cost to an architecture.

TekMol 6y ago
The article does not mention video processing or ML. It describes their setup for what they describe as a generic web app.
- FrozenSynapse 6y ago
  If you want to provide uninterrupted service to your clients you’ll have to spend some $. You want to have redundancy, machines hosted in different different locations, backup prod servers, monitoring, analysis tools. Even if it is for 1k monthly users, if you want reliability - it will increase the costs.
  - TekMol 6y ago
    I beg to differ. In my experience, the complicated setups that are justified by the argument of "reliability" have more downtime then a single VPS. The reason is probably that there are more moving parts and more has to be maintained / can go wrong. These days, a single VPS in the right datacenter has excellent uptime.
    - tristanperry 6y ago
      Agreed. I'll probably be downvoted, but these setups strike me as people who prefer to drink the cool aid than be pragmatic and only use what they need. I've also had very high reliability rates with a single VPS. They've actually given me less downtime than AWS services at times.

caspii 6y ago

I'v added this paragraph to the article: the servers are oversized for the load we're currently seeing. The reason for that is that we tried to solve a production issue by increasing the server specs. It didn't solve the problem, and now we can't down-size the servers without re-provisioning them ️.

mkesper 6y ago
But as they are redundant this should be a no-brainer?
pastage 6y ago
I have found the same with digital ocean it is hard to down size but that is only true because I have so much unicorn data, and a move has become very painfull, so I'm stuck paying for 2x the storage cost. You should really learn to add new servers by provisioning them and remove the old servers. Your app seems a perfect fit for it for moving between servers.
IanCal 6y ago
What's the problem with reprovisioning? Don't you have two that you're switching between?

bigiain 6y ago

I get what you're saying, but I think you're being a bit harsh. In my mind, choosing a dual redundant prod platform so you can do blue/green deployments is _totally_ unnecessary for a 55k MAU site generating no revenue. Same with cloud hosted metabase. You could run that on your own hardware - a spare laptop or probably even a raspberry pi for effectively nothing. On the other hand, a hobby project/side gig where you can demonstrate real world experience in those two things could _easily_ pay off in the first week of a new job it helps you land. If it's _just_ that extra $100/month they're spending there, it seems difficult to justify. If it's commercial experience doing that which is helping him try and land a job paying 20k or more extra per year? That's totally money well spent, in my opinion...

TekMol 6y ago
If we were on a site called "CV News" that teaches people how to get jobs in dull cooperations, I might agree with you. On the other hand, I would not frequent such a site. So we would never have had this discussion.
- bigiain 6y ago
  Yeah fair cop...
- iakov 6y ago
  You can apply the skills that you learn in any job, not just the dull ones. The skills that you learn by implementing this kind of setup are valuable to lots of interesting jobs I'd say.
maxk42 6y ago
Given that he's complaining about the price I don't think the wasteful costs here are justified. I'll also note the article was just edited and the costs are now up to $171 / month. This app could easily be run on the AWS free tier, although even in the paid tier he could probably be managing a lot more than this workload for under $40 / month. (That's for two servers - which as has been pointed-out is wholly unnecessary for an app of this size.) The price he's paying is presently listed at $95.

mjburgess 6y ago

The additional expense in OP had nothing to do with scale; it arises from: redundancy, analytics, off-the-shelf integrations. It is a common misunderstanding amongst software engineers that infrastructure serves "performance". Most of the complexity comes from redundancy and analytics (, realtime especially).

lomereiter 6y ago
Right, but how much do these nice-to-haves add to the revenue? Is it truly worth it to bake them all in from the very beginning? E.g. five minutes of downtime in morning hours will affect only a handful of users, and if so, why bother with blue-green deployments other than out of professional interest?

smt88 6y ago

Dev time is more expensive than server time. If my dev uses a slow stack or doesn't optimize v1, maybe I spent $100/mo instead of $10/mo. Hell, let's say I'm spending $500/mo. My mixed rate is ~$75/hr for dev time, so a week of dev time is equivalent to 6 months of hosting. If optimizing (or using a difficult stack) takes one dev an extra week, then I'd better save $3,000 of hosting.

young_unixer 6y ago

What's your stack?

Andrew_nenakhov 6y ago

This. Before opening a link, I thought to myself, "$5/month ?", because that's what our Django websites with such load cost.

creshal 6y ago

4+k daily visitors on one of the biggest fan sites for a popular mobile game, and I'm still running it with sqlite as DB backend and no caching on a shared VPS.

bdcravens 6y ago

User count matters less than the amount of data and the amount of processing power required per user. Web servers are extremely efficient, so if most of that occurs within the context of requests, it's easy.

CryoLogic 6y ago

Depends on the type of web app. https://anim8.io https://anim8.io - a site for hobby animators could never run on a $10VPS. The bandwidth usage is just way too high and video compression requires GPU compute.

dylz 6y ago
The videos on that site aren't hosted inhouse or first-party. Definitely doable.