6 ms·

> I'm keeping things simple, so I've got mostly Go services, pg + caching, and a svelte webapp. I deployed my Go services on a low-ish end bare metal provider,

by movedx 3y ago

> I'm keeping things simple, so I've got mostly Go services, pg + caching, and a svelte webapp. I deployed my Go services on a low-ish end bare metal provider, and for now it is fine. Deployments are triggered via scripts, and so far so good. Is it sexy, using all the latest and greatest tech? No, its just simple shell scripts. But it works? Don't change a thing. This is perfect. > Am I wrong to think that I could probably scale like crazy and avoid AWS completely with my stack? Why should I pay hundreds/thousands per month plus a premium for bandwidth? I'm also enjoying staying sane avoiding IAM. You're not wrong at all. Check out the hardware stack for Stack Overflow as of 2016: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... Don't over think it. Focus on the software and its features. Focus on getting users and ramping up your MRR. Good luck.

dangus 3y ago
One thing I would point out is that the simplicity of StackOverflow's hardware stack means that it would also be very simple to host in a cloud provider like AWS. I would also point out that StackOverflow isn't an incredibly good performer compared to a number of other websites. Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others.
- tommek4077 3y ago
  I would not give much on those gambled page speed analysis. It comes down to: is ist fast enough? The answer clearly was yes during all those years for the users.
  - movedx 3y ago
    This is exactly right.
- sebazzz 3y ago
  > One thing I would point out is that the simplicity of StackOverflow's hardware stack means that it would also be very simple to host in a cloud provider like AWS. Yes, of course. Every cloud provider also does PaaS, but often it is also not the most cost effective option - often it is the most expensive one! The cheapest option is often the option where toy get reliant on cloud-specific sdks, serverless, etc.
- Scarbutt 3y ago
  This. AWS can be as simple or complex as you want it to be.
  - tommek4077 3y ago
    Just not cheap.
    - slau 3y ago
      Yeah, I have to strongly disagree with that. Sure, AWS can be stupidly expensive, however it can also save you a lot of money. A few years ago, I was the VPoE for a small-ish startup in Denmark. I was the only ops person, and I was able to provide 100% uptime for a year, including a migration from k8s clusters, AWS AZ outages and whatnot. During that time, I also reduced our monthly spend from $15k to $5k. This was a fully auto-scaling, redundant and highly available system. When I started, every night I got woken up by alerts (working alongside the CEO and another engineer to fix things). By the time I had stabilised things, the only thing waking us up were our providers having outages. I used to run a similar business earlier, without AWS. We owned our own hardware and had ops people going to data centres. I can guarantee you we did not spend less than $12-13k/month (AWS fee + my salary) in that company. Think closer to $100-150k/month. AWS can be cheaper when you factor in the cost of employees. It can also be stupidly expensive when you use it the wrong way. Another example: I host a small service that gets very seldom, seldom use. Maybe 20-50 people discover it and use it per month. I have the backend running on a Lambda, and it costs me about $0.5 per month. It took me 20 minutes to write the CloudFormation for that and push it through my CI pipeline to have it deployed. There is no way I could get cheaper hosting, uptime/availability, and faster time to market than with a Lambda. If and when that service becomes more used where it warrants running full time, I’ll rewrite the request handling and throw it in my k8s cluster. But until then, I do believe this is the cheapest solution (for me).
      - hdjjhhvvhga 3y ago
        Yeah that Lambda setup you mention is terribly cheap, and it is one of the very few extremely attractive services pricewise - but many people get lured into using other AWS services that aren't cheap, even though Amazon makes people think so by blurring the total price by splitting it into hours, calculating egress separately, dependencies on services that are billable by the hour (it's not obvious when you start that a private network has the cost of an NAT GW attached to it). Using only Lambda is really cost effective but frankly, few people resist the temptation and use it the way you described.
- pleoxy 3y ago
  Simple, sure, but AWS still charges insane rates. I can get 60k IOPS on a nanode at $5/mo rate. You actually get a random amount between 20k-60k depending on the type of machine they provision your nanode on. You want 60k IOPS on AWS? Be prepared to pay like 4-5k/mo. Want multiple systems with it so you can cluster a reasonably performance database? Pay for each and you need to special request the ability to go past 100k IOPS in a region. Want the privilege of a dedicated vCPU? It's $1500/mo per region just to turn on the feature. I tried to use AWS just as you mention here and the cost was 100-1000x what it would cost me at akamai cloud or Hetzner. All these fees I was unfamiliar with popped out and the cost was crazy high. The system I tried to provision was functionally similar to the bottom end dedicated akamai cloud offering and it was going to cost 15-20k/mo instead of $72. Over 200x. And that was just the hourly provisioned rate, not including egress and other bits...
  - dangus 3y ago
    These high costs are Amazon's way of telling you that you aren't architecting for the cloud. In my ~10 years doing cloud infrastrcuture on AWS I've never worked at a company had to purchase dedicated IOPS or vCPU.
    - pleoxy 3y ago
      Which takes then off the list of providers you can do this with. They could have reasonable prices but they don't. You couple to AWS with your whole system architecture or you stay away.
      - dangus 3y ago
        If you tell me I need to go 250 miles per hour to get to the grocery store then I guess I have to buy a Bugatti.
        pleoxy 3y ago
        Can't carry much in a Bugatti. It's a real problem if you build on a VPS as your base. I was asked to prep such a system for scale on AWS and choked on the pricing. So we settled for what they offer at reasonable pricing and if further growth is needed, will need to hit eject on AWS and just go elsewhere. I recommend starting elsewhere and save the migration of you just use EC2. Most places will let you scale a VPS quite well and there is a logical handoff to your own hardware where you can scale to a system with 512 vCPU and TBs of ram for less than Amazon is charging for step two.
- BoorishBears 3y ago
  > Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others. And yet Stack Overflow has such strong SEO that outright cloning their content on a malware page will get you top 10 results. I think people seriously overestimate the value of Web Core Vitals. At the end of the day a website is usually trying to deliver some value to a visitor: if you deliver more value by rejecting complexity, but sacrifice some speed to do it: users will still value the end result, and search engines will reward you.
  - deleted 3y ago
    [deleted]
- Turskarama 3y ago
  Is StackOverflow slow? Maybe. Has this ever bothered me as a user? Nope.
barelysapient 3y ago
Exactly this. The only thing unmentioned, but that you should be aware of, is the effort to deploy this stack from scratch. You can assume the DB survives, or doesn't, but the ability to go from zero to a running deployment from a recent restore is probably the only thing you need to really worry about. If you can get that to be less than hour, or ideally a few minutes, then you're golden. Don't be afraid to add CDN/Caching when its appropriate. You can always buy a beefier instance, or a dedicated box. Deal with the scale later when its truely needed. Give yourself permission, or plan to hire a team, to handle the scale issues later.
- happytiger 3y ago
  This is the most important advice for an early stage startup trying to scale. Being able to automatically build production from a recent snapshot is a profoundly important point. When we bootstrapped on the cheap we would always maintain a production infrastructure at one company and a test infrastructure at another with a relatively low TTL dns infrastructure hosted at a third party like dnsmadeeasy. Backups would be pushed to the second site. We then would “build our demo” from the backup data (stripping PII at runtime). But the demo served as a second clone production infrastructure. If the primary went down, it was always a quick restore and scale up at the secondary location with recent backups! Worked really well, but you needed to make sure it was automated or it would blow up. Definitely cheap and works like a charm. I’m sure that’s over complicated for a lot of people, but it’s really just a few scripts and a couple of days to set up, and you can be up and rolling with a working system in the time it takes to change DNS.

by movedx 3y ago

dangus 3y ago

One thing I would point out is that the simplicity of StackOverflow's hardware stack means that it would also be very simple to host in a cloud provider like AWS. I would also point out that StackOverflow isn't an incredibly good performer compared to a number of other websites. Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others.

tommek4077 3y ago
I would not give much on those gambled page speed analysis. It comes down to: is ist fast enough? The answer clearly was yes during all those years for the users.
- movedx 3y ago
  This is exactly right.
sebazzz 3y ago
> One thing I would point out is that the simplicity of StackOverflow's hardware stack means that it would also be very simple to host in a cloud provider like AWS. Yes, of course. Every cloud provider also does PaaS, but often it is also not the most cost effective option - often it is the most expensive one! The cheapest option is often the option where toy get reliant on cloud-specific sdks, serverless, etc.
Scarbutt 3y ago
This. AWS can be as simple or complex as you want it to be.
- tommek4077 3y ago
  Just not cheap.
  - slau 3y ago
    Yeah, I have to strongly disagree with that. Sure, AWS can be stupidly expensive, however it can also save you a lot of money. A few years ago, I was the VPoE for a small-ish startup in Denmark. I was the only ops person, and I was able to provide 100% uptime for a year, including a migration from k8s clusters, AWS AZ outages and whatnot. During that time, I also reduced our monthly spend from $15k to $5k. This was a fully auto-scaling, redundant and highly available system. When I started, every night I got woken up by alerts (working alongside the CEO and another engineer to fix things). By the time I had stabilised things, the only thing waking us up were our providers having outages. I used to run a similar business earlier, without AWS. We owned our own hardware and had ops people going to data centres. I can guarantee you we did not spend less than $12-13k/month (AWS fee + my salary) in that company. Think closer to $100-150k/month. AWS can be cheaper when you factor in the cost of employees. It can also be stupidly expensive when you use it the wrong way. Another example: I host a small service that gets very seldom, seldom use. Maybe 20-50 people discover it and use it per month. I have the backend running on a Lambda, and it costs me about $0.5 per month. It took me 20 minutes to write the CloudFormation for that and push it through my CI pipeline to have it deployed. There is no way I could get cheaper hosting, uptime/availability, and faster time to market than with a Lambda. If and when that service becomes more used where it warrants running full time, I’ll rewrite the request handling and throw it in my k8s cluster. But until then, I do believe this is the cheapest solution (for me).
    - hdjjhhvvhga 3y ago
      Yeah that Lambda setup you mention is terribly cheap, and it is one of the very few extremely attractive services pricewise - but many people get lured into using other AWS services that aren't cheap, even though Amazon makes people think so by blurring the total price by splitting it into hours, calculating egress separately, dependencies on services that are billable by the hour (it's not obvious when you start that a private network has the cost of an NAT GW attached to it). Using only Lambda is really cost effective but frankly, few people resist the temptation and use it the way you described.
pleoxy 3y ago
Simple, sure, but AWS still charges insane rates. I can get 60k IOPS on a nanode at $5/mo rate. You actually get a random amount between 20k-60k depending on the type of machine they provision your nanode on. You want 60k IOPS on AWS? Be prepared to pay like 4-5k/mo. Want multiple systems with it so you can cluster a reasonably performance database? Pay for each and you need to special request the ability to go past 100k IOPS in a region. Want the privilege of a dedicated vCPU? It's $1500/mo per region just to turn on the feature. I tried to use AWS just as you mention here and the cost was 100-1000x what it would cost me at akamai cloud or Hetzner. All these fees I was unfamiliar with popped out and the cost was crazy high. The system I tried to provision was functionally similar to the bottom end dedicated akamai cloud offering and it was going to cost 15-20k/mo instead of $72. Over 200x. And that was just the hourly provisioned rate, not including egress and other bits...
- dangus 3y ago
  These high costs are Amazon's way of telling you that you aren't architecting for the cloud. In my ~10 years doing cloud infrastrcuture on AWS I've never worked at a company had to purchase dedicated IOPS or vCPU.
  - pleoxy 3y ago
    Which takes then off the list of providers you can do this with. They could have reasonable prices but they don't. You couple to AWS with your whole system architecture or you stay away.
    - dangus 3y ago
      If you tell me I need to go 250 miles per hour to get to the grocery store then I guess I have to buy a Bugatti.
      - pleoxy 3y ago
        Can't carry much in a Bugatti. It's a real problem if you build on a VPS as your base. I was asked to prep such a system for scale on AWS and choked on the pricing. So we settled for what they offer at reasonable pricing and if further growth is needed, will need to hit eject on AWS and just go elsewhere. I recommend starting elsewhere and save the migration of you just use EC2. Most places will let you scale a VPS quite well and there is a logical handoff to your own hardware where you can scale to a system with 512 vCPU and TBs of ram for less than Amazon is charging for step two.
BoorishBears 3y ago
> Check it out on PageSpeed Insights. I also used a couple other tools that suggest that it loads slower in some regions than others. And yet Stack Overflow has such strong SEO that outright cloning their content on a malware page will get you top 10 results. I think people seriously overestimate the value of Web Core Vitals. At the end of the day a website is usually trying to deliver some value to a visitor: if you deliver more value by rejecting complexity, but sacrifice some speed to do it: users will still value the end result, and search engines will reward you.
- deleted 3y ago
  [deleted]
Turskarama 3y ago
Is StackOverflow slow? Maybe. Has this ever bothered me as a user? Nope.

barelysapient 3y ago

Exactly this. The only thing unmentioned, but that you should be aware of, is the effort to deploy this stack from scratch. You can assume the DB survives, or doesn't, but the ability to go from zero to a running deployment from a recent restore is probably the only thing you need to really worry about. If you can get that to be less than hour, or ideally a few minutes, then you're golden. Don't be afraid to add CDN/Caching when its appropriate. You can always buy a beefier instance, or a dedicated box. Deal with the scale later when its truely needed. Give yourself permission, or plan to hire a team, to handle the scale issues later.

happytiger 3y ago
This is the most important advice for an early stage startup trying to scale. Being able to automatically build production from a recent snapshot is a profoundly important point. When we bootstrapped on the cheap we would always maintain a production infrastructure at one company and a test infrastructure at another with a relatively low TTL dns infrastructure hosted at a third party like dnsmadeeasy. Backups would be pushed to the second site. We then would “build our demo” from the backup data (stripping PII at runtime). But the demo served as a second clone production infrastructure. If the primary went down, it was always a quick restore and scale up at the secondary location with recent backups! Worked really well, but you needed to make sure it was automated or it would blow up. Definitely cheap and works like a charm. I’m sure that’s over complicated for a lot of people, but it’s really just a few scripts and a couple of days to set up, and you can be up and rolling with a working system in the time it takes to change DNS.