4 ms·

Probably not for $200/mo EC2, but AWS/GCP in general * Centralized logging, log search, log based alerting * Secrets manager * Managed kubernetes * Object s

Probably not for $200/mo EC2, but AWS/GCP in general * Centralized logging, log search, log based alerting * Secrets manager * Managed kubernetes * Object store * Managed load balancers * Database HA * Cache solutions ... Can I run all these by myself? Sure. But I'm not in this business. I just want to write software and run that. And yes, I have needed most of this from day 1 for my startup. For a personal toy project, or when you reach a certain scale, it may makes sense to go the other way. U

ozozozd 1y ago
You need database HA and load balancers on day 1? You must be doing truly a lot of growth prior to building. Or perhaps insisting on tiny VMs for your loads?
- swiftcoder 1y ago
  > Or perhaps insisting on tiny VMs for your loads? This happens way too often. Early-stage startups that build everything on the AWS free tier (t2.micro only!), and then when the time comes they scale everything horizontally
  - throwaway7783 1y ago
    I'll repeat what I said above. It's for availability (aka I don't want my customers screaming at me if the machine goes down). And no , scaling out was not our first solution, scaling up was. I have considered going bare metal so many times, but the number of things we need to build/manage by ourselves to function is too much right now. Hopefully when we can afford to do it, we will.
- throwaway7783 1y ago
  HA is for availability. I don't want downtime for my enterprise customers. Are your customers okay with downtime? And as soon as you have more than one nodes, you need some kind of a load balancer in the front.
  - rcxdude 1y ago
    In practice, until you're at a certain scale, software bugs are more of a threat to your availability than hardware failures or maintenance downtime, and the cloud does nothing for you there (in fact, the additional complexity is likely to make it worse). Modern hardware is pretty reliable, more so than a given ec2 instance, for example.
    - throwaway7783 1y ago
      Often, software bugs cause issues with machines (uncontrolled logging overwhelming disk space and killing everything, instead of objectstore absorbing it, or a memory bug that kills the process over a few days and needing manual or custom monitoring scripts instead of k8s handling this until its root caused, and so on)
eska 1y ago
Now imagine your solution is not on a distributed system and go through that list. Centralized logging? There is nothing to centralized. Secrets management? There are no secrets to be constantly distributed to various machines on a network. Load balancing? In practice most people for most work don’t use it because of actually outgrowing hardware, but because they have to provision to shared hardware without exclusivity. Caching? Distributed systems create latency that doesn’t need to exist at all, reliability issues that have to be avoided, thundering herd issues that you would otherwise not have, etc. So while there are areas where you need to introduce distributed systems, this repeated disparaging comment of “toy hobby projects” makes me distrust your judgement heavily. I have replaced many such installations by actually delivering (grand distributed designs often don’t fully deliver), reducing costs, dramatically improving performance, and most importantly reducing complexity by magnitudes.
- b112 1y ago
  Not to mention scaling. Most clients I know never, ever have scaled once. Ever. Or if they do, it's to save money. One server means you can handle the equiv of 100+ AWS instances. And if you're into that turf, then having a rack of servers saves even more. Big corp is pulling back from the cloud for a reason.
  - throwaway7783 1y ago
    I mentioned this in an earlier comment. It is dumb to be on the cloud at a large enough scale.
- viraptor 1y ago
  > Centralized logging? There is nothing to centralized. It's still useful to have the various services, background jobs, system events, etc. in one indexed place which can also manage retention and alerting. And ideally in a place reachable even if the main service goes down. I've got centralised logging on a small homelab server with a few services on it and it's worth the effort. > Load balancing? In practice most people for most work don’t use it because of actually outgrowing hardware, but because they have to provision to shared hardware without exclusivity. Depending on how much you lose in case of downtime, you may want at least 2x of hardware for redundancy and that means some kind of fancy routing (whether it's LB, shared IP, or something else) > Secrets management? There are no secrets to be constantly distributed to various machines on a network. Typically businesses grow to more than one service. For example I've got a slack webhook in 3 services in a small company and I want to update it in one place. (+ many other credentials) > Caching? Distributed systems create latency that doesn’t need to exist at all This doesn't solve the need for caching results of larger operations. It doesn't matter how much latency you have or not, you still don't want that rarely-changing 1sec long query to run on every request. Caching is rarely only about network latency.
troupo 1y ago
> For a personal toy project, which in reality is any project under a few hundred thousand users
- throwaway7783 1y ago
  .. who are okay with services going down here and there
  - troupo 1y ago
    Which is the vast majority of services.
    - throwaway7783 1y ago
      Okay. If the premise is "you don't have to worry about downtimes and only need to serve a few hundred thousand users and no data intensive use cases", then I guess you can do whatever and it'll still be okay.
runako 1y ago
> Centralized logging, log search, log based alerting Do people really use the bare CloudWatch logs as an answer for log search? I find it terrible and pretty much always recommend something like DataDog or Splunk or New Relic.
- throwaway7783 1y ago
  we are on GCP. Logs Explorer on GCP is pretty good

ozozozd 1y ago

You need database HA and load balancers on day 1? You must be doing truly a lot of growth prior to building. Or perhaps insisting on tiny VMs for your loads?

swiftcoder 1y ago
> Or perhaps insisting on tiny VMs for your loads? This happens way too often. Early-stage startups that build everything on the AWS free tier (t2.micro only!), and then when the time comes they scale everything horizontally
- throwaway7783 1y ago
  I'll repeat what I said above. It's for availability (aka I don't want my customers screaming at me if the machine goes down). And no , scaling out was not our first solution, scaling up was. I have considered going bare metal so many times, but the number of things we need to build/manage by ourselves to function is too much right now. Hopefully when we can afford to do it, we will.
throwaway7783 1y ago
HA is for availability. I don't want downtime for my enterprise customers. Are your customers okay with downtime? And as soon as you have more than one nodes, you need some kind of a load balancer in the front.
- rcxdude 1y ago
  In practice, until you're at a certain scale, software bugs are more of a threat to your availability than hardware failures or maintenance downtime, and the cloud does nothing for you there (in fact, the additional complexity is likely to make it worse). Modern hardware is pretty reliable, more so than a given ec2 instance, for example.
  - throwaway7783 1y ago
    Often, software bugs cause issues with machines (uncontrolled logging overwhelming disk space and killing everything, instead of objectstore absorbing it, or a memory bug that kills the process over a few days and needing manual or custom monitoring scripts instead of k8s handling this until its root caused, and so on)

eska 1y ago

Now imagine your solution is not on a distributed system and go through that list. Centralized logging? There is nothing to centralized. Secrets management? There are no secrets to be constantly distributed to various machines on a network. Load balancing? In practice most people for most work don’t use it because of actually outgrowing hardware, but because they have to provision to shared hardware without exclusivity. Caching? Distributed systems create latency that doesn’t need to exist at all, reliability issues that have to be avoided, thundering herd issues that you would otherwise not have, etc. So while there are areas where you need to introduce distributed systems, this repeated disparaging comment of “toy hobby projects” makes me distrust your judgement heavily. I have replaced many such installations by actually delivering (grand distributed designs often don’t fully deliver), reducing costs, dramatically improving performance, and most importantly reducing complexity by magnitudes.

b112 1y ago
Not to mention scaling. Most clients I know never, ever have scaled once. Ever. Or if they do, it's to save money. One server means you can handle the equiv of 100+ AWS instances. And if you're into that turf, then having a rack of servers saves even more. Big corp is pulling back from the cloud for a reason.
- throwaway7783 1y ago
  I mentioned this in an earlier comment. It is dumb to be on the cloud at a large enough scale.
viraptor 1y ago
> Centralized logging? There is nothing to centralized. It's still useful to have the various services, background jobs, system events, etc. in one indexed place which can also manage retention and alerting. And ideally in a place reachable even if the main service goes down. I've got centralised logging on a small homelab server with a few services on it and it's worth the effort. > Load balancing? In practice most people for most work don’t use it because of actually outgrowing hardware, but because they have to provision to shared hardware without exclusivity. Depending on how much you lose in case of downtime, you may want at least 2x of hardware for redundancy and that means some kind of fancy routing (whether it's LB, shared IP, or something else) > Secrets management? There are no secrets to be constantly distributed to various machines on a network. Typically businesses grow to more than one service. For example I've got a slack webhook in 3 services in a small company and I want to update it in one place. (+ many other credentials) > Caching? Distributed systems create latency that doesn’t need to exist at all This doesn't solve the need for caching results of larger operations. It doesn't matter how much latency you have or not, you still don't want that rarely-changing 1sec long query to run on every request. Caching is rarely only about network latency.

troupo 1y ago

> For a personal toy project, which in reality is any project under a few hundred thousand users

throwaway7783 1y ago
.. who are okay with services going down here and there
- troupo 1y ago
  Which is the vast majority of services.
  - throwaway7783 1y ago
    Okay. If the premise is "you don't have to worry about downtimes and only need to serve a few hundred thousand users and no data intensive use cases", then I guess you can do whatever and it'll still be okay.

runako 1y ago

> Centralized logging, log search, log based alerting Do people really use the bare CloudWatch logs as an answer for log search? I find it terrible and pretty much always recommend something like DataDog or Splunk or New Relic.

throwaway7783 1y ago
we are on GCP. Logs Explorer on GCP is pretty good