4 ms·

So less so decisions I have made but I have made a career out of correcting such mistakes. I'm a database systems guy fundamentally so in the previous ~3-4 com

by jpgvm 4y ago

So less so decisions I have made but I have made a career out of correcting such mistakes. I'm a database systems guy fundamentally so in the previous ~3-4 companies I have worked out my main achievement has usually been coming in and ripping out a proprietary, either custom built or poorly selected hosted database and porting it to OSS. Generally to PostgreSQL but also more specialized stores like Apache Druid (replacing a custom TSDB). The issue is that folks seems to get enamored with shiny or think a certain feature of a proprietary database is too hard to replicate on PostgreSQL. Currently that is Google Firestore which has a real-time capability. Replicating such capability on PostgreSQL isn't that difficult if you are aware of logical replication and the tools necessary for scaling it if need be (namely a distributed log like Apache Kafka or Pulsar). The end result is a system that is heinously expensive for what it does, poor at the sort of queries it needs to do and thus the system is riddled with workarounds for it's shortcomings. In the past it also included RethinkDB, MongoDB, Cloud Bigtable, DynamoDB, etc. Some of which can be good datastores if they perfectly match what you are trying to do but most of which you should never touch with a 30ft pole cough MongoDB cough. Generally speaking if you should almost always use PostgreSQL unless you know you need something else. The other big one for me is porting from runtimes like Lambda and CloudFunctions to k8s (either on EKS or GKE). Both of those runtimes result in terrible architecture and aren't worth the ~2-3 days you save with initial setup of k8s. For the more general question of build vs buy I see it this way. If something is core to the flow of your product the selected solution should be maximized for control and ownership. i.e favour build over buy unless there is overwhelming positives to buy or the component is so comoditized it really doesn't matter. Everything purely line of business however should be Buy > Build. i.e Slack, GMail/GSuite, ERP/CRM, etc. Example for buy > build because of superiority is Cloudflare > self-baked CDN. You can't match CF network, it has to be bought. Example of buy because comoditized is low level infra components from AWS/GCP, e.g VMs, networking, LBs, DNS, k8s controllers, etc. Stuff you should never outsource are authz/authn, runtimes (i.e never use Lambda or shit like it), databases (except relatively portable like RDS/CloudSQL), etc. Murky stuff where it's not clear: CI/CD - lots of tradeoffs and spectrum here, hosted controller + self-hosted runners seems good sweet spot, email - generally need multiple upstreams configured with different sending domains to handle being occasionally randomly blacklisted, observability - it's a bitch to run yourself but hosted options are heinously expensive, generally come with restrictions that reduce fidelity directly or indirectly through cost minimization. Feature flagging - hosted services usually result in CORS requests which are slow and the SDKs often block application startup, better handled yourself but maybe getting started with something hosted is OK.

wizwit999 4y ago
I would suggest you to learn about serverless platforms like Lambda and Cloud Functions. Hint: it's not about the setup.
- jpgvm 4y ago
  I have moved multiple latter stage startups off of them thus I am intimately aware of all of their capabilities, limitations, etc. You should calculate the $/CPU/RAM of such platforms vs k8s on spot/pre-emptible instances and it clearly becomes obvious that you can't ever possibly be cost effective on any of these runtimes even with steep 50%+off-list discounts. So no I don't think I need to learn more about serverless, I think you perhaps need to learn more about k8s and modern container orchestration.
  - mikhmha 4y ago
    I work at a enterprise SAAS startup that uses lambda for backend. I agree with a lot of what you say. But I also disagree with you. Lambda ends up being very cost effective at the place I work at. Its a payment processor so lambdas are only ever run in response to approved transactions. For this sort of SAAS the economics of lambda work very well. I'm not sure what would be better. Of course if I were running a public facing free web service I would not use lambda, but for something where every user is a paying customer? I think Lambda is very cost effective. And if it ever got to the point where moving to k8's would result in a massive cost-savings, I would just pay a guy like you to handle the migration. At that point its probably not a small startup anymore.
    - jpgvm 4y ago
      Yeah that is fair. I think the contention I have with it is those runtimes generally don't save time or money but come with more constraints - constraints that usually lead to poor architecture decisions. Examples I have seen include: Massive/mono-lambda - this is caused by the overhead of managing many functions so instead everything gets packed into one with some sort of router logic. The usual cause of this one is managing many Lambda/FaaS things becomes too hard, especially if you are also using API Gateway and friends. Resource misuse - the nature of these FaaS runtimes is they are essentially glorified cgi-bin, in all the worst ways. Namely they tend to mask memory leaks and other poor programming habits until they start leaking through as spurious failures. Poor observability - because FaaS results in unpredictable lifetimes you end up making concessions in your observability stack. Can't do in-process rollup, can't do pull model metrics, i.e Prometheus, etc. You can somewhat get away with this with doing full tracing and sampling on the collector side instead but you have just made the whole thing inefficient for...what exactly? Unnecessary cron-like things - this usually comes from places that try to go all-in on FaaS and squeeze everything into FaaS shaped boxes. Something that would much better be a daemon that spreads it's work over time cleanly becomes a cron-like thing that fires every minute, usually spawning 1000x of sub-FaaS things to accomplish some maintenance task paying huge amounts of overhead. Again... for what? Poor DB connectivity options - again linked to unpredictable lifetimes and "unlimited scale" you don't have as many options for DB connectivity, you have to invest in bouncers immediately even if your application would actually be fine running on just 10 connections or so... if you had 10 containers instead of potentially 100s of lambdas. Poor resource sizing options - sometimes you just need a fk-ton of RAM to solve your problem. Most of the FaaS providers limit not just your max runtime but also max resources you can grant said function. Leading again to sub-optimal work splitting for memory hard problems. The list keeps going on tbh, this is far from exhaustive. At the end of the day the tradeoffs aren't worth it. The only thing of value you get in return is "unlimited scalability" which is easily approximated with k8s primitives. I really don't see why k8s shouldn't be the default option for anyone considering these FaaS things given GKE/AKS/EKS exist and do 99.99% of the work for you and are WAY less complicated than the FaaS runtimes and their associated requirements.

by jpgvm 4y ago

wizwit999 4y ago

I would suggest you to learn about serverless platforms like Lambda and Cloud Functions. Hint: it's not about the setup.

jpgvm 4y ago
I have moved multiple latter stage startups off of them thus I am intimately aware of all of their capabilities, limitations, etc. You should calculate the $/CPU/RAM of such platforms vs k8s on spot/pre-emptible instances and it clearly becomes obvious that you can't ever possibly be cost effective on any of these runtimes even with steep 50%+off-list discounts. So no I don't think I need to learn more about serverless, I think you perhaps need to learn more about k8s and modern container orchestration.
- mikhmha 4y ago
  I work at a enterprise SAAS startup that uses lambda for backend. I agree with a lot of what you say. But I also disagree with you. Lambda ends up being very cost effective at the place I work at. Its a payment processor so lambdas are only ever run in response to approved transactions. For this sort of SAAS the economics of lambda work very well. I'm not sure what would be better. Of course if I were running a public facing free web service I would not use lambda, but for something where every user is a paying customer? I think Lambda is very cost effective. And if it ever got to the point where moving to k8's would result in a massive cost-savings, I would just pay a guy like you to handle the migration. At that point its probably not a small startup anymore.
  - jpgvm 4y ago
    Yeah that is fair. I think the contention I have with it is those runtimes generally don't save time or money but come with more constraints - constraints that usually lead to poor architecture decisions. Examples I have seen include: Massive/mono-lambda - this is caused by the overhead of managing many functions so instead everything gets packed into one with some sort of router logic. The usual cause of this one is managing many Lambda/FaaS things becomes too hard, especially if you are also using API Gateway and friends. Resource misuse - the nature of these FaaS runtimes is they are essentially glorified cgi-bin, in all the worst ways. Namely they tend to mask memory leaks and other poor programming habits until they start leaking through as spurious failures. Poor observability - because FaaS results in unpredictable lifetimes you end up making concessions in your observability stack. Can't do in-process rollup, can't do pull model metrics, i.e Prometheus, etc. You can somewhat get away with this with doing full tracing and sampling on the collector side instead but you have just made the whole thing inefficient for...what exactly? Unnecessary cron-like things - this usually comes from places that try to go all-in on FaaS and squeeze everything into FaaS shaped boxes. Something that would much better be a daemon that spreads it's work over time cleanly becomes a cron-like thing that fires every minute, usually spawning 1000x of sub-FaaS things to accomplish some maintenance task paying huge amounts of overhead. Again... for what? Poor DB connectivity options - again linked to unpredictable lifetimes and "unlimited scale" you don't have as many options for DB connectivity, you have to invest in bouncers immediately even if your application would actually be fine running on just 10 connections or so... if you had 10 containers instead of potentially 100s of lambdas. Poor resource sizing options - sometimes you just need a fk-ton of RAM to solve your problem. Most of the FaaS providers limit not just your max runtime but also max resources you can grant said function. Leading again to sub-optimal work splitting for memory hard problems. The list keeps going on tbh, this is far from exhaustive. At the end of the day the tradeoffs aren't worth it. The only thing of value you get in return is "unlimited scalability" which is easily approximated with k8s primitives. I really don't see why k8s shouldn't be the default option for anyone considering these FaaS things given GKE/AKS/EKS exist and do 99.99% of the work for you and are WAY less complicated than the FaaS runtimes and their associated requirements.