59 ms·

Almost every infrastructure decision I endorse or regret

• shrubble 3y ago
  "Since the database is used by everyone, it becomes cared for by no one. Startups don’t have the luxury of a DBA, and everything owned by no one is owned by infrastructure eventually" I think adding a DBA or hiring one to help you layout your database should not be considered a 'luxury'...
  • winrid 3y ago
    Yeah I mean, hiring one person to own that for 5-10 teams is pretty cheap... Cheaper than each team constantly solving the same problems and relearning the same gotchas/operational stuff that doesn't add much value when writing your application code.
  • steveBK123 3y ago
    There's even consultants you can hire out by the day instead of a full-time DBA. Maybe you need help with setup for a few weeks/months, and then some routine billable hours per month for maintenance / change advice.
• Scubabear68 3y ago
  The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck. Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.
  • sgarland 3y ago
    The downside is then you have many, many DBs to fight with, to monitor, to tune, etc. This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.
    • Scubabear68 3y ago
      Not a downside to me. Each team maintains their own DB and pays for their own choices. In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone. Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.
      • sgarland 3y ago
        I’m a DBRE, which means it’s somehow always my fault until proven otherwise. And even then, it’s usually on me to work around the insane schema dreamt up by the devs. Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit is trying to make 10,000 QPS? Nope, sorry, try again later. EDIT: I say “not my problem,” but as mentioned, it inevitably becomes my problem. Because “just unblock them so the site is functional” is far more attractive to the C-Suite than “slow down velocity to ensure the dev teams are doing things right.”
        • CoolCold 3y ago
          You forgot the modern mantra - dev team is always right!
        • Scubabear68 3y ago
          I agree. My gripe was everybody in the same schema with a global “app” user.
        • dalyons 3y ago
          Or, you just avoid doing multi tenet from the start and none of those become your problem to unblock. What’s the downside?
          • sgarland 3y ago
            Done that as well; it still becomes my problem because teams without RDBMS knowledge eventually break it, and… then I get paged. Full Stack is a lie, and the sooner companies accept that and allow people to specialize again, and to pay for the extra headcount, the better off everyone will be.
  • el_benhameen 3y ago
    Lots of interesting comments on this one. Anyone have any good resources for learning how not to fuck up schema/db design for those of us who will probably never have a DBA on the team?
    • magicalhippo 3y ago
      Good question. We don't have a DBA either. I've learned SQL as needed and while I'm not terrible, it's still daunting when making the schema for a new module that might require 10-20 tables or more. One thing that has worked well for us is to alway include the top-most parent key in all child tables down yhe hierarchy. This way we can load all the data for say an order without joins/exists. Oh and never use natural keys. Each time I thought finally I had a good use-case, it has bitten me in some way. Apart from that we just try to think about the required data access and the queries needed. Main thing is that all queries should go against indexes in our case, so we make sure the schema supports that easily. Requires some educated guesses at times but mostly it's predictable IME. Anyway would love to see a proper resource. We've made some mistakes but I'm sure there's more to learn.
      • AznHisoka 3y ago
        Not to pick on you, but is SQL not basic knowledge for every software engineer these days? Or have times changed?
        • rswail 3y ago
          Times have changed. If you have C# programmers and they can't do it in Entity Framework/LINQ, then they can't do it.
          • neonsunset 3y ago
            This seems like a stereotype from 2010s and disconnected from reality today.
            • 3y ago
  • nitwit005 3y ago
    The moment you have two databases is the moment you need to deal with data consistency problems. If you can't do something like determine if you can delete data, as the article mentions, you won't be able to produce an answer to how to deal with those problems.
  • eduction 3y ago
    Management problem masquerading as a tech problem. Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application. The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it needs to be managed. It could be a dictator DBA or a set of rules designed in meetings and administered by ops, or whatever. But imagine it was money. Different divisions produce and consume money just like data. Would anyone imagine suggesting either every team has their own bank account or total unfettered access to the corporate treasury? Of course not. You would make a system. Everyone would at least mildly hate it. That’s how databases should generally be managed once the company is any real size.
    • dalyons 3y ago
      Why would you make it a shared resource if you don’t have to? Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource . Why on earth would we choose to pay those terrible prices in this day and age, when infrastructure is code, managed databases are everywhere and every team can have their own thing. You didn’t have a choice previously, now you do.
      • eduction 3y ago
        You wouldn’t but in any decent sized organization you will have to. If it is an organization that needs to exist there will be some common set of critical data.
        • webo 3y ago
          In my experience, isolated (repeated) data storage paradigm is even more common at large organizations. They share data via services, ETLs, event buses, etc.
        • 3y ago
• hayst4ck 3y ago
  I would love to see this type of thing from multiple sources. This reflects a lot of my own experience. I think the format of this is great. I suppose it would take a motivated individual to go around and ask people to essentially fill out a form like this to get that.
  • kaycebasques 3y ago
    I also think it's a great format. One suggestion if we're gonna standardize around this format. Avoid the double negatives. In some cases author says "avoided XYZ" and then the judgment was "no regrets". Too many layers for me to parse there. Instead, I suggest each section being the product that was used. If you regret that product, in the details is where you mention the product you should have used. Or you have another section for product ABC and you provide the context by saying "we adopted ABC after we abandoned XYZ". I don't recommend trying to categorize into general areas like logging, postmortems, etc. Just do a top-level section for each product.
• electroly 3y ago
  > The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a full-time in-house DBA. https://calculator.aws/#/estimate?id=48b0bab00fe90c5e6de68d042ff723d7ca16f2af https://calculator.aws/#/estimate?id=48b0bab00fe90c5e6de68d0... Total 12 months cost: 547,441.85 USD Once you get past the point where the markup can pay for one or more full-time employees, I think you should consider doing that instead of blindly paying more and more to scale RDS up. You're REALLY paying for it with RDS. At least re-evaluate the choices you made as a fledgling startup once you reach the scale where you're paying AWS "full time engineer" amounts of money.
  • vasco 3y ago
    That's a huge instance with an enterprise license on top. Most large SaaS companies can run off of $5k / m or cheaper RDS deployments which isn't enough to pay someone. The amount of people running half a million a year RDS bills might not be that large. For most people RDS is worth it as soon as you have backup requirements and would have to implement them yourself.
    • electroly 3y ago
      Definitely--I recommend this after you've reached the point where you're writing huge checks to AWS. Maybe this is just assumed but I've never seen anyone else add that nuance to the "just use RDS" advice. It's always just "RDS is worth it" full stop, as in this article.
      • Aeolun 3y ago
        To some extend that is probably true, because when you’ve built a business that needs a 500k/year database fully on RDS it’s already priced into your profits, and switching to a self-hosted database will seem unacceptably risky for something that works just fine.
• CSMastermind 3y ago
  So by and large I agree with the things in this article. It's interesting that the points I disagree with the author on are all SaaS products: > Moving off JIRA onto linear I don't get the hype. Linear is fine and all but I constantly find things I either can't or don't know how to do. How do I make different ticket types with different sets of fields? No clue. > Not using Terraform Cloud No Regrets I generally recommend Terraform Cloud - it's easy for you to grow your own in house system that works fine for a few years and gradually ends up costing you in the long run if you don't. > GitHub actions for CI/CD Endorse-ish Use Gitlab > Datadog Regret Strong disagree - it's easily the best monitoring/observability tool on the market by a wide margin. Cost is the most common complaint and it's almost always from people who don't have it configured correctly (which to be fair Datadog makes it far too easy to misconfigure things and blow up costs). > Pagerduty Endorse Pagerduty charges like 10x what Opsgenie does and offers no better functionality. When I had a contract renewal with Pagerduty I asked the sales rep what features they had that Opsgenie didn't. He told me they're positioning themselves as the high end brand in the market. Cool so I'm okay going generic brand for my incident reporting. Every CFO should use this as a litmus test to understand if their CTO is financially prudent IMO.
  • crabmusket 3y ago
    We moved from Trello to Linear and it's been fantastic. I hope to never work at an organisation large enough for JIRA to be a good idea.
    • CSMastermind 3y ago
      To be fair Linear does strike me as everything everyone always hoped Trello would be. So if that's the upgrade path you're going down I'd expect it to be fantastic.
    • cqqxo4zV46cp 3y ago
      Newer (aka next gen aka Team-managed) Jira projects are pretty solid.
      • 3y ago
• 005 3y ago
  Interesting read, I agree with adopting an identity platform but this can definitely be contentious if you want to own your data. I wonder how much one should pay attention to future problems at the start of a startup versus "move fast and break things." Some of this stuff might just put you off finishing.
• sakopov 3y ago
  Who's using Pulumi here and how mature is it in comparison to terraform?
  • jryan49 3y ago
    I think currently under the hood it's actually still terraform. I know they are working on their own native providers.
  • dmattia 3y ago
    I'm using Pulumi in production pretty heavily for a bunch of different app types (ECS, EKS, CloudFront, CloudFlare, Vault, Datadog monitors, Lambdas of all types, EC2s with ASGs, etc.), it's reasonably mature enough. As mentioned in the other comment, the most commonly used providers for terraform are "bridged" to pulumi, so the maturity is nearly identical to Terraform. I don't really use Pulumi's pre-built modules (crossroads), but I don't find I've ever missed them. I really like both Pulumi and Terraform (which I also used in production for hundreds of modules for a few years), which it seems like isn't always a popular opinion on HN, but I have and you absolutely can run either tool in production just fine. My slight preference is for Pulumi because I get slightly more willing assistance from devs on our team to reach in and change something in infra-land if they need to while working on app code. We do still use some Pulumi and some Terraform, and they play really nicely together: https://transcend.io/blog/use-terraform-pulumi-together-migrate-infrastructure-tooling https://transcend.io/blog/use-terraform-pulumi-together-migr...
  • rswail 3y ago
    IaaC is one of the worst acronyms ever. Infrastructure should be declared, not coded. Say what you want. The tool then builds that, or changes whats there to match. I've tried Pulumi and understanding the bit that runs before it tries to do stuff and the bit that runs after it tries to do stuff and working out where the bugs are is a PITA. It lulls you into a false sense of security that you can refer to your own variables in code, but that doesn't get carried over to when it is actually running the plan on the cloud service (ie actually creating the infrastructure) because you can only refer to the outputs of other infrastructure. CFN is too far in the other direction, primarily because it's completely invisible and hard to debug. Terraform has enough programmability (eg for_each, for-expressions etc) that you can write "here is what I want and how the things link together" and terraform will work out how to do it. The language is... sometimes painful, but it works. The provider support is unmatched and the modules are of reasonable quality.
• breckenedge 3y ago
  > There are no great FaaS options for running GPU workloads, which is why we could never go fully FaaS. I keep wondering when this is going to show up. We have a lot of service providers, but even more frameworks, and every vendor seems to have their own bespoke API.
  • z3ugma 3y ago
    Check out beam.cloud. They’re impressing me with calling GPU runtimes as a FaaS
  • gfodor 3y ago
    I just started playing with modal.com and so far it seems good. I haven't run anything in production yet, so YMMV.
  • gen220 3y ago
    I don’t think anybody should go “fully FaaS”, it’s like saying screwdrivers are useless, all you need is a hammer. That being said, Cloudflare is on the path to offering a great GPU FaaS system for inference. I believe it’s still in beta, but it’s the most promising option at the moment.
    • breckenedge 3y ago
      Right, I still find it faster to manually provision a specific instance type, install PyTorch on it, and deploy a little flask app for an inference server.
• unknownusererro 3y ago
  [dead]
• hermanradtke 3y ago
  Without some sort of background on cost or scale it is hard to judge any of these decisions.
  • screenoridesaga 3y ago
    [dead]
• cratermoon 3y ago
  Even if others disagree with your endorsements or regrets, this record shows you're actually aware of the important decisions you made over the past four years and tracked outcomes. Did you record the decisions when you made them and revisit later?
• guhcampos 3y ago
  Well it's a bit unfortunate this post was published in Feb 1st, it got really outdated really fast around the "choose flux for gitops" part.
  • CoolCold 3y ago
    Mind sharing bit more of the details?
    • medina 3y ago
      > engineers at Weaveworks built the first version of Flux > Weaveworks donated Flux and Flagger to the CNCF https://fluxcd.io/blog/2022/11/flux-is-a-cncf-graduated-project/ https://fluxcd.io/blog/2022/11/flux-is-a-cncf-graduated-proj... > Weaveworks will be closing its doors and shutting down commercial operations > Alexis Richardson, 5 Feb 2024 https://www.linkedin.com/posts/richardsonalexis_hi-everyone-i-am-very-sad-to-announce-activity-7160295096825860096-ZS67/ https://www.linkedin.com/posts/richardsonalexis_hi-everyone-... If the project has legs, it's now under CNCF.
  • plagiarist 3y ago
    What's the news there? I was just about to try it out this weekend.
  • alexjurkiewicz 3y ago
    Context https://www.silverliningsinfo.com/automation/weaveworks-unravels-heres-what-means-flux https://www.silverliningsinfo.com/automation/weaveworks-unra...
  • zeeZ 3y ago
    So far it seems fine, and the maintainers seem to be doing OK too. Is the project future at risk? https://github.com/fluxcd/flux2/discussions/4544 https://github.com/fluxcd/flux2/discussions/4544
• deleted 3y ago
  [deleted]
• sroussey 3y ago
  If you are startup that can can’t afford a DBA, then why why why are you using Kubernetes?
  • jrockway 3y ago
    Why wouldn't you use Kubernetes? There are basically 3 classes of deployments: 1) We don't have any software, so we don't have a prod environment. 2) We have 1 team that makes 1 thing, so we just launch it out of systemd. 3) We have between 2 and 1000 teams that make things and want to self-manage when stuff gets rolled out. Kubernetes is case 3. Like it or not, teams that don't coordinate with each other is how startups scale, just like big companies. You will never find a director of engineering that says "nah, let's just have one giant team and one giant codebase".
    • otterley 3y ago
      On AWS, at least, there are alternatives such as ECS and even plain old EC2 auto scaling groups. Teams can have the autonomy to run their infrastructure however they like (subject to whatever corporate policy and compliance regime requirements they might have to adhere to). Kubernetes is appealing to many, but it is not 100% frictionless. There are upgrades to manage, control plane limits, leaky abstractions, different APIs from your cloud provider, different RBAC, and other things you might prefer to avoid. It's its own little world on top of whatever world you happen to be running your foundational infrastructure on. Or, as someone has artistically expressed it: https://blog.palark.com/wp-content/uploads/2022/05/kubernetes-iceberg.png https://blog.palark.com/wp-content/uploads/2022/05/kubernete...
      • ezrast 3y ago
        The alternatives aren't frictionless either; many items from that image are not specific to Kubernetes. I personally find AWS API's frustrating to use, so even if I were running a one-person shop (and was bound to AWS for some reason - maybe a warlock has cursed me?) I'd lean towards managing things from EKS to get an interface that fits my brain better. It's just preference, though - EC2 auto-scaling is perfectly viable if that's your jam.
• Castellanavito 3y ago
  [flagged]
• endisneigh 3y ago
  Great post. I do wonder - what are the simplest K8s alternatives? Many say in the database world, "use Postgres", or "use sqlite." Similarly there are those databases that are robust that no one has heard of, but are very limited like FoundationDB. Or things that are specialized and generally respected like Clickhouse. What are the equivalents of above for Kubernetes?
  • tomas789 3y ago
    You can always use old boring AWS EC2 and such. And sprinkle in some Terraform if you feel fancy. That would be my “use sqlite” Kubernetes is probably “use postgres”
  • marcosdumay 3y ago
    Kubernetes aren't like that. It's just that, you should start with a handful of backed-up pet servers. Then manually automate their deployment when you need it. And only then go for a tool that abstracts the automated deployment when you need it. But I fear the simplest option on the Kubernetes area is Kubernetes.
    • doctor_eval 3y ago
      I don’t know that this is good advice. I shunned k8s for a long time because of the complexity, but the managed options are so much easier to use and deploy than pet servers that I can’t justify it any more. For anything other than truly trivial cases, IMO kubernetes or (or similar, like nomad) is easier than any alternative. The stack I use is hosted Postgres and VKS from Vultr. It’s been rock solid for me, and the entire infrastructure can be stored in code.
    • lucw 3y ago
      This is good advice, if you haven't experienced the pain of doing it yourself, you won't know what the framework does for you. There are limits to this reasoning of course, we don't reimplement everything on the stack just for the learning experience. But starting with just docker might be a good idea.
  • 3y ago
• rmccue 3y ago
  > Not using Terraform Cloud We adopted TFC at the start of 2023 and it was problematic right from the start; stability issues, unforeseen limitations, and general jankiness. I have no regrets about moving us away from local execution, but Terraform Cloud was a terrible provider. When they announced their pricing changes, the bill for our team of 5 engineers would have been roughly 20x, and more than hiring an engineer to literally sit there all day just running it manually. No idea what they’re thinking, apart from hoping their move away from open source would lock people in? We ended up moving to Scalr, and although it hasn’t been a long time, I can’t speak highly enough of them so far. Support was amazing throughout our evaluation and migration, and where we’ve hit limits or blockers, they’ve worked with us to clear them very quickly.
• Aeroi 3y ago
  As someone who isn't a developer, readingthis was eye opening. It's interesting just how unbundled the state of running a software company is. And this is only your selection of the tools and options, not imagining the entire landscape.
• LispSporks22 3y ago
  Can any of your engineers run the product locally and iterate fast?
  • cissmayazz 3y ago
    Yeah typically run a single go service or use devspace to combine multiple services using published containers
• ndjshe3838 3y ago
  I’m imagining a developer in the 90s/00s reading this list and being baffled by the complexity/terminology
  • SoftTalker 3y ago
    I am in 2024.
  • kypro 3y ago
    I thought the same reading it – is it really this hard to build an app these days? Things were more far more manual and much less secure, scalable and reliable in the past, but they were also far far simpler.
    • xcrunner529 3y ago
      Agreed. It’s just ridiculous. Some just love to spend money and make things more complex.
  • LispSporks22 3y ago
    I agree. I’m afraid I’m one of those 00s developers and can relate. Back then many startups were being launched on super simple stacks. With all of that complexity/word salad from TFA, where’s the value delivered? Presumably there’s a product somewhere under all that infrastructure, but damn, what’s left to spend on it after all the infrastructure variable costs? I get it’s a list of preferences, but still once you’ve got your selection that’s still a ton of crap to pay for and deal with. Do we ever seek simplicity in software engineering products?
    • bigstrat2003 3y ago
      I think that far too many companies get sold on the vision of "it just works, you don't need to hire ops people to run the tools you need for your business". And that is true! And while you're starting, it may be that you can't afford to hire an ops guy and can't take the time to do it yourself. But it doesn't take that much scale before you get to the point it would be cheaper to just manage your own tools. Cloud and SaaS tools are very seductive, but I think they're ultimately a trap. Keep your tools simple and just run them yourselves, it's not that hard.
• hintymad 3y ago
  > EKS My contrarian view is that EC2 + ASG is so pleasant to use. It’s just conceptually simple: I launch an image into an ASG, and configure my autoscale policies. There are very few things to worry about. On the other hand, using k8s has always been a big deal. We built a whole team to manage k8s. We introduce dozens of concepts of k8s or spend person-years on “platform engineering” to hide k8s concepts. We publish guidelines and sdks and all kinds of validators so people can use k8s “properly”. And we still write 10s of thousands lines of YAML plus 10s of thousands of code to implement an operator. Sometimes I wonder if k8s is too intrusive.
  • xyzzy_plugh 3y ago
    I tend to agree that for most things on AWS, EC2 + ASG is superior. It's very polished. EKS is very bare bones. I would probably go so far as to just run Kubernetes on EC2 if I had to go that route. But in general k8s provides incredibly solid abstractions for building portable, rigorously available services. Nothing quite compares. It's felt very stable over the past few years. Sure, EC2 is incredibly stable, but I don't always do business on Amazon.
    • Noumenon72 3y ago
      At first I thought your "in general" statement was contradicting your preference for EC2 + ASG. I guess AWS is such a large part of my world that "in general" includes AWS instead of meaning everything but AWS.
  • cedws 3y ago
    K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need ISO/PCI/whatever.
    • xyzzy_plugh 3y ago
• LispSporks22 3y ago
  I feel like this is overkill for a startup. Why not dump your application server and dependencies into rented data center (or EC2 if you must) and setup a coarse DR? Maybe start with a monolith in PHP or Rails. None of that word salad sounds like startup to me, but then again everyone loves to refer to themselves as a startup (must be a recruiting tool?), so perhaps muh dude is spot on.
  • icameron 3y ago
    I would like to know what you’re being downvoted for. It’s not bad advice, necessarily… this was the way 20 years ago. I mean isn’t hacker news running kind of like this as a monolith on a single server? People might be surprised how far you can get with a simple setup.
  • charred_patina 3y ago
    I don't want to be negative, but this post reads like a list of things that I want to avoid in my career. I did a brief stint in cloud stuff at a FAANG and I don't care to go back to it. Right now I'm engineer No. 1 at a current startup just doing DDD with a Django monolith. I'm still pretty Jr. and I'm wondering if there's a way to scale without needing to get into all of the things the author of this article mentions. Is it possible to get to a $100M valuation without needing all of this extra stuff? I realize it varies from business to business, but if anyone has examples of successes where people just used simple architecture's I'd appreciate it.
    • krmboya 3y ago
      I bet you can get pretty far with just ec2 and autoscaling, or comparable tech in other cloud platforms. With a managed database service.
      • charred_patina 3y ago
        That I'd be comfortable with.
    • kevinqi 3y ago
• jasoneckert 3y ago
  After reading through this entire post, I'm pleasantly surprised that there isn't one item where I don't mirror the same endorse/regret as the author. I'm not sure if this is coincidence or popular opinion.
• ehPReth 3y ago
  Okta... after everything that's happened recently with them?
  • deskamess 3y ago
    Yeah... this stood out! Do you have any good alternatives? I wish CloudFlare would do it (IDP).
• __turbobrew__ 3y ago
  It is a shame karpenter is AWS only. I was thinking about how our k8s autoscaler could be better and landed on the same kind of design as karpenter where you work from unschedulable pods backwards. Right now we have an autoscaler which looks at resource utilization of a node pool but that doesn’t take into account things like topology spread constraints and resource fragmentation.
  • acedTrex 3y ago
    https://github.com/Azure/karpenter-provider-azure https://github.com/Azure/karpenter-provider-azure there is this in the works for karpenter on aks
    • redrove 3y ago
      It’s actually released in preview, they called it Node Auto Provisioning. Doesn’t work with Azure Linux unfortunately.
      • Xelaz 3y ago
        Azure Linux support has been added recently [1], and so is available for self-hosted Karpenter (you can specify "imageFamily: AzureLinux" in AKSNodeClass), will be rolled out for the managed experience (Node Auto Provisioning) soon. [1] https://github.com/Azure/karpenter-provider-azure/pull/72 https://github.com/Azure/karpenter-provider-azure/pull/72 (source: On the team that is developing the provider)
        • redrove 3y ago
          Awesome, thank you.
• sreeramvenkat 3y ago
  Ironic that the article begins with an image of server chassis with wires running around while the description is entirely about cloud infra.
• ChuckMcM 3y ago
  This is fabulous. I keep lists like this in my notebook(s). The critical thing here is that you shouldn't dwell on your "wrong" choices, instead document the choice, what you thought you were getting, what you got, and what information would have been helpful to know at the time of decision (or which information you should have given more weight at the time of the decision.) If you do this, you will consistently get better and better. And by far "automate all the things" is probably my number one suggestion for DevOps folks. Something that saves you 10 minutes a day pays for itself in a month when you have a couple of hours available to diagnose and fix a bug that just showed up. (5 days a week X 4 weeks X 10 minutes = 200 minutes) The exponential effect of not having to do something is much larger than most people internalize (they will say, "This just takes me a couple of minutes to do." when in fact it takes 20 to 30 minutes to do and they have to do it repeatedly.)
• janfoeh 3y ago
  Almost every time I read someone's insights who works in an environment with IaaS buy-in, my takeaway is the same: oh boy, what an alphabet soup. The initial promise of "we'll take care of this for you, no in-house knowledge needed" has not materialized. For any non-trivial use case, all you do is replace transferrable, tailored knowledge with vendor-specific voodoo. People who are serious about selling software-based services should do their own infrastructure.
• cyounkins 3y ago
  I've climbed the mountain of learning the basics of kubernetes / EKS, and I'm thinking we're going to switch to ECS. Kubernetes is way too complicated for our needs. It wants to be in control and is hard to direct with eg CloudFormation. Load balancers are provisioned from the add-on, making it hard to reference them outside kubernetes. Logging on EKS Fargate to Cloudwatch appears broken, despite following the docs. CPU/Memory metrics don't work like they do on EKS EC2, it appears to require ADOT. I recreated the environment in ECS in 1/10th the time and everything just worked.
  • jacurtis 3y ago
    I've been running ECS for about 5 years now. It has come a long way from a "lightweight" orchestration tool into something thats actually pretty impressive. The recent new changes to the GUI are also helpful for people that don't have a ton of experience with orchestration. We have moved off of it though, you can eventually need more features than it provides. Of course that journey always ends up in Kubernetes land, so you eventually will find your way back there. Logging to Cloudwatch from kubernetes is good for one thing... audit logs. Cloudwatch in general is a shit product compared to even open source alternatives. For logging you really need to look at Fluentd or Kibana or DataDog or something along those lines. Trying to use Cloudwatch for logs is only going to end in sadness and pain.
    • busterarm 3y ago
      GKE is a much better product to me still than EKS but at least in the last two years or so EKS has become a usable product. Back in like 2018 though? Hell no, avoid avoid avoid.
  • el_isma 3y ago
    I started with ECS (because I wanted to avoid the complexity of K8s) and regret it. I feel I wasted a lot of time there. In ECS, service updates would take 15 min or more (vs basically instant in K8s). ECS has weird limits on how many containers you can run on one instance [0]. And in the network mode where you can run more containers on a host, then the DNS is a mess (you need to lookup SRV records to find out the port). Using ECS with CDK/Cloudformation is very painful. They don't support everything (specially regarding Blue/Green deployments), and sometimes they can't apply changes you do to a service. When initially setting up everything, I had to recreate the whole cluster from scratch several times. You can argue that's because I didn't know enough, but if that ever happened to me on prod I'd be screwed. I haven't used EKS (I switched to Azure), so maybe EKS has their own complex painful points. I'm trying to keep my K8s as vanilla as possible to avoid the cloud lock-in. [0] https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/networking-networkmode.html https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesgu...
• amluto 3y ago
  > Zero Trust VPN VPNs can be wonderful, and you can use use Tailscale or AWS VPN or OpenVPN or IPSEC and you can authenticate using Okta or GSuite or Auth0 or Keycloak or Authelia. But since when is this Zero Trust? It takes a somewhat unusual firewall scheme to make a VPN do anything that I would seriously construe as Zero Trust, and getting authz on top of that is a real PITA.
• morsecodist 3y ago
  > Picking AWS over Google Cloud I know this is an unpopular opinion but I think google cloud is amazing compared to AWS. I use google cloud run and it works like a dream. I have never found an easier way to get a docker container running in the cloud. The services all have sensible names, there are fewer more important services compared to the mess of AWS services, and the UI is more intuitive. The only downside I have found is the lack of community resulting in fewer tutorials, difficulty finding experienced hires, and fewer third party tools. I recommend trying it. I'd love to get the user base to an even dozen. The reasoning the author cites is that AWS has more responsive customer service and maybe I am missing out but it would never even occur to me to speak to someone from a cloud provider. They mention having "regular cadence meetings with our AWS account manager" and I am not sure what could be discussed. I must be doing simper stuff.
  • iimblack 3y ago
    I don’t have as much experience with aws but I do hate gcp. The ui is slow and buggy. The way they want things to authenticate is half baked and only implemented in some libraries and it isn’t always clear what library supports it. The gcloud command line tool regularly just doesn’t work; it just hangs and never times out forcing you to kill it manually wondering if it did anything and you’ll mess something up running it again. The way they update client libraries by running code generation means there’s tons of commits that aren’t relevant to the library you’re actually using. Features are not available across all client libraries. Documentation contradicts itself or contradicts support recommendations. Core services like bigquery lack any emulator or Docker image to facilitate CI or testing without having to setup a separate project you have to pay for.
    • arccy 3y ago
      aws is even worse yet somehow people love them, maybe because they get to talk to a support "human" to hand-hold them through all the badness
    • mdaniel 3y ago
      Oh, friend, you have not known UI pain until you've used portal.azure.com. That piece of junk requires actual page reloads to make any changes show up. That Refresh button is just like the close-door elevator button: it's there for you to blow off steam, but it for damn sure does not DO anything. I have boundless screenshots showing when their own UI actually pops up a dialog saying "ok, I did what you asked but it's not going to show up in the console for 10 minutes so check back later". If you forget to always reload the page, and accidentally click on something that it says exists but doesn't, you get the world's ugliest error message and only by squinting at it do you realize it's just the 404 page rendered as if the world has fallen over I suspect the team that manages it was OKR-ed into using AJAX but come from a classic ASP background, so don't understand what all this "single page app" fad is all about and hope it blows over one day
• jiggawatts 3y ago
  Something I’ve noticed with PaaS services like RDS or Azure SQL is that people arguing against it are assuming that the alternative is “competence”. Even in a startup, it’s difficult to hire an expert in every platform that can maintain a robust, secure system. It’s possible, but not guaranteed, and may require a high pay to retain the right staff. Many government agencies on the other hand are legally banned from offering a competitive wage, so they can literally never hire anyone that competent. This cap on skill level means that if they do need reliable platforms, the only way they can get one is by paying 10x the real market rate for an over-priced cloud service. These are the “whales” that are keeping the cloud vendors fat and happy.
• IamLoading 3y ago
  > Go is for services that are non-GPU bound. What are they using for GPU bound services. Python?
  • cissmayazz 3y ago
    Python indeed
• itpragmatik 3y ago
  Not sure the fascination about Go - one can write fully scalable functional readable maintainable upgradable rest api service with Java 17 and above.
  • MarkMarine 3y ago
    I struggle with the type system in both, but today I was going through obscure go code and wishing interfaces were explicitly implemented. Lack of sum types is making me sad
• nickzelei 3y ago
  What are startups using for a logging tool that isn’t datadog?
  • podoman 3y ago
    https://highlight.io https://highlight.io
  • Too 3y ago
    Loki
  • ndr 3y ago
    https://axiom.co/ https://axiom.co/
• bilalq 3y ago
  I love this write-up and the way it's presented. I disagree with some of the decisions and recommendations, but it's great to read through the reasoning even in those cases. It'd be amazing if more people published similar articles and there was a way to cross-compare them. At the very least, I'm inspired to write a similar article.
• roughly 3y ago
  The Bazel one made me chuckle - I worked at a company with an scm & build setup clearly inspired by Google’s setup. As a non-ex-Googler, I found it obviously insane, but there was just no way to get traction on that argument. I love that the rest of this list is pretty cut and dry, but Bazel is the one thing that the author can’t bring themself to say “don’t regret” even though they clearly don’t regret not using it.
  • busterarm 3y ago
    I've seen Bazel reduce competent engineers to tears. There was a famous blog post a half-decade ago called something like "Bazel is the worst build system, except for all the others" and this still seems to ring true for me today. There are some teams I work with that we'll never bother to make use Bazel because we know in advance that it would cripple them.
    • ali_piccioni 3y ago
      Having led a successful Bazel migration, I'd still recommend many projects to stick to the native or standard supported toolchain until there's a good reason to migrate to a build system (And I don't consider GitHub actions to be a build system).
  • dieortin 3y ago
    I’m curious, what do you find insane about Bazel? In my experience it makes plenty of sense. And after using it for some months, I find more insane how build systems like CMake depend on you having some stuff preinstalled in your system and produce a different result depending on which environment they’re run on.
• bayareabadboy 3y ago
  Interesting enough read. But I’m not sure he’s a regretful enough boy to write a blog to merit the title.
• fswd 3y ago
  stuff like this makes me want to experiment with going back to just one huge $100k server and running it all on one box in a server rack.
  • sseagull 3y ago
    I am doing that. I am part of a research group, and don’t have the $$ or ability to pay so much for all these services. So we got a $90k server with 184TB of raw storage (SAS SSD), 64 cores, and 1TB of memory. Put it on a 10GB line at our university and it is rock solid. We probably have less downtime than Github, even with reboots every few months. Have some large (multi-TB) databases on it and web APIs for accessing the data. Would be hugely expensive in the cloud with, especially with egress costs. You have to be comfortable sys-admining though. Fortunately I am.
• hi_hi 3y ago
  I was hoping there would be a section for Search Engines. It's one of those things you tend to get locked in to, and it's hard to clearly know your requirements well enough early on. Any references to something like this with a Search slant would be greatly appreciated.
• brycelarkin 3y ago
  Awesome writeup! Just had a couple comments/questions. > Not adopting an identity platform early on The reason for not adopting an IDP early is because almost every vendor price gouges for SAML SSO integration. Would you say it's worth the cost even when you're a 3-5 person startup? > Datadog What would you recommend as an alternative? Cloudwatch? I love everything about Datadog, except for their pricing.... > Nginx load balancer for EKS ingress Any reason for doing this instead of an Application Load Balancer? Or even HA Proxy?
  • kevinslin 3y ago
    For datadog, unfortunately there's no obvious altnernative despite many companies trying to take marketshare. This is to say, datadog both has second to none DX and a wide breadth of services. Grafana Labs comes closest in terms of breadth but their DX is abysmal (I say this as a heavy grafana/prometheus user) Same comments about new relic though they have better dx than grafana. Chronosphere has some nice DX around prometheus based metrics but lack the full product suite. I could go on but essentially, all vendors either lack breadth, DX, or both.
• hitekker 3y ago
  Props to the author for writing up the results from his exercise. But I think he should focused on a few controversial ones, and not the rotes ones. Many of the decisions presented are not disagreeable (choosing slack) and some lack framing that clarifies the associated loss (Not adopting an identity platform early on). I think they're all good choices worth mentioned; I would have preferred a deeper look into the few that seemed easy and turned out to be hard, or the ones that were hard and got even harder.
  • 8organicbits 3y ago
    > not the rotes ones It helps to hear the validation, although I think almost every decision has a dissenting voice in the HN comments.
• MarchApril 3y ago
  Terraform is great but it's so frustrating sometimes. You just pray that the provider has a specific configuration of whatever resources you're working with, because else when them resources are up on multiple env then you'll have to edit those configs somehow.
• isoprophlex 3y ago
  > There are no great FaaS options for running GPU workloads This hits hard. Someone please take my (client's) money and provide sane GPU FaaS. Banana.dev is cool but not really enterprise ready. I wish there was a AWS/GCP/Azure analogue that the penny pinchers and MBAs in charge of procurement can get behind.
  • karbon0x 3y ago
    I am confused. Doesn't Modal Labs solve this?
    • isoprophlex 3y ago
      Definitely. But the sad reality is that in some corporate environments (incumbent finance, government) if it's not a button click in portal.azure.com away, you can spend 6-12 months in meetings with low energy gloomboys to get your access approved.
      • karbon0x 3y ago
        Ah, I see. Yeah, been victim of that bureaucracy as well.
• Rainymood 3y ago
  As a machine learning platform engineer these sound like technology choices as opposed to infrastructure decisions. I would love to read this post but really with the infrastructure trade-offs that were made. But thanks for the post. Side node: There is a small typo repeated twice "Kuberentes"
• davedx 3y ago
  Utter insanity. So much cost and complexity, and for what? Startups don’t think about costs or runway anymore, all they care about is “modern infrastructure”. The argument for RDS seems to be “we can’t automate backups”. What on earth?
  • isbvhodnvemrwvn 3y ago
    Is spending time to make it reliable worth it vs working on your actual product? Databases are THE most critical things your company has.
    • davedx 3y ago
      All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird
      • isbvhodnvemrwvn 3y ago
        Setting up a multi-az db with automatic failover, incremental backups and PiTR, automated runbooks and monitoring all that doesn't take half a day, not even with RDS.
        • davedx 3y ago
          No, but again, that sounds like a lot of complexity your average startup does not need. Multi-az? Why?
          • marcinzm 3y ago
            Because their Enterprise client requires it on their due diligence paperwork.
            • 3y ago
• tofflos 3y ago
  > Using cert-manager to manage SSL certificates > Very intuitive to configure and has worked well with no issues. Highly recommend using it to create your Let’s Encrypt certificates for Kubernetes. > The only downside is we sometimes have ANCIENT (SaaS problems am I right?) tech stack customers that don’t trust Let’s Encrypt, and you need to go get a paid cert for those. Cert-manager allows you to use any CA you like including paid ones without automation.
• kunley 3y ago
  A fallacy of a "choice" between GCP and AWS never stops to entertain me
• danielovichdk 3y ago
  I would have liked some data around why these technologies were chosen and preferably based on loads from customers. Seems like yagni to me but please prove me wrong
• corentin88 3y ago
  Curious about the mention of buying IPs. Anyone else can share feedback/thoughts on this?
  • cissmayazz 3y ago
    This was done for multiple reasons but mainly security and to allow customers to whitelist a certain ip range.
• throwawaaarrgh 3y ago
  This guy gets it, I agree with it all. The exception being, use Fargate without K8s and lean on Terraform and AWS services rather than the K8s alternatives. When you have no choice left and you have to use K8s, then I would pick it up. No sense going down into the mines if you don't have to.
• michidk 3y ago
  > Code is of course powerful, but I’ve found the restrictive nature of Terraform’s HCL to be a benefit with reduced complexity. No way. We used Terraform before and the code just got unreadable. Simple things like looping can get so complex. Abstraction via modules is really tedious and decreases visibility. CDKTF allowed us to reduce complexity drastically while keeping all the abstracted parts really visible. Best choice we ever made!
• opentokix 3y ago
  After working with infrastructure for 20 years, I fully endorse this post.
• kosolam 3y ago
  What is the cost? With 1/10th of the sum one capable engineer can setup a way better infra on premise. The days of free money is over, guys. Wake up!
• politelemon 3y ago
  > Ubuntu for dev servers I didn't understand this section. Ubuntu servers as dev environment, what do you mean? As in an environment to deploy things onto, or a way for developers to write code like with VSCode Remote?
  • hahnchen 3y ago
    seems like the latter given "Originally I tried making the dev servers the same base OS that our Kubernetes nodes ran on, thinking this would make the development environment closer to prod"
    • runiq 3y ago
      But I thought the whole point of the container ecosystem was to abstract away the OS layer. Given that the kernel is backwards compatible to a fault, shouldn't it be enough to have a kernel that is as least as recent as the one on your k8s platform (provided that you're running with the default kernel or something close to it)?
      • Frivolous9421 3y ago
        [dead]
  • brainzap 3y ago
    My take from this was more: being uniform reduced overhead of maintaining. Being able to write a bash script that runs on ever machine is nice.
• politelemon 3y ago
  > homebrew for Linux No, just no. I see this cropping up now and then. Homebrew is unsafe for Linux, and is only recommended by Mac users that don't want to bother to learn about existing package management.
• erostrate 3y ago
  The author leads infrastructure at Cresta. Cresta is a customer service automation company. His first point is about how happy he is to have picked AWS and their human-based customer service, versus Google's robot-based customer service. I'm not saying there's anything wrong, and I'm oversimplifying a bit, but I still find this amusing.
  • lysecret 3y ago
    Haha very good catch. I prefer GCP but I will admit any day of the week that their support is bad. Makes sense that they would value good support highly.
    • danpalmer 3y ago
      We used to use AWS and GCP at my previous company. GCP support was fine, and I never saw anything from AWS support that GCP didn't also do. I've heard horror stories about both, including some security support horror stories from AWS that are quite troubling.
• yread 3y ago
  > Ubuntu we have dotnet webapp deployed on Ubuntu and it leaves a lot to be desired. The package for .net6 from default repo didn't recognise other dotnet components installed, net8 is not even coming to 22.04 - you have to install from the ms repo. But that is not compatible with the default repo's package for net6 so you have to remove that first and faff around with exact versions to get it installed side by side... At least I don't have to deal with rhel Why is renewing a dev subscription so clunky?!
• f549abd0 3y ago
  Disagree on the point and reasoning about the single database. Sounds like they experienced badly managed and badly constrained database. The described fks and relations: that's what the key constraints and other guard rails and cascades are for - so that you are able to manage a schema. That's exactly how you do it: add in new tables that reference old data. I think the regret is actually not managing the database, and not so much about having a single database. "database is used by everyone, it becomes cared for by no one". How about "database is used by everyone, it becomes cared for by everyone".
  • f549abd0 3y ago
    Reading further > Endorse-ish: Schema migration by Diff Well that explains it... What a terrible approach to migrations for data integrity.
    • cloogshicer 3y ago
      Genuinely curious (I don't have much experiences with DBs), how is schema migration done 'properly' these days?
      • jspdown 3y ago
        Incremental forward-only migrations (non-state based). Then, for the How and When, it mostly depends of your constraints and sizes. There's no silver bullet, it's hard, it require constant thinking, it's a slow and often multi step process. I never saw a successful fully automated one-way-of-doing process.
        • from-nibly 3y ago
          Are you talking about the mechanics? Like more than just run a migration script on boot?
    • jamescontrol 3y ago
      Can you explain? Having a tool to detect changes and create a migration doesn’t sound bad? In a nutshell thats how django migrations work as well, which works really well.
• eadmund 3y ago
  > Startups don’t have the luxury of a DBA … I understand, but I think they don’t have the luxury of not having a DBA. Data is important; it’s arguably more important than code. Someone needs to own thinking about data, whether it is stored in a hierarchical, navigation-based database such as a filesystem, a key-value stored like S3 (which, sure, can emulate a filesystem), or in a relational database. Or, for that matter, in vendor systems such as Google Workspace email accounts or Office365 OneDrive.
  • Draiken 3y ago
    Early on, depending on what you're building, you don't need a fully fleshed DBA and can get away with at least one person that knows DB fundamentals. But if you only want to hire React developers (or swap for the framework of the week) then you'll likely end up with zero understanding of the DB. Down the line you have a mess with inconsistent or corrupted data that'll come back with a vengeance. It's short-sighted for serious endeavors.
• lysecret 3y ago
  Half the stuff is K8s related... Makes me very happy to use Cloud Run.
• gokhan 3y ago
  "Multiple applications sharing a database" and Kubernetes sound really funny together:)
• shp0ngle 3y ago
  I should really learn AWS huh
• knowsuchagency 3y ago
  Using k8s over ECS and raw-dogging Terraform instead of using the CDK? It's no wonder you end up needing to hire entire teams of people just to manage infra
• BrickTamblan 3y ago
  What’s the right way to manage npm installs and deploy it to an AWS ec2 instance from github? Kubernetes? GitOps? EKS? I roll my own solution now with cron and bash because everything seems so bloated.
• ildjarn 3y ago
  Reading this I couldn’t help but think: yeah all of these points make sense in isolation, but if you look at the big picture, this is an absurd level of complexity. Why do we need entire teams making 1000s of micro decisions to deploy our app? I’m hungry for a simpler way, and I doubt I’m alone in this.
  • klabb3 3y ago
    You’re not alone. There is a constant undercurrent of pushback against this craziness. You see it all the time here on hacker news and with people I talk to irl. Does not mean each of these things don’t solve problems. The issue as always about complexity-utility tradeoff. Some of these things have too much complexity for too little utility. I’m not qualified to judge here, but if the suspects have Turing-complete-yaml-templates on their hands, it probably ties them to the crime scene.
  • Sammi 3y ago
    It smells like ZIRP is not over yet. VCs are still burning money in the AWS fire pit.
    • kibwen 3y ago
      ZIRP was never the root problem. The problem was: too much money, too few consequences for burning it. The existence of the uber-wealthy means that markets can no longer function efficiently. Every market remains irrational longer than anyone who's not uber-wealthy can remain solvent. Welcome to the new normal.
      • daxfohl 3y ago
        Now it's "fix it with AI". (And pay lip service to green tech.)
• jgalt212 3y ago
  > We use Okta to manage our VPN access and it’s been a great experience. I have no first hand exerience wtih Okta, but everything I read about it makes me scared to use it. i.e. stability and security.
• rexreed 3y ago
  Sounds like a whole lot of stuff for a startup. Maybe start with a simple stack until there's market fit. Even Amazon didn't start this way.
• iandanforth 3y ago
  For people who enjoyed this post but want to see the other side of the spectrum where self hosted is the norm I'll point to the now classic series of posts on how Stack Overflow runs its infra: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... If anyone has newer posts like the above, please reply with links as I would love to read them.
  • alecthomas 3y ago
    https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0 https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47... is another good one. There are a few different posts on it scattered around: https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa https://world.hey.com/dhh/we-stand-to-save-7m-over-five-year... https://world.hey.com/dhh/our-cloud-exit-has-already-yielded-1m-year-in-savings-db358dea https://world.hey.com/dhh/our-cloud-exit-has-already-yielded... Related, looks like X is doing similar: https://twitter.com/XEng/status/1717754398410240018 https://twitter.com/XEng/status/1717754398410240018
• Shorel 3y ago
  I see more 'Endorse' items than 'Regret' items. Anyway, amazing write up. Learning about alternatives to Jira is always good.
• maccard 3y ago
  I see homebrew in here as a way to distribute <stuff> internally. We have non-developers (artists, designers) on our team, and asking them to manage homebrew is a non-starter. We're also on windows. We current just shove everything (and I mean everything) in perforce. Are there any better ways of distributing this for a small team?
• pavel_lishin 3y ago
  > Discourage private messages and encourage public channels. I wish my current company did this. It's infuriating. The other day, I asked a question about how to set something up, and a manager linked me to a channel where they'd discussed that very topic - but it was private, and apparently I don't warrant an invite, so instead I have to go bother some other engineers (one of whom is on vacation.) Private channels should be for sensitive topics (legal, finance, etc) or for "cozy spaces" - a team should have a private channel that feels like their own area, but for things like projects and anything that should be searchable, please keep things public.
• foxhop 3y ago
  I think kubernetes is a mistake and should have went with AWS ECS (using fargate or backed by autoscaling ec2), if single change he wouldn't need to even thing about a bunch of other topics on his list. Something to think about, AWS Lambda first then fallback to AWS ECS for everything else that needs to really be on 100% of the time.
• pigcat 3y ago
  > My general infrastructure advice is “less is better”. I found this slightly ironic given there are ~50 headers in the article :) I liked the format of the writeup
• thesurlydev 3y ago
  I've seen a lot of comments about how bad DataDog is because of cost but surprisingly I haven't seen open-source alternatives like OpenTelemetry/Prometheus/Grafana/Tempa mentioned. Is it because most people are willing to pay someone else to manage monitoring infrastructure or other reasons?
  • kevinslin 3y ago
    the way I think of datadog is that datadog it provides a second to none DX combined with a wide suite of product offerings that is good enough for most companies most of the time. does it have opaque pricing that can be 100x more expensive than alternatives? absolutely! will people continue to use it? yes! something to keep in mind is that most companies are not like the folks in this thread. they might not have the expertise, time or bandwidth to build invest in observability. the vast majority of companies just want something that basically works and doesn’t take a lot of training to use. I think of Datadog as the Apple of observability vendors - it doesn’t offer everything and there are real limitations (and price tags) for more precise use cases but in the general case, it just works (especially if you stay within its ecosystem)
• data_maan 3y ago
  Noob here - all these are great... but why can't I just use Heroku to radically not have to deal with a large prt if these things?
• nescioquid 3y ago
  > Multiple applications sharing a database [regret] The industry has known this to be a stereotypically bad idea for generations now. It lead to things like the enterprise sevice bus, service-oriented architectures, and finally "micro services". Recently I've seen "micro services" that share the same database, so we've come full-circle. Yet, every place I've worked was either laboring under a project to decouple two or more applications that were conjoined at the DB, or were still at the "this sucks but no one wants to fix it" stage. How do we keep making this same mistake in industry?
• theonething 3y ago
  I don't get why all startups don't just start with a PaaS like Render, Fly.io or Heroku. Why spend time setting up your own infra and potentially have to hire dedicated staff to manage it when you can do away with all that and get on with trying to move your business forward? If and when you start experiencing scaling problems (great!), that's the time to think about migrating to setting up infra.
  • exikyut 3y ago
    Because like every service-oriented offering, each platform differentiates as hard as it can to lock you in to their way of doing things. Things largely look the same on the surface; this takes the most effect at the implementation-detail level, where adjusting and countercorrecting down the track is fiddly and uses an adrenally-draining level of attention span - right when you're at the point where you're scaling and you no longer have the time to deal with implementation detail level stuff. You're on <platform> and you're doing things their way and pivoting the architecture will only be prioritised if the alternative would be bankruptcy.
    • theonething 3y ago
      When you're starting out, you just need a server to run your application and a database. It literally doesn't matter what service you're using at that point. I don't see how you need to be "doing things their way" when that's all you have.
• nuker 3y ago
  AWS ECS is better than Kubernetes, and Cloudformation is better than Terraform. Just my IMHO. Both are more simple and do the same thing.
• gjx324 3y ago
  good
• deleted 3y ago
  [deleted]
• petrusnonius 3y ago
  Nice, I run Kamal on Hetzner with Cloudflare.
• tritiy 3y ago
  I find the amount of services/products used insane. Is this all handled/known by those mythical full-stack-dev-sec-ops developers?