48 ms·

Reclaim the Stack

496 points by dustedcodes 2y ago

sph 2y ago
"Join the Discord server"? Who's the audience of this project?
- mre 2y ago
  Genuinely curious, what's wrong with that? Did you expect a different platform like Slack?
  - fragmede 2y ago
    it would be better at the bottom of the first documentation page, after the reader has a better idea of what this is
  - callalex 2y ago
    Locking knowledge behind something that isn’t publicly searchable or archivable works fine in the short term but what happens when Discord/Slack/whatever gears up for an IPO and limits all chat history to 1 week unless you pay up (oh and now you have a bunch of valuable knowledge stored up their with no migration tool so your only options are “pay up” or lose the knowledge).
    - halfcat 2y ago
      What’s recommended here? Self-hosted Discourse?
      - heavyset_go 2y ago
        Matrix and a wiki would solve the community and knowledge base issues.
        mplewis 2y ago
        Matrix has severe UX issues which drastically limit the community willing to use it on a regular basis.
        Arathorn 2y ago
        historically, yes. matrix 2.0 (due in two weeks) aims to fix this.
      - tacker2000 2y ago
        Github issues or discussions. Or some other kind of forum like Discourse as you mentioned
    - Kiro 2y ago
      No-one complained when projects had IRC channels/servers, which are even worse since they have no history at all.
      - okasaki 2y ago
        All IRC clients have local plain text logging and putting a .txt on a web server is trivial.
        Kiro 2y ago
        Local logging doesn't help much for searchability when you're new and it requires you to be online 24/7. Anyway, that's beside the point. Even if IRC had built-in server history it still has the same problems but I never saw people being outraged about it.
      - Gormo 2y ago
        Good projects still do rely on IRC -- Libera.chat is full of proper channels -- and logging bots are ubiquitous.
        deleted 2y ago
        [deleted]
        Kiro 2y ago
        And you never hear anyone complaining about those. "Locking knowledge" was never an argument before and it's not now.
      - callalex 2y ago
        At least people treat IRC as ephemeral and place all documentation elsewhere. People are writing whole wikis inside of Discord that are not publicly searchable.
  - Gormo 2y ago
    There's a whole FOSS ecosystem of chat/collaboration applications, like Mattermost and Zulip; there's Matrix for a federated solution, and tried-and-true options like IRC. For something called "Reclaim the Stack" to lock discussion into someone else's proprietary walled garden is quite ironic.
- deleted 2y ago
  [deleted]
- mrits 2y ago
  People that don't like wasting money?
  - hobs 2y ago
    Not capturing the information and being able to use it in the future is a huge opportunity cost, and idling on discord pays no bills.
  - Gormo 2y ago
    Wasting money on... better solutions that are also free?
    - mrits 2y ago
      There is a difference between wasting and spending money.
- tacker2000 2y ago
  Also noticed this. Everytime I see a project using discord as main communication tool it makes me think about the “fitness” of the project in the long run. Discord is NOT a benefit. Its not publicly searchable and the chat format is just not suitable to a knowledge base or support based format. Forums are much better in that regard.
  - KronisLV 2y ago
    > Discord is NOT a benefit. Its not publicly searchable and the chat format is just not suitable to a knowledge base or support based format. I don't think people who choose Discord necessarily care about that. Discord is where the people are, so that's where they go. It also costs close to nothing to setup a server and since it has a lower barrier of entry than hosting your own forum, it's deemed good enough. That said, modern forum software like Discourse https://www.discourse.org/ https://www.discourse.org/ or Flarum https://flarum.org/ https://flarum.org/ can be pretty good, though I still miss phpBB.
    - Gormo 2y ago
      > Discord is where the people are, so that's where they go. That doesn't sound right. Each Discord community is its own separate space -- you still need people to join your specific community regardless of whether it is hosted on Discord or something better. > though I still miss phpBB. It hasn't gone away -- the last release was on August 29th, so this is still very much a viable option.
      - dbackeus 2y ago
        It's all in one app and the app has a ton of users. Anyone running the app can join any server with a click of a button. There are no separate accounts required to join different communities. So communities being separate "spaces" doesn't create any meaningful friction with regards to adoption.
evantahler 2y ago
Porter (https://www.porter.run/ https://www.porter.run/) is a great product in the same vein (e.g. turn K8s into a dev-friendly Heroku-like PASS). How does this compare?
- mikeortman 2y ago
  I think the very concept of this is to open source a common stack, instead of relying on a middleman like Porter, which also costs a TON of money at business tier
fragmede 2y ago
having your tool be a single letter, k, seems rather presumptuous.
- rjbwork 2y ago
  Especially given K is already the name of an APL derivative.
- dbackeus 2y ago
  I suppose it is. But no actual users of the platform has had any complaints about it.
chrisweekly 2y ago
This looks great! Thank you for sharing, @dustedcodes. I might set up a playground to gain more hands-on experience w/ the relevant significant parts (k8s, argocd, talos) all of which have been on my radar for some time... Also, the docs look great. I love the Architecture Decision Records (bullet-point pros/cons/context)...
andrewstuart 2y ago
How can a NewsDesk application need kubernetes? Wouldn't a single machine and a backup machine do the job?
- andrewstuart 2y ago
  I just looked it up - its because they run Ruby On Rails.
  - zymhan 2y ago
    and so what?
    - andrewstuart 2y ago
      Ruby On Rails is well known for not being at the fast end of the spectrum, so it needs lots of machines, and lots of machines gives reason to user Kubernetes. A NewsDesk application written in something compiled for example golang would be much faster and likely could run on a single server. The benefit of single server being you don't need kubernetes and can spend that development resource on developing application features.
      - jakelazaroff 2y ago
        I personally prefer Go to Rails, but let’s be real here: the market cap of Rails is probably, like, a hundred times the market cap of Go.
        sien 2y ago
        No doubt with Github, AirBnB, Shopify and other big sites RoR is bigger for the front end. But now if lots of those sites are running on K8s with Argo CD or something or on a cloud platform where the infrastructure is provisioned with Terraform Go is supporting a great deal of things but it's far less visible.
        maxk42 2y ago
appplication 2y ago
This sounds great, I’ll be building our prod infra stack and deploying to cloud for the first time here in the next few weeks, so this is timely. It’s nice seeing some OSS-based tooling around k8s. I know it’s a favorite refrain that “k8s is unnecessary/too complex, you don’t need it” for many folks getting started with their deployments, but I already know and use it in my day job, so it feels like a pretty natural choice.
- briandear 2y ago
  The K8s is unnecessary meme is perpetuated by people that don’t understand it.
  - actionfromafar 2y ago
    True, but also, sometimes it’s not needed.
    - jauntywundrkind 2y ago
      Sometimes it just feels good wearing a fig leaf around my groin, weilding a mid sized log as a crude club, & running through the jungle. You might not need it is the kernel of doubt that can undermine any reasonable option. And it suggests nothing. Sure, you can go write your own kernel! You can make your own database! You might not need to use good well known proven technology that people understand and can learn about online! You can do it yourself! Or cobble together some alternate lesser special stack that just you have distilled out. We don't need civilization. We can go it alone & do our own thing, leave behind shared frames of references. But damn, it just seems so absurdly inadvisable, and it feels so overblown the fear uncertainty & doubt telling us Kubernetes is hard and bad and too much. This article does certainly lend credence to the idea that Kubernetes is complex, but there's so many simpler starting places that will take many teams very far.
      - samatman 2y ago
        Somehow kubernetes and civilization just aren't in the same category of salience to me. Like I think it's reasonable to say that kubernetes is optional in a way which civilization isn't. Like maybe one of those things is more important. than, the other
thetopher 2y ago
“Our basic philosophy when it comes to security is that we can trust our developers and that we can trust the private network within the cluster.” This is not my area of expertise. Does it add a significant amount of complexity to configure this kind of system in a way that doesn’t require trusting the network? Where are the pain points?
- zymhan 2y ago
  It requires encrypting all network traffic, either with something like TLS, or IPSec VPN.
- umvi 2y ago
  Implementing "Zero Trust" architectures are definitely more onerous to deal with for everyone involved (both devs and customers, if on prem). Just Google "zero trust architecture" to find examples. A lot more work (and therefore $) to setup and maintain, but also better security since now breaching network perimeter is no longer enough to pwn everything inside said network.
- nilsherzig 2y ago
  "SSL added and removed here :^)"
- callalex 2y ago
  The top pain point is that it requires setting up SSL certificate infrastructure and having to store and distribute those certs around in a secure way. The secondary effects are entirely dependent on how your microservices talk to their dependencies. Are they already talking to some local proxy that handles load balancing and service discovery? If so, then you can bolt on ssl termination at that layer. If not, and your microservice is using dns and making http requests directly to other services, it’s a game of whack-a-mole modifying all of your software to talk to a local “sidecar”; or you have to configure every service to start doing the SSL validation which can explode in complexity when you end up dealing with a bunch of different languages and libraries. None of it is impossible by any means, and many companies/stacks do all of this successfully, but it’s all work that doesn’t add features, can lead to performance degradation, and is a hard sell to get funding/time for because your boss’s boss almost certainly trusts the cloud provider to handle such things at their network layer unless they have very specific security requirements and knowledge.
fourseventy 2y ago
In my experience you can get pretty far with just a handful of vms and some bash scripts. At least double digit million ARR. Less is more when it comes to devops tooling imo.
- cloudking 2y ago
  +1 or just use App Engine, deploy your app and scale
  - cglace 2y ago
    App engine deploys are soooo slow. I liked cloud run a lot more.
- sanderjd 2y ago
  Of course you can get away with that if your metric is revenue. (I think Blippi makes about that much with, I suspect, nary a VM in sight! The question is what you're doing with your infrastructure, not how much revenue you're making. Some things have higher return to "devops" and others have less.
- lolinder 2y ago
  > you can get pretty far with just a handful of vms and some bash scripts. At least double digit million ARR. Using ARR as the measurement for how far you can scale devops practices is weird to me. Double-digit million ARR might be a few hundred accounts if you're doing B2B, and double-digit million MAUs if you're doing an ad-funded social platform. Depending on how much software is involved your product could be built by a team of anywhere from 1-50 developers. If you're a one-developer B2B company handling 1-3 requests per second you wouldn't even need more than one VM except maybe as redundancy. But if you're the fifty-developer company that's building something beyond simple CRUD, there are a lot of perks that come with a full-fledged control plane that would almost certainly be worth the added cost and complexity.
  - marcosdumay 2y ago
    > double-digit million MAUs I was about to make a similar point, but you made the math, and it's holding-up for the GP's side. You can push vms and direct to ssh synchronization up to double-digit million MAU (unless you are using stuff like persistent web-sockets). It won't be pretty, but you can get that far.
AbuAssar 2y ago
How does this compare to dokku (https://dokku.com/ https://dokku.com/)?
- dbackeus 2y ago
  Main difference is that Dokku is a simple single server platform, geared mostly toward hobby projects. Reclaim the Stack provides a fully highly available multi node platform to host large scale SaaS applications aiming for four nines of uptime.
mikeortman 2y ago
I'm glad we are starting to lean into cloud-agnostic or building back the on-prem/dedicated systems again.
airstrike 2y ago
> We spent 7 months building a Kubernetes based platform to replace Heroku for our SaaS product at mynewsdesk.com. The results were a 90% reduction in costs and a 30% improvement in performance. I don't mean to sound dismissive, but maybe the problem is just that Heroku is/was slow and expensive? Meaning this isn't necessarily the right or quote-unquote "best" approach to reclaiming the stack
zug_zug 2y ago
Seems like a cool premise. Though I guess people building things always want to convince you they are worth-it (sort of a conflict-of-interest), would like to read an unbiased 7-day migration to this.
jonstewart 2y ago
> Having started with Heroku, we have maintained a similar level of security Remember 2022? https://www.bleepingcomputer.com/news/security/heroku-admits-that-customer-credentials-were-stolen-in-cyberattack/ https://www.bleepingcomputer.com/news/security/heroku-admits...
PaulHoule 2y ago
What about “the rest of us” who don’t have time for Kube?
- notpushkin 2y ago
  If you know how to write a docker-compose.yml – Docker Swarm to the rescue! I’m making a nice PaaS-style thing on top of it: https://lunni.dev/ https://lunni.dev/ You can also use Kubernetes with compose files (e.g. with Kompose [1]; I plan to add support to Lunni, too). [1]: https://kompose.io/ https://kompose.io/
- tacker2000 2y ago
  Im using docker compose on every project I have, and it works fine. Of course, I dont have millions of users, but until then this is enough for me.
subarctic 2y ago
I wish _I_ had a business that was successful enough to justify multiple engineers working 7 months on porting our infrastructure from heroku to kubernetes
- bastawhiz 2y ago
  Knowing the prices and performance of Heroku (as a former customer) the effort probably paid for itself. Heroku is great for getting started but becomes untenably expensive very fast, and it's neither easy nor straightforward to break the vendor lock in when you decide to leave.
  - cpursley 2y ago
    Moving from Heroku to Render or Fly.io is very straight forward; it’s just containers.
    - dartos 2y ago
      Unless you relied on heroku build packs.
      - debarshri 2y ago
        Buildpacks is opensource too [1] [1] https://buildpacks.io/ https://buildpacks.io/
    - bastawhiz 2y ago
      If you use containers. If you're big enough for the cost savings to matter, you're probably also not looking for a service like Render or Fly. If your workload is really "just containers" you can save more with even managed container services from AWS or GCP.
      - OJFord 2y ago
        We are talking about moving from Heroku, I don't think being too needy for the likes of Fly is at all a given. (And people will way prematurely think they're too big or needy for x.)
notpushkin 2y ago
It looks like a nice Kubernetes setup! But I don’t see how this is comparable to something like Heroku – the complexity is way higher from what I see. If you’re looking for something simpler, try https://dokku.com/ https://dokku.com/ (the OG self-hosted Heroku) or https://lunni.dev/ https://lunni.dev/ (which I’ve been working on for a while, with a docker-compose based workflow instead). (I've also heard good things about coolify.io!)
Retr0id 2y ago
Based on the title alone, I thought this was going to be people up in arms about -fomit-frame-pointer being used by distros
seungwoolee518 2y ago
Most of the software should work Out-Of-The-Box, but the real problem is coming from hardware.
deisteve 2y ago
i got excited until i saw this was kubernetes. you most certainly do not need to add that layer of complexity. If I can serve 3 million users / month on a $40/month VPS with just Coolify, Postgres, Nginx, Django Gunicorn without Redis, RabbitMQ why should I use Kubernetes?
- itsthecourier 2y ago
  Got a bill from usd10k to usd0.5k a month by moving away from gcp to Kamal in ovh And 30% less latency
  - deisteve 2y ago
    thats 95% in savings!!!! bet you can squueze more with hetzner to ppl who disagree, what business justifies 18x'ing your operating costs? 9.5k USD can get you 3 senior engineers in Canada. 9 in India.
    - deznu 2y ago
      Senior Engineers cost ~$3k a month in Canada?? Seems far-fetched..
      - chaboud 2y ago
        We must have very different definitions of senior engineer from the GP, because I’d put the monthly cost of a senior engineer closer to $30k than $3k, even on a log scale. Employing people requires insurance, buildings, hardware, support, licenses, etc. There are lower cost locations, but I can’t think of a single market on earth where there is a supply of senior engineers that cost $3k/month. And I say this being familiar with costs in India, China, Poland, Spain, Mexico, Costa Rica, and at least a dozen other regions.
        littlestymaar 2y ago
        > even on a log scale. Log scale is just going to distort the picture in favor of your argument nor against it (10 is closer to 3 than to 30, but in log scale 10 becomes closer to 30) so I don't really understand why you're adding that here. Also having hired senior software engineers in Europe (France, Germany, Nederlands), if it cost you 30k a month in India or Poland, you're just being conned. “Hardware, support and license” are just bogus argument, as it's completely negligible unless you're doing exotic stuff requiring expensive licenses for your engineer. “Insurance” costs pretty much nothing in most of Europe because health insurance is mostly covered by the state, and “buildings” is mostly a self-inflicted wounds nowadays, especially since you'd get the better candidates if you supported full working from home. The original 3k is indeed way too low, even for a junior developer, but 30k is equally ridiculous really as you should never really spend more than half of that outside of the US.
strzibny 2y ago
It's good to see new projects. However most people shouldn't start with Kubernetes at all. If you don't need autoscaling, give Kamal[0] a go. It's the tool 37signals made to leave Kubernetes and cloud. Works super well with simple VMs. I also wrote a handbook[1] to get people started. [0] https://kamal-deploy.org https://kamal-deploy.org [1] https://kamalmanual.com/handbook/ https://kamalmanual.com/handbook/
- leohonexus 2y ago
  Bought both your books, they are awesome :)
- mplewis 2y ago
  I’m not going to trust a project like this – made by and for one company – with production workloads.
  - rcaught 2y ago
    hahaha, do you even realize what else this company makes?
- dbackeus 2y ago
  (Reclaim the Stack creator here) We don't do autoscaling. The main reason for Kubernetes for us was automation of monitoring / logs / alerting and highly available database deployments. 37signals has a dedicated operations team with more than 10 people. We have 0 dedicated operations people. We would not have been able to run our product with Kamal given our four nines uptime target. (that said, I do like Kamal, especially v2 seems to smooth out some edges, and I'm all for simple single server deployments)
rglover 2y ago
I made the mistake of falling for the k8s hype a few years back for running all of my indie hacker businesses. Big mistake. Overnight, the cluster config files I used were no longer supported by the k8s version DigitalOcean auto upgraded my cluster to and _boom_. Every single business was offline. Made the switch to some simple bash scripts for bootstrapping/monitoring/scaling and systemd for starting/restarting apps (nodejs). I'll never look back.
- eddd-ddde 2y ago
  So either digital ocean auto updates breaking versions. Or k8s doesn't do versioning correctly. Both very bad. Which was it?
  - poincaredisk 2y ago
    I assume the first one, but it's more complicated. K8s used to have a lot of features (included very important ones) in the "beta" namespace. There are no stability guarantees there, but everyone used them anyway. Over time they graduated to the "stable" namespace, and after some transitory period they were removed from the beta namespace. This broke old deployments, when admins ignored warnings for two or three major releases.
    - dmurray 2y ago
      It's an odd choice to break backwards compatibility by removing them from the beta namespace. Why not keep them available in both indefinitely?
      - pcthrowaway 2y ago
        Probably because the devs understandably can't account for every possible way people might be using it when shipping new features. But in my experience this means k8s is a bag of fiddly bits that requires some serious ops investments to be reliable for anything serious.
      - p_l 2y ago
pton_xd 2y ago
"The results were a 90% reduction in costs and a 30% improvement in performance." What's the scale of this service? How many machines are we talking here?
- internetter 2y ago
  Went from $~7500 to $520/m iirc from the presentation
Summerbud 2y ago
> The results were a 90% reduction in costs and a 30% improvement in performance. I am in a company with dedicated infra team and my CEO is a infra enthusiastic. He use terraform and k8s to build the company's infra. But the results are. - Every deployment take days, in my experience, I need to woke for 24 hr streak to make it work. - The infra is complicated to a level that quite hard to adjust And benefits wise, I can't even think about it. We don't have many users so the claimed scalability is not even there. I will strongly argue startup should not touch k8s until you have fair user base and retention. It's a nightmare to work with.
- tryauuum 2y ago
  ...but why? How many services the deployment requires?
- raziel2p 2y ago
  sounds like your CEO just isn't very good at setting up infra.
  - Summerbud 2y ago
    Maybe, that is one of the possibilities in my mind too.
- cultofmetatron 2y ago
  DAYS??? our infra takes 10 min usually with up to 45 min if we're doing some postgres maintenance stuff. People in a work context should stick to what they are good at.
ksajadi 2y ago
I’ve been building and deploying thousands of stacks on first Docker, then Mesos, then Swarm and now k8s. If I have learned one thing from it, it’s this: it’s all about the second day. There are so many tools that make it easy to build and deploy apps to your servers (with or without containers) and all of them showcase how easy it is to go from a cloud account to a fully deploy app. While their claims are true, what they don’t talk about is how to maintain the stack, after “reclaiming” it. Version changes, breaking changes, dependency changes and missing dependencies, disaster recovery plans, backups and restores, major shifts in requirements all add up to a large portion of your time. If you have that kind of team, budget or problem that deserves those, then more power to you.
- imiric 2y ago
  Agreed, but to be fair, those are general problems you would face with any architecture. At least with mainstream stacks you get the benefit of community support, and relying on approaches that someone else has figured out. Container-based stacks also have the benefit of homogeneizing your infrastructure, and giving you a common set of APIs and workflows to interact with. K8s et al are not a silver bullet, but at this point they're highly stable and understood pieces of infrastructure. It's much more painful to deviate from this and build things from scratch, deluding yourself that your approach can be simpler. For trivial and experimental workloads that may be the case, but for anything that requires a bit more sophistication these tools end up saving you resources in the long run.
- szundi 2y ago
  And what is your take on all those things that you tried? Some experience/examples would benefit us probably.
- wg0 2y ago
  This is absolutely true. I can count easily some 20+ components already. So this is not walk in the park with two willing developers to learn k8s. The underlying apps (Redis, ES) will have version upgrades. Their respective operators themselves would have version upgrades. Essential networking fabric (calico, funnel and such) would have upgrades. The underlying kubernetes itself would have version upgrades. The Talos Linux itself might need upgrades. Of all the above, any single upgrade might lead to infamous controller crash loop where pod starts and dies with little to no indication as to why? And that too no ordinary pod but a crucial pod part of some operator supposed to do the housekeeping for you. k8s is invented at Google and is more suitable in ZIRP world where money is cheap and to change the logo, you have seven designers on payroll discussing for eight months how nine different tones of brand coloring might convey ten different subliminal messages.
mvkel 2y ago
> 90% reduction in costs Curious what accounts are being attributed to said costs. Many new maintenance-related lines will be added, with only one (subscription) removed.
aliasxneo 2y ago
Since there are so many mixed comments here, I'll share my experience. Our startup started on day one with Kubernetes. It took me about six weeks to write the respective Terraform and manifests and combine them into a homogenous system. It's been smooth sailing for almost two years now. I'm starting to suspect the wide range of experiences has to do with engineering decisions. Nowadays, it's almost trivial to over-engineer a Kubernetes setup. In fact, with platform engineering becoming all the rage these days, I can't help but notice how over-engineered most reference architectures are for your average mid-sized company. Of course, that's probably by design (Humanitec sure enjoys the money), but it's all completely optional. I intentionally started with a dead-simple EKS setup: flat VPC with no crazy networking, simple EBS volumes for persistence, an ALB on the edge to cover ingress, and External Secrets to sync from AWS Secrets Manager. No service mesh, no fancy BPF shenanigans, just a cluster so simple that replicating to multiple environments was trivial. The great part is that because we've had such excellent stability, I've been able to slowly build out a custom platform that abstracts what little complexity there was (mostly around writing manifests). I'm not suggesting Kubernetes is for everyone, but the hate it tends to get on HN still continues to make me scratch my head to this day.
hintymad 2y ago
A trajectory question: Is there an acceptable solution to federate k8s clusters, or is there a such need? One thing that EC2 was really powerful is that a company can practically create as many clusters (ASGs) of as many nodes as needed, while k8s by default has this scale limit of 5000 nodes or so. I guess 5000 nodes will far from being enough for a large company that offers a single compute platform to its employees.
thih9 2y ago
> fully open source stack*. *) Except for Cloudflare Are there plans to address that too long term?
- dbackeus 2y ago
  Not from our point of view since Cloudflare's DDOS production and CDN is a crucial part of our architecture. That said, switching out cloudflared for a more traditional ingress like nginx etc would be straight forward. No parts of the RtS tooling as actually dependent on using Cloudflare for ingress in particular.
est 2y ago
> We spent 7 months building a Kubernetes based platform to replace Heroku for our SaaS product And heroku is based on LXC containers. I'd say it's almost the same thing.
jusomg 2y ago
Of course you reduced 90% of the cost. Most of these costs don't come from the software, but from the people and automation maintaining it. With that cost reduction you also removed monitoring of the platform, people oncall to fix issues that appear, upgrades, continuous improvements, etc. Who/What is going to be doing that on this new platform and how much does that cost? Now you need to maintain k8s, postgresql, elasticsearch, redis, secret managements, OSs, storage... These are complex systems that require people understanding how they internally work, how they scale and common pitfalls. Who is going to upgrade kubernetes when they release a new version that has breaking changes? What happens when Elasticsearch decides to splitbrain and your search stops working? When the DB goes down or you need to set up replication? What is monitoring replication lag? Or even simply things like disks being close to full? What is acting on that? I don't mean to say Heroku is fairly priced (I honestly have no idea) but this comparison is not apples to apples. You could have your team focused on your product before. Now you need people dedicated to work on this stuff.
- matus_congrady 2y ago
  Since DHH has been promoting the 'do-it-yourself' approach, many people have fallen for it. You're asking the right questions that only a few people know they need answers to. In my opinion, the closest thing to "reclaiming the stack" while still being a PaaS is to use a "deploy to your cloud account" PaaS provider. These services offer the convenience of a PaaS provider, yet allow you to "eject" to using the cloud provider on your own should your use case evolve. Example services include https://stacktape.com https://stacktape.com, https://flightcontrol.dev https://flightcontrol.dev, and https://www.withcoherence.com https://www.withcoherence.com. I'm also working on a PaaS comparison site at https://paascout.io https://paascout.io. Disclosure: I am a founder of Stacktape.
- dzikimarian 2y ago
  Sorry, but that's just ton of FUD. We run both private cloud and (for a few customers) AWS. Of course you have more maintenance on on-prem, but typical k8s update is maybe a few hours of work, when you know what you are doing. Also AWS is also, complex, also requires configuration and also generates alerts in the middle of the night. It's still a lot cheaper than managed service.
pwmtr 2y ago
Definitely interesting material. I realized, especially in last few years, there is an increased interest on moving away from propriety clouds/PaaS to K8s or even to bare metal, primarily driven by high prices and also interest of having more control. At Ubicloud, we are attacking the same problem, though from a different angle. We are building an open-source alternative to AWS. You can host it yourself or use our managed services (which are 3x-10x more affordable than comparable services). We already built some primitives such as VMs, PostgreSQL, private networking, load balancers and also working on K8s. I have a question to HN crowd; which primitives are required to run your workloads? It seems the OP's list consists of Postgres, Redis, Elasticsearch, Secret Manager, Logging/Monitoring, Ingress and Service Mesh. I wonder if this is representative of typical requirements to run HN crowd's workloads.
- evertheylen 2y ago
  Quite simple, I want to submit a Docker image, and have it accept HTTP requests at a certain domain, with easy horizontal/vertical scaling. I'm sure your Elastic Compute product is nice but I don't want to set it up myself (let alone run k8s on it). Quite like fly.io. PS: I like what you guys are doing, I'd subscribe to your mailing list if you had one! :)
b_shulha 2y ago
Who are your target audience? There are so many components in this system, so it would require a dev-ops team member just to keep it healthy. What are the advantages over the (free) managed k8s provided by DigitalOcean? --- Gosh, I'm so happy I was able to jump of the k8s hype train. This is not something SMBs should be using. Now I happily manage my fleet of services without large infra overhead via my own paas over Docker Swarm. :)
- b_shulha 2y ago
  Oh, thanks for asking. ;) It is a fair source (future Apache 2.0 License) PaaS. I provide a cloud option if you want to manage less and get extra features (soon - included backup space, uptime monitoring from multiple locations, etc) and, of course, you are free to self-host it for free and without any limitations by using a single installation script. ;) https://github.com/ptah-sh/ptah-server https://github.com/ptah-sh/ptah-server But anyway, I'm really curious to know the answers to the questions I have posted above. Thanks!
- KronisLV 2y ago
  > Gosh, I'm so happy I was able to jump of the k8s hype train. This is not something SMBs should be using. Now I happily manage my fleet of services without large infra overhead via my own paas over Docker Swarm. :) I mean, I also use Docker Swarm and it's pretty good, especially with Portainer. To me, the logical order of tools goes with scale a bit like this: Docker Compose --> Docker Swarm --> Hashicorp Nomad / Kubernetes (with maybe Podman variety of tools where needed) I've yet to see a company that really needs the latter group of options, but maybe that's because I work in a country that's on the smaller side of things. All that being said, however, both Nomad and some K8s distributions like K3s https://k3s.io/ https://k3s.io/ can be a fairly okay experience nowadays. It's just that it's also easy to end up with more complexity than you need. I wonder if it's going to be the meme about going full circle and me eventually just using shared hosting with PHP or something again, though so far containers feel like the "right" choice for shipping things reasonably quickly, while being in control of how resources are distributed.
  - b_shulha 2y ago
GaryNumanVevo 2y ago
Potential irony, this site isn't loading for me
sciurus 2y ago
> Replicas are used for high availability only, not load balancing (From https://reclaim-the-stack.com/docs/platform-components/ingress-cloudflared https://reclaim-the-stack.com/docs/platform-components/ingre...) An I reading this right that they built a k8s-based platform where by default they can't horizontally scale applications? This seems like a lot of complexity to develop and maintain if they're running applications that don't even need that.
- dbackeus 2y ago
  This documentation only pertains to the Cloudflared ingress servers, which can handle orders of magnitude more traffic than we actually get. So we have not had any need to look into load balancing of this part of the infrastructure. Our actual application servers can of course be horizontally scaled. That said, there is some kind of balancing across multiple cloudflared replicas. But when we measured the traffic Cloudflare sent ~80% of traffic to just one of the available replicas. We haven't looked into what the actual algorithm is. It may well be that load starts getting better distributed if we were to start hitting the upper limits of a single replica. Or it may be by design that the load balancing is crappy to provide incentive for Cloudflare customers to buy their dedicated Load Balancing product (https://developers.cloudflare.com/load-balancing/ https://developers.cloudflare.com/load-balancing/).
kh_hk 2y ago
> We spent 7 months building a Kubernetes based platform to replace Heroku for our SaaS product at mynewsdesk.com. I thought this was either a joke I was missing, or a rant about Kubernetes. It turned out it was neither, and now I am confused.
noop_joe 2y ago
Heroku and Reclaim are far from the only two options available. The appropriate choice depends entirely on the team's available expertise and the demands of the applications under development. There's a lot of disagreements pitting one solution against another. Even if one hosting solution were better than another, the problem is there are SO MANY solutions that exist on so many axis of tradeoffs, it's determine an appropriate solution (heroku, reclaim, etc) without consideration to its application and context of use. Heroku has all sorts of issues: super expensive, limited functionality, but if it happens to be what a developer team knows and works for their needs, heroku could save them lots of money even considering the high cost. The same is true for reclaim. _If_ you're familiar with all of the tooling, you could host an application with more functionality for less money than heroku.
Havoc 2y ago
Toying with self hosted k8s at home has taught me that it it’s the infra equivalent of happy path coding. Works grand until it blows up in your face for non obvious reasons That’s definitely mostly a skill issue on my end but still would make me very wary betting a startup on it
thesurlydev 2y ago
I was excited about this title until I read it's just another thing on top of Kubernetes. To me, Kubernetes is part of the problem. Can we reduce the complexity that Kubernetes brings and still have nice things?

Reclaim the Stack

496 points by dustedcodes 2y ago

sph 2y ago

"Join the Discord server"? Who's the audience of this project?

mre 2y ago
Genuinely curious, what's wrong with that? Did you expect a different platform like Slack?
- fragmede 2y ago
  it would be better at the bottom of the first documentation page, after the reader has a better idea of what this is
- callalex 2y ago
  Locking knowledge behind something that isn’t publicly searchable or archivable works fine in the short term but what happens when Discord/Slack/whatever gears up for an IPO and limits all chat history to 1 week unless you pay up (oh and now you have a bunch of valuable knowledge stored up their with no migration tool so your only options are “pay up” or lose the knowledge).
  - halfcat 2y ago
    What’s recommended here? Self-hosted Discourse?
    - heavyset_go 2y ago
      Matrix and a wiki would solve the community and knowledge base issues.
      - mplewis 2y ago
        Matrix has severe UX issues which drastically limit the community willing to use it on a regular basis.
        Arathorn 2y ago
        historically, yes. matrix 2.0 (due in two weeks) aims to fix this.
    - tacker2000 2y ago
      Github issues or discussions. Or some other kind of forum like Discourse as you mentioned
  - Kiro 2y ago
    No-one complained when projects had IRC channels/servers, which are even worse since they have no history at all.
    - okasaki 2y ago
      All IRC clients have local plain text logging and putting a .txt on a web server is trivial.
      - Kiro 2y ago
        Local logging doesn't help much for searchability when you're new and it requires you to be online 24/7. Anyway, that's beside the point. Even if IRC had built-in server history it still has the same problems but I never saw people being outraged about it.
    - Gormo 2y ago
      Good projects still do rely on IRC -- Libera.chat is full of proper channels -- and logging bots are ubiquitous.
      - deleted 2y ago
        [deleted]
      - Kiro 2y ago
        And you never hear anyone complaining about those. "Locking knowledge" was never an argument before and it's not now.
    - callalex 2y ago
      At least people treat IRC as ephemeral and place all documentation elsewhere. People are writing whole wikis inside of Discord that are not publicly searchable.
- Gormo 2y ago
  There's a whole FOSS ecosystem of chat/collaboration applications, like Mattermost and Zulip; there's Matrix for a federated solution, and tried-and-true options like IRC. For something called "Reclaim the Stack" to lock discussion into someone else's proprietary walled garden is quite ironic.
deleted 2y ago
[deleted]
mrits 2y ago
People that don't like wasting money?
- hobs 2y ago
  Not capturing the information and being able to use it in the future is a huge opportunity cost, and idling on discord pays no bills.
- Gormo 2y ago
  Wasting money on... better solutions that are also free?
  - mrits 2y ago
    There is a difference between wasting and spending money.
tacker2000 2y ago
Also noticed this. Everytime I see a project using discord as main communication tool it makes me think about the “fitness” of the project in the long run. Discord is NOT a benefit. Its not publicly searchable and the chat format is just not suitable to a knowledge base or support based format. Forums are much better in that regard.
- KronisLV 2y ago
  > Discord is NOT a benefit. Its not publicly searchable and the chat format is just not suitable to a knowledge base or support based format. I don't think people who choose Discord necessarily care about that. Discord is where the people are, so that's where they go. It also costs close to nothing to setup a server and since it has a lower barrier of entry than hosting your own forum, it's deemed good enough. That said, modern forum software like Discourse https://www.discourse.org/ https://www.discourse.org/ or Flarum https://flarum.org/ https://flarum.org/ can be pretty good, though I still miss phpBB.
  - Gormo 2y ago
    > Discord is where the people are, so that's where they go. That doesn't sound right. Each Discord community is its own separate space -- you still need people to join your specific community regardless of whether it is hosted on Discord or something better. > though I still miss phpBB. It hasn't gone away -- the last release was on August 29th, so this is still very much a viable option.
    - dbackeus 2y ago
      It's all in one app and the app has a ton of users. Anyone running the app can join any server with a click of a button. There are no separate accounts required to join different communities. So communities being separate "spaces" doesn't create any meaningful friction with regards to adoption.

evantahler 2y ago

Porter (https://www.porter.run/ https://www.porter.run/) is a great product in the same vein (e.g. turn K8s into a dev-friendly Heroku-like PASS). How does this compare?

mikeortman 2y ago
I think the very concept of this is to open source a common stack, instead of relying on a middleman like Porter, which also costs a TON of money at business tier

fragmede 2y ago

having your tool be a single letter, k, seems rather presumptuous.

rjbwork 2y ago
Especially given K is already the name of an APL derivative.
dbackeus 2y ago
I suppose it is. But no actual users of the platform has had any complaints about it.

chrisweekly 2y ago

This looks great! Thank you for sharing, @dustedcodes. I might set up a playground to gain more hands-on experience w/ the relevant significant parts (k8s, argocd, talos) all of which have been on my radar for some time... Also, the docs look great. I love the Architecture Decision Records (bullet-point pros/cons/context)...

andrewstuart 2y ago

How can a NewsDesk application need kubernetes? Wouldn't a single machine and a backup machine do the job?

andrewstuart 2y ago
I just looked it up - its because they run Ruby On Rails.
- zymhan 2y ago
  and so what?
  - andrewstuart 2y ago
    Ruby On Rails is well known for not being at the fast end of the spectrum, so it needs lots of machines, and lots of machines gives reason to user Kubernetes. A NewsDesk application written in something compiled for example golang would be much faster and likely could run on a single server. The benefit of single server being you don't need kubernetes and can spend that development resource on developing application features.
    - jakelazaroff 2y ago
      I personally prefer Go to Rails, but let’s be real here: the market cap of Rails is probably, like, a hundred times the market cap of Go.
      - sien 2y ago
        No doubt with Github, AirBnB, Shopify and other big sites RoR is bigger for the front end. But now if lots of those sites are running on K8s with Argo CD or something or on a cloud platform where the infrastructure is provisioned with Terraform Go is supporting a great deal of things but it's far less visible.
      - maxk42 2y ago

appplication 2y ago

This sounds great, I’ll be building our prod infra stack and deploying to cloud for the first time here in the next few weeks, so this is timely. It’s nice seeing some OSS-based tooling around k8s. I know it’s a favorite refrain that “k8s is unnecessary/too complex, you don’t need it” for many folks getting started with their deployments, but I already know and use it in my day job, so it feels like a pretty natural choice.

briandear 2y ago
The K8s is unnecessary meme is perpetuated by people that don’t understand it.
- actionfromafar 2y ago
  True, but also, sometimes it’s not needed.
  - jauntywundrkind 2y ago
    Sometimes it just feels good wearing a fig leaf around my groin, weilding a mid sized log as a crude club, & running through the jungle. You might not need it is the kernel of doubt that can undermine any reasonable option. And it suggests nothing. Sure, you can go write your own kernel! You can make your own database! You might not need to use good well known proven technology that people understand and can learn about online! You can do it yourself! Or cobble together some alternate lesser special stack that just you have distilled out. We don't need civilization. We can go it alone & do our own thing, leave behind shared frames of references. But damn, it just seems so absurdly inadvisable, and it feels so overblown the fear uncertainty & doubt telling us Kubernetes is hard and bad and too much. This article does certainly lend credence to the idea that Kubernetes is complex, but there's so many simpler starting places that will take many teams very far.
    - samatman 2y ago
      Somehow kubernetes and civilization just aren't in the same category of salience to me. Like I think it's reasonable to say that kubernetes is optional in a way which civilization isn't. Like maybe one of those things is more important. than, the other

thetopher 2y ago

“Our basic philosophy when it comes to security is that we can trust our developers and that we can trust the private network within the cluster.” This is not my area of expertise. Does it add a significant amount of complexity to configure this kind of system in a way that doesn’t require trusting the network? Where are the pain points?

zymhan 2y ago
It requires encrypting all network traffic, either with something like TLS, or IPSec VPN.
umvi 2y ago
Implementing "Zero Trust" architectures are definitely more onerous to deal with for everyone involved (both devs and customers, if on prem). Just Google "zero trust architecture" to find examples. A lot more work (and therefore $) to setup and maintain, but also better security since now breaching network perimeter is no longer enough to pwn everything inside said network.
nilsherzig 2y ago
"SSL added and removed here :^)"
callalex 2y ago
The top pain point is that it requires setting up SSL certificate infrastructure and having to store and distribute those certs around in a secure way. The secondary effects are entirely dependent on how your microservices talk to their dependencies. Are they already talking to some local proxy that handles load balancing and service discovery? If so, then you can bolt on ssl termination at that layer. If not, and your microservice is using dns and making http requests directly to other services, it’s a game of whack-a-mole modifying all of your software to talk to a local “sidecar”; or you have to configure every service to start doing the SSL validation which can explode in complexity when you end up dealing with a bunch of different languages and libraries. None of it is impossible by any means, and many companies/stacks do all of this successfully, but it’s all work that doesn’t add features, can lead to performance degradation, and is a hard sell to get funding/time for because your boss’s boss almost certainly trusts the cloud provider to handle such things at their network layer unless they have very specific security requirements and knowledge.

fourseventy 2y ago

In my experience you can get pretty far with just a handful of vms and some bash scripts. At least double digit million ARR. Less is more when it comes to devops tooling imo.

cloudking 2y ago
+1 or just use App Engine, deploy your app and scale
- cglace 2y ago
  App engine deploys are soooo slow. I liked cloud run a lot more.
sanderjd 2y ago
Of course you can get away with that if your metric is revenue. (I think Blippi makes about that much with, I suspect, nary a VM in sight! The question is what you're doing with your infrastructure, not how much revenue you're making. Some things have higher return to "devops" and others have less.
lolinder 2y ago
> you can get pretty far with just a handful of vms and some bash scripts. At least double digit million ARR. Using ARR as the measurement for how far you can scale devops practices is weird to me. Double-digit million ARR might be a few hundred accounts if you're doing B2B, and double-digit million MAUs if you're doing an ad-funded social platform. Depending on how much software is involved your product could be built by a team of anywhere from 1-50 developers. If you're a one-developer B2B company handling 1-3 requests per second you wouldn't even need more than one VM except maybe as redundancy. But if you're the fifty-developer company that's building something beyond simple CRUD, there are a lot of perks that come with a full-fledged control plane that would almost certainly be worth the added cost and complexity.
- marcosdumay 2y ago
  > double-digit million MAUs I was about to make a similar point, but you made the math, and it's holding-up for the GP's side. You can push vms and direct to ssh synchronization up to double-digit million MAU (unless you are using stuff like persistent web-sockets). It won't be pretty, but you can get that far.

AbuAssar 2y ago

How does this compare to dokku (https://dokku.com/ https://dokku.com/)?

dbackeus 2y ago
Main difference is that Dokku is a simple single server platform, geared mostly toward hobby projects. Reclaim the Stack provides a fully highly available multi node platform to host large scale SaaS applications aiming for four nines of uptime.

mikeortman 2y ago

I'm glad we are starting to lean into cloud-agnostic or building back the on-prem/dedicated systems again.

airstrike 2y ago

> We spent 7 months building a Kubernetes based platform to replace Heroku for our SaaS product at mynewsdesk.com. The results were a 90% reduction in costs and a 30% improvement in performance. I don't mean to sound dismissive, but maybe the problem is just that Heroku is/was slow and expensive? Meaning this isn't necessarily the right or quote-unquote "best" approach to reclaiming the stack

zug_zug 2y ago

Seems like a cool premise. Though I guess people building things always want to convince you they are worth-it (sort of a conflict-of-interest), would like to read an unbiased 7-day migration to this.

jonstewart 2y ago

> Having started with Heroku, we have maintained a similar level of security Remember 2022? https://www.bleepingcomputer.com/news/security/heroku-admits-that-customer-credentials-were-stolen-in-cyberattack/ https://www.bleepingcomputer.com/news/security/heroku-admits...

PaulHoule 2y ago

What about “the rest of us” who don’t have time for Kube?

notpushkin 2y ago
If you know how to write a docker-compose.yml – Docker Swarm to the rescue! I’m making a nice PaaS-style thing on top of it: https://lunni.dev/ https://lunni.dev/ You can also use Kubernetes with compose files (e.g. with Kompose [1]; I plan to add support to Lunni, too). [1]: https://kompose.io/ https://kompose.io/
tacker2000 2y ago
Im using docker compose on every project I have, and it works fine. Of course, I dont have millions of users, but until then this is enough for me.

subarctic 2y ago

I wish _I_ had a business that was successful enough to justify multiple engineers working 7 months on porting our infrastructure from heroku to kubernetes

bastawhiz 2y ago
Knowing the prices and performance of Heroku (as a former customer) the effort probably paid for itself. Heroku is great for getting started but becomes untenably expensive very fast, and it's neither easy nor straightforward to break the vendor lock in when you decide to leave.
- cpursley 2y ago
  Moving from Heroku to Render or Fly.io is very straight forward; it’s just containers.
  - dartos 2y ago
    Unless you relied on heroku build packs.
    - debarshri 2y ago
      Buildpacks is opensource too [1] [1] https://buildpacks.io/ https://buildpacks.io/
  - bastawhiz 2y ago
    If you use containers. If you're big enough for the cost savings to matter, you're probably also not looking for a service like Render or Fly. If your workload is really "just containers" you can save more with even managed container services from AWS or GCP.
    - OJFord 2y ago
      We are talking about moving from Heroku, I don't think being too needy for the likes of Fly is at all a given. (And people will way prematurely think they're too big or needy for x.)

notpushkin 2y ago

It looks like a nice Kubernetes setup! But I don’t see how this is comparable to something like Heroku – the complexity is way higher from what I see. If you’re looking for something simpler, try https://dokku.com/ https://dokku.com/ (the OG self-hosted Heroku) or https://lunni.dev/ https://lunni.dev/ (which I’ve been working on for a while, with a docker-compose based workflow instead). (I've also heard good things about coolify.io!)

Retr0id 2y ago

Based on the title alone, I thought this was going to be people up in arms about -fomit-frame-pointer being used by distros

seungwoolee518 2y ago

Most of the software should work Out-Of-The-Box, but the real problem is coming from hardware.

deisteve 2y ago

i got excited until i saw this was kubernetes. you most certainly do not need to add that layer of complexity. If I can serve 3 million users / month on a $40/month VPS with just Coolify, Postgres, Nginx, Django Gunicorn without Redis, RabbitMQ why should I use Kubernetes?

itsthecourier 2y ago
Got a bill from usd10k to usd0.5k a month by moving away from gcp to Kamal in ovh And 30% less latency
- deisteve 2y ago
  thats 95% in savings!!!! bet you can squueze more with hetzner to ppl who disagree, what business justifies 18x'ing your operating costs? 9.5k USD can get you 3 senior engineers in Canada. 9 in India.
  - deznu 2y ago
    Senior Engineers cost ~$3k a month in Canada?? Seems far-fetched..
    - chaboud 2y ago
      We must have very different definitions of senior engineer from the GP, because I’d put the monthly cost of a senior engineer closer to $30k than $3k, even on a log scale. Employing people requires insurance, buildings, hardware, support, licenses, etc. There are lower cost locations, but I can’t think of a single market on earth where there is a supply of senior engineers that cost $3k/month. And I say this being familiar with costs in India, China, Poland, Spain, Mexico, Costa Rica, and at least a dozen other regions.
      - littlestymaar 2y ago
        > even on a log scale. Log scale is just going to distort the picture in favor of your argument nor against it (10 is closer to 3 than to 30, but in log scale 10 becomes closer to 30) so I don't really understand why you're adding that here. Also having hired senior software engineers in Europe (France, Germany, Nederlands), if it cost you 30k a month in India or Poland, you're just being conned. “Hardware, support and license” are just bogus argument, as it's completely negligible unless you're doing exotic stuff requiring expensive licenses for your engineer. “Insurance” costs pretty much nothing in most of Europe because health insurance is mostly covered by the state, and “buildings” is mostly a self-inflicted wounds nowadays, especially since you'd get the better candidates if you supported full working from home. The original 3k is indeed way too low, even for a junior developer, but 30k is equally ridiculous really as you should never really spend more than half of that outside of the US.

strzibny 2y ago

It's good to see new projects. However most people shouldn't start with Kubernetes at all. If you don't need autoscaling, give Kamal[0] a go. It's the tool 37signals made to leave Kubernetes and cloud. Works super well with simple VMs. I also wrote a handbook[1] to get people started. [0] https://kamal-deploy.org https://kamal-deploy.org [1] https://kamalmanual.com/handbook/ https://kamalmanual.com/handbook/

leohonexus 2y ago
Bought both your books, they are awesome :)
mplewis 2y ago
I’m not going to trust a project like this – made by and for one company – with production workloads.
- rcaught 2y ago
  hahaha, do you even realize what else this company makes?
dbackeus 2y ago
(Reclaim the Stack creator here) We don't do autoscaling. The main reason for Kubernetes for us was automation of monitoring / logs / alerting and highly available database deployments. 37signals has a dedicated operations team with more than 10 people. We have 0 dedicated operations people. We would not have been able to run our product with Kamal given our four nines uptime target. (that said, I do like Kamal, especially v2 seems to smooth out some edges, and I'm all for simple single server deployments)

rglover 2y ago

I made the mistake of falling for the k8s hype a few years back for running all of my indie hacker businesses. Big mistake. Overnight, the cluster config files I used were no longer supported by the k8s version DigitalOcean auto upgraded my cluster to and _boom_. Every single business was offline. Made the switch to some simple bash scripts for bootstrapping/monitoring/scaling and systemd for starting/restarting apps (nodejs). I'll never look back.

eddd-ddde 2y ago
So either digital ocean auto updates breaking versions. Or k8s doesn't do versioning correctly. Both very bad. Which was it?
- poincaredisk 2y ago
  I assume the first one, but it's more complicated. K8s used to have a lot of features (included very important ones) in the "beta" namespace. There are no stability guarantees there, but everyone used them anyway. Over time they graduated to the "stable" namespace, and after some transitory period they were removed from the beta namespace. This broke old deployments, when admins ignored warnings for two or three major releases.
  - dmurray 2y ago
    It's an odd choice to break backwards compatibility by removing them from the beta namespace. Why not keep them available in both indefinitely?
    - pcthrowaway 2y ago
      Probably because the devs understandably can't account for every possible way people might be using it when shipping new features. But in my experience this means k8s is a bag of fiddly bits that requires some serious ops investments to be reliable for anything serious.
    - p_l 2y ago

pton_xd 2y ago

"The results were a 90% reduction in costs and a 30% improvement in performance." What's the scale of this service? How many machines are we talking here?

internetter 2y ago
Went from $~7500 to $520/m iirc from the presentation

Summerbud 2y ago

> The results were a 90% reduction in costs and a 30% improvement in performance. I am in a company with dedicated infra team and my CEO is a infra enthusiastic. He use terraform and k8s to build the company's infra. But the results are. - Every deployment take days, in my experience, I need to woke for 24 hr streak to make it work. - The infra is complicated to a level that quite hard to adjust And benefits wise, I can't even think about it. We don't have many users so the claimed scalability is not even there. I will strongly argue startup should not touch k8s until you have fair user base and retention. It's a nightmare to work with.

tryauuum 2y ago
...but why? How many services the deployment requires?
raziel2p 2y ago
sounds like your CEO just isn't very good at setting up infra.
- Summerbud 2y ago
  Maybe, that is one of the possibilities in my mind too.
cultofmetatron 2y ago
DAYS??? our infra takes 10 min usually with up to 45 min if we're doing some postgres maintenance stuff. People in a work context should stick to what they are good at.

ksajadi 2y ago

I’ve been building and deploying thousands of stacks on first Docker, then Mesos, then Swarm and now k8s. If I have learned one thing from it, it’s this: it’s all about the second day. There are so many tools that make it easy to build and deploy apps to your servers (with or without containers) and all of them showcase how easy it is to go from a cloud account to a fully deploy app. While their claims are true, what they don’t talk about is how to maintain the stack, after “reclaiming” it. Version changes, breaking changes, dependency changes and missing dependencies, disaster recovery plans, backups and restores, major shifts in requirements all add up to a large portion of your time. If you have that kind of team, budget or problem that deserves those, then more power to you.

imiric 2y ago
Agreed, but to be fair, those are general problems you would face with any architecture. At least with mainstream stacks you get the benefit of community support, and relying on approaches that someone else has figured out. Container-based stacks also have the benefit of homogeneizing your infrastructure, and giving you a common set of APIs and workflows to interact with. K8s et al are not a silver bullet, but at this point they're highly stable and understood pieces of infrastructure. It's much more painful to deviate from this and build things from scratch, deluding yourself that your approach can be simpler. For trivial and experimental workloads that may be the case, but for anything that requires a bit more sophistication these tools end up saving you resources in the long run.
szundi 2y ago
And what is your take on all those things that you tried? Some experience/examples would benefit us probably.
wg0 2y ago
This is absolutely true. I can count easily some 20+ components already. So this is not walk in the park with two willing developers to learn k8s. The underlying apps (Redis, ES) will have version upgrades. Their respective operators themselves would have version upgrades. Essential networking fabric (calico, funnel and such) would have upgrades. The underlying kubernetes itself would have version upgrades. The Talos Linux itself might need upgrades. Of all the above, any single upgrade might lead to infamous controller crash loop where pod starts and dies with little to no indication as to why? And that too no ordinary pod but a crucial pod part of some operator supposed to do the housekeeping for you. k8s is invented at Google and is more suitable in ZIRP world where money is cheap and to change the logo, you have seven designers on payroll discussing for eight months how nine different tones of brand coloring might convey ten different subliminal messages.

mvkel 2y ago

> 90% reduction in costs Curious what accounts are being attributed to said costs. Many new maintenance-related lines will be added, with only one (subscription) removed.

aliasxneo 2y ago

Since there are so many mixed comments here, I'll share my experience. Our startup started on day one with Kubernetes. It took me about six weeks to write the respective Terraform and manifests and combine them into a homogenous system. It's been smooth sailing for almost two years now. I'm starting to suspect the wide range of experiences has to do with engineering decisions. Nowadays, it's almost trivial to over-engineer a Kubernetes setup. In fact, with platform engineering becoming all the rage these days, I can't help but notice how over-engineered most reference architectures are for your average mid-sized company. Of course, that's probably by design (Humanitec sure enjoys the money), but it's all completely optional. I intentionally started with a dead-simple EKS setup: flat VPC with no crazy networking, simple EBS volumes for persistence, an ALB on the edge to cover ingress, and External Secrets to sync from AWS Secrets Manager. No service mesh, no fancy BPF shenanigans, just a cluster so simple that replicating to multiple environments was trivial. The great part is that because we've had such excellent stability, I've been able to slowly build out a custom platform that abstracts what little complexity there was (mostly around writing manifests). I'm not suggesting Kubernetes is for everyone, but the hate it tends to get on HN still continues to make me scratch my head to this day.

hintymad 2y ago

A trajectory question: Is there an acceptable solution to federate k8s clusters, or is there a such need? One thing that EC2 was really powerful is that a company can practically create as many clusters (ASGs) of as many nodes as needed, while k8s by default has this scale limit of 5000 nodes or so. I guess 5000 nodes will far from being enough for a large company that offers a single compute platform to its employees.

thih9 2y ago

> fully open source stack*. *) Except for Cloudflare Are there plans to address that too long term?

dbackeus 2y ago
Not from our point of view since Cloudflare's DDOS production and CDN is a crucial part of our architecture. That said, switching out cloudflared for a more traditional ingress like nginx etc would be straight forward. No parts of the RtS tooling as actually dependent on using Cloudflare for ingress in particular.

est 2y ago

> We spent 7 months building a Kubernetes based platform to replace Heroku for our SaaS product And heroku is based on LXC containers. I'd say it's almost the same thing.

jusomg 2y ago

Of course you reduced 90% of the cost. Most of these costs don't come from the software, but from the people and automation maintaining it. With that cost reduction you also removed monitoring of the platform, people oncall to fix issues that appear, upgrades, continuous improvements, etc. Who/What is going to be doing that on this new platform and how much does that cost? Now you need to maintain k8s, postgresql, elasticsearch, redis, secret managements, OSs, storage... These are complex systems that require people understanding how they internally work, how they scale and common pitfalls. Who is going to upgrade kubernetes when they release a new version that has breaking changes? What happens when Elasticsearch decides to splitbrain and your search stops working? When the DB goes down or you need to set up replication? What is monitoring replication lag? Or even simply things like disks being close to full? What is acting on that? I don't mean to say Heroku is fairly priced (I honestly have no idea) but this comparison is not apples to apples. You could have your team focused on your product before. Now you need people dedicated to work on this stuff.

matus_congrady 2y ago
Since DHH has been promoting the 'do-it-yourself' approach, many people have fallen for it. You're asking the right questions that only a few people know they need answers to. In my opinion, the closest thing to "reclaiming the stack" while still being a PaaS is to use a "deploy to your cloud account" PaaS provider. These services offer the convenience of a PaaS provider, yet allow you to "eject" to using the cloud provider on your own should your use case evolve. Example services include https://stacktape.com https://stacktape.com, https://flightcontrol.dev https://flightcontrol.dev, and https://www.withcoherence.com https://www.withcoherence.com. I'm also working on a PaaS comparison site at https://paascout.io https://paascout.io. Disclosure: I am a founder of Stacktape.
dzikimarian 2y ago
Sorry, but that's just ton of FUD. We run both private cloud and (for a few customers) AWS. Of course you have more maintenance on on-prem, but typical k8s update is maybe a few hours of work, when you know what you are doing. Also AWS is also, complex, also requires configuration and also generates alerts in the middle of the night. It's still a lot cheaper than managed service.

pwmtr 2y ago

Definitely interesting material. I realized, especially in last few years, there is an increased interest on moving away from propriety clouds/PaaS to K8s or even to bare metal, primarily driven by high prices and also interest of having more control. At Ubicloud, we are attacking the same problem, though from a different angle. We are building an open-source alternative to AWS. You can host it yourself or use our managed services (which are 3x-10x more affordable than comparable services). We already built some primitives such as VMs, PostgreSQL, private networking, load balancers and also working on K8s. I have a question to HN crowd; which primitives are required to run your workloads? It seems the OP's list consists of Postgres, Redis, Elasticsearch, Secret Manager, Logging/Monitoring, Ingress and Service Mesh. I wonder if this is representative of typical requirements to run HN crowd's workloads.

evertheylen 2y ago
Quite simple, I want to submit a Docker image, and have it accept HTTP requests at a certain domain, with easy horizontal/vertical scaling. I'm sure your Elastic Compute product is nice but I don't want to set it up myself (let alone run k8s on it). Quite like fly.io. PS: I like what you guys are doing, I'd subscribe to your mailing list if you had one! :)

b_shulha 2y ago

Who are your target audience? There are so many components in this system, so it would require a dev-ops team member just to keep it healthy. What are the advantages over the (free) managed k8s provided by DigitalOcean? --- Gosh, I'm so happy I was able to jump of the k8s hype train. This is not something SMBs should be using. Now I happily manage my fleet of services without large infra overhead via my own paas over Docker Swarm. :)

b_shulha 2y ago
Oh, thanks for asking. ;) It is a fair source (future Apache 2.0 License) PaaS. I provide a cloud option if you want to manage less and get extra features (soon - included backup space, uptime monitoring from multiple locations, etc) and, of course, you are free to self-host it for free and without any limitations by using a single installation script. ;) https://github.com/ptah-sh/ptah-server https://github.com/ptah-sh/ptah-server But anyway, I'm really curious to know the answers to the questions I have posted above. Thanks!
KronisLV 2y ago
> Gosh, I'm so happy I was able to jump of the k8s hype train. This is not something SMBs should be using. Now I happily manage my fleet of services without large infra overhead via my own paas over Docker Swarm. :) I mean, I also use Docker Swarm and it's pretty good, especially with Portainer. To me, the logical order of tools goes with scale a bit like this: Docker Compose --> Docker Swarm --> Hashicorp Nomad / Kubernetes (with maybe Podman variety of tools where needed) I've yet to see a company that really needs the latter group of options, but maybe that's because I work in a country that's on the smaller side of things. All that being said, however, both Nomad and some K8s distributions like K3s https://k3s.io/ https://k3s.io/ can be a fairly okay experience nowadays. It's just that it's also easy to end up with more complexity than you need. I wonder if it's going to be the meme about going full circle and me eventually just using shared hosting with PHP or something again, though so far containers feel like the "right" choice for shipping things reasonably quickly, while being in control of how resources are distributed.
- b_shulha 2y ago

GaryNumanVevo 2y ago

Potential irony, this site isn't loading for me

sciurus 2y ago

> Replicas are used for high availability only, not load balancing (From https://reclaim-the-stack.com/docs/platform-components/ingress-cloudflared https://reclaim-the-stack.com/docs/platform-components/ingre...) An I reading this right that they built a k8s-based platform where by default they can't horizontally scale applications? This seems like a lot of complexity to develop and maintain if they're running applications that don't even need that.

dbackeus 2y ago
This documentation only pertains to the Cloudflared ingress servers, which can handle orders of magnitude more traffic than we actually get. So we have not had any need to look into load balancing of this part of the infrastructure. Our actual application servers can of course be horizontally scaled. That said, there is some kind of balancing across multiple cloudflared replicas. But when we measured the traffic Cloudflare sent ~80% of traffic to just one of the available replicas. We haven't looked into what the actual algorithm is. It may well be that load starts getting better distributed if we were to start hitting the upper limits of a single replica. Or it may be by design that the load balancing is crappy to provide incentive for Cloudflare customers to buy their dedicated Load Balancing product (https://developers.cloudflare.com/load-balancing/ https://developers.cloudflare.com/load-balancing/).

kh_hk 2y ago

> We spent 7 months building a Kubernetes based platform to replace Heroku for our SaaS product at mynewsdesk.com. I thought this was either a joke I was missing, or a rant about Kubernetes. It turned out it was neither, and now I am confused.

noop_joe 2y ago

Heroku and Reclaim are far from the only two options available. The appropriate choice depends entirely on the team's available expertise and the demands of the applications under development. There's a lot of disagreements pitting one solution against another. Even if one hosting solution were better than another, the problem is there are SO MANY solutions that exist on so many axis of tradeoffs, it's determine an appropriate solution (heroku, reclaim, etc) without consideration to its application and context of use. Heroku has all sorts of issues: super expensive, limited functionality, but if it happens to be what a developer team knows and works for their needs, heroku could save them lots of money even considering the high cost. The same is true for reclaim. _If_ you're familiar with all of the tooling, you could host an application with more functionality for less money than heroku.

Havoc 2y ago

Toying with self hosted k8s at home has taught me that it it’s the infra equivalent of happy path coding. Works grand until it blows up in your face for non obvious reasons That’s definitely mostly a skill issue on my end but still would make me very wary betting a startup on it

thesurlydev 2y ago

I was excited about this title until I read it's just another thing on top of Kubernetes. To me, Kubernetes is part of the problem. Can we reduce the complexity that Kubernetes brings and still have nice things?