60 ms·

De-cloud and de-k8s – bringing our apps back home

• vcryan 4y ago
  I have to imagine part of the reason they need to run so many servers is because they are running Ruby. The same application on say, Elixir, probably would require less hardware, reducing the cost of ECS or similar.
• zainhoda 4y ago
  Omg, they could have literally just solved all of this by choosing Cloud Run (a thin wrapper on top of Knative) rather than running GKE directly.
• acedTrex 4y ago
  This seems like an application/stack that didn't have a valid need for k8s in the first place. Don't just use K8s because its what people say you should do. evaluate the pros and the VERY real cons and make an informed decision.
  • bdcravens 4y ago
    That's why we've had good results with ECS. Feels like 80% of the result for 20% of the effort, and I haven't found our use cases needing that missing 20%.
    • sgarland 4y ago
      With EC2 launch types, probably. Setting up ECS for Fargate with proper IaC templates/modules isn't much easier than EKS, IMO.
      • bdcravens 4y ago
        We use Fargate, and what we launch is tightly coupled to our application (background jobs spin down and spin up tasks via the SDK) so for now, we aren't doing anything with IaC, other than CI deployment.
      • brigadier132 4y ago
        On the Google cloud side, using Google cloud build with cloud run with automatic CI/CD is very straightforward. I setup automated builds and deploys for staging in 2 hours. For production I set it up to track tagged branches matching a regex.
      • Aeolun 4y ago
        Mostly because CF and CDK have spawned from the deepest pits of hell. It’s ok when using terraform, and downright pleasant when using Pulumi.
      • jamesfinlayson 4y ago
        When I had to set up ECS with Fargate using CloudFormation the documentation was certainly lacking (in late 2019 I think it was). Now that it's working it's been pretty low maintenance.
        • bdcravens 4y ago
          It has definitely gotten better over time, but we tend to do a lot of stuff ad-hoc that finds its way into production lol, so we aren't yet relying on any infra as code.
      • josevalerio 4y ago
        I recommend everyone take a look at ECS patterns. This is incredibly easy in $CURRENT_YEAR. Give it a dockerfile / image and CDK deploy: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedFargateService.html https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_...
  • birdyrooster 4y ago
    “Need” Eh, I do it because it’s awesome for a single box or thousands. Single sign on, mTLS everywhere, cert-manager, BGP or L2 VIPs to any pod, etc and I can expand horizontally as needed. It’s the best for an at home lab. I pity the people who only use Proxmox.
    • klabb3 4y ago
      Even k3s used 500MB and 2-5% CPU for a single server node with 0 user space pods. That seems incredibly bloated to me.
• guilhas 4y ago
  How easier is mrsk vs k3s?
  • bdcravens 4y ago
    I'm not sure if anyone other than 37Signals is using it at scale yet, so you may get a better idea by looking at the docs yourself. https://github.com/mrsked/mrsk https://github.com/mrsked/mrsk
    • ilyt 4y ago
      Oh, YAML-based DSL to deploy stuff, how original! Now we only need template-based generator for those YAMLs and we will have all worst practices for orchestration right here, just like k8s + helm
  • turtlebits 4y ago
    Looks like an apple vs oranges comparison. They seem to have a low number of distinct services, so there isn't a real need for k3s/k8s (ie orchestration), on the other hand, they need config management.
• mike1o1 4y ago
  To me the really surprising thing is that they still use Capistrano for deploying basecamp!
  • camsjams 4y ago
    Maybe because Capistrano is written in Ruby and the language matches their internal products? That was my only guess.
    • mbreese 4y ago
      I was guessing that they kept using Capistrano because it still worked. No need to change something that’s working… (Somewhat of an ironic comment when talking about an article about ditching K8s…)
  • NetOpWibby 4y ago
    That's a name I haven't heard since I was deploying my Jekyll blog during my apprenticeship at Thoughtbot. Wow.
• ocdtrekkie 4y ago
  De-clouding is going to be a huge trend as companies are pressured to save costs, and they realize on-prem is still a fraction of the cost of comparable cloud services. This whole cloud shift has been one of the most mind-blowing shared delusions in the industry, and I'm glad I've mostly avoided working with it outright.
  • dopylitty 4y ago
    The thing that gets me about it is the very real physical cost of all this cloud waste. The big cloud providers have clear cut thousands of acres in Ohio, Northern VA, and elsewhere to build their huge windowless concrete bunkers in support of this delusion of unlimited scale. Hopefully as the monetary costs become clear their growth will be reversed and these bunkers can be torn down
    • ocdtrekkie 4y ago
      For what it's worth, large providers will always need datacenters. But perhaps datacenters run by public cloud providers today will be sold off to larger businesses running their own infrastructure someday at a discount. Most of the infrastructure itself all will age out in five or ten years, and would've been replaced either way. Heck, datacenters in Virginia are likely to end up being sold directly to the federal government.
    • adamsb6 4y ago
      The big cloud providers are likely packing machines more densely and powering them more efficiently than alternatives like colos.
      • spookthesunset 4y ago
        Much more than efficient. You think AWS is getting the same CPU normal civilians get? No way dude. Those guys are big enough that they can get custom hardware just for their specific needs. They’re cooling systems, power systems, everything is way more efficient. And they are big enough they can afford to measure every single metric that matters and optimize every one.
• lloydatkinson 4y ago
  I've been following their move to on premise with interest and this was a great read. I'm curious how they are wiring up GitHub actions with their on premise deployment. How are they doing this? The best I can think of for my own project is to run one of the self hosted GitHub actions runners on the same machine which could then run an action to trigger running the latest docker image. Without something like that you miss the nice instant push model cloud gives you and you have to use the pull model of polling some service regularly for newer versions.
  • efxhoy 4y ago
    They mentioned their mrsk tool sshes to the boxes to deploy so the action probably runs the tool and does just that?
    • mr_ndrsn 4y ago
      Negative. No external tool/company has ssh access. GHA is strictly for CI, which is decoupled from the actual deploy. If we do decide to tie it in, it will be using the GH Deployment API to inform the local tool on CI status or something.
      • lloydatkinson 4y ago
        What do you do then if you don't mind me asking? I see this problem time and time again for self hosting and and using CI/CD - and every time it seems to either come down to exposing SSH, polling for new versions, or running the github action runner on the same machine as the app or service.
• not_enoch_wise 4y ago
  It's almost as if... ...kubernetes isn't the solution to every compute need...
  • klardotsh 4y ago
    Tell that to the myriad of folks making their money off of peddling it. You'd swear it were the only tool available based on the hype circles (and how many hiring manager strictly look for experience with it).
    • illiarian 4y ago
      Cloud Native Landscape.... https://landscape.cncf.io/images/landscape.pdf https://landscape.cncf.io/images/landscape.pdf It's more than just peddlers at this point. There are peddlers peddling to other peddlers, several layers deep.
    • ilyt 4y ago
      I gotta say from dev perspective it is very convenient solution. But I wouldn't recommend it to anyone that runs anything less complex than "a few services in a database". The tens of minutes you save in writing deploy scrips will be replaced by hours of figuring out how to do it k8s way. From ops perspective let's say I ran it from scratch (as in "writing systemd units to run k8s daemons and setting up CA to feed them", because back then there was not that much reliable automation around deploying it) and the complexity tax is insane. Yeah you can install some automation doing that but if it ever breaks (and I've seen some breaking) good fucking luck, non-veteran will have better chance with reinstalling it from scratch.
  • birdyrooster 4y ago
    Except it was created to model virtually every solution to every compute need. It’s not about the compute itself, it’s about the taxonomy, composability, and verifiability of specifications which makes Kubernetes excellent substrate for nearly any computing model from the most static to the most dynamic. You find kubernetes everywhere because of how flexible it is to meet different domains. It’s the next major revolution in systems computing since Unix.
• Melingo 4y ago
  I'm so lost on so many choices this company did. You de cloud and now use some mini tool like mrskd? I'm running k8s on azure (small), gke (very big), rke2 on a setup with 5 nodes and k3s. I'm totally lost why they would de k8s after investing already so much time? They should be able to work with k8s really well at this point. Sry to say but for me it feels the company has a much bigger issue than cloud vs non cloud: no one with proper experience and strategy.
  • drewda 4y ago
    If one of the co-founders/owners is writing the devops tooling from scratch... well, that's a decision. Not saying it's necessarily a bad decision. But it's potentially driven more by personal interests than a dispassionate and strategic plan.
    • dilyevsky 4y ago
      Means either the company is doing super well or about to fail, no in-between here
      • danjac 4y ago
        I'm not sure how well 37Signals is doing these days - Hey didn't make as big an impact as they had hoped and Basecamp probably has a core of loyal users but I don't think it's getting a ton of new customers. They're small and could probably keep going until their founders decide to retire though.
      • deleted 4y ago
        [deleted]
  • chologrande 4y ago
    It does seem like they just moved all of their infra components, and got rid of autoscaling. Load balancing, logging, and other associated components are all still there. Almost nothing changed in the actual architecture, just how it was hosted. I have a hard time seeing why this was beneficial.
• Melingo 4y ago
  And don't get me wrong: whatever works for the company but k8s experience alone is already super helpful. A lightweight k8s stack out of the box + argocd + cert manager is like I fra steroids.
  • rektide 4y ago
    The whole kubernetes section of this writeup is two sentences. They went with a vendor provided kube & it was expensive & didn't go great. It just sounds like it was poorly executed, mostly? There's enough blogs & YouTube of folk setting up HA k8s on a couple rpi, & even the 2GB model works fine if having not-quite-half the ram as overhead on apiservers/etcd nodes. It's not like 37signals has hundreds of teams & thousands of services to juggle, so it' s not like they need a beefy control plane. I dont know what went wrong & there's no real info to guess by, but 37s seems like a semi-ideal easy lock for k8s on prem.
    • pferde 4y ago
      Or you could look at it like this: It's not like 37signals has hundreds of teams & thousands of services to juggle, so it's not like they need k8s and the complexity it brings.
      • rektide 4y ago
        So they wrote their own bespoke replacement. It seems like a lot of effort to do less. Hopefully it helps others too I guess. But it feels like a problem space with a lot lot of inherent complexity, that's liable to expand over time, & there is a very high skepticism I'd have to folks who opt to greenfield it all.
        • pferde 4y ago
          Sure, there is some inherent complexity, but by writing their own tool, they get to choose exactly how to handle the complexity for their particular use case, instead of having it dictated by a general-purpose tool developed by a consortium of US corporations. I consider that a win. If they have the manpower and expertise to do that, more power to them!
• ethicalsmacker 4y ago
  If only Ruby had real concurrency and the memory didn't bloat like crazy.... you wouldn't need 90% of the hardware.
  • wmf 4y ago
    https://m.signalvnoise.com/only-15-of-the-basecamp-operations-budget-is-spent-on-ruby/ https://m.signalvnoise.com/only-15-of-the-basecamp-operation...
• sys_64738 4y ago
  Larry Ellison was right.
• mfer 4y ago
  I wonder how they are going to handle fault tolerance when machines go offline? So as to avoid being paged in the middle of the night, I grew to really like automation that keeps things online.
  • turtlebits 4y ago
    Their app is running on at least 2 machines, so the load balancer takes care of it.
• electric_mayhem 4y ago
  We used to call this a “sudden outbreak of common sense”
• rad_gruchalski 4y ago
  Good for them. Now they have a one-off to manage themselves. It’s pretty easy to de-cloud using something like k3s. So much value added in Kubernetes to leverage. But they have Chef and they’re a Ruby shop, I guess they’ll be good. TBH, Kubernetes has some really rough edges. Helm charts aren’t that great and Kustomize gets real messy real fast.
  • turtlebits 4y ago
    The scope of their self-developed tool doesn't seem very large, it looks like it could be a wrapper around SSH. I've done similar things using a SSH library with python to deploy and run docker-compose yamls on multiple servers.
    • rad_gruchalski 4y ago
      There are many of these tools out there. When I was working for Technicolor Virdata some years ago, we’ve been heavily invested in https://github.com/infochimps-labs/ironfan https://github.com/infochimps-labs/ironfan. It was extensible, we had support for SoftLayer and IBM SCE, we had some patches to make the bootstrap and the kick command perform faster. But it was still slow and people didn’t like Ruby (I don’t mind it). Even back then I wasn’t a fan of doing a proactive ssh connection to the node. I always leaned towards the machine pulling artefacts and deploying them. Like Flux CD does.
  • klooney 4y ago
    Chef kind of scares me. It's such a tarpit, and it felt like abandonware five years ago
• camsjams 4y ago
  Is this cheaper though? For a medium-to-large app, K8s should offset a lot of the operational difficulties. Also you don't have to use K8s. Cloud is turn-on/turn-off, whereas on-premises you pay up front investment. Here are all of the hidden costs of on-prem that folks forget about when thinking about cloud being "expensive": - hardware - maintenance - electricity - air conditioning - security - on-call and incident response Here are all of the hidden time-consumers of on-prem that folks forget about when thinking about cloud being "difficult": - os patching and maintenance - network maintenance - driver patching - library updating and maintenance - BACKUPS - redundancy - disaster recovery - availability
  • sn0wf1re 4y ago
    Every one of your examples in the second list is relevant to both on-prem and cloud. Also cloud also has on-call, just not for the hardware issues (still likely get a page for reduced availability of your software).
    • vlunkr 4y ago
      “Just not for the hardware issues” is a huge deal though. That’s an entire skillset you can eliminate from your requirements if you’re only in the cloud. Depending on the scale of your team this might be a massive amount of savings.
      • josho 4y ago
        The flip side is there is an entirely new skillset required to successfully leverage the cloud. I suspect those cloud skills are also higher demand and therefore more expensive than hiring for people to handle hardware issues. Personally, I appreciate the contrarian view because I think many businesses have been naive in their decision to move some of their workloads into the cloud. I'd like to see a broader industry study that shows what benefits are actually realized in the cloud.
      • jrockway 4y ago
        At my last job, I would have happily gone into the office at 3am to swap a hard drive if it meant I didn't have to pay my AWS bill anymore. Computers are cheap. Backups are annoying, but you have to do them in the cloud too. (Deleting your Cloud SQL instance accidentally deletes all the automatic backups; so you have to roll your own if you care at all. Things like that; cloud providers remove some annoyance, and then add their own. If you operate software in production, you have to tolerate annoyance!) Self-managed Kubernetes is no picnic, but nothing operational is ever a picnic. If it's not debugging a weird networking issue with tcpdump while sitting on the datacenter floor, it's begging your account rep for an update on your ticket twice a day for 3 weeks. Pick your poison.
• erulabs 4y ago
  Only one sentence about why they chose to abandon K8s: > It all sounded like a win-win situation, but [on-prem kubernetes] turned out to be a very expensive and operationally complicated idea, so we had to get back to the drawing board pretty soon. It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling? Sort of undercuts their main argument that this bit isn't even remotely fleshed out.
  • benatkin 4y ago
    Well to be fair Kubernetes doesn't always pluralize the names of collections, since you can run "kubectl get deployment/myapp". You don't want to do the equivalent of "select * from user" do you? That doesn't make any sense!!! And don't translate that to "get all the records from the user table"! That's "get all the records from the users table". (Rails defaults to plural, Django to singular for table names. Not sure about the equivalent for Kubernetes but in the CLI surprisingly you can use either)
  • mike1o1 4y ago
    To be fair, from the article it says that they built the bulk of the tool and did the first migration in a 6-week cycle. mrsk looks fairly straight forward, and feels like Capistrano but for containers. The first commit of mrsk is only on January 7th of this year. In less than a six-week cycle, we built those operational foundations, shaped mrsk to its functional form and had Tadalist running in production on our own hardware.
    • bastawhiz 4y ago
      They spent a month and a half building tooling _capable of handling their smallest application_, representing an extremely tiny fraction of their cloud usage.
  • imiric 4y ago
    Sometimes there's value in building bespoke solutions. If you don't need many of the features of the off-the-shelf solution, and find the complexity overwhelming and the knowledge and operational costs too high, then building a purpose-built solution to fit your use case exactly can be very beneficial. You do need lots of expertise and relatively simple applications to replace something like k8s, but 37signals seems up to the task, and judging by the article, they picked their least critical apps to start with. It sounds like a success story so far. Kudos to them for releasing MRSK, it definitely looks interesting. As a side note, I've become disgruntled at k8s becoming the defacto standard for deploying services at scale. We need different approaches to container orchestration, that do things differently (perhaps even rethinking containers!), and focus on simplicity and usability instead of just hyper scalability, which many projects don't need. I was a fan of Docker Swarm for a long time, and still use it at home, but I wouldn't dare recommend it professionally anymore. Especially with the current way Docker Inc. is managed.
• asjkaehauisa 4y ago
  They could just use nomad and call it a day
  • tptacek 4y ago
    They could, but instead they're doing something closer to static scheduling. They have a small set of applications and a lot of visibility into what their needs are going to be, so the complexity of a dynamic scheduler might not pay its own freight in their environment. I like Nomad a lot and it's what I would use if I were migrating a "halfheartedly" K8s application to on-prem metal, but I couldn't blame someone who felt burned by K8s complexity for not investing in another dynamic scheduler.
• dustedcodes 4y ago
  If I was Netflix I would de-cloud, but if I was a small team like 37signals then de-clouding is just insanity. I think DHH is either very stupid or extremely naive in his cost calculations or probably a mix of both. Hey and Basecamp customers will see many issues in the next few years and hackers will feast off their on-premise infrastructure.
  • tky 4y ago
    They’ve had non-cloud infrastructure for a very long time. Their new orchestration methods notwithstanding, reliability and security are unlikely to suffer.
• bastawhiz 4y ago
  I'm not going to put this down, because it sounds like they're quite happy with the results. But they haven't written about a few things that I find to be important details: First, one of the promises of a standardized platform (be it k8s or something else) is that you don't reinvent the wheel for each application. You have one way of doing logging, one way of doing builds/deployments, etc. Now, they have two ways of doing everything (one for their k8s stuff that remains in the cloud, one for what they have migrated). And the stuff in the cloud is the mature, been-using-it-for-years stuff, and the new stuff seemingly hasn't been battle-tested beyond a couple small services. Now that's fine, and migrating a small service and hanging the Mission Accomplished banner is a win. But it's not a win that says "we're ready to move our big, money-making services off of k8s". My suspicion is that handling the most intensive services means replacing all of the moving parts of k8s with lots of k8s-shaped things, and things which are probably less-easily glued together than k8s things are. Another thing that strikes me is that if you look at their cloud spend [0], three of their four top services are _managed_ services. You simply will not take RDS and swap it out 1:1 for Percona MySQL, it is not the same for clusters of substance. You will not simply throw Elasticsearch at some linux boxes and get the same result as managed OpenSearch. You will not simply install redis/memcached on some servers and get elasticache. The managed services have substantial margin, but unless you have Elasticsearch experts, memcached/redis experts, and DBAs on-hand to make the thing do the stuff, you're also going to likely end up spending more than you expect to run those things on hardware you control. I don't think about SSDs or NVMe or how I'll provision new servers for a sudden traffic spike when I set up an Aurora cluster, but you can't not think about it when you're running it yourself. Said another way, I'm curious as to how they will reduce costs AND still have equally performant/maintainable/reliable services while replacing some unit of infrastructure N with N+M (where M is the currently-managed bits). And also while not being able to just magically make more computers (or computers of a different shape) appear in their datacenter at the click of a button. I'm also curious how they'll handle scaling. Is scaling your k8s clusters up and down in the cloud really more expensive than keeping enough machines to handle unexpected load on standby? I guess their load must be pretty consistent. [0] https://dev.37signals.com/our-cloud-spend-in-2022/ https://dev.37signals.com/our-cloud-spend-in-2022/
  • tstrimple 4y ago
• wmf 4y ago
  Why are there VMs?
• mannyv 4y ago
  I think RoR is the core problem here.
• denysonique 4y ago
  Seems like the name of their container shipping tool Mrsk is inspired by Maersk a leading container shipping company from Denmark.
  • ivarv 4y ago
    indeed - the presenter (dhh) is Danish, and there was a photo of a container ship near the beginning of the demo video.
  • deleted 4y ago
    [deleted]
  • trololo01001 4y ago
    Plot twist: all of this is just a very clever Maersk marketing campaign.
• AlexITC 4y ago
  I have been saying this to my customers for a long time, most projects do not really benefit from K8s but on the contrary, it is a huge operational/cost compromise to use K8s for a monolith app that does simple CRUD operations where occasional downtime is actually acceptable. In my last project, I removed the unnecessary complexity that K8s was bringing and went back to ansible scripts, which has worked nicely. With another customer, we inherited a frontend application that was being deployed with K8s while vercel is a considerably simpler/faster alternative. K8s certainly has its advantages but I'd bet that many projects using it do not gain much.
  • rossdavidh 4y ago
    My impression is that it makes deploying your 100th server much easier, at the cost of making your first several much harder. If you're going to have 100+ servers, that's probably worth it. If you're not (and most companies aren't), then it's like getting your CDL so that you can go to the grocery store in a semi-tractor trailer, when you should have driven there in a compact car.
• slyall 4y ago
  The thing I noticed is that they are not using any other AWS services. No S3, Elasticache, DynamoDB, etc. They are just running applications and databases. This will not be the case with many people using cloud and a migration to bare metal will be much harder. Each of those services needs an equivalent to be deployed and managed and it's features might be up to what the AWS equivalent has. Even the stuff that they are moving (databases, load balancers, etc) is significant operational overhead. In AWS database fail-over is an option you tick. Self hosting has whole books written about how to do database high availability.
  • wmf 4y ago
    For their more complex products they do use RDS, Elasticache, etc. so that will be a whole other adventure.
• Thaxll 4y ago
  They're saying a VM takes seconds to boot up, yeah only because they run static dedicated servers, of course in the cloud if you wait for the VM to come online it's going to take longer, now how long does it takes them to add a new dedicated server and to add it to that pool of servers? days? The other main issue I see is that they use chef and mrsk to setup applications, how is Filebeat setup? Is it chef that set it up or is it mrsk?
  • dilyevsky 4y ago
    > now how long does it takes them to add a new dedicated server and to add it to that pool of servers? days? Depending on provider and your config it could take minutes to hours these days. It also could take months if you’re ordering your own hw
  • chillfox 4y ago
    I started my career at a company that was excellent at capacity management and prediction. Using physical hardware they never hit a capacity problem, ever, despite growing like crazy. This did require the Marketing department being in close communication with the IT department about upcoming campaigns. Everywhere else though have been terrible at predicting future capacity needs. As far as I can tell that's because they just use tools that gives a prediction based only on historical growth. I guess my point is that it's entirely possible to be good at capacity management, and if you are then the lead time disadvantage of physical hardware can be completely negated.
    • Thaxll 4y ago
      From my experience with static physical capacity you're always way over provisionned, because just in case and how long it takes to get servers.
      • chillfox 4y ago
        It's easier to massively over provision or use the cloud than it is to get good at capacity planning. Same as how it's easier to use a GC than it is to do manual memory management. They are all valid strategies, the key is picking the one that suits your situation. If you need a small to medium amount of resources then the cloud is likely the cheapest option. If you need a medium to high amount of resources then massively over provisioning can still be cheaper than using the cloud. The cheapest option for anything medium size and above is physical servers with good capacity management. Good capacity management requires good internal communication between business units. And making predictions based on expected/planned events not just historical data.
• Ameo 4y ago
  Regardless of the merits or drawbacks of "de-clouding" for this particular company, it seems to me that their ops team is just really bored or permanently unsatisfied with any solution. They say that they've tried deploying their apps in all of: * Their own Datacenter * ECS * GKE * EKS * and now back to their own Datacenter Even with their new "de-clouded" deployment, it seems like they have created an absolutely immense amount of complexity to deploy what seems to be a variety of generic Ruby CRUD apps (I might be wrong but I didn't see anything different in the post). They have a huge list of tools and integrations that they've tried out with crazy names; Capistrano, Chef, mrsk, Filebeat, traefik... It seems well on par complexity-wise with a full K8s deploy with all the bells and whistles (logging, monitoring, networking, etc.) Google says that this company, 37signals, has 34 employees. This seems like such a monumental amount of orchestration and infra stuff unless they're deploying some crazy complex stuff they're not talking about. Idk what the lesson is here, if there is one, but this seems like a poor example to follow.
  • butt__hugger 4y ago
    37signals and RoR have a habit of flip flopping on their decisions. See CoffeeScript.
    • cutler 4y ago
      There's an operator for that.
    • tptacek 4y ago
      Each of these "flip flops" probably lasted a good deal longer than the median 20+ person startup, so that seems pretty facile. But the parallel with CoffeeScript seems valid --- people on message boards are really not OK with nonstandard languages, and are never less happy than when a company they've heard something about does actual computer science of any kind. See, for instance, Fog Creek and Wasabi.
    • mirekrusin 4y ago
      CoffeeScript impacted ECMAScript more than anything else.
• wankle 4y ago
  I so wanted to read the page but after 10 seconds of the high contrast white text on pure solid black background I started developing a headache.
  • rhizome31 4y ago
    Yes, very weird, I got kind of dizzy.
    • pxc 4y ago
      Because I have some light sensitivity issues, I use browser extensions including Dark Reader and Midnight Lizard to enforce my own 'dark mode' across the web. You can also use extensions like that to set the contrast to a more comfortable level on websites that are already dark. I highly recommend this if you have light sensitivity issues like me. Also note that when the contrast on a page is higher, you can generally get away with lower brightness. This is pretty convenient on phones, and probably more necessary as well since on a phone you're more likely to have an OLED screen that really surfaces extreme contrast like white on black.
      • wankle 4y ago
        There are some great web extensions for a lot of things. I don't use any of them because most of them require permissions to read data across all sites, which makes sense for them to work; but I'm not using any of them.
        • pxc 4y ago
          Fair enough. I only use long-lived, open-source browser extensions for that kind of global restyling. But of course there's still a risk that they could be compromised somehow.
• jacooper 4y ago
  Have they tried Swarm?
• vcryan 4y ago
  37signal folks love to put a spin on anything they do as if it's ground breaking or super innovative... but it rarely is. In particular, they love to take a contrarian position. Like their books, there really isn't anything interesting written here.
• tptacek 4y ago
  Skimming the thread here, it seems like there's some confusion about the goals: * They've decided to move from EKS to on-prem largely because of cost. That's logical: almost by definition, it costs more to run workloads on cloud machines than on your own hardware. You can't address that problem by moving from EKS back to ECS, like one commenter suggested. * They've decided to move from K8s to mrsk, a system they developed. They're fuzzier about why they did that, but the two fairly clear claims they made: (1) their deployments under K8s are a lot more complicated, and (2) they slashed their deploy times (because a great deal of their infra is now statically defined). I feel like there's more productive debating to be done about K8s vs mrsk than there is about EKS vs. mrsk. By all means, make the case that applications like Tadalist are best run on K8s rather than a set of conventions around bare-metal container/VMs (which is all mrsk is).
  • lmm 4y ago
    > almost by definition, it costs more to run workloads on cloud machines than on your own hardware. Why should that be so? I'd expect the all-in cost of a cloud machine to be less than my own hardware, for the same reason that buying electricity from the grid is usually cheaper than generating it on-prem. > You can't address that problem by moving from EKS back to ECS, like one commenter suggested. If EKS is more expensive (because it's something they see as a value-add) whereas ECS is a commodity service at commodity prices, then moving there could well solve the cost issue.
    • koolba 4y ago
      A better analogy for cloud vs on-premises is going to a restaurant vs cooking. The markups are about the same too.
      • harpratap 4y ago
        Wouldn't the cost of cooking be higher depending on who you are? If one could spend few hours doing something that has higher ROI than eating pre-cooked food, then you are actually losing money cooking your own food.
• mattgreenrocks 4y ago
  The comments in this thread are quite eye-opening. It really shows what a sacred cow k8s and cloud has become. I’m not much of an ops person so I’m not qualified to comment on what 37 signals has created. But I will say I’m glad to see honest discussion around the costs of choosing k8s for everything even if it has significant mindshare. Perhaps this is the endgame of resume-driven development: cargo culted complexity and everyone using the same tech for similar-ish problems and then wondering why it’s so hard to stand out from both a product and an employee perspective.
  • leetrout 4y ago
    I am much of an ops person and I will say k8s has its place and its in 10-20% of companies max. It is absolutely ridiculous how many places use k8s and rarely for what it can really do. Cargo culted complexity, indeed.
    • mike_d 4y ago
      Some people are really good at writing software, other people are really good at running systems. k8s/cloud allowed the former to pretend to be good at the latter.
  • lmm 4y ago
    I hate kubernetes as much as anyone, but building your own container orchestration platform so that you can deploy a handful of CRUD webapps sounds a lot more like resume-driven development than using a well-known and standard (if somewhat overengineered) solution.
    • mgkimsal 4y ago
      There are other options between k8s and “build your own”.
      • Jnr 4y ago
        What is there really? There is docker swarm, which doesn't seem to be really further developed, and... what else? This whole space seems to be neglected since cloud providers are trying to sell k8s to big company "devops"guys but old school sysadmins don't even know what docker is. Any development in this area is very welcome.
• throwawaaarrgh 4y ago
  A lot of people are kind of missing the forest for the trees here. Ignore the fact that what they're doing is probably a terrible idea for most other people. If it works for them, that's fine. It might only work for them, and that's fine. Don't paint your bike shed orange just because somebody famous painted theirs orange. They have their reasons. Paint yours whatever color works best for you, for your own reasons.
  • ozim 4y ago
    Only if DHH would not be posting marketing pieces where it seems that if you also paint your shed in orange it will save you $8mil in 5 years. It is also that DHH is somewhat renowned in software world so it is also not like Pedro Pascal is telling you to do that. Yes it is marketing and boasting but it is not so easy to see forest for the trees here.
• motoboi 4y ago
  On-prems is wherw companies goes to die.
• cutler 4y ago
  K8s, Docker and AWS/GCP/Azure are to ops what React is to web development, ie. rarely the appropriate tool for the job. Trouble is you now have a generation of devs who have no experience with anything else.
• yafbum 4y ago
  How does the overall cost of infrastructure compare, at the end of something like this? How fast does the move back pay off in terms of ROI?
• karakanb 4y ago
  I find it very interesting that every conversation around k8s turns into a flame war between "just use k8s" and "no you don't need k8s at all". in reality, it is probably more of a spectrum than a boolean value. Also, it seems like people have a different definition on "using kubernetes": * manage your own k8s cluster on your own hardware: probably pretty hardcore, I've never done this, I'd imagine it'd require me to know about the underlying hardware, diagnose issues and make sure the computer itself is running before managing k8s itself. only when the hardware is running properly I can focus on running k8s, which is also operationally expensive as well. Tbh I don't see a reason for a small/mid scale product to go this route unless they have a very specific reason. * manage your own k8s cluster on cloud hardware: this seems like a bit simpler, meaning that I don't actually need to know much about running/managing hardware, that's what the provider does for me. I have done this before with k3s for some small applications, I have 2 small scale applications running like this for ~2 years now on Oracle's free ARM instances, I don't really do any active work/maintenance on them and they are running just fine. I'd probably have a lot of trouble if I wanted to upgrade k3s version for large scale applications, or usecases that have tight SLAs. * use a managed k8s offering from a cloud provider: I've been doing this one the most, and I find it the easiest way to run my applications in a standardized way. I have experience in running applications on this setup for mid-scale as well as multi-national large scale consumer facing applications. Admitted, even though scale has been big, applications themselves have been mostly CRUD APIs, RabbitMQ / Kafka consumers and some scheduled jobs most of the time. The trick seems to lie in the word "standardized" here: it is probably possible to run any application on any sort of hardware/orchestration combination, and MRSK could be a really nice solution for that as well. However, in my personal experience I have never managed to find an easier way of running multiple full applications, e.g. things that have multiple components such as web APIs, async workers, etc, in a standardized, replicable way. I run the following components in one of my cloud-managed k8s clusters: - Vault - A few Laravel applications - A few Golang APIs - Grafana & Loki - Metabase Using k8s for situations like this where the specific requirements from the underlying infra is not very complex actually enables a lot of experimentation / progress simply thanks to the ecosystem. For all of these components there are either ready-made Helm charts that I can simply run a `helm install` and be 90% there, or it is trivial to build a simple K8s deployment configuration to run them. In my experience, I couldn't find anything that comes closer to this experience without having a large engineering team dedicated to solve a very specific problem. In fact, it has been pretty chill to rely on the managed k8s offerings and just rely on my applications. It's a spectrum: there are a billion cases that don't need k8s, and there are probably a similar amount that could actually benefit from it. There's no absolute truth to it other than the fact that k8s is actually useful for certain cases and it is for sure not always "resume driven development". This doesn't mean that we shouldn't be looking out for better approaches, there's probably a lot of accidental complexity around it as well, but we could also acknowledge that it is actually a useful piece of software. I don't know, I feel like I have to pick sides every time these sorts of stuff is being discussed as if there is an objective truth, but I am fairly convinced these days that there is a middleground that doesn't involve fanaticism in either direction.
• spicyramen_ 4y ago
  [dead]
• BackBlast 4y ago
  Online deployments are discussed with some frequency. Tooling is talked about. Always as a "cluster". Why do we need clusters anymore? Scaling containers, scaling functions, scaling, scaling, cluster, cluster. We suffer so much tunnel vision about horizontal scaling when it's just unnecessary anymore for most applications. The cloud products are all about horizontal. Do you really need more than the 400 threads >12TB ram with PBs of storage found in a reasonable high end server?
• ludwigvan 4y ago
  mrsk introduction screencast by DHH explains the process well, and is pretty impressive as usual: https://m.youtube.com/watch?v=LL1cV2FXZ5I https://m.youtube.com/watch?v=LL1cV2FXZ5I
• Joel_Mckay 4y ago
  Depending on your business use-case, sharing a pool of IPs can have detrimental impact on access. For example, you may find the prior users were doing unauthorized security scans, spamming email, or hosting proxies. i.e. if you get an IP block with a bad reputation, than you may find running a mail or VoIP server problematic. If you are running purely user centric web services, than it doesn't matter as long as you are serving under around 30TiB/month. There is also the issue of future offline decryption of sensitive records without using quantum resistant storage cryptography. Rule #4: The first mistake in losing a marathon is moving the finish line. =)
• kristianpaul 4y ago
  Neither cloud or on-prem fits everyones requirements, but at the end you need to know your environment well. One think i like from ECS and fargate is that you could use projects like my_init and get a container behave closer to a vm (run ssh, and other daemons at the same time).
• clepto 4y ago
  Throughout my company’s pursuit of moving everything under the sun into AWS I have done my best to keep everything able to be migrated, we have some systems which are just, simply going to have to be completely rebuilt if we ever needed to move them off of AWS, because there is not a single component of the system that doesn’t rely on some kind of vendor lock-in system AWS provides. I aim to keep everything I’m working on using the simplest services possible, essentially treating AWS like it’s Digital Ocean or Linode with a stupidly complex control panel. This way if we need to migrate, as long as someone can hand me a Linux VM and maybe an S3 interface we can do it. I really just have trouble believing that everyone using Kubernetes and a bunch of infrastructure as code is truly benefiting from it. Linux sysadmin isn’t hard. Get a big server with an AMD Epyc or two and a bunch of RAM, put it in a datacenter colo, and maybe do that twice for redundancy and I almost guarantee you it can take you at least close to 9 figures revenue. If at that point it’s not enough, congratulations you have the money to figure it out. If it’s not enough to get you to that point, perhaps you need to re-think your engineering philosophy(for example, stop putting 100 data constraints per endpoint in your python API when you have zero Postgres utilization beyond basic tables and indexes). If you still really genuinely can’t make that setup work, then congratulations you are in the 10%(maybe) of companies that actually need everything k8s or “cloud native” solutions offer. I would like to note that given these opinions, I do realize there are problems that need the flexibility of a platform like AWS, one that comes to mind is video game servers needing to serve very close to a high number of geographic areas for latency concerns.
  • codethief 4y ago
    To play the devil's advocate here: > I aim to keep everything I’m working on using the simplest services possible, essentially treating AWS like it’s Digital Ocean or Linode with a stupidly complex control panel. What's the benefit of AWS then, if you're not using any of the managed services AWS offers, and are instead treating AWS as an (overly expensive) Digital Ocean or Linode?
    • clepto 4y ago
      I’m arguing there’s not a benefit, it’s just the service I have to use for reasons outside of my control.
• fbdab103 4y ago
  Getting a real dokku vibe from this. Although, currently limited to one container per machine, which is a huge limitation.
• richardw 4y ago
  I see this as an instance of a company discovering value well-described in this article from 2021: "The Cost of Cloud, a Trillion Dollar Paradox" https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/ https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap...
• kgeist 4y ago
  We use k8s to run the app both on AWS, on our own hardware in a few datacenters (in countries with strict personal data laws) and on clients' own servers as well (something like the banking sector or a jewelry company, i.e. companies which don't trust the cloud). From what I heard, AWS is the most stable and easiest to work with of all; the servers which run on our own hardware have more outages and our SRE team often needs to make trips in person to the datacenters to replace hardware etc. Clients' hardware is the faultiest (unsurprisingly). Ideally we'd rather host everything on AWS :)
• ec109685 4y ago
  It’s pretty sad that both Google and AWS price their on premise versions of their hardware at the same price as if you were running in their cloud. Makes it totally a non-starter to be a universal platform.
• lmarcos 4y ago
  I think many have lost the main point of this post: 37signals seems to be a successful tech company, and their employees are having fun. Very few tech companies can say that. So, it doesn't really matter if they ditch k8s for their own solution. Really, it doesn't.
• KingOfCoders 4y ago
  Timing is perfect, several of my clients fell through Series B and now look for cutting cloud costs (all way over provisioned for their traffic and customer numbers).
• KingOfCoders 4y ago
  Cloud is important for those who need it. We did run a fast growing startup (sometimes 100% MoM jumps), 5M active users with 50k concurrent users (not visitors) with DB writes on 6 machines + 2 DB servers and $100M ARR 10 years back. If you're this size, MRSK makes total sense. If you're much larger or growing >50% MoM continuously, K8 in the cloud makes more sense.
• roboben 4y ago
  > cluster-autoscaler > ingress controllers > storage drivers > external-dns > node termination handlers > complex networking concepts around VPN, peering, route tables, NAT, … > where DNS is handled > It also misses the entire sphere around identity and access management for those resources that also needs to be maintained Well, how is this all solved with their new tooling? Like they describe a whole huge complicated problem space and then write a tool for the simplest part of it: deploying an app. :shrug:
• xyzzy_plugh 4y ago
  I've never read such a ridiculous article. I really wanted to give them the benefit of the doubt but good lord. How is any of this simpler or better? It's like they prefer the pain of 2004 mixed with the worst parts of modern infrastructure. The dream of the 2000s really is alive in DHH's head, isn't it? TFA links to their cloud spend for 2022[0], wherein lies the rub: > In total, we spent $3,201,564 on all these cloud services in 2022. That comes out to $266,797 per month. Whew! > For HEY, the yearly bill was $1,066,150 ($88,846/month) for production workloads only. That one service breaks down into big buckets as follows: What the actual fuck? THREE MILLION DOLLARS? A million for their email service?? I have seen bills much larger, but for what 37signals does I am shocked. There is surely a ton of low hanging fruit to drop the bill despite the claim that it's as optimized as it can get. No way. Even then, Hey is $99/year, and they claimed to have 25k users in the first month or so as of 2020, that's nearly $2.5MM. I presume they've grown since then. Another 2020 article[2] mentions 3/4 of their users have the iOS app, and the Android app currently shows "50k+ installs" so let's assume we're talking 200-400k users as a ceiling, ignoring attrition, which would pull $20-40MM. Even if it's half that, the cost doesn't seem unreasonable. They're spending nearly $90k/mo on Hey. Of that the majority is RDS and OpenSearch. TFA makes it clear they know how to run MySQL, why on earth don't they stop running RDS? Both of these can easily be halved if they ran the services manually. EKS is practically free so whatever. They state they have two deployments for ~$23k/mo total -- production is likely larger than staging but let's assume they're equal -- or ~$12k/mo each. A middle of the road EC2 instance like m4.2xlarge is less than $215/mo which gets more than enough cores and memory to run a rails app or two per node. That works out to around 55 nodes per environment. This benchmark[3] shows an m4.2xlarge can serve 172req/s via modern Ruby on Rails. At 500k users that works out to over 1600 request/user/day which seems excessive but likely within an order of magnitude of reality. These are the folks who wrote RoR so I would hope they can optimize this down further. <10000req/s for $12k/mo is pretty awful, and I'm being conservative. Then let's talk about the $1MM/mo S3 bill. I'm not sure how to make 8PB cost that much but even the lightest touch at optimizing storage or compression or caching knocks the cost down. This is all just nuts. There's no reason this all shouldn't be running on AWS or GKE with a much smaller bill. Their apps are predominantly CRUD, some email. Instead they replaced kubernetes with an in-house monstrosity. I am bewildered. [0] https://dev.37signals.com/our-cloud-spend-in-2022/ https://dev.37signals.com/our-cloud-spend-in-2022/ [1] https://www.theverge.com/2020/6/22/21298552/apple-hey-email-app-approval-rules-basecamp-launch https://www.theverge.com/2020/6/22/21298552/apple-hey-email-... [2] https://m.signalvnoise.com/on-apples-monopoly-power-to-destroy-hey/ https://m.signalvnoise.com/on-apples-monopoly-power-to-destr... [3] https://www.fastruby.io/blog/rails/ruby/performance/how-fast-is-ruby-3-on-rails.html https://www.fastruby.io/blog/rails/ruby/performance/how-fast...
• e12e 4y ago
  After reading tfa - I'm actually a bit confused about MRSK. I'm not sure I'd want to run MRSK and Chef? They seem more orthogonal than complementary? Ed: at least between Capistrano and Chef it seems that MRSK is redundant?
• benreesman 4y ago
  So as an industry we've been having some version of this debate (at FB we were having it at least as far back as 2014, my org was IIRC the first big one to test-drive our Borg-alike container solution). These days I think maybe it's just that classic dilemma: over-design and over-build to be ready for contingencies, or build just what we know we need and maybe get caught with our slacks down. This goes by a zillion names WET vs DRY, YAGNI, microservice vs monolith, there are countless variations on the same core idea. If you start with PHP and MySQL and a chain-smoking sysadmin, and you get hit with hyper-growth then you adapt or die, and you have a mountain of data to figure it out. This is paradoxically an easier decision tree (IMHO) even if maybe some of the engineering is harder or at least higher-stress. But by far the more common case is that we're building something that isn't huge yet, and while we hope it goes huge we don't actually know if it will: should we build more features and kinda wing it on the operability/economical/automated can of worms, or should we build for the big time from day one? I think it's a legitimately hard set of questions and reasonable people can disagree. These days I think the only way to fully screw it up is to get ideological rather than pragmatic about it.
• liampulles 4y ago
  I think there is a space between managed K8s on the cloud and e.g Ansible managed docker deploys on-prem. I'm curious to see how it pans out.
• greendestiny_re 4y ago
  Sounds to me like 37signals uses the risk aversion paradigm typical for stagnating businesses — instead of building and refining their strengths they're fixated on mitigating their weaknesses.
• zubairq 4y ago
  37 Signals has many technical people and can afford to de-k8s. But K8s is designed for a totally different use case, for large corps where there are most non IT staff but where IT resources and standards need to be managed in a more central way. Most standard banks or large companies do not want to roll this stuff by hand, they care about STANDARDS!
  • ignoramous 4y ago
    That's the thing with technology though, it goes mainstream as adoption grows. RoR started small at 37Signals and eventually became a standard. MSRK might yet be one, there's no telling right now.
• smetj 4y ago
  It's anecdotal but the sentiment I have is that the Kubernetes ecosystem drains an even bigger part of the collective effort required to provide business value. I believe many engineers have a disconnect on what it means to provide real business value. Solutions like Kubernetes are designed to be able to accommodate an endless number of scenarios out of which you probably only need 1 to provide value for your business. The consequence is that there's a disproportionate ratio of Kubernetes possibilities hence complexities vs. the simplicity of your requirements. Once your workload runs on Kubernetes, you cannot afford to ignore the complexities of Kubernetes so you are automatically sucked into the rabbit hole.
• Lapsa 4y ago
  well... in my book, k8s has always been in "Dinosaur" category. somewhat useful, somewhat versatile, perhaps even good. quick glance on documentation eradicates any desire to learn the tech
• jwmoz 4y ago
  How much users/traffic does Tadalist have?
• DeathArrow 4y ago
  At one of my former workplaces we ran Kubernetes on premises and it worked like a charm. I still think that Kubernetes can be a good fit for microservices even if you use your own hardware.
• namaria 4y ago
  Kubernetes is an amazing tool. Cloud computing is a powerful way to leverage a small team and prototype stuff quickly. Ocean going ships are impressive pieces of kit. CNC machine tools are a powerful way to leverage small teams and manufacture high quality stuff quickly. Now, telling every repair business in town they need robotic lathes and a fleet of major cargo ship is nonsense. Why this kind of discourse thrives in software is beyond me.
• daitangio 4y ago
  K8s has some cognitive overhead. For simple deploy a docker client-server with docker-compose is a winner, see misterio[1] which basically leverage docker-compose +ssh But when you need to guarantee system will auto-restart, and healthchecks and so on, K8s is the de-facto standard. Helm template language (based on Go) is not ideal, but it difficult to replace K8s nowadays with simpler systems. [1] https://github.com/daitangio/misterio https://github.com/daitangio/misterio
• zac23or 4y ago
  Wow. "K8s is simple", it has the same vibes as Linux user vs Dropbox: '...you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem' https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863 https://www.theregister.com/2021/02/25/google_kubernetes_autopilot/ https://www.theregister.com/2021/02/25/google_kubernetes_aut... I love HN's disconnection from reality.
  • cmeacham98 4y ago
    It's not that Kubernetes is simple (it's not), but Kubernetes is relatively simple compared to the task it accomplishes. If you have containers that need to be scheduled and networked and supplied with storage and exposed to the internet across a large set of machines past the scale where you can easily do so with tools like docker-compose, Kubernetes (might) be for you. There's a good chance it will be simpler to understand and reason about than the homegrown kludge you could make to do the same thing, especially once you understand the core design around reconciliation loops. That said, you might not need all that, and then you probably shouldn't use Kubernetes.
• chriszioutas 4y ago
  have they thought to just run openstack on their own servers? everything I saw leads to me to use saltstack + openstack as they dont wanna be on cloud.
• Cwizard 4y ago
  I think it is cool they are developing new tooling. I don’t understand all the negativity. Isn’t it good that people keep innovating in this space? Also how is this different from deploying before k8s and TF etc was a thing? We would write our own scripts to manage and deploy our servers. This is the same no? Just bit more structured and it has a name.