16 ms·

distracting infrastructure (2019)

213 points by ddtaylor 6y ago

nattaylor 6y ago
Maybe it's more important to ask 'why' [do I need this infra]? Complexity shouldn't be the default but it can be the right answer
- bovermyer 6y ago
  As a "devops" engineer, I can say with total confidence that complexity is rarely the right answer. When it is, it's because that complexity is the lowest pain point in a calculation that has itself become complex.
  - agustif 6y ago
    Sometimes you can add complexity to the whole stack by adding a new shiny layer that brings down complexity where you care most about it, in your user-facing UI/UX layer. Developers can handle complexity via code much more efficiently than users operating a site or app... Also, some of the nice developments of the recent explosion of serverless/as a service stuff, is that you can mostly try stuff for free, be happy on the free tier if you don't need scale, and use that scale when you need it without much hassle but paying the overprice of their fat margins. Also you should probably have an ejection plan in case scale + external provider becomes too expensive, you should always have a nice idea on how to run your stack if you need it...
- Zelphyr 6y ago
  Complexity is never the right answer. Complexity, in my experience, comes from two things: The person building that system doesn't understand the problem well enough and/or equates complexity with quality. Neither are good things.
- jbn 6y ago
  that's because there are 2 kinds of complexity: essential (or intrinsic) complexity and accidental complexity. When one uses k8s when one doesn't truly need it, it's accidental complexity. However some problems require k8s, where the engineered complexity of k8s helps solving the problem at hand. Of course the issue is that most people don't need k8s, whereas Google and the like do.
jameshart 6y ago
There is obviously some point, somewhere between ‘just running a website out of my spare bedroom’ and ‘Facebook’, where some of this infrastructure does become necessary. It is very important that you realize when you pass one of those points. It doesn’t help to just tell people ‘you don’t need kubernetes, you are not Facebook’ There are a lot of companies that aren’t Facebook, but they still have multimillion dollar revenue streams riding on service availability, with compliance obligations and employee paychecks on the line, and which need to be able to reliably release changes to production code every day, written by multiple development teams in different time zones, so... yeah, maybe they are a bit beyond just scp’ing their files up to a single Linux VPS.
- tyingq 6y ago
  Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql.
  - ahofmann 6y ago
    This could be the final post to this discussion. You can't really get more to the point than that. We are running a large German browser game on individual VPS instances. For many years, with PHP, MySQL and Redis. It works and it runs just fine.
    - bombcar 6y ago
      It's not one that just recently had to start migrating from Flash, is it?
  - jasode 6y ago
    >Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql. But PHP and MySql are only the surface level tools we see. There is also the unseen and that's where the immense complexity lies hidden under the surface. E.g. To help with scaling (which Friendster failed at), Facebook had separate MySql instances for each school. An interview with Facebook co-founder Dustin Moskovitz mentioned they had constant stress of emergencies and fires to put out. They had tons of custom tooling, Linux scripts, batch jobs, automation/orchestration/dashboard, etc and it's very likely the combination of all that ends up inventing an "invisible internal Kubernetes" without calling it "Kubernetes" to manage their server fleet. In other words, if Docker/Kubernetes/AWS were around, the early Facebook team might have offloaded some of the complexity onto a 3rd-party to reduce the all-nighters. But the tools didn't exist yet so they had to do it the hard way and invent "deployment orchestration" themselves. [EDIT to reply] Thanks for the "Tupperware" comment. A Google search leads to a Facebook post saying they renamed it to "Twine": https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin...
Cthulhu_ 6y ago
As always, it depends. Lots of people wish they had Facebook's challenges when it comes to scaling. 99% of applications simply don't. I've never seen a company I've worked with go for 'fancy' infrastructure because they needed to. It was always because they HOPE that they need to. Granted, re-engineering your application / architecture for scaling is expensive, but not as expensive as building the wrong thing in the first place. They should all have started off with a monolith and grow from there. I mean in all those cases we had just a single, 'monolithic' front-end without any problems.
- mring33621 6y ago
  Aspirational Infrastructure!
fendy3002 6y ago
I assume that premature infrastructure optimization (complexity, complex automation, kubernetes, etc whatever you say) is same with premature optimization, both are bad. However same with optimization, there are some best practices that are good to follow, such as avoiding N+1 queries. For me personally, that will be deploying with docker-compose, which usually yml already be created during development, and it'll be much easier to migrate later.
- jameshart 6y ago
  This is of course a tautology. Premature anything is bad. That’s what ‘premature’ means. It means you did it too soon, before the optimal time. Doing it later would have been better. But there is an optimal time. And even if you did it too soon, if you eventually needed it, at least you’ve done it. Better that than getting to it too late, surely?
  - heroprotagonist 6y ago
    And then there's the "I've already done it this way a dozen times and I'm comfortable with it. There are benefits to it and it's not that much extra effort to set it up from the beginning and save 10x effort down the line."
    - 8note 6y ago
      You need that to go with "also it doesn't add another failure point/upgrading dependencies work before it's needed"
ic0n0cl4st 6y ago
This past year I’ve finally been able to hit the $200/hour rate as a k8s consultant. I feel like I’m selling snake oil. I’m working on my 7th implementation and none of my customers need kubernetes. Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using. My largest implementation was a $3m/year, 9 year old company who could run their entire infrastructure on a couple dozen ec2 instances. However they are under the delusion of “hitting it big and needing to be able to scale 100x”. Paid off my student loans this year, and soon my house, thanks to superfluous k8s integrations!
- droopyEyelids 6y ago
  You are the real job creator, as there will be more work after your migrations than there was before
- deagle50 6y ago
  I've been on the fence about this. If you don't mind, how did you get started?
  - waynesonfire 6y ago
    Oh, I can tell you because I work with these asshats. They come in, deploy the technology, usually get promoted and then leave. The people left behind have to maintain their garbage and because promotions are only given to new feature work we're the suckers holding the bag.
    - ic0n0cl4st 6y ago
      Actually This is exactly how I accidentally started consulting on k8s. Three of my customers had engineers who advocated for k8s, did 50-75% of an implementation then left. I have told all of my customers they don’t need this, but they are convinced this is the way of the future.
      - deagle50 6y ago
1996 6y ago
So true. HN often admires complexity for complexity sake. Even without cloudflare, a static website on 5 VPS from 5 different provider on 5 different continents with a Round Robin DNS and a low TTL will get you most of the bangs for bucks Add geoIP to the mix (like from Azure) and you're golden, even if you are just scp'ing files to 5 VPS. If a DC burns down, remove the A record (or even better: have a script do it for you) and reroute to the remaining 4. Extra bonus: you can fire the DevOps team :)
- tartoran 6y ago
  Why fire the team and not just repurpose them? They’re already familiarized with some aspects of the product.
- tpetry 6y ago
  That‘s easy if you are just serving static html. But a database replicated on 5 continents? That‘s a lot of latency for replication.
- jameshart 6y ago
  That’s funny, sometimes when I visit your site, I get an HTML page that references script files and resources that come back as 404s. I wonder if it’s possible I’m hitting the site during a deployment and getting the new HTML page from one of your servers, but then the resource requests are going to a different server which hasn’t received the new version yet. I guess occasional broken images and script failures are unavoidable though. You’d need like, some dev ops engineers to solve a problem of that complexity.
jFriedensreich 6y ago
3 important factors: - if i use state of the art production grade technology i can talk about it in interviews, if i say oh well i have plenty experience running docker compose manually on a server, not so much. This is also where new technology can be tested without hurting someone, so even if something is totally overkill it can still make sense in the overall picture of all projects you work on - ci/cd is maybe not relevant to your users, but its about your own sanity, it forces complete documentation in code of what is done to build and ship, decouple it from local dev setups and make as deterministic as possible. Even small project benefit, have you ever come back after 2 years and took hours to remember the order of voodoo commands or lost a laptop and then had to install specific versions of cli tools? - the logic is totally changed by the simplicity of faas offerings. Nowadays i would maybe not build a gigantic kubernetes system, but instead of dealing with vms and servers skip right to cloudfare workers or cloud run and have the same benefits of k8 with even less work and more time to work on the features
- rjbwork 6y ago
  >- ci/cd is maybe not relevant to your users, but its about your own sanity, it forces complete documentation in code of what is done to build and ship, decouple it from local dev setups and make as deterministic as possible. Even small project benefit, have you ever come back after 2 years and took hours to remember the order of voodoo commands or lost a laptop and then had to install specific versions of cli tools? This to me is the biggest advantage of this sort of thing. Documentation and notes are great, but I'd rather just have the thing-in-itself available that does the job.
  - jFriedensreich 6y ago
    Yeah and 13 years ago there was a similar argument about git. I heard a lot of developers say well its a small project and its just me, no way i'm dealing with repos, making commits and learn git or mercurial. This completely died out.
  - Person5478 6y ago
    CI/CD doesn't require the cloud or containers.
Traubenfuchs 6y ago
Operations guys, "devops" as they are called now, remain the arch-wizards with grey beards. They love creating highly company-customized, incredibly complex, often times brittle infrastructure and deployments based on obscure scripts written in non mainstream languages based on plugins and libraries without documentation. The same opaqueness as in IT security, but in emergencies you can downsize/ignore IT security, but without the ops guys your product can not survive. Kubernetes, Helm and cloud-platform specific quirks were a godsend that cemented in this status for at least another decade. There is probably no job with higher job security in tech than this.
- breckenedge 6y ago
  > There is probably no job with higher job security in tech than this. Will the last person to leave please turn off AWS.
- supercon 6y ago
  So were it not for this era of cloud platforms and the other technical ”advancements” you mention, would we somehow be in a different place when it comes to maintaining and operating software? As someone in a devops team, I could argue that ops teams are rarely the ones deciding what technologies to use. If we follow the basic sales->development->maintenance -pipeline, as ops your task is simply to keep the service running and secure with the tools you are given. Sure there can be added complexity, but I doubt it is as systemic/general as you make it seem to be. Anyway, if this is seen as a problem the devs could simply take on the responsibility of learning those quirks themselves or produce the level of quality software that requires no maintanance whatsoever. But for some reason that hasn’t happened, eventhough the original idea behind devops was to blur this line. So until that day comes, I’ll sure enjoy whatever security one can really have in todays job market.
  - Traubenfuchs 6y ago
    No, we would simply still use incomprehensible websphere setups. Now we got kubernetes as the application server of the current decade that's application language/framework agnostic. But it shouldn't be like this. Devops shouldn't have to exist. Everything should work like heroku, where even mediocre devs can quickly setup proper pipelines.
mercury_craze 6y ago
This isn't totally related to infrastructure but figuring out what you should build, from what you could build is something I've struggled with as I've moved about the engineering management ladder. My current role is a stone’s throw from solo developer (only developer at 4 person start-up) and while I've got total autonomy in the work I complete and the approach I take, it’s a terrible feeling discovering that you're not super-human and you need constraints like almost every engineer that's come before you. Like the author there's been a number of times where I've felt self-satisfied completing a piece of technical work, only to scratch my head a week later and wonder whether it was worth it. I can say with certainty that having a solid CI/CD pipeline has saved my bacon a few times so I'd probably recommend that as an early investment to make, internal tooling for some of the customer registration was a bit of a mistake and not as critical as I thought it would be which now feels like a waste of 3 weeks. I am yet to come up with any meaningful advice to pass on to developers in the same position. With time you get better at spotting when you can get away with hacking solutions together and when to put the time in, having a forward vision for a product really helps on focusing your thought and attention and time spent on making things visible to the rest of the business is rarely time wasted. Awareness of what’s going on with the rest of the industry is useful, but don’t obsess over the new hotness, build with what you know works and save your innovation tokens for a rainy day.
durnygbur 6y ago
I don't hear you from inside bastion in autoscaling cluster!
eric4smith 6y ago
Sometimes a nice dedicated server and Dokku is good enough. Backups of database offsite, assets in S3 and code in Git means you’re good for 90% of use cases. >> git push dokku master Done.
lukeschlather 6y ago
> Your users to don’t care how your code gets onto your server. I work in B2B and while sure, our users don't necessarily care but they also aren't the ones paying us money. The people paying us money have a huge set of questions about how code gets on our servers and Kubernetes is not everything but it helps a lot answering those questions. (Especially using GKE.)
0xbadcafebee 6y ago
Whether the infra is complex or not isn't going to make a huge difference to whether your product makes money. Some of the highest-revenue-generating products I have seen were tire fires under the hood. One product had multiple people working 24 hours a day to constantly hose down the multiple levels of problems that the incredibly poor design and terrible infrastructure created. But through the toil of those people in the backend, the product made hundreds of millions a year. Whether or not you use complex technology, it matters more whether it's well designed and well operated. If it's not, you will end up with a fire either way. It's just that "simple" fires are a lot easier to put out than "complex" ones. (this is literally true; a lot of fires should not be put out with water)
waynesonfire 6y ago
I went down this exact same path. Started simplifying everything and it did wonders for my velocity. It took me a while to realize that we have teams and teams of people maintaining all this crap I was trying to spin up myself. It just wasn't worth it. I went to basic principles, FreeBSD. systemd has taken Linux in the wrong direction in my opinion. I don't want anything fancy, I don't want to learn another persons resume building framework, I want to use technology that has proven longevity. Our technology space moves so fast, for my personal work, I have to leverage the first principle technologies that I have mastered over my career. The latest framework may make some problems easier to solve, but I don't deploy technology without achieving some level of competency. That takes significant time. I'm getting wiser with how I spend that time, it's not always easy to know. This process has also taught me why we have this proliferation of SaaS solutions. It's tempting, and maybe one day I'll be wise enough to realize that this is the route to take. However, I'm still able to leverage my core skills and open-source technology enough that SaaS solutions don't look attractive to me. I can see the benefits either way, so I suppose maybe in five years when I reflect I'll know if I made the right investment.
- AlchemistCamp 6y ago
  > systemd has taken Linux in the wrong direction in my opinion Can you elaborate on that a bit? Where does it fall short for you and what's your preferred tool?
  - KronisLV 6y ago
    I've often heard people saying that systemd attempts to do too much and as a result is bloated, but personally i've found it to be sufficient for what i want to do - making certain software run and/or automatically restart on a server. Some of the alternatives that i've heard of: - SysVinit (i think older RPM distros used it) https://wiki.archlinux.org/index.php/SysVinit - OpenRC (i believe that Alpine Linux uses it as its default) https://wiki.archlinux.org/index.php/OpenRC - and here are some other links https://alternativeto.net/software/systemd/ For better or worse, i've mostly just stuck to using systemd because it's good enough - and init systems that require scripts to be written manually intimidate me, since it seems like it'd be easy to make mistakes and the scripts wouldn't be standartized enough. Personally, i prefer to create a configuration file that just defines where the software is, how to keep track of its PID, where and how to get configuration for it, as well as what user/group to execute it as (among other things). Maybe the poster you're replying to will provide some arguments against systemd and situations where it isn't very nice to use.
dominotw 6y ago
Its not just engineers that want to build complex infrastructure though. This a massive misconception and shifting the blame to 'nerdy engineers' doing 'nerdy' stuff instead of solving real problems. Startup that worked at raised whole bunch of money and hired tons of engineers. They were all tasked to "build something big". eg: it was expected of data engineers to setup "big data architecture" with complex data-lakes, job schedulers , spark clusters, streaming jobs and all sorts of weird stuff. No one really understood why we are building all this stuff but we never got real answers from the management.
foobarbecue 6y ago
I find there's a bit of an "if you give a mouse a cookie" effect with this stuff. I recently wanted to organize a data processing pipeline at our company. In the past, the system was: there's a bunch of scripts, some of them call each other, and sometimes a person logs in and kicks off one or more scripts without recording their inputs and sending the output to wherever they feel like. I thought implementing this on Argo Workflows would allow us to keep track of previous runs and generally clarify things. Well, Argo runs on Kubernetes. Also, it kinda wants your data in a bucket. And argo kubernetes wants kustomize and skaffold and or / helm and soon I'm installing a package manager that manages package managers and I spend half my time managing systems we don't really need. I'm learning I need to to draw the line and deny Argo some of the things it wants. No, we don't need to run minio and call our files "artefacts." The filesystem works just fine thanks, we don't need another layer of it. This means we can't follow some of the Argo examples and lose some of its functionality.
jwr 6y ago
I ran my B2B SaaS on a single (physical) server for the first three years or so. I then switched to a setup with three production servers, and three staging servers (in a different data center), mostly to have the comfort of a distributed database, and a staging system to try things out. Physical servers provide enormous computing capacity these days. And managing them isn't much different from virtual servers: you should be using a tool like ansible anyway. The only difference is that when I spin up my development cluster I use terraform to generate the ansible inventory, and for physical servers the inventory is static. I do not anticipate needing more capacity or scalability anytime within my planning horizon. I keep seeing people investing tons of time and effort into things like Kubernetes and I honestly have no idea why. It complicates things enormously for no good reason: the benefits, if you aren't Google, aren't interesting at all, and the drawbacks are huge. The biggest drawback for me (apart from the cost of learning, building, and maintaining the infrastructure) is the complexity and resulting unpredictable failure modes. There are things which I'd say are really important for infrastructure, though: * a working and tested backup strategy * disaster recovery plan (what happens when your entire datacenter burns down) * managing servers via a tool like ansible * regular testing of various scenarios (including setting up an entire system from scratch on new servers, with data recovery from backups) * a database that reliably preserves all of your customers' data when any one of your servers goes down (this unfortunately excludes many databases and makes the usual "just use Postgres" a poor choice, now flame me to high heaven) I can't imagine a small business where Kubernetes makes business sense. Do people seriously have problems with scaling to hundreds of machines? Scalability problems with huge traffic peaks that actually cause server load? I'm having trouble keeping my CPUs busy for any meaningful amount of time, and I don't spend much time on optimizations, I just avoid doing stupid things.
- mwcampbell 6y ago
  > * a database that reliably preserves all of your customers' data when any one of your servers goes down (this unfortunately excludes many databases and makes the usual "just use Postgres" a poor choice, now flame me to high heaven) What do you suggest instead? My understanding was that Postgres, when properly set up with backups and replication, is the most reliable open-source database.
- 6y ago
twodave 6y ago
> Your goal, when launching a product, is to build a product that solves a problem for your users. This is the main takeaway, in my opinion. I've been at small companies who were under-architected "because we can" and large companies who were over-architected because someone in the leadership chain was enamored with a particular technical choice. In the end it's a cost curve and depends heavily on the nature of your business. I can say personally if the business is even mildly complex (not necessarily large, this could mean a web app + a mobile app + a single RDBMS) I'd sleep better having some AWS (or other cloud) alerts set up to monitor uptime/etc. Others have mentioned CI/CD, but again, heavily depends on your needs. I worked at one place that survived uploading .zip files to Elastic Beanstalk for _years_ to update our PHP website. For some of the other data compliance/up-time/service availability stuff, I'd expect to have the revenue to cover the sort of overhead that introduces (otherwise you're just not charging enough, or the business isn't really as viable as you think).
ianpurton 6y ago
I've heard the Kubernetes vs VPS a lot now. I think it's the wrong way to think about it. It's more about a learning curve of best practices. - Building your application with a CI pipeline is a best practice. It helps to ensure your code builds from the repo and your tests run. - Building your app into a container is a best practice. The Dockerfile tracks your dependencies as code, and you can version control it. I can tell you from bitter experience these 2 are essential. Once your app is containerized you have many choices in the way you deploy it. You can use PaaS i.e. heroku or azure web apps. This is orchestration. If your service has a few containers and maybe some jobs then Kubernetes starts to look like a good option. K8 is $50 a month on digital ocean, you can run a few projects on that. Continuous deployment/delivery is a best practice. It takes the stress out of releases. Infrastructure as Code is a best practice. It's immutable infrastructure and once you have it gives you so many options in the way you manage failover etc. If you are concerned about uptime during deployments then Blue/Green is a great option too. We're not overcomplicating things, these skills will transfer from a side project to a billion dollar corp concerned with safety critical software. So you have to hit a few learning curves, but after that your a better developer.
- deleted 6y ago
  [deleted]
de6u99er 6y ago
It always depends on the use case and how much vendor lock-in you are willing to accept.
kweinber 6y ago
Here’s an oversimplified rule of thumb: Try not to spend more than a tenth your time on infrastructure or build if you are developing a product. If you find you are spending more than that on manual stuff, try to automate etc. to get back below that. If you are scaling staff, you may want up to a tenth the engineering staff to be dedicated to these tasks.... as you scale up, more automation to stay at or below that level. Oversimplified.... but not bad to think about if you think you are overdoing it. If you are way below a tenth.... really think about if you are counting all the manual workarounds and pieces of overhead honestly....or if you are giving up scalability, reliability or flexibility by not controlling your infrastructure.
throw1234651234 6y ago
There is a fairly in-depth article on which modern deployment strategies are useful, and which are a waste of time for mid-size companies as of 2020 here: https://medium.com/swlh/worthwhile-modern-deployment-strategies-a-full-guide-a692363de186 https://medium.com/swlh/worthwhile-modern-deployment-strateg...
BruceBlacklaws 6y ago
The logic behind this article is just bullshit It's not possible to operate at our scale "using a single Linode VPS" or whatever that may mean
nelsonenzo 6y ago
and when he says "you", he means tiny little side projects that do nearly nothing.
deleted 6y ago
[deleted]
nkotov 6y ago
My team is actually in the process of solving this problem ourselves. At the end of the day, k8s is overkill for most people, and using the nice UI/UX on top of K8s I feel will have devastating effects down the road when you need to troubleshoot, manage, and migrate.
Zelphyr 6y ago
Many people keep saying something to the effect of, "There is a right time to scale a system and we need to focus on that." This is true. There is a right time to scale up. In my experience, you rarely know when that might be except in hindsight. I worked at a startup once and we built the product in a way that it should have been able to scale up pretty well. Then one of the co-founders went on a national news channel and started talking about our company. Our servers got absolutely slammed with traffic. We did the best we could but it didn't go well. Now, we should have known to expect increased traffic from an event like that, and, in fact, we did. We simply didn't know just how much traffic we were going to get, and, honestly, couldn't have known that. We had only had a tiny fraction of that traffic until that event. That was the best basis for comparison we had at that point in time. We updated and upgraded some things after that and the next time he went on that news channel, things were a lot better. Knowing when to scale up is a hard problem. You have hints, sure, but there's never a data point that says, "Ok, you have until July to migrate this system from a single VPS to a full Kubernetes build-out."
- Zelphyr 6y ago
  I would like to add that you'd be foolish to use my post as a reason for building a system to scale to high levels prematurely. I agree with the original article that doing so bogs you down in needless complexity and wastes time and resources. At that startup, we built the system to scale to where we were projecting traffic would be within a given time period. If that co-founder going on national television hadn't happened, our original plan would have been sufficient.
rajatsx 6y ago
That's so true! Not long ago, I wrote an Ansible playbook to install my open source software on a DigitalOcean VPS. I devoted so much time to learn Ansible as I had zero experience with it. I think I almost kept on working on that playbook for almost two months. Even after that, I was buggy. One day, I just woke up and had an epiphany that a BASH script could do the same! On top of that, the end users would not have to install Ansible in order to run the playbook.
dang 6y ago
If curious, past threads: You Don’t Need All That Complex/Expensive/Distracting Infrastructure - https://news.ycombinator.com/item?id=19358259 https://news.ycombinator.com/item?id=19358259 - March 2019 (121 comments) You Don’t Need All That Complex/Expensive/Distracting Infrastructure - https://news.ycombinator.com/item?id=19299393 https://news.ycombinator.com/item?id=19299393 - March 2019 (82 comments)
justizin 6y ago
This article is about a person who needs a brochure. They don't even need a Linode VPS. Also note this starts out by describing a person as making "thousands of dollars per month". If that is your goal, obviously, you should spend as little as possible on all of your overhead.
otikik 6y ago
The same applies to the frontend.
ummonk 6y ago
I mean, with the amount of time spent doing manual work, you often end up spending less time automating it away.
yongjik 6y ago
As of 2016, this was the list of all the machines needed to run stack overflow [1]: 4 Microsoft SQL Servers 11 IIS Web Servers 2 Redis Servers 3 Tag Engine servers 3 Elasticsearch servers 4 HAProxy Load Balancers 2 Networks 2 Fortinet 800C Firewalls 2 Cisco ASR-1001 Routers 2 Cisco ASR-1001-x Routers Next time you're debugging connection issues between your canary webserver kubernetes deployment and redis cluster misc-3, ask yourself: are we serving more traffic than stack overflow? [1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
- bob1029 6y ago
  I feel like we need a campaign to help developers understand just how powerful modern server hardware is when you can get at it directly. Kestrel is pushing 7 million plaintext requests per second on techempower. And their test rig is certainly not the best you can get these days. The cloud has ruined our perception of computing hardware. If I have a choice between owning a physical machine in order to meet a performance objective, or building out a 20 node k8 cluster in AWS for some notion of scalability, I am going to shell out for the physical machine and happily drive to the data center to install it myself. Who would honestly prefer more computers over fewer computers to manage? Even if its in the cloud its still a fucking mess. The entire game of software engineering is about managing complexity. Not creating more of it and making it someone else's problem.
  - debaserab2 6y ago
    So whose going to sign up to be on call 24/7 when your physical hardware breaks?
    - bob1029 6y ago
      Paying for someone to manage the hardware is included in my assumptions. If you are asking "who would consider the role?" the answer is - Anyone who wants the money. Who fixes Amazon's shit when it breaks at 3AM?
bob1029 6y ago
Our problem domain is bounded in such a way that we never have to worry about scaling beyond 1 host in production. For the last 6 years, we have survived on little more than .NET Core and SQLite. We utilize Self-Contained Deployments, so rolling out a new install for a customer is an unzip operation followed by sc.exe create invocation on a blank win2019 server. Because we use SQLite, we don't have to worry about configuring any separate infrastructure to store the business data. It all lives in a convenient folder called "db" right next to the binaries. We also built our software so that it is capable of rebuilding itself from source and performing an in-place upgrade. Then we put a fancy Blazor UI on top of that and made builds & deployments the problems of non-developers. Time/frustration savings here is beyond your wildest imagination. We have project managers competently communicating about client environments in terms of commit hashes now. "Oh Client X's production is still back at 2c8ac0, ill run the build at e8c0a2". This is something that isn't really feasible when you have your solution architecture scattered to the seven winds. Our entire stack lives inside the confines of a single process on a single host. Focusing on the vertical and making it as powerful and capable as possible rather than chasing delusions of grandeur has seemed to pay massive dividends so far. Maybe one day we get screwed and its all over because we never planned to scale to 2+ hosts, but we could never hope to get to that point in the first place if we were to chase a bunch of shiny bullshit around all day.
bedobi 6y ago
At a previous job, our build pipeline * Built the app (into a self contained .jar, it was a JVM shop) * Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment. * Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image. * Combined the AWS .ami with an appropriately sized EC2. * Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style. The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.
asiachick 6y ago
There is no reason it has to be that hard to have the scalable infra out of the box. There should be some setup out there, ideally portable, where setup is one command and now I have a working scalable demo app (multi-user todo, fb clone, instagram clone, etc...) That setup should include unit tests, integration tests, a/b deployment, rolling upgrades, versioning, local dev, staging servers, production servers, data backups, logging infra, etc... The inputs to get started should be as simple 10 questions. Domain names, service names, ... As is now, setting up a new scalable website is similar in some ways to trying to setup a new computer if linux, macos, windows didn't exist and you had to write your own OS from existing parts. There's no reason it should be this way. There's a set a reasonable defaults that would work for most projects. I wouldn't consider that too complex or over engineered. I'd consider it simplified the same way the fact get I get so much from existing OSes simplifies using my computers vs having to write my own OS for every new computer I buy
toss1 6y ago
>>"...every minute spent on infrastructure is a minute less spent shipping features." Completely true. This does not mean that we should spend zero minutes on infrastructure This does mean that we should ensure that every minute & dollar spent on infrastructure is well-spent - will actually produce, for the customers, the developers, and/or the company overall, a return of sufficient value. I used to be all about planning for scale early. But I've actually found it's better to plan to build one or two throw-away versions, built well as we can while being fast and with only minimal scaling features, THEN take the lessons learned (about real features, usage patterns, & loads that will need to be supported) and apply these to building out the more scaleable versions. The initial cheap, cheerful, & disposable versions provide initial traction, a lot of knowledge, and a known & limited technical & financial debt, but the 'killer feature' is zero technical lock-in when you go to design & build the scalable version. YMMV

213 points by ddtaylor 6y ago

Maybe it's more important to ask 'why' [do I need this infra]? Complexity shouldn't be the default but it can be the right answer

bovermyer 6y ago
As a "devops" engineer, I can say with total confidence that complexity is rarely the right answer. When it is, it's because that complexity is the lowest pain point in a calculation that has itself become complex.
- agustif 6y ago
  Sometimes you can add complexity to the whole stack by adding a new shiny layer that brings down complexity where you care most about it, in your user-facing UI/UX layer. Developers can handle complexity via code much more efficiently than users operating a site or app... Also, some of the nice developments of the recent explosion of serverless/as a service stuff, is that you can mostly try stuff for free, be happy on the free tier if you don't need scale, and use that scale when you need it without much hassle but paying the overprice of their fat margins. Also you should probably have an ejection plan in case scale + external provider becomes too expensive, you should always have a nice idea on how to run your stack if you need it...
Zelphyr 6y ago
Complexity is never the right answer. Complexity, in my experience, comes from two things: The person building that system doesn't understand the problem well enough and/or equates complexity with quality. Neither are good things.
jbn 6y ago
that's because there are 2 kinds of complexity: essential (or intrinsic) complexity and accidental complexity. When one uses k8s when one doesn't truly need it, it's accidental complexity. However some problems require k8s, where the engineered complexity of k8s helps solving the problem at hand. Of course the issue is that most people don't need k8s, whereas Google and the like do.

jameshart 6y ago

There is obviously some point, somewhere between ‘just running a website out of my spare bedroom’ and ‘Facebook’, where some of this infrastructure does become necessary. It is very important that you realize when you pass one of those points. It doesn’t help to just tell people ‘you don’t need kubernetes, you are not Facebook’ There are a lot of companies that aren’t Facebook, but they still have multimillion dollar revenue streams riding on service availability, with compliance obligations and employee paychecks on the line, and which need to be able to reliably release changes to production code every day, written by multiple development teams in different time zones, so... yeah, maybe they are a bit beyond just scp’ing their files up to a single Linux VPS.

tyingq 6y ago
Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql.
- ahofmann 6y ago
  This could be the final post to this discussion. You can't really get more to the point than that. We are running a large German browser game on individual VPS instances. For many years, with PHP, MySQL and Redis. It works and it runs just fine.
  - bombcar 6y ago
    It's not one that just recently had to start migrating from Flash, is it?
- jasode 6y ago
  >Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql. But PHP and MySql are only the surface level tools we see. There is also the unseen and that's where the immense complexity lies hidden under the surface. E.g. To help with scaling (which Friendster failed at), Facebook had separate MySql instances for each school. An interview with Facebook co-founder Dustin Moskovitz mentioned they had constant stress of emergencies and fires to put out. They had tons of custom tooling, Linux scripts, batch jobs, automation/orchestration/dashboard, etc and it's very likely the combination of all that ends up inventing an "invisible internal Kubernetes" without calling it "Kubernetes" to manage their server fleet. In other words, if Docker/Kubernetes/AWS were around, the early Facebook team might have offloaded some of the complexity onto a 3rd-party to reduce the all-nighters. But the tools didn't exist yet so they had to do it the hard way and invent "deployment orchestration" themselves. [EDIT to reply] Thanks for the "Tupperware" comment. A Google search leads to a Facebook post saying they renamed it to "Twine": https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin...

Cthulhu_ 6y ago

As always, it depends. Lots of people wish they had Facebook's challenges when it comes to scaling. 99% of applications simply don't. I've never seen a company I've worked with go for 'fancy' infrastructure because they needed to. It was always because they HOPE that they need to. Granted, re-engineering your application / architecture for scaling is expensive, but not as expensive as building the wrong thing in the first place. They should all have started off with a monolith and grow from there. I mean in all those cases we had just a single, 'monolithic' front-end without any problems.

mring33621 6y ago
Aspirational Infrastructure!

fendy3002 6y ago

I assume that premature infrastructure optimization (complexity, complex automation, kubernetes, etc whatever you say) is same with premature optimization, both are bad. However same with optimization, there are some best practices that are good to follow, such as avoiding N+1 queries. For me personally, that will be deploying with docker-compose, which usually yml already be created during development, and it'll be much easier to migrate later.

jameshart 6y ago
This is of course a tautology. Premature anything is bad. That’s what ‘premature’ means. It means you did it too soon, before the optimal time. Doing it later would have been better. But there is an optimal time. And even if you did it too soon, if you eventually needed it, at least you’ve done it. Better that than getting to it too late, surely?
- heroprotagonist 6y ago
  And then there's the "I've already done it this way a dozen times and I'm comfortable with it. There are benefits to it and it's not that much extra effort to set it up from the beginning and save 10x effort down the line."
  - 8note 6y ago
    You need that to go with "also it doesn't add another failure point/upgrading dependencies work before it's needed"

ic0n0cl4st 6y ago

This past year I’ve finally been able to hit the $200/hour rate as a k8s consultant. I feel like I’m selling snake oil. I’m working on my 7th implementation and none of my customers need kubernetes. Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using. My largest implementation was a $3m/year, 9 year old company who could run their entire infrastructure on a couple dozen ec2 instances. However they are under the delusion of “hitting it big and needing to be able to scale 100x”. Paid off my student loans this year, and soon my house, thanks to superfluous k8s integrations!

droopyEyelids 6y ago
You are the real job creator, as there will be more work after your migrations than there was before
deagle50 6y ago
I've been on the fence about this. If you don't mind, how did you get started?
- waynesonfire 6y ago
  Oh, I can tell you because I work with these asshats. They come in, deploy the technology, usually get promoted and then leave. The people left behind have to maintain their garbage and because promotions are only given to new feature work we're the suckers holding the bag.
  - ic0n0cl4st 6y ago
    Actually This is exactly how I accidentally started consulting on k8s. Three of my customers had engineers who advocated for k8s, did 50-75% of an implementation then left. I have told all of my customers they don’t need this, but they are convinced this is the way of the future.
    - deagle50 6y ago

1996 6y ago

So true. HN often admires complexity for complexity sake. Even without cloudflare, a static website on 5 VPS from 5 different provider on 5 different continents with a Round Robin DNS and a low TTL will get you most of the bangs for bucks Add geoIP to the mix (like from Azure) and you're golden, even if you are just scp'ing files to 5 VPS. If a DC burns down, remove the A record (or even better: have a script do it for you) and reroute to the remaining 4. Extra bonus: you can fire the DevOps team :)

tartoran 6y ago
Why fire the team and not just repurpose them? They’re already familiarized with some aspects of the product.
tpetry 6y ago
That‘s easy if you are just serving static html. But a database replicated on 5 continents? That‘s a lot of latency for replication.
jameshart 6y ago
That’s funny, sometimes when I visit your site, I get an HTML page that references script files and resources that come back as 404s. I wonder if it’s possible I’m hitting the site during a deployment and getting the new HTML page from one of your servers, but then the resource requests are going to a different server which hasn’t received the new version yet. I guess occasional broken images and script failures are unavoidable though. You’d need like, some dev ops engineers to solve a problem of that complexity.

jFriedensreich 6y ago

3 important factors: - if i use state of the art production grade technology i can talk about it in interviews, if i say oh well i have plenty experience running docker compose manually on a server, not so much. This is also where new technology can be tested without hurting someone, so even if something is totally overkill it can still make sense in the overall picture of all projects you work on - ci/cd is maybe not relevant to your users, but its about your own sanity, it forces complete documentation in code of what is done to build and ship, decouple it from local dev setups and make as deterministic as possible. Even small project benefit, have you ever come back after 2 years and took hours to remember the order of voodoo commands or lost a laptop and then had to install specific versions of cli tools? - the logic is totally changed by the simplicity of faas offerings. Nowadays i would maybe not build a gigantic kubernetes system, but instead of dealing with vms and servers skip right to cloudfare workers or cloud run and have the same benefits of k8 with even less work and more time to work on the features

rjbwork 6y ago
>- ci/cd is maybe not relevant to your users, but its about your own sanity, it forces complete documentation in code of what is done to build and ship, decouple it from local dev setups and make as deterministic as possible. Even small project benefit, have you ever come back after 2 years and took hours to remember the order of voodoo commands or lost a laptop and then had to install specific versions of cli tools? This to me is the biggest advantage of this sort of thing. Documentation and notes are great, but I'd rather just have the thing-in-itself available that does the job.
- jFriedensreich 6y ago
  Yeah and 13 years ago there was a similar argument about git. I heard a lot of developers say well its a small project and its just me, no way i'm dealing with repos, making commits and learn git or mercurial. This completely died out.
- Person5478 6y ago
  CI/CD doesn't require the cloud or containers.

Traubenfuchs 6y ago

Operations guys, "devops" as they are called now, remain the arch-wizards with grey beards. They love creating highly company-customized, incredibly complex, often times brittle infrastructure and deployments based on obscure scripts written in non mainstream languages based on plugins and libraries without documentation. The same opaqueness as in IT security, but in emergencies you can downsize/ignore IT security, but without the ops guys your product can not survive. Kubernetes, Helm and cloud-platform specific quirks were a godsend that cemented in this status for at least another decade. There is probably no job with higher job security in tech than this.

breckenedge 6y ago
> There is probably no job with higher job security in tech than this. Will the last person to leave please turn off AWS.
supercon 6y ago
So were it not for this era of cloud platforms and the other technical ”advancements” you mention, would we somehow be in a different place when it comes to maintaining and operating software? As someone in a devops team, I could argue that ops teams are rarely the ones deciding what technologies to use. If we follow the basic sales->development->maintenance -pipeline, as ops your task is simply to keep the service running and secure with the tools you are given. Sure there can be added complexity, but I doubt it is as systemic/general as you make it seem to be. Anyway, if this is seen as a problem the devs could simply take on the responsibility of learning those quirks themselves or produce the level of quality software that requires no maintanance whatsoever. But for some reason that hasn’t happened, eventhough the original idea behind devops was to blur this line. So until that day comes, I’ll sure enjoy whatever security one can really have in todays job market.
- Traubenfuchs 6y ago
  No, we would simply still use incomprehensible websphere setups. Now we got kubernetes as the application server of the current decade that's application language/framework agnostic. But it shouldn't be like this. Devops shouldn't have to exist. Everything should work like heroku, where even mediocre devs can quickly setup proper pipelines.

mercury_craze 6y ago

This isn't totally related to infrastructure but figuring out what you should build, from what you could build is something I've struggled with as I've moved about the engineering management ladder. My current role is a stone’s throw from solo developer (only developer at 4 person start-up) and while I've got total autonomy in the work I complete and the approach I take, it’s a terrible feeling discovering that you're not super-human and you need constraints like almost every engineer that's come before you. Like the author there's been a number of times where I've felt self-satisfied completing a piece of technical work, only to scratch my head a week later and wonder whether it was worth it. I can say with certainty that having a solid CI/CD pipeline has saved my bacon a few times so I'd probably recommend that as an early investment to make, internal tooling for some of the customer registration was a bit of a mistake and not as critical as I thought it would be which now feels like a waste of 3 weeks. I am yet to come up with any meaningful advice to pass on to developers in the same position. With time you get better at spotting when you can get away with hacking solutions together and when to put the time in, having a forward vision for a product really helps on focusing your thought and attention and time spent on making things visible to the rest of the business is rarely time wasted. Awareness of what’s going on with the rest of the industry is useful, but don’t obsess over the new hotness, build with what you know works and save your innovation tokens for a rainy day.

durnygbur 6y ago

I don't hear you from inside bastion in autoscaling cluster!

eric4smith 6y ago

Sometimes a nice dedicated server and Dokku is good enough. Backups of database offsite, assets in S3 and code in Git means you’re good for 90% of use cases. >> git push dokku master Done.

lukeschlather 6y ago

> Your users to don’t care how your code gets onto your server. I work in B2B and while sure, our users don't necessarily care but they also aren't the ones paying us money. The people paying us money have a huge set of questions about how code gets on our servers and Kubernetes is not everything but it helps a lot answering those questions. (Especially using GKE.)

0xbadcafebee 6y ago

Whether the infra is complex or not isn't going to make a huge difference to whether your product makes money. Some of the highest-revenue-generating products I have seen were tire fires under the hood. One product had multiple people working 24 hours a day to constantly hose down the multiple levels of problems that the incredibly poor design and terrible infrastructure created. But through the toil of those people in the backend, the product made hundreds of millions a year. Whether or not you use complex technology, it matters more whether it's well designed and well operated. If it's not, you will end up with a fire either way. It's just that "simple" fires are a lot easier to put out than "complex" ones. (this is literally true; a lot of fires should not be put out with water)

waynesonfire 6y ago

I went down this exact same path. Started simplifying everything and it did wonders for my velocity. It took me a while to realize that we have teams and teams of people maintaining all this crap I was trying to spin up myself. It just wasn't worth it. I went to basic principles, FreeBSD. systemd has taken Linux in the wrong direction in my opinion. I don't want anything fancy, I don't want to learn another persons resume building framework, I want to use technology that has proven longevity. Our technology space moves so fast, for my personal work, I have to leverage the first principle technologies that I have mastered over my career. The latest framework may make some problems easier to solve, but I don't deploy technology without achieving some level of competency. That takes significant time. I'm getting wiser with how I spend that time, it's not always easy to know. This process has also taught me why we have this proliferation of SaaS solutions. It's tempting, and maybe one day I'll be wise enough to realize that this is the route to take. However, I'm still able to leverage my core skills and open-source technology enough that SaaS solutions don't look attractive to me. I can see the benefits either way, so I suppose maybe in five years when I reflect I'll know if I made the right investment.

AlchemistCamp 6y ago
> systemd has taken Linux in the wrong direction in my opinion Can you elaborate on that a bit? Where does it fall short for you and what's your preferred tool?
- KronisLV 6y ago
  I've often heard people saying that systemd attempts to do too much and as a result is bloated, but personally i've found it to be sufficient for what i want to do - making certain software run and/or automatically restart on a server. Some of the alternatives that i've heard of: - SysVinit (i think older RPM distros used it) https://wiki.archlinux.org/index.php/SysVinit - OpenRC (i believe that Alpine Linux uses it as its default) https://wiki.archlinux.org/index.php/OpenRC - and here are some other links https://alternativeto.net/software/systemd/ For better or worse, i've mostly just stuck to using systemd because it's good enough - and init systems that require scripts to be written manually intimidate me, since it seems like it'd be easy to make mistakes and the scripts wouldn't be standartized enough. Personally, i prefer to create a configuration file that just defines where the software is, how to keep track of its PID, where and how to get configuration for it, as well as what user/group to execute it as (among other things). Maybe the poster you're replying to will provide some arguments against systemd and situations where it isn't very nice to use.

dominotw 6y ago

Its not just engineers that want to build complex infrastructure though. This a massive misconception and shifting the blame to 'nerdy engineers' doing 'nerdy' stuff instead of solving real problems. Startup that worked at raised whole bunch of money and hired tons of engineers. They were all tasked to "build something big". eg: it was expected of data engineers to setup "big data architecture" with complex data-lakes, job schedulers , spark clusters, streaming jobs and all sorts of weird stuff. No one really understood why we are building all this stuff but we never got real answers from the management.

foobarbecue 6y ago

I find there's a bit of an "if you give a mouse a cookie" effect with this stuff. I recently wanted to organize a data processing pipeline at our company. In the past, the system was: there's a bunch of scripts, some of them call each other, and sometimes a person logs in and kicks off one or more scripts without recording their inputs and sending the output to wherever they feel like. I thought implementing this on Argo Workflows would allow us to keep track of previous runs and generally clarify things. Well, Argo runs on Kubernetes. Also, it kinda wants your data in a bucket. And argo kubernetes wants kustomize and skaffold and or / helm and soon I'm installing a package manager that manages package managers and I spend half my time managing systems we don't really need. I'm learning I need to to draw the line and deny Argo some of the things it wants. No, we don't need to run minio and call our files "artefacts." The filesystem works just fine thanks, we don't need another layer of it. This means we can't follow some of the Argo examples and lose some of its functionality.

jwr 6y ago

I ran my B2B SaaS on a single (physical) server for the first three years or so. I then switched to a setup with three production servers, and three staging servers (in a different data center), mostly to have the comfort of a distributed database, and a staging system to try things out. Physical servers provide enormous computing capacity these days. And managing them isn't much different from virtual servers: you should be using a tool like ansible anyway. The only difference is that when I spin up my development cluster I use terraform to generate the ansible inventory, and for physical servers the inventory is static. I do not anticipate needing more capacity or scalability anytime within my planning horizon. I keep seeing people investing tons of time and effort into things like Kubernetes and I honestly have no idea why. It complicates things enormously for no good reason: the benefits, if you aren't Google, aren't interesting at all, and the drawbacks are huge. The biggest drawback for me (apart from the cost of learning, building, and maintaining the infrastructure) is the complexity and resulting unpredictable failure modes. There are things which I'd say are really important for infrastructure, though: * a working and tested backup strategy * disaster recovery plan (what happens when your entire datacenter burns down) * managing servers via a tool like ansible * regular testing of various scenarios (including setting up an entire system from scratch on new servers, with data recovery from backups) * a database that reliably preserves all of your customers' data when any one of your servers goes down (this unfortunately excludes many databases and makes the usual "just use Postgres" a poor choice, now flame me to high heaven) I can't imagine a small business where Kubernetes makes business sense. Do people seriously have problems with scaling to hundreds of machines? Scalability problems with huge traffic peaks that actually cause server load? I'm having trouble keeping my CPUs busy for any meaningful amount of time, and I don't spend much time on optimizations, I just avoid doing stupid things.

mwcampbell 6y ago
> * a database that reliably preserves all of your customers' data when any one of your servers goes down (this unfortunately excludes many databases and makes the usual "just use Postgres" a poor choice, now flame me to high heaven) What do you suggest instead? My understanding was that Postgres, when properly set up with backups and replication, is the most reliable open-source database.
6y ago

twodave 6y ago

> Your goal, when launching a product, is to build a product that solves a problem for your users. This is the main takeaway, in my opinion. I've been at small companies who were under-architected "because we can" and large companies who were over-architected because someone in the leadership chain was enamored with a particular technical choice. In the end it's a cost curve and depends heavily on the nature of your business. I can say personally if the business is even mildly complex (not necessarily large, this could mean a web app + a mobile app + a single RDBMS) I'd sleep better having some AWS (or other cloud) alerts set up to monitor uptime/etc. Others have mentioned CI/CD, but again, heavily depends on your needs. I worked at one place that survived uploading .zip files to Elastic Beanstalk for _years_ to update our PHP website. For some of the other data compliance/up-time/service availability stuff, I'd expect to have the revenue to cover the sort of overhead that introduces (otherwise you're just not charging enough, or the business isn't really as viable as you think).

ianpurton 6y ago

I've heard the Kubernetes vs VPS a lot now. I think it's the wrong way to think about it. It's more about a learning curve of best practices. - Building your application with a CI pipeline is a best practice. It helps to ensure your code builds from the repo and your tests run. - Building your app into a container is a best practice. The Dockerfile tracks your dependencies as code, and you can version control it. I can tell you from bitter experience these 2 are essential. Once your app is containerized you have many choices in the way you deploy it. You can use PaaS i.e. heroku or azure web apps. This is orchestration. If your service has a few containers and maybe some jobs then Kubernetes starts to look like a good option. K8 is $50 a month on digital ocean, you can run a few projects on that. Continuous deployment/delivery is a best practice. It takes the stress out of releases. Infrastructure as Code is a best practice. It's immutable infrastructure and once you have it gives you so many options in the way you manage failover etc. If you are concerned about uptime during deployments then Blue/Green is a great option too. We're not overcomplicating things, these skills will transfer from a side project to a billion dollar corp concerned with safety critical software. So you have to hit a few learning curves, but after that your a better developer.

deleted 6y ago
[deleted]

de6u99er 6y ago

It always depends on the use case and how much vendor lock-in you are willing to accept.

kweinber 6y ago

Here’s an oversimplified rule of thumb: Try not to spend more than a tenth your time on infrastructure or build if you are developing a product. If you find you are spending more than that on manual stuff, try to automate etc. to get back below that. If you are scaling staff, you may want up to a tenth the engineering staff to be dedicated to these tasks.... as you scale up, more automation to stay at or below that level. Oversimplified.... but not bad to think about if you think you are overdoing it. If you are way below a tenth.... really think about if you are counting all the manual workarounds and pieces of overhead honestly....or if you are giving up scalability, reliability or flexibility by not controlling your infrastructure.

throw1234651234 6y ago

There is a fairly in-depth article on which modern deployment strategies are useful, and which are a waste of time for mid-size companies as of 2020 here: https://medium.com/swlh/worthwhile-modern-deployment-strategies-a-full-guide-a692363de186 https://medium.com/swlh/worthwhile-modern-deployment-strateg...

BruceBlacklaws 6y ago

The logic behind this article is just bullshit It's not possible to operate at our scale "using a single Linode VPS" or whatever that may mean

nelsonenzo 6y ago

and when he says "you", he means tiny little side projects that do nearly nothing.

deleted 6y ago

[deleted]

nkotov 6y ago

My team is actually in the process of solving this problem ourselves. At the end of the day, k8s is overkill for most people, and using the nice UI/UX on top of K8s I feel will have devastating effects down the road when you need to troubleshoot, manage, and migrate.

Zelphyr 6y ago

Many people keep saying something to the effect of, "There is a right time to scale a system and we need to focus on that." This is true. There is a right time to scale up. In my experience, you rarely know when that might be except in hindsight. I worked at a startup once and we built the product in a way that it should have been able to scale up pretty well. Then one of the co-founders went on a national news channel and started talking about our company. Our servers got absolutely slammed with traffic. We did the best we could but it didn't go well. Now, we should have known to expect increased traffic from an event like that, and, in fact, we did. We simply didn't know just how much traffic we were going to get, and, honestly, couldn't have known that. We had only had a tiny fraction of that traffic until that event. That was the best basis for comparison we had at that point in time. We updated and upgraded some things after that and the next time he went on that news channel, things were a lot better. Knowing when to scale up is a hard problem. You have hints, sure, but there's never a data point that says, "Ok, you have until July to migrate this system from a single VPS to a full Kubernetes build-out."

Zelphyr 6y ago
I would like to add that you'd be foolish to use my post as a reason for building a system to scale to high levels prematurely. I agree with the original article that doing so bogs you down in needless complexity and wastes time and resources. At that startup, we built the system to scale to where we were projecting traffic would be within a given time period. If that co-founder going on national television hadn't happened, our original plan would have been sufficient.

rajatsx 6y ago

That's so true! Not long ago, I wrote an Ansible playbook to install my open source software on a DigitalOcean VPS. I devoted so much time to learn Ansible as I had zero experience with it. I think I almost kept on working on that playbook for almost two months. Even after that, I was buggy. One day, I just woke up and had an epiphany that a BASH script could do the same! On top of that, the end users would not have to install Ansible in order to run the playbook.

dang 6y ago

If curious, past threads: You Don’t Need All That Complex/Expensive/Distracting Infrastructure - https://news.ycombinator.com/item?id=19358259 https://news.ycombinator.com/item?id=19358259 - March 2019 (121 comments) You Don’t Need All That Complex/Expensive/Distracting Infrastructure - https://news.ycombinator.com/item?id=19299393 https://news.ycombinator.com/item?id=19299393 - March 2019 (82 comments)

justizin 6y ago

This article is about a person who needs a brochure. They don't even need a Linode VPS. Also note this starts out by describing a person as making "thousands of dollars per month". If that is your goal, obviously, you should spend as little as possible on all of your overhead.

otikik 6y ago

The same applies to the frontend.

ummonk 6y ago

I mean, with the amount of time spent doing manual work, you often end up spending less time automating it away.

yongjik 6y ago

As of 2016, this was the list of all the machines needed to run stack overflow [1]: 4 Microsoft SQL Servers 11 IIS Web Servers 2 Redis Servers 3 Tag Engine servers 3 Elasticsearch servers 4 HAProxy Load Balancers 2 Networks 2 Fortinet 800C Firewalls 2 Cisco ASR-1001 Routers 2 Cisco ASR-1001-x Routers Next time you're debugging connection issues between your canary webserver kubernetes deployment and redis cluster misc-3, ask yourself: are we serving more traffic than stack overflow? [1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

bob1029 6y ago
I feel like we need a campaign to help developers understand just how powerful modern server hardware is when you can get at it directly. Kestrel is pushing 7 million plaintext requests per second on techempower. And their test rig is certainly not the best you can get these days. The cloud has ruined our perception of computing hardware. If I have a choice between owning a physical machine in order to meet a performance objective, or building out a 20 node k8 cluster in AWS for some notion of scalability, I am going to shell out for the physical machine and happily drive to the data center to install it myself. Who would honestly prefer more computers over fewer computers to manage? Even if its in the cloud its still a fucking mess. The entire game of software engineering is about managing complexity. Not creating more of it and making it someone else's problem.
- debaserab2 6y ago
  So whose going to sign up to be on call 24/7 when your physical hardware breaks?
  - bob1029 6y ago
    Paying for someone to manage the hardware is included in my assumptions. If you are asking "who would consider the role?" the answer is - Anyone who wants the money. Who fixes Amazon's shit when it breaks at 3AM?

bob1029 6y ago

Our problem domain is bounded in such a way that we never have to worry about scaling beyond 1 host in production. For the last 6 years, we have survived on little more than .NET Core and SQLite. We utilize Self-Contained Deployments, so rolling out a new install for a customer is an unzip operation followed by sc.exe create invocation on a blank win2019 server. Because we use SQLite, we don't have to worry about configuring any separate infrastructure to store the business data. It all lives in a convenient folder called "db" right next to the binaries. We also built our software so that it is capable of rebuilding itself from source and performing an in-place upgrade. Then we put a fancy Blazor UI on top of that and made builds & deployments the problems of non-developers. Time/frustration savings here is beyond your wildest imagination. We have project managers competently communicating about client environments in terms of commit hashes now. "Oh Client X's production is still back at 2c8ac0, ill run the build at e8c0a2". This is something that isn't really feasible when you have your solution architecture scattered to the seven winds. Our entire stack lives inside the confines of a single process on a single host. Focusing on the vertical and making it as powerful and capable as possible rather than chasing delusions of grandeur has seemed to pay massive dividends so far. Maybe one day we get screwed and its all over because we never planned to scale to 2+ hosts, but we could never hope to get to that point in the first place if we were to chase a bunch of shiny bullshit around all day.

bedobi 6y ago

At a previous job, our build pipeline * Built the app (into a self contained .jar, it was a JVM shop) * Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment. * Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image. * Combined the AWS .ami with an appropriately sized EC2. * Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style. The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.

asiachick 6y ago

There is no reason it has to be that hard to have the scalable infra out of the box. There should be some setup out there, ideally portable, where setup is one command and now I have a working scalable demo app (multi-user todo, fb clone, instagram clone, etc...) That setup should include unit tests, integration tests, a/b deployment, rolling upgrades, versioning, local dev, staging servers, production servers, data backups, logging infra, etc... The inputs to get started should be as simple 10 questions. Domain names, service names, ... As is now, setting up a new scalable website is similar in some ways to trying to setup a new computer if linux, macos, windows didn't exist and you had to write your own OS from existing parts. There's no reason it should be this way. There's a set a reasonable defaults that would work for most projects. I wouldn't consider that too complex or over engineered. I'd consider it simplified the same way the fact get I get so much from existing OSes simplifies using my computers vs having to write my own OS for every new computer I buy

toss1 6y ago

>>"...every minute spent on infrastructure is a minute less spent shipping features." Completely true. This does not mean that we should spend zero minutes on infrastructure This does mean that we should ensure that every minute & dollar spent on infrastructure is well-spent - will actually produce, for the customers, the developers, and/or the company overall, a return of sufficient value. I used to be all about planning for scale early. But I've actually found it's better to plan to build one or two throw-away versions, built well as we can while being fast and with only minimal scaling features, THEN take the lessons learned (about real features, usage patterns, & loads that will need to be supported) and apply these to building out the more scaleable versions. The initial cheap, cheerful, & disposable versions provide initial traction, a lot of knowledge, and a known & limited technical & financial debt, but the 'killer feature' is zero technical lock-in when you go to design & build the scalable version. YMMV