9 ms·
There is obviously some point, somewhere between ‘just running a website out of my spare bedroom’ and ‘Facebook’, where some of this infrastructure does become
by jameshart 6y ago
There is obviously some point, somewhere between ‘just running a website out of my spare bedroom’ and ‘Facebook’, where some of this infrastructure does become necessary.
It is very important that you realize when you pass one of those points.
It doesn’t help to just tell people ‘you don’t need kubernetes, you are not Facebook’
There are a lot of companies that aren’t Facebook, but they still have multimillion dollar revenue streams riding on service availability, with compliance obligations and employee paychecks on the line, and which need to be able to reliably release changes to production code every day, written by multiple development teams in different time zones, so... yeah, maybe they are a bit beyond just scp’ing their files up to a single Linux VPS.
- tyingq 6y agoFacebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql.
- ahofmann 6y agoThis could be the final post to this discussion. You can't really get more to the point than that. We are running a large German browser game on individual VPS instances. For many years, with PHP, MySQL and Redis. It works and it runs just fine.
- bombcar 6y agoIt's not one that just recently had to start migrating from Flash, is it?
- jasode 6y ago>Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql. But PHP and MySql are only the surface level tools we see. There is also the unseen and that's where the immense complexity lies hidden under the surface. E.g. To help with scaling (which Friendster failed at), Facebook had separate MySql instances for each school. An interview with Facebook co-founder Dustin Moskovitz mentioned they had constant stress of emergencies and fires to put out. They had tons of custom tooling, Linux scripts, batch jobs, automation/orchestration/dashboard, etc and it's very likely the combination of all that ends up inventing an "invisible internal Kubernetes" without calling it "Kubernetes" to manage their server fleet. In other words, if Docker/Kubernetes/AWS were around, the early Facebook team might have offloaded some of the complexity onto a 3rd-party to reduce the all-nighters. But the tools didn't exist yet so they had to do it the hard way and invent "deployment orchestration" themselves. [EDIT to reply] Thanks for the "Tupperware" comment. A Google search leads to a Facebook post saying they renamed it to "Twine": https://engineering.fb.com/2019/06/06/data-center-engineering/twine/ https://engineering.fb.com/2019/06/06/data-center-engineerin...
- jeffffff 6y agoit's called 'tupperware'
- tyingq 6y agoThat's recent though. I'm saying they scaled up quite a long way before doing anything like that. From the outside it looks like they didn't really branch out into things like this until 2008 or so.
- jasode 6y ago>From the outside it looks like they didn't really branch out into things like this until 2008 or so. No, even before the Newsfeed feature rollout of September 2006, they already had crazy complexity orchestrating multiple servers for many schools. They were only a "simple PHP+MySql" website when it was February 2004 with only Harvard students on it. Dustin said he was the one in charge of spinning up new servers for each new school and he said it was a nightmare of engineering effort.
- Person5478 6y agoIIRC, they were doing things like patching bittorrent to prefer closer IP's to pull from during their deploys. So it would roll through their network from 1 side to the other without just completely saturating the entire network. But the point still remains, they had challenges to solve, but they were also simple, which made it possible TO solve those challenges.
- mattmanser 6y agoBut today, 16 years of Moore's law later, you could now have gone way, way, way, way past that. They probably could have done most of America on a single server. For example, even in 2009 Stack Overflow was running off two dedicated servers: https://stackoverflow.blog/2009/01/12/new-stack-overflow-server-glamour-shots/ https://stackoverflow.blog/2009/01/12/new-stack-overflow-ser...
- 6y ago
- rualca 6y ago> Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql. Isn't Facebook known for hiring top world experts such as the likes of Andrei Alexandrescu to hyperoptimize all the way down their whole stack? That hardly sounds like running vanilla PHP with your generic nem ache and mysql. And how do you exactly think they get so many 9s in their reliability? Is this plausible with the "a single linode box ought to do" nonsense?
- tyingq 6y agoHe got there in 2009. Facebook did run for a long time with a fairly "vanilla" setup, yes. A hacker got ahold of their "index.php" in 2007: https://gist.github.com/nikcub/3833406 https://gist.github.com/nikcub/3833406 It is decidedly not a tour-de-force of technical prowess, just typical old school "bag of functions in the same namespace" PHP.
- redis_mlc 6y ago> Facebook is kind of an interesting example, as they got pretty far into "hyperscale" with mostly PHP, memcached, and Mysql. Those are fantastic technologies and each is remarkable at what it does. Besides that, those are some of the easiest skill sets to hire for.
- bombcar 6y agoThere are ways to quantify it - but a real telling point is if your service is down because of the complexity of tooling you don’t fully understand. Many problems can be solved with other methods - especially if you do NOT have global all-sync’d requirements like Twitter and Facebook. Many SaaS (example: Salesforce) are perfectly acceptable sharding customers to particular instances thereby limiting downtime if something goes wrong. And always keep in the back of your mind that the largest VPS or servers are very performant today - having to scale sideways may be masking inefficiencies in the code.
- jameshart 6y agoOr if your service is down because the single hosting site with your Linux box in it is on fire Also, I assume sales force engineers aren’t SSHing into each shard and running docker-compose manually on each one. Obviously you need Goldilocks infrastructure - not too little, but not too much. All I’m saying is it doesn’t go ‘one Linux box, one Linux box, one Linux box, reach Facebook size, build your own data center’.
- callmeal 6y ago>Also, I assume sales force engineers aren’t SSHing into each shard and running docker-compose manually on each one. Nothing wrong with that if all you have is 10-15 shards (or whatever your annoyance upper limit is). Might even be faster and cheaper than spending hours learning/debugging/figuring out edge cases on something else.
- protonimitate 6y agoThis is a good point. I wish there was more discussion around when to consider making these decisions, how to transition, what types of systems are in the middle ground between bedroom and facebook, etc. It seems all of the articles/posts that get are attention are the "you don't need it ever" opinions, but those are incredibly short sighted and assume that everyone is recklessly applying complex infrastructure. In general, it would be nice to have more nuanced discussions around these choices rather than the low hanging fruit that is always presented.
- richardw 6y agoNo simple advice will cover all the cases but not a bad place to start: https://basecamp.com/gettingreal/04.5-scale-later https://basecamp.com/gettingreal/04.5-scale-later
- cesaref 6y agoWhen you are in a position to pay someone full time to look after the web infrastructure of your project, that's when you should think this stuff through. Before that, don't worry about the tech, go for what you know, and expect it to need totally re-writing at some point. Let's face it, this stuff moves quickly.
- blacktriangle 6y agoThat's the best metric I've heard. When your Heroku bill is the same as what a full time ops person would cost, start looking to hire.
- jacquesm 6y agoI would say it has to be at least double that and maybe even higher.
- rualca 6y agoIf you are already deploying to Heroku, you are already doing ops. It makes zero sense to presume you don't need to know what you're doing or benefit from automating processes just because you can't hire a full-time expert.
- rualca 6y ago> There is obviously some point, somewhere between ‘just running a website out of my spare bedroom’ and ‘Facebook’, where some of this infrastructure does become necessary. Frankly, that blog post reads terribly miopic. The author essentially looks at his personal experience setting up his single-dev personal project, complains that he spent some time jump-starting a basic CICD pipeline, and from there he proceeds to argue that no one needs any of that. Well, how does he expect to check whether their changes break or not the deployment? How does he expect to deploy his stuff? How does he plan to get multiple commits to play nice prior to deployment? How does he expect to roll back changes? How does he plan to address request spikes, either from normal traffic or attacks? Does he plan to get up at 3AM on a Tuesday to scale up deployments or does he plan to let his site crap out during the night and just get back to it in the morning? And somehow we are supposed to believe that this naive belief that getting features out of the door is all that matters, and basic ops is not needed by anyone?
- thisiszilff 6y agoYeah, I kind of agree with the author, but not 100%. The takeaway shouldn't be "don't use CI, kubernetes, etc." but rather "don't learn something new" when your goal is to get a product out the door. If you already know kubernetes, then it might seem silly to not use it given how it simplifies things. It's only when you don't know kubernetes that you'll run into problems.
- rualca 6y ago> but rather "don't learn something new" when your goal is to get a product out the door. I don't agree at all, because that argument is a poorly put-together strawman. No one ever had to choose between setting up a CICD pipeline and churn out features. Ever. In fact, continuous deployment makes it possible for everyone to get features out if the door as fast as they possibly can, and ensure deployments are two-way door thing instead of a finicky one-way door pray-that-it-works event. The door where features have go get through is not JIRA, but the prod environment. It's inconceivable how some people try to pretend that automating operations adds no value. And hell, CICD nowadays is trivial. You know why? Because of tools like Kubernetes/Cloudformation/CDK/Ansible/etc.
- jrochkind1 6y agoIs there good tooling that's actually in between "just running a website out of my spare bedroom" and "kubernetes"? I think maybe not.
- bearjaws 6y agoServerless compute in your favorite cloud provider IMO. Been using serverless framework for over a year now, handles my deployments, load balancers, API gateway and obviously Lambda. A proper CI/CD setup is only about 8 hours of work with a monolith app.
- salmonlogs 6y agoWhat about a managed service like EKS or GKE? A minimum k8s config is perhaps 20 lines to spin up a service and expose the ports - that feels pretty achievable for bedroom coder?
- skrtskrt 6y agoA couple ansible scripts to deploy the k3s kubernetes distribution onto cloud VMs.* Add LetsEncrypt to the Traefik install for https Use the default Helm chart to make your k8s deployments for your apps - you'll probably only need to tweak a couple values to start - ports and ingress rules. * k3s comes with Traefik pre-baked to listen on ports 80 and 443 so you can run it "bare metal" without a cloud provider's load balancer service or any special tweaks to Traefik config. If you just Helm install any of the standard k8s ingress providers like the standard Traefik chart, Consul, or Nginx, you have to tweak a bunch to get them working on privileged external ports 80 and 443.
- theptip 6y agoI found Dockerizing my app really early to be a good bridge to k8s. Build container locally, test, push. On Linux box, pull the new app container, stop the old app container, start the new app container. Separate nginx reverse proxy also in docker so if you’re fast enough you don’t drop any packets. I’d say docker is worth using from the get-go, but if you are most familiar with raw Linux probably just use that for your first iteration.
- marcosdumay 6y ago
- MR4D 6y ago> There is obviously some point... My take is simple - if Craigslist doesn't need it, then I probably don't either. (I may want it, but that's different). Definitely not perfect, but it works for me. Obviously more/better guideposts would make this a much better yardstick.
- jameshart 6y agoI duunno exactly what Craigslist is running on today, but some slides from 2012[1] include items like running MongoDB over "3 shards across 3 node replica sets" with "duplicate config in 2nd data center"; a "sharded multi-node" redis cluster; multiple "vertically partitioned" MySQL clusters; the actual application stack on top of that being mod_perl on Apache, and it was all operated by a 'small but growing' dedicated team. It might or might not be notable that the growth of that tech team included, one year later, hiring Larry Wall[2]. Given that, I'm not sure how to apply 'if Craigslist doesn't need it, I probably don't' as a general principle.... [1] https://www.slideshare.net/jzawodn/living-with-sql-and-nosql-at-craigslist-a-pragmatic-approach/23-Craigslist_Tech_FAQs_Selfhosted_no https://www.slideshare.net/jzawodn/living-with-sql-and-nosql... [2] http://blog.craigslist.org/2013/10/15/artist-formerly-known/ http://blog.craigslist.org/2013/10/15/artist-formerly-known/
- MR4D 6y agoI see language as a personal choice, and while I wouldn't run perl today (I'm partial to Python, myself), I think there stack is fairly simple - tried & true language (however quirky it may be), no virtualization, simple metal hardware stack, and yet they serve an incredible number of users. In that sense, I'm sure hiring Larry was a no-brainer. They are no FB, but they're bigger than almost anything else on the web [0]. I doubt most people will build anything bigger in their careers. I certainly won't. A huge caveat might be app features. Clearly running something like travelocity could require more pieces, and something like a BI app in a fortune 100 company would have many more connectors to data sources. But in general, they've done pretty well using only newer versions of what worked way back in 1999, and to me, that's just incredible. [0] - https://www.semrush.com/blog/most-visited-websites/ https://www.semrush.com/blog/most-visited-websites/
- Spooky23 6y agoAnother factor is the operating model. People bitching about K8S, systemd, etc miss the point that these systems allow a qualified generalist ops admin pop in and fix stuff without knowing the app well. For a little company with <500 servers, it doesn’t matter. As size and complexity increase, figuring out how the guy left last year setup FreeBSD becomes a liability.
- nmfisher 6y agoMan, I didn't realize 500 servers was still "little company" level!
- Spooky23 6y agoMarginal cost for servers is very small, and with containers, etc, they multiply like bunnies. Many of our devs are running 8-15 VMs just for their dev work. Depending on what you do and how, it’s easy to hockey stick the count!
- marcosdumay 6y agoThat surfaces a very important communication problem. If you consider 500 servers a "small datacenter", then, of course, you will say that K8S adds value to small operations.
- greggyb 6y agoA qualified generalist ops admin should be productive on any modern UNIX or UNIX-like within a couple weeks. Figuring out how the guy last year set up FreeBSD consists of similar effort and process as figuring out how the guy last year set up Debian or CentOS or SUSE.
- Spooky23 6y agoIt can be challenging. I basically bought my house based on figuring that out for people in the old days. It’s not an issue with FreeBSD or anything per se. Usually there’s a genius superhero guy who does whatever, who burns out/quits/otherwise goes away, and the business is stuck.
- jacquesm 6y agoFair point. But people just love to over-complicate things and it is far more often that you come across absolutely ridiculous infrastructure monstrosities for very small business than that you come across the opposite.
- PragmaticPulp 6y agoI’ve found that career FOMO drives a lot of the desire to overcomplicate infrastructure. It’s hard to look at job listings from prestigious companies listing K8s and other buzzwords as requirements without feeling some desire to add those buzzwords to your own resume.
- theptip 6y agoI’d suggest that when you have paying customers, and your duck tape deploy process is starting to feel risky to run during business hours, that’s a good time to do your first taking of stock. Before that point it should be the fastest thing you can wire up given your own personal tooling expertise. (I’d argue that k8s is actually lean enough to be a competitor here if you’re already an expert and you use a managed service like GKE). Your infra should be in service to some uptime/performance/DX goals. Define those (honestly!) and then you can figure out what you need to improve. In particular if no customer has ever noticed downtime, and you don’t think you’d lose business if someone hit a 30-min outage, then you probably don’t need to upgrade. Twitter was notoriously unreliable in the early days and they made it, for example.
- yowlingcat 6y agoI agree that there's a problem with that kind of a false dichotomy, but just as Rails is "good enough" for a vast majority of workloads (where you specialize off the parts that need it when you need it), I'd say the same about Heroku. The DX scales up from 0 to "big enough" very well, and once you get beyond that, it's reasonably straightforward to take the parts you need inside of AWS inside. I've noticed sometimes that there is often an ideological argument for or against open source (and sometimes to an extreme extent) buried in arguments for complex/expensive/distracting infrastructure, whether directly or indirectly. It gets coded as "portability" and so that's why you end up managing your own metal on Kubernetes. But you've got to really ask yourself whether if you had to pick one, you'd index on convenience or portability. There are some businesses where that portability actually is that important. But there are a lot of businesses for which convenience is first in priority. Determining what your priorities are is the first step. From there, I think you have options.
- deckard1 6y ago> It is very important that you realize when you pass one of those points. It. Does. Not. Matter. You're not going to be the same guy that sets up Facebook's garage server in 2004 and then scales it out to a multiple data center monster it is today. If you are that guy, then I really feel sorry for you. Something went seriously wrong with your equity (or lack thereof) and you are doing janitor shit when you should be up in the tower with the other C-levels.
- SiVal 6y agoPart of what you're saying is a good point, IMO. The people who transition the infrastructure from Level 0 to Level 1 won't be the same people in charge of infrastructure when it's time to transition from Level 17 to Level 18. But however many levels there are from garage infrastructure to Big Tech infrastructure--even if there were only 2--someone will always have to decide that the time has come to transition from This Level to Next Level. Different expertise may be required to transition from L0 -> L1 vs L10->L11 vs L30->L31, but that expertise still matters. And if I were the one who had to decide at any of those levels, I would want a well-designed plan for the next transition or two that was of the form: 'When [some measurement] reaches [some threshold] we will [change the infrastructure in the following ways...]'
- deckard1 6y ago> And if I were the one who had to decide at any of those levels, I would want a well-designed plan for the next transition or two Right. This was really my (poorly-expressed) original point. Deciding how and when to scale is going to be a team and organization effort and decision. If it does come down to one guy deciding on something, then that's going to be the CTO. And if he/she is a good CTO, their decision is based on the feedback of a dozen people under them. Scaling is a progression and no matter how fast your growth is, you do have time to come up with a solid plan and execute that plan. You don't need Facebook scale on day one even if you do become the next Facebook.
- oppositelock 6y agoIt's always a judgment call with how to complicate your infrastructure. If you succeed, sometime in the future, you will have lots of engineering resources to manage the complexity of a "web scale" service. However, this level of complexity is high. So, today, what is more important is ease of development, a friendly software iteration cycle, and easy testing on developer machines, since you don't have FB's or Google's test infrastructure yet. You have to shoot for something that you can build fairly simply that'll carry you the next year or two, and think about it in terms of API boundaries, so that you can replace any particular piece, with a faster, more complex piece in the future when you need it. Your appropriate design for today isn't going to come from this site, or from reading AWS or GCP press materials, but from people who have been through this multiple times and have some experience in evolving small systems into big systems. They're hard to find.
- kabdib 6y agoYou would be surprised how far "scp to a bunch of Linux boxes" plus some load balancers will get you.
- rualca 6y agoIn that case just imagine how far you'd go if instead of manually scp to a bunch of servers you instead install a tool like Ansible, add your bunch of linux boxes in a list, write down in a script what you need to scp to those boxes, and from thereon just run the script whenever you need to update them. But somehow that's too much because it's now ops and no one needs to scp to boxes safely and reliably like that.
- kodah 6y agoInfrastructure is important whether you're running a bare metal server or VM's on someone else's computer in the cloud. Different infrastructure solves different problems. Container orchestrators solve a very specific density problem. Back in the day we recklessly threw a machine together and either put one or an infinite amount of apps on it. If you put one then you've wasted all remaining resources on the machine. If you put an infinite amount of apps on it then you've effectively created a hackers wet dream much less noisy neighbor hell. Can you still have containers that demand hoggish resources? Sure, configuration is up to your imagination and your expertise - just like with virtual machines and BSD jails. That said, now you have options. Container orchestrators also made it incredibly easy to pay less for things like load balancers. Instead of paying for that pricey layer 7 load balancer from your favorite cloud provider, you can bootstrap nginx into your stack and have lots of fun. Container orchestrators are also very chatty and invoke a level of network utilization that our predecessors probably would've gawked at. When misconfigured, containers are difficult to troubleshoot and require a large array of supplementary software in order to run them with any efficacy partially due to sheer number (nobody runs just one container). Walking around telling people "you don't need x" is a reactionary statement, because I have only covered the surface level information about container orchestrators in regards to benefits and trade-offs; you can't possibly know the answer without having extensive conversations about these. Saying stuff like this: > ‘Engineers get sidetracked by things that make engineers excited, not that solve real problems for users’ — we’ve heard it all before Engineers do overcomplicate stuff but there's reasonable explanations for why if you dare to ask with any kind of non-judgemental intent. It's fanciful that engineers sit around complicating things for their own amusement, especially professional ones. I run virtual machines and a Kubernetes cluster in my personal playground. They both are part of functional projects, however, I end up deploying applications on either platform based on what the applications needs are. It took me about a day to get that Kubernetes cluster configured the way I wanted it and to implement some basic CI/CD that kept it in shape. It probably doesn't get updates as much as it should, but neither do my VM's, so if you end up on one of my side projects: you've been warned! That day spent has saved me time in the future. When I want to deploy, I'm one kubectl command away from updating my production code. Every platform require effort; effort to build, effort to maintain; the only question is where it's spent in between the time of inception to sunset.
- blobster 6y agoIn the overwhelming majority of cases, you will never pass that point. A well-optimized dedicated server with 90 fast cores, 128GB ram and 4TB SSD goes for $120/mo at Hetzner. That can easily handle 1 billion non-cached monthly page views if you write decent code. More if you cache. How many sites get that much traffic? A few dozen. If you're dedicating this much effort in preparing for 1-in-a-million eventualities, you will never ship anything.