9 ms·
I'm going to scale my foot up your ass (2008)
- cushychicken 4y agoAs someone who runs one of those Rails-on-MySQL apps that the author is talking about: article is 100% right. Getting people to give a shit is totally the harder problem.
- wrldos 4y agoPretty accurate. Everything I've seen that doesn't scale only doesn't because it was written by idiots who read lots of blog posts about scalability. I'm still convinced we could run our entire software platform off 5 EC2 instances and an S3 bucket. But the last 10 years of technical mismanagement turned this into a microservices shit show. Splunk gets more traffic than our clients do.
- hkt 4y agoSimilar situation where I've arrived recently: a big public sector org where the biggest expense each month is cloudwatch.
- durnygbur 4y agoTry to say "no" or even "but" to the logging and monitoring request ticket... do you like having monthly income or not?
- smitty1e 4y agoBut we loves us some info porn.
- wrldos 4y agoData lake after 2 years = data cesspit.
- durnygbur 4y agoSoon consultant will be selling wastedata treatment systems.
- wrldos 4y agoThat's a brilliant idea. /dev/null as a service. Charge per byte.
- MikeDelta 4y ago> Charge per byte of unzipped data
- iso1631 4y agoI'd couldn't afford my backups then!
- wrldos 4y agoI guarantee we will not be upselling AWS egress fees to you though so think of the cost savings on recovery!
- exikyut 4y agohttps://devnull-as-a-service.com/ https://devnull-as-a-service.com/ - https://news.ycombinator.com/item?id=6637480 https://news.ycombinator.com/item?id=6637480 (2013) :v
- jbverschoor 4y agoOr just 1 or 2 physical machines?
- wrldos 4y agoWell we still need to handle AWS losing a region and hardware replacement. Does happen quite regularly.
- imwillofficial 4y agoBut do you? Do you really need those 5 nines? If my bank can go down every week for patching, you probably can too.
- wrldos 4y agoOur customers can't operate at all if our systems are down, so yes we do.
- mechanical_bear 4y agoI’m sure your customers can go down for a minute every other week.
- pixl97 4y agoIt doesn't matter if they can or cannot. This is what happens when you think like an engineer and not like a salesperson. What matters if they will pay someone else for that reliability they don't need.
- itsmeste 4y agoThe last 5 years I found myself developing for companies having a bunch of nonsense "microservices" with totally ridiculous separation, which added more to their problems than solving them. The common denominator of those companies is they all had no significant traffic, had their initial codebase developed by (unmanaged) amateurs, and were later managed by people inexperienced in software. Due to the heavy resistance in all cases, I eventually became tired of proposing simpler solutions and the endless discussions around it, and accepted premature microservice architectures to be part of my infinite source of income.
- rippercushions 4y agoMy favorite microservices pattern is having lots of separate little services, so you get that sweet overhead and complicated debugging, with a shared database, so you can still enjoy systemwide single points of failure and obscure side effects on data.
- itsmeste 4y agoHave those microservices use different patterns (language, framework, infra, etc), and outnumber your devs by a magnitude so everybody gets a piece of that sweet context switching, multiple times a day.
- dopidopHN 4y agoLast job I had to ask to confirm the number: 124 services. And it does not work very well. My team was hired to contains and replace it. We ended up with 3 service. Exact same API is provided. It does not matter : nobody is using it anyway !
- wrldos 4y agoAh I am gorging myself financially on exactly that form of sweet misery.
- majormajor 4y agoI was complaining recently to a manager at another company about a half-assed "microservice breakout" project in a previous job where some teams moved from one big service + one big DB to many big services + single DB and introduced fun new DB connection + perf issues... and was dismayed to learn that this other manager didn't even think "a bunch of services all talking to the same giant DB" seemed like best practices.
- mypastself 4y agoAdd to that an existing monolith that’s only been partly migrated to microservices and brother, you’ve got a stew going.
- Scubabear68 4y ago
- neilv 4y agoOne time, I had to put together an architecture diagram for a big investor presentation (diagram of an unusual bespoke many-servers system). Basically, get boxes on the diagram for each aspect the exec might want to talk about or answer questions about. So that they'd have something to point to, and investors would have some level of understanding, as well as confidence that we thought of and were doing something about each thing. (This is a bit different than the diagrams we were using to discuss and build it internally.) So I mention to a colleague that I "just know", if a room of tech investors see the Web facet of this (even though that's the easiest technical part of the system), and this one part is all going through only one server (which is all it would ever need), someone is gonna be concerned that we might not know what we're doing about scalability, and I bet the exec doesn't want to get derailed debating that with them. Colleague advised we go full "System Design Interview!", so I spewed all the unnecessary vapid things we could do, that some semi-technical person might be tuned to look for. Trying to modify the spew to be sensible enough (if encumbering overkill) that, if investors happened to have a hardcore veteran techie vetting (like one of the people who built AWS or Google from nuts and bolts), that they wouldn't become enraged with disgust, and kill me where I stand. The investor meeting seemed to go well. In a different (real, internal) diagram, for networking architecture, I also broke out out some logical functions into distinct physical servers (some fixed, some variable) in an information/control flow compartment. Then I added a comment on the diagram, something about that these might be combined into a single server for efficiency, for smaller deployments, like customer on-prem. And, I didn't add, if our own data center suddenly needed to host a hugely-popular service for others, and we suddenly had enough hardware for that, the software would be ready (but we'd still only need one server in that one spot that a System Design Interview book-prepped person would think needed a load balancer and workers).
- durnygbur 4y ago...and you don't need kubernetes or docker either...
- groestl 4y agoAlthough second one really helps, if you need to ship Python.
- szastamasta 4y agoFirst one, I totally agree. … but Docker just makes things so much easier…
- quickthrower2 4y agoBut you still need the thing that runs docker. What is that for you? And if you have to set that up anyway, why not install stuff at that level instead?
- szastamasta 4y agoI think that even when you plan to just manually deploy dockers on raw hardware, it’s still worth it. There’s plenty of stuff you can ignore if you do so. Ex: do I test with exactly the same versions of libs/binaries that I have on production? You can also easily do rolling deployments even if you have a single machine. There’s plenty of value while staying reasonably simple. Not something you can say about k8s. It’s nowhere near simple.
- iso1631 4y agoI have an old PHP+C app which only runs on ubuntu 1804 and below And another which requires 2004 plus They both require physical access to the same hardware. I want them to coexist on the same physical machine (I can't add a second as I'm space and power constrained), and switch between them trivially and quickly. Now I haven't actually done this yet, but my feeling is I can dockerise it, push the dev files through, and run one image or another depending on what I need in a given hour.
- jxf 4y agoA good article, but it also makes me wonder what happened to Ted Dziuba. He was everywhere 15 years ago and then suddenly he wasn't and I'm not sure why.
- gnfargbl 4y agoHis brother died, and he decided to spend less of his time creating clickbait: https://web.archive.org/web/20120712014548/http://teddziuba.com/post/24585610978/starting-over https://web.archive.org/web/20120712014548/http://teddziuba....
- ChrisMarshallNY 4y agoThanks for sharing that. I can relate. I had a similar epiphany, but for different reasons.
- cdavid 4y agoHe went into politics: https://www.mtdemocrat.com/news/district-1-senate-race-theodore-dziuba/ https://www.mtdemocrat.com/news/district-1-senate-race-theod...
- hasbot 4y ago“My top issues are stopping illegal immigration, taxes and government waste — and the Second Amendment to the Constitution,” said the candidate during a recent interview.
- mcluck 4y agoI wish I could scream this at the top of my lungs at some of the places I worked. In one place, we spent more time adjusting and refactoring Terraform files than actually writing application code despite having about a thousand users, most of whom used the app once for a few minutes in the morning.
- kfk 4y agoCool take, come back to me on how well this thinking worked when you start landing Enterprise customers and you suddenly need to 10x your talent pool. You use bare metal servers? It will be a lot harder to find devops talent with k8s being a standard. You built your web app on a low code tool? It will be impossible to meet that edge case your big Enterprise client needs. Being scalable can be over thinking, or it can save you from losing customers, context matters and each startup is different.
- ebiester 4y agoI'm reminded of the system that ran on one bare metal application server and one database server (not including backups) that was replaced with 80 services and a 10x ops spend for production. It was also slower than the original system. Build your monolith with modules and APIs and split your applications, not your services. If you need to split a module out, your API just has another transmission protocol instead. As Paul Graham said, do things that don't scale.
- kfk 4y agoTalent is harder to scale than tech, context is important.
- amelius 4y agoAnd I'm going to borrow check my foot up your non-garbage-collected ass.
- LoganDark 4y agoI'm rewriting this comment in Rust. -Emily
- joshxyz 4y agoHa, i love this one. Premature fucking optimization.
- ohgodplsno 4y agoDon't forget the geniuses who setup everything to run on GCP/AWS through K8S because hurr durr autoscaling when their needs are mostly static and the servers have to be up 24/7 anyways so they pay out the ass for mediocre servers. Go on, pay for your xlarge while I get 4 top of the line servers for the same price at hetzner. And with the savings, I can even pay a full time sysadmin.
- JackFr 4y agoHonestly when I read the title I heard it in my father’s lost-his-temper voice and I laughed out loud. Angry dad leaned heavily on the “I’m gonna [X] my foot up your ass!” construction.
- hasbot 4y ago[flagged]
- imwillofficial 4y agoThis was awesome.
- mkl95 4y agoI see it as a long term investment. In a few years many companies will be taking the opposite approach and transforming their 300 microservices into a couple of monoliths, because suddenly "simple systems are more scalable" or some similar claim. And from an engineer's POV it will be a great time to be in the industry.
- amarant 4y agoI've more often been surprised how well applications scale in practice that don't follow the scalability guru's advice. Having worked at a couple of places that actually needed scaling, I've learned that monoliths written in Java, even using Spring boot (which still sucks imnsho, but for other reasons entirely), can actually handle quite a bit of load, if written even half-decently. (I've seen apps with upwards of 1000 req/s written this way, having p99 response times below 30ms) I've also seen a lot of different setups: Minecraft's backend does absolutely everything right IMO, and their sub-10ms p99.9 response times showed as much as well, and I don't have to tell you Minecraft has quite a few users. Or another client I won't name, that used microservices, Kafka message queues everywhere and the most complex architecture I've ever seen, all in the name of scalability, couldn't even calculate their p99 response times because they didn't even answer 99% of their requests of their requests (that is, >1% of requests were dropped), at loads of about 5 requests per minute. Moral of the story: the big picture tech decisions aren't always as important as you might think, implementation details/quality can make or brake any architecture.
- toss1 4y ago>>can make or brake any architecture. HA! I wanted to see "break" (as in fracture) any architecture, but I think "brake" (as in device for arresting or preventing the motion) works even better here — the details of the code increase 'friction' in the machine and prevent it's efficient motion... Overall, I quite agree; the devil is in the details, and those details can swamp the 'architectural' decisions. Best to get both mostly right vs one 'perfect' and be ruined by overlooking the other.
- amarant 4y agoHa! That was actually a typo I hadn't even noticed, but it actually kinda worked!
- semicolon_storm 4y agoFunny, the general sentiment about Minecraft's backend was the exact opposite back when I was doing server hosting a number of years ago. Maybe that's changed though. At the time at least, it was very clear that Minecraft was written for single player in mind. Nearly everything ran in a single main loop, causing virtually every multiplayer server to only be able to utilized 1 core. This meant most servers couldn't scale beyond ~50 players with vanilla Minecraft. Community optimizations could take this to 200-300, but usually sacrificed something for those performance gains. Then a ton of work went into workarounds like Bungee, which enabled the next-gen Minecraft servers to host thousands of players on a "single" logical server. Of course, the main game loop was actually still single threaded and those thousands of players couldn't truly interact with each other as you would expect with vanilla Minecraft.
- naruhodo 4y agoI love a good engineering rant.
- wildcow 4y agoI don't get these cursing blog posts. Why so angry? Is this on the frontpage because of what is discussed or semi because the author curse a lot?
- jjtheblunt 4y agoI wonder if it’s the harsh admonition that keeping anyone interested is much harder than pre-worrying about scaling
- imiric 4y agoCursing sometimes helps get the point across on subjects that are often cargo-culted. It's often done tongue-in-cheek to express some deep frustration. And it's fun. :) Also see the motherfuckingwebsite series, The Best Page in the Universe, etc.
- hintymad 4y agoScaling is just one type of optimization, right? So, one should really focus on getting a system working first and then use data to surgically scale the real bottlenecks (back-of-envelope calculation to identify scaling bottlenecks early on falls into the general category of using data to drive optimization).
- filiperui 4y agoWas expecting a Red Forman reference, got a bit disappointed.