9 ms·
Working on complex systems: What I learned working at Google
- nottorp 1y agoLet's add a post scriptum: Whatever you're working on, your project is not likely to be at Google's scale and very unlikely to be a "complex system".
- pentaphobe 1y agoLet's add a post post scriptum :) Just because your project might not be at Google's scale doesn't mean it is therefore also not complex [^1] Example: I'd say plenty of games fit the author's definition of "complex systems". Even the well-engineered ones (and even some which could fit on a floppy disc) [1]: https://en.m.wikipedia.org/wiki/Affirming_the_consequent https://en.m.wikipedia.org/wiki/Affirming_the_consequent
- globalnode 1y agoSpeaking of games, why hasn't google made a game. They could create a gaming division and well... make one. Amazon did. I wonder why they haven't.
- sdenton4 1y ago/me pours one out for Stadia
- galkk 1y agoGoogle made Ingress and Pokémon go (Niantic was part of google before it was spun off). https://en.wikipedia.org/wiki/Niantic,_Inc https://en.wikipedia.org/wiki/Niantic,_Inc.
- Arelius 1y agoAs someone who has firsthand experience: A. The same reason Amazon had/has such a hard time. B. Google lacking the same persistence of Amazon (Consider all the products that are killed) C. Google's hiring process. (They organizationlly do not know how to hire specialists)
- Ripstad 1y ago>B. Google lacking the same persistence of Amazon (Consider all the products that are killed) Yah, like the Stadia, Google's streaming gaming console thing. They even had a first party game development division for it. So exactly what OP was wondering about.
- fidotron 1y agoGoogle has a really hard time grokking the games industry, to the point they can hire people from it and just almost totally ignore them. Their ideas on how Android game development should be done were utterly hilarious, and it's only because of a couple of their dev relations people going to ludicrous lengths that it is actually viable at all. Fundamentally, and ironically, Google likes to offload complexity on to everyone else in their ecosystems, and they got so used to people being willing to jump through hoops to do this for search ads/SEO they are very confused when faced with a more competitive environment. One reason Google can't make games is they can't conceive of a simple enough platform on which to design and develop one. It would be a far too adventurous constantly moving target of wildly different specifications, and they would insist you support all possible permutations of everything from the start. There are reasons people like targeting games consoles, as it lets you focus on the important bits first.
- nottorp 1y ago> Google has a really hard time grokking the games industry And Apple :) The two mobile duopolists don't understand gaming.
- fidotron 1y agoApple absolutely do get it, they might disagree with you about what that is. Google just can't. I worked in the games industry in exactly this area, and the difference between the two is just enormous, but in ways we cannot go into in this forum.
- octo888 1y agoDon't underestimate my colleagues' abilities to turn the simple into the complex!
- citrin_ru 1y agoFor small scale one can build a simple system but I see many are trying to copy FAANG architecture anyway. IMHO it’s a fallacy - people think that if they’ll would copy architecture used by google their company will be successful like google. I think it other was around - google has to build complex systems because it has many users.
- nottorp 1y agoYes, it's called "cargo cult" and it applies to a lot of architecture and processes decisions in IT :)
- ungreased0675 1y agoIt’s an infectious disease among developers. Some people would spend weeks making a simple landing page, and it would require at least 3 different cloud services.
- nottorp 1y agoI'll raise you 7. If i recall correctly. At my tiny workplace we once hired someone to do a simple industrial web site that wasn't likely to ever have more than 2000 users and 10k "entities" in the database. 7 is the number of different AWS services he managed to use for just a prototype. We thanked him and rewrote the thing.
- vendiddy 1y agoManaging complexity pays off sooner than one would think. Even a project that's like 15k lines of code would benefit from a conscious effort to fight against complexity.
- 0xKelsey 1y ago100%
- nasretdinov 1y agoIMO even a more interesting observation is that even Google itself doesn't necessarily work on large scale, e.g. many regionalised services in Google Cloud don't have _that_ many requests in each region, allowing for a much simpler architecture compared to behemoths like GMail or Maps
- p_v_doom 1y agoIMO what we term "complex" tends to be that which the current setup/system struggles to deal with or manage. Relatively speaking google has much much higher complexity, but it doesnt matter as much, because even in simpler cases we are dealing with huge amount of variety and possible states, and the principles of managing that remain the same regardless of scale.
- prmph 1y agoComplex is orthogonal to Large. Some small to medium scale systems address an incredibly complex problem space. Some large systems are solving relatively simple problems. Of course I do agree that size introduces it's own complexity.
- dmoy 1y ago> My immediate reaction in my head was: "This is impossible". But then, a teammate said: "But we're Google, we should be able to manage it!". Google, where the impossible stuff is reduced to merely hard, and the easy stuff is raised to hard.
- dijit 1y agoThis is probably the most accurate statement possible. “I just want to store 5TiB somewhere” “Ha! Did you book multiple bigtable cells” https://youtu.be/3t6L-FlfeaI?si=C5PJcrvLepABZsVF https://youtu.be/3t6L-FlfeaI?si=C5PJcrvLepABZsVF
- Phelinofist 1y agoWhat are peer-bonuses?
- dijit 1y agoThe idea is if someone helps you in a really big way that you’re able to reward that. So you can ask the company to give the person either credits for an internal store, or a direct addition to their salary for one month. Obviously, there are limits to how many pay bonuses you can give out and if it’s direct money or store credits. Directly asking for a peer bonus’ is not very “googly” (and yes, this is a term they use- in case you needed evidence of Google being a bit cultish). There are companies who help do this “as a service”; https://bonusly.com/ https://bonusly.com/
- teivah 1y agoI wasn't aware of bonuses-as-a-service. Thanks for sharing.
- Sharlin 1y agoMy last workplace had a similar institution, only the reward was candy bar or similar that you could go grab from a bowl in the kitchen (working on an honor code basis), in addition to getting some praise on Slack for general warm fuzzies. It was more of a symbolic gesture for recognizing small everyday things, of course, but it was nice IMO.
- kossTKR 1y ago"This is one possible characteristic of complex systems: they behave in ways that can hardly be predicted just by looking at their parts, making them harder to debug and manage." To be honest this doesn't sound too different from many smaller and medium sized internetprojects i've worked on, because of the asynchronous nature of the web, with promises, timing issues and race conditions leading to weirdness that's pretty hard to debug because you have to "playback" with the cascading randomness of request timing, responses, encoding, browser/server shenanigans etc.
- braza 1y agoOne of my pet peeves with the usage of complex(ity) out of the traditional time/space in computer science is that most of the time the OPs of several articles over the internet do not make the distinction between boundaried/arbitrary complexity, where most of the time the person has most of the control of what is being implemented, and domain/accidental/environmental complexity, which is wide open and carries a lot of intrinsic and most of the time unsolvable constraints. Yes, they are Google; yes, they have a great pool of talent around; yes, they do a lot of hard stuff; but most of the time when I read those articles, I miss those kinds of distinctions. Not lowballing the guys at Google, they do amazing stuff, but in some domains of domain/accidental/environmental complexity (e.g. sea logistics, manufacturing, industry, etc.) where most of the time you do not have the data, I believe that they are way more complex/harder than most of the problems that the ones that they deal with.
- kubb 1y agoI’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.
- repeekad 1y agoThe phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%… In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the text on a button, ugh
- pclmulqdq 1y agoThere's a culture of "I won't approve this unless it does something for me" at Google. So now changing the text on a button comes with 2 minor refactors, 10 obvious-but-ignored bugfixes, and 5 experiments that it is actually better.
- 1y ago
- gilleain 1y agoMostly overlapping definition of what a 'complex system' is with : https://en.wikipedia.org/wiki/Complex_system https://en.wikipedia.org/wiki/Complex_system although I understood the key part of a system being complex (as opposed to complicated) is having a large number of types of interaction. So a system with a large number of parts is not enough, those parts have to interact in a number of different ways for the system to exhibit emergent effects. Something like that. I remember reading a lot of books about this kind of thing a while ago :)
- ggm 1y agoI think there are two myths applicable here. Probably more. One myth is that complex systems are inherently bad. Armed forces are incredibly complex. That's why it can take 10 or more rear echelon staff to support one fighting soldier. Supply chain logistics and materiel is complex. Middle ages wars stopped when gunpowder supplies ran out. Another myth is that simple systems are always better and remain simple. They can be, yes. After all, DNA exists. But some beautiful things demand complexity built up from simple things. We still don't entirely understand how DNA and environment combine. Much is hidden in this simple system. I do believe one programming language might be a rational simplification. If you exclude all the DSL which people implement to tune it.
- zmb_ 1y agoFollowing the definition from the article, armed forces seems like a complicated system, not a complex one. There is a structured, repeatable solution for armed forces. It does not exhibit the hallmark characteristics of complex systems listed in the article like emergent behaviors.
- cowboylowrez 1y agonot a fan of the article for this reason alone. good points made, but no reason to redefine perfectly good words when we already have words that work fine.
- p_v_doom 1y agoAgreed. The problem is not complexity. Every system must process a certain amount of information. And the systems complexity must be able to match that amount. The fundamental problem is about designing systems that can manage complexity, especially runaway complexity.
- jajko 1y ago> Middle ages wars stopped when gunpowder supplies ran out Ukraine would be conquered by russia rather quickly if russians weren't so hilariously incompetent in these complex tasks, and war logistics being the king of them. Remember that 64km queue of heavy machinery [1] just sitting still? This was 2022, and we talk about fuel and food, the basics of logistics support. [1] https://en.wikipedia.org/wiki/Russian_Kyiv_convoy https://en.wikipedia.org/wiki/Russian_Kyiv_convoy
- Zoethink 1y ago[dead]
- polotics 1y agoI think you are using hysteresis when actually meaning more general path-dependency.
- Pavilion2095 1y agoThe cookie banner reappears indefinitely on this website when I click 'only necessary' lol.
- teivah 1y agoSorry about that, I'm my newsletter provider (Substack) which is very buggy sometimes.
- romanovcode 1y agoProbably because it is overly complex system.
- codydkdc 1y agoif only there were some simple solution to host a static website without cookies and other garbage https://cloud.google.com/storage/docs/hosting-static-website https://cloud.google.com/storage/docs/hosting-static-website + pick your favorite OSS CMS
- owebmaster 1y agoby option or incompetence because serving text over http is very well abstracted nowadays.
- nonethewiser 1y agoThankfully I dont see a cookie banner at all. Did you try moving continents?
- EasyMark 1y agoor ublock and adding all the cookie blocker lists
- CommenterPerson 1y agoInteresting, Thanks to the writer. However, all this amazing stuff in the service of .. posting ads ?
- RenThraysk 1y agoThere is a certain amount of irony when the cookie policy agreement is buggy on a story about complicated & complex systems. Clicking on "Only Necessary" causes the cookie policy agreement to reappear.
- jajko 1y agoNot for me, on Chrome now
- CommenterPerson 1y agoIt didn't appear on DuckDuckGo either, Thanks.
- nonethewiser 1y agoI dont see a cookie banner. Thankfully.
- amdivia 1y agoHad the same issue
- wooque 1y agoSame here, it's because you have third party cookies blocked.
- junto 1y agoMy assumption with bugs like this is that they are “geographically based edge cases” that have been poorly tested due to engineers not being in the right location to test it, but affects a large number of users without throwing a error that can be logged. GDPR banner only to be used in EU, with conditional of only accepting non-essential cookies, and the engineer or QA is based in the U.S. As a side note, as someone that lives in the EU my pattern of usage here is: - choose only non-essential, but if not a presented option then - reject all cookies, but if no reject all available then - switch the reader mode (or hide distracting items), or if not possible then - close tab I’m getting much more aggressive when dealing with cookie banners dark patterns. I will not a third third party advertising cookies as much as possible and support websites that allow me an easy way to opt out of them.
- ninetyninenine 1y agoExcept computers attempt to model mathematics in an ideal world. Unless your problem comes from something side effects on a computer that can’t be modeled mathematically there is nothing technically stopping you from modeling the problem as mathematical problem then solving that problem via mathematics. Like the output of the LLM can’t be modeled. We literally do not understand it. Are the problems faced by the SRE exactly the same? You give a system an input of B and you can’t predict the output of A mathematically? It doesn’t even have to be a single equation. A simulation can do it.
- ratorx 1y agoI think the vast majority of SRE problems are in the “side effects” category. But higher level than the hardware-level side effects of the computer that you might be imagining. The core problem is building a high enough fidelity model to simulate enough of the real world to make the simulation actually useful. As soon as you have some system feedback loops, the complexity of building a useful model skyrockets. Even in “pure” functions, the supporting infrastructure can be hard to simulate and critical in affecting the outputs. Even doing something simple like adding two numbers requires an unimaginable amount of hidden complexity under the hood. It is almost impossible for these things to not have second-order effects and emergent behaviour under enough scale.
- ninetyninenine 1y agoCan you give me an example of some problem that emerged that was absolutely unpredictable.
- hiddencost 1y agoThis is all exacerbated by a ton of the ML stack being in Python, for some god Forsaken reason.
- bitpush 1y agoHow is the choice of language the cause of anything complex/complicated? Both python and rust (for instance) are both turing complete, and equally capable
- neilv 1y ago> My immediate reaction in my head was: "This is impossible". But then, a teammate said: "But we're Google, we should be able to manage it!". "We can do it!" confidence can be mostly great. (Though you might have to allow for the possibility of failure.) What I don't have a perfect rule for is how to avoid that twisting into arrogance and exceptionalism. Like, "My theory is correct, so I can falsify this experiment." Or "I have so much career potential, it's to everyone's advantage for me to cheat to advance." Or "Of course we'll do the right thing with grabbing this unchecked power, since we're morally superior." Or "We're better than those other people, and they should be exterminated." Maybe part of the solution is to respect the power of will, effort, perseverance, processes, etc., but to be concerned when people don't also respect the power and truth of humility, and start thinking of individual/group selves as innately superior?
- yodsanklai 1y agoSorry to say that, but this sounds a bit like a fantasy. I think the vast majority of Google employees don't see themselves as particularly brillant or special. Even there, lots of people have imposter syndrome. Actually, I've found this is a constant in life, whatever you achieve, you end up in a situation where you're pretty average among your peers. You may feel proud to get into Google for a few months, and then you're quickly humbled down.
- neilv 1y agoUnderstood, but I meant to ask about the more general problem -- not specific to Google, only prompted by that quote. (Also, to be clear about my examples: I don't think Google is fabricating their dissertation research, nor do I think Google is genocidal.) If you're suggesting there's a lot of humility, yet "But we can do it!" still works, that's great, and I'd be interested in more nuances of that.
- tunesmith 1y agoI think the definitions of complex/complicated get muddled with the question of whether something is truly a closed system. Often times something is defined as "complex" when all they mean is that their model doesn't incorporate the externalities. But I don't know if I've come across a description of a truly closed system that has "emergent behavior". I don't know if LLMs qualify.
- carom 1y agoThere are typos and rough grammar in the first few paragraphs and I am actually very happy about that because I know I'm not reading LLM slop.
- destring 1y agoI come from a math background so it’s a bit surprising when software engineers don’t have such basic that systems modeling vocabulary. Everyone should give Thinking in Systems by Meadows a read
- quantum_state 1y agoThe mentioned characteristics and more of complex systems have been known and studied since about half century ago in physics and biology. The deal with large organizations or large code bases is yet another example.
- gilbetron 1y agoI've interviewed many current and ex Googlers, and one thing we've discovered is that we have to be careful overindexing on the scale and complexity of systems they work on. Google is insanely huge and complex, but have insane and complex tooling to help developers. "I worked on a project that affected 250 million users" is something we'll here and sounds amazing, but in reality, from their perspective, they spent months working through the complex Google dev, QA, and deployment process and pushed out a relatively straightforward change, but that change was for a massive system. They have a unique and distinctive experience, but it usually isn't what you expect. It is rare to encounter someone from Google that actually built something of significance, and those that have are always at the staff+ level and had been there 10+ years. If I were to make another generalization, the [g|x]ooglers that worked on relatively "small" projects are often the most interesting, as they had resources to build something from the ground up and do attempt some really interesting projects.