16 ms·
You are not Google (2017)
- hermanradtke 5y agoFrom 2017
- Ma8ee 5y agoAnd even more relevant today, even if the buzzwords are slightly different.
- qntty 5y agoI didn't know that there were video lectures on youtube by Richard Hamming: https://www.youtube.com/playlist?list=PL2FF649D0C4407B30 https://www.youtube.com/playlist?list=PL2FF649D0C4407B30
- dionysus8 5y agoHe even has a twitter account! https://twitter.com/richard_hamming https://twitter.com/richard_hamming
- jconley 5y agoThe paradoxical thing about this for me is that both the most junior and senior engineers fall victim to this line of thinking. I have done so myself, on both ends of the experience/skill spectrum. As a junior it was because the tech was exciting, and the cool kids were doing it so obviously it was the Right Way to do things. As a senior because I'd been a part of the scramble to scale short-sighted systems in a startup that had found product/market fit and didn't want that pain again. Turns out you can build the most scalable thing and people still won't flock to your product. And you wasted all that iteration and learning time.
- lbriner 5y agoThis is true! What I have learned is only to plan for the next order of magnitude from a sensible starting point. If you are creating a new e.g. AirBnB, you will need a minimim viable performance, let's say 1000 users at a time on the site. After that, you plan for 10000, then 100000 etc. You might get multiple orders of magnitude from the same improvements or changes but you only need to get one more step. Why? Your team and expertise will be different in 3 years. You might get bought-out or a new thing might come along that suits your model really well. Eventually, you might have enough money to have an entire data centre with 100 support staff but you certainly can't plan for that on day one.
- twistedpair 5y ago> You might get bought-out or a new thing might come along that suits your model really well. Once you get bought out, you'll likely need to integrate or replatform to the buyer's tech. Just look @ Google acquisitions. They spend the next few years paying down tech debt and scaling, so don't waste your time on that pre liquidity event. Spend that period of time on product improvements and market expansion.
- jconley 5y agoI do the same. But I've done it enough times now that I usually start at the 10k number, just to survive a huge press surge without trouble. Hardware and software is good enough now that this isn't usually much of an overhead.
- hintymad 5y ago> What I have learned is only to plan for the next order of magnitude from a sensible starting point. This is exactly what Google suggested. In particular, it was Jeff Dean who shared this piece of advice in one of his talks on scaling Google's infrastructure.
- bcrosby95 5y agoMy friends and I call this the "complexity hump". Some people never get over it.
- hinkley 5y agoI have a growing pile of anecdotes about smart people getting trapped in this hump, possibly confirmation bias based on recent experience. But it has made me revise my esteem for former coworkers who were very smart and avoided the trap.
- fibers 5y agoIsn't this a symptom of insanely crazy programming whiteboarding interviews?
- xboxnolifes 5y agoWhy do you think that is the case?
- jconley 5y agoI refuse to do those and have fallen victim to it as a founder, so not in my case.
- hinkley 5y agoI have spent more time in my career than many people think is healthy trying to plot out contingency plans for 'what if' situations. It stops being irritating the moment the building is on fire and all of a sudden people want to listen to me. My relationship with YAGNI has ebbed and flowed over the years, and as with any 'problematic' relationship, you may or may not see your own relationship clearly but people who have it worse are still easy to identify. Whether you then ask if you're like that is a matter of wisdom. What we hear about, what we interview about, what we write about is these cool huge projects from the heroes we want, but the heroes we need are the people who figure out how to solve problems in ways that leave the option of solving them differently in the future. It's a little dangerous to say things like that out loud because lots of people hear that as "I'm going to build a configuration engine that I can use to swap out implementations at startup/runtime," which is pretty much the merry-go-round we are all stuck on half the time. No, what I mean is go back way old school, taking some notes from Bertrand Meyer, and arranging your code so that there are 'spots' where major changes in functionality seem to naturally fit. I may have mangled this story in my head over the years into a parable, but I still recall hearing my uncles waxing poetic about, the Chevy straight 6 engine block that was popular during the height of the muscle car era. This engine did not have a particularly high horse power to displacement ratio. But in those days engine bays were fairly empty, and the Straight 6 was overbuilt just enough that it was a dream to modify it, and easy enough to work on that many people did. They bored it out for higher displacement, modified it for higher compression ratios (naturally aspirated or blown), hung additional accessories off of it, you name it. Many people were running around with cars that had over 50% more horse power than the stock version, and some crazy bastards who went considerably higher. In short, this engine was not that particularly great, but it was full of potential. As someone who may or may not become the next Google, (it's been a long time since I worked for anyone who shared opinions like that, when Google was smaller and only dreamed of being as big as they are now), I don't want a system full of features. I want a system full of possibilities. IF we wanted to do this, you would add it here, and verify it here. But we don't, not yet, so if you could just not break 'here' please, that would be great. Nobody is collecting those stories and patterns. They're hidden away in Meyer, Fowler, dropped as throw-away lines in lectures, and discussed at length over coffee, noodles, or beers but never written down. Elsewhere they are really hidden in older aphorisms like 'premature optimization', "single responsibility" and that ilk, so much so that they risk becoming bad advice.
- mellosouls 5y agoDiscussion from 2019 https://news.ycombinator.com/item?id=19576092 https://news.ycombinator.com/item?id=19576092
- dang 5y agoThanks! Macroexpanded: You Are Not Google (2017) - https://news.ycombinator.com/item?id=19576092 https://news.ycombinator.com/item?id=19576092 - April 2019 (572 comments) How to avoid picking the wrong technology just because it's cool - https://news.ycombinator.com/item?id=14508264 https://news.ycombinator.com/item?id=14508264 - June 2017 (158 comments)
- mattgreenrocks 5y agoI don't think the reason I'm about to discuss is behind all of these stories, but I think it is a persistent force in our industry. I am somewhat convinced that a not-insignificant portion of engineers is under-employed in an intellectual sense. When $BIGCO announces the release of a new tool that supposedly helps with their planet-scale needs, the engineer just lights up at the thought of needing to dig into a new challenge. (The impulse to grow is well and good!) It also helps that places like HN buzz with new tools, it is good resume material, and a vague feeling of not being left behind if said thing takes off. The real fix is to get away from domains/companies that bore you in this way, but it is not an easy realization. There are a ton of interesting problem domains out there! I did this and am much happier with my career.
- pjerem 5y agoI'm not sure I agree. I'd prefer working on a boring MVC software, being proud of it being reliable and having the time to work on features my customers will genuinely like rather than extinguishing fires every weeks and adding band-aids everywhere..
- nomaxx117 5y agoI did the same and I am so much happier for it.
- schmichael 5y agoAround 2010 I was working at Urban Airship (now just Airship) offering an API for mobile app developers to send push notifications. We put a ton of effort into making sure we didn’t drop pushes and our API was HA. At one point very early on I was restoring pushes out of error emails in mutt (not great, I know, I know). The reason? Not some unrealistic view of ourselves as Google or Amazon, but because we knew we served pushes for a popular pill reminder app (among other reasons). So just because you don’t have Google or Amazons scale, the data you store and process may be as precious to your users as Amazon’s shopping carts are to them. Over 10 years later there is a lot more medication in my life, and I’m really proud of the sometimes-ridiculous lengths we went to to get the pushes delivered.
- lbriner 5y agoThat isn't really the counter-argument to the OP. Their examples were all about large complexity for the gain in throughput. Clearly, having HA and reliable transport is reasonable for your use-case but that isn't the same as microservices, hadoop or Kafka.
- schmichael 5y agoDefinitely a bit off topic from me, apologies! I think I was mainly writing in agreement with the article’s closing: > What we’re all imploring you to do is to think! And to actually understand the problem you are trying to solve.
- hn_throwaway_99 5y agoI would actually go farther and say the comment you are responding to supports the article's thesis. They talk about remediating issues with email alerts. That is exactly the kind of approach you can take when you are nowhere near Google's scale, instead of building a complex system to ensure all of these remediations can happen automatically without any human intervention. Sounds like they took exactly the right approach of not over-engineering a solution prematurely.
- schmichael 5y ago
- nova22033 5y agoThis is a very good piece except for this part.. Did you know you can buy a terabyte of RAM for around $10,000? Even if you had a billion users, this would give you 1kB of RAM per user to work with Not sure if the author is being facile..
- dekhn 5y ago142 bytes per person, right? A terabyte of ram has been a technical solution to a number of problems in genomics for quite some time and the players were quite willing to pny up the $250K it took to buy that a decade ago. Amusingly, the problem can be converted to... mapreduce and it doesn't need the RAM.
- pc86 5y agoI think the point was that that kind of conclusion shows a staggeringly impressive misunderstanding of how RAM works with regard to web applications.
- tashoecraft 5y agoI can guarantee Oz knows more about how RAM works than you do.
- dekhn 5y agoDo they?
- TuringNYC 5y agoI think his comment is spot on. Many problems have a naturally limited number of users/objects. You can scale up. Scaling up is seen as sacrilege, but scaling-out is expensive in human labor. Scaling up is inexpensive in human labor, and often inexpensive in hardware, and sufficient for many problems.
- shahbaby 5y agoAlso applies to interviews. If you copy Google's interview process, don't be surprised when your best new hires leave for Google.
- pc86 5y agoThis is a good point but the end is wrong. Your new hires won't go to Google, you just won't get any hires. Because I guarantee you're not paying Google wages.
- shahbaby 5y agoI've seen this happen on multiple occasions but of course it won't happen for most people. I suspect that these smaller companies don't really care if copying the industry leaders means they might lose a few of their people to them.
- productceo 5y agoI like UNPHAT idea, as long as it is executed in an agile way. That is, at a given moment, perform multiple stages that can be done safely without feeling blocked by other stages, then iterate, where in each iteration, one tries to do all possible stages better. For example, after reading papers, don't consider previous stages "DONE", and revisit understanding the problem based on the new understanding, and so on.
- softwarebeware 5y agoI interpret the problem a little differently. I don't think the reason people want to copy Google is out of a misguided delusion that their app is going to have the same scale. At least not most of the time. The reason is that software is so bad. Plain and simple. Google is one company whose products seem to be reliable. Most software is a soul-sucking exercise in futility to use them. They error out and do unexpected things randomly. In contrast, Google does a great job in providing a quality experience. People naturally want to know how they do that so well. And to do so at such high scale just adds another layer of impressiveness.
- mirceal 5y agoi sort of disagree. people want to copy google because 1) they like smart, complex things 2) they want to work on complex things. nobody got an award and praise for choosing the simplest thing that could work, doing that and building a service that is so reliable that nobody knows it actually exists (ie no problems or incidents). also, the tragedy of software development is that the squeaky wheel gets the grease and that pyromaniacs are working as firefighters. and are rewarded and promoted based on their firefighting skills. in contrast a well build house, with proper fire retardant materials, sprinkler systems and proper ventilation is meh and not exciting.
- softwarebeware 5y ago> nobody got an award and praise for choosing the simplest thing that could work Well, if that's true, that's where young software engineers have gone wrong. The giants among us used to talk specifically about the power of doing the simplest thing that could possibly work: https://www.artima.com/articles/the-simplest-thing-that-could-possibly-work https://www.artima.com/articles/the-simplest-thing-that-coul...
- mirceal 5y agothere’s theory and there is practice. i constantly have to ask the “kids” at work: do you need this? what tps do you need to support? how are you going to maintain this (at what cost)? do you really want to reinvent the wheel? constantly. i am afraid that most people nowadays practice resume driven development. my favorite pet subject is k8s.
- cgearhart 5y agoWhat I’ve seen more often is software engineers preparing for their next job (or the job they want next) by pushing to build a project with a tool chain that they want to have on their resume. It’s not so much that they want to use X because Google does and it solves their problem, but that they want to look like they know how to use X because then maybe Google will be interested in hiring them—and if not then at least they know that they’re as smart as all those stuck up Googlers anyway.
- yongjik 5y agoThe funny thing is, Google doesn't care that you have three years' Kubernetes experience. They want you to solve leetcode problems. So, it would have been better for everybody if you chose a boring technology, kept the system running with no hassle and no 3am pages, and spent the quiet evening studying on algorithm questions. But apparently that's not sexy enough.
- jacobsenscott 5y agoThis is the main reason, and you can't blame engineers. The only way to raise your salary is to job hop, so the only point of having a job is to train for your next job. Also, as a company trying to hire people you need to use the latest fads to attract new employees. It is a vicious cycle for sure. There is no room in this dysfunctional industry to use simple proven technology that is appropriate to your size.
- weatherlite 5y ago> and you can't blame engineers I think you can definitely blame them and in fact I'd expect CTO's and managers to discourage employees from doing that. A small silly decision can later cost millions in rewrites or maintenance. In the end we'll all benefit if not every brochure website needs to have 50 micro services and use the latest nosql solution. We're the ones who end up maintaining these systems.
- deleted 5y ago[deleted]
- jstummbillig 5y agoThe reason a lot of people chose something out of Google book is not because they get confused about not being Google (which is actually a pretty insulting and also idiotic assumption), but because they wanna benefit from Google putting a stamp of approval on certain tech and pouring resources into it. Tech changes fast, disappears quickly, there is a lot of it out there, most of it is readily available, and so we make use whatever signals we can gather to navigate this space. Which doesn't mean that you can close your eyes and pick whatever tech for whichever use case (duh?) but Amazon or Google doing it differently in the past or a company not being as big as them is certainly not a great argument for not using parts of their stack.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- alecbz 5y agoBut you do need to have _an_ argument for why using their tech is good. > benefit from Google putting a stamp of approval on certain tech and pouring resources into it On its own this is still not a great argument; Googling pouring resources into solving a problem does not help you unless you _also_ have that problem.
- staticassertion 5y agoChoosing technology that's battle tested at scale and under crazy load seems extremely reasonable. Just from a "I need this thing not to have a shitload of bugs" perspective. If you were choosing between two libraries, and one was used by 5 people and one was used by 5 million, it'd be hard to go with the one that is used by 5 people even if it is a better fit for the problem.
- alecbz 5y agoSure, but I think this is not the shape of situation people are generally facing when we say "you're not Google". E.g., often you see choices between NoSQL-esque solutions like BigTable, Dynamo, etc. and traditional relational DBs.
- darkwater 5y agoYou might know you are not Google but you might still love, unconsciously or not, CV Driven Development.
- crate_barre 5y agoI can one can even say for those in these larger companies that you are also not the part of the elite core products. You are not on Google search, or AWS cloud, and so on. You are on that crappy Analytics dashboard that’s slow and confusing.
- CalChris 5y agoThis thinking you're Google is the opposite of technical debt, avoiding doing something and then paying a recurring price for the expediency. This is paying a price up front by doing something which is then never used.
- photochemsyn 5y agoThis is such a great thing to keep in mind. As a junior dev trying to build independent projects for the first time, one of the biggest issues is figuring out what the appropriate level of complexity should be. Building a shed in the backyard doesn't require advanced skyscraper architectural design software. You just want the shed to robust, not flimsy, and for that much simpler tools (a level, a square, a plumb line) work fine. Otherwise you end up never building anything because all your time goes into trying to configure the advanced tools you read about, and it turns into another dead-end mess. It doesn't hurt to know that such advanced tools exist, but I'd think that part of getting an introductory job at any decent firm would involve being trained on how their advanced tools and systems are used in practice.
- JTbane 5y ago> ... one student’s company had chosen to architect their system around Kafka. This was surprising because, as far as I could tell, their business processed just a few dozen very high value transactions per day. I found this immensely funny as a business case that could (and should) be done with a basic SQL-style database like SQLite.
- alexashka 5y agoWhen the people up the dominance hierarchy are impressed with fancy tech, plebs use fancy tech. Maybe call it what it is - a bunch of clueless idiots with money. Fix that, the rest will fix itself.
- alecbz 5y agoWow there used to be good content on Medium?
- tharne 5y agoThe author is 100% correct here, but it's not going to change anything, and I'll tell you why. Companies LOVE requiring all of these sexy over-engineered tools in the requirements for the jobs they post. I've definitely used these overkill solutions in my own work, sometimes because they were the right tool for the job, but many times simply because I knew that one day in the future, I'd be applying for another job, and that job would likely require that I have experience in these technologies. It sucks, but it's the way the world works for now. This has a couple of downstream effects too. If I'm trying to hire, I need to include at least some of those sexy tech stacks in my posting if I want to get good applicants. Why? for the same reason I need to use those things myself even when they're not the best tool for the job. The folks applying to my role know that they're not going to work for me for the rest of their lives and want to been able to check the right boxes when it comes time apply for their next job. Then of course when they get hired in we have to use at least some of these technologies even when we don't totally need them, because after all, it said right on the job posting that we use those technologies. I saw the inverse of this too. There was a team managing a series of small-ish but incredibly important databases. They had these thing running on Microsoft sql server and were doing a beautiful job. There was exactly zero reason to migrate them to something else. But....when the folks in this department went to apply for other jobs inside the company or out, they were greeted with: "You're not using Snowflake??!!", "you're not doing Cloud?!!", "you don't even have an EMR cluster??!!". Not surprisingly, after a while this group had a hard time hiring because folks saw it as a career dead end. TL;DR people do what they're incentivized to do. Stop incentivizing dumb behaviors.
- spaetzleesser 5y ago“There was a team managing a series of small-ish but incredibly important databases. They had these thing running on Microsoft sql server and were doing a beautiful job. There was exactly zero reason to migrate them to something else. But....when the folks in this department went to apply for other jobs inside the company or out, they were greeted with: "You're not using Snowflake??!!", "you're not doing Cloud?!!", "you don't even have an EMR cluster??!!". Exactly. A lot of people who are doing a great job for years wouldn’t even get hired by their own company. A few years ago I realized this was true for myself. Since then I am making an effort to keep myself hireable even if new tech wouldn’t necessarily make sense for the company. But it makes sense for my own career survival.
- dehrmann 5y ago> MapReduce/Hadoop is a soft target at this point because even the cargo culters have realized that the planes ain’t en route. Side question: what is the state of Hadoop? I remember it had a huge ecosystem, but a lot of it seemed half-baked and is dying, so I'm curious how popular it still is, what other tools people are using for massive data transforms, and which parts of the ecosystem (Pig, Hive, etc.) ended up being popular vs. dead ends.
- timr 5y agoFrom what I've seen, the trend amongst today's same class of user (i.e. startups with analytics/log data) is to use Redshift or BigQuery, or just to farm out the equivalent work to a third party (e.g. Datadog).
- omoikane 5y agoI wonder if the title and content ("you can't work the same way big companies work") were inspired by this earlier article from 2014: "No, You Can’t Manufacture That Like Apple Does" https://beneinstein.medium.com/no-you-cant-manufacture-that-like-apple-does-93bea02a3bbf https://beneinstein.medium.com/no-you-cant-manufacture-that-...
- hardwaregeek 5y agoThere's also "You Are Not Microsoft", i.e. you do not have millions of users who would be impacted if this API changed. Sure, if you're at a scale where a few thousand people would be impacted, or at the severity where whoever is affected would be really affected (e.g. medical software), do not break compatibility. But if your product is used by a dozen people and it's not keeping them alive, it's maybe okay if it breaks. Sure, you alienate those dozen people, but you can't keep a company running on a dozen users. And besides, if you have a dozen users, you can literally afford to spend time fixing their specific setups. That's the other flip side. People assume that because Microsoft/Google/etc. provide impersonal, poor customer service, they should provide equally impersonal, poor customer service. Bad customer service is a trade-off you make to have scale. If you don't have scale, you can avoid that trade-off.
- thunderbong 5y agoFrom the article - (and I find this attitude rampant!) Software engineers go crazy for the most ridiculous things. We like to think that we’re hyper-rational, but when we have to choose a technology, we end up in a kind of frenzy — bouncing from one person’s Hacker News comment to another's blog post until, in a stupor, we float helplessly toward the brightest light and lay prone in front of it, oblivious to what we were looking for in the first place.
- blenderdt 5y agoSome developers underestimate the cost of building slow webapps. For example Magento. Magento is extremely slow. I know companies who pay hundreds of dollars for hosting per month to make it faster. Working with Magento always means thinking about scalability and complex architecture. But in reality webservers are extremely fast, even cheap ones. If Magento was built as a good app you would only start to worry about scalability at the time you had a million dollar webshop.
- kache_ 5y agop9999999 is alot easier when you process 100000000000000000000000 requests
- jameshart 5y agoYou may not have google’s scale problems, but even if you’re much smaller than then, if you have a 24/7 web presence you do still have the same browser compatibility, security, accessibility, latency, analytics, monitoring, dependency management, compliance, etc. etc. problems that Google has. Not every piece of technical complexity is designed to solve a scaling problem.
- Adiqq 5y agoI agree, I learned Kubernetes, we use it in our company and to be honest, we could live without it, but it was just ugly, due to lack of good standardization. Now we have GitOps and can easily use tools like Argo Workflows, Argo CD and Argo Events to execute practically any kind of reasonable activity, make it reusable and everyone can see how everything works, at least on dynamically created k3d cluster provisioned with Terraform for development/testing/learning purposes. It's also easy to show everything you create on high level and communicate it to others, because it's easily presentable with e.g. draw.io It doesn't of course solve all kind of issue, because software development is very complex area, but it's making it easier to reason about some solution with others, so we can make what actually client wants without many dirty hacks and rediscovering the wheel. Do we need all these tools and concepts to just build cooperatively, some secure APIs, some front-end, automatically test them and ensure that solution will be highly available under light or moderate load? Probably no, but it would be much harder, even if our solution will be used only internally to handle up to few hundred requests per second and not millions of them. With Kubernetes I can setup almost everything on my own, with just virtual machines or servers and some networking change requests and assuming that infrastructure is stable, solution deployed with Kubernetes will probably be also stable. There are also managed Kubernetes clusters provided in cloud, where cost might seem high for smaller companies or there might be worries about security of using cloud, but once you get it how to create proper cloud-native solutions with Kubernetes, everything becomes much simpler. However learning curve is steep, so beginner might still have issues with efficient use of Kubernetes or other advanced setups, if there won't be good guidance on solution from documentation and/or meetings.
- mrweasel 5y agoThere's a little too many of his examples I've seen in the real world. We where trying to bid on hosting an application that required Cassandra and Kubernetes. The application was built that way, that was how it currently ran. The customer wanted to move to Azure and try to save a bit on hosting. We reached the conclusion that they could use CosmosDB with the Cassandra API enabled and given that it was just a single container it could actually just run in Azure Websites, no need to Kubernetes. Then we started to dig into what the application did, and how much traffic it received. It was just a few hundred http requests per day, and a stable 2GB of data. This should just have been a tiny Java application and SQLite or MariaDB. We did not win the bid, neither the customer nor the developers seemed please with our conclusions. Kafka is another fan favorite with developers. It's not that Kafka isn't great, it is, but if you are only pushing a 2500 messages per day, it's a bit heavy on infrastructure, maybe just stuff those messages in your database. We've build infrastructure for large national projects (in a small country) and you can easily service an entire population on a single MariaDB on a modern server and few VM for the application.
- djmips 5y agoOff topic! This author combined the folksy 'aint' with the fancy 'en route'. Was it jolting for you too? Not only for the weird combo but for it's awkward meter. :) -- I apologize in advance for this comment.
- staticassertion 5y agoOK. But... some of us do need to use Cassandra? Some of us do need to use Kafka? Like, more than 5 companies definitely do work that requires that level of scale... Of course bad engineers make bad engineering decisions, and most engineers are bad, but that isn't really interesting to point out imo. > How much data do you have exactly? Petabytes or exabytes at every company I've worked at, none of which are names mentioned in this article. > But have you done the math? Yeah. I mean, I've read this before, so I'm gonna stop here. I get it. Engineers are bad at their jobs so they default to picking the technologies they've heard of instead of the technologies that make sense. But like... working with massive amounts of data is not that rare anymore. It wasn't in 2017, it definitely isn't in 2022.
- zcdziura 5y agoYou're essentially agreeing with the author's premise. They're saying that you should think about your business domain problem and choose technologies that solve that problem, even if they aren't the big, flashy technologies used by Big Tech. If your business domain involves handling peta/exabytes of data, then by all means Cassandra is right for you! Most companies don't handle nearly that much data however, so using Cassandra for a database to manage only a few gigabytes of data is overkill.
- staticassertion 5y agoOf course I agree with that. I just think that it boils down to: a) These tools are for companies that do XYZ b) Lots of engineers are bad at choosing tools both of which are pretty obvious. Except it also sorta makes this other point "you aren't Google" but... a lot of us actually do stuff at scale. A lot of us. A lot don't, I'm sure.
- riddleronroof 5y agoPeople dont do this because they misunderstand they aren’t google. But because it’s cool and will look good on a resume. Esp when, you know, apply to Google.
- jfax 5y agoIn the words of Terry A Davis: "scalability works both ways".