10 ms·

Requiring permission to do work is the enemy of progress and engineering dignity. It creates a presumption of incompetence and an atmosphere of low trust that p

by fdsfsaa 10y ago

Requiring permission to do work is the enemy of progress and engineering dignity. It creates a presumption of incompetence and an atmosphere of low trust that punishes people who want to push the envelope of what's possible. Google's design document culture is bad. Google has succeeded in spite of it. In my experience, having worked at many large tech companies, design documents obfuscate, not enlighten. They become increasingly out-of-date as the code evolves, creating anti-documentation that makes it take longer to understand code. Yes, yes, people should update design documents as the code evolves. Everyone knows that in practice, nobody updates old design documents. Design documents make it too easy for other developers to shoot down ideas. Sometimes the worth of code isn't apparent until it's made. It's far too easy for someone to comment "this will never work" on a proposed change. It's much harder for someone to deny benchmarks attached to a proposed change. It's too easy for reviewers to knock out functionality. Design documents turn every feature into a half-assed, lowest-common-denominator risk-minimized barely-adequate shell of itself. The real reason everyone at Google writes design documents is that promotion committees demand documents as "evidence of complexity". No design document, no impact. No impact, no promotion. Code is just code. Bad changes can be backed out. It's much better to move fast and iterate quickly than to create an illusion of care and add friction to every aspect of the development process. Up-front design of software just does not work. If it did, waterfall project planning would be successful. These questions you highlight --- Why are you making this change? What impact will it have? --- can be asked during review of actual code. There is no need to build a speedbump, not if you trust your people. Developers should be able to choose to create design documents and solicit feedback. Most changes don't need this process. You should trust developers to know what changes require a more extensive discussion and which ones don't. A culture that requires permissions and signoffs before work can begin is a culture that leaves products stagnant for years.

stinkytaco 10y ago
I can't comment on Google's culture, but: >Design documents turn every feature into a half-assed, lowest-common-denominator risk-minimized shell of itself. Sounds like a sentence written by someone who is an engineer and not a support staff or a user, i.e the people who have to deal with the fallout of every feature change and every engineering decision. I could just as easily substitute "feature driven design" into most of your points above. Requiring someone to put thought into the design of a product and to subject that thought to rigorous scrutiny is, on the whole, a good thing. One that leads to products end users want to use.
- fdsfsaa 10y ago
  > Sounds like a sentence written by someone who is an engineer and not a support staff or a user Do support staff and users sign off on design documents? "No" is the universal answer. Are you claiming that engineers aren't reasonable human beings who can take support staff and user concerns into account on their own? What makes you think the people reviewing design documents can do that? Is it that you just trust a subset of engineers to understand the big picture? Are most engineers just drones? You shouldn't hire people who don't give enough of a shit to take the big picture into account. I am an engineer. I am also a user. I do support my software. I've been programming for twenty years. I find that "rigorous scrutiny" hurts more often than it helps. Maybe this rigid scrutiny was more appropriate in a world with release cycles measured in years, but we don't live in that world anymore. When you ship every week, you can easily undo mistakes, and you're better off erring toward iteration.
  - zardeh 10y ago
    >You shouldn't hire people who don't give enough of a shit to take the big picture into account. At a certain point you can't. I was recently asked to implement a feature for a usecase for another engineer. It required a design doc and review by a few representatives from related teams. He and I wrote the doc, and the initial review was that any solution that would fix his usecase would break the testing infrastructure for practically every other developer in the building. He could write a few fewer loc when testing, at the cost of tests failing due to unrelated changes. Neither he nor I had the awareness of the impact, and realistically couldn't have, it was neither of our responsibilities. And because I spent an hour filling out a template document, I didn't need to waste my time doing that.
    - fdsfsaa 10y ago
      > break the testing infrastructure for practically every other developer in the building Why wouldn't continuous integration have caught that bug? If a diff breaks the build, it shouldn't land. If a diff causes tests to fail, it shouldn't land. If a diff lands anyway and causes problems, any developer negatively affected should be able to insta-revert the diff. How does a design document help?
zardeh 10y ago
>Requiring permission to do work is the enemy of progress and engineering dignity. It creates a presumption of incompetence and an atmosphere of low trust that punishes people who want to push the envelope of what's possible. Its not a permission to do work, its permission to merge the results of the work to master. Those are different. >Google's design document culture is bad. Then why do we have JEPs, PEPs, and rfcs in perhaps every other major project out there?
- fdsfsaa 10y ago
  Counterexample: the Linux kernel.
  - teacup50 10y ago
    Which has succeeded in spite of being a hot, churning mess.
  - zardeh 10y ago
    Which leads to problems: see systemd. Also, to be clear, Linux is relatively small compared to Google or Microsoft, or indeed many corporations codebases. Someone could conceivably read the entire Linux kernel codebase. That's not true for BigCorp.
    - unixhero 10y ago
      I beg to differ. https://youtu.be/yVpbFMhOAwE https://youtu.be/yVpbFMhOAwE
      - zardeh 10y ago
        Right, a single book has on the order of 4-500 pages, longer ones have more, but we'll take the average, and an average book has....30 lines of text per page, so 13500 lines per book. That makes the kernel ~1000 books, which is a lot of books, but avid readers do read 50+ books per year (my father probably does close to double this). That makes reading the entire source absolutely possible, in the order of 10s of years, which is a long time, but then, the kernel has been around for what, 25 now? So there are a number of maintainers who have been around long enough to have read through the entire kernel. Compare that to google, where[1] there are almost as many unique source files as the kernel has lines (though to be fair, many are autogenerated). [1]: http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext http://cacm.acm.org/magazines/2016/7/204032-why-google-store...
skj 10y ago
We write design docs at Google to communicate ideas with each other. They're useful for promo committees because effective communication between engineers is something that is prerequisite for effective engineering. Of course a full design doc is not needed for every change. Many changes are small and straightforward. Only big things, where the team needs to discuss and understand options. Or bigger things, where directors etc need to sign off (not really design docs anymore, but same basic thing). The fact is that, on average, design doc+code takes less time than code without design (that is, without communication). Again, only for certain kinds of changes. Things go faster because problems are found, approaches are adjusted, or unmotivated features are axed.
- fdsfsaa 10y ago
  > We write design docs at Google to communicate ideas with each other. I have no problem with individual developers choosing to circulate documents in order to solicit feedback. My objection is to rigid processes that force engineers to write documents. Mandatory design documents for "communication" invariably morph into checklists of required signoffs from people who have little incentive to say "yes". In this way, a culture of design documents breeds a culture of extreme risk avoidance. It's a tragedy, really. Through numerous small steps, each apparently reasonable, a nimble organization becomes an ossified nightmare in which it takes six months to add a checkbox. > The fact is that, on average, design doc+code takes less time than code without design Prove it. Provide evidence. In my experience, your claim is not the case for most changes in most projects. For the changes where design documents facilitate development, my experience is that developers will choose to circulate documents even when not required to do so. > Unmotivated features are axed. One developer's "unmotivated feature" is another developer's essential use case.
  - zardeh 10y ago
    To clarify here, what kinds of changes do you think require a design document?
    - fdsfsaa 10y ago
viraptor 10y ago
I think it really depends on the project, the community, etc. Some projects seem to be quite successful with design documents. (go, python) Some make it the process where changes go to die unless you're part of the core team and can poke the right people personally to review them. (openstack) The culture is everything. But I think you're missing some very practical things here: > Why are you making this change? What impact will it have? --- can be asked during review of actual code. Yes, and they're going to be asked every single time. And if the reviewers disagree with the impact, you'll have to either drop the change or rewrite it. So why not ask first? > There is no need to build a speedbump, not if you trust your people. "your people" works on a small scale with people you work with continuously. It doesn't work on the internet when someone called "fdsfsaa" submits the code to your project and you hear of them for the first time. > Most changes don't need this process. Depends on the stage of development goals, etc. Some projects will get bugfixes and maintenance. Some will be constantly changing. You can't generalise.
deleted 10y ago
[deleted]
edmundhuber 10y ago
At a large Internet Company that I worked at, my experience was (seemingly?) quite unlike yours. While writing a CEP, my gut reaction was like yours, "this is a waste of time, code is art, and I am programmer Picasso," etc. But since my manager and my technical leads at the time were very good, I bit my tongue and did the work of writing up a CEP anyway. In that environment, it didn't take that long to write a CEP. It was about 2 pages. And since I was proposing to change how a critical piece of infrastructure worked, it was really important for the oncall people to be on board, and it had to make sense, and it was important to identify all the failure points in advance, etc. From the egocentric perspective, I'm certain that it was much less work for me to write those 2 pages than it would have been to explain to 20 different people what I'm doing and why, either at the water cooler or during code review. Trust is often earned and not given. Your coworkers may not know you or the quality of your work. Under the right conditions, I don't think being asked to write a CEP is being asked to dilute your vision, it's merely being asked to define it, and describe how it fits in with how things already work. If that antithetical to you, then you need to work at or found a small company, where everyone is most concerned (hopefully) with making something work, rather than trying to make something that is already working and already making money better. I have some points of agreement with you, although I wholeheartedly disagree with the conclusions you make. "Everyone knows that in practice, nobody updates design documents after the fact." Even if this is completely true (and it isn't -- I have written and read many documents that closely follow current practice) does that make it better to not try? "Sometimes the worth of code isn't apparent until it's made." Does this mean that it is too hard to explain why it's worth doing? "These questions you highlight --- Why are you making this change? What impact will it have? --- can be asked during review of actual code." You're right, these questions will certainly be asked -- in which case, you can link them to the CEP which will probably answer their questions plus other ones that they didn't even think of to ask. I can't disagree more that code review can replace a CEP.
debaserab2 10y ago
It sounds like you have some experience dealing with some serious bureaucratic red tape manifesting itself in design doc's. I don't know if design documents need to be a huge thing with stakeholder sign off and approval processes necessarily - maybe when you're google size. At my (small) company, we use something similar (we simply call them spec's) and it's more of a way to express your concept than it is to get sign off. Sometimes just spelling out an idea helps you validate it. Really, you should be able to defend a certain amount of criticism if you're writing code that someone (and probably not you, eventually) will have to maintain. Bad changes can be backed out if caught early, but if left they find ways to creep into other areas and cement themselves when they are built upon. I don't disagree with your main point though, but I do believe there is a balance.
tlogan 10y ago
Design, functionality, and test specifications are very important. But the process must be monitored and be very flexible. I'm not familiar with Google system, but in other big corporations the problem is that you end up with system which does not allow exceptions - so engineers end up requiring writing 10 different documents to make a small change which can be explain on one page and everybody will get it: support, testing, doc writers, etc. And these documents ended up begin written so that management is happy and not actual intended audience: tester, support, maintenance, documentation writers, etc. So these documents ended up being useless so .... The worst result is that engineers which do not write good code but write good essays end up with promotion and eventually destroying the entire product (not their fault - they just not talented programers).
- fdsfsaa 10y ago
  I think the pathologies you mention are inevitable once it becomes acceptable to address technical problems by adding process like mandatory design document signoff. The only way to avoid these pathologies is to take a hard stand against process.
tytso 10y ago
At least in the part of Google where I work (Technical Infrastructure), in general most of the obvious optimizations that can be made at only one level (within the scope of a single programmer, or a single team), have been made long ago. So most of the changes to make the system more efficient will require coordinated changes across multiple teams, and multiple pieces of software, and in some cases, may impact more than one SRE team. In that kind of situation, you betcha we need to have design docs! And in terms of making it easy for other developers to shoot down ideas, very often they may know about some dependency or key assumption in some other piece of code that you didn't know about it. And it's better to find out about it during the design phase, than to have to rework 50% of your work when you find out about it at code review time, or worse, if it gets deployed and you get angry notes from SRE's that were woken up at 3am and you need to send them a bottle of whiskey to apologize for your f*ck up....
scottlamb 10y ago
> A culture that requires permissions and signoffs before work can begin is a culture that leaves products stagnant for years. I don't see such a culture. I often create one or more prototypes as part of my design. One of those might become the final result, or I might throw away everything. As long as that's understood, all is well. The way I think about it is this: don't do work you aren't willing to throw away until earlier steps have been reviewed. How much work you're willing to throw away is a personal preference. Your design reviewer(s) might say "did you consider this other way that has these advantages?" You shouldn't reply with "No, and I've invested too much time to consider other approaches now. Stop holding me up and lgtm already." No one wants to work with someone like that. Your argument should be based on what's best. What's already done should only be considered if neither approach is (believed to be) significantly better.
dismantlethesun 10y ago
> In my experience, having worked at many large tech companies, design documents obfuscate, not enlighten. They become increasingly out-of-date as the code evolves, creating anti-documentation that makes it take longer to understand code. Yes, yes, people should update design documents as the code evolves. Everyone knows that in practice, nobody updates old design documents. My experience has been the reverse, precisely because as you said.... no one updates old design documents. A piece of code with a design document at least has a historical record of what the original aims of the project were, and a written rationale for why they took certain approaches. Often you'll find a piece of code with a seemingly inane architecture and wonder "why is this so inane? were the the developers on drugs?". By reading the design document, you find out that sadly no the water fountains were not spiked with LSD in the 70s, but rather they were working around the performance characteristics of hardware that no longer exists and thus these baked in assumptions had reason and merit. Understanding the original why often illuminates the entire architecture, even if undergone a lot of changes because the original skeleton still remains. I sort of look at it as Code Archeology, or maybe literary deconstruction as applies to code.
- fdsfsaa 10y ago
  > A piece of code with a design document at least has a historical record of what the original aims of the project were, and a written rationale for why they took certain approaches. That information is very useful, particularly as a comment or a commit message. Storing this information in a separate unversioned document off on some enterprise management system makes it harder to find. If you put the rationale for a change in the commit message, the rationale is right there when you run blame!
  - vertex-four 10y ago
    And what about when your commit is part of a 5-patch series, as part of a 3-series project to bring some internal framework code up to scratch in order to start implementing a new feature? Which commit do you stick your design rationale on, and how do you ensure people will see it? Commit messages are really good for e.g. "patch that due to this bug" or "refactor this so it's decoupled from that feature", but not quite so good for describing arches in development. On the other hand, you can reference an issue in every commit message, and that issue can lead to the design document.
optionalparens 10y ago
This attitude strikes me as very SV/HN and while I can appreciate certain elements here, the answer is the typical one - "It depends." Rather than regurgitate what most people here already said, let me list a few programming projects, domains, and tasks where at least thinking about design if not writing design documents or spending days, weeks, or months figuring it all out is worthwhile. * Programming Languages * Databases * Operating Systems * Medical Devices * Safety Equipment * Streaming containers/formats * Encryption * Security * Manufacturing/Robotics * Aerospace / Space * App Dev Frameworks * Game Engines I could go on. The point here is that there are plenty of things where thinking about it up front is beneficial, if not required, especially if some combination (but not limited to) the following are true: * Lives are at stake * Changing it later would be hard (programming languages are an egregious offender, I won't name names) * Customer adoption will completely derail or forbid architectural changes * Fixing it will require essentially doing it again from scratch * Changes will force the creation of patches that will incrementally kill the project or slow future development Frankly, I think we have too many things that are poorly designed. Most projects I see in nearly any domain are mostly set in stone once time and money is added to the mix. Everyone talks about redoing or fixing things, but it rarely happens except for minor changes. As projects scale up, few people can afford to constantly back out lots of changes and rearchitect everything. Those that do usually fail or don't get a good ROI, and those that don't change fail anyway. I've worked with all kinds of people and though there are people I have great admiration for, I can safely say that 99% of them are idiots and have no business being programmers. I know it sounds harsh, but I've been doing this a long time and have worked with all kinds of people. Too often I see the programmer's equivalent of an illiterate child that gets pushed through high school. So no, I don't trust people to do the right thing, I merely trust most people I work with to not act maliciously. Most of all, I don't trust myself. As the progression goes as a programmer - your code sucks -> my code sucks -> all code sucks -> my code sucks but I'll live with it, hope it is better than most, and ask people smarter than me for help. Most better developers I know do in fact right some form of design documents, even if it's just notes and justification why X or Y won't work, but Z "might" work. Many also take a lot of time to think about something before writing any code, but once they do, they actually finish much quicker with less bugs than the young programmers who want to "move fast." Of course none of this is universal, and as I said, it all just "depends." What do I know?
int_19h 10y ago
> Code is just code. Bad changes can be backed out. Not if they have already been shipped.
dcosson 10y ago
> A culture that requires permissions and signoffs before work can begin is a culture that leaves products stagnant for years. I've also seen the exact opposite being the case. A culture where everyone does whatever they're in the mood for without running ideas by other teammates who will have different experience in different areas of the codebase, can leave products stagnant for years. Tech debt builds up and the size of change that anyone is comfortable doing gets smaller and smaller until large new features become unfeasible. If the culture on the team is to mostly work on your own without spending time designing & brainstorming up front except in rare circumstances, then nobody wants to be the only person enforcing process on themselves. You might worry that you come across as a weak engineer if you ask for a lot of feedback and nobody else does, and it does slow you down some so you'll get less work done on individual projects than your teammates (and your solutions might be of a higher quality, but that tends to show itself as the absence of problems or only be visible in the future to the next person who works on your code, and those things are easy to overlook). So IMO it's not a matter of a presumption of incompetence vs competence, the goal is aligning incentives so that collaboration and taking the time to come up with the best solutions become the most natural way to work. (FWIW I've never worked at Google and can't speak at all to their implementation of a design document culture).
- beat 10y ago
  I call that "The inmates running the asylum".
vincnetas 10y ago
Talking about desing documments i think internet would be a mess without RFC's (https://en.wikipedia.org/wiki/Request_for_Comments https://en.wikipedia.org/wiki/Request_for_Comments) which i think are formal design documents...
- fdsfsaa 10y ago
  RFCs are usually protocol specifications. Specifications are usually intended to facilitate interoperability. They document protocols or grammars or some other artifact. Specifications need to be well-written and precise. The kind of design document that I frequently find superfluous isn't a specification of some protocol, but a prose description of the code one intends to write. In concrete terms, an RFC might describe TCP header flags and the TCP state machine, but it'd be silent on the Linux kernel's sk_buff structure. A design document would describe sk_buff in detail.
Confusion 10y ago
It creates a presumption of incompetence No, it acknowledges the reality, that people, including you and I, don't know what is good for them. Being forced to come up with design documents is very similar to forcing doctors to use checklists. They complained, and still complain, that they are professionals and don't need the bureaucracy and 'assumption of incompetence'. But the numbers speak for themselves: there are much fewer medical errors when they are forced to use simple checklists. My colleagues are all competent, yet they produce much better work if they are forced to first come up with a design document.
euyyn 10y ago
I agree with what you're saying, but what you're saying is not how it works at Google. At least the teams I've been at. Design docs aren't a prerequisite to start working, and in many cases a design is nonsensical if you haven't actually at least prototyped what you want. It's just a tool to help you be comprehensive when you want to decide between alternatives, and let other eyeballs help you decide. "this will never work", without an actual argument, isn't Googley :) A big design doc is a liability for sure, in the same way as code is.
beat 10y ago
I can tell you've never dealt with one of those trivial backouts on a product that moves a hundred billion dollars a day and is a key component of the entire US economy. Or a product where a bad software deployment can actually kill people. Process is the scar tissue of the enterprise. Those scars are there because the enterprise was wounded. Lots of process, many scars.
- fdsfsaa 10y ago
  I can choose not to work in an environment riddled with "scar tissue". Instead, I can go work at a startup and eat that enterprise's lunch with a tenth of the budget. Unfortunately, thanks to inflexible and sanctimonious attitudes some programmers adopt about what is and is not "responsible" engineering, the only way to change practices is to beat the old practices in the marketplace.
  - beat 10y ago
    You're welcome to try eating Google's lunch, since their process is your favorite example. But try tackling banking, or insurance, or transportation logistics, or any other big-boy problems with that attitude. You won't last long.
yawz 10y ago
> "It's much better to move fast and iterate quickly than to create an illusion of care and add friction to every aspect of the development process." This!
leepowers 10y ago
Wait, you don't spec out project changes? You just start writing code and attempt to figure it out as you go along? That seems like a recipe for disaster, or at least a lot of wasted effort. How do you even quantify that you've done something useful if you haven't set a goal that everyone can agree on?
imagist 10y ago
> Code is just code. Bad changes can be backed out. It's much better to move fast and iterate quickly than to create an illusion of care and add friction to every aspect of the development process. Up-front design of software just does not work. If it did, waterfall project planning would be successful. Bad changes can be backed out, but in an infrastructure with any size, this isn't a trivial task. I've worked in a million-line codebase with multiple separate deployments, and in that case, design documents were much cheaper to create and maintain than a rollback or even writing working code and then discarding it. In an infrastructure of Google's age and size, the trade off is even more clearly in favor of up-front design. You probably think the "move fast and break things" ideology you're espousing is agile, but it's not. It's just another plan, and agile is about responding to change over following a plan. I hope that if you ever work on a project of Google's size that you can adapt.

by fdsfsaa 10y ago

stinkytaco 10y ago

I can't comment on Google's culture, but: >Design documents turn every feature into a half-assed, lowest-common-denominator risk-minimized shell of itself. Sounds like a sentence written by someone who is an engineer and not a support staff or a user, i.e the people who have to deal with the fallout of every feature change and every engineering decision. I could just as easily substitute "feature driven design" into most of your points above. Requiring someone to put thought into the design of a product and to subject that thought to rigorous scrutiny is, on the whole, a good thing. One that leads to products end users want to use.

fdsfsaa 10y ago
> Sounds like a sentence written by someone who is an engineer and not a support staff or a user Do support staff and users sign off on design documents? "No" is the universal answer. Are you claiming that engineers aren't reasonable human beings who can take support staff and user concerns into account on their own? What makes you think the people reviewing design documents can do that? Is it that you just trust a subset of engineers to understand the big picture? Are most engineers just drones? You shouldn't hire people who don't give enough of a shit to take the big picture into account. I am an engineer. I am also a user. I do support my software. I've been programming for twenty years. I find that "rigorous scrutiny" hurts more often than it helps. Maybe this rigid scrutiny was more appropriate in a world with release cycles measured in years, but we don't live in that world anymore. When you ship every week, you can easily undo mistakes, and you're better off erring toward iteration.
- zardeh 10y ago
  >You shouldn't hire people who don't give enough of a shit to take the big picture into account. At a certain point you can't. I was recently asked to implement a feature for a usecase for another engineer. It required a design doc and review by a few representatives from related teams. He and I wrote the doc, and the initial review was that any solution that would fix his usecase would break the testing infrastructure for practically every other developer in the building. He could write a few fewer loc when testing, at the cost of tests failing due to unrelated changes. Neither he nor I had the awareness of the impact, and realistically couldn't have, it was neither of our responsibilities. And because I spent an hour filling out a template document, I didn't need to waste my time doing that.
  - fdsfsaa 10y ago
    > break the testing infrastructure for practically every other developer in the building Why wouldn't continuous integration have caught that bug? If a diff breaks the build, it shouldn't land. If a diff causes tests to fail, it shouldn't land. If a diff lands anyway and causes problems, any developer negatively affected should be able to insta-revert the diff. How does a design document help?

zardeh 10y ago

>Requiring permission to do work is the enemy of progress and engineering dignity. It creates a presumption of incompetence and an atmosphere of low trust that punishes people who want to push the envelope of what's possible. Its not a permission to do work, its permission to merge the results of the work to master. Those are different. >Google's design document culture is bad. Then why do we have JEPs, PEPs, and rfcs in perhaps every other major project out there?

fdsfsaa 10y ago
Counterexample: the Linux kernel.
- teacup50 10y ago
  Which has succeeded in spite of being a hot, churning mess.
- zardeh 10y ago
  Which leads to problems: see systemd. Also, to be clear, Linux is relatively small compared to Google or Microsoft, or indeed many corporations codebases. Someone could conceivably read the entire Linux kernel codebase. That's not true for BigCorp.
  - unixhero 10y ago
    I beg to differ. https://youtu.be/yVpbFMhOAwE https://youtu.be/yVpbFMhOAwE
    - zardeh 10y ago
      Right, a single book has on the order of 4-500 pages, longer ones have more, but we'll take the average, and an average book has....30 lines of text per page, so 13500 lines per book. That makes the kernel ~1000 books, which is a lot of books, but avid readers do read 50+ books per year (my father probably does close to double this). That makes reading the entire source absolutely possible, in the order of 10s of years, which is a long time, but then, the kernel has been around for what, 25 now? So there are a number of maintainers who have been around long enough to have read through the entire kernel. Compare that to google, where[1] there are almost as many unique source files as the kernel has lines (though to be fair, many are autogenerated). [1]: http://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext http://cacm.acm.org/magazines/2016/7/204032-why-google-store...

skj 10y ago

We write design docs at Google to communicate ideas with each other. They're useful for promo committees because effective communication between engineers is something that is prerequisite for effective engineering. Of course a full design doc is not needed for every change. Many changes are small and straightforward. Only big things, where the team needs to discuss and understand options. Or bigger things, where directors etc need to sign off (not really design docs anymore, but same basic thing). The fact is that, on average, design doc+code takes less time than code without design (that is, without communication). Again, only for certain kinds of changes. Things go faster because problems are found, approaches are adjusted, or unmotivated features are axed.

fdsfsaa 10y ago
> We write design docs at Google to communicate ideas with each other. I have no problem with individual developers choosing to circulate documents in order to solicit feedback. My objection is to rigid processes that force engineers to write documents. Mandatory design documents for "communication" invariably morph into checklists of required signoffs from people who have little incentive to say "yes". In this way, a culture of design documents breeds a culture of extreme risk avoidance. It's a tragedy, really. Through numerous small steps, each apparently reasonable, a nimble organization becomes an ossified nightmare in which it takes six months to add a checkbox. > The fact is that, on average, design doc+code takes less time than code without design Prove it. Provide evidence. In my experience, your claim is not the case for most changes in most projects. For the changes where design documents facilitate development, my experience is that developers will choose to circulate documents even when not required to do so. > Unmotivated features are axed. One developer's "unmotivated feature" is another developer's essential use case.
- zardeh 10y ago
  To clarify here, what kinds of changes do you think require a design document?
  - fdsfsaa 10y ago

viraptor 10y ago

I think it really depends on the project, the community, etc. Some projects seem to be quite successful with design documents. (go, python) Some make it the process where changes go to die unless you're part of the core team and can poke the right people personally to review them. (openstack) The culture is everything. But I think you're missing some very practical things here: > Why are you making this change? What impact will it have? --- can be asked during review of actual code. Yes, and they're going to be asked every single time. And if the reviewers disagree with the impact, you'll have to either drop the change or rewrite it. So why not ask first? > There is no need to build a speedbump, not if you trust your people. "your people" works on a small scale with people you work with continuously. It doesn't work on the internet when someone called "fdsfsaa" submits the code to your project and you hear of them for the first time. > Most changes don't need this process. Depends on the stage of development goals, etc. Some projects will get bugfixes and maintenance. Some will be constantly changing. You can't generalise.

deleted 10y ago

[deleted]

edmundhuber 10y ago

At a large Internet Company that I worked at, my experience was (seemingly?) quite unlike yours. While writing a CEP, my gut reaction was like yours, "this is a waste of time, code is art, and I am programmer Picasso," etc. But since my manager and my technical leads at the time were very good, I bit my tongue and did the work of writing up a CEP anyway. In that environment, it didn't take that long to write a CEP. It was about 2 pages. And since I was proposing to change how a critical piece of infrastructure worked, it was really important for the oncall people to be on board, and it had to make sense, and it was important to identify all the failure points in advance, etc. From the egocentric perspective, I'm certain that it was much less work for me to write those 2 pages than it would have been to explain to 20 different people what I'm doing and why, either at the water cooler or during code review. Trust is often earned and not given. Your coworkers may not know you or the quality of your work. Under the right conditions, I don't think being asked to write a CEP is being asked to dilute your vision, it's merely being asked to define it, and describe how it fits in with how things already work. If that antithetical to you, then you need to work at or found a small company, where everyone is most concerned (hopefully) with making something work, rather than trying to make something that is already working and already making money better. I have some points of agreement with you, although I wholeheartedly disagree with the conclusions you make. "Everyone knows that in practice, nobody updates design documents after the fact." Even if this is completely true (and it isn't -- I have written and read many documents that closely follow current practice) does that make it better to not try? "Sometimes the worth of code isn't apparent until it's made." Does this mean that it is too hard to explain why it's worth doing? "These questions you highlight --- Why are you making this change? What impact will it have? --- can be asked during review of actual code." You're right, these questions will certainly be asked -- in which case, you can link them to the CEP which will probably answer their questions plus other ones that they didn't even think of to ask. I can't disagree more that code review can replace a CEP.

debaserab2 10y ago

It sounds like you have some experience dealing with some serious bureaucratic red tape manifesting itself in design doc's. I don't know if design documents need to be a huge thing with stakeholder sign off and approval processes necessarily - maybe when you're google size. At my (small) company, we use something similar (we simply call them spec's) and it's more of a way to express your concept than it is to get sign off. Sometimes just spelling out an idea helps you validate it. Really, you should be able to defend a certain amount of criticism if you're writing code that someone (and probably not you, eventually) will have to maintain. Bad changes can be backed out if caught early, but if left they find ways to creep into other areas and cement themselves when they are built upon. I don't disagree with your main point though, but I do believe there is a balance.

tlogan 10y ago

Design, functionality, and test specifications are very important. But the process must be monitored and be very flexible. I'm not familiar with Google system, but in other big corporations the problem is that you end up with system which does not allow exceptions - so engineers end up requiring writing 10 different documents to make a small change which can be explain on one page and everybody will get it: support, testing, doc writers, etc. And these documents ended up begin written so that management is happy and not actual intended audience: tester, support, maintenance, documentation writers, etc. So these documents ended up being useless so .... The worst result is that engineers which do not write good code but write good essays end up with promotion and eventually destroying the entire product (not their fault - they just not talented programers).

fdsfsaa 10y ago
I think the pathologies you mention are inevitable once it becomes acceptable to address technical problems by adding process like mandatory design document signoff. The only way to avoid these pathologies is to take a hard stand against process.

tytso 10y ago

At least in the part of Google where I work (Technical Infrastructure), in general most of the obvious optimizations that can be made at only one level (within the scope of a single programmer, or a single team), have been made long ago. So most of the changes to make the system more efficient will require coordinated changes across multiple teams, and multiple pieces of software, and in some cases, may impact more than one SRE team. In that kind of situation, you betcha we need to have design docs! And in terms of making it easy for other developers to shoot down ideas, very often they may know about some dependency or key assumption in some other piece of code that you didn't know about it. And it's better to find out about it during the design phase, than to have to rework 50% of your work when you find out about it at code review time, or worse, if it gets deployed and you get angry notes from SRE's that were woken up at 3am and you need to send them a bottle of whiskey to apologize for your f*ck up....

scottlamb 10y ago

> A culture that requires permissions and signoffs before work can begin is a culture that leaves products stagnant for years. I don't see such a culture. I often create one or more prototypes as part of my design. One of those might become the final result, or I might throw away everything. As long as that's understood, all is well. The way I think about it is this: don't do work you aren't willing to throw away until earlier steps have been reviewed. How much work you're willing to throw away is a personal preference. Your design reviewer(s) might say "did you consider this other way that has these advantages?" You shouldn't reply with "No, and I've invested too much time to consider other approaches now. Stop holding me up and lgtm already." No one wants to work with someone like that. Your argument should be based on what's best. What's already done should only be considered if neither approach is (believed to be) significantly better.

dismantlethesun 10y ago

> In my experience, having worked at many large tech companies, design documents obfuscate, not enlighten. They become increasingly out-of-date as the code evolves, creating anti-documentation that makes it take longer to understand code. Yes, yes, people should update design documents as the code evolves. Everyone knows that in practice, nobody updates old design documents. My experience has been the reverse, precisely because as you said.... no one updates old design documents. A piece of code with a design document at least has a historical record of what the original aims of the project were, and a written rationale for why they took certain approaches. Often you'll find a piece of code with a seemingly inane architecture and wonder "why is this so inane? were the the developers on drugs?". By reading the design document, you find out that sadly no the water fountains were not spiked with LSD in the 70s, but rather they were working around the performance characteristics of hardware that no longer exists and thus these baked in assumptions had reason and merit. Understanding the original why often illuminates the entire architecture, even if undergone a lot of changes because the original skeleton still remains. I sort of look at it as Code Archeology, or maybe literary deconstruction as applies to code.

fdsfsaa 10y ago
> A piece of code with a design document at least has a historical record of what the original aims of the project were, and a written rationale for why they took certain approaches. That information is very useful, particularly as a comment or a commit message. Storing this information in a separate unversioned document off on some enterprise management system makes it harder to find. If you put the rationale for a change in the commit message, the rationale is right there when you run blame!
- vertex-four 10y ago
  And what about when your commit is part of a 5-patch series, as part of a 3-series project to bring some internal framework code up to scratch in order to start implementing a new feature? Which commit do you stick your design rationale on, and how do you ensure people will see it? Commit messages are really good for e.g. "patch that due to this bug" or "refactor this so it's decoupled from that feature", but not quite so good for describing arches in development. On the other hand, you can reference an issue in every commit message, and that issue can lead to the design document.

optionalparens 10y ago

This attitude strikes me as very SV/HN and while I can appreciate certain elements here, the answer is the typical one - "It depends." Rather than regurgitate what most people here already said, let me list a few programming projects, domains, and tasks where at least thinking about design if not writing design documents or spending days, weeks, or months figuring it all out is worthwhile. * Programming Languages * Databases * Operating Systems * Medical Devices * Safety Equipment * Streaming containers/formats * Encryption * Security * Manufacturing/Robotics * Aerospace / Space * App Dev Frameworks * Game Engines I could go on. The point here is that there are plenty of things where thinking about it up front is beneficial, if not required, especially if some combination (but not limited to) the following are true: * Lives are at stake * Changing it later would be hard (programming languages are an egregious offender, I won't name names) * Customer adoption will completely derail or forbid architectural changes * Fixing it will require essentially doing it again from scratch * Changes will force the creation of patches that will incrementally kill the project or slow future development Frankly, I think we have too many things that are poorly designed. Most projects I see in nearly any domain are mostly set in stone once time and money is added to the mix. Everyone talks about redoing or fixing things, but it rarely happens except for minor changes. As projects scale up, few people can afford to constantly back out lots of changes and rearchitect everything. Those that do usually fail or don't get a good ROI, and those that don't change fail anyway. I've worked with all kinds of people and though there are people I have great admiration for, I can safely say that 99% of them are idiots and have no business being programmers. I know it sounds harsh, but I've been doing this a long time and have worked with all kinds of people. Too often I see the programmer's equivalent of an illiterate child that gets pushed through high school. So no, I don't trust people to do the right thing, I merely trust most people I work with to not act maliciously. Most of all, I don't trust myself. As the progression goes as a programmer - your code sucks -> my code sucks -> all code sucks -> my code sucks but I'll live with it, hope it is better than most, and ask people smarter than me for help. Most better developers I know do in fact right some form of design documents, even if it's just notes and justification why X or Y won't work, but Z "might" work. Many also take a lot of time to think about something before writing any code, but once they do, they actually finish much quicker with less bugs than the young programmers who want to "move fast." Of course none of this is universal, and as I said, it all just "depends." What do I know?

int_19h 10y ago

> Code is just code. Bad changes can be backed out. Not if they have already been shipped.

dcosson 10y ago

> A culture that requires permissions and signoffs before work can begin is a culture that leaves products stagnant for years. I've also seen the exact opposite being the case. A culture where everyone does whatever they're in the mood for without running ideas by other teammates who will have different experience in different areas of the codebase, can leave products stagnant for years. Tech debt builds up and the size of change that anyone is comfortable doing gets smaller and smaller until large new features become unfeasible. If the culture on the team is to mostly work on your own without spending time designing & brainstorming up front except in rare circumstances, then nobody wants to be the only person enforcing process on themselves. You might worry that you come across as a weak engineer if you ask for a lot of feedback and nobody else does, and it does slow you down some so you'll get less work done on individual projects than your teammates (and your solutions might be of a higher quality, but that tends to show itself as the absence of problems or only be visible in the future to the next person who works on your code, and those things are easy to overlook). So IMO it's not a matter of a presumption of incompetence vs competence, the goal is aligning incentives so that collaboration and taking the time to come up with the best solutions become the most natural way to work. (FWIW I've never worked at Google and can't speak at all to their implementation of a design document culture).

beat 10y ago
I call that "The inmates running the asylum".

vincnetas 10y ago

Talking about desing documments i think internet would be a mess without RFC's (https://en.wikipedia.org/wiki/Request_for_Comments https://en.wikipedia.org/wiki/Request_for_Comments) which i think are formal design documents...

fdsfsaa 10y ago
RFCs are usually protocol specifications. Specifications are usually intended to facilitate interoperability. They document protocols or grammars or some other artifact. Specifications need to be well-written and precise. The kind of design document that I frequently find superfluous isn't a specification of some protocol, but a prose description of the code one intends to write. In concrete terms, an RFC might describe TCP header flags and the TCP state machine, but it'd be silent on the Linux kernel's sk_buff structure. A design document would describe sk_buff in detail.

Confusion 10y ago

It creates a presumption of incompetence No, it acknowledges the reality, that people, including you and I, don't know what is good for them. Being forced to come up with design documents is very similar to forcing doctors to use checklists. They complained, and still complain, that they are professionals and don't need the bureaucracy and 'assumption of incompetence'. But the numbers speak for themselves: there are much fewer medical errors when they are forced to use simple checklists. My colleagues are all competent, yet they produce much better work if they are forced to first come up with a design document.

euyyn 10y ago

I agree with what you're saying, but what you're saying is not how it works at Google. At least the teams I've been at. Design docs aren't a prerequisite to start working, and in many cases a design is nonsensical if you haven't actually at least prototyped what you want. It's just a tool to help you be comprehensive when you want to decide between alternatives, and let other eyeballs help you decide. "this will never work", without an actual argument, isn't Googley :) A big design doc is a liability for sure, in the same way as code is.

beat 10y ago

I can tell you've never dealt with one of those trivial backouts on a product that moves a hundred billion dollars a day and is a key component of the entire US economy. Or a product where a bad software deployment can actually kill people. Process is the scar tissue of the enterprise. Those scars are there because the enterprise was wounded. Lots of process, many scars.

fdsfsaa 10y ago
I can choose not to work in an environment riddled with "scar tissue". Instead, I can go work at a startup and eat that enterprise's lunch with a tenth of the budget. Unfortunately, thanks to inflexible and sanctimonious attitudes some programmers adopt about what is and is not "responsible" engineering, the only way to change practices is to beat the old practices in the marketplace.
- beat 10y ago
  You're welcome to try eating Google's lunch, since their process is your favorite example. But try tackling banking, or insurance, or transportation logistics, or any other big-boy problems with that attitude. You won't last long.

yawz 10y ago

> "It's much better to move fast and iterate quickly than to create an illusion of care and add friction to every aspect of the development process." This!

leepowers 10y ago

Wait, you don't spec out project changes? You just start writing code and attempt to figure it out as you go along? That seems like a recipe for disaster, or at least a lot of wasted effort. How do you even quantify that you've done something useful if you haven't set a goal that everyone can agree on?

imagist 10y ago

> Code is just code. Bad changes can be backed out. It's much better to move fast and iterate quickly than to create an illusion of care and add friction to every aspect of the development process. Up-front design of software just does not work. If it did, waterfall project planning would be successful. Bad changes can be backed out, but in an infrastructure with any size, this isn't a trivial task. I've worked in a million-line codebase with multiple separate deployments, and in that case, design documents were much cheaper to create and maintain than a rollback or even writing working code and then discarding it. In an infrastructure of Google's age and size, the trade off is even more clearly in favor of up-front design. You probably think the "move fast and break things" ideology you're espousing is agile, but it's not. It's just another plan, and agile is about responding to change over following a plan. I hope that if you ever work on a project of Google's size that you can adapt.