22 ms·
Ten years against division of labor in software
- phendrenad2 5y agoDo not use a library without reading it's source code. Software problems are hard, and you can't hide from the complexity by letting someone else do the thinking. You have to know what the library author knows, so you can determine what's possible, and decide if the library provides you with the right abstraction.
- cortesoft 5y agoThere is no way I have time to read and understand every library I use. I have to read all 100k+ lines of SQLite before I can use it?
- draw_down 5y ago
- fatbird 5y agoEven in my own work, when I'm moving back and forth across the code, I'll set up interfaces and contracts where appropriate, because it's important to have stability somewhere in the evolving mess. One of the problems you can run into while free-range programming is chasing the consequences of a change back and forth, where a fix in one place requires a fix in another, requiring a fix in another... When I mentor coders one of the core lessons I try to get them to internalize is to code from known working state to known working state. Always be able to run something, either the code or a unit test, so that you know it works as expected. Add some code and get to the next known working state. This requires no undefined dependencies. If you're suddenly in a position where you're saying "I don't know why this doesn't work", you've bitten off more change than you can handle and need to back up and stabilize by decomposing the working state you're aiming for. Interfaces and contracts help immensely with that as long term stable scaffolding, as a skeleton for your codebase.
- nyanpasu64 5y agoInterfaces and contracts are good. Telling people to not care about the code that implements the interface and contract, making it difficult to jump into a Cargo crate's source code and add debug logging, making it hard to tinker with your dependencies, is bad.
- deleted 5y ago[deleted]
- TameAntelope 5y agoIt is genuinely frustrating to work with someone who insists on building tools that others have built better and actively maintain. I have, in the recent past, told someone if they can't get used to using 3rd party libraries, they should seek employment elsewhere. I'm just tired of having the argument. Maybe some shops can afford to build everything in house, but nowhere I've ever worked can.
- hinkley 5y agoI simply don’t understand why a junior developer would want to work at a company that uses as much proprietary software as some shops do. What have they learned at the end of their tenure that they can apply somewhere else?
- TameAntelope 5y agoI agree, by learning how to use tools they'd have access to for the rest of their careers, they've learned how to be useful to the rest of the world. Turning your talent into something that makes people's lives better is immensely useful and satisfying.
- politician 5y agoIf employers hire junior developers for reasons other than that they have some capacity to train them (such as cheap labor), then, agreed, they are doing them some disservice. But for folks that are not so junior, their insistence on rewriting libraries often is a consequence of a lack of confidence or misunderstanding about the purpose of the software project. If you’re building a new database for use with your new language, yeah, it’s worth building it from scratch. But for many older shops, simple was made easy long ago through selection of libraries that don’t need to be rebuilt for pedagogical reasons.
- hinkley 5y agoWe had a guy who left (a mess), before he left he asked one of the office Nice Guys his opinion on open sourcing a framework he wrote. Even the nice guy said it was a bad reimplementation of something that already exists. The logical conclusion is that he was already looking to leave and wanted to take his homunculus with him, but my head cannon is that he realized his play time was over and people weren’t going to kiss his butt anymore. There are distractions I find healthy and ones I find unhealthy and most weeks I’m trying to juggle the mix of both. Among the ones I don’t like are people asking me questions about my code that they should be able to answer themselves. That means I’m more open to questions from junior devs, because they can’t, but if people are stuck I want to make it easier for the next person. Or me in a year when I wander back into a project that was some definition of “done” before new requirements came in or my perspective on “good” has shifted (these two are not independent variables).
- erikerikson 5y ago> Software libraries suck. Here's why, in a sentence: they promise to be abstractions, but they end up becoming services. The file abstraction seems fine. Not having to understand physical wiring of disks, the specific disk manufacturer's wire protocol, and so on is Labor saving. Many of the abstractions don't even force the seek and location concepts but allow "just put my string there" semantics. The addition abstraction seems fine though if you have a limited length number type there are a few things to consider. Arbitrary precision libraries eliminate those nuances though perhaps move you to others like numbers that cannot fit in available memory. Yet those are rather extreme corner cases. It's a provocative claim but the universal declaration and the confidence it seems to project seem as weak as some of those abstractions being railed against.
- deleted 5y ago[deleted]
- cxr 5y agoThis is not someone making a provocative post claiming that abstractions are bad. It is someone who understands that abstractions are good but claims that it is bad to rely on libraries to solve the need for abstraction.
- deleted 5y ago[deleted]
- datavirtue 5y agoI have been ruminating on this very thought for some time... "I started out mocking NPM for the left-pad debacle, but over time I've actually grown more sympathetic to left-pad. The problem isn't that somebody wrote a library for left-pad. It is that even something as trivial as left-pad has triggered our division-of-labor instincts. A library like left-pad is just begging to be saved into projects that use it so users have one less moving part to worry about at deployment time. Turning it into a first-class dependency also ensured that a lot less people looked inside it to understand its five lines of code or whatever." I think we should be extracting code from libraries until we need the whole thing. Dragging around black box dependencies as binaries has a lot of serious drawbacks. If people had extracted the parts they needed from log4j they wouldn't have even had to think about the exploits.
- Jtsummers 5y agoPulling out just the subcomponents you need is actually easier with a more modular design. But akkartik also opposes modularity, or at least has expressed a less-than-positive disposition towards it: > I think namespaces and modules are socially counter-productive even when implemented in a technically impeccable manner, because they encourage division of labor. Namespaces and modules are precisely what you need to be able to select parts of a larger system and ignore other parts. Without them, you're faced with numerous "monoliths", excessively large systems that don't decompose into reusable, isolatable bits. That forces you to import too much, or to recreate just the part that you care about (often a waste of time).
- ReactiveJelly 5y agoI was very surprised by that line, too. I make namespaces and modules in my own projects just so that I can use simple names like "update" or "load" un-ambiguously. Projects that nobody else has ever touched and maybe nobody ever will. It's been a popular C++ style since before I did C++, and now it's also idiomatic Rust style. Is OP really advocating for a C style where everything must be fully-qualified? It's horrifying to read GTK+ C code because of that. Everything is `gtk_do_a_thing` and all code operates on a `GtkTypeOfGtkObject`. One of these days I'll get RSI, I only have so many keystrokes left in me! Have mercy! And you can't rename things, so good luck mixing incompatible major versions of Qt or GTK+ in a C or C++ project. (Actually, Qt has a compile-time flag to put it in a namespace. Maybe GTK+ has some similar ugly macro. But this requires you to compile your own dependencies, a task which is especially annoying, inconvenient, and error-prone in the C family)
- abvdasker 5y agoThat’s a wildly impractical position to take and isn’t really true in my experience. I spent several years at a seed stage startup where it was me on backend and one other engineer on mobile. We would specify a JSON API and then could work independently (and very fast) to realize the functionality because he’s an expert at mobile programming and I am an expert at backend. We didn’t have the luxury of working at 50% output for a year to learn each other’s specialization while trying to get the product off the ground. He didn’t need to learn the ins and outs of rate limiting with redis and I didn’t need to know how to animate a custom UI component for each of us to be maximally productive. Division of labor and the interfaces which enable it exist for a reason. I suspect it’s because it reduces the complexity of collaboration between engineers with different specializations. It facilitates the separation of concerns.
- zem 5y agowhile knowing your tools is always a good idea, i disagree with the overall notion of "don't divide labour" because a lot of the real-world problems software has to deal with are fractally complex. for instance, to pick a seemingly well-contained problem: if i wanted to write a collaborative text editor i would absolutely want to rely on someone else's work for font rendering or fault-tolerant networking, because each of those are problems that i could spend years on, and still not have covered all the failure modes and corner cases.
- popularonion 5y agoI would like to point out the contrast between the mostly-negative comments in this thread and the overwhelmingly-positive comments about SerenityOS: 1. https://news.ycombinator.com/item?id=27317655 https://news.ycombinator.com/item?id=27317655 2. https://news.ycombinator.com/item?id=28817599 https://news.ycombinator.com/item?id=28817599 3. https://news.ycombinator.com/item?id=28206840 https://news.ycombinator.com/item?id=28206840 4. https://en.wikipedia.org/wiki/SerenityOS https://en.wikipedia.org/wiki/SerenityOS
- deleted 5y ago[deleted]
- jeff-davis 5y agoWhat if we think of libraries and dependencies in a different way? For instance: dependencies would be at the granularity of a single function or data type. And when you take a dependency, in your editing environment it would look just like your code and fit right in, maybe even with your preferred autoformatted style. The editor would realize that the function has an external author, and could git-pull new changes in, or you could fork the function (and maybe some merges would conflict). The advantage is that the code would feel a lot tighter and less mysterious, though still allow collaborative development. Rust isn't way off here. It already does a lot of tracking of individual functions during the compilation process, and docs.rs already provides quick access to a lot of source code for dependencies. Closing the last few gaps to make it feel like part of your program, rather than outside magic, could be awesome.
- SergeAx 5y agoDid I miss some important context here? What is wrong with the division of labor in the first place?
- mjburgess 5y agoThere is something wrong with division of labour in creative projects... the story of the service industry seems to be how this is relearnt over-and-over. A "silo" is incomprehsible in a factory: the pin-machinist isnt silo'd. Knowledge, skills, vision, creative engagement... these are silo'd. And this follows from misconceiving creative work as a form of goods-producing labour.
- akkartik 5y agoYes! I'd like to reiterate my link to https://creators.vice.com/en_uk/article/animator-hand-draws-film-in-4-years-nova-seed https://creators.vice.com/en_uk/article/animator-hand-draws-... in OP.
- aspenmayer 5y ago> Knowledge, skills, vision, creative engagement... these are silo'd. And this follows from misconceiving creative work as a form of goods-producing labour. It’s also related to the division of decision-making, usually between workers, with little or no executive authority, and management, with some to all of the executive authority. Creative roles are typically closer to management, and their knowledge bases are siloed, as is the business’s operational experience and expertise. Everyone wants to be close to power, but how many actually wield power effectively? What types of hierarchy are amenable to diffuse power structures? A lot of business are run like petty fiefdoms, and may not benefit from a minor change, and yet may not have the stomach for radical reform either. Worker owned co-ops and DAOs give me hope for new ways of working.
- dpark 5y ago> A "silo" is incomprehsible in a factory Efficient factories are built on silos. You do your job in your area and the next person does theirs. You might collaborate, but you absolutely have a clear division of labor. If you don’t have a division of labor and silos, you don’t have a factory. You have artisans.
- hinkley 5y agoBecause a lot of the profit comes in vertical integration. I mean that’s why you built the factory right? To take materials in one end and put something that is greater than the sum if its parts out the other?
- Msw242 5y agoFrom my reading it is less about being "against" division of labor than it is about being against "premature" division of labor. Amazon's API Mandate[0] was arguably incredibly successful, and created the necessary conditions for AWS to emerge. [0]: https://nordicapis.com/the-bezos-api-mandate-amazons-manifesto-for-externalization/ https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...
- imbnwa 5y agoYeah I lot of this thread is arguing against a strawman
- ginko 5y agoSo should all projects implement their own kernel and network stack?
- throwaway984393 5y agoAnd why aren't web developers writing compilers and linkers?
- HideousKojima 5y agoI wish they had, instead of the nightmares they created like Webpack
- NAR8789 5y agoIt's interesting to contrast this against the concepts in Team Topologies (https://teamtopologies.com/ https://teamtopologies.com/) - Kartik argues for better mutual understanding and additional communication - Team Topologies states you should match system boundaries to team boundaries, specifically when cognitive and/or communications load have grown unmanageable due to team size and/or software complexity On the surface, Kartik is arguing against abstractions and Team Topologies is arguing for abstractions, but both stances feel valid to me. It feels like there's a spectrum of tradeoffs here. Is this essentially an analogue to the tradeoffs of monolith vs services? - For small teams, by all means keep abstractions low and communication high--monolithic systems and less division of labor is ideal and your team will gel better for it. Your team is probably not large enough to warrant either services or division of labor, so stick with the monolith. - As an organization scales, you end up with an n^2 communication edges problem, and at this point division of labor is necessary to limit the cognitive and communications loads.
- mprovost 5y agoThis sounds like embracing Conway's Law [0] instead of observing it as emergent behaviour. [0] https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law
- NAR8789 5y agoPrecisely. Team Topologies strongly advocates the "Inverse Conway Maneuver", where you shape your team and communications structures towards your desired architecture.
- akkartik 5y ago"On the surface, Kartik is arguing against abstractions.." Nope. To clarify: https://news.ycombinator.com/item?id=30019146#30038853 https://news.ycombinator.com/item?id=30019146#30038853
- redredyupdy 5y agoAs someone who deals with APIs, I understand the point of abstraction. But when you start to deal with performance problems, a high level view of the internals can be invaluable for finding the problem. Something like a CRC diagram is invaluable http://www.agilemodeling.com/artifacts/crcModel.htm http://www.agilemodeling.com/artifacts/crcModel.htm To clarify, this then allows you to look inside of the API (to bunch of things talking over more APIs, yay microservices). Then you can see which bit might be causing the issue with the bit of the larger black box you are talking to
- codeflo 5y agoI think Hyrum’s Law should be best understood as a warning for users rather than as handcuffs for implementors: don’t rely on non-obvious implementation details. I’ve seen software break due to a race condition when an upgrade made something faster. Hyrum’s Law tells us the performance was a hidden part of the interface. Is the solution to never optimize anything? Instead, I claim the client code was broken all along.
- fragmede 5y agoWhere client code is something someone outside the organization implemented and is using, that makes a certain amount of sense. But when the client is another internal team, and the SaaS product the company sells broke, the customer doesn't really care that the database team and the API team don't get along - the product is down and while it's down, no sales are happening. The point, thus, of looking into other teams is to get a sense for the challenges they face and to have a less adversarial working relationship with them.
- kibwen 5y agoIn such a scenario I'd say that the distinction between the two teams is essentially imaginary. If management is forcing the downstream team to use the new API even if it's broken, and then management is forcing the upstream team not to break the API for the benefit of the downstream team, then the API is an illusion.
- eternityforest 5y ago...
- eternityforest 5y agoThe fact that a library discourages you from learning it's internals is how you know it's well done. If you needed to understand it, you could. The reason you don't is because you've never had a compelling reason to. If only one person knows how a library works, that is a problem. If a 100 person team maintains it and you're not on that team.... it's probably because you have other stuff to do. Software is an engineering discipline. Computer science is a science. If you are in programming because you want to advance our understanding, great, go work in one of the many fields with large novel algorithms that need to be understood. For a typical programmer... Look around. Modern software is one of the best things about the modern world. It does SO much for us. Do you really think we, with a real world distribution of programmer abilities, could do all this without massive division of labor? How much would be missing if we insisted on understanding everything before we use it? I suspect very few alternative approaches to software would work as well to truly build the modern software defined world where everything from cars to taxes to photography is digital. Because... whenever someone tries an alternative approach, there usually seems to be the hidden unspoken assumptions that they actually don't want software to be as big as it is. The end product of software itself these days is a service(in the sense of this article, not the SASS sense), wheras software built with an understanding-first mindset seems to usually wind up being closer to a digital version of a passive analog tool.
- imbnwa 5y ago"Division of labor is an extremely mature state for a society. Aiming prematurely for it is counterproductive. Rather than try to imitate more mature domains, start from scratch and see what this domain ends up needing." This is more the gist of the article than "lets not have libraries or abstractions at all", its pointing more towards "lets defer those questions until the right time rather than from the jump". I think Enterprise Java and the insane complexity of modern web dev are indicative of a consequence. >Because... whenever someone tries an alternative approach, there usually seems to be the hidden unspoken assumptions that they actually don't want software to be as big as it is. I really don't think that OP is making this assumption
- hinkley 5y ago
- wrnr 5y agoIn my lifetime Apple has changes hardware platform twice while still running on the Next Foundation API. Same thing with politicians, they never simplify the complicated laws but instead expect software to manage and work around this complexity. Nothing to do with the division of labor but more the packing order of software engineers.
- ape4 5y agoDon't think about how a library works - until you have to
- WJW 5y agoHow do you know when you have to? There might be dozens of libraries in just the current project that we could be using more effectively if only we'd taken the time to read the source.
- Jtsummers 5y agoWe have this wonderful organ called a "brain". It appears to be the center of reasoning. We apply that to the situation and end up drawing a line where we say, "Beyond this point, I will not look. I may learn what it does, but I will not dwell on how it does it, neither in principle (conceptual model) nor in fact (actual code). Because if I were to investigate this, and every other such system, I would find that it is turtles all the way down and I will never actually complete my objective. Unless I choose to freeze all these dependencies at this one moment in time and never permit them to change. If, in the course of my work, I find that this particular area is a source of trouble or a possible source of solution, then I will investigate further. On my own, with my team, directing a team member, or contacting the responsible party." It's a noble goal to make systems that are comprehensible, and one that should be striven for more often. It's also a noble goal to try to fully understand the stack of dependencies. But short of freezing time (this subsystem will never be permitted to change after I have studied it, except by my own hand), spending a lifetime on a more narrow field, being in the top 0.0001% of intellects, or working on relatively simple systems, it's infeasible in the general case. Once you try and do something even as "simple" as build out an avionics system for an aircraft, it will quickly explode into too many domains for any one person to truly comprehend every detail.
- WJW 5y agoYou are certainly much smarter (or more inexperienced in aircraft software design) than me if you think avionics systems are simple. But yes, I also have a brain and can reason with it. You don't have to restate the obvious as if it is some deep wisdom. But parent comment made the point that you should only look at library code when you have to and I countered that you cannot know when you have to without looking. Your counter-counter that you must think about what you have seen does not help with deciding when to start looking in the first place.
- lkrubner 5y agoThe best counter-argument was given by Alfred North Whitehead back in 1912: "By relieving the brain of all unnecessary work, a good notation sets it free to concentrate on more advanced problems, and in effect increases the mental power of the race. It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments." https://en.wikiquote.org/wiki/Alfred_North_Whitehead https://en.wikiquote.org/wiki/Alfred_North_Whitehead
- akkartik 5y agoOP was me annoyingly quoting myself, and here's my refutation annoyingly quoting OP: "Software libraries suck. Here's why, in a sentence: they promise to be abstractions, but they end up becoming services. An abstraction frees you from thinking about its internals every time you use it. A service allows you to never learn its internals." "Confusingly, I've been calling Iverson's notion of subordination of detail 'abstraction', and Iverson's notion of abstraction 'service' or 'division of labor'. Though lately I try to avoid the term "abstraction" entirely. That seems on the right track. Regardless of terminology, this is a critical distinction." As I hope these clarify, I have nothing against true abstractions as you describe them.
- passion__desire 5y agoYou can’t do much carpentry with your bare hands and you can’t do much thinking with your bare brain. (i.e. using appropriate Thinking Tools and Intuition Pumps) As Daniel Dennett says, "we are productive now because we have downloaded more apps to our necktops (i.e. brains)" Intuition Pumps and Other Tools for Thinking : https://www.goodreads.com/en/book/show/18378002 https://www.goodreads.com/en/book/show/18378002
- ramesh31 5y ago>Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments. I love this analogy. People who aren't accustomed to doing long form, extended, in-depth thought about a single problem don't seem to understand this. That your attention and mental effort is a finite resource, that can be frivolously wasted or put to good use. It requires real concerted effort, and it isn't easy. Thus, why most people never really think about anything.
- throwaway984393 5y agoHN trope #317: blog post preclaiming that the one thing we all know makes sense and has been proven to work for decades is actually wrong.
- deleted 5y ago[deleted]
- togaen 5y agoSound, sage advice that no one will listen to.
- axiosgunnar 5y agoThe UX designer doesn‘t need to know how React works. The React developer doesn't need to know how TCP/IP networking works. The network expert doesn't need to know how a kernel works. The kernel developer doesn't need to know how a CPU works. The CPU engineer doesn't need to know how silicon metallurgy works. And whoever smites metals all day sure as hell does not need to know how to draw pretty icons. etc. But all put together we are eating the world. It‘s beautiful.
- gtsop 5y agoThis isn't entirely true. Knowing how react works (or doesn't work) helps a designer design a valid experience that can actually be implemented by the developers. Ok maybe they don't need to know React, but surely they need to know what a browser can and can't do. A react developer (web developer) needs to know how tcp/ip networking works to avoid spending hours of debuging communication problems with XYZ service. And so on. Surely people can go about doing their jobs without necesserily knowing how the adjecent stacks work, but they would be way better professionals of they did. I would agree more with your argument if you restricted the distance of irrelevant tech-stack to let's say 3 levels. Eg, the web developer truly won't do any better if they knew more about kernels.
- ItsMonkk 5y agoI like thinking of it like the Olympic rings. The blue ring(all the way to the left) doesn't need to know anything about the red ring(all the way to the right), but should absolutely know the yellow ring(second ring from the left), and possibly bits of the black ring(middle ring). The leakier the abstraction, the more you need to reach.
- TameAntelope 5y agoBut it's mostly true, and that's what matters. Nobody's trying to build a super rigid structure around the division of labor, they're just trying to create a general set of guidelines that will help answer questions quickly.
- ginko 5y ago
- ajuc 5y agoAbstraction is necessary, but I think OOP as practiced in many companies likes abstraction a bit too much. At the extreme people treat everything except the code they currently work on as black boxes and code like this is written: //this was written long ago in one class List<Book> getBooks(BookRepository bookRepository, String author) { var books = bookRepository.getBooksByAuthor(author); if (books.isEmpty()) return null; return books; } //this too but in a different task and in a different class Collection<Book> getBookAboutKeyword(String author, String keyword) { var books = getBooks(this.bookRepo, author); if (books == null) { return null; } // let's ignore the fact this should use filtering on db return books.stream().filter(Book b -> b.about(keyword)).collect(Collections::toList()); } //this is written now and there's a NPE List<Book> getBooksAndFooBarThem(List<Books> books, String author, String keyword) { var books = getBookAboutKeyword(books, author, keyword); //foobar them return books; } And of course the solution to NPE was to put ANOTHER null check in the getBooksAndFooBarThem function because anything up or down the callstack is black magic.
- somethingAlex 5y agoI feel like all of the downsides given in the quotes can be solved by making sure people get a variety of work. Define an interface and split the work into chunks. Just make sure people are being assigned to different kind of chunks should their skill allow it. This is really just the crux of managing people and bringing out their potential. I’m not gonna split a CRUD endpoint or class between two people but I’ve also had two people write the same exact function because it was so complex.
- dpark 5y agoThis screams “toy projects” to me. Separation of concerns is a hard-won insight. We call code without this “spaghetti” because it eventually becomes nearly impossible trace or to reason about. Even if you have all your engineers working all over the whole codebase, the codebase itself still needs separation of concerns to simplify reasoning about behavior. This is considered a good engineering practice because the alternative is that you need to hold the whole system in your head to understand any behaviors. I’ve seen codebases like this. It’s not pretty. Use of abstractions to simplify cognitive overhead is not a weakness. The idea that projects should take source copies instead of library dependencies is just kind of nuts, at least for large libraries. Yes, the proliferation of tiny libraries is obnoxious and adds a ton of pointless overhead. But for large libraries, taking a source copy is absolutely crazy. Imagine taking a source clone of Chromium into your project because you need a rendering engine. Yes, you can probably cut a bunch of code out as a result, but you will also now need to deal with security concerns and incorporation of new features until the end of time. And this is now way more expensive and you can’t just do a straightforward merge because you changed the code. 5 line package? Sure, just copy the code in (assuming licensing is in the clear). 500 dev-year codebase? Cloning that code is by far not a win.
- akkartik 5y ago"Separation of concerns is a hard-won insight." Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people. It's certainly reasonable to consider my projects toy. I consider them research: * https://github.com/akkartik/mu https://github.com/akkartik/mu * https://github.com/akkartik/teliva https://github.com/akkartik/teliva "The idea that projects should take source copies instead of library dependencies is just kind of nuts..." The idea that projects should take copies seems about symmetric to me with taking pointers. Call by value vs call by reference. We just haven't had 50 years of tooling to support copies. Where would we be by now if we had devoted equal resources to both branches? "...at least for large libraries." How are these large libraries going for ya? Log4j wasn't exactly a shining example of the human race at its best. We're trying to run before we can walk.
- hinkley 5y ago