13 ms·

The Failed Commodification of Technical Work

• andy99 3y ago
  He doesn't explicitly cover the whole AI angle to this - in the spring I wrote something about the obvious parallel between AI tools and low code programming which I think is relevant, basically that both make something easier but anything outside of that something harder so they don't really add efficiency https://gist.github.com/rbitr/3294819148316df3ed90a2a1ce8a9188 https://gist.github.com/rbitr/3294819148316df3ed90a2a1ce8a91...
• edent 3y ago
  I think - and it's only a think - that the author has ignored that a large part of what used to be called technical work is now commodified. I remember when a mail merge literally meant printing out lots of address labels and then manually sticking them onto a letter & envelope. Word 2.0 (?) solved that problem for the 1990s and MailChimp has commodified it for the 21st century. Double-entry book-keeping was technical work and was usually run by highly trained individuals. Nowadays every shop keeper just scans a barcode and has the customer tap-to-pay. There's not yet a drag-and-drop like interface for anything more complex than Scratch (wither Visual Basic!) but the hard part isn't the technical work of stringing together libraries; it's requirements gathering. Speaking of which, it has never been easier to drop in a high-quality cryptographic library, or import an interactive map on a website, or WYSIWYG edit a website. So, the author is right that you can't stick a bored 18 year old in front of an IDE and have them create you an ERP. But a lot of the "grunt work" of IT is now firmly a commodity.
  • steveBK123 3y ago
    I guess the point may be that after 30-40 years of this, the low hanging fruit of commodification may be gone. Further, the more we commodify, the higher order our problems become and the specialist engineers you hire move further and further up the stack. Also, not sure if it's always been the case.. but the latest vintage of SaaSified startups have a high % of products that don't actually do any of the things you want them to do yet. They want you to pay them for their service, so they can capture your use cases for implementation and then commodify them for other customers. So you end up with long lead times and IP leakage. Neat! I think the example of templating SQL is always this misunderstood target for management. I dunno, the language in particular has survived an incredible length of time in our industry... it actually does a pretty good job. 99% of wrappers/DSLs/etc put on top of it make it far worse and still require you to dip into SQL for anything remotely non-vanilla. Further, instead of hiring SQL experts (there's many) you need to train up SaaSified DSL SQL wrapper X experts (none exist).
  • Dardania 3y ago
    to bounce on your point, there's the distinction between creative & non creative work. Sure, tools might help the creativity, but it can't replace it, and their article discusses how it can make it worse. Accurate requirements gathering requires a spark of creativity...
  • danwee 3y ago
    What you say is true, but the amount of "grunt work" is not constant over the years. In fact, I think the amount of "grunt work" in teh tech industry is just growing and not shrinking; I think the following look is quite obvious: - amount of current grunt work: X - new tech Z appears that makes X be reduced to 0.1X - at the same time Z enables new ways of doing things. Some things become grunt work because they are a byproduct of Z - amount of current grunt work: Y (where Y ~= X) - ... If the technological progress had stopped in the 2000s, then all the grunt work (originated in the 90s) would be esentially zero today. New tech just brings automation and grunt work. I don't think we will live in a society where there's practically no grunt work. The most recent example is AI: there are AI tools that generate sound, images, video and text... but if you want to create a differentiating product/experience, you need to combine (do the grunt work) all the available tools (chatgpt, stable difussion, etc.)
    • steveBK123 3y ago
      Right!! 10/20/x years ago we didn't have DevOps, CloudOps, CloudFinOps, CloudSecOps, IaC experts, Cloud Architects, Cloud transformation experts, Observability architects, SREs, plus all the permutations of roles around "data" that didn't exist discretely, etc etc etc.
      • esafak 3y ago
        We did not have web scale products, which enabled new possibilities. E-mailing documents and collaborating offline sucked.
    • BurningFrog 3y ago
      > I think the amount of "grunt work" in the tech industry is just growing and not shrinking... Not sure, but isn't this just another way of saying that the tech industry keeps growing?
      • csydas 3y ago
        I'm not sure what the parent post meant exactly, but I do agree there is tons of grunt work -- I've seen big name SV companies where large parts of their work flow include parts like "and then just every hour you need to do something in a slow UI that can't be automated" to keep vital systems working. I would say that's really grunt work, and there are even persons in such companies where their only task is doing such grunt work. Truly I've been told by clients I work with they have entire double-digit sized teams where the members only responsibility is to reboot VMs that breach specific resource thresholds -- easily automated and even built into most hypervisors, but for whatever reason these tech giants opted for a human to do it -- the only semi-reasonable explanation I got from one client was that their infrastructure team got outsourced and they laid off the only people who knew how to use the automation tooling. It's a dumb reason for sure, but at least I can understand why they opted for the manual grunt work. Similarly, keep in mind a lot of this grunt work is just to satisfy some reporting requirement from somewhere -- some person(s) in the company want to see at least X% of uptime or Y LOC every day, so you get people trying to write a lot of yak shaving code that basically does nothing except satisfy the metrics or ensure that uptime % always looks good (i.e., they don't fix the cause of the downtime entirely, they just get the endpoint that is checked to determine update working well enough so it reports to the monitoring system and they leave it at that)
  • loloquwowndueo 3y ago
    Nah, WordPerfect and wordstar had mail merge way before word did.
    • suoduandao3 3y ago
      I miss WordPerfect. Maybe it was the novelty but it had such enjoyable fonts.
      • loloquwowndueo 3y ago
        Well I remember when it had no fonts :)
      • anonymouskimmer 3y ago
        It's still there if you want to buy it. At least in name.
  • fulafel 3y ago
    I wonder if there's a term for Amdahl's law but applied to human processes. Like the other side of "law of diminishing returns".
    • davidgerard 3y ago
      "Adding manpower to a late software project makes it later." - Fred Brooks, 1975
    • random3 3y ago
      Amdahl's Law applies cleanly to human processes. Perhaps the most revealing example is the origin of "computer" as a human occupation and how scaling the compute process happened at Los Alamos https://ahf.nuclearmuseum.org/ahf/history/human-computers-los-alamos/ https://ahf.nuclearmuseum.org/ahf/history/human-computers-lo... The more general aspect of Amdahl's law is captured by certain scaling laws and limits generally related to communication (see full bisection bandwidth) and certain architectures (e.g. Cray) meant to optimize for this
      • fulafel 3y ago
        It semantically works but it has not been adopted by people outside computing, I'd guess think the definition isn't relatable or understandable enough for people coming from humanities/biz backgrounds, so it might well be possible that there's a parallel concept there.
  • martingalex2 3y ago
    the people you're supposed to be eliciting requirements from are just regurgitating what ChatGPT told them are the requirements hehehe
    • dexwiz 3y ago
      This is becoming so true. I have read so many documents in the last year that are obviously from a GPT, especially when it’s about something new to a group. But in the end, I would rather get a half baked GPT doc than a quarter baked junior analyst doc. I just worry that GPTs are going to kick the rungs out of the bottom of any knowledge work later. Being bad but junior used to be a learning environment without too many repercussions. But how do you compete with peers using AI? You use it also. But now you have robbed yourself of a learning opportunity. Yeah you can learn someway by doing it, but it’s like doing homework by looking at the answers. Sure it can help you double check, but if you don’t put the effort into constructing your own answer, then you have only cheated yourself. I think the AI alignment issues are probably over blown in the short term, but what about the long term when the average person has regressed so far as to be unable to live without AI. They will just do whatever is told to them.
  • ludicity 3y ago
    This is a fair critique, and I'm giving it some thought now. I'll need to stew on it a little bit. Maybe the fundamental issue is that many of these products are designed to, as one of the other commenters noted, appear to purchasers that don't work in the field as simply appliances one purchases and then the problem is solved. The issue is that a lot of the stuff out there doesn't actually solve the problem - it just appears to because other people buy it, and then lie about the implementation being successful to get promoted. Things like mail merge -are- like kettles, they're solved problems, and the only way to solve them is to try things. The broader issue is that my employer purchased Workday because they believed it's like a kettle, but it can't actually fix the fact that our org structure is so horrendous that it can't be modeled. (Incidentally, this year is the first year that I've realized that a sufficiently bad org structure, in a large company, tech debt of a sort. You end up doing all sorts of crazy things just to work out who works for who, and what can this user see in this database, etc.)
  • atmosx 3y ago
    Some aspects have indeed been commodified. But what about the bigger picture? How simple is it to run a business, a website, organise a travel around or pay a bill nowadays, compared to 1994 or 2004? At times, I can't help but feel that the previous generation had a more leisurely pace of life, which led to a more fulfilling lifestyle. Nowadays, time seems to pass at a rapid pace, with high levels of stress. Allow me to share two experiences: a) The other day, while at the bank, I witnessed at least three individuals over the age of 60 struggling to complete simple tasks, aimlessly wandering around and pleading with the staff for assistance. These tasks are supposed to be easily accessible through online banking, but due to certain exceptions, the system did not support their specific needs. As a result, they were forced to make appointments, with the earliest available slot being three to four months away. One of them needed to withdraw money from a blocked account to purchase wood and heat her home, but the bank's staff refused to budge, insisting that she wait three months to solve the problem. b) Just two years ago, my father was in Sicily and could not find a way to make a simple phone call back home. Yet, in the 1970s, all he had to do was walk into the bar in the area with a few coins. Not to mention that while once upon a time your average person could fix the lights, the car, the heating, the non-automatic door, etc. by themselves now they need to call the professionals.
• mattgreenrocks 3y ago
  I think he’s right. The tech industry has been trying to commodify devs for a long time (COBOL, Java). But there’s a sort of essential quality that reasserts itself no matter what you abstract. Despite the seemingly simple requirements paired with high level frameworks, a lot of our software still doesn’t even work well. As the author notes. The only real fix is talented devs that care. You can make a career out of that.
  • steveBK123 3y ago
    The problem is that beyond the boilerplate we aren't solving commodity problems most of the time. If you've worked in the same field long enough, you'll see that things certainly rhyme, but everyone has slightly different business requirements. At each level of the stack this grows, and so in total there's a ton of non-commodity, bespoke work to do. That's why no two products are exactly alike and we don't just have 1 giant all world megacorp producing everything.
    • booleandilemma 3y ago
      The problem is that beyond the boilerplate we aren't solving commodity problems most of the time. The only people that see it is a problem are the ones that wish programmers were commoditized. Your complexity is my salary.
      • pixl97 3y ago
        That complexity is how companies typically differentiate their product, generally purposely to keep it from interoperability with other software so their entire business is not commodified. Just look at the IOT space to see this in play.
      • TeMPOraL 3y ago
        > Your complexity is my salary. Then your salary paid out of broken window fallacy. But that's fine. All the nice, humane things happen where the market optimized things a little bit, but not too much.
    • 3y ago
• fredgrott 3y ago
  Reminds me of some of my startup skeleton stories.... A founder so dumb he cannot understand the basics of one app equals one domain...wanted to fill each domain he owned with a part of a major app and then sell each domain and not understanding that would destroy the full app if we could ever break the one app one domain equation in the first place!
• gitgud 3y ago
  About 10 years ago a few friends of mine (mechanical engineers) were surprised that I was studying software development. They said something along the lines of “is there much left to do? we can just use existing systems to do everything we need right?” The misconception is that building systems to tackle new problems are easy and thus have been “commodified” meaning nobody needs to write code anymore. The reality is that building software is rarely as easy as configuring a UI. You end up needing text which represents logical rules and flows, you need version control see how the system changes, rollbacks… which means you need programmers Coding doesn’t disappear, it just moves up the levels of abstractions
• treprinum 3y ago
  I once worked for a company that ran on the Phoenix Project ideas, treating devs as mechanical cogs in a factory; last time I've heard about them they were shutting down their main subsidiary and laying off most of the staff (already cherry picked from all around the world to minimize costs by automated tests more difficult than interviews at Google) while the leadershi* that led to that kept their positions intact.
• simonbarker87 3y ago
  I agree that the full commodification of technical work is a bad idea and will, hopefully, continue to fail. However, having read the Phoenix Project twice and hating most of Scrum, I disagree that’s what the Phoenix Project is advocating for. My main takeaways from the PP are: 1. Have clear systems in place to carry out and manage your repeatable work, automate where possible 2. Minimise the time work is in progress for so people aren’t bogged down with a million tasks 3. Share information widely and have multiple members of the team able to carry out the same task 4. Make sure the work being done is what the business actually needs doing 5. Reduce noise and unplanned work so staff can get on with the higher value work they actually enjoy rather than wading through a quagmire of disorganised chaos The point of PP isn’t to turn people into interchangeable automatons - it is to put a system in place to allow people the headspace and time to do the really valuable work that can’t be automated or systematised. I’ve run a factory and been a dev so I see it from both sides and making devs production factory workers isn’t sensible but likewise where work looks like factory work (known work, repeatable steps etc) it should be treated in a similar way.
  • Pet_Ant 3y ago
    > hating most of Scrum I'll just say that if you look at Scrum itself there is really nothing objectionable to it. https://scrumguides.org/scrum-guide.html https://scrumguides.org/scrum-guide.html It's the other shit people pack on top of calling it Scrum that usually sucks ass. I've found the best way to fight back against shitty-Scrum is not to fight it, but actually feign puritanical allegiance to the actual doctrine, it's much less repulsive. I makes you look like less of contrarian and it's easier to make an impact that way.
    • simonbarker87 3y ago
      Very true, after being told many times “that’s not what it says in the Scrum guide, we need to do it like …” by a Scrum Master and a PO I decided to read it. I was astounded to find that the guide says very little and they were just using it as a weapon to push their own controlling desires on to the team. All the devs read it and the next time they used that line we asked them what it actually does say, they just made stuff up, clearly they had never read it either.
• tboyd47 3y ago
  It requires an extreme level of selective amnesia to make McDonald's the operations standard for anyone. Try "the ice cream machine is broken today" on your next I.T. contract and see how far that gets you. But to the author's point, I remember checking out a book from 1980 on software engineering from a highly well-known author (I forget who). I was shocked to read the author state unsarcastically that software development labor would be fully itemized in a standard catalogue like auto mechanic by the year 2000.
  • peteradio 3y ago
    > that software development labor would be fully itemized in a standard catalogue like auto mechanic by the year 2000. That's pretty dumb... Software doesn't "break" and if it could be itemized to such a degree it could be made self-healing.
    • acuozzo 3y ago
      > Software doesn't "break" It most certainly does because most software is written atop dynamic systems and every possible change cannot be pre-empted. Extreme example: RAM writes accidentally flipping bits ala RowHammer. Broken, yes, but is it a bug?
      • peteradio 3y ago
        I'd still describe the first one as "non-line-item-able" how can you set a lineitem time to "re-jigger the dependencies"? Those take open-ended investigation that no experienced "dev" would commit small amounts of time to like replacing the alternator in a 2003 Ford Ranger. The second one is clearly an automation issue, are you going to assign a lineitem to "fix the latest rowhammer attack"?
• solardev 3y ago
  I was really confused by the title, because doesn't "commodify" mean "to make saleable", like commercialize? Hasn't tech made billions? I think the author is talking about "commoditization", eg genericizing tech work so that any replaceable employee can do it. From https://en.wikipedia.org/wiki/Commoditization https://en.wikipedia.org/wiki/Commoditization: > This is not to be confused with commodification, which is the concept of objects or services being assigned an exchange value which they did not previously possess by their being produced and presented for sale, as opposed to personal use. One way to summarize the difference is that commoditization is about proprietary things becoming generic, whereas commodification is about nonsaleable things becoming saleable. In social sciences, particularly anthropology, the term is used interchangeably with commodification to describe the process of making commodities out of anything that was not available for trade previously. Am I being pedantic? I thought the two had different meanings? Edit: Ah, but wait... from Wiktionary instead: https://en.wiktionary.org/wiki/commodification https://en.wiktionary.org/wiki/commodification > Sometimes used interchangeably with commodification Guess it's just a common mixup.
  • mikepurvis 3y ago
    I've always heard the term used in the sense that the author is using it. Oil is a commodity because there are many producers of it and it doesn't matter which one you get it from since they're all making the same thing.
    • solardev 3y ago
      Which term? Commodotize? Aren't we in agreement with the author then, that tech work isn't quite the same as oil or burgers (ie it hasn't been commodotized, even though it's been commodifed?) Or are you saying the two words mean the same thing?
      • radicalbyte 3y ago
        The work many were doing 30 years ago - making shitty PHP websites - has been completely commodotized to the point where as a dev many of us don't touch it. Why would we now that business people have enough literacy to click something which looks pretty together with Wix or that other website-as-a-service which funds 95% of podcasts :-) Or hire a designer / commodity PHP-shop to make them a Wordpress.
• marcus_holmes 3y ago
  I think every engineering manager has either worked for or interviewed with a company that believes this stuff. Software dev is still at the craftsman[0] level. It might move out of that, eventually. But not yet, and probably not in the next 20 years or so. We haven't solved some intrinsic problems around defining a problem completely, precisely and succinctly without having to write code[1]. And getting five engineers to write a single piece of software is exactly as complex as it was when Fred Brooks wrote about it, I think the only improvement we've had since then is Git. [0] craftsperson? that doesn't feel like the right gender-neutral expression. I guess "artisanal" but that looks rude. Suggestions? [1] The "I got ChatGPT to write this application without writing a single line of code" phenomenon is interesting, but it seems like an alternate skill path - you can write code, or you can write prompts. The complexity is the same, and the amount of effort is within an order of magnitude. I'm not sure, though - I haven't managed to get ChatGPT to solve a single technical problem successfully yet.
  • banku_brougham 3y ago
    Chat always writes me something that includes perfecttly_suited_package_that_doesnt_exist(my_inputs)
    • dbetteridge 3y ago
      It's great isn't it. Npm packages that don't exist and yet always have the same parameters as your data.
      • namtab00 3y ago
        eventually it will create and publish it for you, they say... not only the package definition, but the implementation also. I don't know exactly why, but I doubt it will.
    • TeMPOraL 3y ago
      Ironically, that's one of the common practices in programming - writing calls to nonexistent functions that you implement later.
• lysecret 3y ago
  Well written and enjoyable read. The image of a Burgermaster 5000 operated by a Highschool kid producing burgers is one that really gets stuck in your head. I am wondering to which degree something like this might or might not come true with the advent of AI driven coding. I have built a full swift app using Ai (not writing a single line of code) but it took a lot of coding knowledge to pull it off. I wonder if this will change.
  • zerbinxx 3y ago
    What was the full app?
    • lysecret 3y ago
      Calorie counting using ChatGPT API. Sadly, burns too much money to publish it but I have been using it consistently for a few months.
      • zerbinxx 3y ago
        Oh so you say “I had a burger” and it estimates what kind of calories that would entail?
• absoluteunit1 3y ago
  What a fun read; thank you! I’m still very early on in my career in software dev (~3 years in) and you articulated the idea I’ve been grappling with extremely well in this piece. I could never find the word for it but “commodification of technical work” is the perfect way describe it.
• banku_brougham 3y ago
  Consider the Business Intelligence/Analyst roles. The industry is trying to replace people who write SQL with people who can use Tableau or similar. "Just connect to whatever datastore you have and non technical people can drag and drop." Its got some problems: 1. They forget you need to hire many more (lower paid) people, because your output now linearly scales. Human hands have to turn the crank because its all UI-based work. 2. You still end up with very complex and disorganized business logic transform code, and now its buried in the UI. The PMs or business teams are the only ones who know what that logic is. The engineering org delivers high quality, tested datasets that are pretty raw for the purpose of answering business team questions. The hard part was always solving the weird way business outputs are obtained from raw upstream data. Now that solution gets stored in a tableau workbook and cant be used as input for something else. It has to be copy pasted from the UI into a new tableau workbook. Well, now we bought the Tableau cloud service and our BI team can build and maintain SQL extracts in a more rigorous way. Tableau now looks like its trying to take a chunk of Databricks business here, but now its a non-engineering team doing it. Not sure its going to work out.
  • artyom 3y ago
    Respectfully disagree. Meaningful answers from raw data is hard, but the hardest part was always making business people to _ask the right question_. Take as an example a typical business question: "which countries are our users from?" But do they mean the country the user declared in the registration form? Or country they're currently accessing your service from? Or the one from their payment method? Or the one where they where born? Or the one they have citizenship from? Or the one they're currently residing on? Or their shipping address? Or they billing address? Or..? If your dataset is sufficiently big, every one of those countries will output a different answer. I'm sure that in the "global fintech" space those weird cases become the norm. Your typical lower-wage Tableau user will just look for whatever country codes show up and run a count, then declare it the truth. A slightly smarter Tableau user will bribe an engineer to write SQL for them. It'll take someone with knowledge of the dataset and probably the systems where the data is sourced from to push back and force the business to ask the proper question, and provide proper context. Tableau and the like are good to replace "technical work" which is a guy copy/pasting the same SQL query into pgAdmin and emailing the resulting CSV on a daily basis, and then some, but it's not making "less skilled" UI-oriented workers to think better.
• ReptileMan 3y ago
  You can't abstract away risk, you can't abstract away complexity. You can either reduce, increase or shuffle them. Shuffling with layers of abstraction is the manager class preferred approach. Which leads to tech stacks of the type - "only me and god knew in the beginning what was going on, now only god knows"
• rswail 3y ago
  Programming is still a craft, not engineering, or manufacturing. A software house should work like bespoke tailoring, or fine cabinetry, or glass blowing. There's still no better training for programming than the equivalent of master/journeyman/apprentice. Apologies for the gender specific terms, but they are specific to how tradespeople operated from medieval times. The worst thing to ever happen to the practice of business is the invention of the MBA. MBAs are imbued with the misleading axiom that management is a craft and science of its own, independent of the type of process or practice that is being managed. Combined with endless selling of the latest buzzword theories by consultants is why we end up with JIRA-Driven-Development, nonsense like t-shirt sizes, 2 hour wankfests called "Sprint Reviews", let alone all the scrumming and standing-up and backlog-refining and endless make work.
  • plz-no-soy 3y ago
    [flagged]
  • pif 3y ago
    > Apologies for the gender specific terms, It doesn't make any sense to apologize for using a language in its current form.
    • dsr_ 3y ago
      All living languages are in some state of flux. That's how you can tell they are still alive.
      • pif 3y ago
        I've not stated the opposite.
      • plz-no-soy 3y ago
        [flagged]
    • TeMPOraL 3y ago
      It comes out of an abundance of caution, necessary in these uncertain times.
• csydas 3y ago
  This is a real issue I've seen at my past jobs in Support -- in theory, the better the support team got at closing issues without needing RND, the better the case load should have gotten and the better overall time for everyone involved (support, rnd, C-levels, customers). We had many huge enterprise customers using the product (usual big names across the globe), and everything was working pretty damn well and people were happy. Then the acquisition came, hordes of new C-Levels and directors added, and suddenly, the system everyone wanted was no longer enough; we needed more out of the Support/RND team for some reason, more sales, more renewals, more special contracts with ENT clients with exhausting demands from the company. And the expectation was just "be more efficient". All of this came over slow time with small changes (back porting features for ENT clients who refused to upgrade, forbidden solutions for the Support team because "it upset the major clients", allowing sales/renewals to force RND engagement if they demanded it, new CRM because it was too hard to do marketing campaigns in the old one, even though people were calling _us_ and asking to just send them a quote they liked the product so much) I honestly think trying to commodify and extract even more from what was a very successful system financially and just overall ended up ruining pretty much everything. Sales are slumping as are renewals, we're churning RND and Support folk, and because of slump in sales, there's belt-tightening everywhere. We've implemented so many new systems needlessly with huge implementation/consultant fees that absolutely no one knows how to use -- workday is a prime example of this as out of the box it doesn't do _anything_ we needed, but we had to use it anyways, and absolutely no way to pay for the features we really needed. Same with ServiceNow implementation, it was supposed to be a near 0-code experience we were sold, but naturally that was not the case. Why did we get it? No idea except that it came down from on-high from persons that don't use the CRM and now we're stuck with it. For me what it comes down to is too many people having to be like the CEO and Bill from the article -- for some reason such Clevels need to show they're doing "something", but I didn't think that something should include implementing wild changes to workflows in the company the C's know nothing about, or even worse, responding to customer complaints and demanding we "fix the issue" without knowing what the issue is. The conference experience from the article resonates with me heavily, cause we have all these systems that do "everything but nothing", all these workflow changes without listening to the people having to use the work, it's really awful.
• lloydatkinson 3y ago
  You know it’s going to be a good read when they post a new one
• gavinhoward 3y ago
  I'm building a software business. My bet agrees with the author: I am betting that I can spend time and brainpower designing and implementing a good solution to a problem, and people will pay enough to support me and my wife. You can't commoditize design, and that's where my work actually is. This is also why I'm not scared of AI taking my job.
• vsskanth 3y ago
  There's an entire spectrum of options here from complete outsourcing (banks and gov) all the way to writing everything yourself (Google). I've lately found a middle ground that seems to work well - licensing source code or libraries instead of entire closed web solutions. It enables a small team of good engineers to be really productive and ship something to production rather quickly. You pay for maintenance and bugfixes for what you license and can still retain control of your data and interfaces.
  • ludicity 3y ago
    I read something great to this effect recently, but I can't remember where. The gist of it is that adding people increases overhead, so the absolute smartest play is to keep the tiniest team physically possible and give them an absurd amount of leverage - which frequently takes the form of licensing the things that you really -can- commodify well. The fundamentals of data engineering (https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/ https://www.oreilly.com/library/view/fundamentals-of-data/97...) is adamant on this. If you're a new data engineering team, just buy things that download the data you need, because fetching data from APIs is (usually) pretty simple to do with automated tooling. Then your team can focus on the part that we haven't nailed, like making sane models.
• a3c9 3y ago
  The way I think about it is that as programmers we build the process for the kitchen. The software is designed around commodifying the repetitive work for non-programmer domain experts. In my experience this was building a process for data scientists (math PhDs) to train and deploy ML models, and currently chemical engineers to build and deploy process simulations. Data scientists and chemical engineers will have to excuse my comparing their work to flipping burgers :)
• jwie 3y ago
  One key problem is nobody, none of the suits anyway, want to believe that there are essential, hard problems that can't be outsourced, can't be commodified, can't be shortcut in any way. It's the business version of the get-rich-quick scam course hucksters. The truth that there's no silver bullets can't compete.
• wxd2351 3y ago
  In the McDonalds analogy, developers are not the teenagers working at the machines, we're the engineers that designed the machines. In the McDonalds analogy, the computer is the teenager. Programming isn't work, it's meta work, you come up with a list of instructions once, and then the computer does the work 24/7 indefinitely. Meanwhile you go on to write another set of instructions for something else. If you ever write the same set of instructions more than once, you're basically doing it wrong. So it's hard to know how long things will take, because you're always doing something new. You never do something more than once.
  • tonnydourado 3y ago
    > Programming isn't work, it's meta work This is either profoundly insightful of absolute bollocks, but I don't think I'll be able to tell which before the next 5 years.
    • TeMPOraL 3y ago
      Programming is 5% of painstaking work to break down a problem into hyper-detailed description that leaves no space for ambiguity (that's the "meta work"), and 95% of dealing with self-inflicted bullshit like build systems, package managers, platforms, service architectures, devops, devsecops, secdevops, and all the procedural nonsense necessary to give people paying us a modicum of control over delivery.
      • fwungy 3y ago
        It's a series of simple tasks interrupted by two week diversions.
      • SoftTalker 3y ago
        Wow. This may explain why I don’t really like programming anymore. After doing it for years the solutions often seem immediately obvious. It’s the modern programming process (the “self-inflicted bullshit”) that sucks.
  • vegetablepotpie 3y ago
• epups 3y ago
  I don't find this argument very compelling. Technical work has been commodified to a great extent, even in bleeding edge applications. Does anybody have any doubt that OpenAI has timetables with clear action items that need to happen so they can release their next model? The counter-example of a software to abstract SQL queries is weird. This is exactly what we have been doing with other levels of abstraction, happily so. Why write Python instead of just using a compiled language? Because it's easier, and allows you to hire different types of people and focus on higher order problems. Maybe that's offensive if you are a world-class programmer like the author seems to think he is.
  • x86x87 3y ago
    Yes, yes. In case of OpenAI: not only clear action but all the research and the disciveries needed are planned at least a few years ahead. They know exactly when AGI is going to be here!!!
• pbourke 3y ago
  > You can pay people to churn out bad self-help all day, but none of those are going to be worth a damn without allowing the human element to flourish. But you still need a factory which can print the things, and as clinical as a mass bookbinding operations sounds, I really believe that you're only going to get a beautiful binding when that factory is run by people who have the connection to the work necessary to exercise taste. Operation, innovation and maintenance. Pick any 3. Better if done by the same team or teams that are very close to each other. Ideally, all done to varying degrees by each person on the team because that’s where inspiration for improvement comes from. We can split the 3 functions up into 3 groups, with 3 sets of management hierarchy. Often the result is mediocre and the people are unhappy (especially those stuck on the maintenance team - thankless but crucial work, that). We can try to make teams responsible for all 3 functions, but often hire managers who overfit on one of them (coincidentally the one that leads to better rewards next quarter). It’s hard to champion operational excellence, diligent maintenance and hammock time in a single culture. No wonder it’s hard.
• justincredible 3y ago
  [dead]
• zubairq 3y ago
  Having read the phoenix project while working at red hat I can see some value of the process for large organisations with a lot of people and teams that need to be aligned, but for smaller projects with few people then just steer clear of all this software factory stuff
• highfrequency 3y ago
  If you look at the development of any technology there is a consistent evolution from high skilled use of simple tools to low skilled use of complex tools. Consider the evolution of human weapons. It was very easy to make a crude throwing spear (break off a branch and sharpen the wooden tip against a rock) but a ton of strength and precision was necessary to take down a target with it. It took a while to figure out how to make a decent bow, but once invented it spread all over the world because momentum could be imparted to the arrow instantaneously when releasing the string. Strength was still required to hold the bow at full tension; this requirement was eliminated with the crossbow. As a tradeoff, the crossbow required more significant manufacturing expertise and economies of scale, particularly with regard to precise metal locking mechanisms. The evolution continues with guns replacing crossbows, tanks replacing horses, and ending with nuclear weapons. Each technological development requires a much greater level of manufacturing organization, and in turn bestows more power on an unskilled user. The resulting economies of scale are why only a small set of huge states are militarily relevant today, and why the idea of nomad bands posing a threat suddenly seems laughable (whereas it was a dominant worry for civilized society up until a few hundred years ago). It is natural to glorify the work of the artisan whose disorganized but brilliant insight has not yet been commoditized. But for better or worse, progress usually takes the form of painstakingly systematizing those insights, detail by detail, until they are reproducible and robust, so that a broader range of people can consistently churn out similarly good work. It is this systematization and rationalization of the manufacturing process that really brings quality of life improvements to large numbers of people. It is also what can seem to take the magic out of the original insight, eg as Edison’s discovery of electricity becomes as mundane as flipping on a light switch. Progress happens in the transition from magic to technology.
• sublimefire 3y ago
  I am coming from structural engineering and see software more on par with city planning rather than other engineering disciplines. It is just too vast. There is no surprise that after years we start to get many more diverse positions in the field that describe specific types of work, e.g. backend/frontend/firmware/ml engineers, data scientists, security analysts, kernel devs etc. I think it will get to a point where you will need very specific certification to be allowed to work in one of these spots as standards move forward. Again, it might change to something else but the crystallisation effects are visible nonetheless. Thinking about software as a factory is also possible and might be useful. But not everything fits into that analogy, for instance, when you think about integrations in the product or when it is a service rather than an app. Factory implies something is being made but software is not the end result in many cases but is an enabler in itself.
• arthurofbabylon 3y ago
  > My man is out here advocating of a workflow that consists of feeding your subconscious mind research for four hours, then meditating on it for another two, then sleeping and praying that the Gods of Design simply bless you with an answer in the morning Delightful.
• mathattack 3y ago
  I’m sure this fellow is a pleasure to work with. In reality most of the world operates somewhere between “we’re a factory” and “we are unmeasurable Picassos.” It’s not a huge stretch to combine thinking that “software is unpredictable” and “people pay up for predictability” so we should still try to tilt that way.
• bane 3y ago
  I get what this post is saying, and I feel it too, deep in my bones. But technical work, especially in software, is being commoditized. If you view commoditization as a process, the end point of it is "free" (as in beer). I've been in tech long enough to remember when you had to carve literally every line of code out of the firmament of the heavens to get anything done. Today the bulk of technical work is plumbing together literally millions of lines of "free" (as in freedom and beer) code to get some behavior that checks all the requirements boxes. Going from the post some more, modern developers are more akin to line cooks that put ingredients together. There's value to that skill, but it's a reduced value as its easier to learn to plug together APIs than to craft a cache aware data structure and reduce the big-O of some algorithm while fitting it into limited RAM. In most restaurants line cooks are almost a commodity (even the sous and executive chefs are to a point), customers don't know if the cook from yesterday quit in a rage and a new guy got picked up this morning who had also quit in a rage from the place next door. Follow the recipe and you get the same product to the customer. News for the tech folks here, line cooks (even experienced ones) make shockingly less than the average fell off a bus tech-bro with a couple bootcamps and a github repo. The average developer is today thousands of times more productive than the developers from 30 years ago because they don't have to build the ingredients anymore -- the industry is well bootstrapped by now. But if some future Co-Pilot LLM can turn one developer into 4, and effectively "auto-plumb" with minimal human direction then...
  • ludicity 3y ago
    I had a great chat with a reader yesterday, where I became very concerned that we aren't incentivizing young people (like myself!) to pick up the "carve code out of firmament" skills. We just rely on certain personality types getting into it, whereas we fling prestige and money at surgeons. Meanwhile, where I am, they'll only fling money at me if I memorize details about the Snowflake billing model and API - not very useful to society if we need to innovate! You're right though. As someone near the top pointed out - some things actually do get commoditized, like mail merges. That actually works and is probably never going away. Maybe the real issue is that a lot of these products aren't innovative at all, have no shot at being the next mail merge, and our leaders largely don't understand how to tell the difference.
• Nevermark 3y ago
  In software, the vast majority of work has asymptotically been automated. So we forget that it was ever work. Consider the humble file copy. Or “automated scribe” if you will. Copying is automated to the point where the enormous amount of copying our systems do has becomes invisible to us - and also lost as a point of economic differentiation. And it’s meta useful! All our copying programs are themselves easily copied - with the same algorithms! The point is that software writing, like mathematics, will always spend its time futzing around the edge of the known and unknown, because every area that gets fully known/characterized will get automated in a way that both solves that problem for everyone and scorches the economic earth of that activity. Software work will always include an element of quest (research) into the unknown, in search of riches (economic value), in some aspect, whether that new area is glorious or tragically mundane. Whatever area of software work a manager without software expertise can do themselves automagically, is like a fruit tree tamed so that it is easy to pluck fruit from, because it no longer has fruit. The manager would no longer be doing anything differentiated or valuable if their problem statement wasn’t upgraded back into difficult to automate territory requiring pesky creativity and expertise.
• solatic 3y ago
  OP is correct when a business decides against standardization. > For example, one pitch in particular was for a product which promised to remove the need for me to write SQL in exchange for being able to set up all my dependencies from a drag-and-drop editor, with the sales pitch consisting of "You can get rid of thousands of lines of all that SQL you hate!" - no I can't, fucko, because your application is still connecting to Postgres so it's just writing the SQL for me with another layer of licensed abstraction on top of it. Why would I pay to have more abstractions designed for you to sell software to multiple clients, you blue-suited dementor? Eight times out of ten, I want to pay you to remove them from my codebase. See, the problem is that you can't fully remove the original SQL. It's still there, in the code. The database is still there. So instead of having one interface to access the database - SQL - now you have two, the low-level SQL and this bastardized abstraction. Some of your employees will write SQL, and some other of your employees will use the abstraction. They use different tools, so the tools will fall out of sync, and the employees will fall out of sync, and you'll get discord. Pick one tool. As far as it's technically feasible, pick one database, one programming language, one UI framework, one wiki vendor, one CRM, one ops visualization framework that is right for the business. Don't pick up some fad-of-the-day, and don't let your "artisan" engineers do "research" projects to see whether some other framework might suit their needs "better". Tell your engineers, here's the business problem, here's the pre-existing stack, solve the problem with the already-existing and already-supported tools. If the engineers are truly artisans - guess what, an artisan doesn't blame his tools. Pick new tools only slowly, deliberately, when you have no other choice, and only with a solid plan for standardizing the tool for full adoption across the enterprise. Why? Because high-performing organizations ensure that they are always building upon prior work. Five engineers with five years' experience with React will be a stronger team - more productive, faster, more accurate at forecasting, have an easier time reviewing and supporting each others' code - than will John with 5 years React experience, Sally with 5 years Vue experience, Wang with 5 years Svelte experience, Cynthia with 5 years Angular experience, and Ilya with 5 years jQuery experience. If Ilya gets hit by a bus? The other engineers will have difficulty picking up the load. Not much of that React, Vue, Svelte, Angular experience will help with supporting the jQuery parts of the codebase. Commodification requires a standard. We appreciate the value of standards when we succeed at establishing them - tabs vs. spaces, Docker containers, infrastructure providers, project management tooling - because they eliminate discussions that do not revolve around the more fundamental question of how to deliver value to users. Shouldn't you ask yourself, if you're evaluating a new language, tool, or framework, whether you really benefit from breaking the company standard?
• Simon_ORourke 3y ago
  Doing some project lit. review a few years back, I came upon the area of component-based software engineering (CBSE), where the idea was to mirror the same kind of manufacturing approach as in electronics. You'd write a software component to do one thing, and have well defined inputs and outputs, and you could then "simply" compose complex systems by chaining lots of these software components together. Nice idea by maybe software engineering wasn't/isn't that formulaic as more well-established engineering disciplines.
  • gustavus 3y ago
    Congratulations you've reinvented the concept of a library.
    • Simon_ORourke 3y ago
      Funnily enough, that's where that particular write-up ended up in.
• pylua 3y ago
  1) developers are not fungible. 2) things that are created are not the same , otherwise it would just be reused. 3) to understand how to build on the system you must learn the domain it is built on, which is also not fungible knowledge. You are fundamentally building items , not reproducing them in software. A factory reproduces the same items. There is a crisis in industry where legal contracts and business forecasts do not align with what is reasonable and predictable for software development .
• arthurofbabylon 3y ago
  The key decisions of a manager entail when and where to collapse complexity into simplicity (and inversely expand a simple system into something more complex). So you can pick out a foolish manager by their unwillingness to work with complexity in areas core to their business/craft/discipline, and you can find bad ass collaborators by examining what sorts of complexity they're enthusiastic about. (Note: I'm using the term "manager" broadly. It could be an executive, a team lead, an entrepreneur... basically any contributor making choices.) I personally love being around people who are willing to get dirt under their finger nails, smell the soil, ask questions, and read the documentation (ie, willing to step into complex systems). It's inspiring. It also is a good indicator that the environment I'm in is being well-cared for. Think of a human caretaker – those with empathy (a beautiful example of willingness to engage complexity) far surpass others in both ability and impact. Think of a barista observing details like humidity while adjusting their process. And now keep this analysis while moving to higher levels of abstraction. (BTW, this is a phenomenal essay. I love how the author left it to the reader to realize all the lessons contained in the McDonald's anecdote.)
  • ludicity 3y ago
    I was already going to write that this is a beautifully written comment, and then the high praise at the end sealed the deal! Thank you! I must confess that I am still in the reactionary phase of having watched Simple Made Easy two years ago, and am relentlessly trying to remove complexity, but your comment made me think about whether I've gone too far the other way. Complexity is frustrating because it is frequently the result of bad choices (at least where I work), but the ability to handle it probably is a fair marker for excellence. "Make it simpler" isn't an option sometimes, and I still instinctively flinch away from those instances. Oh, and the description of "empathy" as really being the willingness to engage with a kind of complexity is a very beautiful parallel between the worlds of software engineering and being a decent human being. I really love that.
• 082349872349872 3y ago
  Judging both by the office->cubicle->open plan progression, and by https://www.workatastartup.com/jobs/62929 https://www.workatastartup.com/jobs/62929 there's been some degree of commodification. The low end of that salary range is less (inflation adjusted) than offers (that had options on top) which I was getting last century just out of school, and the high end is less than I was making as a "software engineer" with two years experience.
  • ludicity 3y ago
    I can't remember if I saw this in Slack or Peopleware (either way, very evidence-light books), but they made the great point that there's basically no good data on open plans being better than offices. It's just that you can measure the cost of floor space and it makes manager lives easier. Programmers benefit from thinking, managers benefit from being able to ask people for updates on a whim and lowering costs in the short-term, and guess who decides on the seating arrangements? I have some other hot takes on how, despite I believe Peopleware being a Bill Gates favourite and my executives drooling at the thought of being as wealthy as him, they don't actually read or do anything of the things he recommends when it doesn't result in an immediate superficial victory.
• manvillej 3y ago
  I think the problem is that the outcomes of technical work is not equivalent to a product with a more static value. Simply put, software is not the same as a burger. Software development to me is a financial investment strategy. Some risky investments, some safe bonds, some good debt to invest, some bad credit card debt that will be cleared later. To me, these are the same as a risky overhaul of a critical system adding some automated tests, skipping some type annotations, or launching what was supposed to be a demo/MVP to be first to market. a piece of software that took months to build could be worthless tomorrow if a competitor comes out with a better solution. There's a time value and strategy to it. speeding up development with CD lowers the investment cost & time to market of an individual strategy. Automated tests lowers risk. Feature flag & AB testing diversifies the portfolio. One line of code be the most valuable piece of a product, like Doom's Fast Inverse square root. and millions of lines of code could be worthless. the value of a software engineer's work is extremely contextual. Two automation can be structurally the same, developed by the same person, written in the same language, take the same amount of time and resources; and still be COMPLETELY different in value. until IT/software strategies are treated like an investment strategy, companies & leadership will continue to flounder when managing technical teams.
  • JackMorgan 3y ago
    I agree whole-heartedly! I am sometimes called upon to audit codebases for acquisitions. I've spent enough time in the strategic accounting world to realize that we almost have no concrete units of measurement to talk about software. Lines of code is as meaningful as the count of boards in a house. In a house, we can at least get total square feet, number of legally defined bedrooms and bathrooms, lot size, etc. In software, we've got almost nothing. I've seen tiny codebases that can CRUD hundreds of different records, and I've seen multi-million line codebases that can barely handle saving a dozen different records. Consider all the ways data flows in and out of a system: asynchronous processes, batch processing, auditing, third party integrations, permissioning, reporting, and legacy system integrations. Each of these change the value and technical debt of a system. I suspect we will have an increasing need to perform an accurate audit of a codebase. And provide the value, depreciation, and debt of a system in dollars. I would love to see a standardized metric for accounting for assets and liabilities.
• danielovichdk 3y ago
  Reading the comments I haven't found any evidence for anything. And that is the trouble with software in a nutshell. The assumptions are too many and the subjective opinions too deep. I rarely submit to assumptions any longer without the other person submitting at least some evidence for their claims. Experience is important but gathering information on everyone's experience is more important to form some sort of experience based evidence. I know everything everyone writes is well meant but damn I miss someone wrote a book and made a course on how professionalism shoukd ne based on evidence gathered from our fields experience. The amount of managers incapable of commitment towards serving towards that are just what fred brooks said it was - they rarely exist.
  • bradleyjg 3y ago
    I find that some evidence is worse than useless. There’s always some evidence for anything and now you’ve got people with their deep subjective opinions held unshakably because they are SCIENCE.
• rawgabbit 3y ago
  What the author failed to mention is that the software industry itself is also responsible for this state of affairs. I am talking about the consultants and the salesmen of enterprise software. In many cases, they convince CEOs that they simply need to sign a contract and all their problems will be solved. They don't care to explain the high rate of failure of software implementations or how much manpower is needed on an ongoing basis to maintain the shitty software. CEOs think they are buying an appliance. e.g., I buy a coffee maker, I pour water into it, I put in the coffee pod, I press the button, and coffee comes out. In the best scenario (from a CEO perspective), they are getting an assembly line someone else put together for them. e.g., the CEO hires a general contractor who lines up all the subcontractors, installs the different pieces of equipment including the "glue" that connects them, and will maintain equipment going forward. The CEO simply needs to pay for the capital expense and provide the operators for the assembly line. In the typical scenario (bad from a CEO perspective and bad for the software industry), what the CEO gets is a stack of re-purposed software modules that have been pressed together in a slip-shod fashion that works when it wants to and fails without a trace as no logging exists. The CEO has to hire IT developers just to get the thing to run. And he will now look for his next job because the board of directors told him, you failed.
  • ludicity 3y ago
    Thank you for taking the time to write this. "CEOs think they are buying an appliance" sums it all up perfectly. I hold them accountable because they're paid so much that they should know better, but you're right that a lot of people try very hard to mislead them, and sometimes they're even incentivized to be misled. If you build in-house to suit your data models and workflow, there is a chance you won't ever get the thing deployed, then you look bad. You can always sign up for Salesforce, and it might be bad, but it's running so you can hide the details of the badness from the board. This is why when I wrote another post on saving 500K for my company, the two-slide presentation I was asked to write went through a lead, a director, and a lower-level C-suite member before being allowed to reach the top of the pyramid.
• ang_cire 3y ago
  > and that giving someone a salary is enough reason for them to subjugate the entirety of themselves as they turn up to work every day... > Well, you're wrong, and you can fucking bite me. Hear, hear!
• athrowaway3z 3y ago
  (Enterprise) Java grew a lot because the non-technical message managers got was something more like: An engineer can produce objects, and we can swap out objects and their factories. That way we might one day in the future decide we want a faster object or choose a cheaper object. In the decades since, the 'progress' we've made can be summed up as dropping the J in JVM.
• zem 3y ago
  the author does make some good points, but I was a bit taken aback by the whole "who could possibly get excited about logistics of all things?!" bit. makes me wonder what else he simply doesn't "get" because he has written entire areas off as intrinsically boring and lacking in value.
  • ludicity 3y ago
    Not at all - I probably just wrote badly and gave off the wrong tone there. I love that people can find excitement in almost anything. But uh, yeah, the book is very strange. There are a lot of characters that just take all sorts of insane abuse at work, and then with a grim face think: "This company is my family. I will not fail. Parts Unlimited will conquer the opposition." Like, dude, this is pathological. Yes, you can work hard and try to save people's jobs, but where's the part where your friends strongly advise you to simultaneously look for work elsewhere because this place is toxic? The "excited for logistics" thing was just uh... well, the first quote about the "excited" person just reads horribly in the context above, and The Goal wasn't even a fast-paced thriller. I believe the energy is best described as "Hail Corporate!"
• 8crazyideas 3y ago
  This post resonates with me as a radiologist with 40 years experience, and a son who founded and runs his own company centered around machine learning, and now, LLMs. I frequently hear about how "AI" is going to replace radiologists any day now, but I do not believe it, for some of the same reasons described by the author, though in a different context. I recently wrote a post "Will Artificial Intelligence Replace Radiologists": https://anordinarydoctor.substack.com/p/will-artificial-intelligence-replace https://anordinarydoctor.substack.com/p/will-artificial-inte... that explains why I think the answer is "Yes, when it replaces engineers and everybody else".
• __turbobrew__ 3y ago
  The most successful technical organizations (US Navy, HP, Xerox, Bell Labs) let people own their own solutions and work. You give someone a problem with context on why that problem needs to be solved and then you let them own it with near complete autonomy. Not everyone can work in this environment which is why technologists will not be commodified in the near future. Until I can tell an AI that they need to reduce my AWS spend by 20% or they need to add an extra 9 on my service’s reliability there are going to be technologists involved. https://govleaders.org/rickover.htm https://govleaders.org/rickover.htm