11 ms·

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and

by nataz 6y ago

This guy gets it. Development is literally the simplest part of the problem set. The intersection of domain expertise, contracting/procurement creativity, and technical evaluation is the tough part to wrangle. Getting to the point where development can start is 95% of the battle in my experience. Not to trivialize the actual development portion of the project, but most missteps happen early, dooming projects before anyone sits down to code.

belter5 6y ago
There was just an article on here last week about how software development is a learning process and programs are just artifacts of that process. Big, old, messy, and highly regulated domains will obviously be hard to learn.
- ethbr0 6y ago
  Here's an example I learned on Friday: If money is left over, apply to the next available item via (logic). Unless money is sent twice, for two separate sub-items on the same item, in which case if the first amount is insufficient to cover the entirety of the first sub-item, the excess (if any) from the second amount, after being applied to the second sub-item, must be applied to the remainder of the first sub-item. There are legal ramifications of doing the first way, without the second. And only the actual in-the-trenches people were able to tell us about the second. Using outsourced process documentation is @&$&ing useless.
  - Aeolun 6y ago
    To be fair, this sounds like someone having a hard time saying you should apply the money sent to the entire balance.
    - ethbr0 6y ago
      Nope. That's pretty much what the specs (written by outsourced consultants) said. In reality, there's one edge case where things needs to be applied in a different, distinct, and very specific order, otherwise (legal consequences).
      - Aeolun 6y ago
        Yeah, this is probably just my lack of context around the subject. That said, if I get requirements like that I always have a strong feeling the case can be generalized. Laws generally aren’t as specific as that.
        ethbr0 6y ago
        This is around refund payments and interest, in healthcare. So probably an outlier. ;)
- bdavis__ 6y ago
  not to mention the fact that by definition the software developers know very little about the domain (given that 1/2 of them have less than 5 years of experience)
ryanmarsh 6y ago
On engagements like these most billable hours are spent waiting on the client to make trivial decisions or give access to data/systems. Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature. The high bill rate is for your pain and suffering from having to deal with them.
- ethbr0 6y ago
  Not entirely untrue. I've billed 50% of a project's hours as "still waiting for system access." Granted, that was a pathologically bad customer, but it happens. The ironic thing is the question you usually get from them is "Do you have access yet?" It's your system and your company! How do you not know the answer to that question?!
- acdha 6y ago
  > Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature. This has not been my experience, which is really to say that the world is big enough that you can find any combination of blame imaginable somewhere. Once you have misaligned incentives no part of the dynamic can be relied on not to cause problems.
  - j45 6y ago
    When billing hourly it is more profitable (and not always ethical) to hire average people than top performers in every role. This is true, too often, in poorly run and ineffective consulting firms who optimize best practices to mean what is best for them.
    - acdha 6y ago
      It’s not a great look but it’s usually quite literally what the client is paying them for. I view the companies maximizing revenue the same way I’d view a shark: it’s what they are, and you can always choose not to jump into the water.
colechristensen 6y ago
But most of the rest of it in these huge contracts could just not be done. The myth is that all of the rest is necessary when only a tiny fraction really is. I have worked in tiny very early stage startups and old dinosaur companies. What comes across loud and clear is that early stage companies could use a bit more organization and dinosaurs could cut huge portions of what they “do” and then both would be in a similar situation to get things done reasonably well. The bulk of the waste is an attempt to get people who don’t know what they’re doing into a position of getting things done. Sometimes it works.
- ethbr0 6y ago
  I look at red tape / internal rules and policies as proofreading code for genetic replication. On one hand, you limit (beneficial) mutation. On the other hand, you limit (harmful) mutation. An organization usually doesn't scale past a certain size without stabilizing its ability to copy itself. For better and worse.
- mpfundstein 6y ago
  Whats the prob? Its money changing hands. It flows from the Corp down to developers, managers, etc. also it is good that corps and governments try to innovate and spend money on it. Can it be more efficient? Always(!) Can it be more focused? Sure. But a big innovation takes a lot of factors to be well balanced and that alone is often not guaranteed nor possible given the circumstances. You just might not find the kind of devs it really needs (and not even knowing about it). So you hire the best thats available but it won't work... Bad luck. At least you tried. I see it that way. Whether vcs pump millions into 100 startups so that one succeeds or corps pump millions into 100 projects... Who cares? The money is not burned. It hopefully ends in my pocket
  - colechristensen 6y ago
    What's the problem? People spending their lives pushing a boulder up a hill. People paying taxes to pay for people pushing boulders up hills. The problem is wasting human life on wildly inefficient endeavors.
eckza 6y ago
> Development is literally the simplest part of the problem set. This is the single most important and insightful sentence that a lot of HN-ers are going to read today, and they don't even know it.
- whynotminot 6y ago
  This guy governments.
- eaton 6y ago
  I'd never in a million years argue that Deloitte did an awesome job on a large government contract, but it's mind-boggling to watch the regular ritual of HN posters blithely insist that anything more over $20K and a pile of pizzas for government infrastructure is a scam.
  - saagarjha 6y ago
    Try offering it to them to do the job and see how quickly they change their tune.
  - arminiusreturns 6y ago
    Not only do I think thats a strawman (I don't see that base level argument given that often on hn), but even if it wasn't, the truth is there is a lot, and I mean a fuckton, of money waste at the government level on IT projects. Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed, and there are plenty of examples in even recent history that prove this point. More than just waste, the intention to fill good ol boy pockets in priority over getting the job done has significantly weakened the function of the projects, for example, the national security implications of abandoning thinthread that William Binney and Thomas Drake talk about... as just one in a myriad of examples. Virginia didn't have multiple counties become the new primary growth centers of millionares because of fair and balanced bids... and anyone pretending otherwise doesn't know how DC works.
    - matthewdgreen 6y ago
eigenvector 6y ago
Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do? "Because we don't have the expertise to do software development." The comments in this thread, which I fully agree with by the way, just cement my belief that if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house.
- nataz 6y ago
  Ha, I'm currently a federal government program manager, so I am thinking about it from the government's side right now. I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides. Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.
  - rorykoehler 6y ago
    I’ve been working in enterprise/government sector for a few years now and my observation is that they need to focus on better systems design. Everything is too convoluted. Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear is where most of the money is wasted.
    - eecc 6y ago
      Oh, my last disappointment of the week was being lectured via 2 screenfuls of Confluence with a “it’s more complex that what you make it” exposure of 2-3 layers of encrusted legacies. The PO was flexing and glowing how their managed to build into the new product this synthesis of bad designs. Which means entire departments get a list of rotating passwords by email every couple weeks. The network of ossified legacies just grew one more sprout.
    - Aeolun 6y ago
      > Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear “We need a new system, but it has to function exactly like the old one.” Including any bugs and craziness.
fractallyte 6y ago
Wait! If there was a sane set of APIs and data interchange formats that was standard across the whole system, development would be the simplest part. It's because they have to shoehorn their solution into a morass of garbage software, that this turned into such a complex task. The real answer is to dump ALL these huge contractors, and their 'solutions', and make something new and better, with proper engineers directing the process. Imagine if Deloitte had been contracted to 'invent' the Internet...
- NearAP 6y ago
  >>> with engineers directing the process This is usually not the solution. You have to understand the requirements/know your actual users. Engineers don’t necessarily do this well. They build with the latest tech but as you see in this example, the primary users are older people who can’t use latest tech
  - Aeolun 6y ago
    > Engineers As far as I’ve seen there’s people that build first, and ask questions later, and the reverse. Only the latter are engineers.
    - nitrogen 6y ago
      Agreed. This is not a No True Scotsman -- understanding true requirements is a fundamental attribute of engineering. If you aren't doing that, whatever you are doing it isn't engineering.
- flarg 6y ago
  Ha ha I know, but it isn't like that. APIs are always written as REST but the systems serving those APIs are not REST so everyone starts off building lots of GET APIs and then progress steams to a halt because it's tough to write loads of APIs for the many different types of write data flows in most systems.
keanebean86 6y ago
This is depressing but oh so true. I've worked at a few companies and they're all the same. The business team schedules an hour meeting with the project manager once a week for two years to figure out the requirements. I spend a sprint implementing the backend in parallel with a frontend person. We spend another sprint testing and fixing bugs. Then the project sits in UAT for 6 months while the business suggests contradictory things. Finally the project is shelved indefinitely due to budget issues. Edit: fixed typo
- 2sk21 6y ago
  This sketch comes to mind: https://youtu.be/BKorP55Aqvg https://youtu.be/BKorP55Aqvg
manquer 6y ago
Yet that is the part that is broken now. Sure all the requirement gathering and other costlier project parts can doom the success of the project because you built the wrong thing. Here however the tech debt and bugs is why the project is abandoned
grey-area 6y ago
The project management part, which I agree is the hard bit, could be made tractable too. Other orgs manage this all the time and build sophisticated software which continues to improve. This is a booking management system for appointments, the initial problem statement is trivial, the difficulties lie in interfacing with other systems, simplifying UI, getting data in and out, providing some customisation but not too much, etc. These are all things you don't know how to do till you start using it. The process you describe of getting 'domain experts' in a room to decide on a spec without contact with the real users then throwing it over the wall to build is fundamentally broken. You won't get to even 50% done without real users - that's the problem, and it sounds like that's what happened here. You simply can't build a system like this without frequent contact with real users (old people booking appointments, volunteers entering data in a hurry, pharmacies who need to add extra info for patients), experts on the other hand will often add negative value unless they are trying to use it. If there was a will within the org commissioning the work to appoint one project lead and then get out of the way apart from demanding weekly builds from week 1 onward this would be tractable. Collect feedback from real users and deliver every week. This is not rocket science and a small team could and should do it if allowed the space to do so.
ddingus 6y ago
A good team, having done this just once, assisted by domain experts, would bulld a code and knowledge base able to get the jump on future problems. Wash, rinse repeat and it won't take long for things to run very nicely. Bonus, code can be published, standards made, all that and bot rented by the year.
Joeri 6y ago
What they need to do is flip the script. Instead of only looking at the input side of these systems and how to plug into older systems not designed for it, look at the output side of everything that’s built and make sure it is api first so future projects can integrate much easier. At the city of antwerp we flipped over in 2015 from a typical assortment of poorly integrated consultant-built monoliths to a microservices api-first approach. It became a hard requirement that all functionality provided by new systems had to be offered by a documented api, and most of the newly built systems dogfood their own api’s from the ui. In 5 years this caused a fundamental shift, where it gave us the ability to set up a covid contact tracing app for schools in weeks, because it was mostly the piecing together of well documented and battle tested api’s.
- overkalix 6y ago
  Could you elaborate on this experience? How did the city deal with institutional reluctance/resistance?
estaseuropano 6y ago
Or to mention they have to price in the tender development and the cost of all the failed tenders that they made for similar projects.
znpy 6y ago
Indeed, that guy really gets it. One of my previous employers had a very good software and it sold such software to various other companies around the world. Such software was highly complex and really the top money could buy in the field. A yearly seat for a single user could cost up to around 5-7 k$. Yet for years the company made most of its money from "installation projects" and "professional services", that is working with the client to understand its needs and implement customisation to the software to make it work well in the customer environment.
xondono 6y ago
What you mean is that 95% of the time and money spent is about different bureaucrats fighting over turf wars and trying to get some credit in ,to be able to claim responsibility if it works, but at the same time not exposing themselves too much, just in case it blows badly.
deleted 6y ago
[deleted]

by nataz 6y ago

belter5 6y ago

There was just an article on here last week about how software development is a learning process and programs are just artifacts of that process. Big, old, messy, and highly regulated domains will obviously be hard to learn.

ethbr0 6y ago
Here's an example I learned on Friday: If money is left over, apply to the next available item via (logic). Unless money is sent twice, for two separate sub-items on the same item, in which case if the first amount is insufficient to cover the entirety of the first sub-item, the excess (if any) from the second amount, after being applied to the second sub-item, must be applied to the remainder of the first sub-item. There are legal ramifications of doing the first way, without the second. And only the actual in-the-trenches people were able to tell us about the second. Using outsourced process documentation is @&$&ing useless.
- Aeolun 6y ago
  To be fair, this sounds like someone having a hard time saying you should apply the money sent to the entire balance.
  - ethbr0 6y ago
    Nope. That's pretty much what the specs (written by outsourced consultants) said. In reality, there's one edge case where things needs to be applied in a different, distinct, and very specific order, otherwise (legal consequences).
    - Aeolun 6y ago
      Yeah, this is probably just my lack of context around the subject. That said, if I get requirements like that I always have a strong feeling the case can be generalized. Laws generally aren’t as specific as that.
      - ethbr0 6y ago
        This is around refund payments and interest, in healthcare. So probably an outlier. ;)
bdavis__ 6y ago
not to mention the fact that by definition the software developers know very little about the domain (given that 1/2 of them have less than 5 years of experience)

ryanmarsh 6y ago

On engagements like these most billable hours are spent waiting on the client to make trivial decisions or give access to data/systems. Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature. The high bill rate is for your pain and suffering from having to deal with them.

ethbr0 6y ago
Not entirely untrue. I've billed 50% of a project's hours as "still waiting for system access." Granted, that was a pathologically bad customer, but it happens. The ironic thing is the question you usually get from them is "Do you have access yet?" It's your system and your company! How do you not know the answer to that question?!
acdha 6y ago
> Deloitte doesn’t usually hire dummies. IME it’s always the client being a total swamp creature. This has not been my experience, which is really to say that the world is big enough that you can find any combination of blame imaginable somewhere. Once you have misaligned incentives no part of the dynamic can be relied on not to cause problems.
- j45 6y ago
  When billing hourly it is more profitable (and not always ethical) to hire average people than top performers in every role. This is true, too often, in poorly run and ineffective consulting firms who optimize best practices to mean what is best for them.
  - acdha 6y ago
    It’s not a great look but it’s usually quite literally what the client is paying them for. I view the companies maximizing revenue the same way I’d view a shark: it’s what they are, and you can always choose not to jump into the water.

colechristensen 6y ago

But most of the rest of it in these huge contracts could just not be done. The myth is that all of the rest is necessary when only a tiny fraction really is. I have worked in tiny very early stage startups and old dinosaur companies. What comes across loud and clear is that early stage companies could use a bit more organization and dinosaurs could cut huge portions of what they “do” and then both would be in a similar situation to get things done reasonably well. The bulk of the waste is an attempt to get people who don’t know what they’re doing into a position of getting things done. Sometimes it works.

ethbr0 6y ago
I look at red tape / internal rules and policies as proofreading code for genetic replication. On one hand, you limit (beneficial) mutation. On the other hand, you limit (harmful) mutation. An organization usually doesn't scale past a certain size without stabilizing its ability to copy itself. For better and worse.
mpfundstein 6y ago
Whats the prob? Its money changing hands. It flows from the Corp down to developers, managers, etc. also it is good that corps and governments try to innovate and spend money on it. Can it be more efficient? Always(!) Can it be more focused? Sure. But a big innovation takes a lot of factors to be well balanced and that alone is often not guaranteed nor possible given the circumstances. You just might not find the kind of devs it really needs (and not even knowing about it). So you hire the best thats available but it won't work... Bad luck. At least you tried. I see it that way. Whether vcs pump millions into 100 startups so that one succeeds or corps pump millions into 100 projects... Who cares? The money is not burned. It hopefully ends in my pocket
- colechristensen 6y ago
  What's the problem? People spending their lives pushing a boulder up a hill. People paying taxes to pay for people pushing boulders up hills. The problem is wasting human life on wildly inefficient endeavors.

eckza 6y ago

> Development is literally the simplest part of the problem set. This is the single most important and insightful sentence that a lot of HN-ers are going to read today, and they don't even know it.

whynotminot 6y ago
This guy governments.
eaton 6y ago
I'd never in a million years argue that Deloitte did an awesome job on a large government contract, but it's mind-boggling to watch the regular ritual of HN posters blithely insist that anything more over $20K and a pile of pizzas for government infrastructure is a scam.
- saagarjha 6y ago
  Try offering it to them to do the job and see how quickly they change their tune.
- arminiusreturns 6y ago
  Not only do I think thats a strawman (I don't see that base level argument given that often on hn), but even if it wasn't, the truth is there is a lot, and I mean a fuckton, of money waste at the government level on IT projects. Trying to act like just because they aren't all a scam doesn't mean there isn't a systemic problem of waste that still needs to be addressed, and there are plenty of examples in even recent history that prove this point. More than just waste, the intention to fill good ol boy pockets in priority over getting the job done has significantly weakened the function of the projects, for example, the national security implications of abandoning thinthread that William Binney and Thomas Drake talk about... as just one in a myriad of examples. Virginia didn't have multiple counties become the new primary growth centers of millionares because of fair and balanced bids... and anyone pretending otherwise doesn't know how DC works.
  - matthewdgreen 6y ago

eigenvector 6y ago

Now think of the problem in reverse, from the government's side. Why are they contracting it out when all of the most difficult aspects involve integrating with government systems, policies and personnel - things that government already knows how to do? "Because we don't have the expertise to do software development." The comments in this thread, which I fully agree with by the way, just cement my belief that if government wants to deliver high-quality IT projects they need to start developing the expertise to do it in-house.

nataz 6y ago
Ha, I'm currently a federal government program manager, so I am thinking about it from the government's side right now. I've also been a consultant cog in a mega-corp and run my own small consulting business, so I've seen it from both sides. Tech folks underestimate non tech challenges (basically anything outside of their domain), and clients don't actually know what they want. It's an age old problem in lots of industries.
- rorykoehler 6y ago
  I’ve been working in enterprise/government sector for a few years now and my observation is that they need to focus on better systems design. Everything is too convoluted. Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear is where most of the money is wasted.
  - eecc 6y ago
    Oh, my last disappointment of the week was being lectured via 2 screenfuls of Confluence with a “it’s more complex that what you make it” exposure of 2-3 layers of encrusted legacies. The PO was flexing and glowing how their managed to build into the new product this synthesis of bad designs. Which means entire departments get a list of rotating passwords by email every couple weeks. The network of ossified legacies just grew one more sprout.
  - Aeolun 6y ago
    > Trying to make the software fit the poorly thought out and implemented legacy ERP design from yesteryear “We need a new system, but it has to function exactly like the old one.” Including any bugs and craziness.

fractallyte 6y ago

Wait! If there was a sane set of APIs and data interchange formats that was standard across the whole system, development would be the simplest part. It's because they have to shoehorn their solution into a morass of garbage software, that this turned into such a complex task. The real answer is to dump ALL these huge contractors, and their 'solutions', and make something new and better, with proper engineers directing the process. Imagine if Deloitte had been contracted to 'invent' the Internet...

NearAP 6y ago
>>> with engineers directing the process This is usually not the solution. You have to understand the requirements/know your actual users. Engineers don’t necessarily do this well. They build with the latest tech but as you see in this example, the primary users are older people who can’t use latest tech
- Aeolun 6y ago
  > Engineers As far as I’ve seen there’s people that build first, and ask questions later, and the reverse. Only the latter are engineers.
  - nitrogen 6y ago
    Agreed. This is not a No True Scotsman -- understanding true requirements is a fundamental attribute of engineering. If you aren't doing that, whatever you are doing it isn't engineering.
flarg 6y ago
Ha ha I know, but it isn't like that. APIs are always written as REST but the systems serving those APIs are not REST so everyone starts off building lots of GET APIs and then progress steams to a halt because it's tough to write loads of APIs for the many different types of write data flows in most systems.

keanebean86 6y ago

This is depressing but oh so true. I've worked at a few companies and they're all the same. The business team schedules an hour meeting with the project manager once a week for two years to figure out the requirements. I spend a sprint implementing the backend in parallel with a frontend person. We spend another sprint testing and fixing bugs. Then the project sits in UAT for 6 months while the business suggests contradictory things. Finally the project is shelved indefinitely due to budget issues. Edit: fixed typo

2sk21 6y ago
This sketch comes to mind: https://youtu.be/BKorP55Aqvg https://youtu.be/BKorP55Aqvg

manquer 6y ago

Yet that is the part that is broken now. Sure all the requirement gathering and other costlier project parts can doom the success of the project because you built the wrong thing. Here however the tech debt and bugs is why the project is abandoned

grey-area 6y ago

The project management part, which I agree is the hard bit, could be made tractable too. Other orgs manage this all the time and build sophisticated software which continues to improve. This is a booking management system for appointments, the initial problem statement is trivial, the difficulties lie in interfacing with other systems, simplifying UI, getting data in and out, providing some customisation but not too much, etc. These are all things you don't know how to do till you start using it. The process you describe of getting 'domain experts' in a room to decide on a spec without contact with the real users then throwing it over the wall to build is fundamentally broken. You won't get to even 50% done without real users - that's the problem, and it sounds like that's what happened here. You simply can't build a system like this without frequent contact with real users (old people booking appointments, volunteers entering data in a hurry, pharmacies who need to add extra info for patients), experts on the other hand will often add negative value unless they are trying to use it. If there was a will within the org commissioning the work to appoint one project lead and then get out of the way apart from demanding weekly builds from week 1 onward this would be tractable. Collect feedback from real users and deliver every week. This is not rocket science and a small team could and should do it if allowed the space to do so.

ddingus 6y ago

A good team, having done this just once, assisted by domain experts, would bulld a code and knowledge base able to get the jump on future problems. Wash, rinse repeat and it won't take long for things to run very nicely. Bonus, code can be published, standards made, all that and bot rented by the year.

Joeri 6y ago

What they need to do is flip the script. Instead of only looking at the input side of these systems and how to plug into older systems not designed for it, look at the output side of everything that’s built and make sure it is api first so future projects can integrate much easier. At the city of antwerp we flipped over in 2015 from a typical assortment of poorly integrated consultant-built monoliths to a microservices api-first approach. It became a hard requirement that all functionality provided by new systems had to be offered by a documented api, and most of the newly built systems dogfood their own api’s from the ui. In 5 years this caused a fundamental shift, where it gave us the ability to set up a covid contact tracing app for schools in weeks, because it was mostly the piecing together of well documented and battle tested api’s.

overkalix 6y ago
Could you elaborate on this experience? How did the city deal with institutional reluctance/resistance?

estaseuropano 6y ago

Or to mention they have to price in the tender development and the cost of all the failed tenders that they made for similar projects.

znpy 6y ago

Indeed, that guy really gets it. One of my previous employers had a very good software and it sold such software to various other companies around the world. Such software was highly complex and really the top money could buy in the field. A yearly seat for a single user could cost up to around 5-7 k$. Yet for years the company made most of its money from "installation projects" and "professional services", that is working with the client to understand its needs and implement customisation to the software to make it work well in the customer environment.

xondono 6y ago

What you mean is that 95% of the time and money spent is about different bureaucrats fighting over turf wars and trying to get some credit in ,to be able to claim responsibility if it works, but at the same time not exposing themselves too much, just in case it blows badly.

deleted 6y ago

[deleted]