21 ms·

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees wi

by tw04 6y ago

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.

blunte 6y ago
Yes. And after the boat is righted, fire every exec from IT Director upward through CEO. That last part may not be necessary though, as customers will jump ship and the company will probably sink anyway.
- Nextgrid 6y ago
  > customers will jump ship and the company will probably sink anyway. Depends, they might have a captive market or a customer base that's non technical and doesn't understand how bad this is. If their systems are like this, chances are their offering isn't anything groundbreaking and the competition already provides a better service for possibly even cheaper, so the customers who are willing to jump ship would've done so long ago and the fact they're still in business suggests they have a complacent customer base that's likely to stick to them even despite this incident. The same reason this technical debt was left unchecked for ages applies to customers. The task of migrating to another provider will most likely rot forever in a Jira board somewhere and they will keep paying their bill in the meantime.
viraptor 6y ago
There's the possibility that those employees are happily retired and don't care. Or they're just not available in many ways. I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.
- notyourday 6y ago
  I bet if the hospital is to pay Bob/Charlie $1M for 6 months, post tax to fix it they would gladly do it. Note: this does not mean "We will pay you up to a million dollars, Net 90, after an invoice is approved and signed off by fifteen people". This means "We will wire a deposit of $250,000 today. The rest will be wired every month in 24 hours after an invoice is issued" The reality is company execs like to talk about fixing problems with money but rarely are they actually willing to fix a problem with money. In my career I dealt with the exact three companies where execs decided to throw money at the problem bypassing all the fiefdoms and bureaucracies created to avoid rapidly solving problems using money in favor of slow moving, don't step on Jackson the IT manager's toes, approaches. All three solved their problems, destroying the local fiefdoms in a process.
  - viraptor 6y ago
    For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want. Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day. In context of this case - if they don't have money to fix their backups (https://news.ycombinator.com/item?id=26539988 https://news.ycombinator.com/item?id=26539988) they likely couldn't drop lots of cash without a serious credit line.
    - notyourday 6y ago
      That's called priorities. I'm sure as every public hospital it pays a lot money in salaries to those protecting and growing their personal fiefdoms within the hospital.
    - tw04 6y ago
      >For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want. A new system isn't going to do them much good when they don't even know who their customers are. I doubt any customer is going to hang around for a year while you rebuild your system from scratch, ignoring the fact that if they're an MSP the customers may not even have their own copy of the data. All of that is ignoring the likely contractual obligations they're likely under to maintain the data on their systems. >Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day. Hospital? He said they're an MSP on reddit: >Yeah we're an awful MSP. I've been trying to find a new job for quite some time. https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqljsc?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
drclau 6y ago
Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.
- intrasight 6y ago
  That is, of course, the solution everywhere that companies complain about not being able to hire skilled employees. Fund it and they will come.
  - smnrchrds 6y ago
    This kinda sounds like saying "why are poor countries poor? Can't they just pay everyone more and then they won't be poor?" I get what you are saying but I don't think this will work if the problem is country-wide. You can't just magically increase everyone's wages in a country.
    - intrasight 6y ago
      1. The problem isn't country wide. 2. Don't have to increase everyone's - just those with a "shortage". Companies want a market economy - except when it comes to salaries.
      - smnrchrds 6y ago
        > 1. The problem isn't country wide Sorry, I though I was on the Romania subthread.
- filleduchaos 6y ago
  > The solution to their problem is really simple: make better offers. That's certainly a simple solution. The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
ghaff 6y ago
Even if you open a checkbook wide, there's no guarantee that someone remembers the specifics of some kludgy system they worked on years ago. And if things are really snafu-d, there's no guarantee that even someone who does have domain knowledge can recover things.
- WJW 6y ago
  And, of course, there's the scenario where the people with domain knowledge are already dead. Money won't help in that case.
omgJustTest 6y ago
If business not make money, close business. Business make money, big boat of money. Engineers not make money.
joezydeco 6y ago
then it shouldn't be a tough decision to make You'll continually be amazed at the human tendency to double down instead of admitting a mistake.
- m463 6y ago
  They could also hire expensive consultants (who will just call up the ex-employee, and listen. maybe to laughter.)
geoduck14 6y ago
Let's not forget about the Mythical Man Month. Sure, specific people amd money can fix the problem, but we can't just assume that throwing people and money can fix this. I would anticipate a mix of: the people who know where the right bodies are buried, the right incentive (yes, that includes money), but also people to look over the entire program, a designer to make sure it is designed correctly, an organized person to make sure it is properly documented, and someone to make sure it is adhered to in the future.
JKCalhoun 6y ago
Is there another industry where a few employees with "domain knowledge" can sink a company? Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation? Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
- ISL 6y ago
  > Or rather, how does one avoid getting into this situation. Documentation?
  - rantwasp 6y ago
    nah. documentation only goes so far. you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.
    - foobiekr 6y ago
      This. I actually think just documentation is, well not completely worthless but close to. Imagine telling someone they could learn to weld airframes by reading. Just no.
      - twobitshifter 6y ago
        For software, it’s the maintenance task that’s helped by documentation not building. They still need to know how to build but it’s a lot easier with documentation. I know that when we use software packages we all rely on the documentation to understand how to use it and code our desired behaviors. It’s obvious that our own code can benefit from documentation in the same way.
        6y ago
Accujack 6y ago
It's slightly more complicated than that, depending on how much time has passed since the creators of the system left, but you're correct in general. Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.
- criddell 6y ago
  The person who started the thread added this comment: > The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..). Sounds like finding their old employees wouldn't help. They need to find old employees of the company that wrote the system and hope they kept a copy of the source code.
  - yellowapple 6y ago
    > and hope they kept a copy of the source code Which from a legal perspective might be fraught with peril, at least in theory. In practice, if the company folded then it'd depend entirely on the litigiousness of the current owner of the copyright, which could range from "nobody" to "some holding company consisting almost entirely of lawyers chomping at the bit to eat you for dinner".
- at_a_remove 6y ago
  Yes. I had an ancient system where the holders of domain knowledge kept either dying or retiring, one by one. I was frantic, but nobody listened to me. The system began to fail on the day I announced my departure. I would later find that one of the previous architects had reached out to my manager to offer assistance, but this was never made known to us. It wouldn't have even cost anything. Anything but pride, I guess.
- hinkley 6y ago
throwthescene 6y ago
Throwaway purely so that I can comment. I've had this happen before. Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda. Fast forward 6 months.. a contractual obligation existed to release the software. They still hadn't successfully built (yup autoconf, make and friends), let alone know where to build it. I got the call you always dream about, 30 days prior to the release being due. My boss asked me to do him a favour - I reminded him that I quit due to a lack of promotion and raises. He asked what I want, and I asked for 125K, which clearly he scoffed at. Then I reminded him that it was a 10M deal they'd signed, because we had an all hands. The CEO got involved and screamed at me for 'torpedoing the company'. I walked out and calmly informed them both along the way the price was now 250K, non negotiable. I ignored every single mail (and legal letter) sent me way. With 10 days left, they agreed. I had the company deposit a cheque in escrow with lawyers of my choosing. We signed contracts, and I did my thing. Four days of work. It was well worth it. Dougie, if you're reading this I hope your cringing.
- tofuahdude 6y ago
  I love this with all my heart. You did exactly the right thing at every step.
  - ycombinete 6y ago
    Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.
    - Judgmentality 6y ago
      > Funnily enough it made me sick to my stomach. That an overworked employee who had already left the company would stick up for himself? Or that a company would go to such efforts to try to manipulate a former employee? I realize my tone doesn't hide my feelings, but even if we disagree I'd still be curious to hear more.
temp8964 6y ago
What if they lost contact information of their former employees?

by tw04 6y ago

blunte 6y ago

Yes. And after the boat is righted, fire every exec from IT Director upward through CEO. That last part may not be necessary though, as customers will jump ship and the company will probably sink anyway.

Nextgrid 6y ago
> customers will jump ship and the company will probably sink anyway. Depends, they might have a captive market or a customer base that's non technical and doesn't understand how bad this is. If their systems are like this, chances are their offering isn't anything groundbreaking and the competition already provides a better service for possibly even cheaper, so the customers who are willing to jump ship would've done so long ago and the fact they're still in business suggests they have a complacent customer base that's likely to stick to them even despite this incident. The same reason this technical debt was left unchecked for ages applies to customers. The task of migrating to another provider will most likely rot forever in a Jira board somewhere and they will keep paying their bill in the meantime.

viraptor 6y ago

There's the possibility that those employees are happily retired and don't care. Or they're just not available in many ways. I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.

notyourday 6y ago
I bet if the hospital is to pay Bob/Charlie $1M for 6 months, post tax to fix it they would gladly do it. Note: this does not mean "We will pay you up to a million dollars, Net 90, after an invoice is approved and signed off by fifteen people". This means "We will wire a deposit of $250,000 today. The rest will be wired every month in 24 hours after an invoice is issued" The reality is company execs like to talk about fixing problems with money but rarely are they actually willing to fix a problem with money. In my career I dealt with the exact three companies where execs decided to throw money at the problem bypassing all the fiefdoms and bureaucracies created to avoid rapidly solving problems using money in favor of slow moving, don't step on Jackson the IT manager's toes, approaches. All three solved their problems, destroying the local fiefdoms in a process.
- viraptor 6y ago
  For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want. Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day. In context of this case - if they don't have money to fix their backups (https://news.ycombinator.com/item?id=26539988 https://news.ycombinator.com/item?id=26539988) they likely couldn't drop lots of cash without a serious credit line.
  - notyourday 6y ago
    That's called priorities. I'm sure as every public hospital it pays a lot money in salaries to those protecting and growing their personal fiefdoms within the hospital.
  - tw04 6y ago
    >For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want. A new system isn't going to do them much good when they don't even know who their customers are. I doubt any customer is going to hang around for a year while you rebuild your system from scratch, ignoring the fact that if they're an MSP the customers may not even have their own copy of the data. All of that is ignoring the likely contractual obligations they're likely under to maintain the data on their systems. >Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day. Hospital? He said they're an MSP on reddit: >Yeah we're an awful MSP. I've been trying to find a new job for quite some time. https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolute_worst_case_scenario_happened_what/grqljsc?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

drclau 6y ago

Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.

intrasight 6y ago
That is, of course, the solution everywhere that companies complain about not being able to hire skilled employees. Fund it and they will come.
- smnrchrds 6y ago
  This kinda sounds like saying "why are poor countries poor? Can't they just pay everyone more and then they won't be poor?" I get what you are saying but I don't think this will work if the problem is country-wide. You can't just magically increase everyone's wages in a country.
  - intrasight 6y ago
    1. The problem isn't country wide. 2. Don't have to increase everyone's - just those with a "shortage". Companies want a market economy - except when it comes to salaries.
    - smnrchrds 6y ago
      > 1. The problem isn't country wide Sorry, I though I was on the Romania subthread.
filleduchaos 6y ago
> The solution to their problem is really simple: make better offers. That's certainly a simple solution. The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.

ghaff 6y ago

Even if you open a checkbook wide, there's no guarantee that someone remembers the specifics of some kludgy system they worked on years ago. And if things are really snafu-d, there's no guarantee that even someone who does have domain knowledge can recover things.

WJW 6y ago
And, of course, there's the scenario where the people with domain knowledge are already dead. Money won't help in that case.

omgJustTest 6y ago

If business not make money, close business. Business make money, big boat of money. Engineers not make money.

joezydeco 6y ago

then it shouldn't be a tough decision to make You'll continually be amazed at the human tendency to double down instead of admitting a mistake.

m463 6y ago
They could also hire expensive consultants (who will just call up the ex-employee, and listen. maybe to laughter.)

geoduck14 6y ago

Let's not forget about the Mythical Man Month. Sure, specific people amd money can fix the problem, but we can't just assume that throwing people and money can fix this. I would anticipate a mix of: the people who know where the right bodies are buried, the right incentive (yes, that includes money), but also people to look over the entire program, a designer to make sure it is designed correctly, an organized person to make sure it is properly documented, and someone to make sure it is adhered to in the future.

JKCalhoun 6y ago

Is there another industry where a few employees with "domain knowledge" can sink a company? Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation? Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.

ISL 6y ago
> Or rather, how does one avoid getting into this situation. Documentation?
- rantwasp 6y ago
  nah. documentation only goes so far. you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.
  - foobiekr 6y ago
    This. I actually think just documentation is, well not completely worthless but close to. Imagine telling someone they could learn to weld airframes by reading. Just no.
    - twobitshifter 6y ago
      For software, it’s the maintenance task that’s helped by documentation not building. They still need to know how to build but it’s a lot easier with documentation. I know that when we use software packages we all rely on the documentation to understand how to use it and code our desired behaviors. It’s obvious that our own code can benefit from documentation in the same way.
      - 6y ago

Accujack 6y ago

It's slightly more complicated than that, depending on how much time has passed since the creators of the system left, but you're correct in general. Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.

criddell 6y ago
The person who started the thread added this comment: > The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..). Sounds like finding their old employees wouldn't help. They need to find old employees of the company that wrote the system and hope they kept a copy of the source code.
- yellowapple 6y ago
  > and hope they kept a copy of the source code Which from a legal perspective might be fraught with peril, at least in theory. In practice, if the company folded then it'd depend entirely on the litigiousness of the current owner of the copyright, which could range from "nobody" to "some holding company consisting almost entirely of lawyers chomping at the bit to eat you for dinner".
at_a_remove 6y ago
Yes. I had an ancient system where the holders of domain knowledge kept either dying or retiring, one by one. I was frantic, but nobody listened to me. The system began to fail on the day I announced my departure. I would later find that one of the previous architects had reached out to my manager to offer assistance, but this was never made known to us. It wouldn't have even cost anything. Anything but pride, I guess.
hinkley 6y ago

throwthescene 6y ago

Throwaway purely so that I can comment. I've had this happen before. Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda. Fast forward 6 months.. a contractual obligation existed to release the software. They still hadn't successfully built (yup autoconf, make and friends), let alone know where to build it. I got the call you always dream about, 30 days prior to the release being due. My boss asked me to do him a favour - I reminded him that I quit due to a lack of promotion and raises. He asked what I want, and I asked for 125K, which clearly he scoffed at. Then I reminded him that it was a 10M deal they'd signed, because we had an all hands. The CEO got involved and screamed at me for 'torpedoing the company'. I walked out and calmly informed them both along the way the price was now 250K, non negotiable. I ignored every single mail (and legal letter) sent me way. With 10 days left, they agreed. I had the company deposit a cheque in escrow with lawyers of my choosing. We signed contracts, and I did my thing. Four days of work. It was well worth it. Dougie, if you're reading this I hope your cringing.

tofuahdude 6y ago
I love this with all my heart. You did exactly the right thing at every step.
- ycombinete 6y ago
  Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.
  - Judgmentality 6y ago
    > Funnily enough it made me sick to my stomach. That an overworked employee who had already left the company would stick up for himself? Or that a company would go to such efforts to try to manipulate a former employee? I realize my tone doesn't hide my feelings, but even if we disagree I'd still be curious to hear more.

temp8964 6y ago

What if they lost contact information of their former employees?