3 ms·

This kind of story always makes me think I'll never really be out of work. I like to think that given this level of funding, I could do a much better job in a s

by japhyr 12y ago

This kind of story always makes me think I'll never really be out of work. I like to think that given this level of funding, I could do a much better job in a situation like this than the firm that was hired did. I've been programming recreationally since I was a kid in the '70s, and I've started to do some professional work in the last few years. I started out in BASIC, and then became familiar with C, Fortran, and Pascal in college. I've since learned Java, and now my language of choice is Python. I have a basic competence in SQL, git, and any number of other tools we use in modern development approaches. I know work like this isn't easy. I know the data is stored in esoteric structures on long-outdated hardware, running on OSes I've never touched. I know there are issues with security clearances and background checks. But I have to think - give me $28m, let me work on this full time for a couple years, and I'll have some meaningful work done and some usable end products. I'd give myself a modest salary, and hire people with way more experience than myself in the areas needed. I can even see a path into this line of work - complete a series of successively more complex and sensitive data-migration projects. I like to think that doing that work wouldn't turn me into someone running a firm like the one hired in this situation. For people who've worked on projects like this that turned out well, is this something any of us with reasonable competence could get into, if we build an appropriate portfolio? Also, I live in Alaska so I find this story particularly frustrating. Not just because of the wasted money, but also because I want our law enforcement personnel to have a well-functioning system at their disposal.

zaroth 12y ago
If you ever had the opportunity to look under the covers on a project like this, you would find that all the complexity is manufactured. It's not even the 80/20 rule. It's more like 99/1 rule. As in, I see this requirements list, and I have two options for you. A) I can deliver 99% of what you want for 1% of your stated budget, or B) I can take 1000% of your budget and still fail to deliver 100%.
- omouse 12y ago
  The sooner you abandon any software design and engineering principles such as estimation and planning you will find that your project will start seeing more bugs and will be developed far more slowly. High quality code = rapid development.
  - zaroth 12y ago
    Either you have an incredible sense of irony, or there's a bug in your comment.
x0x0 12y ago
But the trick is, you probably couldn't do a better job. There is little technical challenge in a product like this. Building crud apps that scale to low tens of thousands of users is not that challenging. Since (I believe) we all share that belief, then why do projects like this keep failing? I've worked on large medical information systems. The problems those encounter are (1) you essentially never get to bulk replace entire systems; you will have to support ongoing data ingestion and export to other running systems, often systems not designed to support this or with poor apis; (1a) if / when you fuck up, critical processes are disrupted. You have to engineer your systems and create backup processes to make this very unlikely to happen. There is a case I can't really share too many details of where a database was accidentally dropped in a datacenter of a hospital chain you've heard of. Said hospital chain was unable to access medical records for 20+ hours; this was in the 90s running on giant hpux machines and it took a long time to reimport data off tape. My employer was on the hook for $30M iirc, and I wouldn't be surprised if this actually contributed to serious patient injuries or deaths. Now obviously law enforcement isn't quite as critical as medical records, but it's not far behind. (2) over the life of such systems, lots of workflows / processes have developed around using them inside hospitals or large doctors offices. You have to discover these processes, mostly stored in the heads of line level workers and their immediate supervisors, and figure out how to enable them in your shiny new system. To be concrete: the medical record system didn't support some requirement. That didn't stop users from needing it. So employees or doctors decided to use a field in a certain way, and they know how to interpret the presence and/or data in that field. You don't even know a field is being repurposed, let alone how they actually use it. This will be a large problem importing all the old data and in recreating the necessary functionality. (3) no one person, and probably no small set of people, know or understand all the ways the system is used. Somehow, you have to capture that knowledge and use it to create a functioning new system. At least the medical record company I worked for hired lots of doctors and nurses out of hospitals to help them understand how users used the system in practice. You essentially don't understand at all how prosecutors, police, secretaries, and all the other people involved in law enforcement use this system. There are clear parallels to replacing systems like this. Particularly when you have to do a live transition.
- japhyr 12y ago
downandout 12y ago
The problem is that even relatively experienced contractors tend to not get these kinds of contracts. You really have to be well known and connected to be chosen for this kind of thing. It's no accident that they chose a multi-billion dollar defense contractor for this. It's also no accident that it has taken so long and cost so much. Being efficient and delivering on time - something they certainly have the capability to do - simply isn't as profitable as dragging it out.
Sophistifunk 12y ago
You're under the impression that a) these projects are meant to succeed, and b) merit in developing the solutions is important in the selection. As far as the people who are actually in charge are concerned, the important outcome is to spend (thus justify) their budget, and to be seen to take action. As for selection, it's about shmoozing the right people, who you know, and navigating labyrinthine government conditions that have more to do with bureaucratic ass-covering and factory production schedules than software.

by japhyr 12y ago

zaroth 12y ago

If you ever had the opportunity to look under the covers on a project like this, you would find that all the complexity is manufactured. It's not even the 80/20 rule. It's more like 99/1 rule. As in, I see this requirements list, and I have two options for you. A) I can deliver 99% of what you want for 1% of your stated budget, or B) I can take 1000% of your budget and still fail to deliver 100%.

omouse 12y ago
The sooner you abandon any software design and engineering principles such as estimation and planning you will find that your project will start seeing more bugs and will be developed far more slowly. High quality code = rapid development.
- zaroth 12y ago
  Either you have an incredible sense of irony, or there's a bug in your comment.

x0x0 12y ago

But the trick is, you probably couldn't do a better job. There is little technical challenge in a product like this. Building crud apps that scale to low tens of thousands of users is not that challenging. Since (I believe) we all share that belief, then why do projects like this keep failing? I've worked on large medical information systems. The problems those encounter are (1) you essentially never get to bulk replace entire systems; you will have to support ongoing data ingestion and export to other running systems, often systems not designed to support this or with poor apis; (1a) if / when you fuck up, critical processes are disrupted. You have to engineer your systems and create backup processes to make this very unlikely to happen. There is a case I can't really share too many details of where a database was accidentally dropped in a datacenter of a hospital chain you've heard of. Said hospital chain was unable to access medical records for 20+ hours; this was in the 90s running on giant hpux machines and it took a long time to reimport data off tape. My employer was on the hook for $30M iirc, and I wouldn't be surprised if this actually contributed to serious patient injuries or deaths. Now obviously law enforcement isn't quite as critical as medical records, but it's not far behind. (2) over the life of such systems, lots of workflows / processes have developed around using them inside hospitals or large doctors offices. You have to discover these processes, mostly stored in the heads of line level workers and their immediate supervisors, and figure out how to enable them in your shiny new system. To be concrete: the medical record system didn't support some requirement. That didn't stop users from needing it. So employees or doctors decided to use a field in a certain way, and they know how to interpret the presence and/or data in that field. You don't even know a field is being repurposed, let alone how they actually use it. This will be a large problem importing all the old data and in recreating the necessary functionality. (3) no one person, and probably no small set of people, know or understand all the ways the system is used. Somehow, you have to capture that knowledge and use it to create a functioning new system. At least the medical record company I worked for hired lots of doctors and nurses out of hospitals to help them understand how users used the system in practice. You essentially don't understand at all how prosecutors, police, secretaries, and all the other people involved in law enforcement use this system. There are clear parallels to replacing systems like this. Particularly when you have to do a live transition.

japhyr 12y ago

downandout 12y ago

The problem is that even relatively experienced contractors tend to not get these kinds of contracts. You really have to be well known and connected to be chosen for this kind of thing. It's no accident that they chose a multi-billion dollar defense contractor for this. It's also no accident that it has taken so long and cost so much. Being efficient and delivering on time - something they certainly have the capability to do - simply isn't as profitable as dragging it out.

Sophistifunk 12y ago

You're under the impression that a) these projects are meant to succeed, and b) merit in developing the solutions is important in the selection. As far as the people who are actually in charge are concerned, the important outcome is to spend (thus justify) their budget, and to be seen to take action. As for selection, it's about shmoozing the right people, who you know, and navigating labyrinthine government conditions that have more to do with bureaucratic ass-covering and factory production schedules than software.