6 ms·

Alaska government spent $28M on database project before it was scrapped

37 points by japhyr 12y ago

falcolas 12y ago
> The replacement system was to be built from scratch The downfall of many a project. Most just don't accumulate $28m in costs before they fold.
- x0x0 12y ago
  Given the ratio of expense to scale of the system, I doubt that's true. Many similarly scaled projects fail inside private companies, but we just don't hear about them.
us0r 12y ago
Ah, SAIC (now known as Leidos). When you think of shady, corrupt government contractors - these guys are basically the arm pit of that lot. I always wondered why they take on such small projects. Northrop just botched a system in Tennessee[0]. http://www.tennessean.com/story/news/2015/01/12/tenncare-fires-northrop-grumman/21643813/ http://www.tennessean.com/story/news/2015/01/12/tenncare-fir...
- spiritplumber 12y ago
  Because it's basically free money AND free on-the-job training/evaluation for their junior guys.
- hudibras 12y ago
  Yup, when I saw "Virginia-based defense contractor" my first thought was SAIC.
deleted 12y ago
[deleted]
yeukhon 12y ago
As much as I hate Oracle / IBM contracted software (my alma mater paid Oracle more than $600 millions to replace our university portal, and half of what they did was customize existing ecommerce system to fit our need), compare to this project, I feel much better about the new portal Oracle delivered to my alma mater. Look, I am okay with the contractor subcontracts to 34 other contractors. I will take that as hiring consultants from 34 other consulting agencies. But I can bet with my life they were all working in vacuum and in a full water fall model. No doubt we need to start with analysis, and propose a possible solution like database schema and migration, but one can still work with other parts of the project iteratively, e.g. front-end and some other micro-services. I also suspect contractors hire offshore which can further complicate the development process as one has to deal with language barrier (kind of a minor issue) and timezone difference. FYI I do work with offshore day-to-day and the one I work with are excellent. Like healtcare.gov, and the fictional (but based on real life stories) Phoenix Project, there was little to no human resource and IT capacity planning, and management oversight is completely broken with 35 contractors involved. I cannot underestimate the complexity involved in migrating old data, but I bet if we gather a bunch of database professors they can probably come up with a pretty good database migration plan and a new database schema design.
- zaroth 12y ago
  Just like healthcare.gov, the reality is you start with an absolutely trivial problem and turn it into a nightmare. CRUD is CRUD. The complexity is imposed by project managers and engineers that can't help themselves or simply don't know better. Seen it happen over and over, and often just the effort to stop the madness is more work than a fully working sane implementation. More than once I've seen stealth mode side-projects run on a single engineer's own time out-deliver and out-perform the fully funded project because the standard approach is to take absolutely trivial problems and heap on absurd layers of complexity which of course are full of bugs and utterly fail in the field. I wonder if taxpayers have ever tried to sue for damages, but I'm sure it's impossible.
  - x0x0 12y ago
    governments -- essentially taxpayers -- often sue eg oregon is suing oracle over their failed implementation of the state obamacare site http://www.reuters.com/article/2014/08/22/us-usa-oregon-oracle-idUSKBN0GM21V20140822 http://www.reuters.com/article/2014/08/22/us-usa-oregon-orac...
deleted 12y ago
[deleted]
japhyr 12y ago
This kind of story always makes me think I'll never really be out of work. I like to think that given this level of funding, I could do a much better job in a situation like this than the firm that was hired did. I've been programming recreationally since I was a kid in the '70s, and I've started to do some professional work in the last few years. I started out in BASIC, and then became familiar with C, Fortran, and Pascal in college. I've since learned Java, and now my language of choice is Python. I have a basic competence in SQL, git, and any number of other tools we use in modern development approaches. I know work like this isn't easy. I know the data is stored in esoteric structures on long-outdated hardware, running on OSes I've never touched. I know there are issues with security clearances and background checks. But I have to think - give me $28m, let me work on this full time for a couple years, and I'll have some meaningful work done and some usable end products. I'd give myself a modest salary, and hire people with way more experience than myself in the areas needed. I can even see a path into this line of work - complete a series of successively more complex and sensitive data-migration projects. I like to think that doing that work wouldn't turn me into someone running a firm like the one hired in this situation. For people who've worked on projects like this that turned out well, is this something any of us with reasonable competence could get into, if we build an appropriate portfolio? Also, I live in Alaska so I find this story particularly frustrating. Not just because of the wasted money, but also because I want our law enforcement personnel to have a well-functioning system at their disposal.
- zaroth 12y ago
  If you ever had the opportunity to look under the covers on a project like this, you would find that all the complexity is manufactured. It's not even the 80/20 rule. It's more like 99/1 rule. As in, I see this requirements list, and I have two options for you. A) I can deliver 99% of what you want for 1% of your stated budget, or B) I can take 1000% of your budget and still fail to deliver 100%.
  - omouse 12y ago
    The sooner you abandon any software design and engineering principles such as estimation and planning you will find that your project will start seeing more bugs and will be developed far more slowly. High quality code = rapid development.
earleybird 12y ago
If the organization is dysfunctional the project is doomed from the outset. From personal experience I've been in or seen more dysfuntion than function. Probably the most illuminating quote from the article: "The failed project exposed simmering layers of tension within the Department of Public Safety and its division that supports law enforcement technology and criminal history information statewide, documents show, and has triggered numerous restructuring efforts, including a new internal governance board."
smoyer 12y ago
One of these days I'd like to volunteer (with a large enough group of peers) to build a system like this for the government free-of-charge. I can't imagine that a hundred good developers working part-time couldn't get this system migrated in a reasonable amount of time. One clue to why if failed describes there being a tension between departments and a new governance board. Providing a system like this based only on best practices should be pretty feasible if you throw out all the petty politics and "perceived but useless" pet features. I'd imagine the current system (as it's mainframe-based) pretty much follows the typical forms entry (perhaps they're still using green-screens?). Moving the existing system as-is to a modern version of an RDBMS with an equivalent web-based front-end should be the first step. Once you're current, you can add features as you go.
- japhyr 12y ago
  I'm not sure that a hundred good developers would complete a project like this. I'd rather take a small team who could dig deeply into the project, and commit full time to it. One or more of those people would have to be pretty politically savvy to help pull it off.
  - acveilleux 12y ago
    It's death by a thousand tiny procedural cuts. There's thousands of users, dozens of processes and as many external dependencies. Money is always subject to whomever got elected last November and a great many people resent you. They're 57, been working with the previous system since the mid 80s and they like it dammit. They see the writing on the wall and the mainframe operator or data entry specialist job they have is a goner. Shy of a full pension too. Or the prima Donna department that always got their way is blue balling the analysts the whole way through because they don't own this anymore. Problems in this type of projects are all people and processes. I also work in Healthcare IT on the vendor side and I see projects that get shit done and project that die. It's about management.
    - spiritplumber 12y ago
      Yep, I can confirm that. My strategy is to tell people "I am going to make it work and you are going to either help me, or watch helplessly as I make it work without you and make you look like an idiot". It helps to take assertions like "you cannot do that" literally, even though they tend to be said to mean "you may not do that" or "you should not do that", and prove that you can in fact, do that all you like because there are no laws of physics preventing it. It has caused a few surreal meetings, like the time when a guy did a 40 minute long presentation on "I need more people and more money in my workgroup to make this work" while the first batch of completed, working, tested production units (built by me and one other guy after we got sick of the BS, I even paid for the manufacturing out of pocket) was in the middle of the conference table. Imagine a half dozen people trying to ignore what was literally in front of their eyes. Another time it also has caused me to escort a security guard out of the building because he was being disruptive. He had been told to escort me out of the building, grabbed me without asking my permission, and found himself in a headlock, then was informed that if he did want to fight me, he was welcome to. Then I got back inside and finished my business. Sometimes you just have to say "full speed ahead and damn the torpedos", or nothing gets done. (My father's rule of fights: Never hit first, always hit last)
TYPE_FASTER 12y ago
Here's another large contract failure: http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-in-citytime-corruption-trial.html?_r=0 http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-in... Multiple large organizations on a large scale consulting engagement seems to lower the chances of success.
AndrewKemendo 12y ago
There are literally thousands of projects like this throughout the US federal government.
coldcode 12y ago
"In an email, a spokeswoman for Leidos would only say that “the Leidos system was delivered on time, on-scope and is fully functioning.”" I can't imagine saying that with a straight face.
- jobu 12y ago
  Yeah, given the end result I can't imagine what they're thinking by saying that. To their credit, the auditors said this was their first negative review of the company: "Charles Collins Jr., the lead author of the MTG report, said in an interview that the consulting group had encountered SAIC in the past. He said he believed this to be MTG’s first critical report on the contractor." Their review of another department (Division of Statewide Services) was particularly damning though: "In the December 2013 “organizational review” of the Division of Statewide Services, also conducted by MTG Consulting Group, auditors found evidence of what they called “a seriously dysfunctional IT support organization.” The review found that the division lacked governance and communication, its benefits to other divisions and departments were unclear, and there was a serious aversion to risk or change." Sounds like the place was being run by Mordac: http://goo.gl/o7JRvr http://goo.gl/o7JRvr
- omouse 12y ago
  If you're a manager, salesperson or work on the business side, you have to tell tall tales all the time. It's called people-skills.
fspacef 12y ago
Bureaucracy at it's best
imperialdrive 12y ago
I'm just starting to flesh out a new database system for an SF government related agency, ground level, replacing something almost 15 years old. If there is a single person or two out there that wants to sit in on a meeting, brainstorming, it would be much welcomed. 501c3 org. I'm not even a DB guy, more of a general tech solutions architect, but I'm curious (as others on here are) just how much waste is necessary with these kind of projects. So far we've spent about 20k on maintaining the current system, maybe even more over the past 2 years. Anyway, I could go on and on, but if you're in SF, drop a line for lunch/coffee on me, dan at sfpretrial . org
jonas21 12y ago
$28 million? That's nothing. The Air Force recently spent over $1 billion on a cancelled logistics software project. http://www.nytimes.com/2012/12/09/technology/air-force-stumbles-over-software-modernization-project.html http://www.nytimes.com/2012/12/09/technology/air-force-stumb...
- adwf 12y ago
  $1 billion? That's nothing. In the UK we spent over £10 billion on our abandoned NHS IT system ;) http://www.theguardian.com/society/2013/sep/18/nhs-records-system-10bn http://www.theguardian.com/society/2013/sep/18/nhs-records-s...
  - arikrak 12y ago
    That's something, even more than the US government spent on the Obamacare website.
nova22033 12y ago
>The company is also known for big-ticket software development contracts with federal government agencies, from the National Security Administration to the Department of Defense to the FBI. big ticket projects for the FBI? Like VCF? http://www.washingtonpost.com/wp-dyn/content/article/2006/08/17/AR2006081701485.html http://www.washingtonpost.com/wp-dyn/content/article/2006/08... > the system delivered by SAIC was so incomplete and unusable that it left the FBI with little choice but to scuttle the effort altogether.
omouse 12y ago
This is why software estimation and as much honesty as possible is required when dealing with clients. For all the talk of risk management, clients still believe the most optimistic schedules! Whatever a sales person or manager says, pad it by 50-100% and if you can live with that, then you can ask for updated estimates every month to increase accuracy.
aaron695 12y ago
IT Conversations had a good talk on large scale databases and why they fail. ($170 million FBI database) http://cdn.itconversations.com/ITC.IEEE-FBISoftware-2006.06.01.mp3 http://cdn.itconversations.com/ITC.IEEE-FBISoftware-2006.06.... TL;DL No one actually sat down and specifically told the contractors what was needed because they didn't bother to even work it out for themselves. So the contractors could get away with what they wanted.

37 points by japhyr 12y ago

> The replacement system was to be built from scratch The downfall of many a project. Most just don't accumulate $28m in costs before they fold.

x0x0 12y ago
Given the ratio of expense to scale of the system, I doubt that's true. Many similarly scaled projects fail inside private companies, but we just don't hear about them.

us0r 12y ago

Ah, SAIC (now known as Leidos). When you think of shady, corrupt government contractors - these guys are basically the arm pit of that lot. I always wondered why they take on such small projects. Northrop just botched a system in Tennessee[0]. http://www.tennessean.com/story/news/2015/01/12/tenncare-fires-northrop-grumman/21643813/ http://www.tennessean.com/story/news/2015/01/12/tenncare-fir...

spiritplumber 12y ago
Because it's basically free money AND free on-the-job training/evaluation for their junior guys.
hudibras 12y ago
Yup, when I saw "Virginia-based defense contractor" my first thought was SAIC.

deleted 12y ago

[deleted]

yeukhon 12y ago

As much as I hate Oracle / IBM contracted software (my alma mater paid Oracle more than $600 millions to replace our university portal, and half of what they did was customize existing ecommerce system to fit our need), compare to this project, I feel much better about the new portal Oracle delivered to my alma mater. Look, I am okay with the contractor subcontracts to 34 other contractors. I will take that as hiring consultants from 34 other consulting agencies. But I can bet with my life they were all working in vacuum and in a full water fall model. No doubt we need to start with analysis, and propose a possible solution like database schema and migration, but one can still work with other parts of the project iteratively, e.g. front-end and some other micro-services. I also suspect contractors hire offshore which can further complicate the development process as one has to deal with language barrier (kind of a minor issue) and timezone difference. FYI I do work with offshore day-to-day and the one I work with are excellent. Like healtcare.gov, and the fictional (but based on real life stories) Phoenix Project, there was little to no human resource and IT capacity planning, and management oversight is completely broken with 35 contractors involved. I cannot underestimate the complexity involved in migrating old data, but I bet if we gather a bunch of database professors they can probably come up with a pretty good database migration plan and a new database schema design.

zaroth 12y ago
Just like healthcare.gov, the reality is you start with an absolutely trivial problem and turn it into a nightmare. CRUD is CRUD. The complexity is imposed by project managers and engineers that can't help themselves or simply don't know better. Seen it happen over and over, and often just the effort to stop the madness is more work than a fully working sane implementation. More than once I've seen stealth mode side-projects run on a single engineer's own time out-deliver and out-perform the fully funded project because the standard approach is to take absolutely trivial problems and heap on absurd layers of complexity which of course are full of bugs and utterly fail in the field. I wonder if taxpayers have ever tried to sue for damages, but I'm sure it's impossible.
- x0x0 12y ago
  governments -- essentially taxpayers -- often sue eg oregon is suing oracle over their failed implementation of the state obamacare site http://www.reuters.com/article/2014/08/22/us-usa-oregon-oracle-idUSKBN0GM21V20140822 http://www.reuters.com/article/2014/08/22/us-usa-oregon-orac...

deleted 12y ago

[deleted]

japhyr 12y ago

This kind of story always makes me think I'll never really be out of work. I like to think that given this level of funding, I could do a much better job in a situation like this than the firm that was hired did. I've been programming recreationally since I was a kid in the '70s, and I've started to do some professional work in the last few years. I started out in BASIC, and then became familiar with C, Fortran, and Pascal in college. I've since learned Java, and now my language of choice is Python. I have a basic competence in SQL, git, and any number of other tools we use in modern development approaches. I know work like this isn't easy. I know the data is stored in esoteric structures on long-outdated hardware, running on OSes I've never touched. I know there are issues with security clearances and background checks. But I have to think - give me $28m, let me work on this full time for a couple years, and I'll have some meaningful work done and some usable end products. I'd give myself a modest salary, and hire people with way more experience than myself in the areas needed. I can even see a path into this line of work - complete a series of successively more complex and sensitive data-migration projects. I like to think that doing that work wouldn't turn me into someone running a firm like the one hired in this situation. For people who've worked on projects like this that turned out well, is this something any of us with reasonable competence could get into, if we build an appropriate portfolio? Also, I live in Alaska so I find this story particularly frustrating. Not just because of the wasted money, but also because I want our law enforcement personnel to have a well-functioning system at their disposal.

zaroth 12y ago
If you ever had the opportunity to look under the covers on a project like this, you would find that all the complexity is manufactured. It's not even the 80/20 rule. It's more like 99/1 rule. As in, I see this requirements list, and I have two options for you. A) I can deliver 99% of what you want for 1% of your stated budget, or B) I can take 1000% of your budget and still fail to deliver 100%.
- omouse 12y ago
  The sooner you abandon any software design and engineering principles such as estimation and planning you will find that your project will start seeing more bugs and will be developed far more slowly. High quality code = rapid development.

earleybird 12y ago

If the organization is dysfunctional the project is doomed from the outset. From personal experience I've been in or seen more dysfuntion than function. Probably the most illuminating quote from the article: "The failed project exposed simmering layers of tension within the Department of Public Safety and its division that supports law enforcement technology and criminal history information statewide, documents show, and has triggered numerous restructuring efforts, including a new internal governance board."

smoyer 12y ago

One of these days I'd like to volunteer (with a large enough group of peers) to build a system like this for the government free-of-charge. I can't imagine that a hundred good developers working part-time couldn't get this system migrated in a reasonable amount of time. One clue to why if failed describes there being a tension between departments and a new governance board. Providing a system like this based only on best practices should be pretty feasible if you throw out all the petty politics and "perceived but useless" pet features. I'd imagine the current system (as it's mainframe-based) pretty much follows the typical forms entry (perhaps they're still using green-screens?). Moving the existing system as-is to a modern version of an RDBMS with an equivalent web-based front-end should be the first step. Once you're current, you can add features as you go.

japhyr 12y ago
I'm not sure that a hundred good developers would complete a project like this. I'd rather take a small team who could dig deeply into the project, and commit full time to it. One or more of those people would have to be pretty politically savvy to help pull it off.
- acveilleux 12y ago
  It's death by a thousand tiny procedural cuts. There's thousands of users, dozens of processes and as many external dependencies. Money is always subject to whomever got elected last November and a great many people resent you. They're 57, been working with the previous system since the mid 80s and they like it dammit. They see the writing on the wall and the mainframe operator or data entry specialist job they have is a goner. Shy of a full pension too. Or the prima Donna department that always got their way is blue balling the analysts the whole way through because they don't own this anymore. Problems in this type of projects are all people and processes. I also work in Healthcare IT on the vendor side and I see projects that get shit done and project that die. It's about management.
  - spiritplumber 12y ago
    Yep, I can confirm that. My strategy is to tell people "I am going to make it work and you are going to either help me, or watch helplessly as I make it work without you and make you look like an idiot". It helps to take assertions like "you cannot do that" literally, even though they tend to be said to mean "you may not do that" or "you should not do that", and prove that you can in fact, do that all you like because there are no laws of physics preventing it. It has caused a few surreal meetings, like the time when a guy did a 40 minute long presentation on "I need more people and more money in my workgroup to make this work" while the first batch of completed, working, tested production units (built by me and one other guy after we got sick of the BS, I even paid for the manufacturing out of pocket) was in the middle of the conference table. Imagine a half dozen people trying to ignore what was literally in front of their eyes. Another time it also has caused me to escort a security guard out of the building because he was being disruptive. He had been told to escort me out of the building, grabbed me without asking my permission, and found himself in a headlock, then was informed that if he did want to fight me, he was welcome to. Then I got back inside and finished my business. Sometimes you just have to say "full speed ahead and damn the torpedos", or nothing gets done. (My father's rule of fights: Never hit first, always hit last)

TYPE_FASTER 12y ago

Here's another large contract failure: http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-in-citytime-corruption-trial.html?_r=0 http://www.nytimes.com/2013/11/23/nyregion/3-found-guilty-in... Multiple large organizations on a large scale consulting engagement seems to lower the chances of success.

AndrewKemendo 12y ago

There are literally thousands of projects like this throughout the US federal government.

coldcode 12y ago

"In an email, a spokeswoman for Leidos would only say that “the Leidos system was delivered on time, on-scope and is fully functioning.”" I can't imagine saying that with a straight face.

jobu 12y ago
Yeah, given the end result I can't imagine what they're thinking by saying that. To their credit, the auditors said this was their first negative review of the company: "Charles Collins Jr., the lead author of the MTG report, said in an interview that the consulting group had encountered SAIC in the past. He said he believed this to be MTG’s first critical report on the contractor." Their review of another department (Division of Statewide Services) was particularly damning though: "In the December 2013 “organizational review” of the Division of Statewide Services, also conducted by MTG Consulting Group, auditors found evidence of what they called “a seriously dysfunctional IT support organization.” The review found that the division lacked governance and communication, its benefits to other divisions and departments were unclear, and there was a serious aversion to risk or change." Sounds like the place was being run by Mordac: http://goo.gl/o7JRvr http://goo.gl/o7JRvr
omouse 12y ago
If you're a manager, salesperson or work on the business side, you have to tell tall tales all the time. It's called people-skills.

fspacef 12y ago

Bureaucracy at it's best

imperialdrive 12y ago

I'm just starting to flesh out a new database system for an SF government related agency, ground level, replacing something almost 15 years old. If there is a single person or two out there that wants to sit in on a meeting, brainstorming, it would be much welcomed. 501c3 org. I'm not even a DB guy, more of a general tech solutions architect, but I'm curious (as others on here are) just how much waste is necessary with these kind of projects. So far we've spent about 20k on maintaining the current system, maybe even more over the past 2 years. Anyway, I could go on and on, but if you're in SF, drop a line for lunch/coffee on me, dan at sfpretrial . org

jonas21 12y ago

$28 million? That's nothing. The Air Force recently spent over $1 billion on a cancelled logistics software project. http://www.nytimes.com/2012/12/09/technology/air-force-stumbles-over-software-modernization-project.html http://www.nytimes.com/2012/12/09/technology/air-force-stumb...

adwf 12y ago
$1 billion? That's nothing. In the UK we spent over £10 billion on our abandoned NHS IT system ;) http://www.theguardian.com/society/2013/sep/18/nhs-records-system-10bn http://www.theguardian.com/society/2013/sep/18/nhs-records-s...
- arikrak 12y ago
  That's something, even more than the US government spent on the Obamacare website.

nova22033 12y ago

>The company is also known for big-ticket software development contracts with federal government agencies, from the National Security Administration to the Department of Defense to the FBI. big ticket projects for the FBI? Like VCF? http://www.washingtonpost.com/wp-dyn/content/article/2006/08/17/AR2006081701485.html http://www.washingtonpost.com/wp-dyn/content/article/2006/08... > the system delivered by SAIC was so incomplete and unusable that it left the FBI with little choice but to scuttle the effort altogether.

omouse 12y ago

This is why software estimation and as much honesty as possible is required when dealing with clients. For all the talk of risk management, clients still believe the most optimistic schedules! Whatever a sales person or manager says, pad it by 50-100% and if you can live with that, then you can ask for updated estimates every month to increase accuracy.

aaron695 12y ago

IT Conversations had a good talk on large scale databases and why they fail. ($170 million FBI database) http://cdn.itconversations.com/ITC.IEEE-FBISoftware-2006.06.01.mp3 http://cdn.itconversations.com/ITC.IEEE-FBISoftware-2006.06.... TL;DL No one actually sat down and specifically told the contractors what was needed because they didn't bother to even work it out for themselves. So the contractors could get away with what they wanted.