11 ms·

Healthcare.gov crashing because backend doomed in the requirements stage

45 points by blurpin 13y ago

wheaties 13y ago
I can't wait to see what the report of it didn't work actually says. It's quite possible that one of the services that the website hit couldn't handle the traffic which in turn caused a bottleneck in other systems.
mrharrison 13y ago
I know the people who worked on this. This is not what happened. Just yet another propaganda piece by Forbes.
- pfraze 13y ago
  Can you elaborate on what did happen?
- briandear 13y ago
  Well something happened. You might want to tell those guys that 500 million lines of code is a bit ridiculous.
  - dodyg 13y ago
    You know what's cool? 1 billion lines of code.
  - twistedpair 13y ago
    I wonder what those LoC were for? XML parsing remote API calls between various disparate systems? Sounds like it's well beyond operation system complexity. Linux kernel is currently 15M LoC. Then again, kernels and operating systems are quite logical and laid out by architects. Systems like these are rats nests defined by bureaucrats and integration nightmares.
- darkarmani 13y ago
  > the interface kept asking me to reenter the information for our dependents. Farther on in the process it doubled the number of our children (to 6!) Even though it listed our children’s names twice (identically) and even though these clones had identical SS numbers, the validation software was not smart enough to realize that the duplicate records were describing the same people! This didn't happen? Are you suggesting he made this whole story up?
  - sp332 13y ago
    The article, as the headline states, is about poor requirements dooming the project.
- GVIrish 13y ago
  Can they elaborate on what actually did happen? If I were your friends I would be trying to contact news agencies with corrections if at all possible because this is shaping up to be the highest profile government IT failure in US history.
- sunraa 13y ago
  Per the NYT: "But the government was so slow in issuing specifications that the firm did not start writing software code until this spring, according to people familiar with the process." Source: http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html http://www.nytimes.com/2013/10/13/us/politics/from-the-start... (October 12) And from today (Oct 21): http://www.nytimes.com/2013/10/21/us/insurance-site-seen-needing-weeks-to-fix.html?hp http://www.nytimes.com/2013/10/21/us/insurance-site-seen-nee... In a previous life I worked on similar software development projects at the US State level. This project was several orders of magnitude larger in scope and in complexity than what I worked on. Assuming a generous start date for Spring - say March 1 - this boils down to ~7 months to design and develop to the Oct 1 hard deadline. Nearly impossible in my opinion. I'm not discounting that there couldn't be issues else where in whole process. These would include: communication overhead between all the various agencies and contractors, integration nightmares, and not enough skilled folks at Medicare to manage all of this. But having the requirements delivered several months before roll-out seems to me a recipe for disaster.
  - Retric 13y ago
    You don't need 100% final requirements to start building the system. If nothing else start working though the interfaces with other systems, and try and get as close as possible to a working prototype. Granted funding is often an issue, but even a tiny team given a few months head start can get a lot done.
    - Plasmoid 13y ago
      On government contracts that's actually a problem. You can't bill for any work you did without 'authorization'. Which means you can't do anything until a finalized spec, signed in triplicate, is physically delivered to you.
      - betterunix 13y ago
        I have wondered for some time now if it is legal to develop tools and libraries that are relevant to an upcoming contract, but which are also general purpose and could be used elsewhere.
- rco8786 13y ago
  I don't know if I would use the word propaganda...but speculation definitely fits the bill.
gry 13y ago
Thoughtback [https://thoughtback.com/ https://thoughtback.com/] sent me a pertinent reminder this morning. No affiliation, just a great tool. Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness. [1] [1] The Influence of Organizational Structure On Software Quality: An Empirical Case Study http://research.microsoft.com/apps/pubs/default.aspx?id=70535 http://research.microsoft.com/apps/pubs/default.aspx?id=7053... -- edit: formatting
- joshdick 13y ago
  "If you have four groups working on a compiler, you'll get a 4-pass compiler" -- esr
gesman 13y ago
Could anyone share which platform / framework they used?
- Iftheshoefits 13y ago
  There probably isn't any one "platform/framework" they used. This beast touches too many systems, many of which are 'legacy' for the phrase "platform/framework" to really apply. If it's anything like what I saw at the one (and only) government contractor I ever worked for, the site is a frankensteinian nightmare of COTS ('Commercial Off The Shelf') crap (which federal contracts encourage, when they don't outright require it) and sweatshop-style integration development run by contractors who hire anybody with a heartbeat and a handful of competent people, and pay them all peanuts while billing $200/hr in labor to the government.
- jonknee 13y ago
  I'm sure tons. The web site's code is pretty straight forward, it's that interacting with so many other parties that is not (besides anything having to do with actual insurance, Healthcare.gov interacts with the IRS, social security, homeland security, Equifax, etc etc). I'm not sure what the solution is since these checks are not optional. It would be a huge political scandal if just one illegal immigrant was able to buy insurance using Healthcare.gov and there are surely many groups trying every day to make sure this happens. Is there any large benefits system that works well? All that I have seen from MegaCorps are also awful and they don't even have all the complex stuff to make sure you qualify (since you work at the company, you qualify).
- babs474 13y ago
  mnsure.org was throwing springframework exceptions the day it launched
  - uxp 13y ago
    healthcare.gov itself was a Ruby Jekyll static site. It was on GitHub[0] for a while, but mysteriously disappeared. Here[1] is a clone of it, and what seems to be the currently agreed upon canonical version. My state appears to be using Joomla based on the meta-generator tag in it's head tag. http://www.avenueh.com/ http://www.avenueh.com/ [0] https://github.com/CMSgov/HealthCare.gov-Open-Source-Release https://github.com/CMSgov/HealthCare.gov-Open-Source-Release [1] https://github.com/blencorp/HealthCare.gov-Open-Source-Release https://github.com/blencorp/HealthCare.gov-Open-Source-Relea...
powertower 13y ago
> In my case, even the most expensive plan for our family in the State of Maine, where we live, is half the cost of what we are currently paying, exclusive of any subsidies. The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-down-obamacare-will-increase-avg-individual-market-insurance-premiums-by-99-for-men-62-for-women/ http://www.forbes.com/sites/theapothecary/2013/09/25/double-...
- chimeracoder 13y ago
  > The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-.. http://www.forbes.com/sites/theapothecary/2013/09/25/double-.... It still boggles my mind that people are surprised by this. Mathematically, including people with pre-existing conditions will mean either that those people are required to pay for the (expected, not actual) costs of their own care, or that other people will be required to pay for those costs[0]. This effect is much larger than the effect of mandating insurance for younger, healthier people who otherwise elected out of getting insurance[1]. Note: this is not a normative statement about whether this is a "good" outcome. This is just a a mathematical observation about the way insurance works, from a mathematical and economic perspective. [0] Subsidies come from taxpayer money, so they have no net effect on this accounting - it changes how the money changes hands, but the money still has to come from somewhere. [1] I'm waving my hands around the math here, but if people are interested I can dig up the more detailed blog post draft that I did a while back explaining this in full detail.
  - lukasb 13y ago
    The effect that you're ignoring is that insurance premiums have a hefty amount of margin built into them, and the ACA produces a collective bargaining effect that gives insurers an incentive to offer a competitive price on the exchange.
  - powertower 13y ago
    > It still boggles my mind that people are surprised by this... I was surprised because I was told that this new coverage affordability came from getting insurance companies greed in check. I didn't know that insurance companies were actually lobying for Obamacare, not against it. http://www.policymic.com/articles/62541/here-s-the-proof-obamacare-is-for-the-insurance-companies-and-not-for-you http://www.policymic.com/articles/62541/here-s-the-proof-oba...
mbesto 13y ago
If there is anything that can come of good of my taxes (a ridiculous amount of it in fact) is that we should educate people on the difficulties of software development. This would make a fantastic case study on what I presume are the issues at hand: - Maintaining complex code (+1mil LOC) - Classic "waterfall" approaches to software development (and the inherent problem with trying to blueprint requirements up front) - The mythical man month[1] - Change management Funny thing is...these are all human problems. [1] - http://en.wikipedia.org/wiki/The_Mythical_Man-Month http://en.wikipedia.org/wiki/The_Mythical_Man-Month
drakaal 13y ago
I didn't see anything that meant it was doomed from the requirements stage. This was more of an Obama love fest and defense of ObamaCare than a technical article describing who the backend was doomed. For what this cost to build it could have been much better, much simpler, and handled the scale gracefully. Yes privacy concerns keep it off the elastic cloud. (not a point mentioned in the article) Yes authenticated users limit caching and make scaling harder. (not really mentioned) But when you are talking about everyone in the US, you only have so many combinations. Income in something like 32 ranges, number of kids 1 through 12 (greater than 12 they are all free) a bunch of ages. We just saw a post that only 15 searches are new to google each day, so the number of combinations is much less than search which does lots of caching. The requirements were messed up, the implementation was. Clearly not built by people who had ever done "Facebook scale" stuff.
- GVIrish 13y ago
  Well, some reports are that the requirements were changing until 1 week before launch, and that development didn't start until March of this year due to requirements delays. Just those two factors alone would doom just about any project of this scale.
- coldcode 13y ago
  The primary contractor has had plenty of large gov projects including some for Medicare and SS. Of course so have all large consulting firms and they fail often as well. What the issue is isn't clear from random musings. Unless someone who worked on the project at a high enough level can explain all we have is guesswork.
- twistedpair 13y ago
  They screwed the pooch, clearly. However I wouldn't assume that the insurance landscape is so simple. You're talking about an insurance industry and legal system that has had a fifty year head start making the process of insurance buying as complicated and obfuscated as possible.
chucknelson 13y ago
Wow, this particular excerpt from the article is stretching things a bit too thin: "Clearly there are problems with the government’s IT procurement processes, but if the same level of dysfunction had been present in the software behind Obama’s political campaign (as it was, apparently, behind Romney’s) the President would have never been reelected." Really? I doubt the campaign websites were the deciding factor. Anyway, a lot of speculation in this article...not very insightful or informative.
- ebiester 13y ago
  The software in question was the software used to determine the best media buys, the volunteer organization software, and similar that was known to be part of his advantage.
- nightpool 13y ago
  See http://arstechnica.com/information-technology/2012/11/built-to-win-deep-inside-obamas-campaign-tech/ http://arstechnica.com/information-technology/2012/11/built-... and http://awsofa.info/ http://awsofa.info/ (https://news.ycombinator.com/item?id=5542368 https://news.ycombinator.com/item?id=5542368) as well as http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/ http://arstechnica.com/information-technology/2012/11/inside... (https://news.ycombinator.com/item?id=4764864 https://news.ycombinator.com/item?id=4764864) for comparison.
- nightpool 13y ago
  See http://arstechnica.com/information-technology/2012/11/built-to-win-deep-inside-obamas-campaign-tech/ http://arstechnica.com/information-technology/2012/11/built-... and http://awsofa.info/ http://awsofa.info/ (https://news.ycombinator.com/item?id=5542368 https://news.ycombinator.com/item?id=5542368) as well as http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/ http://arstechnica.com/information-technology/2012/11/inside... (https://news.ycombinator.com/item?id=4764864 https://news.ycombinator.com/item?id=4764864) for comparison.
- skwirl 13y ago
  They aren't talking about the campaign websites. Both campaigns had complex internal software systems to help target field worker campaign activities, get out the vote efforts, monitor and respond to turnout data on election day, etc. According to various reports, the Obama campaign was hugely successful in this endeavor [1] while the Romney campaign's election day voter turnout system was a disaster [2]. [1] http://www.drdobbs.com/architecture-and-design/software-development-in-the-obama-campai/240146307 http://www.drdobbs.com/architecture-and-design/software-deve... [2] http://www.businessinsider.com/romney-project-orca-disaster-2012-11 http://www.businessinsider.com/romney-project-orca-disaster-...
Plasmoid 13y ago
Why government software projects go wrong [http://www.mrobinson.ca/2012/09/why-government-software-projects-going.html http://www.mrobinson.ca/2012/09/why-government-software-proj...] ibid - Part 2 [http://www.mrobinson.ca/2013/08/government-projects-going-wrong-part-2.html http://www.mrobinson.ca/2013/08/government-projects-going-wr...]
teyc 13y ago
I bet you half the problem is the software being specified in a synchronous and not asynchronous manner. If a flow chart is expressed synchronously, it immediately bottlenecks around the slow backend systems. e.g. check the applicant has 4 kids, and if that checks out, present them with these options. I attribute some of these problems to inadequate advances in today's mainstream programming languages. For instance, if futures/promises were common parlance in programming, some of these bottlenecks would have been identified earlier. I once read an interesting essay on what future programming languages could work on. One of the interesting ideas is a language that allows programmers to implement undo/redo easily. Recently there was a blog post about fuzzy computing, where a function is able to return approximate values when it is busy. The Knight Capital incident teaches us that if we architected our monitoring systems the same was as intrusion systems are done, we could have stopped activity when suspicious incidents occur outside the normal operating range. There was another incident where a Japanese trader accidentally keyed in the wrong number and promptly lost his employers a lot of money. Languages that incorporate ideas like validation and expected values as first class constructs might be able to help here. The Adriane reminded us how badly humans do when reasoning about systems where multiple agents perform the same task.

45 points by blurpin 13y ago

I can't wait to see what the report of it didn't work actually says. It's quite possible that one of the services that the website hit couldn't handle the traffic which in turn caused a bottleneck in other systems.

mrharrison 13y ago

I know the people who worked on this. This is not what happened. Just yet another propaganda piece by Forbes.

pfraze 13y ago
Can you elaborate on what did happen?
briandear 13y ago
Well something happened. You might want to tell those guys that 500 million lines of code is a bit ridiculous.
- dodyg 13y ago
  You know what's cool? 1 billion lines of code.
- twistedpair 13y ago
  I wonder what those LoC were for? XML parsing remote API calls between various disparate systems? Sounds like it's well beyond operation system complexity. Linux kernel is currently 15M LoC. Then again, kernels and operating systems are quite logical and laid out by architects. Systems like these are rats nests defined by bureaucrats and integration nightmares.
darkarmani 13y ago
> the interface kept asking me to reenter the information for our dependents. Farther on in the process it doubled the number of our children (to 6!) Even though it listed our children’s names twice (identically) and even though these clones had identical SS numbers, the validation software was not smart enough to realize that the duplicate records were describing the same people! This didn't happen? Are you suggesting he made this whole story up?
- sp332 13y ago
  The article, as the headline states, is about poor requirements dooming the project.
GVIrish 13y ago
Can they elaborate on what actually did happen? If I were your friends I would be trying to contact news agencies with corrections if at all possible because this is shaping up to be the highest profile government IT failure in US history.
sunraa 13y ago
Per the NYT: "But the government was so slow in issuing specifications that the firm did not start writing software code until this spring, according to people familiar with the process." Source: http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html http://www.nytimes.com/2013/10/13/us/politics/from-the-start... (October 12) And from today (Oct 21): http://www.nytimes.com/2013/10/21/us/insurance-site-seen-needing-weeks-to-fix.html?hp http://www.nytimes.com/2013/10/21/us/insurance-site-seen-nee... In a previous life I worked on similar software development projects at the US State level. This project was several orders of magnitude larger in scope and in complexity than what I worked on. Assuming a generous start date for Spring - say March 1 - this boils down to ~7 months to design and develop to the Oct 1 hard deadline. Nearly impossible in my opinion. I'm not discounting that there couldn't be issues else where in whole process. These would include: communication overhead between all the various agencies and contractors, integration nightmares, and not enough skilled folks at Medicare to manage all of this. But having the requirements delivered several months before roll-out seems to me a recipe for disaster.
- Retric 13y ago
  You don't need 100% final requirements to start building the system. If nothing else start working though the interfaces with other systems, and try and get as close as possible to a working prototype. Granted funding is often an issue, but even a tiny team given a few months head start can get a lot done.
  - Plasmoid 13y ago
    On government contracts that's actually a problem. You can't bill for any work you did without 'authorization'. Which means you can't do anything until a finalized spec, signed in triplicate, is physically delivered to you.
    - betterunix 13y ago
      I have wondered for some time now if it is legal to develop tools and libraries that are relevant to an upcoming contract, but which are also general purpose and could be used elsewhere.
rco8786 13y ago
I don't know if I would use the word propaganda...but speculation definitely fits the bill.

gry 13y ago

Thoughtback [https://thoughtback.com/ https://thoughtback.com/] sent me a pertinent reminder this morning. No affiliation, just a great tool. Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness. [1] [1] The Influence of Organizational Structure On Software Quality: An Empirical Case Study http://research.microsoft.com/apps/pubs/default.aspx?id=70535 http://research.microsoft.com/apps/pubs/default.aspx?id=7053... -- edit: formatting

joshdick 13y ago
"If you have four groups working on a compiler, you'll get a 4-pass compiler" -- esr

gesman 13y ago

Could anyone share which platform / framework they used?

Iftheshoefits 13y ago
There probably isn't any one "platform/framework" they used. This beast touches too many systems, many of which are 'legacy' for the phrase "platform/framework" to really apply. If it's anything like what I saw at the one (and only) government contractor I ever worked for, the site is a frankensteinian nightmare of COTS ('Commercial Off The Shelf') crap (which federal contracts encourage, when they don't outright require it) and sweatshop-style integration development run by contractors who hire anybody with a heartbeat and a handful of competent people, and pay them all peanuts while billing $200/hr in labor to the government.
jonknee 13y ago
I'm sure tons. The web site's code is pretty straight forward, it's that interacting with so many other parties that is not (besides anything having to do with actual insurance, Healthcare.gov interacts with the IRS, social security, homeland security, Equifax, etc etc). I'm not sure what the solution is since these checks are not optional. It would be a huge political scandal if just one illegal immigrant was able to buy insurance using Healthcare.gov and there are surely many groups trying every day to make sure this happens. Is there any large benefits system that works well? All that I have seen from MegaCorps are also awful and they don't even have all the complex stuff to make sure you qualify (since you work at the company, you qualify).
babs474 13y ago
mnsure.org was throwing springframework exceptions the day it launched
- uxp 13y ago
  healthcare.gov itself was a Ruby Jekyll static site. It was on GitHub[0] for a while, but mysteriously disappeared. Here[1] is a clone of it, and what seems to be the currently agreed upon canonical version. My state appears to be using Joomla based on the meta-generator tag in it's head tag. http://www.avenueh.com/ http://www.avenueh.com/ [0] https://github.com/CMSgov/HealthCare.gov-Open-Source-Release https://github.com/CMSgov/HealthCare.gov-Open-Source-Release [1] https://github.com/blencorp/HealthCare.gov-Open-Source-Release https://github.com/blencorp/HealthCare.gov-Open-Source-Relea...

powertower 13y ago

> In my case, even the most expensive plan for our family in the State of Maine, where we live, is half the cost of what we are currently paying, exclusive of any subsidies. The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-down-obamacare-will-increase-avg-individual-market-insurance-premiums-by-99-for-men-62-for-women/ http://www.forbes.com/sites/theapothecary/2013/09/25/double-...

chimeracoder 13y ago
> The elephant in the room is that this is not the typical case - http://www.forbes.com/sites/theapothecary/2013/09/25/double-.. http://www.forbes.com/sites/theapothecary/2013/09/25/double-.... It still boggles my mind that people are surprised by this. Mathematically, including people with pre-existing conditions will mean either that those people are required to pay for the (expected, not actual) costs of their own care, or that other people will be required to pay for those costs[0]. This effect is much larger than the effect of mandating insurance for younger, healthier people who otherwise elected out of getting insurance[1]. Note: this is not a normative statement about whether this is a "good" outcome. This is just a a mathematical observation about the way insurance works, from a mathematical and economic perspective. [0] Subsidies come from taxpayer money, so they have no net effect on this accounting - it changes how the money changes hands, but the money still has to come from somewhere. [1] I'm waving my hands around the math here, but if people are interested I can dig up the more detailed blog post draft that I did a while back explaining this in full detail.
- lukasb 13y ago
  The effect that you're ignoring is that insurance premiums have a hefty amount of margin built into them, and the ACA produces a collective bargaining effect that gives insurers an incentive to offer a competitive price on the exchange.
- powertower 13y ago
  > It still boggles my mind that people are surprised by this... I was surprised because I was told that this new coverage affordability came from getting insurance companies greed in check. I didn't know that insurance companies were actually lobying for Obamacare, not against it. http://www.policymic.com/articles/62541/here-s-the-proof-obamacare-is-for-the-insurance-companies-and-not-for-you http://www.policymic.com/articles/62541/here-s-the-proof-oba...

mbesto 13y ago

If there is anything that can come of good of my taxes (a ridiculous amount of it in fact) is that we should educate people on the difficulties of software development. This would make a fantastic case study on what I presume are the issues at hand: - Maintaining complex code (+1mil LOC) - Classic "waterfall" approaches to software development (and the inherent problem with trying to blueprint requirements up front) - The mythical man month[1] - Change management Funny thing is...these are all human problems. [1] - http://en.wikipedia.org/wiki/The_Mythical_Man-Month http://en.wikipedia.org/wiki/The_Mythical_Man-Month

drakaal 13y ago

I didn't see anything that meant it was doomed from the requirements stage. This was more of an Obama love fest and defense of ObamaCare than a technical article describing who the backend was doomed. For what this cost to build it could have been much better, much simpler, and handled the scale gracefully. Yes privacy concerns keep it off the elastic cloud. (not a point mentioned in the article) Yes authenticated users limit caching and make scaling harder. (not really mentioned) But when you are talking about everyone in the US, you only have so many combinations. Income in something like 32 ranges, number of kids 1 through 12 (greater than 12 they are all free) a bunch of ages. We just saw a post that only 15 searches are new to google each day, so the number of combinations is much less than search which does lots of caching. The requirements were messed up, the implementation was. Clearly not built by people who had ever done "Facebook scale" stuff.

GVIrish 13y ago
Well, some reports are that the requirements were changing until 1 week before launch, and that development didn't start until March of this year due to requirements delays. Just those two factors alone would doom just about any project of this scale.
coldcode 13y ago
The primary contractor has had plenty of large gov projects including some for Medicare and SS. Of course so have all large consulting firms and they fail often as well. What the issue is isn't clear from random musings. Unless someone who worked on the project at a high enough level can explain all we have is guesswork.
twistedpair 13y ago
They screwed the pooch, clearly. However I wouldn't assume that the insurance landscape is so simple. You're talking about an insurance industry and legal system that has had a fifty year head start making the process of insurance buying as complicated and obfuscated as possible.

chucknelson 13y ago

Wow, this particular excerpt from the article is stretching things a bit too thin: "Clearly there are problems with the government’s IT procurement processes, but if the same level of dysfunction had been present in the software behind Obama’s political campaign (as it was, apparently, behind Romney’s) the President would have never been reelected." Really? I doubt the campaign websites were the deciding factor. Anyway, a lot of speculation in this article...not very insightful or informative.

ebiester 13y ago
The software in question was the software used to determine the best media buys, the volunteer organization software, and similar that was known to be part of his advantage.
nightpool 13y ago
See http://arstechnica.com/information-technology/2012/11/built-to-win-deep-inside-obamas-campaign-tech/ http://arstechnica.com/information-technology/2012/11/built-... and http://awsofa.info/ http://awsofa.info/ (https://news.ycombinator.com/item?id=5542368 https://news.ycombinator.com/item?id=5542368) as well as http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/ http://arstechnica.com/information-technology/2012/11/inside... (https://news.ycombinator.com/item?id=4764864 https://news.ycombinator.com/item?id=4764864) for comparison.
nightpool 13y ago
See http://arstechnica.com/information-technology/2012/11/built-to-win-deep-inside-obamas-campaign-tech/ http://arstechnica.com/information-technology/2012/11/built-... and http://awsofa.info/ http://awsofa.info/ (https://news.ycombinator.com/item?id=5542368 https://news.ycombinator.com/item?id=5542368) as well as http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/ http://arstechnica.com/information-technology/2012/11/inside... (https://news.ycombinator.com/item?id=4764864 https://news.ycombinator.com/item?id=4764864) for comparison.
skwirl 13y ago
They aren't talking about the campaign websites. Both campaigns had complex internal software systems to help target field worker campaign activities, get out the vote efforts, monitor and respond to turnout data on election day, etc. According to various reports, the Obama campaign was hugely successful in this endeavor [1] while the Romney campaign's election day voter turnout system was a disaster [2]. [1] http://www.drdobbs.com/architecture-and-design/software-development-in-the-obama-campai/240146307 http://www.drdobbs.com/architecture-and-design/software-deve... [2] http://www.businessinsider.com/romney-project-orca-disaster-2012-11 http://www.businessinsider.com/romney-project-orca-disaster-...

Plasmoid 13y ago

Why government software projects go wrong [http://www.mrobinson.ca/2012/09/why-government-software-projects-going.html http://www.mrobinson.ca/2012/09/why-government-software-proj...] ibid - Part 2 [http://www.mrobinson.ca/2013/08/government-projects-going-wrong-part-2.html http://www.mrobinson.ca/2013/08/government-projects-going-wr...]

teyc 13y ago

I bet you half the problem is the software being specified in a synchronous and not asynchronous manner. If a flow chart is expressed synchronously, it immediately bottlenecks around the slow backend systems. e.g. check the applicant has 4 kids, and if that checks out, present them with these options. I attribute some of these problems to inadequate advances in today's mainstream programming languages. For instance, if futures/promises were common parlance in programming, some of these bottlenecks would have been identified earlier. I once read an interesting essay on what future programming languages could work on. One of the interesting ideas is a language that allows programmers to implement undo/redo easily. Recently there was a blog post about fuzzy computing, where a function is able to return approximate values when it is busy. The Knight Capital incident teaches us that if we architected our monitoring systems the same was as intrusion systems are done, we could have stopped activity when suspicious incidents occur outside the normal operating range. There was another incident where a Japanese trader accidentally keyed in the wrong number and promptly lost his employers a lot of money. Languages that incorporate ideas like validation and expected values as first class constructs might be able to help here. The Adriane reminded us how badly humans do when reasoning about systems where multiple agents perform the same task.