7 ms·
How Healthcare.gov Changed the Software Testing Conversation
- diminoten 13y agoI chuckled when we had the president of the United States of America announcing software delays on national television.
- larrys 13y agoAnything that is done by a public official at that level (not just the President) is well thought out and implemented after much discussion to achieve a particular goal. You can reverse engineer it and draw your conclusions as to why they said what they said. In this case my interpretation is that it's to shift the blame a bit and also to appear to be out in front of the issue and let people know that "things are being handled help is on the way and we're on it". An analogy I would use is this. Let's say you are hosted somewhere (say Rackspace or AWS) and you lose connectivity and your site is down. Better for RS or AWS to "appear to be on top of it" (whether they are or not) and to be "doing something" than to have this black box of "hey exactly what is going on and when will it be fixed". The anxiety is what kills you. Once you know that at least (using another example) "a spare airplane is on the way but it is currently delayed by a snow storm that should clear in 2 hours" you are feeling much better than if you don't have any answers at all.
- larrys 13y ago"School children and grandparents alike are now worrying about whether their passwords are being passed in the clear now. Imagine that." Even if they are "worrying" (and I really doubt that they are actually) they would only be worrying the same way they do about flying right after a major plane accident. That anxiety tends to go away very quickly.
- lstamour 13y agoYep. With every password breach the word gets out that you should have unique, strong passwords for each site. And yet with every breach we learn of more shared, common passwords. Not to mention how little attention most pay to entering a password on a site without SSL. Credit cards they'll worry about but passwords, less so. And imagine explaining session cookie hijacking as we tried to when Firesheep launched? ;-) Once the big players launched with SSL, problem "solved". That said, most do now secure their WiFi, so the public can learn, and I think has learned from healthcare.gov.
- larrys 13y ago"That said, most do now secure their WiFi" Then there is the linksys wifi camera which you can access over the net but doesn't have https. So in order to get to it securely I have to ssh in to a machine on the network with a tunnel and connect by proxy. A real pain.
- lstamour 13y agoOr the devices you don't check. In my case, I've yet to intercept traffic from a Dropcam to see if it's SSL'd correctly everywhere. Let's not start on the alert screenshots it emails me....
- jeffdavis 13y agoWith so many problem's, I doubt it's just an issue of testing. They probably have serious design issues and other engineering problems.
- yggdrasil 13y agoExactly. The cardinal rule of competent software development is that you don't test quality into the software. The root causes were requirements churn, split development, and no project coordinator. Those would be nearly insurmountable on an average size project but were hopeless on this project. It's a marvel they were able to stabilize it to the point it's at.
- danso 13y agoSoftware developers should better appreciate the uniqueness of their testing situation: like any other field, writing, maintaining, and executing takes time. But in no other field that I can think of is testing so relatively cost-free to continue doing..."regression" testing is either not possible or feasible elsewhere. The ability to automate and near effortlessly run tests with every code change has a fundamental impact. To give a non-programming example...journalists make silly mistakes all the time about dates and misspelled names. It's not always, or even usually, because the reporter is incompetent. Sometimes they hurriedly typed in a fact from the top of their head and forgot to put a "TODO" near it (some in-house CMSes do not make meta-comments easy). Or, just as frequently, something got changed as the text moved from one editor's desk to another...there's pretty much no such thing as diff software, which is fundamentally different than keeping revisions. Spellcheck can help detect and auto-fix some problems. But generally, you need to manually proof-read things to verify them, and at some point, you just assume that no one is going to change what you've verified, and then you press "Publish". It's not that continuous-proofing isn't possible, it's just not feasible. It can be frustrating working with people who think testing software is to keep "proofreading" it over and over again...user testing is vital, obviously, but I'm talking about people who test the wrong, already-verified things, and then sap their energy not checking for other variations, and this is understandable and very human, of course. I guess the big picture to understand is that screwups are frequent in every field, all the time. The best surgeons forget to wash their hands...not because they're idiots, but because emergency surgery will cause all kinds of things to go haywire, including basic procedure. And obviously, very basic mistakes can occur in production code. But as programmers, we uniquely benefit from what we can do to prevent that. And this superpower of ours is something that I wish was more conveyable to the greater world.
- jeffdavis 13y agoAs a counterpoint, software development arguably suffers most from the combinatorial explosion of behaviors. In other words, testing software is easier but less effective (or at least less complete).
- powertower 13y agoI tried to get coverage on the 22nd. The process is very long and tedious. I spent a solid 5 hours going through the entire process about 6 or 7 times, from start to finish, only to get told each time afterwards that it's not complete for some odd reason, and to start over again on step #1! I tried multiple things, deleting the application, using Chrome instead of IE, etc. On the 23rd you could not even log in - they took that option away. What bothered me the most was, besides your social security number and a slew of personal information, they even demand that if you are a naturalized or derived citizen, you locate your naturalization or citizenship certificate and enter numbers (Alien # and Cert #) from it that -- get this - their Javascript refuses to validate (it kind of looked like their rule match is off by a digit). And you get stuck on that step. I also noticed that if you enter your income as below $10,000, the system tells you that you qualify for no benefits. But the moment you enter $11,500, you get a $260 tax credit. Go figure that one out.
- bmelton 13y ago> I also noticed that if you enter your income as below $10,000, the system tells you that you qualify for no benefits! I know that the common plight there is that some are poor enough that they qualify for Medicare vs. Obamacare, though I admittedly don't know what the distinction is. If you need insurance, and Healthcare.gov is telling you you're too poor, it might simply be failing to tell you to go check with Medicare and see if you're eligible there.
- powertower 13y agoNo. It was rather - Enter $10,000 - you get no credit. Enter $10,500 - you get big credit. Enter $11,000 - you get no credit. Enter $11,500 - you get big credit. Etc.
- bmelton 13y agoAh. I misread then. That's... odd, to say the least. About in line with the expectation though, I suppose.
- 13y ago
- a3n 13y agoInteresting post, probably true enough, but I don't think testing is the major problem. The law itself was rushed through, to the point that the President and members who voted either way didn't really know what was in it. It was huge, and rushed because they had to beat the clock of the potential coming of a hostile Republican House. Even champions of the bill in Congress admitted that it would take years to fully understand what the bill does. The mind reels. The law has many hostile stakeholders, and many turf grabbers, and the real requirements were not managed by a single entity who wanted to get it done the best and most economical way possible. Members of Congress (keepers of the purse) were and are fighting it at every turn. Government agencies had to have a piece of it, providing data for deciding customer eligibility and policy level; law enforcement and immigration concerns play a part. The requirements were a result of compromise (in the political sausage making sense) and turf dominance, some of which are part of any project, but at a hyper level when government agencies are involved. There was probably no technical adult in the room telling political people that this and that just can't be done in any economical and working way. The government does not normally provide web sites to manage and coordinate anything between government, citizens and corporations on this scale. The most you'll usually see, from the outside as a citizen/consumer, is listing of information in text or pdf format, possibly filling in a form to make an appointment or communicate with a representative. They don't know what they're doing, in a profound sense and on a grand scale. The government just doesn't have experience managing and running such a site with such requirements. They're trying to do something like Facebook in its current incarnation right from the start. It's like never having gone to the moon and strapping some astronauts on the top of an ICBM and hoping for the best, which is a formual to "fail fast" in the worst sense. Because they didn't know what they were doing, and because it was probably thought of as just another IT contract effort, they followed standard procurement practices and used standard ("we know them") vendors. And that failed miserably. And because of all of the above, and general inexperience for such a project, and relatively unlimited funds to get it as wrong as possible, they started actual specification and implementation way, way late, with the added complication that the possibility of a time overrun was just not there, because of a hard political deadline. It's no wonder that testing was done so poorly and ineptly, they just had no time at all.
- jeffdavis 13y ago
- dannyrosen 13y agoIt should be noted that testing consists of such a wide array of tasks that it's often as hard to find the right people than it is to identify the right type of testing they should perform. While performance and load testing on Healthcare.gov should be obvious tasks we get into murky territory when asking questions like: Does the UI need to be tested more than the APIs that power it? Will the value of automated end-to-end testing be deprecated after a UI refresh takes place? (Will there be a UI refresh?) How much time should be put into scalable UI tests? Are automated smoke tests enough? I could go on, but the question I'd like answered is what did the actual testing plan consist of?
- undoware 13y ago"Hey Frank, you test this?" "Naw, Larry, government contract." "Shit, I think this one's important though." "We'll fix it later. It's easier to beg forgiveness than permission to change spec."
- robomartin 13y agoThinking that this is about TDD or software management is to grotesquely miss reality. This whole fiasco has demonstrated, to an incredible level of clarity, why we've reached a point where government is too big to work. This has nothing whatsoever to do with who's President or which party controls what. This is a simple case of an entity that has grown so large, complex, ignorant and bureaucratic that it simply can't figure out how to produce anything useful, from laws to websites. The difference this time around is that this has been very public. This is a problem that forces people to pay attention and get involved. Healthcare affects everyone directly and people care about it. Most of us who have understood the devolution of government over the years have recognized the incompetence of the organization as a whole for quite some time. Some have been more vocal than others in trying to highlight the issue. It's a difficult position to hold because it is relentlessly attack by those who, through indoctrination or religious-like following, stick to their respective parties and simply won't even admit they are being screwed despite mounting evidence to the contrary. The President lies about healthcare to the country and the world dozens of times and Liberal media contorts itself to try to figure out a way to spin it into some alternate reality that makes sense. We had exactly the same kind of thing happen with Bush and his wars, but this is about the ACA. We let them lie. Some of us see it and call them on it while others shoot us down based on party loyalty and continue to support the effectively criminal behavior. The difficulty in gaining mass awareness for these kinds of problems has been in that most people, at the end of the day, couldn't care less about what's going on inside the sausage-making factory. They are too busy trying to earn a living and going through their daily lives. There are things in most people's lives that are far more interesting to them than what government does and how it works. True to this I think it is fair to say that most US voters are utterly uninformed and get their opinions (and voting decisions) from the media --mostly TV. Now things are different. This is something that is important to everyone. Health, food and housing are top-level concerns for everyone. This ACA/Obamacare mess is achieving something no government critic could possibly buy for any amount of money: Bring to the forefront the incompetence, waste and mismanagement that has become part and parcel of what our government has been about for years. Everything in our government is done this way. Everything. You just don't see it or don't care to dig into it for other issues. Healthcare just made it first page news for everyone. Everyone now sees how the sausages are being made. There is no way to hide it. Everyone can now balance the equation of what politicians said and promised them against what they've actually delivered. A family who's healthcare premiums doubled, who's deductible skyrocketed and who lost the ability to see their doctors at their hospital has no way to satisfy a promise of lower premiums, better coverage and keeping what you like. I still remember Los Angeles Mayor Villaraigosa deciding to change the outcome of a vote in a very public way right in front of the cameras while a woman came over to him and said "Let them do what they are going to do", all of it picked-up by the microphones and cameras. There is no shame or respect any more. Politicians know they can get away with murder because there are no consequences for such things as publicly and visibly changing a vote or spewing out lies. Hopefully people are starting to think about just how ridiculous it is to give a blank check or a pass to anyone in government. I also hope they are starting to become more convinced that if we don't make those who lie to us responsible for their lies we are never going to improve government. If it had not been for Snowden who in government would have told us what was going on with the NSA? They've been lying to us for years and they would have continued to do so had it not been for him. They lied knowing there are no measurable consequences for their actions. At worst they move elsewhere in government and live goes on. It's a joke. Imagine if you were involved in a contract negotiation for you business, signing the papers for your home purchase, auto purchase or a lease rental agreement and the other party said something akin to "Don't worry. We have to sign the contract to see what's in it". How fast would you run from that one? You would, right? Then, why is it that we let those in government play such games? By extension they --the governing class-- over time, start feeling they can do anything, to anyone at any time. I would not be surprised if this is the sort of thing that leads to mass surveillance programs not being given a second thought within that community. If you can get away with murder you keep murdering, plain and simple. The solution, I am afraid, is not technological, it's political. I've said this before: If you are in tech you need to be smart about politics because bad government and bad policy can be more destructive to your efforts than bad code. The NSA fiasco alone will probably cost US companies billions. Trust is a very difficult thing to regain, whether it is with your girlfriend, boyfriend, wife, husband, company or country. Once trust is broken it takes orders of magnitude more work to regain it. If ever. So, there you are, becoming a highly skilled technologist thinking that government and political issues are of no importance to your mission, only to find out that your international trust and market potential keeps getting boned by ignorant fools in government who really have no clue what they are doing. Government is almost the only entity that can completely destroy your startup with the stroke of an ignorant pen. Look at the startups fighting against the transportation and housing unions, rules and regs. How different would things be in terms of competition and progress if those barriers where not there? Bottom line: You have to be informed and get involved. You have to fight against indoctrination to see what reality looks like. Stop taking other's opinions as your own. Question everything, research, critically observe, measure and use your CS and mathematical skills to model. In other words, be the high information citizen. The devil isn't in the details, it's in politics.
- evolve2k 13y agoAnyone have a link to the github repo she mentions? Also was there an announcement of the open sourcing of code that I missed (been travelling and off the grid for a while).
- Datsundere 13y agoI'm not sure what people that earn below the required threshold do if their state isn't expanding on medicaid. How do they apply for medicaid?
- yeukhon 13y agoI am just an undergraduate, so what the heck do I know about agile in big project. Please feel free to critique. What do guys from big projects have to say? Here is how I imagine people arguing about agile vs water fall. WF: It probably will take the same amount of hours and people to hash system together in agile, making sure all components are integrated properly without a strong and detail system requirements. AG: But you don't wait until the end to test your software and wait until the end to discover shit and changes. You can iteratively change stuff. WF: Ah. Fine. I will just tell my co-workers to test more often. Now, you agile people just want everything to be short-term. How do you ensure people don't block other teams from completing works? you see when blockers appear, the time requires for completion is the same as everything done at once. AG: But if you finish 80% while waiting on the 20% to block forever, you still have 80% done. Plus, in agile you have some short term and long term goals. You order them by importance and blocker. When the whole sprint is blocked, just do more testing and making sure existing systems are working as said. WF: Fine. Now what about people going monkey on their own? The point of water fall is to ensure no one is outside the design and minimize risk. Now everyone goes agile and shit going to come because some teams are not playing nice. AG: This is a tough problem. But the only way out is by scrum, well, at least meeting daily or weekly. The benefit of agile plus regular meeting is that everyone knows what the heck is going on and changes can be made. Once you made a specification forever, it's hard to change. WF: But it is the same in agile that a chance can be tough to make because people didn't make the right decision at one of the early sprints. AG: Yeah. That certainly is. I won't lie agile people don't make that easier. But I argue it is easier if you just work on smart parts. WF: Are you sure it will work for multiple-billion dollars project? With a lot of legal blockers and a lot of other software to work with? You need to do a thorough analysis. You need to conduct research how existing healthcare registration works, find out the pros and cons. That's a lot of time. Now if you go agile, you will just keep doing research. AG: .... YEUKHON: I will fill in the blanks, but hey what the heck do I know about big project software engineering. I think agile doesn't require people to start right away. Agile just ensure nothing blocks forever and that goals are more or less short-terms. it is always required one to have good knowledge of how existing solutions works. So if you need to find out how to integrate the new healtcare.gov with the rest of the solutions out there, do that research first. Obviously you can't just develop a site like a baby punching keyboard, hoping the baby compose some legendary music. It takes real talent to do that. So I say myth buster, agile still requires professionals to do priori research, have lengthy discussions about how to go about implementing the system, give a pretty good sketch about how things supposed to work, and then go off and start implementing something. But don't come up with the full solution so detail that you ought to obey that contract. WF: Yeah, but your client is spending 100 millions and this is impacting tens of millions of people. Any example out there like healthcare.gov did finish on time and running successfully using agile? I know WF sucks, but real examples of agile out there? You know, people tend to do a lot of talking and sketching when they are working with so many people and so many things involved. "Oh Bob and his teams are doing research on existing solutions works and my team is working on consumer APIs? Okay, let's talk about what fields we want, what schema we want in the database, etc"