7 ms·

Does anyone know what the author meant by this? "Software that assigned identities to enrollees and ensured that they saw only their own personal data, known i

Does anyone know what the author meant by this? "Software that assigned identities to enrollees and ensured that they saw only their own personal data, known internally as the EIdM, was being quickly overwhelmed. Customers could not log in to create accounts." "Experts disagree on what went wrong. But several said that errors in the software code written to stitch the Oracle product into the online system and improperly configured hardware trapped users in endless technological loops. It would take eight days to resolve just that one bottleneck."

adventured 13y ago
Consultant cover-your-ass speak for we set everything up really poorly and had to do it all over again.
- growupkids 13y ago
  Or possibly it's they-paid-us-for-x-and-told-their-bosses-to-expect-y situation.
  - nickff 13y ago
    All branches of government contract out a wide variety of services, the only things they all have in common are cost overruns, delays, and under-performance. It is astonishing that all these different contractors are always to blame, and the government has done nothing wrong. Fool me once, blame the contractor; fool me twice, blame the management.
    - hga 13y ago
      And in this case, the government's Centers for Medicare and Medicaid Services (CMS, a part of HHS) took upon itself the role of integrator, including integration testing. For which they manifestly didn't have what it took, right down to not assigning any one person the full time responsibility for getting it done. Towards the end of October they got "fired" from this role and replaced with QSSI ... who are responsible for the "data hub" and this Oracle EIDM system (which is part of the data hub? Development has been so politicized that they refused to publish architectural diagrams, not even supplying them to the state exchanges).
    - bmelton 13y ago
      Having done it, I can attest that it is indeed very hard to work a government contract to completion, coming in on time and on budget. It isn't impossible, but it is indeed very, very hard, and the general inclination is to fail, through no fault of the contractor. Government workers have interesting employment metrics. They are, for lack of a more profound term, extremely risk averse. If you, as a contractor, propose a new solution to an existing problem, you are looked at like a watch practitioner, or perhaps a voodoo doctor. All claims are treated with a level of skepticism that can only be described as paranoia and sadly, given the general trend on government projects, that isn't necessarily unwarranted. Building a website to service a total user base of 30,000 users is, for most of us I'm sure, pretty easy. That's not exactly a difficult scaling target to hit, and could easily be done on commodity hardware. However, in implementing something similar, I was forced to order 4 Sun Fire boxes, at a cost of over $10k each, to mitigate concerns of speed. Certificate based SSL is also very easy, but we were coerced into purchasing a dedicated SSL module to offload SSL handling to hardware, to mitigate concerns of security, which were then compounded by anecdotal experience that "ssl is slow". So, not only did we have to implement SSL-only for a site with absolutely no public facing endpoint, we had to do so with hardware crypto, that we had to hustle to learn at the last minute -- all reasonable sounding objectives, but compounded with the notion that the contract was a firm-fixed price, bid with different hardware and software SSL, and you can see where the project was trending towards either failure, or a lawsuit after an overrun. The phases of a government contract too, are curious. It starts with skepticism, in which everybody wants the project to die before implementation because its public failure would make everybody look bad, then, if it's somehow able to assuage enough fears to get a go-ahead, you're generally assigned a COTR (Contracts Officer) that ostensibly wants the project to succeed, but would happily wager a failure before implementation than a bigger failure post-implementation. Assuming you can get to the point of actually convincing somebody that the project might actually succeed, and then it might actually deliver good results, the exact opposite happens -- EVERYBODY wants to become a stakeholder, so that they can have their names attached to the success. This has its own faults, because now everybody wants to put their stamp on the project -- which means week-long meetings with various groups of stakeholders all arguing over what image should be used in the page header -- I've had day-long meetings in which the only topic of discussion was "In the picture of the building, the summer photos have too many leaves on the trees, and we can't see the building, but in the winter photos, there are too few leaves, making the scenery look dead... how do we fix this?" Assuming your project hasn't failed yet, you get to a point where now, all the other contractors in the building hate you, and want you to fail. In a normal business, that would never be an issue -- but in the government, the DNS changes are handled by company A, while the active directory is managed by company B. You're sitting there, company C that you are, requesting changes from company A and company B -- both of which will happily obey the exact letter of every request, so that they can indicate full compliance, but which are happy to interpret those letters in such a way that set you up for failure, so that if the project is at this point, you've gotten to the level where the idea is sold, and failure can be blamed on the contractor -- YOU! This means that if it fails, Company A and Company B will swoop in with the exact same idea, only they'll offer their superior experience and intimate knowledge of the project, its failures, and the existing infrastructure to why the project will succeed in their hands. This should be a blog post -- I'll shut up now.
DougWebb 13y ago
The first part sounds like they created a user management system: normally a single table with fields like username, password-hash, real name, address, etc, and sometimes a couple of other tables for groups/roles and cross-ref tables for relationships. In this case the user table would also probably have ids/references to the various government backend systems that the site used to verify your identity, income, etc as you signed up. The rest of it sounds like they botched making this system scalable, to handle both the number of possible users (every adult in the US) and the rate at which they'd be signing up. "Endless technological loops" might refer to either a circular reference in an overly-complex schema, or to normal dependency checking in the schema which takes too long to evaluate within the database when the database is under enormous load.
- VladRussian2 13y ago
  no offense, man, it is just sound like you've never developed or deployed Enterprise Identity and Access Management. I envy you :) The keyword here is "Enterprise" - insanely complex [lets just don't touch whether such complexity is really necessary, it is just the fact of a design to meet any possible RFP in the space :) ] and [in many aspects as a result of the complexity] slow non-scalable products. >The rest of it sounds like they botched making this system scalable these systems just don't scale. In a BigCo with local replication to regional sites it takes noticeable time for such systems to perform. Trying to build such a website - well, it is a government procurement, though i don't think this is a worse one around (Halliburton's and the likes' cost+ in Iraq/Afganistan definitely bigger :). >normally a single table with fields like It is "normal" from another world :) The simple way you described would be done by some startup, and they would horizontally partition it when time come to scale. It is not the way the "Enterprise" things are done.
  - throwaway9848 13y ago
    I would guess the user database is 11NF and has 1000 tables. The UML form of it cannot exist in our limited 4 dimensions of space and time :)
  - makmanalp 13y ago
cobralibre 13y ago
I wonder if "the EIdM" is one or more off-the-shelf enterprise identity management products from Oracle? For example: http://www.oracle.com/technetwork/middleware/id-mgmt/overview/index-098451.html http://www.oracle.com/technetwork/middleware/id-mgmt/overvie...
vellum 13y ago
EIDM is the consolidated Identity and Access Management System in CMS which is one of the largest Oracle 11gR2 Identity and Access Management deployment in the world ... Services available from EIDM have been grouped into four main services areas (registration service, authorization service, ID lifecycle management service, and access management service). [1] Optum/QSSI used the EIDM to make accounts. Their part of the system crashed, and they blamed it on the feds, for making people register to compare plans.[2] This reddit post describes the infinite loop: i'm now stuck in some infinite loop where it can't identify me, asks me more questions, verify fails, repeat. [3] 1 - http://www.civicagency.org/2013/10/learning-from-the-healthcare-gov-infrastructure/ http://www.civicagency.org/2013/10/learning-from-the-healthc... 2 - http://www.inforisktoday.com/obamacare-website-security-questioned-a-6171/op-1 http://www.inforisktoday.com/obamacare-website-security-ques... 3 - http://www.reddit.com/r/AskReddit/comments/1oq0ds/has_anyone_successfully_signed_up_for_obamacare/ http://www.reddit.com/r/AskReddit/comments/1oq0ds/has_anyone...

adventured 13y ago

Consultant cover-your-ass speak for we set everything up really poorly and had to do it all over again.

growupkids 13y ago
Or possibly it's they-paid-us-for-x-and-told-their-bosses-to-expect-y situation.
- nickff 13y ago
  All branches of government contract out a wide variety of services, the only things they all have in common are cost overruns, delays, and under-performance. It is astonishing that all these different contractors are always to blame, and the government has done nothing wrong. Fool me once, blame the contractor; fool me twice, blame the management.
  - hga 13y ago
    And in this case, the government's Centers for Medicare and Medicaid Services (CMS, a part of HHS) took upon itself the role of integrator, including integration testing. For which they manifestly didn't have what it took, right down to not assigning any one person the full time responsibility for getting it done. Towards the end of October they got "fired" from this role and replaced with QSSI ... who are responsible for the "data hub" and this Oracle EIDM system (which is part of the data hub? Development has been so politicized that they refused to publish architectural diagrams, not even supplying them to the state exchanges).
  - bmelton 13y ago
    Having done it, I can attest that it is indeed very hard to work a government contract to completion, coming in on time and on budget. It isn't impossible, but it is indeed very, very hard, and the general inclination is to fail, through no fault of the contractor. Government workers have interesting employment metrics. They are, for lack of a more profound term, extremely risk averse. If you, as a contractor, propose a new solution to an existing problem, you are looked at like a watch practitioner, or perhaps a voodoo doctor. All claims are treated with a level of skepticism that can only be described as paranoia and sadly, given the general trend on government projects, that isn't necessarily unwarranted. Building a website to service a total user base of 30,000 users is, for most of us I'm sure, pretty easy. That's not exactly a difficult scaling target to hit, and could easily be done on commodity hardware. However, in implementing something similar, I was forced to order 4 Sun Fire boxes, at a cost of over $10k each, to mitigate concerns of speed. Certificate based SSL is also very easy, but we were coerced into purchasing a dedicated SSL module to offload SSL handling to hardware, to mitigate concerns of security, which were then compounded by anecdotal experience that "ssl is slow". So, not only did we have to implement SSL-only for a site with absolutely no public facing endpoint, we had to do so with hardware crypto, that we had to hustle to learn at the last minute -- all reasonable sounding objectives, but compounded with the notion that the contract was a firm-fixed price, bid with different hardware and software SSL, and you can see where the project was trending towards either failure, or a lawsuit after an overrun. The phases of a government contract too, are curious. It starts with skepticism, in which everybody wants the project to die before implementation because its public failure would make everybody look bad, then, if it's somehow able to assuage enough fears to get a go-ahead, you're generally assigned a COTR (Contracts Officer) that ostensibly wants the project to succeed, but would happily wager a failure before implementation than a bigger failure post-implementation. Assuming you can get to the point of actually convincing somebody that the project might actually succeed, and then it might actually deliver good results, the exact opposite happens -- EVERYBODY wants to become a stakeholder, so that they can have their names attached to the success. This has its own faults, because now everybody wants to put their stamp on the project -- which means week-long meetings with various groups of stakeholders all arguing over what image should be used in the page header -- I've had day-long meetings in which the only topic of discussion was "In the picture of the building, the summer photos have too many leaves on the trees, and we can't see the building, but in the winter photos, there are too few leaves, making the scenery look dead... how do we fix this?" Assuming your project hasn't failed yet, you get to a point where now, all the other contractors in the building hate you, and want you to fail. In a normal business, that would never be an issue -- but in the government, the DNS changes are handled by company A, while the active directory is managed by company B. You're sitting there, company C that you are, requesting changes from company A and company B -- both of which will happily obey the exact letter of every request, so that they can indicate full compliance, but which are happy to interpret those letters in such a way that set you up for failure, so that if the project is at this point, you've gotten to the level where the idea is sold, and failure can be blamed on the contractor -- YOU! This means that if it fails, Company A and Company B will swoop in with the exact same idea, only they'll offer their superior experience and intimate knowledge of the project, its failures, and the existing infrastructure to why the project will succeed in their hands. This should be a blog post -- I'll shut up now.

DougWebb 13y ago

The first part sounds like they created a user management system: normally a single table with fields like username, password-hash, real name, address, etc, and sometimes a couple of other tables for groups/roles and cross-ref tables for relationships. In this case the user table would also probably have ids/references to the various government backend systems that the site used to verify your identity, income, etc as you signed up. The rest of it sounds like they botched making this system scalable, to handle both the number of possible users (every adult in the US) and the rate at which they'd be signing up. "Endless technological loops" might refer to either a circular reference in an overly-complex schema, or to normal dependency checking in the schema which takes too long to evaluate within the database when the database is under enormous load.

VladRussian2 13y ago
no offense, man, it is just sound like you've never developed or deployed Enterprise Identity and Access Management. I envy you :) The keyword here is "Enterprise" - insanely complex [lets just don't touch whether such complexity is really necessary, it is just the fact of a design to meet any possible RFP in the space :) ] and [in many aspects as a result of the complexity] slow non-scalable products. >The rest of it sounds like they botched making this system scalable these systems just don't scale. In a BigCo with local replication to regional sites it takes noticeable time for such systems to perform. Trying to build such a website - well, it is a government procurement, though i don't think this is a worse one around (Halliburton's and the likes' cost+ in Iraq/Afganistan definitely bigger :). >normally a single table with fields like It is "normal" from another world :) The simple way you described would be done by some startup, and they would horizontally partition it when time come to scale. It is not the way the "Enterprise" things are done.
- throwaway9848 13y ago
  I would guess the user database is 11NF and has 1000 tables. The UML form of it cannot exist in our limited 4 dimensions of space and time :)
- makmanalp 13y ago

cobralibre 13y ago

I wonder if "the EIdM" is one or more off-the-shelf enterprise identity management products from Oracle? For example: http://www.oracle.com/technetwork/middleware/id-mgmt/overview/index-098451.html http://www.oracle.com/technetwork/middleware/id-mgmt/overvie...

vellum 13y ago

EIDM is the consolidated Identity and Access Management System in CMS which is one of the largest Oracle 11gR2 Identity and Access Management deployment in the world ... Services available from EIDM have been grouped into four main services areas (registration service, authorization service, ID lifecycle management service, and access management service). [1] Optum/QSSI used the EIDM to make accounts. Their part of the system crashed, and they blamed it on the feds, for making people register to compare plans.[2] This reddit post describes the infinite loop: i'm now stuck in some infinite loop where it can't identify me, asks me more questions, verify fails, repeat. [3] 1 - http://www.civicagency.org/2013/10/learning-from-the-healthcare-gov-infrastructure/ http://www.civicagency.org/2013/10/learning-from-the-healthc... 2 - http://www.inforisktoday.com/obamacare-website-security-questioned-a-6171/op-1 http://www.inforisktoday.com/obamacare-website-security-ques... 3 - http://www.reddit.com/r/AskReddit/comments/1oq0ds/has_anyone_successfully_signed_up_for_obamacare/ http://www.reddit.com/r/AskReddit/comments/1oq0ds/has_anyone...