4 ms·

> particular technologies being used, including something called MarkLogic, an XML database XML database ? For a critical service used by 300 M people? Brrrrr

> particular technologies being used, including something called MarkLogic, an XML database XML database ? For a critical service used by 300 M people? Brrrrrr.....

glaucon 3y ago
It was around high-tide of 'no-sql' but surely it cannot literally be true ? I suspect business rules in XML, so effectively read-only, while dynamic data in a rdbms ? Maybe that's just my hopeful heart talking :-)
- ajcp 3y ago
  I had the same thought
- mdickers47 3y ago
  nope it was exactly that; transactional customer state in giant xml documents in an xml-document-storing database. Bugs would cause them to grow to 10, 20, 100MB.
  - glaucon 3y ago
    Wow, strange stuff, indeed.
    - deleted 3y ago
      [deleted]
  - qohen 3y ago
    According to Paul Smith, who has an AMA about healthcare.gov at Lobster.rs[0], the database was MarkLogic, which started out as an XML database, according to its Wikipedia page[1] (Paul also mentions this briefly his 10th anniversary post). According to another commenter[2], who says he met a MarkLogic manager on a plane flight, MarkLogic had passed some sort audit/certification that only Oracle had previously passed, which may explain why it was used for healthcare.gov. [0] https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_gov#c_0fu122 https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_go... [1] https://en.wikipedia.org/wiki/MarkLogic https://en.wikipedia.org/wiki/MarkLogic [2] https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_gov#c_f2khfm https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_go...
  - heelix 3y ago
    Was working for MarkLogic and got pulled into this a couple days before launch. That sort of thing was the least of their worries. They got the model wrong. With ACA, you were not allowed to do policy pricing on preexisting conditions. Age, sex, smoking, and location. That really only leaves one variable - location - for the actuarial folks to create unique policies. You might suspect how this breaks things. A healthy 18yo female in Tampa could have hundreds of thousands of policies. Live on a superfund site... hey, about those 3 spendy options. Those files could be massive - hundreds of MB. The log files... ye gods... those things grew in GB fast. The other gotcha was Java is single inheritance. Foo > XML. Bar extends Foo > XML. Baz extends Foo > /dev/null the Bar bits in the persistence layer. The MarkLogic bits did OK. When everything went sideways at launch, almost no traffic was getting all the way to the back end. Because things were getting profiled, there was way more time to tune and tweak the indexes as they tried to re-engineer things. The NoSQL bits helped there, as it made it a bit easier to morph things as it was refactored. Not to say there were not pain points or they did not have a silly amount of hardware in it in the end. One of the more memorable things for me was working in the XOC, having an update go sideways in the early AM, and seeing CNN pop up a ticker saying ACA was down. Good times.
ajcp 3y ago
I doubt that was the main datastore. It might have been a configuration database though, which I can see it being an issue given that it sounds like it was business logic causing the issues.
- heelix 3y ago
  It was. The way they built it out was UML > generated Java, which persisted data as XML. The MarkLogic database could then do xpath/queries (think stored procedures) for the Java layer - so not just POJO > XML, though there was some of that too. It ended up being a non-trivial cluster.
  - ajcp 3y ago
    Your explanation made me dry-heave. UML > Java generation explains why their deployments seemed to contain obscene amounts of code that no one understood.
- hugs 3y ago
  You would be surprised. :) Source: I was also a member of the Tech Surge. (Paul Smith recruited me to help with testing.)

> particular technologies being used, including something called MarkLogic, an XML database XML database ? For a critical service used by 300 M people? Brrrrrr.....

glaucon 3y ago

It was around high-tide of 'no-sql' but surely it cannot literally be true ? I suspect business rules in XML, so effectively read-only, while dynamic data in a rdbms ? Maybe that's just my hopeful heart talking :-)

ajcp 3y ago
I had the same thought
mdickers47 3y ago
nope it was exactly that; transactional customer state in giant xml documents in an xml-document-storing database. Bugs would cause them to grow to 10, 20, 100MB.
- glaucon 3y ago
  Wow, strange stuff, indeed.
  - deleted 3y ago
    [deleted]
- qohen 3y ago
  According to Paul Smith, who has an AMA about healthcare.gov at Lobster.rs[0], the database was MarkLogic, which started out as an XML database, according to its Wikipedia page[1] (Paul also mentions this briefly his 10th anniversary post). According to another commenter[2], who says he met a MarkLogic manager on a plane flight, MarkLogic had passed some sort audit/certification that only Oracle had previously passed, which may explain why it was used for healthcare.gov. [0] https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_gov#c_0fu122 https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_go... [1] https://en.wikipedia.org/wiki/MarkLogic https://en.wikipedia.org/wiki/MarkLogic [2] https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_gov#c_f2khfm https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_go...
- heelix 3y ago
  Was working for MarkLogic and got pulled into this a couple days before launch. That sort of thing was the least of their worries. They got the model wrong. With ACA, you were not allowed to do policy pricing on preexisting conditions. Age, sex, smoking, and location. That really only leaves one variable - location - for the actuarial folks to create unique policies. You might suspect how this breaks things. A healthy 18yo female in Tampa could have hundreds of thousands of policies. Live on a superfund site... hey, about those 3 spendy options. Those files could be massive - hundreds of MB. The log files... ye gods... those things grew in GB fast. The other gotcha was Java is single inheritance. Foo > XML. Bar extends Foo > XML. Baz extends Foo > /dev/null the Bar bits in the persistence layer. The MarkLogic bits did OK. When everything went sideways at launch, almost no traffic was getting all the way to the back end. Because things were getting profiled, there was way more time to tune and tweak the indexes as they tried to re-engineer things. The NoSQL bits helped there, as it made it a bit easier to morph things as it was refactored. Not to say there were not pain points or they did not have a silly amount of hardware in it in the end. One of the more memorable things for me was working in the XOC, having an update go sideways in the early AM, and seeing CNN pop up a ticker saying ACA was down. Good times.

ajcp 3y ago

I doubt that was the main datastore. It might have been a configuration database though, which I can see it being an issue given that it sounds like it was business logic causing the issues.

heelix 3y ago
It was. The way they built it out was UML > generated Java, which persisted data as XML. The MarkLogic database could then do xpath/queries (think stored procedures) for the Java layer - so not just POJO > XML, though there was some of that too. It ended up being a non-trivial cluster.
- ajcp 3y ago
  Your explanation made me dry-heave. UML > Java generation explains why their deployments seemed to contain obscene amounts of code that no one understood.
hugs 3y ago
You would be surprised. :) Source: I was also a member of the Tech Surge. (Paul Smith recruited me to help with testing.)