5 ms·

Apparently last year's major failure was caused by their Oracle data warehouse going down hard: http://foia.state.gov/_docs/PIA/ConsularConsolidatedDatabase_CC

Apparently last year's major failure was caused by their Oracle data warehouse going down hard: http://foia.state.gov/_docs/PIA/ConsularConsolidatedDatabase_CCD.pdf http://foia.state.gov/_docs/PIA/ConsularConsolidatedDatabase... "The Consular Consolidated Database (CCD) is one of the largest Oracle based data warehouses in the world that holds current and archived data from the Consular Affairs (CA) domestic and post databases around the world. As of December 2009, it contains over 100 million visa cases and 75 million photographs, utilizing billions of rows of data, and has a current growth rate of approximately 35 thousand visa cases every day" Unclear what's gone wrong this time, but the mention of "biometrics" (like photographs) makes me suspect it's the same system.

jpatokal 11y ago
An interview with the director responsible for the system, props to @kalleboo for finding this: http://fcw.com/Articles/2014/10/20/State-Department-database-crash-1.aspx?Page=all&p=1 http://fcw.com/Articles/2014/10/20/State-Department-database... Typical quote: "We knew we could run it on one node. We needed to have one very powerful node."
- BinaryIdiot 11y ago
  > Typical quote: "We knew we could run it on one node. We needed to have one very powerful node." As someone who has done government contracting...it's always alarming when this path is suggested. The last time was a beefy server with about 50TB of ram to keep all data in memory with multiple hard drives to keep backups. Ugh.
  - codecamper 11y ago
    The WOPR? How about a nice game of chess?
  - crdb 11y ago
    Mind talking more about what the challenges in running a 50TB relational database? I and I'm sure many others would be very interested in a blog post or long-ish comment with anecdotes and lessons learnt...
    - BinaryIdiot 11y ago
      > Mind talking more about what the challenges in running a 50TB relational database? I wish I could but it wasn't my direct project so I wasn't part of the team implementing it. I'm not even sure it was successfully delivered.
- marcosdumay 11y ago
  One of the problems with proprietary databases is that licensing issues are another barrier to creating proper clusters (and to have equal environments on development, testing and production).
boca 11y ago
They did mention it's not the same thing this time. "This is not the same problem we had with the CCD last year, which was a problem with the database caused by a software patch. This is a hardware failure, and we are working to restore system functions."
- jpatokal 11y ago
  If you read that carefully, that does not preclude it from being a different problem with the CCD.
cm2187 11y ago
A Visa application system doesn't strike me as an obvious candidate for a relational database. I would rather store all the information related to a visa application in a document store with a relational database only used as a sort of index. I wonder if lots of systems are not built using relational just out of habit. And then bump into these problems.
- andreasklinger 11y ago
  I assume the technical fundaments are older than document databases were popular in enterprise and reliable on scale
- icebraining 11y ago
  Say you're the tech lead for the project, built before 2001 [1], and you need to hire a vendor that can provide a cluster that won't break with 100TB+ and guarantee long term support, possibly for decades. Who would you call? [1] http://www.law.umaryland.edu/marshall/crsreports/crsdocuments/RL31512_02112004.pdf http://www.law.umaryland.edu/marshall/crsreports/crsdocument...
  - bigiain 11y ago
    I don't remember the exact timeframes, but I suspect Digital and Sun Microsystems were likely looking choices around that time. (Sun was the default choice for Telco billing systems around then...)
  - needusername 11y ago
    Why do you need a cluster?
    - jpatokal 11y ago
      Back in 2001, 100TB was massive. The first 1 TB HDD didn't roll out until 2007.
      - needusername 11y ago
notNow 11y ago
What happened to "No one was fired for using Oracle or Microsoft" meme?
- dagw 11y ago
  The meme doesn't imply that those products are good or never break, just that you won't get fired for choosing them. We'll have to wait and see if anybody involved in selecting Oracle gets fired over this.
- dotBen 11y ago
  I think it was "no one was fired for choose IBM" :p
  - notNow 11y ago
    and IBM as well, true :)
dataker 11y ago
I'd assume Oracle would still have a century long contract to ensure they still have a monopoly.

jpatokal 11y ago

An interview with the director responsible for the system, props to @kalleboo for finding this: http://fcw.com/Articles/2014/10/20/State-Department-database-crash-1.aspx?Page=all&p=1 http://fcw.com/Articles/2014/10/20/State-Department-database... Typical quote: "We knew we could run it on one node. We needed to have one very powerful node."

BinaryIdiot 11y ago
> Typical quote: "We knew we could run it on one node. We needed to have one very powerful node." As someone who has done government contracting...it's always alarming when this path is suggested. The last time was a beefy server with about 50TB of ram to keep all data in memory with multiple hard drives to keep backups. Ugh.
- codecamper 11y ago
  The WOPR? How about a nice game of chess?
- crdb 11y ago
  Mind talking more about what the challenges in running a 50TB relational database? I and I'm sure many others would be very interested in a blog post or long-ish comment with anecdotes and lessons learnt...
  - BinaryIdiot 11y ago
    > Mind talking more about what the challenges in running a 50TB relational database? I wish I could but it wasn't my direct project so I wasn't part of the team implementing it. I'm not even sure it was successfully delivered.
marcosdumay 11y ago
One of the problems with proprietary databases is that licensing issues are another barrier to creating proper clusters (and to have equal environments on development, testing and production).

boca 11y ago

They did mention it's not the same thing this time. "This is not the same problem we had with the CCD last year, which was a problem with the database caused by a software patch. This is a hardware failure, and we are working to restore system functions."

jpatokal 11y ago
If you read that carefully, that does not preclude it from being a different problem with the CCD.

cm2187 11y ago

A Visa application system doesn't strike me as an obvious candidate for a relational database. I would rather store all the information related to a visa application in a document store with a relational database only used as a sort of index. I wonder if lots of systems are not built using relational just out of habit. And then bump into these problems.

andreasklinger 11y ago
I assume the technical fundaments are older than document databases were popular in enterprise and reliable on scale
icebraining 11y ago
Say you're the tech lead for the project, built before 2001 [1], and you need to hire a vendor that can provide a cluster that won't break with 100TB+ and guarantee long term support, possibly for decades. Who would you call? [1] http://www.law.umaryland.edu/marshall/crsreports/crsdocuments/RL31512_02112004.pdf http://www.law.umaryland.edu/marshall/crsreports/crsdocument...
- bigiain 11y ago
  I don't remember the exact timeframes, but I suspect Digital and Sun Microsystems were likely looking choices around that time. (Sun was the default choice for Telco billing systems around then...)
- needusername 11y ago
  Why do you need a cluster?
  - jpatokal 11y ago
    Back in 2001, 100TB was massive. The first 1 TB HDD didn't roll out until 2007.
    - needusername 11y ago

notNow 11y ago

What happened to "No one was fired for using Oracle or Microsoft" meme?

dagw 11y ago
The meme doesn't imply that those products are good or never break, just that you won't get fired for choosing them. We'll have to wait and see if anybody involved in selecting Oracle gets fired over this.
dotBen 11y ago
I think it was "no one was fired for choose IBM" :p
- notNow 11y ago
  and IBM as well, true :)

dataker 11y ago

I'd assume Oracle would still have a century long contract to ensure they still have a monopoly.