4 ms·

Does anyone else think that, all conspiracy theory aside, the information published is only partly true? If it's a purely hw issue, say a piece of highly speci

Does anyone else think that, all conspiracy theory aside, the information published is only partly true? If it's a purely hw issue, say a piece of highly specialized hardware (hw crypto, piece of old mainframe, etc), it can take a very long time to source it (although a SPOF is surprising in such a critical infrastructure), but it does not take 100 people to work on it, 24/7. It requires a dozen guys making angry phone calls every 2 hours to some suppliers... If it really takes 100 people 24/7, mostly likely explanation to me is that it's software related and they have to rewrite a critical sw in record time. Causes: -bugfix (which may be entirely unrelated to secrity) -emergency migration from one hw vendor to another, for sourcing reason which entails rewrite of a part of the system.

wisty 11y ago
Maybe there are 2,147,483,647 rows are in a table with the id signed int32? 2 billion isn't such a big number.
- raphman_ 11y ago
  Hmm, given they already had 'billions of rows' [1] (though probably distributed across many tables), they might have been wise enough to plan for this. [1] https://news.ycombinator.com/item?id=9756534 https://news.ycombinator.com/item?id=9756534
bigiain 11y ago
"100 people to work on it, 24/7" sounds like manual data entry to me - who wants to bet against a loss of a db without usable backups available, and a bunch of people now hand entering hardcopy forms and data out of emails back into some critical database?
- userbinator 11y ago
  I would be surprised if they didn't use OCR with possibly human checking.
  - mnx 11y ago
    Deploying OCR in an emergency seems like an insane thing to do, to me. Especially if you CAN just hire a 100 people to do it.
- eps 11y ago
  ... which means that somebody somewhere didn't have backups of some database server that died.
  - pdkl95 11y ago
    I'm guessing a backup technically existed (to satisfy any requirements or boss's-orders to the "letter of the law"), but nobody bothered giving any thought to restoring everything from that backup. Of course, a backup that can't be restored (or nobody knows how to restore) is more or less equivalent to not having a backup, so this distinction probably doesn't matter. // always remember to test your restore plan
    - codecamper 11y ago
      Damn I wish this would happen at the IRS. Tired of paying taxes to do nothing to help. Spend hundreds of millions on computer systems but they can't figure out how to do a backup.
    - bigiain 11y ago
      Indeed - "usable" backups - they are ones you've tested restoring from (recently enough to be assured they still work on the latest version of your system). I _hope_ they don't have someone saying "we had a backup - it was on a RAID set!". It would surprise me less to discover the reported migration from Oracle on Windows to Oracle on Linux was still partly done, and they were taking solid reliable useable backups - of the old not-yet-decommissioned windows db servers... (For the record, I've made both of those mistakes (and more) in my career... Fortunately neither represented weeks of 24x7 remedial work by 100s of people.)
- kalleboo 11y ago
  > sounds like manual data entry to me It could also be manual verification. Say they had two or more database nodes, a hardware failure on one could cause them to go out of sync and then they need to verify all the divergent changes.
pimlottc 11y ago
Not necessarily; replacing the physical hardware is just one step of the process of getting a system back to normal operation. They could have a spare already, but they haven't been keep up their disaster recovery drills and now have to figure out how to setup it up again from scratch. It doesn't even have to be custom hardware, it could just be a hard drive failure on commodity server that was just never backed up properly.
buro9 11y ago
Or the system has a long and complex work flow and some parts of that involve human labour. When any part of the system has a backlog, the laborious part needs people thrown at it to clear the backlog. This certainly is the case with the UK Passport office when it has issues.
- deleted 11y ago
  [deleted]
jpatokal 11y ago
I think you're taking that a bit too literally: I interpreted that as "we've got 100 tech guys and we've told them to fix this ASAP". Which, this being government, likely means 99 of them frantically trying to cover their asses and 1 guy doing the work.
- franciscop 11y ago
  Lol at least one is working, not like in Spain (; Edit: forgot to put a reference. This is the official website to renew the National Identity Document: - https://www.citapreviadnie.es/ https://www.citapreviadnie.es/ You also got the big warning notice saying 'This Connection is Untrusted' on Firefox and Chrome? Yep, it's been a feature for many years.
  - deleted 11y ago
    [deleted]
- codecamper 11y ago
  I've worked for the government before & that is more like it.
mattlutze 11y ago
"More than 100 experts across the country are working on this problem 24/7" is PR-speak. There's a pile of folks between a few contractor firms, and the staff in the offices that manage those contracts, whose current top priority is fixing whatever the issue is. They said specifically that embassies and consulates were having problems doing biometric checks, which I would imagine requires a State system to talk to a number of other federal/intel/defense systems to do records look-ups. If we do some further supposing, that web of interconnected systems may have had some underlying issues and I could definitely see it taking some time to get each firm involved in building or maintaining those systems together to figure out where the issues are and to figure out how to fix it.
marcosdumay 11y ago
I'm betting on the "we told all are IT workers to solve it" explanation, but their phrasing could still be true literally. Imagine that software depends on complex hardware, but it's not manufactured anymore, or the gov can not legally buy it anymore (contract expired). If it fails, it must be ported ASAP, what can take that kind of work. I've never seen something like this happen, but I've seen enough instances of it being possible to imagine it would happen once in a while.
wheaties 11y ago
Let me describe the structure of a project I witnessed but thankfully was never part of (and see if I can get the formatting right) Subcontractor which did the work: - 2 devs - 1 accountant - 1 project manager Contracted under: - 1 project lead - 1 dev lead - 4 QA engineers - 1 lead QA engineer - 3 accountants - 1 lead accountant - 2 document reviewers - 1 lead document reviewer The actual people doing the work spent 1 month writing documentation and then changed ...10 lines of code. This software was not mission critical, no lives were at stake, and if there was a bug the danger was negligible. That's government in action. I whole heartedly believe that there are 100 people at work here.
- lukeschlather 11y ago
  That sounds typical for any large legacy codebase serving hundreds of customers. It's not a government thing, it's just the nature of large projects. It's also not as simple as saying that it's waste. Changing 10 lines of code in an app that has been live for years can be the result of months of work and careful ruling out of 10,000 lines of code you decided not to change.
- harperlee 11y ago
  > 1 project lead - 1 dev lead - 4 QA engineers - 1 lead QA engineer - 3 accountants - 1 lead accountant - 2 document reviewers - 1 lead document reviewer Were all these people 100% on the project? My experience is that these roles usually are overhead for lot's of projects.
  - wheaties 11y ago
    100% that project alone. Remember, qa people in government contracts often are concerned less with qa'ing the code and more qa'ing the process within which the code is developed.
deleted 11y ago
[deleted]

wisty 11y ago

Maybe there are 2,147,483,647 rows are in a table with the id signed int32? 2 billion isn't such a big number.

raphman_ 11y ago
Hmm, given they already had 'billions of rows' [1] (though probably distributed across many tables), they might have been wise enough to plan for this. [1] https://news.ycombinator.com/item?id=9756534 https://news.ycombinator.com/item?id=9756534

bigiain 11y ago

"100 people to work on it, 24/7" sounds like manual data entry to me - who wants to bet against a loss of a db without usable backups available, and a bunch of people now hand entering hardcopy forms and data out of emails back into some critical database?

userbinator 11y ago
I would be surprised if they didn't use OCR with possibly human checking.
- mnx 11y ago
  Deploying OCR in an emergency seems like an insane thing to do, to me. Especially if you CAN just hire a 100 people to do it.
eps 11y ago
... which means that somebody somewhere didn't have backups of some database server that died.
- pdkl95 11y ago
  I'm guessing a backup technically existed (to satisfy any requirements or boss's-orders to the "letter of the law"), but nobody bothered giving any thought to restoring everything from that backup. Of course, a backup that can't be restored (or nobody knows how to restore) is more or less equivalent to not having a backup, so this distinction probably doesn't matter. // always remember to test your restore plan
  - codecamper 11y ago
    Damn I wish this would happen at the IRS. Tired of paying taxes to do nothing to help. Spend hundreds of millions on computer systems but they can't figure out how to do a backup.
  - bigiain 11y ago
    Indeed - "usable" backups - they are ones you've tested restoring from (recently enough to be assured they still work on the latest version of your system). I _hope_ they don't have someone saying "we had a backup - it was on a RAID set!". It would surprise me less to discover the reported migration from Oracle on Windows to Oracle on Linux was still partly done, and they were taking solid reliable useable backups - of the old not-yet-decommissioned windows db servers... (For the record, I've made both of those mistakes (and more) in my career... Fortunately neither represented weeks of 24x7 remedial work by 100s of people.)
kalleboo 11y ago
> sounds like manual data entry to me It could also be manual verification. Say they had two or more database nodes, a hardware failure on one could cause them to go out of sync and then they need to verify all the divergent changes.

pimlottc 11y ago

Not necessarily; replacing the physical hardware is just one step of the process of getting a system back to normal operation. They could have a spare already, but they haven't been keep up their disaster recovery drills and now have to figure out how to setup it up again from scratch. It doesn't even have to be custom hardware, it could just be a hard drive failure on commodity server that was just never backed up properly.

buro9 11y ago

Or the system has a long and complex work flow and some parts of that involve human labour. When any part of the system has a backlog, the laborious part needs people thrown at it to clear the backlog. This certainly is the case with the UK Passport office when it has issues.

deleted 11y ago
[deleted]

jpatokal 11y ago

I think you're taking that a bit too literally: I interpreted that as "we've got 100 tech guys and we've told them to fix this ASAP". Which, this being government, likely means 99 of them frantically trying to cover their asses and 1 guy doing the work.

franciscop 11y ago
Lol at least one is working, not like in Spain (; Edit: forgot to put a reference. This is the official website to renew the National Identity Document: - https://www.citapreviadnie.es/ https://www.citapreviadnie.es/ You also got the big warning notice saying 'This Connection is Untrusted' on Firefox and Chrome? Yep, it's been a feature for many years.
- deleted 11y ago
  [deleted]
codecamper 11y ago
I've worked for the government before & that is more like it.

mattlutze 11y ago

"More than 100 experts across the country are working on this problem 24/7" is PR-speak. There's a pile of folks between a few contractor firms, and the staff in the offices that manage those contracts, whose current top priority is fixing whatever the issue is. They said specifically that embassies and consulates were having problems doing biometric checks, which I would imagine requires a State system to talk to a number of other federal/intel/defense systems to do records look-ups. If we do some further supposing, that web of interconnected systems may have had some underlying issues and I could definitely see it taking some time to get each firm involved in building or maintaining those systems together to figure out where the issues are and to figure out how to fix it.

marcosdumay 11y ago

I'm betting on the "we told all are IT workers to solve it" explanation, but their phrasing could still be true literally. Imagine that software depends on complex hardware, but it's not manufactured anymore, or the gov can not legally buy it anymore (contract expired). If it fails, it must be ported ASAP, what can take that kind of work. I've never seen something like this happen, but I've seen enough instances of it being possible to imagine it would happen once in a while.

wheaties 11y ago

Let me describe the structure of a project I witnessed but thankfully was never part of (and see if I can get the formatting right) Subcontractor which did the work: - 2 devs - 1 accountant - 1 project manager Contracted under: - 1 project lead - 1 dev lead - 4 QA engineers - 1 lead QA engineer - 3 accountants - 1 lead accountant - 2 document reviewers - 1 lead document reviewer The actual people doing the work spent 1 month writing documentation and then changed ...10 lines of code. This software was not mission critical, no lives were at stake, and if there was a bug the danger was negligible. That's government in action. I whole heartedly believe that there are 100 people at work here.

lukeschlather 11y ago
That sounds typical for any large legacy codebase serving hundreds of customers. It's not a government thing, it's just the nature of large projects. It's also not as simple as saying that it's waste. Changing 10 lines of code in an app that has been live for years can be the result of months of work and careful ruling out of 10,000 lines of code you decided not to change.
harperlee 11y ago
> 1 project lead - 1 dev lead - 4 QA engineers - 1 lead QA engineer - 3 accountants - 1 lead accountant - 2 document reviewers - 1 lead document reviewer Were all these people 100% on the project? My experience is that these roles usually are overhead for lot's of projects.
- wheaties 11y ago
  100% that project alone. Remember, qa people in government contracts often are concerned less with qa'ing the code and more qa'ing the process within which the code is developed.

deleted 11y ago

[deleted]