4 ms·

I wonder how much data can be processed compared to a modern "hackable" system? My guess is close to nil. How much of it gets lost? How much of the paper is dup

I wonder how much data can be processed compared to a modern "hackable" system? My guess is close to nil. How much of it gets lost? How much of the paper is duplicated for backups? How much time is spent looking through filling cabinets for relevant data? How is data transmitted - physically mailed, or faxed, or scanned and emailed, which is hackable anyway? There are costs to both practices.

jacobush 7y ago
How much useful data hides forever "lost" in "big data".
- mulmen 7y ago
  I'm not sure I follow. Do you mean it's hard to find a single record out of many or that new insights can be discovered in aggregate? The former is simply not true, databases exist for this purpose. The latter seems obvious.
  - jacobush 7y ago
    I mean, gathering all kinds of junk may make you loose sight of what you really are about. See for instance various self-criticism from within the three letter agencies, about how operational capabilities were lost in favour of hoovering up as much data as possible. Most of which is useless. When compiling and processing data is very painful, you make sure you compile and process the best data you can find. I'm not saying we should go back to clay tablets or anything. I'm just saying in any rapid technology transition, we risk loosing the good with the bad.
    - mulmen 7y ago
      I don't think I agree with this line of thinking. How can we determine data is "useless" if we do not analyze the data? New technology may have good and bad aspects but that doesn't mean it prevents us from doing what works.
      - jazzyjackson 7y ago
        Analyzing the data becomes more difficult as you put quantity over quality wrt how you're structuring data and where its coming from. There's a database somewhere of Do Not Fly suspicious people, but how was it compiled? Should we put any weight on these lists of names? I think this is what OP was alluding to, when it is more painful to collect and organize data, you make sure the data is worth collecting and organizing. See also "Worse is Better"
        mulmen 7y ago
        I don’t agree that we have to make a trade off between quality and quantity of data. That seems like a false choice to me.
dsfyu404ed 7y ago
People are getting their retirement checks. Clearly it's processing enough data.
- mulmen 7y ago
  Do you know that though? does the system know if it misses someone? How long does it take? The linked article suggests employees are banking months of vacation time in anticipation of the long delays in processing.
Johnny555 7y ago
I wonder how much data can be processed compared to a modern "hackable" system Probably not much -- I'd expect that single record lookups are relatively fast, but large scale analysis across large datasets is nearly impossible. Probably more of a "feature" of a government database than a drawback - the government should be doing as little as possible with our data.
ptah 7y ago
filing systems are fairly robust. it's down to trade-offs. valuable data is better kept offline and the costs can actually be less than keeping it in cloud database

jacobush 7y ago

How much useful data hides forever "lost" in "big data".

mulmen 7y ago
I'm not sure I follow. Do you mean it's hard to find a single record out of many or that new insights can be discovered in aggregate? The former is simply not true, databases exist for this purpose. The latter seems obvious.
- jacobush 7y ago
  I mean, gathering all kinds of junk may make you loose sight of what you really are about. See for instance various self-criticism from within the three letter agencies, about how operational capabilities were lost in favour of hoovering up as much data as possible. Most of which is useless. When compiling and processing data is very painful, you make sure you compile and process the best data you can find. I'm not saying we should go back to clay tablets or anything. I'm just saying in any rapid technology transition, we risk loosing the good with the bad.
  - mulmen 7y ago
    I don't think I agree with this line of thinking. How can we determine data is "useless" if we do not analyze the data? New technology may have good and bad aspects but that doesn't mean it prevents us from doing what works.
    - jazzyjackson 7y ago
      Analyzing the data becomes more difficult as you put quantity over quality wrt how you're structuring data and where its coming from. There's a database somewhere of Do Not Fly suspicious people, but how was it compiled? Should we put any weight on these lists of names? I think this is what OP was alluding to, when it is more painful to collect and organize data, you make sure the data is worth collecting and organizing. See also "Worse is Better"
      - mulmen 7y ago
        I don’t agree that we have to make a trade off between quality and quantity of data. That seems like a false choice to me.

dsfyu404ed 7y ago

People are getting their retirement checks. Clearly it's processing enough data.

mulmen 7y ago
Do you know that though? does the system know if it misses someone? How long does it take? The linked article suggests employees are banking months of vacation time in anticipation of the long delays in processing.

Johnny555 7y ago

I wonder how much data can be processed compared to a modern "hackable" system Probably not much -- I'd expect that single record lookups are relatively fast, but large scale analysis across large datasets is nearly impossible. Probably more of a "feature" of a government database than a drawback - the government should be doing as little as possible with our data.

ptah 7y ago

filing systems are fairly robust. it's down to trade-offs. valuable data is better kept offline and the costs can actually be less than keeping it in cloud database