3 ms·

No, but I think it's unlikely that anybody, including the NSA, can collect, index, catalog and parse meaning out of hundreds of GB _per second_ (I'm using Stati

No, but I think it's unlikely that anybody, including the NSA, can collect, index, catalog and parse meaning out of hundreds of GB _per second_ (I'm using Statista numbers for global IP traffic). I would be skeptical that it's even physically possible with any amount of hardware in existence. If you really want a concentrated haul of the most interesting stuff, then I would say that platforms which openly market themselves as safe venues for criminals and dissidents would be strong candidates. You would still have extraordinary data warehousing needs just for that limited subset of traffic.

posix_me_less 5y ago
> unlikely that anybody, including the NSA, can collect, index, catalog and parse meaning out of hundreds of GB _per second No, it's actually very likely. I've noticed people often make this argument as if it is obvious that the data that is being processed is somehow too large to handle. But they misunderstand the ways this can be done and underestimate current technological capabilities. One big switch in single datacenter can handle all traffic of that datacenter, hundreds of GB/s. Make a special one with custom FPGA/cpus and in addition to switching, it can do decrypting, parsing, matching and filtering the way you like, hundreds of GB per second, real-time. This is a highly paralelizable job and the hardware is made to exploit that. The resulting stream of interesting data is probably much thinner and can be easily sent to some central location, realtime. The rest can be stored locally and sent on demand.
- grumblenum 5y ago
  I can see how hardware handling of switching, some filtering and maybe stripping TLS is feasible. Since you're knowledgeable of the subject, how would you index that information for future retrieval, which could be anything in the set of all digital representations of information in all possible languages and encodings? How would you determine what information is important?
  - fjrueosjdksn 5y ago
    packet metadata, intercept location and timestamp, will form the index.
- fjrueosjdksn 5y ago
  You are not incorrect about that hardware. Splice off fibre optics, copy packets, localised hardware processes like you say, directs to data center for future processing or active surveillance or active legal interception, etc.
  - posix_me_less 5y ago
    Which part is incorrect?
int_19h 5y ago
I don't think they can parse all of it in real time, but storing it with basic metadata (like IP addresses) is another matter. Then it can be selectively extracted and parsed when needed.

posix_me_less 5y ago

> unlikely that anybody, including the NSA, can collect, index, catalog and parse meaning out of hundreds of GB _per second No, it's actually very likely. I've noticed people often make this argument as if it is obvious that the data that is being processed is somehow too large to handle. But they misunderstand the ways this can be done and underestimate current technological capabilities. One big switch in single datacenter can handle all traffic of that datacenter, hundreds of GB/s. Make a special one with custom FPGA/cpus and in addition to switching, it can do decrypting, parsing, matching and filtering the way you like, hundreds of GB per second, real-time. This is a highly paralelizable job and the hardware is made to exploit that. The resulting stream of interesting data is probably much thinner and can be easily sent to some central location, realtime. The rest can be stored locally and sent on demand.

grumblenum 5y ago
I can see how hardware handling of switching, some filtering and maybe stripping TLS is feasible. Since you're knowledgeable of the subject, how would you index that information for future retrieval, which could be anything in the set of all digital representations of information in all possible languages and encodings? How would you determine what information is important?
- fjrueosjdksn 5y ago
  packet metadata, intercept location and timestamp, will form the index.
fjrueosjdksn 5y ago
You are not incorrect about that hardware. Splice off fibre optics, copy packets, localised hardware processes like you say, directs to data center for future processing or active surveillance or active legal interception, etc.
- posix_me_less 5y ago
  Which part is incorrect?

int_19h 5y ago

I don't think they can parse all of it in real time, but storing it with basic metadata (like IP addresses) is another matter. Then it can be selectively extracted and parsed when needed.