4 ms·

I struggle to believe that GCHQ has the ability to store 3 or even 30 days worth of internet traffic flowing through the UK borders and also make reasonable sen

by a2d9s22 13y ago

I struggle to believe that GCHQ has the ability to store 3 or even 30 days worth of internet traffic flowing through the UK borders and also make reasonable sense of it all. It's such a mind boggling amount of data. I looked at some bandwidth graphs from places like LINX, which is just one of presumably many, and they are moving around a terrabyte a second - how can you possibly store, let alone sift through that kind of volume? 30 days at a terrabyte a second is nearly 3,000 petabytes of data. And that's just in one exchange. Is this kind of thing feasible? Facebook allegedly deals with 200PB of photo data - 3000PB is a lot more than that.

femto 13y ago
Three seconds of data at 1TB/s, will fit on the largest consumer hard drive (3TB). There are 3 x 86400 seconds in a day = 259200 seconds. Therefore three days' data requires 86400 drives. Cost of said drives is about $100 each, meaning the cost of disks to store 3 days of data is $8.64 million, which is negligible to the government. Even with multiple links at 1TB/s the problem is tractable.
- a2d9s22 13y ago
  I know, i did the numbers too - but it's still an almost unthinkable amount of data to try and make sense of. If we conservatively say that there are 5 others like linx, that's nearly half a million 3TB drives. And that's only for 3 days! edit: clarity
  - Zigurd 13y ago
    > but it's still an almost unthinkable amount of data to try and make sense of. It's only money, and compared with keeping a division supplied in Afghanistan or putting an enormous reconnaissance satellite in space, it's an obvious winner in the minds of the people running the security apparatus.
- malandrew 13y ago
  Storing it really is the trivial and cheap part. There are other really hard problems to solve: (1) Where do you place collection points so you get a full take of not only international traffic (moving across your borders) and domestic traffic (all traffic within your borders but that doesn't leave your borders) (2) Since the number of collection points is limited, that means there is a lot of data that has to be recorded at select points. How do you record that data to disk in real time? (3) How do you avoid the duplication of packets that travel through multiple collection points. (4) Lastly, the most difficult problem is figuring out how to query all that data and not end up with a haystack. When you have millions and millions of pieces of communication from people with no involvement in the criminal activity, then all that communication becomes noise.
  - femto 13y ago
    1) If it's your own country's traffic, at the choke point in the network, which you (ie. govt) have full control over, since they are domestic. 2) Since you already have 100,000 disks, run 5000 of them in parallel. Each disk interface does 200MB/s, for an aggregate of 1TB/s. If necessary, use fibre to transport data from the collection point to the storage point. 3) Don't. Record the lot. 4) Spend a billion on a super computer? I think it comes down to the bandwidth of computers now vastly exceeding the bandwidth of human thought. It is now possible to record an person's entire life, with plenty of headroom to spare, to account for any attempt to overwhelm the recording system.

by a2d9s22 13y ago

femto 13y ago

Three seconds of data at 1TB/s, will fit on the largest consumer hard drive (3TB). There are 3 x 86400 seconds in a day = 259200 seconds. Therefore three days' data requires 86400 drives. Cost of said drives is about $100 each, meaning the cost of disks to store 3 days of data is $8.64 million, which is negligible to the government. Even with multiple links at 1TB/s the problem is tractable.

a2d9s22 13y ago
I know, i did the numbers too - but it's still an almost unthinkable amount of data to try and make sense of. If we conservatively say that there are 5 others like linx, that's nearly half a million 3TB drives. And that's only for 3 days! edit: clarity
- Zigurd 13y ago
  > but it's still an almost unthinkable amount of data to try and make sense of. It's only money, and compared with keeping a division supplied in Afghanistan or putting an enormous reconnaissance satellite in space, it's an obvious winner in the minds of the people running the security apparatus.
malandrew 13y ago
Storing it really is the trivial and cheap part. There are other really hard problems to solve: (1) Where do you place collection points so you get a full take of not only international traffic (moving across your borders) and domestic traffic (all traffic within your borders but that doesn't leave your borders) (2) Since the number of collection points is limited, that means there is a lot of data that has to be recorded at select points. How do you record that data to disk in real time? (3) How do you avoid the duplication of packets that travel through multiple collection points. (4) Lastly, the most difficult problem is figuring out how to query all that data and not end up with a haystack. When you have millions and millions of pieces of communication from people with no involvement in the criminal activity, then all that communication becomes noise.
- femto 13y ago
  1) If it's your own country's traffic, at the choke point in the network, which you (ie. govt) have full control over, since they are domestic. 2) Since you already have 100,000 disks, run 5000 of them in parallel. Each disk interface does 200MB/s, for an aggregate of 1TB/s. If necessary, use fibre to transport data from the collection point to the storage point. 3) Don't. Record the lot. 4) Spend a billion on a super computer? I think it comes down to the bandwidth of computers now vastly exceeding the bandwidth of human thought. It is now possible to record an person's entire life, with plenty of headroom to spare, to account for any attempt to overwhelm the recording system.