2 ms·
There are a mixture of issues at play here. People tend to underestimate collection capability and overestimate processing and analysis capability. The former
by jandrewrogers 3y ago
There are a mixture of issues at play here.
People tend to underestimate collection capability and overestimate processing and analysis capability. The former greatly outstrips the capacity of the latter in practice. This is fundamentally a technology gap. For example, the open source stacks cannot handle the scale and velocity of the data nor the complexity of the data analysis required. The tech gap is qualitative.
A major driver behind the increasing use of open source intelligence (OSINT) is data freshness, latency of access, and the ability to easily do mash-ups of different sources. Use of classified collections requires a bureaucratic process to even know it exists, never mind get permission to use it, or to blend it with other collection sources. Furthermore, data handling restrictions introduce high data processing latencies at an architectural level. As a consequence, the classified collection may have amazing data but you may not be able to put it all together for days, which makes it difficult to use for intelligence analysis that is very time sensitive. OSINT, by contrast, is largely permission-less with few gatekeepers, so it has a huge speed advantage in terms of time-to-insight that can often provide more value than having exceptional data that you can't access quickly enough to matter.
Intelligence agencies have been trying to tackle these challenges for many years. It does not have an easy answer because it is a confluence of independent technology, process, and cultural issues.