3 ms·
I've designed systems that do this on continental scales (i.e. hundreds of millions of cell phones simultaneously, in real-time). The devil is in the details an
by jandrewrogers 9y ago
I've designed systems that do this on continental scales (i.e. hundreds of millions of cell phones simultaneously, in real-time). The devil is in the details and non-trivial; this is not a "an intern and 6 months" job. Mobile telemetry is not nearly as ideal in practice as assumed here and it typically takes a couple years to learn how to handle the numerous peculiar artifacts of that data that will damage the quality of a naive implementation. Reconstructing a model of the population from the cleaned data that approximates the ground truthing is surprisingly difficult and requires quite a bit of clever data science and maths.
It takes a lot of work and expertise to build a population model from mobile telemetry that approximately reflects reality. Far fewer people know how to do this well than you might assume by looking at the requirements for a naive implementation. Even most mobile carriers have limited ability.
- frandroid 9y agoHave you posted about this at length somewhere? If not, care to elaborate on what it took to design this system?
- jandrewrogers 9y agoI have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing it, it just requires far more domain expertise to do correctly than I think people expect. You also need to be willing to write some of your own tooling to deal with the data efficiently and effectively.
- dajohnson89 9y agoI'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.
- rl3 9y agoThe Snowden leaks confirmed NSA has the ability to conduct co-traveler inference.[0] In other words: finding mobile devices proximate to a targeted mobile device, based on similar vectors. Perhaps even making associations in absence of targeting via patterns in device proximity over time. It probably gets real interesting when they're trying to distinguish between various modes of transit, such as a city bus, an Uber/Lyft/taxi, and a private vehicle not participating in rideshares. Of those examples, the latter would suggest the highest degree of association. Pure speculation, but I wouldn't doubt they take a peek at ridesharing data for co-traveler inference purposes. Knowing if a rideshare driver is on or off the clock would be incredibly valuable information in that context. [0] https://www.washingtonpost.com/apps/g/page/world/how-the-nsa-is-tracking-people-right-now/634/ https://www.washingtonpost.com/apps/g/page/world/how-the-nsa...
- jandrewrogers 9y agoGraph reconstruction from space-time event data goes far beyond the above in terms of capability. You can infer relationships between people that never co-travel, infer that people have been places that are not in the event data, etc by stitching together large numbers of orthogonal event streams over long periods of time. It is straightforward to distinguish between various modes of transit analytically. The "metadata" that simply indicates an event in space-time is far more valuable analytically than the data because it is possible to reconstruct so much with it that isn't contained within the data per se. I was doing all of this five years ago, the capability has been around for a while.
- pjc50 9y agoDo you ever worry that this data might be used for bad purposes?
- deleted 9y ago[deleted]
- user5994461 9y agoMapping is wide and common industry, just like web or finance. The NSA only recruits in the USA. It's a fraction of the talent pool of the planet.
- user5994461 9y agoA great PoC is fairly doable by an intern. Increasing the precision by tenfold will likely increase the effort by a hundred fold or more. Just because it can be made harder and more expensive doesn't mean it has to be. At the end of the day, a bit of precision doesn't change the nature of an effective planet scale mass surveillance system.
- awkwarddaturtle 9y ago> which requires additional data sources beyond the mobile telemetry. So it really isn't technically difficult. It's just lacking in data? In your original post, you made it seem like it was extremely difficult. From my perspective, it seems like child's play from a technical standpoint. > It isn't that difficult technically, if you have experts doing it But your top comment implied it was extremely technically difficult.
- jandrewrogers 9y agoThe are some things that are technically very difficult if you have no domain expertise, which applies to the subject matter. Most people that try to do this without experience fail in practice, it takes a lot of time and effort to become competent at it, but once you figure it out it is repeatable without too much effort. There is a much smaller set of things that are technically difficult to execute even if you are highly experienced at doing it -- each time is a challenge. This is not one of those cases, it just has a severe learning curve.
- zkms 9y ago> the numerous peculiar artifacts of that data that will damage the quality of a naive implementation. Do you have examples (or a link/reference to something that has such examples) of those types of artifacts?
- pdoege 9y agoCarrier iQ demonstrated GPS level tracking of 100m+ phones nearly 10 years ago.