Several years ago I joined a stealth-mode spinoff of a large telecom equipment manufacturer, and worked on deep-packet inspection (DPI) on 10Gb/s fiber-optic ne

Several years ago I joined a stealth-mode spinoff of a large telecom equipment manufacturer, and worked on deep-packet inspection (DPI) on 10Gb/s fiber-optic networks. The initial motivation for the project was a desire by telecom companies to obtain personally identifiable information and profiles for targeted advertising, and as we learned, a number of other interesting use cases, like reducing customer churn. At the NOCs there were a number of various servers in racks that were getting a feed of raw traffic for a variety of purposes. It wasn't a free-for-all, but it was eye-opening. I recall reading through a Cisco router manual for example (not the company I worked for) and seeing its port-mirroring capabilities blatantly promoted as a means for customer profiling and targeted advertising. Outside of the telecom industry itself there was quite a bit of resistance to this sort of thing, and we had to go before the US Congress to explain what we were up to. Profiling for the sake of profiling was not smooth sailing, but if it was for the purpose of "security" then it was more or less a free pass. The forcus of our DPI technology turned to the task of network-based threat detection as its primary raison d'être, with customer profiling being an opt-in service by which users could obtain the security service in exchange for targeted ads. In the years since, I don't expect that Telecom's desire to be much more than a "dumb pipe" has diminished in the least. They view the traffic they carry on their networks to be their property, in a way. They feel entitled to inspect it, throttle it, slice and dice it any conceivable way they can to maximise their profits. Its one of the reasons I quit. Imagine the US postal service steaming open every letter and opening every package that went through their system, so they could plug your mailbox with targeted special offers or increase the delivery fees for certain things. Its all similar BS with ISPs, but it's all techie stuff and heavily lobbied so the public gets bamboozled. You raise a really good point about wide-spread encryption being an impediment to ISP profiling. But there is a LOT you can surmise from user traffic even if you don't know the exact content of the encrypted payloads. Just analyzing IP addresses and times can reveal a ton of information about a person. My first patent [US20100161795] was in fact a NAT session detection and tracking technique to identify and track individual users within a household through TCP/IP analysis. Using this technique someone could get a pretty clear picture of how many people were in a household, their ages, genders, interests and patterns of activity, even without delving into the http payload of the packets. We didn't, but this kind of thing is most definitely possible, and I wouldn't trust other shady entities not to do it.