5 ms·

Related/relevant question. A number of months ago I decided to finally begin researching a project I've had on the backburner for ages: a simple domain+IP band

Related/relevant question. A number of months ago I decided to finally begin researching a project I've had on the backburner for ages: a simple domain+IP bandwidth throughput counter for everything within my LAN. Basically I wanted to make a dashboard saying "you exchanged this much data with this many IPs within the last 24h; here's the per-IP and per-domain breakdown; here are the domains each IP served", that sort of thing. Part of the motivation for this was to track down the headscratch-generating mysterious small amount of "unmetered data" my ISP reports each month (I don't watch TV over the internet and cannot think of what else would be it), and the other part of the motivation was to satisfy idle curiosity and observe domain<->IP cohesion/fragmentation over time. At the exact point I began my research TLS 1.3 had just been approved, and I rapidly realized my nice little idea was going to get majorly stomped on: the inspiration for the project, years ago, was the SNI field itself (hah). 5-10 years ago, I would have been able to route all my traffic through a gateway box, and passively watch/trace (slightly behind realtime) everything via tcpdump or similar, catching DNS lookups and sniffing SNI fields. That would have handled everything. Now, with encrypted SNI, and the upcoming DNS-over-HTTPS (seems I need to think ahead :/), it looks like my (now irritatingly stupid-seeming) project will require me to generate a local cert, install it on all my devices, and get a machine beefy enough to keep line-rate speed while doing MITM cert-resigning. And then, for the fraction of traffic my system reports as "not signed by local cert", I'm just going to have to let that go, or I'll break everything that uses cert pinning. ...which I wouldn't be surprised if some programs eventually use with DoH lookups (using pinned certs fished out of DNSSEC for DoH makes sense). I'm very very interested to know if there are any alternatives I can use. I realize, for now, that I can sniff for and collect DNS lookup results and associate these with IPs, but when will this break?(!) Honestly HTTPS and security in general just seems like a gigantic mess. I say this not as a disrespect of the architecture, but more as a "wow, this is just so hard to get right".

xg15 8y ago
Even on desktop, some non-browser programs already seem to use cert-pinning. E.g., there doesn't seem to be a way at all to inspect the traffic of the Dropbox client on Windows, short of patching the executable. The situation is likely worse on mobile. So no, I don't think that's an easily solvable problem. And yes, I also think it's a giant mess. Even from a security standpoint it doesn't seem to make sense: We're fortifying the connections itself against sniffing but at the same time force everything to go through centralized cloud services which have a growing reputation to be hacked and also force everyone to blindly trust the programs they have running on their own devices. To put on my tinfoil hat, this kind of threat model sometimes seems better designed to protect surveillance capitalism against attacks instead of protecting the users.
- zeta0134 8y ago
  I mean, no one's forcing me to blindly trust programs I run on my own devices. I make a point of only running open source software for this reason. I don't need to inspect the traffic if I can simply read the source code and look for outgoing calls. I won't run Dropbox for this exact reason. Instead I prefer SyncThing, which has no data caps, no central server, and is open source and easy to inspect. Obviously I'm not actually reading all this source code (who has the time?), but I strongly prefer it to be available, so that I have the ability if desired. Also for reasonably popular stuff the community tends to suss out suspicious activity pretty quickly. I absolutely do not trust proprietary blobs; they don't get to run on any of my real workstations. I might run them on my Windows install (primarily for games) if I must, and the computer I use for work gets to run anything my company declares that I need (thankfully not much), but I certainly don't trust them.
  - paulddraper 8y ago
    > Also for reasonably popular stuff the community tends to suss out suspicious activity pretty quickly. Unless it's ImageMagick, OpenSSH, WordPress, or any number of other things.
- danjoc 8y ago
  >The situation is likely worse on mobile. You might think so, but not really. Any app you install on Android can watch what you do online in any other app, with no permissions at all. https://hackernoon.com/android-gives-apps-full-access-to-your-network-activity-f0fd92ee1824 https://hackernoon.com/android-gives-apps-full-access-to-you...
brazzledazzle 8y ago
This is pretty similar to my own thoughts on the matter that came from thoughts about doing something similar (mostly to make sure there’s not random malware from family members). Imagine being in corp information security though. It seems like it won’t be long before command and control or exfiltration happens via certificate pinned apps you learn you can’t inspect. Corporate networks will have to start banning them if they don’t have a way for them to inspect.
- mr_toad 8y ago
  Honestly, if corporate IT has to drop connections to a bank, because they can’t MiTM the connection, I see this as a good thing. Too many people are unaware that their connections to their banks etc from work are insecure because of corporate firewalls and what are (essentially) compromised certificate stores.
  - tialaramex 8y ago
    It's not uncommon with TLS 1.2 to have a setup where what actually happens is this: You make an outgoing TLS connection, the MITM Proxy Meg intervenes, but you actually wanted MyBank You->Meg (thinking she's MyBank): Hi, I'm looking for MyBank? Meg->MyBank: Hi, I'm looking for MyBank? MyBank->Meg: Yes, we're MyBank <cert for MyBank> [ Meg inspects the cert ] Meg->You: Yes, we're MyBank <cert for MyBank> Now Meg stitches the TCP connection back together so you're talking to MyBank. She no longer eavesdrops. This works for the business because now they're not paying for a box that can proxy your connection _and_ they're less likely to find that one of their "security" employees emptied your bank account. They feel comfortable that MyBank aren't bad guys, so they aren't scared that you'll, say, exfiltrate stolen corporate secrets via MyBank or download Malware from it. In TLS 1.3 this doesn't work, because the conversation goes like this: You->Meg (thinking she's MyBank): Hi, I'm looking for MyBank and I picked 14, and the letter Q Meg->MyBank: Hi, I'm looking for MyBank and I picked 14, and the letter Q MyBank->Meg: Cool, I picked a Banana and 209. <Unintelligible encrypted data> [ Meg can't do anything with this ] Or Meg could try picking her own Elliptic Curve Diffie Hellman values: You->Meg (thinking she's MyBank): Hi, I'm looking for MyBank and I picked 14, and the letter Q Meg->MyBank: Hi, I'm looking for MyBank and I picked 81, and the letter Z MyBank->Meg: Cool, I picked a Banana and 209. <We're MyBank - encrypted data> [ Meg now has a connection to MyBank, but she can't give it you because it doesn't work if you picked different values ] In the medium term we expect corporations which do this will just clamp everything to TLS 1.2 with full proxying. This will be very expensive, and worse, but well, we did tell you not to do this, so now that's broken you get to keep both halves. Ordinary folks who aren't trying to put middleboxes everywhere get TLS 1.3, with better performance and more security.
crtasm 8y ago
Maybe it's a deliberate lie from your ISP? if everyone has the nice words 'unmetered data' on their bill it might help with overall customer retention.
- exikyut 8y ago
  This is Telstra, Australia's juggernaut/cartel^Hmajor player (owns all the copper and 4G; "most definitely >51% market and then some"). So I'd actually mostly blame bureaucracy. There are (and have been for some time) some known-unmetered sites. (There was a small stink raised where an FTP/HTTP file mirror got removed some years ago.) Some digging around I did some months ago turned up a (semi-public, as in "obscurely published but uninteresting") list of supposedly-whitelisted IP addresses pointing to school/university backend systems and apparently used by rural schools sitting on (expensive) satellite. Suffice to say that I wasn't able to find anything that could be used as a proxy :P
  - filmor 8y ago
    iinet has a few domains they explicitly do not meter and list in the stats, including a mirror for open source software. Was very useful while I was down under to keep my Gentoo system nicely updated without killing the quota of our shared flat. I so don't miss this :)
  - crtasm 8y ago
    Ah, to clarify I meant lying about you accessing any unmetered sites/IPs, not that any existed.
tialaramex 8y ago
You will still be able to measure how much data goes to and from particular IPs, since that can't be screened. TLS 1.3 approval doesn't stomp on SNI; on the contrary, for the first time it actually makes SNI mandatory - in previous versions it was optional, leading to some HTTP implementations still not working with SNI late in the 21st century. I had a corporate Python system I was responsible for that couldn't do SNI even earlier this year. You probably shouldn't attempt your idea of proxying everything. Proxies are very fragile, and this will probably inadvertently worsen your security, while also introducing weird non-security misbehaviour that's hard to track down. For example Firefox will go "Oh, a MITM proxy :(" and it will switch off all the safety measures built up over years for the public Internet because this isn't the public Internet. In principle if your MITM proxy is enforcing the same or tougher rules, this wouldn't be worse, but let's face it, you're going to cobble together the minimum possible to make it work. Every time somebody tries to get fancy and _think_ about the bits inside the packets their network is supposed to be moving they screw up. That's the most cynical reason why there's an impetus to encrypt everything. If the bits are all encrypted gibberish the temptation to meddle with them goes away.
- ryanlol 8y ago
  >leading to some HTTP implementations still not working with SNI late in the 21st century So how did IPv6 adoption go?
- xg15 8y ago
  And yet a number of recent privacy breaches where apps were collecting and transmitting data they had no business in were discovered by sniffing the traffic. > Every time somebody tries to get fancy and _think_ about the bits inside the packets their network is supposed to be moving they screw up. How do you propose we do security otherwise? (Apart from delegating it to other parties so they think about the bits)
  - wang_li 8y ago
    >How do you propose we do security otherwise? (Apart from delegating it to other parties so they think about the bits) The clear and obvious solution is to make developers, managers, executives, and major investors personally responsible for the damage their products do. If Google is going to encrypt everything using pinned certificates, making it impossible for me to implement security features/boundaries of my choice, then they should be 100% liable for all damages that come from the use of their products. Additionally, if an analysis of the problem shows that is is related to aspects of the product that do not directly related to what the end user believes the purpose of the product to be, then there should also be criminal liabilities for all parties as well.
the8472 8y ago
Maybe just run your own DNS resolver and log that, then correlate with IP traffic? Additionally you can trace which applications open which connections on each machine.

xg15 8y ago

Even on desktop, some non-browser programs already seem to use cert-pinning. E.g., there doesn't seem to be a way at all to inspect the traffic of the Dropbox client on Windows, short of patching the executable. The situation is likely worse on mobile. So no, I don't think that's an easily solvable problem. And yes, I also think it's a giant mess. Even from a security standpoint it doesn't seem to make sense: We're fortifying the connections itself against sniffing but at the same time force everything to go through centralized cloud services which have a growing reputation to be hacked and also force everyone to blindly trust the programs they have running on their own devices. To put on my tinfoil hat, this kind of threat model sometimes seems better designed to protect surveillance capitalism against attacks instead of protecting the users.

zeta0134 8y ago
I mean, no one's forcing me to blindly trust programs I run on my own devices. I make a point of only running open source software for this reason. I don't need to inspect the traffic if I can simply read the source code and look for outgoing calls. I won't run Dropbox for this exact reason. Instead I prefer SyncThing, which has no data caps, no central server, and is open source and easy to inspect. Obviously I'm not actually reading all this source code (who has the time?), but I strongly prefer it to be available, so that I have the ability if desired. Also for reasonably popular stuff the community tends to suss out suspicious activity pretty quickly. I absolutely do not trust proprietary blobs; they don't get to run on any of my real workstations. I might run them on my Windows install (primarily for games) if I must, and the computer I use for work gets to run anything my company declares that I need (thankfully not much), but I certainly don't trust them.
- paulddraper 8y ago
  > Also for reasonably popular stuff the community tends to suss out suspicious activity pretty quickly. Unless it's ImageMagick, OpenSSH, WordPress, or any number of other things.
danjoc 8y ago
>The situation is likely worse on mobile. You might think so, but not really. Any app you install on Android can watch what you do online in any other app, with no permissions at all. https://hackernoon.com/android-gives-apps-full-access-to-your-network-activity-f0fd92ee1824 https://hackernoon.com/android-gives-apps-full-access-to-you...

brazzledazzle 8y ago

This is pretty similar to my own thoughts on the matter that came from thoughts about doing something similar (mostly to make sure there’s not random malware from family members). Imagine being in corp information security though. It seems like it won’t be long before command and control or exfiltration happens via certificate pinned apps you learn you can’t inspect. Corporate networks will have to start banning them if they don’t have a way for them to inspect.

mr_toad 8y ago
Honestly, if corporate IT has to drop connections to a bank, because they can’t MiTM the connection, I see this as a good thing. Too many people are unaware that their connections to their banks etc from work are insecure because of corporate firewalls and what are (essentially) compromised certificate stores.
- tialaramex 8y ago
  It's not uncommon with TLS 1.2 to have a setup where what actually happens is this: You make an outgoing TLS connection, the MITM Proxy Meg intervenes, but you actually wanted MyBank You->Meg (thinking she's MyBank): Hi, I'm looking for MyBank? Meg->MyBank: Hi, I'm looking for MyBank? MyBank->Meg: Yes, we're MyBank <cert for MyBank> [ Meg inspects the cert ] Meg->You: Yes, we're MyBank <cert for MyBank> Now Meg stitches the TCP connection back together so you're talking to MyBank. She no longer eavesdrops. This works for the business because now they're not paying for a box that can proxy your connection _and_ they're less likely to find that one of their "security" employees emptied your bank account. They feel comfortable that MyBank aren't bad guys, so they aren't scared that you'll, say, exfiltrate stolen corporate secrets via MyBank or download Malware from it. In TLS 1.3 this doesn't work, because the conversation goes like this: You->Meg (thinking she's MyBank): Hi, I'm looking for MyBank and I picked 14, and the letter Q Meg->MyBank: Hi, I'm looking for MyBank and I picked 14, and the letter Q MyBank->Meg: Cool, I picked a Banana and 209. <Unintelligible encrypted data> [ Meg can't do anything with this ] Or Meg could try picking her own Elliptic Curve Diffie Hellman values: You->Meg (thinking she's MyBank): Hi, I'm looking for MyBank and I picked 14, and the letter Q Meg->MyBank: Hi, I'm looking for MyBank and I picked 81, and the letter Z MyBank->Meg: Cool, I picked a Banana and 209. <We're MyBank - encrypted data> [ Meg now has a connection to MyBank, but she can't give it you because it doesn't work if you picked different values ] In the medium term we expect corporations which do this will just clamp everything to TLS 1.2 with full proxying. This will be very expensive, and worse, but well, we did tell you not to do this, so now that's broken you get to keep both halves. Ordinary folks who aren't trying to put middleboxes everywhere get TLS 1.3, with better performance and more security.

crtasm 8y ago

Maybe it's a deliberate lie from your ISP? if everyone has the nice words 'unmetered data' on their bill it might help with overall customer retention.

exikyut 8y ago
This is Telstra, Australia's juggernaut/cartel^Hmajor player (owns all the copper and 4G; "most definitely >51% market and then some"). So I'd actually mostly blame bureaucracy. There are (and have been for some time) some known-unmetered sites. (There was a small stink raised where an FTP/HTTP file mirror got removed some years ago.) Some digging around I did some months ago turned up a (semi-public, as in "obscurely published but uninteresting") list of supposedly-whitelisted IP addresses pointing to school/university backend systems and apparently used by rural schools sitting on (expensive) satellite. Suffice to say that I wasn't able to find anything that could be used as a proxy :P
- filmor 8y ago
  iinet has a few domains they explicitly do not meter and list in the stats, including a mirror for open source software. Was very useful while I was down under to keep my Gentoo system nicely updated without killing the quota of our shared flat. I so don't miss this :)
- crtasm 8y ago
  Ah, to clarify I meant lying about you accessing any unmetered sites/IPs, not that any existed.

tialaramex 8y ago

You will still be able to measure how much data goes to and from particular IPs, since that can't be screened. TLS 1.3 approval doesn't stomp on SNI; on the contrary, for the first time it actually makes SNI mandatory - in previous versions it was optional, leading to some HTTP implementations still not working with SNI late in the 21st century. I had a corporate Python system I was responsible for that couldn't do SNI even earlier this year. You probably shouldn't attempt your idea of proxying everything. Proxies are very fragile, and this will probably inadvertently worsen your security, while also introducing weird non-security misbehaviour that's hard to track down. For example Firefox will go "Oh, a MITM proxy :(" and it will switch off all the safety measures built up over years for the public Internet because this isn't the public Internet. In principle if your MITM proxy is enforcing the same or tougher rules, this wouldn't be worse, but let's face it, you're going to cobble together the minimum possible to make it work. Every time somebody tries to get fancy and _think_ about the bits inside the packets their network is supposed to be moving they screw up. That's the most cynical reason why there's an impetus to encrypt everything. If the bits are all encrypted gibberish the temptation to meddle with them goes away.

ryanlol 8y ago
>leading to some HTTP implementations still not working with SNI late in the 21st century So how did IPv6 adoption go?
xg15 8y ago
And yet a number of recent privacy breaches where apps were collecting and transmitting data they had no business in were discovered by sniffing the traffic. > Every time somebody tries to get fancy and _think_ about the bits inside the packets their network is supposed to be moving they screw up. How do you propose we do security otherwise? (Apart from delegating it to other parties so they think about the bits)
- wang_li 8y ago
  >How do you propose we do security otherwise? (Apart from delegating it to other parties so they think about the bits) The clear and obvious solution is to make developers, managers, executives, and major investors personally responsible for the damage their products do. If Google is going to encrypt everything using pinned certificates, making it impossible for me to implement security features/boundaries of my choice, then they should be 100% liable for all damages that come from the use of their products. Additionally, if an analysis of the problem shows that is is related to aspects of the product that do not directly related to what the end user believes the purpose of the product to be, then there should also be criminal liabilities for all parties as well.

the8472 8y ago

Maybe just run your own DNS resolver and log that, then correlate with IP traffic? Additionally you can trace which applications open which connections on each machine.