4 ms·

> Lumen identified over 330,000 unique IP addresses that communicated with one of 75 observed C2 nodes How does Black Lotus Labs global telemetry know which IP

> Lumen identified over 330,000 unique IP addresses that communicated with one of 75 observed C2 nodes How does Black Lotus Labs global telemetry know which IP communicated with which other IP if they have control of neither endpoint? Who/what is keeping traffic logs? If these guys can do it, remind me again how Tor is secure because nobody could possibly be able to follow packets from your machine, through the onion hops, to the exit node where the same packet is available unencrypted...

hedora 2y ago
This is reasonably standard functionality for backbone routers. They have to parse the TCP headers in hardware anyway, and can track common endpoints with O(1) state. Of course, on the other end of the spectrum, the NSA has tapped into core internet links, is recording everything it possibly can, and is keeping it forever.
- Hikikomori 2y ago
  Yup. Pretty much all ISPs collect sflow/netflow from their devices to be able to debug problems or detect ddos.
- sebzim4500 2y ago
  Is that actually feasible with their budget? If we are generous and assume there a zettabyte of data a year that they want to store. At consumer prices, you would have to pay $10B per year just buying hard drives yet alone the operational costs/redundancy. The budget for all of the US intelligence services is ~$65B. I think if they wanted to actually do what you are describing it would be the single biggest intelligence expense they have and I don't see how you hide that.
  - deleted 2y ago
    [deleted]
  - choilive 2y ago
    Its not. They don't store the raw IP packet data, instead they store the metadata (this was revealed in a leak a long time ago), like the type in this article (data source and destination, timestamps, size of the data, etc.) the metadata is orders of magnitude less data than the raw packets and likely easily compressible, so I wouldn't be surprised if they keep it all for a decent chunk of time.
    - sebzim4500 2y ago
      Presumably this becomes increasingly useless in an eSNI world? i.e. they can prove I visited a bunch of cloudflare sites, but there are millions of those so who cares?
- oasisbob 2y ago
  > They have to parse the TCP headers in hardware anyway Backbone routers have no need to implement stateful TCP inspection or deal with the transport layer for TCP, dealing with IP is enough.
luma 2y ago
Presumably, Windstream is logging customer traffic as a matter of course. It might just be metadata (NetFlow/sFlow/IPFIX/etc), but one way or the other the only way they have this information is if they are recording and retaining it. Hopefully this is made clear in Windstream's contract terms.
- londons_explore 2y ago
  These aren't likely 'top flows', since the C&C data will probably only be a few kilobytes. So to capture this, you at a minimum need to be logging every TCP connection's SRC IP and DST IP. And they seem pretty confident in their worldwide map and fairly exact counts, so I would guess they must have probes covering most of the world doing this monitoring, and it likely isn't just 1-in-a-million sampling either...
  - luma 2y ago
    For whatever it's worth, what you describe above is specifically what IPFIX/NetFlow etc does. Not full-take, just metadata for each flow such as the time, src/dst ip/port, tcp sequence #, octets sent, etc. This is common in datacenters for traffic and flow analysis for troubleshooting, capacity planning, and the occasional incident response. More details: https://en.wikipedia.org/wiki/IP_Flow_Information_Export https://en.wikipedia.org/wiki/IP_Flow_Information_Export
shrubble 2y ago
Lumen (merger of Level3 and CenturyLink) sells services to a large part of the Internet and may provide a lot of the backhaul for Windstream. In which case they would be in the path for monitoring.
vieinfernale 2y ago
I'm quite disenchanted here. So this means that it is practically impossible to avoid IP fingerprints in any way ? Even with Tor, VMs, etc ? You'll always be at the mercy of whoever runs the show unless you own the physical servers
- miohtama 2y ago
  The physical servers do not matter. Someone owns the physical cable.
- semiquaver 2y ago
  Of course a backbone provider can directly inspect the source and destination IP addresses of any traffic transiting its network. How could it be otherwise? Thats not fingerprinting, it’s just pulling fields out of a struct. Tor does defeat this though. Rather than seeing the true destination of your traffic they see that of a Tor exit node.
  - londons_explore 2y ago
    But... That tor exit node then sends the traffic onwards... Again via the internet, and the backbone provider can inspect it again. Seeing a packet heading to a tor exit node and then a similarly sized packet heading onwards a fraction of a millisecond later is a pretty surefire way to spy on individual tor users.
    - Thorrez 2y ago
      I think Tor tries to resize/split/join packets a bit. And each Tor node will in theory be carrying traffic for many different users simultaneously. And Tor uses 3 nodes, each in a different country. So it's not quite as trivial as you make it sound. If 1, 2, or possibly all 3 nodes are run by a malicious actor, deanonymization becomes easier. At one point 10% of nodes were run by a single malicious actor: https://therecord.media/a-mysterious-threat-actor-is-running-hundreds-of-malicious-tor-relays https://therecord.media/a-mysterious-threat-actor-is-running...
    - 2y ago
codexon 2y ago
Lumen is a tier 1 network so a lot of traffic passes through them. They can man-in-the-middle the traffic and see the TCP packets going through their network.
- perlgeek 2y ago
  "They can man-in-the-middle the traffic" could be interpreted as them having to actively do something to become the man in the middle, when they already are. It's likely they just do sampling (think netflow) to get some statistics over the data that's already transiting their network.
rpcope1 2y ago
I have a friend who works at Black Lotus (and who may have written this blog post, who knows). Black Lotus is part of Lumen which is Level3 and CenturyLink and is one of the biggest (if not the biggest) backbone traffic provider in the world, with a huge percentage of the worlds traffic transiting their network, and thus I think they get direct insight into the traffic including metrics on it.
kbenson 2y ago
> If these guys can do it, remind me again how Tor is secure because nobody could possibly be able to follow packets from your machine, through the onion hops, to the exit node where the same packet is available unencrypted... You're supposed to be protected by the fact that you're going through multiple nodes before exiting TOR, and traffic should be mixed. Can you find some streams if you have most/all the nodes within your network and can analyze the traffic? Probably some, but the more traffic a node handles the harder it would be. There is a simpler approach though, which is to just run exit nodes.[1] 1: https://en.wikipedia.org/wiki/Tor_(network)#Exit_node_eavesdropping https://en.wikipedia.org/wiki/Tor_(network)#Exit_node_eavesd...
- 1oooqooq 2y ago
  the only problem with tor are exit nodes. they should not exist. both for safety and ending abusers.
  - Thorrez 2y ago
    What do you mean? Can Tor even function if exit nodes don't exist?
- Thorrez 2y ago
  What do you mean by just run exit nodes? The linked section says that just running exit nodes allows the exit node to steal data sent over plain HTTP. Is that actually a problem? Who's using plain HTTP? The linked section says just running exit nodes doesn't allow deanonymization.
  - kbenson 2y ago
    I didn't mean to imply it was exactly analogous, just a lot simpler, and there is a lot to be gleaned from that data. In fact, I would assume the more data you pass over it (e.g. if you proxy all your traffic across TOR) the easier it is to make assumptions about the source, unless it actively splits it across different exit nodes, which seems like it could be problematic in a lot of cases. If you have a DigitalOcean personal server you run some services on and you access it through TOR... well you might have just made the job of anyone trying to deanonymize you much easier.

hedora 2y ago

This is reasonably standard functionality for backbone routers. They have to parse the TCP headers in hardware anyway, and can track common endpoints with O(1) state. Of course, on the other end of the spectrum, the NSA has tapped into core internet links, is recording everything it possibly can, and is keeping it forever.

Hikikomori 2y ago
Yup. Pretty much all ISPs collect sflow/netflow from their devices to be able to debug problems or detect ddos.
sebzim4500 2y ago
Is that actually feasible with their budget? If we are generous and assume there a zettabyte of data a year that they want to store. At consumer prices, you would have to pay $10B per year just buying hard drives yet alone the operational costs/redundancy. The budget for all of the US intelligence services is ~$65B. I think if they wanted to actually do what you are describing it would be the single biggest intelligence expense they have and I don't see how you hide that.
- deleted 2y ago
  [deleted]
- choilive 2y ago
  Its not. They don't store the raw IP packet data, instead they store the metadata (this was revealed in a leak a long time ago), like the type in this article (data source and destination, timestamps, size of the data, etc.) the metadata is orders of magnitude less data than the raw packets and likely easily compressible, so I wouldn't be surprised if they keep it all for a decent chunk of time.
  - sebzim4500 2y ago
    Presumably this becomes increasingly useless in an eSNI world? i.e. they can prove I visited a bunch of cloudflare sites, but there are millions of those so who cares?
oasisbob 2y ago
> They have to parse the TCP headers in hardware anyway Backbone routers have no need to implement stateful TCP inspection or deal with the transport layer for TCP, dealing with IP is enough.

luma 2y ago

Presumably, Windstream is logging customer traffic as a matter of course. It might just be metadata (NetFlow/sFlow/IPFIX/etc), but one way or the other the only way they have this information is if they are recording and retaining it. Hopefully this is made clear in Windstream's contract terms.

londons_explore 2y ago
These aren't likely 'top flows', since the C&C data will probably only be a few kilobytes. So to capture this, you at a minimum need to be logging every TCP connection's SRC IP and DST IP. And they seem pretty confident in their worldwide map and fairly exact counts, so I would guess they must have probes covering most of the world doing this monitoring, and it likely isn't just 1-in-a-million sampling either...
- luma 2y ago
  For whatever it's worth, what you describe above is specifically what IPFIX/NetFlow etc does. Not full-take, just metadata for each flow such as the time, src/dst ip/port, tcp sequence #, octets sent, etc. This is common in datacenters for traffic and flow analysis for troubleshooting, capacity planning, and the occasional incident response. More details: https://en.wikipedia.org/wiki/IP_Flow_Information_Export https://en.wikipedia.org/wiki/IP_Flow_Information_Export

shrubble 2y ago

Lumen (merger of Level3 and CenturyLink) sells services to a large part of the Internet and may provide a lot of the backhaul for Windstream. In which case they would be in the path for monitoring.

vieinfernale 2y ago

I'm quite disenchanted here. So this means that it is practically impossible to avoid IP fingerprints in any way ? Even with Tor, VMs, etc ? You'll always be at the mercy of whoever runs the show unless you own the physical servers

miohtama 2y ago
The physical servers do not matter. Someone owns the physical cable.
semiquaver 2y ago
Of course a backbone provider can directly inspect the source and destination IP addresses of any traffic transiting its network. How could it be otherwise? Thats not fingerprinting, it’s just pulling fields out of a struct. Tor does defeat this though. Rather than seeing the true destination of your traffic they see that of a Tor exit node.
- londons_explore 2y ago
  But... That tor exit node then sends the traffic onwards... Again via the internet, and the backbone provider can inspect it again. Seeing a packet heading to a tor exit node and then a similarly sized packet heading onwards a fraction of a millisecond later is a pretty surefire way to spy on individual tor users.
  - Thorrez 2y ago
    I think Tor tries to resize/split/join packets a bit. And each Tor node will in theory be carrying traffic for many different users simultaneously. And Tor uses 3 nodes, each in a different country. So it's not quite as trivial as you make it sound. If 1, 2, or possibly all 3 nodes are run by a malicious actor, deanonymization becomes easier. At one point 10% of nodes were run by a single malicious actor: https://therecord.media/a-mysterious-threat-actor-is-running-hundreds-of-malicious-tor-relays https://therecord.media/a-mysterious-threat-actor-is-running...
  - 2y ago

codexon 2y ago

Lumen is a tier 1 network so a lot of traffic passes through them. They can man-in-the-middle the traffic and see the TCP packets going through their network.

perlgeek 2y ago
"They can man-in-the-middle the traffic" could be interpreted as them having to actively do something to become the man in the middle, when they already are. It's likely they just do sampling (think netflow) to get some statistics over the data that's already transiting their network.

rpcope1 2y ago

I have a friend who works at Black Lotus (and who may have written this blog post, who knows). Black Lotus is part of Lumen which is Level3 and CenturyLink and is one of the biggest (if not the biggest) backbone traffic provider in the world, with a huge percentage of the worlds traffic transiting their network, and thus I think they get direct insight into the traffic including metrics on it.

kbenson 2y ago

> If these guys can do it, remind me again how Tor is secure because nobody could possibly be able to follow packets from your machine, through the onion hops, to the exit node where the same packet is available unencrypted... You're supposed to be protected by the fact that you're going through multiple nodes before exiting TOR, and traffic should be mixed. Can you find some streams if you have most/all the nodes within your network and can analyze the traffic? Probably some, but the more traffic a node handles the harder it would be. There is a simpler approach though, which is to just run exit nodes.[1] 1: https://en.wikipedia.org/wiki/Tor_(network)#Exit_node_eavesdropping https://en.wikipedia.org/wiki/Tor_(network)#Exit_node_eavesd...

1oooqooq 2y ago
the only problem with tor are exit nodes. they should not exist. both for safety and ending abusers.
- Thorrez 2y ago
  What do you mean? Can Tor even function if exit nodes don't exist?
Thorrez 2y ago
What do you mean by just run exit nodes? The linked section says that just running exit nodes allows the exit node to steal data sent over plain HTTP. Is that actually a problem? Who's using plain HTTP? The linked section says just running exit nodes doesn't allow deanonymization.
- kbenson 2y ago
  I didn't mean to imply it was exactly analogous, just a lot simpler, and there is a lot to be gleaned from that data. In fact, I would assume the more data you pass over it (e.g. if you proxy all your traffic across TOR) the easier it is to make assumptions about the source, unless it actively splits it across different exit nodes, which seems like it could be problematic in a lot of cases. If you have a DigitalOcean personal server you run some services on and you access it through TOR... well you might have just made the job of anyone trying to deanonymize you much easier.