12 ms·

Ask HN: Was I pwned? [resolved]

189 points by wasipwned 4y ago

A few days ago, I noticed that my home network performance would degrade substantially to the point of being unusable. I would just power-cycle all my switches, and the issue would resolve for a while. It happened again this morning, so I decided to try to look closer at what could be causing the issue. That's when I noticed that my Linux desktop was doing a lot of traffic, and here's what I observed: - My desktop has a private IP address, let's say 10.0.0.2. - Running `iftop`, I saw all the traffic coming from a different source IP address, 10.0.0.3. It was transferring ~300Mbps. - Running `tcpdump`, I saw that all of this traffic was going to a public IP address (AT&T). All of the source port/dest were ipsec-nat-t. - I saw that `10.0.0.3` showed up as a client on my switch with a randomized MAC address (presumably, since I couldn't find the MAC prefix in a vendor list). - I could not find any references to `10.0.0.3` or the random MAC address on my desktop (looking at kernel logs, system logs, ip a, ifconfig). - During this period, my network was degraded (high packet loss across my switches). It was at this point that I decided to try blocking the MAC address from my switch, and performance immediately returned to normal. I tried unblocking the MAC a few minutes later, but it has yet to return. That plus the fact that the issue happens at seemingly random times (especially the middle of the night) makes me think that it's not automatically connecting and instead being triggered remotely. I've since disconnected my desktop from the network and am in the process of rotating keys. I'm especially perplexed at the traffic showing up from a different source IP on my desktop, but I did not see any interface that matched. I tried to look and see if it was potentially a VM running, but I didn't see anything in virsh. I did have Docker containers running, but I assume I would have seen the IP address show up on one of my interfaces. I'm at a bit of a loss and was wondering if anyone has ever seen anything like this before, and if there is any suggestions for things I should check.

jatin2302 4y ago
Looks like RAT Or it could be torrent running in background or some sync services for any storage app.
cyberpunk 4y ago
I mean. Yes, you're pwned. You need to reflash all possible firmware, dd your disks to zeros and start again.. I wouldn't even trust it then, personally. What's the IP address it was talking to? Maybe we can help find out what it was?
- wasipwned 4y ago
  Yeah. That's definitely the plan, but I want to see if there's anything I can learn from the machine before I even do so. The IP address was 107.122.31.71.
  - cyberpunk 4y ago
    It looks like a consumer IP address (AT&T US); ungood. Either your attacker is unsophisticated and they actually had that IP, or they're bouncing through a pwned machine. Kill it. Kill it with fire.
  - ehPReth 4y ago
    one open port it seems: https://www.shodan.io/host/107.122.31.71 https://www.shodan.io/host/107.122.31.71
    - deleted 4y ago
      [deleted]
  - KomoD 4y ago
    Well, I've dug around, there's no hostname associated with it or pointing to it. There's no obvious connections to any orgs or sites, and no entries in virustotal or abuseipdb, however there is an open port 179 (looks like BGP??).
    - Atlas22 4y ago
      Its a carrier grade NAT IP for ATT's cellular service, the BGP is likely just the ATT router.
      - wasipwned 4y ago
        Yeah, I'm now realizing that this might be multicast traffic I'm seeing from another device, and I do use AT&T which is making me think this may not actually be malicious.
      - 4y ago
yabones 4y ago
I think we need more information. Do you run any services on that machine that would be exposed? Do you port-forward to that box? Use a VPN or something like Tailscale? Or perhaps a sync client like syncthing, onedrive, nextcloud, etc. could be to blame. One option would be to log all traffic on that machine to a .pcap and feed it through some IDS analyzers.
- wasipwned 4y ago
  Nothing is exposed directly to the internet, but I had some development services that were accessible on my private network. I do use Dropbox, but the odd part was it seemingly IPsec traffic. I really should have grabbed a pcap when it was occurring. I only have a screenshot of tcpdump which is not very useful.
  - yabones 4y ago
    Was it IPsec traffic, or just something running on the IPsec port?
    - wasipwned 4y ago
      Unfortunately, I did not save any packet captures, so I only saw that it was on the IPsec port.
  - CamperBob2 4y ago
    Dropbox would be my #1 suspect. They like to play fast and loose with your networking resources. I have seen Dropbox open lots of weird, undocumented ports for purposes their own support people aren't aware of. Case in point: https://pdfhost.io/v/1C9zgOibj_port_1298_1299_dialogue_with_dropbox https://pdfhost.io/v/1C9zgOibj_port_1298_1299_dialogue_with_... (23 kB .PDF) Kill your Dropbox process(es) -- there will probably be several, again for no clear reason -- and I'll bet this behavior stops. Whenever my system behaves in an unexpected way, I've learned to start the troubleshooting process by temporarily killing Dropbox.
bguebert 4y ago
Some cell phones will generate a random MAC address. Is 10.0.0.3 given out from your dhcp server? Maybe a phone syncing video to a cloud service.
- wasipwned 4y ago
  It was given an IP via DHCP (but not that specific IP, that was more for illustrative purposes). I'm currently ruling out that it is any other device given I'm seeing the traffic from my desktop, and it shouldn't be acting as a router for another physical device. But I'd like to know if I could be wrong about htat.
tetraca 4y ago
Do you have a corporate laptop or computer? Things like crowdstrike love to scan your network and phone home.
- wasipwned 4y ago
  There are corporate laptops on my network. I had originally thought maybe that could be related. But I can't explain how this source IP was showing up on a tcpdump from my desktop if that was the case, so at this point I'm assuming it's an issue originating from my desktop.
- userbinator 4y ago
  Another often-overlooked side of WFH. Devices you don't actually own should be kept on an isolated network away from everything else you do own. I also throttle their bandwidth and disconnect them outside of working hours. If companies can say "our network, our rules", I'll also do the same with mine.
erdos4d 4y ago
How is your desktop connected to the switch (ethernet or wifi)? If the computer is wired, maybe you have a virus on it and that is somehow using the wifi to get another IP? I would suggest you backup your user data and wipe/restore the desktop. If it comes back after that, I'd bet someone has cracked your wifi password and is getting in that way, or some other device on the network is the culprit and reinfecting your desktop.
- wasipwned 4y ago
  Ethernet, and wifi is disabled. But even if it was wifi or another network interface getting the IP, I should have been able to find it in ip a / ifconfig I'm assuming
  - erdos4d 4y ago
    Maybe I misunderstood what you wrote, but you feel the traffic is actually originating from your desktop? I am completely unaware how someone could get 2 IP addresses for your desktop without that showing up somewhere on the box (agreed, you should have found it with the tools you ran). I still think wipe and wait, but this seems like you got seriously hacked. Good luck.
    - wasipwned 4y ago
      I probably am mistaken that it's originating from my desktop. It is entirely possible that it is multicast traffic I am seeing. See https://news.ycombinator.com/item?id=33821387 https://news.ycombinator.com/item?id=33821387
deusum 4y ago
At least block the external IP and ports where the transfers are happening. Change the router password, some neighbor might be in the network. It sounds like it might be part of a DDoS campaign, as well. Hard to diagnose here.
Kikawala 4y ago
Do you happen to have a mobile phone with AT&T and are near Fremont, CA?
- wasipwned 4y ago
  Yes and yes.
  - Kikawala 4y ago
    See if the traffic goes away if you disconnect the phone from your wifi. It's most likely AT&T's wifi calling feature.
    - wasipwned 4y ago
      So interestingly, it looks like Unifi did classify the traffic as wifi calling, but it was doing a lot of traffic in the middle of the night when I was asleep. And the biggest question mark in my head is: how is this traffic looking like it's coming from my desktop?
      - wasipwned 4y ago
        And by looking it's coming from my desktop, I mean the tcpdump was run directly on my desktop and I saw the traffic there. So I assume it had to be routed through my desktop unless I am missing something.
        Kikawala 4y ago
        You can run the below command to see which process or PID is talking over ipsec-nat-t sudo lsof -n -i :4500
        FollowingTheDao 4y ago
        You are missing something. You are seeing BGP routing table updates on your network from an AT&T router. You have BGP running on your network somewhere.
kevin_thibedeau 4y ago
Keep it as a honeypot and run the replacement in a vlaned subnet off the currently owned router.
deleted 4y ago
[deleted]
FollowingTheDao 4y ago
Are any of your switches running a BGP service you do not know about? Could it be trying to send or receive a huge routing table?
- wasipwned 4y ago
  Not running BGP that I'm aware of. Network is comprised of mostly UniFi switches and one MikroTik switch.
  - FollowingTheDao 4y ago
    > Not running BGP that I'm aware of You need to find out because that is what is happening. Both UniFi and MikroTik switches support BGP. https://help.mikrotik.com/docs/display/ROS/BGP https://help.mikrotik.com/docs/display/ROS/BGP
itake 4y ago
Is your router patched? Maybe they hacked your router. Not sure why it would need to assign itself a new IP. Maybe there is a docker container running on the router?
- wasipwned 4y ago
  I'm using a UDM Pro, and I had just recently patched. The only container running on the router is unifi-os itself. I keep repeating myself because I want to make sure I can't be missing something, but tcpdump on my desktop is showing the traffic which is why I'm assuming that it's my desktop that is compromised.
  - FollowingTheDao 4y ago
    > I'm using a UDM Pro, and I had just recently patched. This is the issue. The patch either turned on BGP or has a BGP bug.
maineldc 4y ago
Older versions of Unifi controller were subject to mining hacks because of the Log4J compromise. I would check to make sure you are running a recent version of the Unifi Controller.
andrewf 4y ago
Are you running any virtual machines on your desktop? Because "My machine is 10.0.0.2 but there's something on the same physical host communicating as 10.0.0.3" is what it'd look like if a virtual machine with a "bridged" ethernet interface got its own IP address via DHCP and talked to the Internet. This is speculation, I don't know whether you were owned.
- wasipwned 4y ago
  Yeah, that was exactly what I was thinking as well. I do use KVM to run a few VMs, but they were the only VMs I could find, and they were both stopped the whole time.
guerby 4y ago
iproute2 things you could look at: ip ne # show the IP/MAC table ip rule # show the source routing state ip netns list # show network namespaces You could also transfer a trusted "ip" binary from another system in case yours is compromised (kernel could be compromised too)
nonane 4y ago
> - I saw that `10.0.0.3` showed up as a client on my switch with a randomized MAC address (presumably, since I couldn't find the MAC prefix in a vendor list). MAC address randomization is enabled by default on iOS: https://www.linksys.com/support-article?articleNum=317709 https://www.linksys.com/support-article?articleNum=317709
- wasipwned 4y ago
  Thanks for reminding me of this. This is looking less and less malicious at this point as `10.0.0.3` is my phone (using AT&T, which is where all the traffic was destined).
  - ghostbrainalpha 4y ago
    That still seems like an awful lot of traffic for your phone to be using. Do you mind sharing what gets you that kind of data usage? Just hours of FaceTime calling or something else?
    - wasipwned 4y ago
      I can't explain this part yet. I was asleep when this happened, so I wasn't even using my phone. I may be wrong about 300Mbps being to the AT&T. public IP, as my router shows a much lower rate. That might have just been the total traffic I was seeing internally on my private network from multicast.
      - smoyer 4y ago
        Maybe they've got your phone generating traffic to bill you for going over your limit?
wasipwned 4y ago
False alarm. Really appreciate everyone helping me sanity check this. The randomized MAC is part of iOS' Wi-Fi privacy, and my phone is using Wi-Fi calling for AT&T. The randomized MAC and the fact that I thought I saw the traffic originating from my desktop (it wasn't, it was just multicast traffic) really threw me off.
- CamperBob2 4y ago
  At 300 Mb/sec?
  - wasipwned 4y ago
    I can't fully explain this part yet, but I am currently expecting that I will see the issue again even with my desktop disconnected. I am also probably wrong about it being 300Mbps to AT&T. It was probably 300Mbps of multicast traffic internally.
    - kevin_nisbet 4y ago
      One possibility, although unlikely, is do you have a loop in your network. I think you mentioned multiple switches... does anything have multiple paths on the switching layer to form a loop?
      - wasipwned 4y ago
        Not to my knowledge, but I will double checking to make sure. The odd part is the issue only started recently, but I haven't made any major changes to my network recently.
        kevin_nisbet 4y ago
        Well, the fun thing about network loops, is they can run for years without a problem on a switched network. It's only when you run into a learning issue that switches fall back to flooding all ports except the source port, which then allows packets to flow through the loop indefinitely. This is why so many people footgun themselves disabling spanning tree, is it seems to work without it... until it doesn't.
guerrilla 4y ago
> I'm especially perplexed at the traffic showing up from a different source IP on my desktop, but I did not see any interface that matched This is easy to do with a raw socket, you just ARP for the IP. See fantaip in Unicornscan for example an example of software that can do that for you. So, all you need is root.
xur17 4y ago
I experienced a somewhat similar issue yesterday on my network that I described in detail here [0]. Essentially one of the computers (running ubuntu) on my network started sending a VERY high volume (it measured 20gb for the day, and I think it was all over a 10 minute period) of DNS traffic to my router, which runs an unbound instance for my network. That traffic (or at least I think it was that traffic) brought down my network to the point where I could even ping an external or internal ip address. Does tcpdump show the destination ip address the traffic was sent to on AT&T's network? Curious if that could be a dns server.. Also, what version of ubuntu is your desktop running, and what software does it have on it? Are you using canonical's livepatch service? [0] https://forum.opnsense.org/index.php?topic=31284.0 https://forum.opnsense.org/index.php?topic=31284.0
- wasipwned 4y ago
  Yes I posted the IP address here: https://news.ycombinator.com/item?id=33820749 https://news.ycombinator.com/item?id=33820749 and it appears to be AT&T's CGNAT IP address and communicating over port 4500 (IPsec), so the likely culprit is Wi-Fi calling which uses IPsec. I'm running Ubuntu 20.04. I don't use livepatch, but I do update/reboot frequently. I'm mostly running Chrome, Firefox, and Docker. Occasionally GIMP and LibreOffice.
LocalPCGuy 4y ago
Sounds like you found the issue but for future reference or for anyone stumbling across this later on, another thing to check is network ports in the "cheap" (i.e. generally most < $100) USB/USB-C hubs/port expanders with a power passthrough and a network port. I had a pretty bizarre experience where it would work just fine during the day while the computer was on, but when I'd shut the lid of my work MacBook, the network port on that little USB-C hub would just start sending off ACK signals like crazy, killing my network for anything else trying to use it (effectively denial of service myself). It was really hard to track down also because it wasn't "traffic" really, and it didn't happen on the devices that were impacted (i.e. I'd be using my Windows PC in the evening and that was attached to my work computer). Even more perplexing because it was semi-random - turned out it wasn't "random", it was when I shut the lid of my work laptop vs. just leaving it up and walking away. I finally saw the flood of traffic by dumping network traffic and was able to trace it back to that hub (first I thought my laptop was pwned and was doing something like exfiltrating data or mining when I wasn't logged in, but it was very definitely the hub after a bit more digging). Since discovering that, I have come across others that have written up the same or similar issues. With the power passthrough, the hub still has power, and if the network interface is flaky as many are, it can cause issues, particularly when the machine it's plugged into stops using it. This post has links to a few various write-ups: https://mjtsai.com/blog/2022/05/11/usb-c-hubs-breaking-ethernet-networks/ https://mjtsai.com/blog/2022/05/11/usb-c-hubs-breaking-ether...
harel 4y ago
Even though this was a false alarm in the end, the processes taken to investigate this merit an upvote and a save for future reference.
seussfrank 4y ago
i agree there have problems.
deleted 4y ago
[deleted]

189 points by wasipwned 4y ago

jatin2302 4y ago

Looks like RAT Or it could be torrent running in background or some sync services for any storage app.

cyberpunk 4y ago

I mean. Yes, you're pwned. You need to reflash all possible firmware, dd your disks to zeros and start again.. I wouldn't even trust it then, personally. What's the IP address it was talking to? Maybe we can help find out what it was?

wasipwned 4y ago
Yeah. That's definitely the plan, but I want to see if there's anything I can learn from the machine before I even do so. The IP address was 107.122.31.71.
- cyberpunk 4y ago
  It looks like a consumer IP address (AT&T US); ungood. Either your attacker is unsophisticated and they actually had that IP, or they're bouncing through a pwned machine. Kill it. Kill it with fire.
- ehPReth 4y ago
  one open port it seems: https://www.shodan.io/host/107.122.31.71 https://www.shodan.io/host/107.122.31.71
  - deleted 4y ago
    [deleted]
- KomoD 4y ago
  Well, I've dug around, there's no hostname associated with it or pointing to it. There's no obvious connections to any orgs or sites, and no entries in virustotal or abuseipdb, however there is an open port 179 (looks like BGP??).
  - Atlas22 4y ago
    Its a carrier grade NAT IP for ATT's cellular service, the BGP is likely just the ATT router.
    - wasipwned 4y ago
      Yeah, I'm now realizing that this might be multicast traffic I'm seeing from another device, and I do use AT&T which is making me think this may not actually be malicious.
    - 4y ago

yabones 4y ago

I think we need more information. Do you run any services on that machine that would be exposed? Do you port-forward to that box? Use a VPN or something like Tailscale? Or perhaps a sync client like syncthing, onedrive, nextcloud, etc. could be to blame. One option would be to log all traffic on that machine to a .pcap and feed it through some IDS analyzers.

wasipwned 4y ago
Nothing is exposed directly to the internet, but I had some development services that were accessible on my private network. I do use Dropbox, but the odd part was it seemingly IPsec traffic. I really should have grabbed a pcap when it was occurring. I only have a screenshot of tcpdump which is not very useful.
- yabones 4y ago
  Was it IPsec traffic, or just something running on the IPsec port?
  - wasipwned 4y ago
    Unfortunately, I did not save any packet captures, so I only saw that it was on the IPsec port.
- CamperBob2 4y ago
  Dropbox would be my #1 suspect. They like to play fast and loose with your networking resources. I have seen Dropbox open lots of weird, undocumented ports for purposes their own support people aren't aware of. Case in point: https://pdfhost.io/v/1C9zgOibj_port_1298_1299_dialogue_with_dropbox https://pdfhost.io/v/1C9zgOibj_port_1298_1299_dialogue_with_... (23 kB .PDF) Kill your Dropbox process(es) -- there will probably be several, again for no clear reason -- and I'll bet this behavior stops. Whenever my system behaves in an unexpected way, I've learned to start the troubleshooting process by temporarily killing Dropbox.

bguebert 4y ago

Some cell phones will generate a random MAC address. Is 10.0.0.3 given out from your dhcp server? Maybe a phone syncing video to a cloud service.

wasipwned 4y ago
It was given an IP via DHCP (but not that specific IP, that was more for illustrative purposes). I'm currently ruling out that it is any other device given I'm seeing the traffic from my desktop, and it shouldn't be acting as a router for another physical device. But I'd like to know if I could be wrong about htat.

tetraca 4y ago

Do you have a corporate laptop or computer? Things like crowdstrike love to scan your network and phone home.

wasipwned 4y ago
There are corporate laptops on my network. I had originally thought maybe that could be related. But I can't explain how this source IP was showing up on a tcpdump from my desktop if that was the case, so at this point I'm assuming it's an issue originating from my desktop.
userbinator 4y ago
Another often-overlooked side of WFH. Devices you don't actually own should be kept on an isolated network away from everything else you do own. I also throttle their bandwidth and disconnect them outside of working hours. If companies can say "our network, our rules", I'll also do the same with mine.

erdos4d 4y ago

How is your desktop connected to the switch (ethernet or wifi)? If the computer is wired, maybe you have a virus on it and that is somehow using the wifi to get another IP? I would suggest you backup your user data and wipe/restore the desktop. If it comes back after that, I'd bet someone has cracked your wifi password and is getting in that way, or some other device on the network is the culprit and reinfecting your desktop.

wasipwned 4y ago
Ethernet, and wifi is disabled. But even if it was wifi or another network interface getting the IP, I should have been able to find it in ip a / ifconfig I'm assuming
- erdos4d 4y ago
  Maybe I misunderstood what you wrote, but you feel the traffic is actually originating from your desktop? I am completely unaware how someone could get 2 IP addresses for your desktop without that showing up somewhere on the box (agreed, you should have found it with the tools you ran). I still think wipe and wait, but this seems like you got seriously hacked. Good luck.
  - wasipwned 4y ago
    I probably am mistaken that it's originating from my desktop. It is entirely possible that it is multicast traffic I am seeing. See https://news.ycombinator.com/item?id=33821387 https://news.ycombinator.com/item?id=33821387

deusum 4y ago

At least block the external IP and ports where the transfers are happening. Change the router password, some neighbor might be in the network. It sounds like it might be part of a DDoS campaign, as well. Hard to diagnose here.

Kikawala 4y ago

Do you happen to have a mobile phone with AT&T and are near Fremont, CA?

wasipwned 4y ago
Yes and yes.
- Kikawala 4y ago
  See if the traffic goes away if you disconnect the phone from your wifi. It's most likely AT&T's wifi calling feature.
  - wasipwned 4y ago
    So interestingly, it looks like Unifi did classify the traffic as wifi calling, but it was doing a lot of traffic in the middle of the night when I was asleep. And the biggest question mark in my head is: how is this traffic looking like it's coming from my desktop?
    - wasipwned 4y ago
      And by looking it's coming from my desktop, I mean the tcpdump was run directly on my desktop and I saw the traffic there. So I assume it had to be routed through my desktop unless I am missing something.
      - Kikawala 4y ago
        You can run the below command to see which process or PID is talking over ipsec-nat-t sudo lsof -n -i :4500
      - FollowingTheDao 4y ago
        You are missing something. You are seeing BGP routing table updates on your network from an AT&T router. You have BGP running on your network somewhere.

kevin_thibedeau 4y ago

Keep it as a honeypot and run the replacement in a vlaned subnet off the currently owned router.

deleted 4y ago

[deleted]

FollowingTheDao 4y ago

Are any of your switches running a BGP service you do not know about? Could it be trying to send or receive a huge routing table?

wasipwned 4y ago
Not running BGP that I'm aware of. Network is comprised of mostly UniFi switches and one MikroTik switch.
- FollowingTheDao 4y ago
  > Not running BGP that I'm aware of You need to find out because that is what is happening. Both UniFi and MikroTik switches support BGP. https://help.mikrotik.com/docs/display/ROS/BGP https://help.mikrotik.com/docs/display/ROS/BGP

itake 4y ago

Is your router patched? Maybe they hacked your router. Not sure why it would need to assign itself a new IP. Maybe there is a docker container running on the router?

wasipwned 4y ago
I'm using a UDM Pro, and I had just recently patched. The only container running on the router is unifi-os itself. I keep repeating myself because I want to make sure I can't be missing something, but tcpdump on my desktop is showing the traffic which is why I'm assuming that it's my desktop that is compromised.
- FollowingTheDao 4y ago
  > I'm using a UDM Pro, and I had just recently patched. This is the issue. The patch either turned on BGP or has a BGP bug.

maineldc 4y ago

Older versions of Unifi controller were subject to mining hacks because of the Log4J compromise. I would check to make sure you are running a recent version of the Unifi Controller.

andrewf 4y ago

Are you running any virtual machines on your desktop? Because "My machine is 10.0.0.2 but there's something on the same physical host communicating as 10.0.0.3" is what it'd look like if a virtual machine with a "bridged" ethernet interface got its own IP address via DHCP and talked to the Internet. This is speculation, I don't know whether you were owned.

wasipwned 4y ago
Yeah, that was exactly what I was thinking as well. I do use KVM to run a few VMs, but they were the only VMs I could find, and they were both stopped the whole time.

guerby 4y ago

iproute2 things you could look at: ip ne # show the IP/MAC table ip rule # show the source routing state ip netns list # show network namespaces You could also transfer a trusted "ip" binary from another system in case yours is compromised (kernel could be compromised too)

nonane 4y ago

> - I saw that `10.0.0.3` showed up as a client on my switch with a randomized MAC address (presumably, since I couldn't find the MAC prefix in a vendor list). MAC address randomization is enabled by default on iOS: https://www.linksys.com/support-article?articleNum=317709 https://www.linksys.com/support-article?articleNum=317709

wasipwned 4y ago
Thanks for reminding me of this. This is looking less and less malicious at this point as `10.0.0.3` is my phone (using AT&T, which is where all the traffic was destined).
- ghostbrainalpha 4y ago
  That still seems like an awful lot of traffic for your phone to be using. Do you mind sharing what gets you that kind of data usage? Just hours of FaceTime calling or something else?
  - wasipwned 4y ago
    I can't explain this part yet. I was asleep when this happened, so I wasn't even using my phone. I may be wrong about 300Mbps being to the AT&T. public IP, as my router shows a much lower rate. That might have just been the total traffic I was seeing internally on my private network from multicast.
    - smoyer 4y ago
      Maybe they've got your phone generating traffic to bill you for going over your limit?

wasipwned 4y ago

False alarm. Really appreciate everyone helping me sanity check this. The randomized MAC is part of iOS' Wi-Fi privacy, and my phone is using Wi-Fi calling for AT&T. The randomized MAC and the fact that I thought I saw the traffic originating from my desktop (it wasn't, it was just multicast traffic) really threw me off.

CamperBob2 4y ago
At 300 Mb/sec?
- wasipwned 4y ago
  I can't fully explain this part yet, but I am currently expecting that I will see the issue again even with my desktop disconnected. I am also probably wrong about it being 300Mbps to AT&T. It was probably 300Mbps of multicast traffic internally.
  - kevin_nisbet 4y ago
    One possibility, although unlikely, is do you have a loop in your network. I think you mentioned multiple switches... does anything have multiple paths on the switching layer to form a loop?
    - wasipwned 4y ago
      Not to my knowledge, but I will double checking to make sure. The odd part is the issue only started recently, but I haven't made any major changes to my network recently.
      - kevin_nisbet 4y ago
        Well, the fun thing about network loops, is they can run for years without a problem on a switched network. It's only when you run into a learning issue that switches fall back to flooding all ports except the source port, which then allows packets to flow through the loop indefinitely. This is why so many people footgun themselves disabling spanning tree, is it seems to work without it... until it doesn't.

guerrilla 4y ago

> I'm especially perplexed at the traffic showing up from a different source IP on my desktop, but I did not see any interface that matched This is easy to do with a raw socket, you just ARP for the IP. See fantaip in Unicornscan for example an example of software that can do that for you. So, all you need is root.

xur17 4y ago

I experienced a somewhat similar issue yesterday on my network that I described in detail here [0]. Essentially one of the computers (running ubuntu) on my network started sending a VERY high volume (it measured 20gb for the day, and I think it was all over a 10 minute period) of DNS traffic to my router, which runs an unbound instance for my network. That traffic (or at least I think it was that traffic) brought down my network to the point where I could even ping an external or internal ip address. Does tcpdump show the destination ip address the traffic was sent to on AT&T's network? Curious if that could be a dns server.. Also, what version of ubuntu is your desktop running, and what software does it have on it? Are you using canonical's livepatch service? [0] https://forum.opnsense.org/index.php?topic=31284.0 https://forum.opnsense.org/index.php?topic=31284.0

wasipwned 4y ago
Yes I posted the IP address here: https://news.ycombinator.com/item?id=33820749 https://news.ycombinator.com/item?id=33820749 and it appears to be AT&T's CGNAT IP address and communicating over port 4500 (IPsec), so the likely culprit is Wi-Fi calling which uses IPsec. I'm running Ubuntu 20.04. I don't use livepatch, but I do update/reboot frequently. I'm mostly running Chrome, Firefox, and Docker. Occasionally GIMP and LibreOffice.

LocalPCGuy 4y ago

Sounds like you found the issue but for future reference or for anyone stumbling across this later on, another thing to check is network ports in the "cheap" (i.e. generally most < $100) USB/USB-C hubs/port expanders with a power passthrough and a network port. I had a pretty bizarre experience where it would work just fine during the day while the computer was on, but when I'd shut the lid of my work MacBook, the network port on that little USB-C hub would just start sending off ACK signals like crazy, killing my network for anything else trying to use it (effectively denial of service myself). It was really hard to track down also because it wasn't "traffic" really, and it didn't happen on the devices that were impacted (i.e. I'd be using my Windows PC in the evening and that was attached to my work computer). Even more perplexing because it was semi-random - turned out it wasn't "random", it was when I shut the lid of my work laptop vs. just leaving it up and walking away. I finally saw the flood of traffic by dumping network traffic and was able to trace it back to that hub (first I thought my laptop was pwned and was doing something like exfiltrating data or mining when I wasn't logged in, but it was very definitely the hub after a bit more digging). Since discovering that, I have come across others that have written up the same or similar issues. With the power passthrough, the hub still has power, and if the network interface is flaky as many are, it can cause issues, particularly when the machine it's plugged into stops using it. This post has links to a few various write-ups: https://mjtsai.com/blog/2022/05/11/usb-c-hubs-breaking-ethernet-networks/ https://mjtsai.com/blog/2022/05/11/usb-c-hubs-breaking-ether...

harel 4y ago

Even though this was a false alarm in the end, the processes taken to investigate this merit an upvote and a save for future reference.

seussfrank 4y ago

i agree there have problems.

deleted 4y ago

[deleted]