7 ms·

Analysis of a large brute force attack campaign against Windows Remote Desktop

71 points by rmdoss 4y ago

tg180 4y ago
I had to ban Flyservers on many occasions too. Given the frequency with which these incidents happen, I am more inclined to think they are a bulletproof hoster of some kind ... At a certain point it becomes difficult to attribute things to “incompetence” rather than malicious intent
technion 4y ago
As someone who supports a lot of Windows environments - yes, it's incredible the rate at which this occurs. I'd strongly point people at the RD Gateway MFA plugin as a requisite for publishing RDS these days: https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extension-rdg https://docs.microsoft.com/en-us/azure/active-directory/auth... It's also supported to publish behind the Azure App Proxy, avoiding the need for any port forwards at all: https://docs.microsoft.com/en-us/azure/active-directory/app-proxy/application-proxy-integrate-with-remote-desktop-services https://docs.microsoft.com/en-us/azure/active-directory/app-...
- vladvasiliu 4y ago
  > I'd strongly point people at the RD Gateway MFA plugin Is this working well? I've tried setting this up a few weeks ago and found it to be somewhat brittle. The MFA requests would sometimes be delayed long enough that the GW would timeout. I've tested this on Windows 2022.
  - technion 4y ago
    I've a whole range of clients running it without an issue. I will say however, the only MFA it supports is push (ie, TOTP doesn't work), and if you're not familiar with that, be aware many Android installations snooze that app and the push doesn't work unless you specifically have the app open at the time you logon.
  - ptk 4y ago
    We’ve been using it for years and it’s always been irritating in the way you describe. It seems if you miss the first push notification, you’ll receive a flurry more, many of which don’t really seem to do anything even if you do approve them. It’s possible that I have the timeouts set incorrectly, but I’ve followed Microsoft’s guidelines. Unfortunately we don’t have a better solution, so we suffer with it.
  - greatquux 4y ago
    We use Rublon for adding MFA to Remote Desktop and other VPNs too and it works very well. It’s not an over engineered monstrosity like Microsoft solutions.
- stinos 4y ago
  You'd recommend a standard trick like changing the port? Just doing that dropped the number of attempts to almost zero here.
  - technion 4y ago
    I think that's down to the problem you're trying to solve. Much like SSH brute force, if you know the only account is yours and it has an appropriately reasonable password and you can trust yourself not to get phished (say, you have a system managing the logon) then that's probably fine. The second there are other people with dumb passwords, or many other risk factors at play, I'd really suggest you need something doing MFA which unfortunately, MS doesn't offer out of the box on RDP.
  - r00f 4y ago
    You never know which 0-day gets discovered tomorrow, so even strong password might not be enough. I always change default ports on both SSH and RDP. It helps falling out of scope of unsofisticatrd scanners, keeps logs a bit cleaner plus doesn't waste CPU on useless auth attempts. Keeping ports behind VPN or port knocking is even better, but not always feasible
Neil44 4y ago
There’s a pretty cheap program that will do auto blocking, it’s easily googleable. Also changing the default port will cut down the numbers. I’ve had one hacked over RD gateway so that alone isn’t the answer. On our cloud windows servers we use a remote management agent to access, this obvs has its own trust issues but overall works well.
TazeTSchnitzel 4y ago
The difference between type 3 and type 10 is probably the difference between providing the username and password when connecting to RDP or after already connecting. In the latter case it's “interactive”: you enter the password via the GUI of the server you connected to.
gambiting 4y ago
Yep, I have a single Windows Server machine with an RDP port(non default) exposed to the world and it gets tens of thousands of failed logins every day. It has been for the past few years. I have no idea who is wasting their time on this or why, but as far as I can tell it's costing me nothing, so other than setting up an alert for a successful login I decided to simply ignore it.
frogger8 4y ago
https://github.com/devnulli/EvlWatcher https://github.com/devnulli/EvlWatcher README… It's basically a fail2ban for windows. Its goals are also mainly what we love about fail2ban: pre-configured no-initial-ducking-around-with-scripts-or-config-files install-and-forget You can download it here ( v2.1.5 - April 2022 ) . Also, we love issues! If anyone needs something or has questions about something, please feel free to open an issue. We are especially happy to get issues about log-entry samples we don't react on, or ideas of how we can support more protocols. A bit more detailed description of what EvlWatcher does. Scenario: there are those bad people out there, hammering your service (RDP and whatnot) with brute force attempts. You can see them and their IPs clearly in the Windows Event-Log. You have searched the web and yea, there are plenty of tools, scripts, and all that, to read the event-log and automatically ban the attackers IP. You however, are lazy. You need something like fail2ban, with a preconfigured set of rules to just RUN right away and it works. But then, it still needs enough flexibility for you to completely configure it, should you wish to do so. EvlWatcher does that. It scans the Windows-Event-Log, and reacts. It works by installing a service that scans the event log for unsuccessful login attempts. When one of its rules are violated (e.g. trying to log in without correct credentials, more than 5 times in 2 minutes), it will place that poor bastard into a generic firewall rule, and thereby ban the attacker for 2 hours. Also, when someone is repeatedly trying, there is a permanent ban list for that, where people defaultly land on when they've had three strikes. You can, of course, adjust the rules to your liking. They are basically a consisting of an event source, and a Regex to extract an IP, its pretty simple.
- EvanAnderson 4y ago
  My old ts_block[0] project does something similar to yours, albeit for RDP only and with much less sophisticated customization. I opted to go with a WMI Event Sink rather than polling the Event Log. I've never done a benchmark to see which architecture would use less CPU, but I can say the WMI event sink causes nearly instantaneous reaction. As an aside: I'd love to hear if somebody tries ts_block on Windows Server 2022. It works fine on 2012 R2 thru 2019 but I've never tried it on 2022. [0] https://github.com/EvanAnderson/ts_block https://github.com/EvanAnderson/ts_block
  - 4y ago
300bps 4y ago
Windows do not log the passwords they tried, so we can't know which ones were used Would be interesting to create a server daemon that listens on TCP port 3389 and implements the login portion of the RDP protocol to 1) Be able to gather information about the passwords used and 2) Experiment with ways to lock up or stall the scanners.
- h4waii 4y ago
  A few links which provide some good insight for your idea. https://research.nccgroup.com/2021/10/21/cracking-rdp-nla-supplied-credentials-for-threat-intelligence/ https://research.nccgroup.com/2021/10/21/cracking-rdp-nla-su... https://github.com/nccgroup/nlahoney https://github.com/nccgroup/nlahoney
dontbenebby 4y ago
Why don't more people use Fail2Ban? https://www.fail2ban.org/wiki/index.php/Main_Page https://www.fail2ban.org/wiki/index.php/Main_Page I avoided running servers since people are jerks but if I did that would be on my checklist of things to set up if I intended one to stick around long term. People seem to reinvent concepts from long ago for reasons I do not understand.
- droopyEyelids 4y ago
  Aside from cleaner logs what benefit do you expect from fail2ban?
  - zamadatix 4y ago
    It cuts down on the feasibility of generic internet scan + brute force attacks like described in this article. It's not an answer to specialized/focused attacks though cleaner logging may help you notice those.
    - droopyEyelids 4y ago
      If you use brute-forcable passwords it’s better to address that problem directly with password requirements rather than rely on fail2ban
      - zamadatix 4y ago
        It's best to do both as security isn't something delivered by a single solution. After all it's not like protocols such as RDP and SSH have been found to be infallible over the years, they've had side-channels, vulnerabilities detectable by scanning, and other weaknesses regardless whether your password was theoretically impossible to brute force when you made it. You never reach the point "I'm secure" you simply reach the point "adding more security measures costs more than the risk it removes".
        droopyEyelids 4y ago
LinuxBender 4y ago
They do not appear to be ramping up on any of my VPS nodes. I see the same averages to my tarpit ports on all my nodes. # VPS They all look similar to this iptables -L INPUT -n -v | grep TAR | sort -r | awk '{print $1"\t" $3"\t" $11}' 2680 TARPIT dpt:23 2657 TARPIT dpt:6379 170K TARPIT dpt:445 1186 TARPIT dpt:80 882 TARPIT dpt:22 757 TARPIT dpt:443 591 TARPIT dpt:5900 311 TARPIT dpt:21 296 TARPIT dpt:3389 119 TARPIT dpt:873 Some poorly coded bots appear to get stuck on 445 indefinitely. On the other hand I do see a vast increase on my home network to 5900 VNC. It is usually about double the hits to 3389 RDP # Home iptables -L INPUT -n -v | grep TAR | sort -r | awk '{print $1"\t" $3"\t" $11}' 9013 TARPIT dpt:5900 1628 TARPIT dpt:22 1274 TARPIT dpt:23 1092 TARPIT dpt:445 119 TARPIT dpt:3389 40 TARPIT dpt:21 [Edit] Here is a second home connection that has a higher uptime. Running tcpdump it looks like the latest VNC bots do not recognize and avoid tarpits. # Second Home connection iptables -L INPUT -n -v | grep TAR | sort -r | awk '{print $1"\t" $3"\t" $11}' 50532 TARPIT dpt:22 44301 TARPIT dpt:445 33520 TARPIT dpt:23 6718 TARPIT dpt:80 6439 TARPIT dpt:3389 184K TARPIT dpt:5900 1091 TARPIT dpt:21 271 TARPIT dpt:1900
ozim 4y ago
Well I never ever expose RDP to the internet directly. Allow only office IP or setup some VPN/jumphost via ssh whatever. I trust SSH much more than RDP to not have RCE/auth bypass bugs. SSH is also much easier to setup key auth or 2FA than in RDP if one like they mentioned in post are more used to setting up Linux boxes. After reading this blog post - I would not send any logs to their servers.
- greatquux 4y ago
  I think though it may be better without 2FA to block attacking IPs no matter what the protocol.
- CyanLite4 4y ago
  Exactly! I at first thought this was a rogue or disgruntled employee who wrote a blog post on the way out. But it appears to be a legit article sanctioned by their company. They literally opened a server without firewall rules for anyone to connect to their RDP port, and wrote a blog article to brag about it. They also have a Windows agent but they brag about not using Windows in the past 20 years…
johnklos 4y ago
You absolutely should blame Flyserver. They allow bots to run, and they don't do anything about it. There are too many network operators like this. Imagine if any networks where constant and ongoing illegal activity happens, and the network operators do nothing, eventually result in those networks being taken away. Things would change drastically.

71 points by rmdoss 4y ago

I had to ban Flyservers on many occasions too. Given the frequency with which these incidents happen, I am more inclined to think they are a bulletproof hoster of some kind ... At a certain point it becomes difficult to attribute things to “incompetence” rather than malicious intent

technion 4y ago

As someone who supports a lot of Windows environments - yes, it's incredible the rate at which this occurs. I'd strongly point people at the RD Gateway MFA plugin as a requisite for publishing RDS these days: https://docs.microsoft.com/en-us/azure/active-directory/authentication/howto-mfa-nps-extension-rdg https://docs.microsoft.com/en-us/azure/active-directory/auth... It's also supported to publish behind the Azure App Proxy, avoiding the need for any port forwards at all: https://docs.microsoft.com/en-us/azure/active-directory/app-proxy/application-proxy-integrate-with-remote-desktop-services https://docs.microsoft.com/en-us/azure/active-directory/app-...

vladvasiliu 4y ago
> I'd strongly point people at the RD Gateway MFA plugin Is this working well? I've tried setting this up a few weeks ago and found it to be somewhat brittle. The MFA requests would sometimes be delayed long enough that the GW would timeout. I've tested this on Windows 2022.
- technion 4y ago
  I've a whole range of clients running it without an issue. I will say however, the only MFA it supports is push (ie, TOTP doesn't work), and if you're not familiar with that, be aware many Android installations snooze that app and the push doesn't work unless you specifically have the app open at the time you logon.
- ptk 4y ago
  We’ve been using it for years and it’s always been irritating in the way you describe. It seems if you miss the first push notification, you’ll receive a flurry more, many of which don’t really seem to do anything even if you do approve them. It’s possible that I have the timeouts set incorrectly, but I’ve followed Microsoft’s guidelines. Unfortunately we don’t have a better solution, so we suffer with it.
- greatquux 4y ago
  We use Rublon for adding MFA to Remote Desktop and other VPNs too and it works very well. It’s not an over engineered monstrosity like Microsoft solutions.
stinos 4y ago
You'd recommend a standard trick like changing the port? Just doing that dropped the number of attempts to almost zero here.
- technion 4y ago
  I think that's down to the problem you're trying to solve. Much like SSH brute force, if you know the only account is yours and it has an appropriately reasonable password and you can trust yourself not to get phished (say, you have a system managing the logon) then that's probably fine. The second there are other people with dumb passwords, or many other risk factors at play, I'd really suggest you need something doing MFA which unfortunately, MS doesn't offer out of the box on RDP.
- r00f 4y ago
  You never know which 0-day gets discovered tomorrow, so even strong password might not be enough. I always change default ports on both SSH and RDP. It helps falling out of scope of unsofisticatrd scanners, keeps logs a bit cleaner plus doesn't waste CPU on useless auth attempts. Keeping ports behind VPN or port knocking is even better, but not always feasible

Neil44 4y ago

There’s a pretty cheap program that will do auto blocking, it’s easily googleable. Also changing the default port will cut down the numbers. I’ve had one hacked over RD gateway so that alone isn’t the answer. On our cloud windows servers we use a remote management agent to access, this obvs has its own trust issues but overall works well.

TazeTSchnitzel 4y ago

The difference between type 3 and type 10 is probably the difference between providing the username and password when connecting to RDP or after already connecting. In the latter case it's “interactive”: you enter the password via the GUI of the server you connected to.

gambiting 4y ago

Yep, I have a single Windows Server machine with an RDP port(non default) exposed to the world and it gets tens of thousands of failed logins every day. It has been for the past few years. I have no idea who is wasting their time on this or why, but as far as I can tell it's costing me nothing, so other than setting up an alert for a successful login I decided to simply ignore it.

frogger8 4y ago

https://github.com/devnulli/EvlWatcher https://github.com/devnulli/EvlWatcher README… It's basically a fail2ban for windows. Its goals are also mainly what we love about fail2ban: pre-configured no-initial-ducking-around-with-scripts-or-config-files install-and-forget You can download it here ( v2.1.5 - April 2022 ) . Also, we love issues! If anyone needs something or has questions about something, please feel free to open an issue. We are especially happy to get issues about log-entry samples we don't react on, or ideas of how we can support more protocols. A bit more detailed description of what EvlWatcher does. Scenario: there are those bad people out there, hammering your service (RDP and whatnot) with brute force attempts. You can see them and their IPs clearly in the Windows Event-Log. You have searched the web and yea, there are plenty of tools, scripts, and all that, to read the event-log and automatically ban the attackers IP. You however, are lazy. You need something like fail2ban, with a preconfigured set of rules to just RUN right away and it works. But then, it still needs enough flexibility for you to completely configure it, should you wish to do so. EvlWatcher does that. It scans the Windows-Event-Log, and reacts. It works by installing a service that scans the event log for unsuccessful login attempts. When one of its rules are violated (e.g. trying to log in without correct credentials, more than 5 times in 2 minutes), it will place that poor bastard into a generic firewall rule, and thereby ban the attacker for 2 hours. Also, when someone is repeatedly trying, there is a permanent ban list for that, where people defaultly land on when they've had three strikes. You can, of course, adjust the rules to your liking. They are basically a consisting of an event source, and a Regex to extract an IP, its pretty simple.

EvanAnderson 4y ago
My old ts_block[0] project does something similar to yours, albeit for RDP only and with much less sophisticated customization. I opted to go with a WMI Event Sink rather than polling the Event Log. I've never done a benchmark to see which architecture would use less CPU, but I can say the WMI event sink causes nearly instantaneous reaction. As an aside: I'd love to hear if somebody tries ts_block on Windows Server 2022. It works fine on 2012 R2 thru 2019 but I've never tried it on 2022. [0] https://github.com/EvanAnderson/ts_block https://github.com/EvanAnderson/ts_block
- 4y ago

300bps 4y ago

Windows do not log the passwords they tried, so we can't know which ones were used Would be interesting to create a server daemon that listens on TCP port 3389 and implements the login portion of the RDP protocol to 1) Be able to gather information about the passwords used and 2) Experiment with ways to lock up or stall the scanners.

h4waii 4y ago
A few links which provide some good insight for your idea. https://research.nccgroup.com/2021/10/21/cracking-rdp-nla-supplied-credentials-for-threat-intelligence/ https://research.nccgroup.com/2021/10/21/cracking-rdp-nla-su... https://github.com/nccgroup/nlahoney https://github.com/nccgroup/nlahoney

dontbenebby 4y ago

Why don't more people use Fail2Ban? https://www.fail2ban.org/wiki/index.php/Main_Page https://www.fail2ban.org/wiki/index.php/Main_Page I avoided running servers since people are jerks but if I did that would be on my checklist of things to set up if I intended one to stick around long term. People seem to reinvent concepts from long ago for reasons I do not understand.

droopyEyelids 4y ago
Aside from cleaner logs what benefit do you expect from fail2ban?
- zamadatix 4y ago
  It cuts down on the feasibility of generic internet scan + brute force attacks like described in this article. It's not an answer to specialized/focused attacks though cleaner logging may help you notice those.
  - droopyEyelids 4y ago
    If you use brute-forcable passwords it’s better to address that problem directly with password requirements rather than rely on fail2ban
    - zamadatix 4y ago
      It's best to do both as security isn't something delivered by a single solution. After all it's not like protocols such as RDP and SSH have been found to be infallible over the years, they've had side-channels, vulnerabilities detectable by scanning, and other weaknesses regardless whether your password was theoretically impossible to brute force when you made it. You never reach the point "I'm secure" you simply reach the point "adding more security measures costs more than the risk it removes".
      - droopyEyelids 4y ago

LinuxBender 4y ago

They do not appear to be ramping up on any of my VPS nodes. I see the same averages to my tarpit ports on all my nodes. # VPS They all look similar to this iptables -L INPUT -n -v | grep TAR | sort -r | awk '{print $1"\t" $3"\t" $11}' 2680 TARPIT dpt:23 2657 TARPIT dpt:6379 170K TARPIT dpt:445 1186 TARPIT dpt:80 882 TARPIT dpt:22 757 TARPIT dpt:443 591 TARPIT dpt:5900 311 TARPIT dpt:21 296 TARPIT dpt:3389 119 TARPIT dpt:873 Some poorly coded bots appear to get stuck on 445 indefinitely. On the other hand I do see a vast increase on my home network to 5900 VNC. It is usually about double the hits to 3389 RDP # Home iptables -L INPUT -n -v | grep TAR | sort -r | awk '{print $1"\t" $3"\t" $11}' 9013 TARPIT dpt:5900 1628 TARPIT dpt:22 1274 TARPIT dpt:23 1092 TARPIT dpt:445 119 TARPIT dpt:3389 40 TARPIT dpt:21 [Edit] Here is a second home connection that has a higher uptime. Running tcpdump it looks like the latest VNC bots do not recognize and avoid tarpits. # Second Home connection iptables -L INPUT -n -v | grep TAR | sort -r | awk '{print $1"\t" $3"\t" $11}' 50532 TARPIT dpt:22 44301 TARPIT dpt:445 33520 TARPIT dpt:23 6718 TARPIT dpt:80 6439 TARPIT dpt:3389 184K TARPIT dpt:5900 1091 TARPIT dpt:21 271 TARPIT dpt:1900

ozim 4y ago

Well I never ever expose RDP to the internet directly. Allow only office IP or setup some VPN/jumphost via ssh whatever. I trust SSH much more than RDP to not have RCE/auth bypass bugs. SSH is also much easier to setup key auth or 2FA than in RDP if one like they mentioned in post are more used to setting up Linux boxes. After reading this blog post - I would not send any logs to their servers.

greatquux 4y ago
I think though it may be better without 2FA to block attacking IPs no matter what the protocol.
CyanLite4 4y ago
Exactly! I at first thought this was a rogue or disgruntled employee who wrote a blog post on the way out. But it appears to be a legit article sanctioned by their company. They literally opened a server without firewall rules for anyone to connect to their RDP port, and wrote a blog article to brag about it. They also have a Windows agent but they brag about not using Windows in the past 20 years…

johnklos 4y ago

You absolutely should blame Flyserver. They allow bots to run, and they don't do anything about it. There are too many network operators like this. Imagine if any networks where constant and ongoing illegal activity happens, and the network operators do nothing, eventually result in those networks being taken away. Things would change drastically.