9 ms·
Crowdsec: A Fail2Ban alternative written in Go
- miked85 6y agoIs the fact that it is "written in Go" a selling point?
- justin_oaks 6y agoSome people prefer using a single binary rather than requiring a python installation.
- adkadskhj 6y agoYup. Perhaps the language shouldn't matter as much, ideally maybe we'd talk about "runtime features" - which matter to the readers. "Single Binary, static link, No GC" etc. But the language serves to me as a small proxy for most of those attributes. Tell me a game engine is written in Python or Go and i can infer a lot about the intended audience or runtime performance characteristics. The HN crowd seems to be so annoyed by language recently, but to me they just scream of missing the point entirely. /shrug
- omginternets 6y agoYes! For those who know Go, it means they'll be able to hack/improve/fix the software themselves. This question gets posted on every single "X written in Y", and I can't help but think it's an effortless way to broadcast some strange form of superiority (namely: by showing my exasperation with Go enthusiasts, I place myself in the category of people unimpressed by Go. Bonus points for mentioning Rust or Haskell.) This feeling is at odds with open source culture, where the ability to understand the code you're running is absolutely central. If you value Open Source, it should be pretty easy to understand how "written in language X" is a valuable piece of information.
- jdashg 6y agoThe title doesn't say that it's open source, though. (nor its license) It's specifically advertising itself based on its language, not its qualities.
- omginternets 6y agoGitHub wasn't a dead giveaway? How about the MIT License badge on the page? Come on, now... I think it's safe to say that one should check these (obvious) things before posting a snarky comment. It seems to me that this is part of the HN community ethos.
- bityard 6y ago> GitHub wasn't a dead giveaway? There have been numerous source-available but proprietary github projects posted to HN.
- omginternets 6y agoGood thing you can check the license!
- Fnoord 6y agoCorrect me if I am wrong but doesn't Go yield better performance than Python?
- FridgeSeal 6y agoYes, by quite a decent margin.
- gregoriol 6y agoYes, but it's just a trend. Could as well have been written in React. Joke apart, the trend to rewrite any single thing in Go/Rust is scary: why take something that is working and standard, and make it new? People have tried this hundreds of times as per hackernews history, and it's mostly not worth it. However, it is a good training, tutorial for Go.
- sieabahlpark 6y agoPeople vote it up, Google gets more people to use the SJW golang. The person feels validated for doing it and will promptly abandon it in a few months. It's equivalent to steam shovelware
- deleted 6y ago[deleted]
- eeZah7Ux 6y ago> the trend to rewrite any single thing in Go/Rust is scary Many useful projects get abandoned just because someone made a more popular alternative. In 2-3 years the rewrite in go/rust fad will fade and both the new and the old projects end up abandoned. I'll be downvoted to hell for this: jumping on fads harms the FLOSS ecosystem. Additions: also, static liking and embedding many dependencies harms Linux distributions.
- deleted 6y ago[deleted]
- hda111 6y agoOf course not. It's the same as with the Docker hype a few years ago. I don't get it. Still today everyone seems to think an app is better when it's "dockerized". Now with Go everyone can write buggy, memory leaking programs. I prefer apps written in C whenever a I can.
- eeZah7Ux 6y agowe need a fork of HN
- snazz 6y agoGarbage collection should make apps written in Go less memory-leaking and less buggy than those written in C, no?
- the_only_law 6y agoNot sure why on earth you think something would be non-buggy or non-leaky just because its written in C. I avoid C for almost any user mode work save a few things.
- LambdaComplex 6y agoI'd expect the C to be buggier, personally
- ashtonkem 6y agoOnly if you had issues with fail2ban's prior performance or ease of installation. Otherwise we start getting into partisan point scoring competitions over what critical services are written in what languages, which is incredibly tedious.
- ahendriksen 6y agoFor me definitely! I have a tiny VPS running ARM that I use as a build server. I get so much SSH spam that it noticeably slows down the VPS (both with and without Fail2Ban).. "Written in Go" signifies to me that it can be faster and more resource-efficient than Fail2Ban: a good reason to check it out.
- abdusco 6y agoYou should consider changing default SSH port. It helps a lot with the spam.
- blibble 6y agoand/or add an iptables rule that limits the rate, set high enough such that you'll never hit it 1 line in your iptables config
- coolspot 6y agoAlso adding port knocking. https://netslovers.com/2018/02/28/port-knocking-server-securing-ssh-connection-centos-7/ https://netslovers.com/2018/02/28/port-knocking-server-secur...
- InvaderFizz 6y agoIs this really much of a problem? I have a VPS that's been online for years, serving port 22. I average about 200k attempts per year. I have it set to pubkey only, root can't login at all. If you connect without sending a pubkey, it pretty much instantly tells you to go away. I don't bother with fail2ban. Maybe I should start logging attempted pubkeys as a side project just to see what pops up.
- justin_oaks 6y agoI like the idea of a Fail2Ban alternative, but I'm not sold on the idea of sending information to a third party. It may be optional to upload/download the malicious IP addresses, but the text on this project's GitHub page doesn't indicate that it is optional. My immediate thought is "Fail2ban works. I don't need or want communication with a 3rd party right now."
- golem14 6y agoI think that's fair, and I had the same initial reaction. However, people are generally fine with DMARC or other crowdsourced spam signals, so it's not clearly bad. I haven't looked at the source, but it should be easy to turn the crowdfiltering off or self-host if you have your own larger enterprise, no ? It also depends on what exactly is transmitted. Transmitting all IP addresses in the clear might be bad, sending out hashes might be better ? I think it's actually a good idea to try to detect those IP at scale. On your own individual server, you just see a million login attempts from a million different nodes on a botnet. But if you have millions of servers report the bad IPs and aggregate, you can see patterns and hopefully catch whole botnets.
- klodolph 6y ago> However, people are generally fine with DMARC or other crowdsourced spam signals, so it's not clearly bad. DMARC isn't a crowdsourced spam signal. And from my memories chatting with people running mail servers—the crowdsourced spam filtering is fairly controversial.
- golem14 6y agoApologies, I was thinking of the spamhaus list, which I'm also not sure now is crowdsourced in the sense most people would use. But most email providers build spamfilters using 'crowdsourced' techniques between their users (if enough users mark a sender spam it might be marked spam for all users). And while that may be controversial, it's also highly effective.
- 6y ago
- EdwinLarkin 6y agoMarketing riding on the COVID19 wave is distasteful (Let’s achieve a “digital herd immunity")
- Bishop_ 6y agoThe concept of "herd immunity" far predates COVID-19 and is an accurate enough analogy for what they're trying to accomplish.
- hda111 6y agoI don't like fail2ban because with IPv6 it becomes useless.
- sigio 6y agoThat should be just a matter of making firewall blocks at least a /64, and considering scans/source-ips also as a netblock instead of individual ip's.
- TrueDuality 6y agoIt's still relevant even if you don't switch to network blocking. I haven't heard of any bots brute forcing over IPv6 yet, which will be much harder due to the size of the address space but those two aside... A bot is unlikely to reconfigure the host's network stack to grab or rotate additional IPv6 addresses. That type of behaviour would be very easy to detect by endpoint protection systems and shut down. When scanning, scraping, and/or brute forcing service passwords they're likely to remain using the same IPv6 address either permanently or on a daily rotation, most likely this will be mostly impacted by OS defaults on privacy addresses as I don't actually expect many normal users to know and/or care about them. So if you're attacked on IPv6, you'll likely be equally protected by fail2ban as you are on IPv4.
- Philippe_H 6y agoHi Guys, thanks for all your feedbacks. (I'm part of the CS team) I'll try to address some few questions. 1/ You don't have to communicate. If you don't, you get a modern, fast, decoupled fail2ban with many various remediations (instead of just drop) and observability. What you don't get though are the IPs spotted by the crowd and curated by us. You don't contribute, you don't get them, fair. If you contribute, only offending IP / timestamp / scenario triggered are sent back to us to establish what we call a consensus (to avoid false positives and poisoning) 2/ We are super vigilant and sensitive about privacy. We made the architecture and many other crucial points compatible with GDPR (EU Law framework regarding private data handling) 3/ IP sent: We could hash it, but it's very easy to reverse. Maybe have a public/private key encryption, quite a good point, I'll tell the team, thx. 4/ You can contribute scenario in YAML or data source connectors in Grok. We are not hardcore for or against any language, but Go allows portability (we'll release Win & Macos binaries) and is container friendly, plus super fast, easy to read and scalable. Ever since we released, tons of proposal were made to port it to a 'real' language, sorry we are fine with that choice, no intent to change, no intent to convert anyone either ;) 5/ Herd immunity is what we want to create indeed. We tried to explain the combination of Behavior + Reputation by using an analogy with Waze. It worked but is less accurate. I prefer the one with Immune system. We are available for direct dialog on gitter. allow just some delays depending on your time zone, we are based in France, so CEST. (https://gitter.im/crowdsec-project/community https://gitter.im/crowdsec-project/community) we answer in French & English. Try it, it's free, MIT licensed and stable: https://github.com/crowdsecurity/crowdsec https://github.com/crowdsecurity/crowdsec Thanks, Philippe.
- dgrin91 6y agoHow is reversing the hash of an IP easy? Are you saying because there are only 4billion ipv4s? Hash should be fine for ipv6 still, right?
- Philippe_H 6y agoWell hashing is (usually) a symmetric function and we are open source... Meaning you could recover the key in the code (or intercept it during transfer). I think Private/Public key is a simpler approach, reusable elsewhere in the code and it's known to be safe. But I'm not the CTO either, I could be mistaken.
- kristianpaul 6y agoI rather use ossec http://ossec.net/ http://ossec.net/
- Philippe_H 6y agodifferent approach, but I'm sure at some point we'll get close to one another.
- kristianpaul 6y agoThat would be nice, its good to have alternatives
- Fnoord 6y agoOr Wazuh (ELK stack fork)
- ed25519FUUU 6y agoI love reading these go projects. I don't use Go professionally and spent minimal time with it on side projects. Apparently that was enough for the language because I can hop right into the codebase and basically understand everything in one-shot. A big language benefit, even if it maybe it was frustrating at times for the author.
- francislavoie 6y agoPlenty of people have asked for fail2ban support for Caddy, but since Caddy v2 has transitioned to structured logging, this hasn't been very easy. I'd love to see integration with Caddy here, I'm sure many people would appreciate a Caddy plugin that can do what they'd typically use fail2ban for.
- toxik 6y agoFyi, I looked into where the attacks came from. 99.5% China, no joke. I blocked whole B networks from China, and wouldn’t you know it - less break in attempts, less vulnerability scans, less SMTP spam. By orders of magnitude, night and day difference.
- UI_at_80x24 6y agoNot enough people do this. Using country-level block-lists dropped the number of IP/Port scans we have received down to sub 1% of totals. It may not be an elegant approach to the problem but it is VERY effective.
- Philippe_H 6y agoWe'll (soon) provide a Backoffice, where you can choose which IP you decide to ban (based on their activities, like bot scrapping, bruteforcing, etc.) but also add some 3rd party blacklist, block some AS or ranges, Tor exit nodes or VPN. This is all being builded right now, but in a couple of months, should be available.
- 10000truths 6y agoChanging the default port takes a lot less effort than finding and using a country-level block list, and is just as effective in cutting down intrusion attempts.
- gingerlime 6y agolooks really interesting! thanks for sharing. How does it fare in terms of performance compared to fail2ban? as far as I can see, fail2ban can chrun for quite a bit of cpu digesting logs. Is crowdsec faster/lighter? another question regarding backwards compatibility... eg porting fail2ban configs, action scripts, jails etc. I guess it’s not going to be a drop in replacement, but are there any porting efforts, recipes, scripts, docs to help with the transition? sorry for the crappy formatting, but using my mobile atm and couldn’t wait to ask :)
- Philippe_H 6y agoPerfwise, we have a user that previously used fail2ban to block some http botnets. He crunches 7000 IPs worth of logs in 50 mins with F2B. under a minute with CrowdSec. Another block 3000 IPs doing credit card stuffing directly at payment page, very quickly as well.
- dschulz 6y agoAlready replaced fail2ban for sshguard [1], which I like better. But I'll be testing this even not being a fan of crowd sex :-) [1] https://www.sshguard.net/ https://www.sshguard.net/
- Philippe_H 6y agoCrowdSec is for all protocoles / system generating logs (can be Cloud trail, syslog, kafka, etc.) and can ban at an applicative, user or IP level.
- nacs 6y agoFrom the sshguard site: "Started for SSH, now protects a wide range of services out of the box"
- worik 6y agoWhy?
- amanzi 6y agoHi - this looks great, but can you share what your plans are for the premium features in the futures? It feels like the big value here for you is the crowd-sourced info that you can aggregate from all users. What concerns me is investing time in a tool that builds value for you but then the features get stripped back when the premium offering lands. Cheers.
- Philippe_H 6y agono risk here. Tool is MIT, if community doesn't like our approach, you fork it. So we'll be faithful to our commitments and this licensing model is the best insurance for it. Now, I can also tell you that people using the free software and contributing IPs will get back, for free, the IPs dangerous for their technology footprint. (like if you use Wordpress / SSH & Nginx scenario, you'll get the IP attacking those). Free. Period. We monetize the aggregated, curated data and the features we offer that cost us infrastructure to run.
- mister_hn 6y agoBeware: > Crowdsec is in BETA version. It shouldn't, and didn't crash any production so far we know, but some features might be missing or undergo evolutions. IP Blocklists are limited to very-safe-to-ban IPs only (~5% of the global database so far, will grow soon)
- Philippe_H 6y agoAbsolutely. We owe that transparency to our users. The 1.0 should be out in a month from now, and it will include a Local API, an abstraction layer between the core and the bouncers & data sources. This will help the community to dev their own scenario, bouncers & data source connectors. But at that stage, we had no report of CrowdSec daemon bugging a server, crashing it or over consume resources. It's even used by some hosting companies, to process their reverse proxies logs, without any meaningful perf impact. That being said, it's still beta because some features or architectural points could vary a lot at that early stage. We only distribute 5 to 10% of the IP rep DB because we are over cautious and don't want any false positive to happen. Our Consensus algorithm is getting better by the week by we are cautious by nature (and experience)
- deeblering4 6y agoSshguard is another fail2ban alternative that is worth a look.
- Philippe_H 6y agoCrowdSec is not designed specifically for SSH. It can ingest any type of logs and answer with a bouncer at pretty much any level. IP/Session/User/software stack. Ie, we are working on Magento to parse all logs (apache, magento's logs, etc.) and provide a bouncer that is user aware, at an applicative level. Some people are experimenting it to parse logs from airplane communications, to see if pilots behavior is close to a standard or deviate. We have experimentations on BGP protocol, etc.
- deeblering4 6y agoNeither is sshgurd, jftr
- kazinator 6y agoFor seven years, I've been using a home-grown Fail2Ban alternative called txrban: http://www.kylheku.com/cgit/txrban/tree/ http://www.kylheku.com/cgit/txrban/tree/
- Philippe_H 6y agoI should maybe have told you also, team members are from pentesting and high security hosting background. We also have created some other OSS components before, like NAXSI (Waf over Nginx), Snuffleupagus, PHP malware finder, etc. So we faced the hurdles of assembling, deploying, configuring, handling and maintaining sectools in our Devops & Secops environments, and we thought this tool with our years of experience in mind.