17 ms·

Ask HN: How do you continuously monitor web logs for hack attempts?

477 points by sandGorgon 9y ago

what is the generally accepted best practice to be monitoring web logs for anomalous accesses ? do you guys just throw cloudflare in front and forget about it ? Or do you have engineers who work like data scientists - eyeball the logs ? I have heard suggestions of using a firewall - but I'm genuinely curious on how do security focus companies discover things like "oh, we got attacked by Bitcoin miners from North Korea". Are there sophisticated tools that do this for you.. or is there a generally accepted practice that has evolved for even regular engineers to do this ? P.S. I'm personally more interested in an API-focused answer, but I guess the same thing applies for websites.

jenkstom 9y ago
fail2ban, but I'm not sure that's what you are looking for. You'd have to customize it for an API.
hluska 9y ago
I assume that I suck at security, everything I have ever built is badly compromised and I'm too dumb to know it. I'm so bad at security that evil criminals actually celebrate my birthday. Since I'm this bad, I assume that any monitoring tool I could install on the boxes themselves, up to and including all my logs are compromised. I'll usually still install monitoring tools for compliance, basic diligence and because they make wicked graphs that impress my superiors. Since I'm that bad, I'd rather not invest 40 hours a week combing through fake logs unless I'm validating that I've taken care of the basic security fundamentals for my threat level. Rather, I like to make sure that my security fundamentals are taken care of so that I can stop the simple attacks. If my threat/compliance model dictates, from there I'll possibly step things up into a web application firewall (WAF). But, through all of this, I know that there are people out there who are so much smarter than me that the best I can do is little more than an annoyance for them. So, again, I like to assume that I'm badly compromised. This implies that I'm very careful about what data I keep, and I take efforts to protect my data at rest.
summadat 9y ago
2nd'ing fail2ban, it's simple and awesome. Beyond that, tripwire or it's modern equivalent. Traffic monitoring, "deep packet" inspection on the network side. Run all outbound through a proxy.
- kazinator 9y ago
  Not a lot of people seem to know what tripwire is nowadays. There is this Linux thing called IMA which in some ways reminds me of tripwire (Integrity Measurement Architecture).
- tbrock 9y ago
  Nah, I’ve waned to use fail2ban multiple times and failed. Unless something has recently changed it is some of the most poorly documented software on the planet. I’ve never been able to configure the thing or figure out what it’s doing or... really anything besides find the homepage. It’s also just not worth the trouble. Globally open SSH port? That’s like 1992 level Linux box in your parent’s basement stuff.
  - semanticist 9y ago
    While it monitors SSH by default out of the box, people here are talking about creating custom filters which will monitor their Apache or Nginx logs. (Although, that said, we also have it monitor our non-standard SSH port with v. aggressive blocking rules, even though it's also behind a firewall. Defence in depth, and all that.) The documentation isn't the greatest, but this page explains the concepts well enough and was easy to find using Google. https://www.fail2ban.org/wiki/index.php/MANUAL_0_8 https://www.fail2ban.org/wiki/index.php/MANUAL_0_8
    - tbrock 9y ago
      I understand the concepts but... how do you actually use it?
      - hluska 9y ago
        I like Fail2Ban, but as I remember it, my learning process looked a lot like this: 1.) Read through the documentation. 2.) Pour myself a scotch. 3.) Read through more documentation. 4.) Start drinking right out of the bottle. 5.) Try something. 6.) Get an error or experience a spectacular failure. 7.) Use StackOverflow. 8.) GOTO 5. ------------- In good news, it worked - the light went on and now I'm confident that I could adapt Fail2Ban to suit any use. But, in bad news, honestly, I feel your pain and understand where/why you are getting stuck. Fail2Ban is a very complicated, yet extremely useful piece of software.
kazinator 9y ago
I use a home-grown solution called txrban: http://www.kylheku.com/cgit/txrban/tree/ http://www.kylheku.com/cgit/txrban/tree/ It consists of several processes written in the TXR language. One for monitoring Apache, one for sshd and one for Exim (mail server). TXR's standard library has "tail streams": streams which automatically follow a rotating log file. These txrban scripts simply open tail streams on their respective log files, use TXR pattern matching to scan for what they are a looking for, and then use functions in the tiny txrban utility library to do the banning. Banning is instigated by issuing iptables commands. The banning isn't persisted, but when you restart the scripts, they retroactively replay everything by scanning the available logs from the beginning. Banning events are reported via syslog. A "dry run" configuration variable allows testing for the rules without actually modifying your iptables. I never packaged this program into a redistributable form. It contains some hard-coded log-file paths that work on my server, and no documentation.
originalsimba 9y ago
use fail2ban. Every month or whenever you choose, review your access log and see what kinds of attacks are happening, then use fail2ban to block those. If you target 404s for non-existing PHP files, especially things like "setup.php" you can nail most would-be hackers before they can find vulnerabilities. Use a low-retry count and a high ban time, that'll kill the scanner-bots. Other than that, make sure your webserver is secure. If your server is secure then you don't need to do anything because nobody can get in. Secure first. Aggressive next.
eli 9y ago
I used to spend a lot of time combing my logs with custom scripts, but honestly I think the effort is better spent elsewhere. There're countless automated attacks a day on a big site but they really aren't anything you need to worry about.
wepple 9y ago
most successful organizations ive seen don’t think about their API/app, but rather their stuff of value: usually data. If someone lifts your entire database, can you spot that on the way out the door? How quickly would you notice 100% of your fattest internet pipes spewing data at some poor souls website? You mention bitcoin; how long until you spot all your machines at 100% CPU?
gmuslera 9y ago
Computers are better at the eyeballs-on-logs tasks. You teach them what is "normal" to be ignored and they may warn you about what is left, even the old logwatch is pretty capable for that. Of course, there some basic security practices. Firewall, fail2ban or similar to stop/ban misbehaving IPs, monitoring systems metrics to detect abnormal activity/usage (not just for intrusion detection), and maybe feeding logs to i.e. ELK to notice new/unknown patterns.
ChuckMcM 9y ago
Fail2ban looking for web pages that don't exist on your web site (like ../wpadmin) works pretty well at getting rid of script kiddies. I suppose you could add a honeypot page that took form submissions and looked for any sequence attempting a CSRF, XSS, or SQL attack and ban the source there as well.
loblollyboy 9y ago
In short - security is such a big and multifaceted topic that it’s hard to know where to start, but I think that learning Kali Linux tools would cover a lot. I started working at a cybersecurity company - our product is ridiculously stupid and I try and work as little as possible. In the time I’m not working I try to at least learn stuff. Since I’m at a cybersecurity place I sometimes try to learn about cybersecurity. It seems that even though for profit cs companies are making crazy dough, a lot of open source solutions out there do the same thing. There are so many attacks - they can be anywhere from the application level to the level of physical devices like routers having some kind of exploit. For any type of attack there is a good or multiple good open source tools that can be used to execute or defend (depending on what color hat you have), and a ton of them come standard installed on Kali Linux, but before you learn these you have to understand networks. As a python/Django dev I didn’t really have a deep knowledge and still don’t have a deep knowledge of this. Nor have I done more than dabble with any of these tools. But I know that there are a lot of videos on how to use these - including this 15 hour (!) one https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM https://m.youtube.com/watch?t=1177s&v=vg9cNFPQFqM - if you were to go through that you’d probably be a security wizard
- khamba 9y ago
  Thanks for the youtube link. Seems interesting. I am going to watch it in full.
dsl 9y ago
I break into stuff professionally. There is nothing I love more than the "just throw {Cloudflare,Akamai,Fastly,etc} in front of it" crowd. If you are going to deploy a WAF [1] it _needs_ to be at the edge of your datacenter or cloud environment. No matter what fancy feature your provider wants to sell you to fix this, there is a way to bypass it. * Make sure you are logging centrally, and logging the right things. * Make sure someone is looking at the logs, it doesn't have to be a full time person, but it needs to be someones job. * Depending on how big you are, hire people to break in on some regular basis. It could be anything from hiring a cheap contractor to run a bunch of automated tools, to full on 10 person adversarial teams. They will find some stuff for you to fix, but more importantly you will know whether the first two points are working based on what you see. 1. https://www.owasp.org/index.php/Web_Application_Firewall https://www.owasp.org/index.php/Web_Application_Firewall
- dom_hutton 9y ago
  > Make sure you are logging centrally, and logging the right things. Could you please elaborate on what some of the right things generally are?
  - deleted 9y ago
    [deleted]
- BinaryIdiot 9y ago
  > to full on 10 person adversarial teams Okay this sounds incredibly interesting. Are there any open source groups that do this "for free" sorta? Not asking to get free working, asking so I could possibly jump in and learn.
  - pvtmert 9y ago
    we call it hacking forums/teams. main motivation is some idelogical/political belief tho. imho try to hack yourself first
    - MandieD 9y ago
gesman 9y ago
Using Splunk to monitor traffic either via web logs or via more detailed packet capture solutions. Just good web logs is enough to detect all kind of malicious anomalies. Splunk machine learning functions can cluster data and detect anomalies in anything without rules or signatures: https://imgur.com/a/IFhUZuH https://imgur.com/a/IFhUZuH
gzur 9y ago
You don't monitor web logs, you ship the logs to cold storage every [crontab spec units for time :)] and monitor your hosts and network for unusual activity. If anything happens, THEN you look at the logs. The signal/noise ratio of the average web host logs makes combing through them just not worth it. There are tons of products out there - have a look at OSSEC (https://www.ossec.net/ https://www.ossec.net/), it's open source and free.
semanticist 9y ago
We also use Fail2Ban like a poor man's Web Application Firewall - for a Rails app you know you'll never have legit traffic to anything 'cgi' or 'php' or 'exe', so crafting a couple of regexps to match those and then ban them works well. We use CloudFlare so we feed the IP bans into CloudFlare's firewall through their API as well as local IPTables. (We also use CloudFlare's WAF - we like redundancy in our paranoia.) If you had more demanding requirements you could look at mod_security for Apache or Nginx. We also implement rate-limiting using Fail2Ban, which helps to trip the dumber automated scans. Every time Fail2Ban blocks an IP address, it emails our admin team with the matching log entries and a whois on the IP address, which we then eyeball to make sure there's nothing weird going on. TBH, easily 90% of the IP addresses we block got caught looking for 'wp-admin/admin.php', which is probably a good reason to never use WordPress.
- _Chief 9y ago
  I like this approach. Do you permaban the IPs? because a ton of ISPs rotate IPs across users
  - semanticist 9y ago
    CloudFlare has a maximum number of firewall rules, based on the types of paid accounts you have. Every couple of months we purge the list when it gets close to being full. 99% of IP addresses we block are from outside our target market - we're a very local/regional company - so we don't really have to worry much about blocking legitimate customers. Very rarely our support team will get a request from someone that's been blocked and we'll unban the IP address, or set it to a JS/Captcha challenge at CloudFlare.
- inopinatus 9y ago
  Just be aware when crafting those regexes that user-generated URLs (e.g. slug-based content paths, or even just query-string params if one is very careless) can thereby be used to build a denial of service attack against internal infrastructure, third-party integrations/webhooks, and even regular customers. Reminds me that back in the day we used to be able to train email sanitizing appliances for false positives by sending them EICAR or GTUBE with spoofed headers.
SnowingXIV 9y ago
I’ve learned that at the very least to ensure that everything is being properly backed up and that the restore works. Testing this regularly puts my mind at ease that even if my site, application, whatever gets destroyed, stolen, etc I can bring it back to life in a very short time. It’s very difficult to ensure that something is 100% secure (not even sure if possible since the major players like Twitter make mistakes). Of course, this doesn’t solve the issue of data being taken but that’s why it’s important to understand the data you actually store and if you truly need to store it or if some could be handled another way.
elchief 9y ago
OSSEC is a nice FOSS tool for this
jwatte 9y ago
What is "an attack?" You need to define this better to get more focused answers. Log monitoring is good for finding bad access patterns, but can't be trusted to find a problem as it happens. Look for repeated errors. Hitting the same service over an over? Login attempt failures from the same few IPs? It's either a script, or the Bing crawler. Write application level code and network level configuration to mitigate the problem. How about legitimate users doing social engineering attacks on each other? Sybil attacks in ratings or voting? Attempt to reuse mobile purchase tokens on different accounts? Some new SSHD credential leak zero day? Using your cat picture upload service for storing pictures of dogs? Each of these need different approaches, and a good logging system that can change with your needs is super helpful. But you can't read the logs in real time, looking for bad patterns. That needs customized analytics.
guessmyname 9y ago
> I'm personally more interested in an API-focused answer OSSEC [1] is my go-tool for this type of tasks, it offers a language-agnostic interface to monitor any type of events. I used it along with my team to power the early version of the Sucuri Firewall, which later became the GoDaddy's security system. The project has matured a lot and is heavily tested using millions of HTTP requests every day. We also implemented a behavior-based algorithm to detect malicious requests, this part is not open-source but I can tell you that it was relatively easy to integrate it with the core interface. Splunk [2] is also a good option, slightly more difficult to configure though. I have had the opportunity to meet some of the developers working in this company during a short visit to their Vancouver office and was genuinely surprised with the level of engineering that goes around this product. Definitely, worth checking it. NewRelic [3] while the purpose has nothing to do with security, it also offers an easy-to-use interface to monitor events in the system. With some extra configuration you can make it work as a rudimentary firewall and benefit from all the great features that the product provides. Disclaimer: I am an active contributor to the core features, but I am neutral on my recommendation. [1] https://en.wikipedia.org/wiki/OSSEC https://en.wikipedia.org/wiki/OSSEC [2] https://en.wikipedia.org/wiki/Splunk https://en.wikipedia.org/wiki/Splunk [3] https://en.wikipedia.org/wiki/New_Relic https://en.wikipedia.org/wiki/New_Relic
busterarm 9y ago
1) fail2ban 2) log everything to an ELK stack (or your preferred alternative) and set up dashboards and reports to feed you malicious crafted URLs 3) Reject traffic in your web proxy based on a maintained list of hostile URLs. Ban the source of this traffic. Don't send back 500s, just slow down and eventually stop answering the requests. HAproxy lets you do this easily with a few simple rules and a stick table.
tptacek 9y ago
The answer isn't quite "don't bother" but it's definitely not "install these tools and get started". Most of the things you can easily set up to watch for security events are looking for problems you simply shouldn't have. For instance: you can set up fail2ban, sure. But what's it doing for you? If you have password SSH authentication enabled anywhere, you're already playing to lose, and logging and reactive blocking isn't really going to help you. Don't scan your logs for this problem; scan your configurations and make sure the brute-force attack simply can't work. The same goes for most of the stuff shrink-wrap tools look for in web logs. OSSEC isn't bad, but the things you're going to light up on with OSSEC out of the box all mean something went so wrong that you got owned up. Same with URL regexes. You can set up log detection for people hitting your admin interfaces. But then you have to ask: why is your admin interface available on routable IPs to begin with? What you want is an effectively endless sink for developer logs (Splunk is fine but expensive, ELK is fine but complicated to manage), and you want to add structured instrumenting logs to your applications in all the security-sensitive places, like authorization checks (which should rarely fail), and especially a very solid, very long-term audit trail of admin-like actions, so you have some prayer of tracking insider misuse. You could instrument your SQL queries and catch any query failures --- which should also be rare. That kind of stuff. I don't think there's a good shrink-wrap package that does that. But really: focus on removing attack surface, rather than continuously monitoring it.
- koolba 9y ago
  > But really: focus on removing attack surface, rather than continuously monitoring it. You buried the lede on this one.
- monkpit 9y ago
  I think you meant “Splunk” :)
  - tptacek 9y ago
    thx
- peterwwillis 9y ago
cat199 9y ago
previously worked at a major shared webhost a few years back (which used Apache) we used mod_security with common rulesets + a custom list which was reviewed/updated based on what hacks were coming in (with hacks both passively monitored by customer base/compromised spam reports and via active scans for malware deposited by attackers which were then cross-correlated/traced back to application logs to find the entry point/compromise. support staff was ~600 (including front end call center and billing people), with probably 250 of those doing L2+ support and ~50 doing the security work which fed into the ruleset, for a server farm of ~2-4000ish (and exponentially more sites/accounts since it was shared webhosting). In some ways our network could have been viewed as a giant honeypot because of all of the outdated webapps that were being run by the customers, plus their compromised PC's saving passwords, etc.
- yani 9y ago
  I see a lot of mod_security misconfiguration across hostings. While I think it is a great tool, these misconfigs make it difficult for me to recommend it.
twunde 9y ago
There's really no way to eyeball logs in any sort of complex environment. What you're looking for is a system that will automatically analyze logs and/or traffic for malicious events and alert on them. What you're looking for are Introsion Detection Systems (IDS) and Security Information and Event Management (SIEM). For IDS there are a whole bunch of open source solutions. Essentially these systems ingest logs into a centralized location and run them through a rules engine, which gets updated with new rules on a weekly or daily basis.
- jlgaddis 9y ago
  > What you're looking for is a system that will automatically analyze logs and/or traffic for malicious events and alert on them. (I don't pick on you, so I hope this comment doesn't come off that way.) What are "malicious events" and how will you know them when you see them? I would argue that, instead, your system should automatically analyse logs for benign (a.k.a., completely normal, expected, innocent) events, toss them out, and leave humans to review anything that's left over. With any new "system", it will take a bit to build up a complete "whitelist" so that the amount of data to review is down to a manageable level but you'll be rewarded with much more meaningful and actionable data when you get there.
  - chrisper 9y ago
    Instead of the term "malicious events" he should have used the term "anomalies".
  - amelius 9y ago
    > What are "malicious events" and how will you know them when you see them? I suspect they are simply defined as "rare" events.
strictnein 9y ago
> I'm genuinely curious on how do security focus companies discover things like "oh, we got attacked by Bitcoin miners from North Korea". Are there sophisticated tools that do this for you.. I work in a Fortune 100 Security Operations Center (SOC). We have a dev team that builds tools for our Incident Response team, Threat Intel team, etc, etc and an engineering team to manage those tools. And lots of other teams that specialize on lots of other things (malware reverse engineering, spam/phishing, red team, etc). We have to build a fair amount of our own stuff because we operate at levels above what a lot of commercial tools can do. We have hundreds and hundreds of custom alerts that fire based on various network traffic and sysmon activity that our threat team has written rules around. This, of course, is out of the question for most. So, what should you do? First, get your logs all in one place. The ELK stack is probably the easiest way to do that. ES makes things searchable and Kibana is a pretty good UI for a lot of this stuff. Second, if you aren't monitoring your internal network, you really have no idea what is going on, so look at what you can accomplish quickly with distributions like Security Onion (which includes the ELK stack): https://securityonion.net/ https://securityonion.net/ Third, there's a fair amount of Open Source Intel (OSINT) on threats. You'll need to find a way to integrate that information and scan your logs for it. Malware, like Coinminers, will have certain Indicators that will tell you if they are on your network. If you're on the ELK platform, look at something like ElastAlert. You write rules in Yaml and then set up Alerts to fire into your Slack channel or Email you or whatever. There is no real shortcut here. Commercial tools that can do a lot of this stuff includes FireEye's TAP, but they definitely aren't free: https://www.fireeye.com/solutions/threat-analytics-platform.html https://www.fireeye.com/solutions/threat-analytics-platform.... And, as mentioned elsewhere, absolutely remove any attack surface that you can. Scan your IP ranges and domains with tools like Shodan.io, HackerTarget.com's DNS tools (DNSDumpster.com has a nice, free DNS search), etc and make sure you have a handle on what, exactly, you're exposing. And, finally, if you're big enough then every single day a computer will be compromised in some way. There is no company in the world who doesn't experience this so you need to have a clear set of tools and playbooks to handle that. If you can respond to most incidents in days (not weeks), you'll be better than 90%+ of the companies out there. Edit: clarified acronym
- jrochkind1 9y ago
realandreskytt 9y ago
SpectX is a tool that can run fast sql-like queries on top of distributed text files. It was initially built to parse logs by an infosec team. Hope it helps!
StreamBright 9y ago
For me monitoring the logs is already too late to do anything about the hacking attempt. Usually web servers write logs after the request was served. One approach (I have seen commercial products do it) is to hook into the web server code and inspect every request real time before it hits the application servers and make a decision if it is a legitimate request or not.
- yani 9y ago
  It does take a number of malicious requests before someone can find an oppening. If you ban them after the 1st or 2nd request, you will be safe. It is going to be a miracle if some vulnerability scanner finds and oppening after 1 request, not impossible but HIGHLY unlikely
  - hluska 9y ago
    > If you ban them after the 1st or 2nd request, you will be safe. Sadly, it is too easy to get a new IP address now. I think as part of our security mindset, we should consider it trivial to scan for several hundred vulnerabilities using relatively unique IP addresses. I'm not saying that you shouldn't ban traffic as it's required under several compliance programs and it is another form of demonstrated diligence. But, unfortunately, banning individual IP addresses only works against the absolute lowest hanging fruit of the criminal world.
subcosmos 9y ago
A tad off-topic, but one thing I've enjoyed doing in the past is giving my attackers something to get distracted on, allowing me to observe their behavior. https://www.youtube.com/watch?v=smvwW3vW5rM https://www.youtube.com/watch?v=smvwW3vW5rM
- archi42 9y ago
  Wow, it's interesting to see how bad this hacker is with a shell. Even the "better" attacker from the video you linked in the comments doesn't seem to really know what he is doing at times. Sadly, I feel not all hackers are dilettantes.
cft 9y ago
Another question- how do you comply with GDPR and do that? What if a black hat sends you a SAR request?
crucialfelix 9y ago
I use logwatch. It gives a large informative report every day. Mostly it shows me that fail2ban and my secure configuration are doing what they are supposed to do.
cm2187 9y ago
I sleep a lot better since I use IP white lists on any port/is that is not public facing. Central reliable location with a signed whitelist that the servers checks regularly and adds to its firewall. No security measure is bulletproof, particularly against zero days but that reduces the risk by orders of magnitude.
- archi42 9y ago
  Nice idea, but how does it improve over "have important stuff on an administrative VPN"? To me the whitelist seems more effort (but we often have non-static IPs at home; with dual stack lite aka native(?) ipv6 plus a private ipv4 behind a huge, ISP-wide NAT making things even worse), and it doesn't give you additional encryption.
  - cm2187 9y ago
    Agree. I have scripts regularly updating the whitelist with dynamic IPs every 5 min. But I don't use VPNs for various reasons. OpenVPN on windows is very slow (unlike on Linux), it's hard to get above 5-7MB/s, which is a problem for large file transfers (and that was pre meltdown and spectre patches). It's also hard to configure, and the server still has WAN connectivity so it's easy to misconfigure it and get no protection. I also found that having an OpenVPN LAN with no gateway in addition to your default LAN on win7 tends to interfere with Chrome on the client, which will occasionally complain that it has no connectivity. Being a belt and suspenders guy, and only dealing with windows machines, I found that a good solution is to combine it with IPSec, which is straightforward to configure on windows, fast, and adds encryption when the underlying protocol doesn't have any (SMB2). You can configure the firewall to only use IPSec for certain ports. But it's a nightmare to interop with non windows machines.
xer 9y ago
It's called SIEM, Security Information Event Management, there is a lot of companies doing this and providing cheap and expensive solutions.
tgsovlerkhgsel 9y ago
For web logs specifically, even if you had a system that would 100% tell you what is a hack attempt and what isn't, you would probably not want to look at the output. The web server will be hit by thousands of "hack attempts" a day. Most of them will be directed at software you don't even use. You care about successful hacks, not random attempts (targeted attempts may be different, but good luck finding them in the noise of web logs). Web logs are probably the noisiest, least useful place to look for those (although they can be useful to determine how an attacker got in/where they came from after the fact). The only thing that could make sense for web logs is looking for brute-force attempts, but that is better done with application-level logging (and countermeasures like CAPTCHAs). Things you could look at instead are e.g. firewall/netflow logs: If your server is making unexpected outbound connections, there's a good chance it got owned and someone is now downloading their toolkit onto it.
hannob 9y ago
Don't. You shouldn't care about hack attempts. There will be thousands each day. They don't matter. You should care about your systems not being easy to hack. Unless you're a very high value target then "practice basic security hygiene, install your updates, don't reuse passwords" is enough to achieve that.
tegeek 9y ago
There is a product called Unomaly [1] which does exactly what you wrote. 1. https://unomaly.com/ https://unomaly.com/
FundThrowaway 9y ago
Ha I remember awhile ago doing something similar with our AWS server logs, turned out we were always under attack 24/7, rather amusing but not very useful apart from scaring juniors a little bit.
unixhero 9y ago
I can not recommend watching this Defcon talk enough: https://www.youtube.com/watch?v=_SsUeWYoO1Y https://www.youtube.com/watch?v=_SsUeWYoO1Y - "You Spent All That Money"
- freehunter 9y ago
  Dude I'm a security consultant for a major vendor and I can't tell you the number of times I've A) Found an incident while installing new equipment B) Installed new equipment last year and came back this year because they still got hacked and/or C) Ripped out their old gear and put in my company's gear after a hack because the sales guy told them they wouldn't get hacked with our (passive, monitor-only) gear. You need the right tools, no doubt. But you need to understand your needs first. Centralized logging is the first step, but without knowing what you want to get out of it, those logs just sit there unnoticed. Even intelligence platforms like Splunk or ArcSight, unless you know what you're looking for and write it into rules, it's just sitting there. Knowledge is what matters, the tools just make it easier.
vectorEQ 9y ago
The most important things for this are: - A good idea of what is normal traffic and behaviour on the web service. - Good threat intel. These 2 combined will make it fairly obvious what needs to be done, but obtaining these 2 items is not a trivial task, infact, they are both continuous processes.
andyjpb 9y ago
Marcus J. Ranum's "Artificial Ignorance" concept is pretty good. I did an implementation and have been running it for more than 10 years: http://www.ranum.com/security/computer_security/papers/ai/ http://www.ranum.com/security/computer_security/papers/ai/ Someone's also done a Python implementation: https://github.com/lchamon/ai https://github.com/lchamon/ai ...and it looks like there's one in syslog-ng Premium Edition: https://syslog-ng.com/documents/html/syslog-ng-pe-latest-guides/en/syslog-ng-pe-guide-admin/html/concepts-artificial-ignorance.html https://syslog-ng.com/documents/html/syslog-ng-pe-latest-gui...
bsenftner 9y ago
I used to run a web service, an API, that created 3D models of people from photographs. It was a character creation service that made likeness models of real people from a photo. I knew we'd get a lot of hack attempts, and a consultant recommended we get a high end Sonic Wall appliance, a $20K piece of hardware that cleanses all network traffic to and from my data center. Best decision EVER! We'd get hack attempts, we'd get DoS attacks to the tune of 130K hits per hour, and that Sonic Wall shrugged it all oof like nothing. For a while, it looked like every script kiddie was trying to hack into our servers. No one got through, and we could care less how or what they were doing in their attempts. We had real work to do, that did not include fucking around with those idiots. We also had a block list for customers from China. Not a single transaction from China was legitimate, all forged credit card data, so we just blocked their billions of potential customers because not a single purchase from China was real.
rebekk99 9y ago
We use a combination of https://www.templarbit.com https://www.templarbit.com and Splunk. Templarbit allows us to get real-time notifications when someone is running a vuln scan, violates one of our security headers or is trying payloads that look funny.
bkovacev 9y ago
Could something like poor man's primitive check work? - Create a custom http header and find a unique way to create/identify it. So theoretically another layer of authorization, just like APP ID for instance. - Create a middleware that would scan the request and check whether it has the required custom header - If not log everything and create a notification/slack/allow/deny and if it's good let through the request.
lmeyerov 9y ago
Separating this out: 1. Monitoring logs for security -- people don't, but machines do. Initial security comes more from segmentation architectures (bastions, whitelisting, ..) and network/host intrusion detection systems. Logs do come into play later for app-specific rules (unpatched apps, fraud tipoffs, ...) and eventually automated UEBAs (exabeam, ..) and analyst threat hunting (plug: http://www.graphistry.com http://www.graphistry.com !). 2. Forensics for post-breach attribution & containment/mitigation. Let's say something does happen (... it will.) Logs are important here. -- Threat intel: Known indicators of compromise (IOCs) -- IPs, URLs, downloaded file hashes, ... -- get published. Team setup always-on correlation rules against their internal logs via say splunk or hadoop, and if hit, they know the threat intel of it. Or they'll get alerts from OpenDNS that an internal device is talking to a known bad actor. That's generally pretty shallow threat intel on the attack (ex: VirusTotal.com entries), but a start. There are multi-step ways to iteratively correlate across these sources, but takes effort. Attacker attribution happens here, but typically more important is identifying further incident IOCs in case of a bigger breach. -- IR / DFIR: Teams care more about defense and explaining what happened internally (incident killchain, scope, ...). Say there was a malware alert from AV, if the dwell time was hours or the device important, teams should investigate what else happened for the device/user/etc, or if anyone else got hit. THAT'S where logs really shine. Small/immature teams will check individual tools (firewall device, ...), while more mature teams check centralized logs. We find manual log analysis shallow in practice for a decently sized org... so another plug for http://www.graphistry.com http://www.graphistry.com for setting up more comprehensive multi-step analysis flows for the human side of looking through stuff like malware, phishing, and account hijacking. Tools like ours work on top of log stores like Splunk/ELK/Hadoop/etc. (more scalable) or bring their own DB (less scalable.) This can get pretty intense: some companies log all the way through process activity. At the extreme end (FBI, Mandiant, ...), companies will quarantine a device for activities like analyzing RAM & disk images.
arca_vorago 9y ago
ossec + syslog , how you display the data (zabbix, elk, etc) is up to you.
linsomniac 9y ago
I'd recommend only really addressing the most obvious attacks, rather than trying to follow up on everything that even looks like it. I once got an "emergency" call at 7am (I had been up doing something rather late), about a machine I run attacking this guy. After some digging with him, the machine was our public-service NTP server, and the packets were on udp/123, in response to packets his system was sending out. Ended up with him yelling at me and hanging up, and me shutting down our public NTP server for good.
e12e 9y ago
What you need is information, what's easy to get is data. Generally, if you have data on what "normal" looks like, and data on what the current state is, you can view the difference as information that indicate an abnormal state; be that a full/failing disk - or someone downloading all tour data via blind sql injection. That's the theory. I've yet to achieve this in a timely fashion, within budget limitations - but on my list is to have another look at graylog2 for logging (if you have the money, splunk is probably easier - but they do charge for the value they provide). But I'm hoping to find the time to mash graylog2 with something like collectd, and maybe a general event collector for exceptions/stack traces. The goal would be to have it all on a similar timeliness in a unified interface (this spike of exceptions happened after this series of requests from there, and after the disk went full). And ideally with a sensible way to automate "normal" vs "exceptional" (alert on high cpu, low disk, high network bandwidth etc). Sure, people should read logs/alerts - but it's a challenge to filter out irrelevant (ssh pw bruteforce against a ssh server with only key auth, Web app crashing due to #wontfix (yet) bug) - so the relevant becomes obvious (successful key based login from different continent - where none of your team are). https://github.com/Graylog2/graylog2-server https://github.com/Graylog2/graylog2-server
kwi 9y ago
I highly recommend you try out Sqreen. https://www.sqreen.io/ https://www.sqreen.io/ It's basically newrelic for security. Takes 2m to add to your web app and it works super well. It blocked an attack automatically on our service recently!
zallarak 9y ago
error/exception monitoring
pilooch 9y ago
Dont do this in production without backup tools but we use unsupervised learning for anomaly detection. We're machine learnists so this is natural for us and we originally built the system for a mega Corp and made it open source (though not all parts are yet public). Catch is it's not real time but he system outputs outliers every few hours or daily depending of the size of your traffic (daily for around 50M proxy daily logs). Outliers are then reviewed by scripts and few operators / analysts.
sprkyco 9y ago
I work for: https://www.signalsciences.com/ https://www.signalsciences.com/ and our tool is specifically designed for use cases like this. Let me know if you would like more info cody-at-signalsciences-dot-com
deleted 9y ago
[deleted]
HauptJ 9y ago
WAFs like CloudFlare help, but stuff still gets through. Last night an obvious brute force attack went undetected through CloudFlare. I just finished writing a post about it: https://hauptj.com/2018/05/08/brute-force-login-attacks/ https://hauptj.com/2018/05/08/brute-force-login-attacks/
hkr_mag 9y ago
This is a list of what you can do for application security with Nginx (mostly with open source tools): https://github.com/wallarm/awesome-nginx-security https://github.com/wallarm/awesome-nginx-security My talk from Nginx conference: https://www.nginx.com/blog/build-application-security-shield-with-nginx-wallarm/ https://www.nginx.com/blog/build-application-security-shield... Important note. Care about vulnerabilities. Not about attacks. Buy Burp license. Run appsec training for all of your developers; it's easy while you're small. Disclaimer: I am a co-founder of Wallarm mentioned in preso.
tmaly 9y ago
I use Fail2ban http://www.fail2ban.org/wiki/index.php/Main_Page http://www.fail2ban.org/wiki/index.php/Main_Page for auto blocking. I take an 80/20 type approach and see what types of access is not covered by current rules, then I add a custom rule to cover that type of access attempt.
mejakethomas 8y ago
During my time at Wanderu we set up Filebeat on all app machines, forwarding json logs to Kafka, and then on to ELK and Pipelinedb for analysis/alerting/session killing/ etc. It worked very well, as there are almost always application edge cases that should not be blocked immediately with Cloudflare or at a FW level. ELK was used for exploration and/or setting alerts on thresholds that we had found/defined beforehand. Pipelinedb was used to run continuous aggregates on a stream, augment (pop json log lines off a stream, enrich them with Maxmind, etc) logs in "realtime", and immediately surface malicious behavior. We'd then programmatically kill sessions or add Cloudflare rules upstream, based on the behavior that surfaced in a Pipelinedb continuous aggregate. Wanderu: https://www.wanderu.com/ https://www.wanderu.com/ Filebeat: https://www.elastic.co/products/beats/filebeat https://www.elastic.co/products/beats/filebeat PipelinedB: https://www.pipelinedb.com/ https://www.pipelinedb.com/
- alexland 8y ago
  I'm curious how you got Filebeat to push to Kafka. Did you write a custom modification to Filebeat/libbeat?
jgimenez 8y ago
I maybe have a counter-question. Once you identify an attack, what do you do? Firewalling the attacker is an option but not really effective, the attacker might just change IP address and keep going. We've only inconvenienced them a bit. What else can be done?

eli 9y ago

Are these bots causing problems like consuming excess resources? If not it doesn't seem worth the effort to ban them. These days I only block bots when they're e.g. filling out forms or posting junk comment that a human has to moderate.

PeterisP 9y ago
Some kid running an automated vulnerability scan can be automatically detected and blocked this way, which can protect you if that scan would have succeeded otherwise. Of course, the proper way would be just to scan all that yourself and actually fix the problems, but that may be too difficult in some cases.
yani 9y ago
Scanning bots consume resources and traffic. I pay for both. They increase my bill so it makes sense to restrict them.
- eli 9y ago
  Having gone down this path before, I think time is better spent making your site more efficient so it doesn't matter. Developer time and site complexity are much more important commodities than bandwidth fees.
  - yani 9y ago
    I usually leave it to the devops to deal with it. If it has to be a developer on my team to do it then you are right, I will spend their time elsewhere.
raverbashing 9y ago
It seems you never had to deal with the amount of crap bots throw at every minor website out there. If you don't want your logs flooded with vuln. scanning and referrer spam, use fail2ban or a waf.
- eli 9y ago
  I have, in fact, been doing this for years. Why should I care about having pristine logs?
  - raverbashing 9y ago
    Yeah, who cares about the extra disk space, the extra overhead, the waste of resources, filtering through noise to understand some issues with the app, right? Isn't it lovely when every other referrer is trying to promote some online pharmacy or testing to see if you have a wp vulnerability? Sure you can filter it, but I'd rather deal with garbage at the source.
    - eli 9y ago
      A lesson I learned the hard way: if random bots are causing significant resourcing issues, then you need to address that directly. And if their impact isn't significant, why are you spending time on it? The disk space my log files consume is trivial.
semanticist 9y ago
While it's mostly bots trying to attack Wordpress, it's also blocked a few people spidering us aggressively (probably competitors) and other people trying attacks. Once it's scripted through a Chef cookbook, it's applied to all of our infrastructure with almost no effort, including any new boxes we launch. The biggest effort is running a script every few months to clear out the CloudFlare firewall, and I could probably stick that in a cron job, really. Our technical team is very small and the company entirely dependent on our website and our reputation, so we take multiple approaches to securing the site and servers. Each little thing we add helps my boss sleep better, and that alone is probably worth the initial effort. I definitely wouldn't do what some people are suggesting, manually reviewing logs/fail2ban matches and blocking IP addresses by hand. That's a waste of effort - and would probably be a full time job, too.

tptacek 9y ago
Don't do most of this.
- andyfleming 9y ago
  Care to elaborate?
  - tptacek 9y ago
    You almost certainly don't want to set up a WAF. Don't use any of those scanners, all of which look almost exclusively for vulnerabilities you don't have. Do look into getting a Burp license for every developer on your team. By all means, harden Nginx. Don't pay SpiderLabs to do it for you.
    - peterwwillis 9y ago
      A WAF will stop a lot of the stupid junk that a kid with a "how 2 hax0r" book can make work a disturbing amount of the time.
      - tptacek 9y ago
        If any of that stuff is actually working on your site, not having a WAF is the least of your problems. Don't waste time with this stuff.
        nxc18 9y ago
        I assume you are writing in an attempt to persuade. Your message is being lost/ignored as a result of your delivery. We want to understand your message, but you aren't giving enough background. I don't trust your message because there is no supporting background. If you're going to tell me to not do something, either: A) provide a compelling alternative (not doing $it is not a compelling alternative to $it in nearly all cases) B) explain in detail why you shouldn't do $it. Provide more details if asked. I want to believe what you are saying and be convinced by it, please help me. (Note: I'm not trying to be a dick, I've just known a lot of people who are perpetually ignored because of this communication style and take forever to learn that lesson because no one bothers to say anything)
        tptacek 9y ago
        In this instance, I'm just interested in getting the right answer written down and moving on.
        pvg 9y ago
        I'm not trying to be a dick You are succeeding without trying. You're essentially complaining some free advice does not fit your exact expectations and specifications. Which is fine, lots of free advice doesn't. Trying to pass this complaint off as some kind of favour to the person giving you free advice, though, is... less than great.
        irundebian 9y ago
        No, tptacek just doesn't give proper answers here because he thinks he can afford it with his status here. He could have just left links to further links/literature.
        pvg 9y ago
        There's nothing 'improper' about the answers, you just don't like how they're presented. As I said, that's a totally fair objection. It's just that nobody owes you free advice in the precise format you find acceptable. That's usually a feature of paid advice.
        TeMPOraL 9y ago
        I think tptacek's implicit point is this: all the problems you're trying to address with the tools listed are the problems you shouldn't be having in the first place, so, to quote, "focus on removing attack surface, rather than continuously monitoring it".
        anothergoogler 9y ago
        > We want Please speak for yourself alone. tptacek's comments in the thread are quite clear.
        hueving 9y ago
        tptacek is using an appeal to authority given his standing as a security expert on this forum. That's the only reason his comment containing no justification is being taken seriously.
        Alex3917 9y ago
        > If any of that stuff is actually working on your site, not having a WAF is the least of your problems. Is this advice assuming that it's devs who want the WAF, rather than ops or management? For devs clearly it's not what they should be focusing on. But saying something isn't worth doing because it shouldn't work is a terrible security principle.
        tptacek 9y ago
        I wrote a longer comment here, but decided it wasn't helping. So, I'm just going to go with: no, if you're really building an application and care about application security, no, you shouldn't waste time setting up a WAF tool designed to help 10,000 employee insurance companies make sure their Wordpress marketing site isn't exposing some vulnerability from 3 years ago. Not wasting time with dumb stuff like this isn't "terrible security"; every dumb thing you do exacts a cost from good things you could be doing instead.
        subcosmos 9y ago
        "Don't invest in a home security system because your windows should be completely shatterproof"
        balls187 9y ago
        More like, "Don't invest in home security, because your most precious valuables are stored in a safety deposit box at a bank, and everything else is covered by home insurance."
        pennaMan 9y ago
        "Don't invest in a home security system before you make sure your house has a door"
        subcosmos 9y ago
        Something tells me those taking objection to my comment are missing the point that I am criticizing the parent.... If your counterexample is just reinforcing the parents erroneous point that WAFs only protect against specific applications with known vulnerabilities, and that not all websites have "doors", then you are missing the point that many WAFs have detectors for fairly generic sql injection and path traversal techniques that could very well find problems with even custom application endpoints....
      - sebcat 9y ago
        A WAF will also add an attack surface, just like anti-virus software before them. Company X, a content publisher, has a wordpress site with a bunch of plugins. They hear that the security record of such a setup is less than ideal. They buy a WAF solution to protect them. Company Y, also a content publisher in the same space, realizes that they don't need dynamically generated content. They don't need tons of JS. They start publishing static content and what little dynamic functionality they have is well-compartmentalized, with interactions to the outside world carefully audited. Company Y could also get a WAF, but why? It increases the attack surface, Company Y probably don't have the time/expertise to audit it, it is not clear what benefits it will have.
    - sandGorgon 9y ago
      This is brilliant. Thanks !
    - _pdp_ 9y ago
      There are good burp alternatives too :)
      - tptacek 9y ago
        There definitely are! Your stuff included! Burp is just the industry standard.
        sandGorgon 9y ago
        So is it "fix whatever Burp points out" ? Is that a good first step ? Especially for api endpoints (versus websites)
        tptacek 9y ago
        You'd ideally want a dev process where all your developers were capable of conducting integration test runs with Burp; you'd probably set up a checklist of authz tests, and set up some custom wordlists for Intruder for database, filesystem, and DOM injection vectors.
        sandGorgon 9y ago
        Thanks. This was a superuseful suggestion. I can now Google a lot of stuff on my own.
      - deleted 9y ago
        [deleted]
  - peterwwillis 9y ago
    The problem with throwing tools at the thing is you have no idea what it's doing or how it works or IF it works, and it becomes forgotten about. But it's better than nothing in the short term while you figure out how to secure your application. Slap ModSecurity in front of your app while you go over OWASP and managing cloud infrastructure securely.
  - technion 9y ago
    In addition to the points discussed.. If you follow the Geekflare link to modsecurity, and then follow the link there to a download page, there's this warning: >NOTE: Some instabilities in the Nginx add-on have been reported (see the Github issues page for details). Please use the "nginx_refactoring" branch where possible for the most up to date version and stay tuned for the ModSecurity version 4. That's not something I would suggest throwing in production, given that warning has been there for years. The post itself describes Shadow daemon as "probably defunct". Ironbee's last commit was two years ago and I get an nxdomain looking for their website. AWS WAF is both expensive and and extremely manual. You won't get anything out of it without a major labour investment. Nginx's built in WAF is only in the pro edition, and outside the price of many people. This whole discussion however depends on your customer. I have an agency I support where I have to provide a report on every single hack attempt. That means if my Rails application gets hit with said three year old Wordpress exploit, and the web server "404", I have to report on how we "blocked" it. And no, "not applicable" is not an answer. So we throw money at commercial products. And sometimes I recommend doing so. But like security can involve just being aware of your risks, this use of a WAF is about being aware of the real problem you're solving, which is a paper work based one.

ivanr 9y ago

I came here to add an important detail about WAFs that's missing in this thread. For context, I am the original author of ModSecurity (but haven't worked on it since at least 2009). I wrote ModSecurity for three use cases that still make a lot of sense sense; they are: 1) full HTTP transaction logging (meaning request and response bodies), 2) rule-based analysis of HTTP traffic (largely to trigger logging and otherwise highlight anomalies) and 3) virtual patching. The first two use cases are all about traffic visibility, which has been, and remains a big problem. If we're talking about improving security, the third use case—virtual patching—is the exciting part. The story goes something like this: you discover a vulnerability in your code but you can't fix it immediately, for whatever reason (take your pick). In practice, it may be weeks until the problem is fixed. However, if you have ModSecurity installed, you can write a virtual patch for your problem, often to fix it 100% reliably. However, it's very important to understand that virtual patching doesn't mean just slapping together some rule that appears to be doing the job. Blacklisting doesn't work. To properly virtual-patch, you have to fully understand the vulnerability and the entry point to the application. You have to carefully whitelist all parameters in that particular location. In other words, you have to almost replicate the work that should have been done by the developer. Even in this context, it's not all rosy. Virtual patching requires a great deal of expertise to do properly. It's almost always better to spend your time fixing the actual problem. Sometimes, however, that's not possible to do in a reasonable time frame, and that's when a WAF (according to my definition), can be a life saver.

tptacek 9y ago
Thanks for this. I admire ModSecurity! But the subtext of my comment is the needs of typical HN people, most of whom manage applications they wrote, for small companies. Overwhelmingly, those people should spend any time they would spend on things like ModSecurity instead shoring up the security of their code.
FuckOffNeemo 9y ago
TLDR: Ivan, thank you for everything you've provided to the IT community. - - - - There's few people I am more pleased to see commenting here on a topic like Sec more than yourself Ivan, your work's exceptional. And the tools you've provided like Hardenize and SSL Labs have saved me countless hours. Your books have been an immense source of knowledge for me (and I'm relatively a newcomer by my colleagues standards). Really pleased to see you here on HN. Though, I'm not entirely surprised. The best of the best seem to haunt this place.

iuguy 9y ago

The parent posted what is pretty much the right answer, but to add: * Your logs should be there to help you find out what happened in the event of a compromise, not to detect it in progress. What OP should be monitoring is the activity that occurs on the system - logins, processes spawned (e.g. php or mysql suddenly spawning shells for no reason), changes to content, resource volume spikes and as the parent said, configuration changes. Much of this can be done with a variety of tools but there isn't really a one-size fits all approach. I'd start with resource spikes as it's easiest and work up to an OSSEC configuration reflecting the actual environment. Also by way of middle ground on the log front, Graylog2 is a happy medium between Splunk and ELK, but it's better to log fewer (but more useful) things than a load of noise.

viraptor 9y ago
> What OP should be monitoring is the activity that occurs on the system - logins, processes spawned (e.g. php or mysql suddenly spawning shells for no reason), In my experience, monitoring this takes about as much effort as adding a new apparmor / selinux / firejail / ... rule which would block it in the first place. I mean, with processes spawned we're almost at auditd level already.
unethical_ban 9y ago
>logs are not there.... to detect an attack in progress I vehemently disagree with this assertion. I also disagree with tptacek's asseration that any admin interface should be inaccessble from the public Internet. That's just not how a lot of small businesses, non-profits, etc. operate. 2FA? Sure. Geo-fencing? Keeps out noise. Basic auth / some front-end to minimize app-specific attack surface? Perhaps. Saying "you shouldn't look at logs to detect anomalous activity before the fact" is awful. So is "If you have and admin availability publicly, you're screwed". My saying is "Security is a spectrum". A bank should not have their firewall admin interface publicly available. A community newsletter blog would probably be alright.
- tptacek 9y ago
  I don't know what the maturity cutoff you're using here is, but I'll draw my line at "deploying on AWS". If you're on AWS, you have really no business exposing an admin interface on a routable address. Deploy it on an internal address and have admins use a VPN to reach it. There are organizations on AWS that have publicly-routable admin interfaces. They should all be working on plans to migrate away from that, and, in the meantime, they should be extra careful about authentication and monitoring. If you're too small to have a non-routable admin interface, you are definitely too small to pay for security products.
- iuguy 9y ago
  Mixing discussions between people makes it harder to respond to your comments. It's better to have one comment aimed at me and one aimed at tptacek. As for the content of your comment, it's hard to distinguish what's aimed at me and what's aimed at him, so I'll assume everything aside from your first sentence is aimed at him.

dalore 9y ago

Yeah I find young naive devops people implementing security watching and stuff like fail2ban but in the long run it reduces productivity. Stuff like fail2ban can block legitimate traffic (it blocked me on many occasion trying to work from home). When I was young I too used to try implement detection and banning but realised it's a waste of time. Key only access to ssh and make sure you at least subscribe to security updates for all packages is enough. For our own app we went the ELK route due to the number of log messages coming from our own application. But yes as you said it's a pain to manage. Also creating kibana graphs/views/desktop for each type of log is also painful. A good service I believe would be someone to offer a turnkey/shrink wrapped managed ELK for self hosting. Not only managing the system side of it, but also offering services to help create custom kibana graphs.

danieltillett 9y ago
Just set a short ban time in fail2ban. I have it set to 5 minutes. The aim is just to slow down the kiddies so that my logs don’t fill with a zillion pointless attempts.
- spurgu 9y ago
  And/or just reconnect through another server or VPN.
  - danieltillett 9y ago
    Yes, but my aim is more to create a molasses pit to slow down the log fill rate rather than to ban anyone for long. If I manage to lock myself out I just get up and get a drink of water. Using a short ban time keeps the load of running fail2ban to a minimum. I at one point was banning the kiddies/bots for a week and 40% of the server load was chewed up just running fail2ban.
    - gog 9y ago
      Fail2ban has issues when you have a lot of IPs blocked in memory. The solution I came up with to this problem is to use a custom ban action that adds an IP to the blocked ipset list with a given TTL, but set fail2ban to think that the IP has to be only banned for 5 seconds or so. The result is that ipset purges the IP from the list after the TTL expires, but fail2bans working set is a 5 second rolling windows of blocked IPs. This works great and it's being used in production for a couple of years.
      - danieltillett 9y ago
        Sounds like a great solution. Have you written about this in any detail?
- atomical 9y ago
  Do you really need the space or is it just some anxiety issue / ocd issue?
  - fragmede 9y ago
    If a sufficiently interested botnet discovers your SSH server, it'll hammer your server with 10's of connections a second, from many different IPs. That doesn't take up enough bandwidth to be noticeable, but it can fill up the disk, when you're least expecting, which can cause all sorts of trouble. Of course there are ways to deal with it, but that's just extra work.
    - atomical 9y ago
      Fill up a terabyte? Doubt it, especially since logs are compressed nightly.
      - ishaan12345 9y ago
        Are you serious? There are others https://riskemy.com/free-cloud-vulnerability-scanning-tools/ https://riskemy.com/free-cloud-vulnerability-scanning-tools/
        atomical 9y ago
        What does your comment have to do with mine? You seriously registered an account just to reply to me? What's going on?
        atomical 9y ago
        .
- dalore 9y ago
  When your server is down and you need to ssh into an arbitrary 5 minute wait is going to frustrate you. You don't really need fail2ban at all. Just don't log failed password attempts. Key access only, no one is going to get in.
  - danieltillett 9y ago
    I am not worried about people getting in via ssh, I am worried about some zero-day on some other service.
- combatentropy 9y ago
  > so that my logs don’t fill with a zillion pointless attempts What about moving SSH to a different port?
  - danieltillett 9y ago
    I do this, but it is the port scanning that I am tracking. It is hard to find your ssh port if you get banned port scanning.

danieltillett 9y ago

Removing attack surface make it viable to continuously monitor. I don’t think it is an either/or. Just out of curiosity has anyone with a cryptographically strong ssh password ever been hacked via the password authentication path unless they gave this password to someone else?

tptacek 9y ago
I don't know what "cryptographically strong" means in the context of a password, but I do know people with strong SSH passwords that got owned up, yes. Don't use "cryptographically strong" passwords with SSH. Use keys, disable passwords.
- danieltillett 9y ago
  I mean a password with enough entropy that also hasn't been used before and is not shared with anyone. Do you know how they were owned? Is there a write up anywhere as this is a super interesting topic.
- IncRnd 9y ago
  For passwords "cryptographically strong" normally means a password with appropriate complexity and sometimes unpredictability requirements, since entropy is almost nonexistent in passwords.
  - namibj 9y ago
    Something out of pwgen or diceware would do though. If your application has length requirements that preclude the latter, make sure they are liable if you loose something due to preventing you from using a secure password. (No one can remember a short secure Password for more than a small handful of applications.)
- weavie 9y ago
  > but I do know people with strong SSH passwords that got owned up By having their password brute forced, or some other means?
- sunstone 9y ago
  So for example, a 16 character random password is still vulnerable?
  - danieltillett 9y ago
    This is what I would like to know. If strong ssh passwords can be broken then there is a bigger issue than just turn off ssh password log in.
    - sunstone 8y ago
      I am pretty sure that 16 character random passwords are safe from dictionary attacks, and I suspect you and I are using the same process for protecting these passwords. tptacek is leary of our process while you and I are leary of the certs getting out of our control without our knowledge. Security is a handful of trade offs.

westpfelia 9y ago

Im currently building a SoC for a fairly large company in Germany and before that I worked for what probably by now through M&A is the largest MSSP in the world. I dont know if I agree with you that having a MSSP is out of the question. we had plenty of clients who only paid us maybe 2 to 3 thousand a year if even that. Most MSSPs will charge on a contract to contract basis revolving around log volume and estimated daily event creation. If you are only creating an estimated .1 events per day you could very well have a rather cheap contract.

zxcmx 9y ago
If it costs 2 or 3k a year your MSSP is probably not spending more than a few minutes a year understanding your logs. Realistically at that price the most likely number is zero minutes. There is room in the industry for players that aggregate logs in interesting ways and can share threat intel and track attack patterns between customers, but a lot of the time with MSSPs you will just find yourself paying for a really expensive splunk instance. Logs are fairly useless without occasional review (to at least verify you are collecting the things you think you are) and a decent understanding of the infrastructure they support. I don't know your business model or capabilities though so this might not apply in your case. Hopefully not. But there are plenty of MSSPs out there that will ingest logs, charge out the %]%{^{ for "monitoring" and add very little value. After the nth time you get alerts for PHP vulns on stuff which is clearly not running it, or something stopped sending logs months ago and no one noticed, or you find that firewalls logs from anything not in an unpublished support list don't really get parsed (yeah they will ingest it but just not tell you that nothing meaningful will happen with it..)... To be fair, monitoring by third parties is one of those things that you can't "do and forget about it", you need shared responsibility and a clear understanding of who is doing what. None of this is cheap though.

evgen 9y ago
It is also being taken seriously because 1) he is correct, and 2) those of us who do this for a living are literally sick to fucking death of having to explain to management, developers, and sysadmins that there is no silver bullet that you can buy that will fix the mistakes you should have cleaned up five years ago. You still have to do the work. If you need to hear this ("reduce attack surface rather than spend time/money on monitoring attack surface") more than once you are beyond help and it is time for us to move on to people for whom there is at least a possibility that they won't be p0wned in the next 12 months.
- irundebian 9y ago
  This isn't a religious/esoteric forum here. Technical statements should be backed by proper reasons regardless of who is stating something. If you are sick to tell technical reasons then why bothering posting on HN?
  - lvh 9y ago
    Because there are people who are willing to believe that something is good or bad without needing to understand the rationale behind it? Most people making the case for WAFs aren't posting detailed rationales of why WAFs are a good way to spend your resources either. Example, same author: Cryptographic Right Answers. (Disclaimer: I work with tptacek.)
    - irundebian 9y ago
      That's whataboutism. I'm not interested in anyone who fails at giving proper arguments. Neither when tptacek is it or someone else. The sense of giving rational arguments lies not only in the persuasion of the parent poster but also on the information of anybody else who is reading. And simple "Don't do most of this." and "You almost certainly don't want to set up a WAF." are just crappy answers which are based on authority and therefore meaningless.
      - tptacek 9y ago
        You are being extremely rude. Nobody on this thread has been as personally abrasive as you have. If you have a technical argument to make, make it; nobody is going to call you names for doing that. Technical arguments, even those you don't feel are adequately supported with evidence, are not insults. But "whataboutism" and "appeals to authority" and "religion" and "arrogance" are insults, and speak to motives and intents, which are things you have no business bringing into this discussion. Stop.
        irundebian 9y ago
        My posts aren't meant to be rude. I feel very sorry if I hurt you or others feelings. I'm not making a point on your general credibility and technical statements - quite the opposite - but some your answers today just seemed to be ignorant for me (and probably some others). If you're giving a simple statement without background (just based on your opinion) and you're friendly asked for your information, then imo you should either don't answer or give back more information, but not just continuing giving answers without background. That's all, and that's not meant to be rude. Just my opinion on your way of posting today, as others have giving me today.
        c22 9y ago
        No one means to be rude. Rudeness is the byproduct of a lack of self awareness and concern for social mores. Gp (and others in this ridiculous subthread) are trying to give you pointers that will improve the quality of your interactions with this community, which is ironic given the comments that sparked this discussion.
  - stronglikedan 9y ago
    You're relatively new, but after a while, you get to learn the credibility of some HN'ers by their usernames. Often, you can click their username to find previous comments from the same user where they expounded on the technical details, particularly for subjects that come up here a lot. I don't blame them for not wanting to go into detail each and every time, especially when their reputation gives their comments credibility in place of details. I'd also like to add that you may want to work on the tone of your comments if you'd like to get enough karma to really participate in the community. Your comment history seems a bit confrontational and argumentative. There are better, friendlier ways to make the same points. I'll defer to the guidelines[0] to explain how you may improve your communication skills. [0]https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
    - irundebian 9y ago
      I'm quite aware of some credibility of some HN users. That doesn't give them the right to give arrogant or non-argument-backed answers. Besides that: There is no reason to base your argument on the age of my account.
      - stronglikedan 9y ago
        You may have missed my edit to my previous comment regarding the tone of your comments. I suggest you go back and check it out. (Yet another sign of the age of your account, which is most definitely relevant.)
        irundebian 9y ago
        Sure, thanks for your advice. But that's very unrelated to this topic here. Furthermore I don't see much value in karma points from random strangers and a need for building some strange internet personality who avoids conflicts to prevent a decrease in its karma points. I also don't see how I can even more participate in this community with more karma points.
        9y ago
SmellyGeekBoy 9y ago
Nope. I upvoted it before even seeing who wrote it. The advice is absolutely spot on. Harden your servers, don't just install a load of tools that look for intrusions after-the-fact and call it a day. I'm not sure why some people aren't getting it.
raverbashing 9y ago
It's not an appeal to authority when you're really an expert. Or you go to the doctor and expect him to justify your treatment down to the minor detail? You sound like you talk to your doctor about $overpriced_drug.
- irundebian 9y ago
  And still there are doctors who are prescribing homeopathy and other effect-less "medicine". It's just reasonable to ask for rational arguments. That's the whole point of the rants here.
srean 9y ago
Its not that he owes a justification or an explanation or that readers can legitimately feel entitled to one. Paid support is another matter.
tptacek 9y ago
If you think I'm wrong, rebut.
- peterwwillis 9y ago
  The fan boys came out to rep you, but they're wrong. First of all - it takes less than an hour to set up a WAF, and it doesn't worsen your security position. Not only is it free, it's fast and easy and will demonstrably be better than no security at all. Saying "don't put up a firewall because it isn't perfect" is stupid. You don't have to be a security guru to get that, which obviously several people have, or they wouldn't all be giving you crap about it. Securing your application long term is a complex, expensive, time consuming process. Suggesting that spending a couple months to be "serious about security" is somehow better than first taking one hour to do the most bare minimum effort is ridiculous. If I have a computer on the internet, I put a firewall up. Is it defense in depth? No. But that doesn't mean it won't stop the most basic, most common attacks, and it takes no time and no skill to apply it. I don't know what weird thing you have against WAFs other than apparently they aren't cool or sophisticated enough, but 1) they remove attack surface, 2) they filter the most common drive-bys, 3) they can be both passive and active, 4) they're free, 5) they're easy, 6) they're quick to deploy. How is it 2018 and I have to explain to you of all people that the web equivalent of iptables might be a good idea if you have zero security?
  - viraptor 9y ago
    > WAFs (...) 1) they remove attack surface They do what you configure them to do. Without knowledge of what's allowed and what's not, they're going to have either false positives or false negatives. And if you have that knowledge, you can fix those issues in the web app rather than hide them with the WAF. A lot of preconfigured WAFs will remove specific matches you don't care about anyway rather than attack surface. > 2) they filter the most common drive-bys Common as in known proxy or webapp exploits? If that's the case, why not patch them? (There's a reason to use WAF if you can't patch but that costs time/money to get right and test) > 3) they can be both passive and active I'm not even sure what passive WAF means... just logging? > 4) they're free Only for very small traffic. > 5) they're easy 6) they're quick to deploy. See point 1. If you can't put the right restrictions on the app, why do you think you can put them in WAF? If they're not in the app in the first place, why do you think this way is quicker? It's not a magic box will know your requirements. You need a proper setup or the only thing you achieve is removing obviously silly cgi-bin requests from your log files. I'm not even against WAFs. Just think the gain from them is been none and minimal if you reduce it to "turn it on, it's cheap and easy". It's really similar to iptables - you can turn it on, but why are you doing it? (On a server anyway) If you have a specific reason/route to protect - great. If not, make sure you bind non-public services to local interfaces and you don't need iptables.

lvh 9y ago

I'm going to assume that whenever you say firewall you're talking about a WAF, since you mention "web equivalent of iptables". Disclaimer: I also work with 'tptacek, but opinions my own. > it takes less than an hour to set up a WAF Not if you want it to do anything useful. Default WAF rules are mostly useless and come with bypasses -- kinda like XSS filters in browsers, and browsers are in a way better position to judge if XSS is going on or not. You can sort-of get useful WAF rules if you encode most of the invariants of your system in your WAF -- but your app is in a strictly better position to do that sort of validation for you instead. > and it doesn't worsen your security position. Yes, except for the shiny new TLS MITM, the pathological parser performance cases, the new attack surface in that WAF itself (hands up if you've seen XSS in a WAF interface) and the continued operational cost of having new infrastructure that sees plaintext... Sure! And then I'm being generous in assuming we're not talking about a virtual appliance or as I like to call them "mystery *nix box you can't patch or audit". > Securing your application long term is a complex, expensive, time consuming process. Suggesting that spending a couple months to be "serious about security" is somehow better than first taking one hour to do the most bare minimum effort is ridiculous. Also, unlike WAFs, it's effective. To be clear: I am not conceding your point about cost. It does not take "months" to identify systemic appsec issues. It also does not take an hour to configure a WAF and have it do something vaguely useful for your application. Anecdotal evidence: Latacora literally does this systemic-discovery-and-remediation-plan thing for a living as a part of our engagement. "Months" is not our timeline for any kind of systemic appsec issue like SQLi or XSS. > ... the web equivalent of iptables might be a good idea Are you really sure you want to stand by that equivalence? How often does one come across iptables bypasses?

peterwwillis 9y ago
> Are you really sure you want to stand by that equivalence? Yes. Both require some knowledge of how to use the tool and rules, but both are not difficult for even a novice to pick up quickly. Not to mention both can be done by either a dev or a sysadmin in the cloud right now, neither of them require in-depth knowledge of the code, and the scope of the change is simple enough that it can be tested and applied quickly. > "Months" is not our timeline for any kind of systemic appsec issue like SQLi or XSS. The discussion was about a dev or sysadmin fixing all this themselves ("taking appsec seriously" etc). Not just running Burp on their website. After they identify the vulns they have to have meetings to discuss fixes, figure out how to patch them and test that it works, go through QA and then roll into production. A fix may involve several systems and most people's deploy process is painful. I've never seen that take less than three weeks, and six is more reasonable. Can you do it in less time? Of course. Does that actually happen? Not without paying a consultancy $300k to hold your hand for two weeks. It is completely plausible to get a WAF up and running and have it provide real protection immediately. CloudFlare's is probably the best example of this, considering the extensive add-ons they provide based on the intrusions they've seen. I don't like CloudFlare's TLS MITM either, but tens of thousands of their customers don't seem to mind. Just like iptables, ModSecurity doesn't do anything at first. You have to actually make some decisions. Does that mean you shouldn't spend the extra 15 minutes to read a manual and enable some protection? Earlier it was suggested that protecting against Drupal or Wordpress vulns is unnecessary. I disagree! There's no reason not to protect against drive-by vulns. Filters for XSS/SQLi, unnecessary HTTP features, malicious bots and scanners, directory traversal, RCE, etc are well worth preventing whenever possible. As for false positives, they're not as bad as they used to be. There are thorough guides designed to help you identify and remove them from your site. https://www.netnea.com/cms/apache-tutorial-7_including-modsecurity-core-rules/ https://www.netnea.com/cms/apache-tutorial-7_including-modse... https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/ https://www.netnea.com/cms/apache-tutorial-8_handling-false-... For a sysadmin this may take more time, just like it would if you were using iptables to secure a black box. A dev can change the filters immediately to suit their app. But even if you don't do any active filtering, the passive filters can at least alert you that something is happening so you can go fix it. This is so standard that Kubernetes includes it as an option in their Nginx Ingress Controller. This guide can get you up and running on your existing K8s install in 10 minutes: https://karlstoney.com/2018/02/23/nginx-ingress-modsecurity-and-secchatops/ https://karlstoney.com/2018/02/23/nginx-ingress-modsecurity-...