14 ms·

AWS GuardDuty – the Good, the Bad, and the Ugly

132 points by bnchandrapal 4y ago

m1keil 4y ago
GuardDuty is another example of brilliance of AWS pricing scheme and how they manage to twist your hand to pay extra which can cost quite a lot in the end of the month. When comparing EC2 to servers, nobody adds the added premiums of the extras. Things like CloudTrail, Support, GuardDuty, CloudWatch. All of these things have a variable cost that grows with usage and very hard to predict ahead of time. Just last week I discovered our GuardDuty bills went up from $15/month to $400/month. Inspecting closer, the issue was a small script that did AWS API call at a tight while loop in a couple of EC2 instances. So if you choose to enable GD, make sure to have your monitoring in place, gradually enable GD across the infra and establish clear baselines and alerting in place for costs.
- deleted 4y ago
  [deleted]
- bushbaba 4y ago
  Until you realize how much the alternatives cost. You think Palo Alto networks is cheap?
  - albert_e 4y ago
    Is there a feature and price comparison between the native AWS offering and Palo Alto somewhere? Would be super handy for one of our projects right now. Thanks!
  - Bluecobra 4y ago
    At least you can get a fixed cost if you go that route vs. your bills growing at an exponential rate. From what I have found, PA really wants you to commit to annual pricing with their VM appliances. If you go hourly you pay 73% more.
  - unethical_ban 4y ago
    Do you think Palo Alto's offerings and GuardDuty are comparable?
- philliphaydon 4y ago
  Na. At some point you scale to the point that $400/m is peanuts for the benefits you get. We used it at previous job and realised we were under constant attack and and it reduced our cou usage by 15% in reduced requests for the amount of traffic we were getting. No more random spikes. Improved our overall security and ultimately reduced costs on the AWS bill. But if you’re gonna switch it on and walk away then you’re not really using it.
  - m1keil 4y ago
    Yes of course, $400/month is peanuts so does $8000/month for certain companies. The point is not the absolute sum but how easy is it to spike your bill by 30 times. The way GD works, you are pretty much guaranteed to overpay for it sooner or later. If you can afford it.. great.
    - philliphaydon 4y ago
      One thing that we noticed was after switching it on, the EC2 instances were being hit directly, so we moved those into a private security group only accessible to the load balancer. RDS got restricted. S3 buckets fixed. Coupled with AWF to block on inregular activity, resulted in GD bill going down, not up. This is no different from programming. PHP has some awful code out in the wild, it doesn't mean PHP is shit just because people write bad code. The issue with AWS is it's far too easy for people to just spin stuff up and it works and they don't look at what they are being billed for, don't analysis their infrastructure, don't optimize. They just throw servers, containers, etc up into the wild then when the bill comes: "OH AWS BAD I got billed cos I just set it up and forgot about it, then when it worked they charged me for it, AWS is wrong, just go baremetal."
      - bnchandrapal 4y ago
        Out of curiosity, did you have any data lakes on S3? Did you find optimization techniques for the same?
        4y ago
- ramraj07 4y ago
  The biggest “hidden” cost for me was Elastic Blockstore IO charges! It started adding up quite quickly and I realized I had to think twice about doing IO intensive calculations on EC2! I switched over to lightsail (obviously these are for personal projects).
  - mkesper 4y ago
    Just use gp3, shouldn't be needed to request higher IO with it (except for DBs, probably cheaper to use DynamoDB etc then).
  - Bedon292 4y ago
    Yeah, provisioned IOPS can add up quick. Never had justification to spend on them myself. If I need IO I use instances with attached storage, otherwise I purely use the GP SSDs.
- ransom1538 4y ago
  Just make sure everything is multi-az that way if an entire zone goes down you will still be fine /s
- psanford 4y ago
  Its quite annoying that you can't disable parts of guardduty you don't get much value from. I think the CloudTrail monitoring is quite useful and the VPC flowlog monitoring is basically useless (to me, I have other means of doing host and network based monitoring, I'm sure that there are a lot of people who get meaningful value out of it). I'd like to be able to turn off flowlog monitoring and just use guardduty for cloudtrail monitoring, but that isn't an option. So I can either overpay for a bunch of extra things I don't get any value from or not enable guardduty at all.
- JCM9 4y ago
  I hear you but how is this any different than any alternative approaches? I mean yes you could throw a server under your desk and it would be cheaper. Should you run a server in prod with sensitive workloads without all these extra security bells and whistles (on prem or in the cloud)? No. No you should not. Compared to alternative security appliances and offerings this is still a good deal.
  - Jensson 4y ago
    > Should you run a server in prod with sensitive workloads without all these extra security bells and whistles (on prem or in the cloud)? No. No you should not. Why not? Have people forgotten how to run servers in the past decade?
    - Spivak 4y ago
      People haven’t forgotten, companies just don’t hire infra people. “Fire all your ops people do the cloud” has made it so that organizations have forgotten how to run servers. Working in infra but not at supermassive scale does really feel like the new COBOL programmer.
- Cthulhu_ 4y ago
  Managing and occasionally slashing costs is why AWS consultants can make really good money - if you're into this kinda thing, it's worthwhile learning how to analyze and optimize AWS bills.
  - Jensson 4y ago
    Well, before AWS big companies tended to have the same problem, but instead of paying for many AWS features they paid for a thousand different products and trying to make sense of all those bills for each product was hard. I know some people who worked as consultants to clean up all the services the company was paying for but didn't really need any longer, paying for around thousand different software products was the norm and not some unrealistically large number.
- stunt 4y ago
  how did you debug it? We have this weird thing where the cost of GuardDuty varies on different days of the month and stays the same from month to month. But we can't figure out what causes them. Can you actually see the events that GuardDuty has processed?
judge2020 4y ago
> The Ugly > Cost can get sky-high Is there _any_ service on AWS where you feel like you're getting more value than the dollars you're paying with (other than IAM and Free tier services)? It's no secret that AWS is one of the most successful and profitable modern businesses, but perhaps there's a hidden offering that does something, does it well, and costs very little compared to the value it brings.
- ceejayoz 4y ago
  RDS is one for me. I’ve set up a regionally replicated, point-in-time backuped DB cluster and have no desire to run one again.
- bnchandrapal 4y ago
  There are a few AWS security services which are free/priced reasonably. Some free services: 1. AWS Org (Disable services and enforce guardrails) 2. VPC (Create private networks) 3. IAM (User access and IAM policy analyzer to help with least priv) 4. IAM Access Analyzer (Alert on resources with cross account & public access) 5. SSM Inventory & Patch manager (Basic check if all VMs have security updates installed) Reasonably priced IMO: 1. AWS WAF with free managed rules (when rightly configured you get lesser FP and high ROI)
  - discodave 4y ago
    Those services are gateway, or requirements for using other services though. A VPC isn't useful without EC2 instances in it. AWS Organizations allows you to create more accounts, with more instances, databases etc in them!
- czbond 4y ago
  Most if not all services, I feel are a good enough value...when I consider my "time" or hiring consultants time to do the same, etc. A single example was GuardDuty above. I know how expensive similar to GuardDuty services are per month when you have to have a well planned strategy, implementation, execution, operations for threats... no matter if one does it "in house" or with "consultants" - the cost is very high to implement anything similar to GuardDuty. No matter if it is duct taped open source or enterprise offerings. And then you have to do that same iterative business process loop across infrastructure (servers/database), data centers, code deployment, DevOps, security, etc etc.
solatic 4y ago
> Disable access to services in all non-active regions using SCPs. This is key advice anyway. When setting up new AWS infrastructure for a new company, set up an AWS organization, and only enable us-east-1 (required for some global services like CloudFront) and maybe one additional region (if you don't want to put all your eggs in the us-east-1 basket). Don't enable additional regions that you don't need. Because most AWS APIs are regional, it makes finding aberrant infrastructure much, much easier, even if you're just combing through the console manually.
- didip 4y ago
  I would not start by default in us-east-1, unless you want chaos monkey as a feature.
  - bcjordan 4y ago
    Are there any historical metrics out there on uptime/stability by region? Would be interesting to see a top-line comparison (GCP and Azure regions would also be neat)
  - acdha 4y ago
    It’s not bad advice but I’d do it for latency for western clients more than this — I’ve been running in us-east-1 since the 2000s and there’ve been only a handful of times where we had a production outage on a properly-designed application (a network routing issue in 2011 or 2012, and a couple regional S3 or IAM issues). No, those weren’t perfect but during the same time period our professionally-managed data center resources had multiple weeks of complete downtime versus maybe a day cumulatively.
- thedougd 4y ago
  I recommend AWS Control Tower for getting this all setup. It's also compatible with Terraform in more than one way.
  - bnchandrapal 4y ago
    Ahhh. AWS Control tower has not cost but it requires AWS Config to be enabled. Config is yet another AWS service that can get costly over time (if continuous monitoring of changes is enabled)
yrgulation 4y ago
AWS managed to make things so much easier and cheaper compared to “classic” hosting that you now need twice as many devops employees and spend 10x in bills. Fortunately tech people aren’t financially literate so amazon can keep on squeezing all the while using free software made by the very same people. Congrats.
- arinlen 4y ago
  > AWS managed to make things so much easier and cheaper compared to “classic” hosting that you now need twice as many devops employees and spend 10x in bills. I'm far from a AWS fan but this take can't even be deemed an apples-to-orange comparison. "Classical" hosting at best matches EC2. The absolute high-end "classical" hosting offers at best also offer something resembling EC2's VPC. Forget about regions, let alone anything resembling availability zones. Everything else that AWS offers ends up being nice-to-have conveniences. Stuff like object storage and pub-sub and message queues and managed nosql and classical RDBMS services and managed kubernetes and integrated infrastrucure-as-code systems are way outside what a "classical" hosting company offers
  - deleted 4y ago
    [deleted]
- Uehreka 4y ago
  I mean like, this is fun to say, but y’all know this isn’t actually true right? (Well, the free software part is, but the implication in the first sentence isn’t) The past 10-15 years has seen enormous growth in eCommerce productivity as a portion of the overall economy (just google “gdp attributed to internet commerce” and similar phrases, there’s tons of data on this). The rise in the number of devops engineers and the amounts businesses spend on AWS hosting are often in service of business models that were simply impossible to even try before cloud computing. Sure, everyone on HN seems to have a story about an organization going all “architecture astronaut”nuts with Kubernetes and then ending up with slower/more expensive infra, and it’s fun to tell those to each other, but there’s clearly a huge amount of economic activity being enabled here that wasn’t happening before.
- 4y ago
pid-1 4y ago
My personal experience: GuardDuty is a very expensive false positive generator with a bad UI (because AWS). If money allows, I'd look at wiz.io instead
acdha 4y ago
I think this is a natural tendency to get a large bill and want it to be somebody else’s fault. The two cost drivers mentioned are S3 access logs and VPC flow logs. S3 access logs are required by most security standards and are the only way to get that feature: if you need it, you’re going to be setting those up in whatever cloud security tool you pick or build. This is also odd with the request to only monitor some buckets - not enabling logging is exactly how you’re intended to do that. VPC flow logs are odd, too: you actually don’t need to enable them for Guard Duty - one of its selling points is that you can globally enable it without the possibility of one of your organization’s accounts having it disabled due to accident or malice – but again, if your security policy requires this there’s no shortcut for any tool: you can get figures quickly before you run through the free tier and use those for your budgets. The DNS logging points are handled by separate products: if you want those features, check out the Route 53 resolver logging and firewall docs: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-dns-firewall.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/re... https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-dns-firewall-vpc-protections.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/re...

132 points by bnchandrapal 4y ago

GuardDuty is another example of brilliance of AWS pricing scheme and how they manage to twist your hand to pay extra which can cost quite a lot in the end of the month. When comparing EC2 to servers, nobody adds the added premiums of the extras. Things like CloudTrail, Support, GuardDuty, CloudWatch. All of these things have a variable cost that grows with usage and very hard to predict ahead of time. Just last week I discovered our GuardDuty bills went up from $15/month to $400/month. Inspecting closer, the issue was a small script that did AWS API call at a tight while loop in a couple of EC2 instances. So if you choose to enable GD, make sure to have your monitoring in place, gradually enable GD across the infra and establish clear baselines and alerting in place for costs.

deleted 4y ago
[deleted]
bushbaba 4y ago
Until you realize how much the alternatives cost. You think Palo Alto networks is cheap?
- albert_e 4y ago
  Is there a feature and price comparison between the native AWS offering and Palo Alto somewhere? Would be super handy for one of our projects right now. Thanks!
- Bluecobra 4y ago
  At least you can get a fixed cost if you go that route vs. your bills growing at an exponential rate. From what I have found, PA really wants you to commit to annual pricing with their VM appliances. If you go hourly you pay 73% more.
- unethical_ban 4y ago
  Do you think Palo Alto's offerings and GuardDuty are comparable?
philliphaydon 4y ago
Na. At some point you scale to the point that $400/m is peanuts for the benefits you get. We used it at previous job and realised we were under constant attack and and it reduced our cou usage by 15% in reduced requests for the amount of traffic we were getting. No more random spikes. Improved our overall security and ultimately reduced costs on the AWS bill. But if you’re gonna switch it on and walk away then you’re not really using it.
- m1keil 4y ago
  Yes of course, $400/month is peanuts so does $8000/month for certain companies. The point is not the absolute sum but how easy is it to spike your bill by 30 times. The way GD works, you are pretty much guaranteed to overpay for it sooner or later. If you can afford it.. great.
  - philliphaydon 4y ago
    One thing that we noticed was after switching it on, the EC2 instances were being hit directly, so we moved those into a private security group only accessible to the load balancer. RDS got restricted. S3 buckets fixed. Coupled with AWF to block on inregular activity, resulted in GD bill going down, not up. This is no different from programming. PHP has some awful code out in the wild, it doesn't mean PHP is shit just because people write bad code. The issue with AWS is it's far too easy for people to just spin stuff up and it works and they don't look at what they are being billed for, don't analysis their infrastructure, don't optimize. They just throw servers, containers, etc up into the wild then when the bill comes: "OH AWS BAD I got billed cos I just set it up and forgot about it, then when it worked they charged me for it, AWS is wrong, just go baremetal."
    - bnchandrapal 4y ago
      Out of curiosity, did you have any data lakes on S3? Did you find optimization techniques for the same?
      - 4y ago
ramraj07 4y ago
The biggest “hidden” cost for me was Elastic Blockstore IO charges! It started adding up quite quickly and I realized I had to think twice about doing IO intensive calculations on EC2! I switched over to lightsail (obviously these are for personal projects).
- mkesper 4y ago
  Just use gp3, shouldn't be needed to request higher IO with it (except for DBs, probably cheaper to use DynamoDB etc then).
- Bedon292 4y ago
  Yeah, provisioned IOPS can add up quick. Never had justification to spend on them myself. If I need IO I use instances with attached storage, otherwise I purely use the GP SSDs.
ransom1538 4y ago
Just make sure everything is multi-az that way if an entire zone goes down you will still be fine /s
psanford 4y ago
Its quite annoying that you can't disable parts of guardduty you don't get much value from. I think the CloudTrail monitoring is quite useful and the VPC flowlog monitoring is basically useless (to me, I have other means of doing host and network based monitoring, I'm sure that there are a lot of people who get meaningful value out of it). I'd like to be able to turn off flowlog monitoring and just use guardduty for cloudtrail monitoring, but that isn't an option. So I can either overpay for a bunch of extra things I don't get any value from or not enable guardduty at all.
JCM9 4y ago
I hear you but how is this any different than any alternative approaches? I mean yes you could throw a server under your desk and it would be cheaper. Should you run a server in prod with sensitive workloads without all these extra security bells and whistles (on prem or in the cloud)? No. No you should not. Compared to alternative security appliances and offerings this is still a good deal.
- Jensson 4y ago
  > Should you run a server in prod with sensitive workloads without all these extra security bells and whistles (on prem or in the cloud)? No. No you should not. Why not? Have people forgotten how to run servers in the past decade?
  - Spivak 4y ago
    People haven’t forgotten, companies just don’t hire infra people. “Fire all your ops people do the cloud” has made it so that organizations have forgotten how to run servers. Working in infra but not at supermassive scale does really feel like the new COBOL programmer.
Cthulhu_ 4y ago
Managing and occasionally slashing costs is why AWS consultants can make really good money - if you're into this kinda thing, it's worthwhile learning how to analyze and optimize AWS bills.
- Jensson 4y ago
  Well, before AWS big companies tended to have the same problem, but instead of paying for many AWS features they paid for a thousand different products and trying to make sense of all those bills for each product was hard. I know some people who worked as consultants to clean up all the services the company was paying for but didn't really need any longer, paying for around thousand different software products was the norm and not some unrealistically large number.
stunt 4y ago
how did you debug it? We have this weird thing where the cost of GuardDuty varies on different days of the month and stays the same from month to month. But we can't figure out what causes them. Can you actually see the events that GuardDuty has processed?

judge2020 4y ago

> The Ugly > Cost can get sky-high Is there _any_ service on AWS where you feel like you're getting more value than the dollars you're paying with (other than IAM and Free tier services)? It's no secret that AWS is one of the most successful and profitable modern businesses, but perhaps there's a hidden offering that does something, does it well, and costs very little compared to the value it brings.

ceejayoz 4y ago
RDS is one for me. I’ve set up a regionally replicated, point-in-time backuped DB cluster and have no desire to run one again.
bnchandrapal 4y ago
There are a few AWS security services which are free/priced reasonably. Some free services: 1. AWS Org (Disable services and enforce guardrails) 2. VPC (Create private networks) 3. IAM (User access and IAM policy analyzer to help with least priv) 4. IAM Access Analyzer (Alert on resources with cross account & public access) 5. SSM Inventory & Patch manager (Basic check if all VMs have security updates installed) Reasonably priced IMO: 1. AWS WAF with free managed rules (when rightly configured you get lesser FP and high ROI)
- discodave 4y ago
  Those services are gateway, or requirements for using other services though. A VPC isn't useful without EC2 instances in it. AWS Organizations allows you to create more accounts, with more instances, databases etc in them!
czbond 4y ago
Most if not all services, I feel are a good enough value...when I consider my "time" or hiring consultants time to do the same, etc. A single example was GuardDuty above. I know how expensive similar to GuardDuty services are per month when you have to have a well planned strategy, implementation, execution, operations for threats... no matter if one does it "in house" or with "consultants" - the cost is very high to implement anything similar to GuardDuty. No matter if it is duct taped open source or enterprise offerings. And then you have to do that same iterative business process loop across infrastructure (servers/database), data centers, code deployment, DevOps, security, etc etc.

solatic 4y ago

> Disable access to services in all non-active regions using SCPs. This is key advice anyway. When setting up new AWS infrastructure for a new company, set up an AWS organization, and only enable us-east-1 (required for some global services like CloudFront) and maybe one additional region (if you don't want to put all your eggs in the us-east-1 basket). Don't enable additional regions that you don't need. Because most AWS APIs are regional, it makes finding aberrant infrastructure much, much easier, even if you're just combing through the console manually.

didip 4y ago
I would not start by default in us-east-1, unless you want chaos monkey as a feature.
- bcjordan 4y ago
  Are there any historical metrics out there on uptime/stability by region? Would be interesting to see a top-line comparison (GCP and Azure regions would also be neat)
- acdha 4y ago
  It’s not bad advice but I’d do it for latency for western clients more than this — I’ve been running in us-east-1 since the 2000s and there’ve been only a handful of times where we had a production outage on a properly-designed application (a network routing issue in 2011 or 2012, and a couple regional S3 or IAM issues). No, those weren’t perfect but during the same time period our professionally-managed data center resources had multiple weeks of complete downtime versus maybe a day cumulatively.
thedougd 4y ago
I recommend AWS Control Tower for getting this all setup. It's also compatible with Terraform in more than one way.
- bnchandrapal 4y ago
  Ahhh. AWS Control tower has not cost but it requires AWS Config to be enabled. Config is yet another AWS service that can get costly over time (if continuous monitoring of changes is enabled)

yrgulation 4y ago

AWS managed to make things so much easier and cheaper compared to “classic” hosting that you now need twice as many devops employees and spend 10x in bills. Fortunately tech people aren’t financially literate so amazon can keep on squeezing all the while using free software made by the very same people. Congrats.

arinlen 4y ago
> AWS managed to make things so much easier and cheaper compared to “classic” hosting that you now need twice as many devops employees and spend 10x in bills. I'm far from a AWS fan but this take can't even be deemed an apples-to-orange comparison. "Classical" hosting at best matches EC2. The absolute high-end "classical" hosting offers at best also offer something resembling EC2's VPC. Forget about regions, let alone anything resembling availability zones. Everything else that AWS offers ends up being nice-to-have conveniences. Stuff like object storage and pub-sub and message queues and managed nosql and classical RDBMS services and managed kubernetes and integrated infrastrucure-as-code systems are way outside what a "classical" hosting company offers
- deleted 4y ago
  [deleted]
Uehreka 4y ago
I mean like, this is fun to say, but y’all know this isn’t actually true right? (Well, the free software part is, but the implication in the first sentence isn’t) The past 10-15 years has seen enormous growth in eCommerce productivity as a portion of the overall economy (just google “gdp attributed to internet commerce” and similar phrases, there’s tons of data on this). The rise in the number of devops engineers and the amounts businesses spend on AWS hosting are often in service of business models that were simply impossible to even try before cloud computing. Sure, everyone on HN seems to have a story about an organization going all “architecture astronaut”nuts with Kubernetes and then ending up with slower/more expensive infra, and it’s fun to tell those to each other, but there’s clearly a huge amount of economic activity being enabled here that wasn’t happening before.
4y ago

pid-1 4y ago

My personal experience: GuardDuty is a very expensive false positive generator with a bad UI (because AWS). If money allows, I'd look at wiz.io instead

acdha 4y ago

I think this is a natural tendency to get a large bill and want it to be somebody else’s fault. The two cost drivers mentioned are S3 access logs and VPC flow logs. S3 access logs are required by most security standards and are the only way to get that feature: if you need it, you’re going to be setting those up in whatever cloud security tool you pick or build. This is also odd with the request to only monitor some buckets - not enabling logging is exactly how you’re intended to do that. VPC flow logs are odd, too: you actually don’t need to enable them for Guard Duty - one of its selling points is that you can globally enable it without the possibility of one of your organization’s accounts having it disabled due to accident or malice – but again, if your security policy requires this there’s no shortcut for any tool: you can get figures quickly before you run through the free tier and use those for your budgets. The DNS logging points are handled by separate products: if you want those features, check out the Route 53 resolver logging and firewall docs: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-dns-firewall.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/re... https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-dns-firewall-vpc-protections.html https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/re...