4 ms·

We received this as well except 1. We don’t use Java 2. The instances in the email are not in our accounts We later received another email providing some vag

by taf2 5y ago

We received this as well except 1. We don’t use Java 2. The instances in the email are not in our accounts We later received another email providing some vague reasons for this… On a Friday evening I was not very happy spending time trying to hunt down the invalid instances

gunapologist99 5y ago
Wow... I'd be curious as to whether or not this has occurred to anyone else. What were the vague reasons?
- mtmail 5y ago
  We had a non-AWS false positive: when sending the log4j exploit string with prepared unique subdomain via email there was an anti-spam system which did a DNS request on the subdomain. Apparently it looks for any content remotely looking like a URL and probes it. Similar to what was reported about Protonmail here https://news.ycombinator.com/item?id=29537549 https://news.ycombinator.com/item?id=29537549
- taf2 5y ago
  Here it is below - for us it was #4 which i should clearify was not vague just my tired memory of it - since once i received this i was very relieved and proceeded to sleep. I can tell they are definitely trying and IMO doing the right thing with their efforts below was the followup email that let us get sleep. Earlier today, we provided you a notification with a list of instances that may still be running a version of log4j that has a known security vulnerability and needs to be patched. We want to provide you additional details about that email. The list of instances was obtained by monitoring for specific DNS queries which are typically used when targeting the log4j vulnerability. These DNS lookups can indicate that someone is attempting to exploit the log4j vulnerability on your instance. Each of the instances provided has made a DNS lookup to one of the suspect domains between 12:00 AM and 11:59 PM PST on December 16th 2021. While we are not able to tell whether the instance was compromised, we strongly recommend that you take action to update log4j across all of your Java environments, whether they are publicly accessible or not. In some cases, the list of instances included instance IDs that may not currently be present within your EC2 environment. This happened for a number of reasons: 1. EC2 instances may have been terminated since the scan was completed at 11:59 AM PST on December 16th 2021; 2. EC2 instances that have been stopped and restarted may appear with the incorrect instance ID; 3. ECS, EKS, and Fargate containers have been included with the underlying instance ID instead of the container ID; 4. EC2 instances used by underlying network services were erroneously included in the list of instances. These services are not themselves running unpatched log4j, but can be indicative of these DNS queries coming from within your VPC. While it is not always possible for us to pinpoint the exact instances making these DNS queries, it is critically important that you patch all Java environments for the log4j issue, whether they are publicly accessible or not. To help you, we are also providing a 15-day free trial to Amazon Inspector which can assist you in finding vulnerable resources by scanning your Amazon Elastic Compute Cloud (EC2) instances and container images for the log4j vulnerability. Please take the steps explained in this security blog post [1] to protect your resources. You also can find more information about log4j in our security bulletin [2]. If you have any questions or concerns, please reach out to AWS Support [3]. [1] https://aws.amazon.com/blogs/security/using-aws-security-services-to-protect-against-detect-and-respond-to-the-log4j-vulnerability/ https://aws.amazon.com/blogs/security/using-aws-security-ser... [2] https://aws.amazon.com/security/security-bulletins/AWS-2021-006/ https://aws.amazon.com/security/security-bulletins/AWS-2021-... [3] https://console.aws.amazon.com/support https://console.aws.amazon.com/support
- jhealy 5y ago
  It happened to us too. There's no JVM (or log4j) in our environment, but we received this notification listing 6 instance IDs as having done DNS lookups to suspect domains. No trace of those instance IDs in our account over the past few weeks, and after following up with a ticket we were told they're actually the instance IDs that sit underneath some fargate tasks (no info on what tasks or ECS service of course, because that would be sensible). We've rechecked the bits of our stack that run on Fargate and confirmed there's definitely no JVM, so we figure it must be a false positive. Maybe DNS lookups to customer controlled hostnames (which we support as part of a feature, and sandbox carefully).
bink 5y ago
We received notifications for instances that do not and have never run java as well. These instances don't provide DNS resolution for any other hosts or act as proxies. I'm curious if there was some data corruption on the AWS side.
tgmatt 5y ago
We also received that email, but it was also followed up by one of our TAMs. For some inexplicable reason, the instance IDs in the email are their AWS internal IDs and is not visible to us, so it's actually pretty useless. They followed up and gave the actual instance for us to investigate, but we think it's a false positive as there is no Java on that instance.
OJFord 5y ago
I had it relating to images in ECR, and similarly spent a while hunting them down. They were in my account, but surely the email could've told me which region, or even linked them directly!

by taf2 5y ago

gunapologist99 5y ago

Wow... I'd be curious as to whether or not this has occurred to anyone else. What were the vague reasons?

mtmail 5y ago
We had a non-AWS false positive: when sending the log4j exploit string with prepared unique subdomain via email there was an anti-spam system which did a DNS request on the subdomain. Apparently it looks for any content remotely looking like a URL and probes it. Similar to what was reported about Protonmail here https://news.ycombinator.com/item?id=29537549 https://news.ycombinator.com/item?id=29537549
taf2 5y ago
Here it is below - for us it was #4 which i should clearify was not vague just my tired memory of it - since once i received this i was very relieved and proceeded to sleep. I can tell they are definitely trying and IMO doing the right thing with their efforts below was the followup email that let us get sleep. Earlier today, we provided you a notification with a list of instances that may still be running a version of log4j that has a known security vulnerability and needs to be patched. We want to provide you additional details about that email. The list of instances was obtained by monitoring for specific DNS queries which are typically used when targeting the log4j vulnerability. These DNS lookups can indicate that someone is attempting to exploit the log4j vulnerability on your instance. Each of the instances provided has made a DNS lookup to one of the suspect domains between 12:00 AM and 11:59 PM PST on December 16th 2021. While we are not able to tell whether the instance was compromised, we strongly recommend that you take action to update log4j across all of your Java environments, whether they are publicly accessible or not. In some cases, the list of instances included instance IDs that may not currently be present within your EC2 environment. This happened for a number of reasons: 1. EC2 instances may have been terminated since the scan was completed at 11:59 AM PST on December 16th 2021; 2. EC2 instances that have been stopped and restarted may appear with the incorrect instance ID; 3. ECS, EKS, and Fargate containers have been included with the underlying instance ID instead of the container ID; 4. EC2 instances used by underlying network services were erroneously included in the list of instances. These services are not themselves running unpatched log4j, but can be indicative of these DNS queries coming from within your VPC. While it is not always possible for us to pinpoint the exact instances making these DNS queries, it is critically important that you patch all Java environments for the log4j issue, whether they are publicly accessible or not. To help you, we are also providing a 15-day free trial to Amazon Inspector which can assist you in finding vulnerable resources by scanning your Amazon Elastic Compute Cloud (EC2) instances and container images for the log4j vulnerability. Please take the steps explained in this security blog post [1] to protect your resources. You also can find more information about log4j in our security bulletin [2]. If you have any questions or concerns, please reach out to AWS Support [3]. [1] https://aws.amazon.com/blogs/security/using-aws-security-services-to-protect-against-detect-and-respond-to-the-log4j-vulnerability/ https://aws.amazon.com/blogs/security/using-aws-security-ser... [2] https://aws.amazon.com/security/security-bulletins/AWS-2021-006/ https://aws.amazon.com/security/security-bulletins/AWS-2021-... [3] https://console.aws.amazon.com/support https://console.aws.amazon.com/support
jhealy 5y ago
It happened to us too. There's no JVM (or log4j) in our environment, but we received this notification listing 6 instance IDs as having done DNS lookups to suspect domains. No trace of those instance IDs in our account over the past few weeks, and after following up with a ticket we were told they're actually the instance IDs that sit underneath some fargate tasks (no info on what tasks or ECS service of course, because that would be sensible). We've rechecked the bits of our stack that run on Fargate and confirmed there's definitely no JVM, so we figure it must be a false positive. Maybe DNS lookups to customer controlled hostnames (which we support as part of a feature, and sandbox carefully).

bink 5y ago

We received notifications for instances that do not and have never run java as well. These instances don't provide DNS resolution for any other hosts or act as proxies. I'm curious if there was some data corruption on the AWS side.

tgmatt 5y ago

We also received that email, but it was also followed up by one of our TAMs. For some inexplicable reason, the instance IDs in the email are their AWS internal IDs and is not visible to us, so it's actually pretty useless. They followed up and gave the actual instance for us to investigate, but we think it's a false positive as there is no Java on that instance.

OJFord 5y ago

I had it relating to images in ECR, and similarly spent a while hunting them down. They were in my account, but surely the email could've told me which region, or even linked them directly!