7 ms·

Blacklisted as Malware: a Downside of using Amazon EC2

36 points by dpapathanasiou 17y ago

jmount 17y ago
I wish the issue was explained a bit more in the article. In my opinion if the blocking was due to host-names then it had nothing to do with EC2. If the blocking was due to re-use of IP addresses then maybe it had something to do with EC2 (but the article didn't explicitly claim that). As far as I can tell the effect was due to using a outsider server at all (instead of carrying one into the client's own private data center), which I think has nothing to do with having used EC2.
- dpapathanasiou 17y ago
  It was based on IP address. You're right in that all hosting services suffer this problem to some degree, but it's a particular problem on EC2 since it's so easy to request IP for an instance that does something bad, then release back into the pool for other instances to use, a short time later. It only costs a few dollars to that on EC2, whereas most other hosting services require at least a month's commitment and higher fee(s) before you can defile one of their IP addresses. BTW, the reason I posted was to find out what other blacklists might be out there, so if your company is blocking access, I'd appreciate letting me know which service they're using.
  - talison 17y ago
    Are you sure it was based on IP address? This domain was used to host questionable links a few years ago, including suspicious antivirus software: http://web.archive.org/web/*/workstax.com http://web.archive.org/web/*/workstax.com http://web.archive.org/web/20070127212047/http://workstax.com/ http://web.archive.org/web/20070127212047/http://workstax.co...
    - dpapathanasiou 17y ago
      You're right about the prior activity for the domain, but both Blue Coat and McAfee complained based on the IP address we're using now (i.e., the EC2 address). But my purpose in posting this is to ask: what other filtering lists are out there? Getting ourselves off is easy now, since we are a legitimate site; the hard part is figuring out who else may be blocking us.
tcc619 17y ago
what if you get an amazon elastic IP which isn't blacklisted?
blantonl 17y ago
This is a very large problem that Amazon needs to spend some time addressing. I run a fairly large production site on EC2 (14 instances) and when we went into production we quickly found out we were unable to send email from our servers to anyone... just about every email provider has blacklisted EC2's address space. I literally have to run an "off site" server from Amazon as a mail relay to work around this issue. Word of warning, EC2 is awesome, but this is an important lesson learned if you are moving to the cloud.
datums 17y ago
The blacklisting you experienced, could have happened with any other host. I've seen this happen with dedicated hosting companies, where a server was previously used by spammers and the ips allocated to a new client. When they try to send mail, guess what they're on all the major rbls. As far as sending mail out from the EC2 environment, that's an on going problem. Take a look at this http://www.spamhaus.org/sbl/sbl.lasso?query=SBL79954 http://www.spamhaus.org/sbl/sbl.lasso?query=SBL79954 . I think AWS users would be willing to pay for clean ips that are rate limited. Maybe an EC2 SMTP Service. Your block is using a browser bl and I don't know how those are shared by http filters.
holdenk 17y ago
You can try something like http://www.mxtoolbox.com/blacklists.aspx http://www.mxtoolbox.com/blacklists.aspx , it checks a bunch of RBLs all at once.
ckinnan 17y ago
Amazon needs to check its IPs against the blacklists before releasing them back into the AWS pool.
- shpxnvz 17y ago
  Seems like this points to a longer term problem with blacklisting by IP address as they continue to become more transient with respect to the hosted service.
  - petewarden 17y ago
    There's a similar problem with some API providers like Twitter throttling usage from App Engine and EC2 IP ranges: http://news.ycombinator.com/item?id=793939 http://news.ycombinator.com/item?id=793939
- talison 17y ago
  Let's not rush to judge Amazon. It's not clear this is an IP address issue. This domain name has a questionable history: http://web.archive.org/web/20070127212047/http://workstax.com/ http://web.archive.org/web/20070127212047/http://workstax.co...
jgilliam 17y ago
I recently found http://sendgrid.com/ http://sendgrid.com/ which is designed to solve the email sending problem from EC2, and includes a REST API.
- petewarden 17y ago
  The founders Isaac and Jose were in Techstars with me this summer, and I highly recommend SendGrid. They've got an awesome plugin architecture too that's an immense time saver for things like adding and managing unsubscribe links.
spudlyo 17y ago
define(`SMART_HOST', `some.host.with.a.clean.ip')dnl
- mxtoolbox 17y ago
  When creating mail severs for testing purposes, we too have noticed that many of the IP addresses in the AWS pool are indeed on blaclists or private reputation lists. A smarthost is certainly recommended if you want mail from one of these IP addresses to make it past spam filters. In theory, once you've had the IP address for a while, you should be able to get it delisted and cleared up, but some of these lists and toosl will just block entire subnets that are known to frequently change hands. I think the only real solution would be for Amazon to set aside some IP space and call it premium, and only give it out to validated or longstanding customers, IMHO. @MxToolbox
akamaka 17y ago
I think this speaks more to the low quality of these blacklists than to a problem at Amazon. Did anyone who noticed the offending site contact Amazon to have it removed, or are they simply content to keep adding more blocked IPs as the offender jumps from server to server on EC2?
ntoper 17y ago
I am the founder of http://critsend.com http://critsend.com, we make your email ends up into its appropriate inbox. We have a special offer for EC2 customers, we have some servers directly at AWS so latency is low and bandwidth is free for them. We are currently in closed beta but if you subscribe and add say you saw us on Hacker News, we will send you an invite asap. Our first 1,000 emails are free and after that we are pay per use.

I wish the issue was explained a bit more in the article. In my opinion if the blocking was due to host-names then it had nothing to do with EC2. If the blocking was due to re-use of IP addresses then maybe it had something to do with EC2 (but the article didn't explicitly claim that). As far as I can tell the effect was due to using a outsider server at all (instead of carrying one into the client's own private data center), which I think has nothing to do with having used EC2.

dpapathanasiou 17y ago
It was based on IP address. You're right in that all hosting services suffer this problem to some degree, but it's a particular problem on EC2 since it's so easy to request IP for an instance that does something bad, then release back into the pool for other instances to use, a short time later. It only costs a few dollars to that on EC2, whereas most other hosting services require at least a month's commitment and higher fee(s) before you can defile one of their IP addresses. BTW, the reason I posted was to find out what other blacklists might be out there, so if your company is blocking access, I'd appreciate letting me know which service they're using.
- talison 17y ago
  Are you sure it was based on IP address? This domain was used to host questionable links a few years ago, including suspicious antivirus software: http://web.archive.org/web/*/workstax.com http://web.archive.org/web/*/workstax.com http://web.archive.org/web/20070127212047/http://workstax.com/ http://web.archive.org/web/20070127212047/http://workstax.co...
  - dpapathanasiou 17y ago
    You're right about the prior activity for the domain, but both Blue Coat and McAfee complained based on the IP address we're using now (i.e., the EC2 address). But my purpose in posting this is to ask: what other filtering lists are out there? Getting ourselves off is easy now, since we are a legitimate site; the hard part is figuring out who else may be blocking us.

tcc619 17y ago

what if you get an amazon elastic IP which isn't blacklisted?

blantonl 17y ago

This is a very large problem that Amazon needs to spend some time addressing. I run a fairly large production site on EC2 (14 instances) and when we went into production we quickly found out we were unable to send email from our servers to anyone... just about every email provider has blacklisted EC2's address space. I literally have to run an "off site" server from Amazon as a mail relay to work around this issue. Word of warning, EC2 is awesome, but this is an important lesson learned if you are moving to the cloud.

datums 17y ago

The blacklisting you experienced, could have happened with any other host. I've seen this happen with dedicated hosting companies, where a server was previously used by spammers and the ips allocated to a new client. When they try to send mail, guess what they're on all the major rbls. As far as sending mail out from the EC2 environment, that's an on going problem. Take a look at this http://www.spamhaus.org/sbl/sbl.lasso?query=SBL79954 http://www.spamhaus.org/sbl/sbl.lasso?query=SBL79954 . I think AWS users would be willing to pay for clean ips that are rate limited. Maybe an EC2 SMTP Service. Your block is using a browser bl and I don't know how those are shared by http filters.

holdenk 17y ago

You can try something like http://www.mxtoolbox.com/blacklists.aspx http://www.mxtoolbox.com/blacklists.aspx , it checks a bunch of RBLs all at once.

ckinnan 17y ago

Amazon needs to check its IPs against the blacklists before releasing them back into the AWS pool.

shpxnvz 17y ago
Seems like this points to a longer term problem with blacklisting by IP address as they continue to become more transient with respect to the hosted service.
- petewarden 17y ago
  There's a similar problem with some API providers like Twitter throttling usage from App Engine and EC2 IP ranges: http://news.ycombinator.com/item?id=793939 http://news.ycombinator.com/item?id=793939
talison 17y ago
Let's not rush to judge Amazon. It's not clear this is an IP address issue. This domain name has a questionable history: http://web.archive.org/web/20070127212047/http://workstax.com/ http://web.archive.org/web/20070127212047/http://workstax.co...

jgilliam 17y ago

I recently found http://sendgrid.com/ http://sendgrid.com/ which is designed to solve the email sending problem from EC2, and includes a REST API.

petewarden 17y ago
The founders Isaac and Jose were in Techstars with me this summer, and I highly recommend SendGrid. They've got an awesome plugin architecture too that's an immense time saver for things like adding and managing unsubscribe links.

spudlyo 17y ago

define(`SMART_HOST', `some.host.with.a.clean.ip')dnl

mxtoolbox 17y ago
When creating mail severs for testing purposes, we too have noticed that many of the IP addresses in the AWS pool are indeed on blaclists or private reputation lists. A smarthost is certainly recommended if you want mail from one of these IP addresses to make it past spam filters. In theory, once you've had the IP address for a while, you should be able to get it delisted and cleared up, but some of these lists and toosl will just block entire subnets that are known to frequently change hands. I think the only real solution would be for Amazon to set aside some IP space and call it premium, and only give it out to validated or longstanding customers, IMHO. @MxToolbox

akamaka 17y ago

I think this speaks more to the low quality of these blacklists than to a problem at Amazon. Did anyone who noticed the offending site contact Amazon to have it removed, or are they simply content to keep adding more blocked IPs as the offender jumps from server to server on EC2?

ntoper 17y ago

I am the founder of http://critsend.com http://critsend.com, we make your email ends up into its appropriate inbox. We have a special offer for EC2 customers, we have some servers directly at AWS so latency is low and bandwidth is free for them. We are currently in closed beta but if you subscribe and add say you saw us on Hacker News, we will send you an invite asap. Our first 1,000 emails are free and after that we are pay per use.