7 ms·

A large (10m uniques/mo) web site I used to work for had a complete DMCA takedown process in place. Links on every page, web forms, contact emails, physical add

by aphyr 15y ago

A large (10m uniques/mo) web site I used to work for had a complete DMCA takedown process in place. Links on every page, web forms, contact emails, physical addresses, etc. Then our site went dark one evening. I spent an hour frantically trying to figure out where our main web servers had gone, only to discover that a "online anti-terrorism team" had taken issue with some user-submitted content that didn't seem very friendly towards Americans. They contacted Softlayer, our hosting provider, and said that anti-American content was in violation of some commerce law--don't recall the exact details. It wasn't what I would describe as a "credible legal threat"--they left no name, no physical address, their web site looked like a vigilante operation. Softlayer, in turn, sent us a "generic you have a support ticket" email and b.) 72 hours later, unplugged the web server NICs. We suffered hours of downtime without any idea what was happening. We juggled dozens of Softlayer tickets at the time, so another anonymous tracking number just got lost in the shuffe. Never underestimate the power of unaccredited strangers to fuck you through your hosting provider.

aptwebapps 15y ago
Please tell me they switched providers after that.
- aphyr 15y ago
  We established multiple datacenters across different providers with active-active DNS, so neither of them could take us down. It's not really for the faint of heart, though. We spent a fair amount of time trying to get the right services to communicate correctly across DCs.
  - georgieporgie 15y ago
    I have no idea what you're talking about, but I find it very interesting. I think it would be well received if you wrote a blog post expanding on this experience.
  - ComputerGuru 15y ago
    Would you care to/be kind enough to share more info on how this is done and any obstacles you faced in deploying this?
    - aphyr 15y ago
      Sure. This is a really quick off the cuff summary, so I'm gonna say some things that are loose, out of date, or maybe flat out wrong. Comments welcome. :) The ideal situation is that both datacenters can handle your total load, but when one fails, the other doesn't explode under the thundering herd of traffic rerouted its way. So you need to plan your systems in such a way that they're elastic under load; response times rise within these limits, but you won't see outright failures. You use a DNS failover service to provide each client with the appropriate DNS. There are various issues around caching and preferred A records--for instance, some name servers or DNS clients will pick the first A record, sorted, which can send all your traffic to one datacenter. Typically you hand out different combinations of A records depending on locality, so clients are hitting, say, the two closest datacenters to them. When a DC fails, you remove the DNS entries which pointed to that datacenter's IPs, and lookups start returning only the known good ones. Clients which already have your multiple A records can detect the failure and fail over immediately. Where client software doesn't support that, they have to wait until DNS caching expires to get the new records. The datacenters themselves need to contain enough of your infrastructure to function autonomously, but also should share state. Cassandra, Oracle, Riak MDC... there are lots of options out there. We were on MySQL at the time, and maintained a slave in the secondary DC which could be promoted in the event that the primary DC was, say, nuked from orbit. This system was not partition-tolerant; if the mysql link between datacenters failed, one DC would become functionally read-only. We proxied DB traffic back and forth over SSH tunnels managed by upstart init jobs. This was shockingly reliable. We actually started off using mysql's SSL support but as it turns out mysql will segfault if it gets more than, say, 8 ssl connections in a short timeframe. So we tunneled everything--redis, mysql, stats, over SSH. The rest of the infrastructure had little shared state, so we ran the typical web stack: two identical boxes running nginx (static content) -> haproxy (load balancing) -> rails and ramaze apps spread across various boxes. Each nginx forwarded to both haproxies, both haproxies forwarded to all the app servers, so you could lose either machine in a given DC and service would keep running. We used heartbeat to manage a shared virtual IP interface between the two forwarding boxes, so you'd drop TCP conns but failover switch time was generally in the tens of milliseconds--however long it took to ifup and gratuitous-arp the rack's L3 switch. We ran memcache independently in both DCs--since user sessions almost never switched between DCs it was OK for us to just have two distinct pools. Queues were split up as well. Some services weren't critical enough to split across DCs so we just accepted that if the primary DC died they'd be down for a few hours, until we could deploy another copy on the backup DC. Non-critical things like statistics, garbage collection, etc. Automated deployment made that a lot less painful. I wouldn't recommend doing this at an early stage--dual environments, especially on different hardware, takes a lot of testing to get right. You have to worry about doing everything twice--two DNS zones, two Redis clusters, etc. You also have to worry about asymmetries if you're doing master-slave replication. All of this comes with operational and development overhead; your app needs to be aware that might might running in a partitioned state, that writes might take much longer than reads if you're doing master->slave across DCs, etc. I'm a strong believer in planning for that stage of your growth--but you always have to strike a balance between the ultimate reliable configuration and getting other things done. Does that help answer your question? :)
seanp2k2 15y ago
Hosting provider sysadmin here. We're required to do exactly what soft layer did here, and 72 hours is pretty generous. We typically call the client if they don't respond to the DMCA takedown notice within 24 hours an give them another 24 to take down whatever content they're being DMCA'd for. It's shit and I hate it, but if they don't respond, we do the very minimum damage possible to make the content unavailable, as required of us by DMCA safe harbor provisions. If it is a managed service, we'll just chmod the image to 000 or whatever does the least damage to their site. Unfortunately, if we don't have the login to the server (unmanaged) or if it's a colo, we just have to disable that IP on the switch or router (or null-route their IP for a bit) until they contact us and can take their "illegal" content down. My point: hate DMCA, not SoftLayer, for this. They (assuming unmanaged service) just did what they were legally required to do.
- aphyr 15y ago
  [edit] It's good that you call the clients. Nothing drives me nuts more than trying to sift through a painful ticket system where everything is tracked only by reference number. What pisses me off was that this wasn't even a DMCA request--SL had no legal responsibility to take action. It was just some random internet vigilantes making an unsubstantiated threat. Come to think of it, SL may have taken us down for DMCA as well. We had an obvious path for handling abuse that both the accusers and SL could have used. After some negotiation I think we were able to convince them to just forward abuse emails to our address--but it took some doing.
  - seanp2k2 15y ago
    Ahh, Gotcha. When that ("frivolous legal threats")happen to us, we just don't even reply to the person reporting it in most cases. If they're persistent, we say in so many words "get a court order, then we'll talk". We've had India law firms call us screaming at 6PM on a friday, and we told them: - we require that they submit all abuse matters to our abuse@ e-mail address per RFC 2142 (kinda, but this sounds official when you say it to some law intern chump) - they can scream all they want; we're not taking it down unless they submit a "valid legal order" to us (I don't even mention DMCA because I don't want to give them ideas) - we're not responsible for the content of our clients, so they need to take it up with them >"We had an obvious path for handling abuse that both the accusers and SL could have used." They (accusers) don't, and they do this on purpose. They don't really want the content just gone, they want collateral damage as revenge for your "violation". >"After some negotiation I think we were able to convince them to just forward abuse emails to our address--but it took some doing." Sorry that that even required negotiation. We forward all abuse e-mail besides spam complaints for managed services. With anything like this, we try our best to do as little damage as possible to our clients.

by aphyr 15y ago

aptwebapps 15y ago

Please tell me they switched providers after that.

aphyr 15y ago
We established multiple datacenters across different providers with active-active DNS, so neither of them could take us down. It's not really for the faint of heart, though. We spent a fair amount of time trying to get the right services to communicate correctly across DCs.
- georgieporgie 15y ago
  I have no idea what you're talking about, but I find it very interesting. I think it would be well received if you wrote a blog post expanding on this experience.
- ComputerGuru 15y ago
  Would you care to/be kind enough to share more info on how this is done and any obstacles you faced in deploying this?
  - aphyr 15y ago
    Sure. This is a really quick off the cuff summary, so I'm gonna say some things that are loose, out of date, or maybe flat out wrong. Comments welcome. :) The ideal situation is that both datacenters can handle your total load, but when one fails, the other doesn't explode under the thundering herd of traffic rerouted its way. So you need to plan your systems in such a way that they're elastic under load; response times rise within these limits, but you won't see outright failures. You use a DNS failover service to provide each client with the appropriate DNS. There are various issues around caching and preferred A records--for instance, some name servers or DNS clients will pick the first A record, sorted, which can send all your traffic to one datacenter. Typically you hand out different combinations of A records depending on locality, so clients are hitting, say, the two closest datacenters to them. When a DC fails, you remove the DNS entries which pointed to that datacenter's IPs, and lookups start returning only the known good ones. Clients which already have your multiple A records can detect the failure and fail over immediately. Where client software doesn't support that, they have to wait until DNS caching expires to get the new records. The datacenters themselves need to contain enough of your infrastructure to function autonomously, but also should share state. Cassandra, Oracle, Riak MDC... there are lots of options out there. We were on MySQL at the time, and maintained a slave in the secondary DC which could be promoted in the event that the primary DC was, say, nuked from orbit. This system was not partition-tolerant; if the mysql link between datacenters failed, one DC would become functionally read-only. We proxied DB traffic back and forth over SSH tunnels managed by upstart init jobs. This was shockingly reliable. We actually started off using mysql's SSL support but as it turns out mysql will segfault if it gets more than, say, 8 ssl connections in a short timeframe. So we tunneled everything--redis, mysql, stats, over SSH. The rest of the infrastructure had little shared state, so we ran the typical web stack: two identical boxes running nginx (static content) -> haproxy (load balancing) -> rails and ramaze apps spread across various boxes. Each nginx forwarded to both haproxies, both haproxies forwarded to all the app servers, so you could lose either machine in a given DC and service would keep running. We used heartbeat to manage a shared virtual IP interface between the two forwarding boxes, so you'd drop TCP conns but failover switch time was generally in the tens of milliseconds--however long it took to ifup and gratuitous-arp the rack's L3 switch. We ran memcache independently in both DCs--since user sessions almost never switched between DCs it was OK for us to just have two distinct pools. Queues were split up as well. Some services weren't critical enough to split across DCs so we just accepted that if the primary DC died they'd be down for a few hours, until we could deploy another copy on the backup DC. Non-critical things like statistics, garbage collection, etc. Automated deployment made that a lot less painful. I wouldn't recommend doing this at an early stage--dual environments, especially on different hardware, takes a lot of testing to get right. You have to worry about doing everything twice--two DNS zones, two Redis clusters, etc. You also have to worry about asymmetries if you're doing master-slave replication. All of this comes with operational and development overhead; your app needs to be aware that might might running in a partitioned state, that writes might take much longer than reads if you're doing master->slave across DCs, etc. I'm a strong believer in planning for that stage of your growth--but you always have to strike a balance between the ultimate reliable configuration and getting other things done. Does that help answer your question? :)

seanp2k2 15y ago

Hosting provider sysadmin here. We're required to do exactly what soft layer did here, and 72 hours is pretty generous. We typically call the client if they don't respond to the DMCA takedown notice within 24 hours an give them another 24 to take down whatever content they're being DMCA'd for. It's shit and I hate it, but if they don't respond, we do the very minimum damage possible to make the content unavailable, as required of us by DMCA safe harbor provisions. If it is a managed service, we'll just chmod the image to 000 or whatever does the least damage to their site. Unfortunately, if we don't have the login to the server (unmanaged) or if it's a colo, we just have to disable that IP on the switch or router (or null-route their IP for a bit) until they contact us and can take their "illegal" content down. My point: hate DMCA, not SoftLayer, for this. They (assuming unmanaged service) just did what they were legally required to do.

aphyr 15y ago
[edit] It's good that you call the clients. Nothing drives me nuts more than trying to sift through a painful ticket system where everything is tracked only by reference number. What pisses me off was that this wasn't even a DMCA request--SL had no legal responsibility to take action. It was just some random internet vigilantes making an unsubstantiated threat. Come to think of it, SL may have taken us down for DMCA as well. We had an obvious path for handling abuse that both the accusers and SL could have used. After some negotiation I think we were able to convince them to just forward abuse emails to our address--but it took some doing.
- seanp2k2 15y ago
  Ahh, Gotcha. When that ("frivolous legal threats")happen to us, we just don't even reply to the person reporting it in most cases. If they're persistent, we say in so many words "get a court order, then we'll talk". We've had India law firms call us screaming at 6PM on a friday, and we told them: - we require that they submit all abuse matters to our abuse@ e-mail address per RFC 2142 (kinda, but this sounds official when you say it to some law intern chump) - they can scream all they want; we're not taking it down unless they submit a "valid legal order" to us (I don't even mention DMCA because I don't want to give them ideas) - we're not responsible for the content of our clients, so they need to take it up with them >"We had an obvious path for handling abuse that both the accusers and SL could have used." They (accusers) don't, and they do this on purpose. They don't really want the content just gone, they want collateral damage as revenge for your "violation". >"After some negotiation I think we were able to convince them to just forward abuse emails to our address--but it took some doing." Sorry that that even required negotiation. We forward all abuse e-mail besides spam complaints for managed services. With anything like this, we try our best to do as little damage as possible to our clients.