5 ms·

Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. a

by gecko 14y ago

Kiln alone has >4 TB of data; you want to SCP that with 90 minutes heads-up? Having power and/or rack space is not the same as having servers, switches, etc. anyway. Hopefully it will not be down that long. We'll let you know more when we know more.

hellmans 14y ago
That 90 minutes claim is a bit disingenuous. We're at the same data center. Internap told us that the fuel pumps were flooded and asked us to shut down at ~9:30pm last night. The generators lasted until 10:52am. So 12 hours+ of warning. To be fair, at 9:30 they did warn that the generators would only last for 4-5 hours, but customers like us who proactively shut everything down extended that significantly.
ynniv 14y ago
You could have rsync'd it with a few days heads up, and freshened that in the last 90min.
modoc 14y ago
Do you guys do off-site backups?
- shadytrees 14y ago
  Yes we do. Unfortunately, that's all they are -- data backups, sans infrastructure.
  - modoc 14y ago
    Sure, but that invalidates the complaint of it being infeasible to scp out 4 TB of data before your NYC DC runs out of power. Those 4 TB of data are already out of NYC, safely in some other DC where you have hardware. You just need new servers/VMs in/near that DC to restore the backups to. I'm not trying to 2nd guess your ops team, but the whole point in having off-site backups is to facilitate a your RTO plan in case you lose your primary DC with no warning. I guess I'd be surprised if you don't have a < 24 hour RTO plan in place. With how quickly you can get VMs and even dedicated server provisioned by many hosting providers (minutes to a couple hours), the idea of physically moving servers off-site into a new racks, with new networking, etc... seems kinda nutty...
- kevingessner 14y ago
  Yes, we have multiple off-site backups (cloud as well as an offsite storage DC) for all customer data. All data is still safe in NYC -- we've just brought down service to prevent problems in case of an abrupt power failure.
bunderbunder 14y ago
Ha, I wouldn't worry too much about Kiln. I can push when the servers come back. Heck, considering how often some folks I know of push their changesets I wouldn't be surprised if a lot of them never even notice that anything happened. FogBugz is more of a problem. Some days the "Resolve" button is my only source of job satisfaction.
alkaramba 14y ago
>4 TB of data? I didn't realize Kiln's gotten that big already. When this whole thing is sorted out, @gecko @kevingessner - would love to see a "State of the Kiln" post and some stats!

by gecko 14y ago

hellmans 14y ago

That 90 minutes claim is a bit disingenuous. We're at the same data center. Internap told us that the fuel pumps were flooded and asked us to shut down at ~9:30pm last night. The generators lasted until 10:52am. So 12 hours+ of warning. To be fair, at 9:30 they did warn that the generators would only last for 4-5 hours, but customers like us who proactively shut everything down extended that significantly.

ynniv 14y ago

You could have rsync'd it with a few days heads up, and freshened that in the last 90min.

modoc 14y ago

Do you guys do off-site backups?

shadytrees 14y ago
Yes we do. Unfortunately, that's all they are -- data backups, sans infrastructure.
- modoc 14y ago
  Sure, but that invalidates the complaint of it being infeasible to scp out 4 TB of data before your NYC DC runs out of power. Those 4 TB of data are already out of NYC, safely in some other DC where you have hardware. You just need new servers/VMs in/near that DC to restore the backups to. I'm not trying to 2nd guess your ops team, but the whole point in having off-site backups is to facilitate a your RTO plan in case you lose your primary DC with no warning. I guess I'd be surprised if you don't have a < 24 hour RTO plan in place. With how quickly you can get VMs and even dedicated server provisioned by many hosting providers (minutes to a couple hours), the idea of physically moving servers off-site into a new racks, with new networking, etc... seems kinda nutty...
kevingessner 14y ago
Yes, we have multiple off-site backups (cloud as well as an offsite storage DC) for all customer data. All data is still safe in NYC -- we've just brought down service to prevent problems in case of an abrupt power failure.

bunderbunder 14y ago

Ha, I wouldn't worry too much about Kiln. I can push when the servers come back. Heck, considering how often some folks I know of push their changesets I wouldn't be surprised if a lot of them never even notice that anything happened. FogBugz is more of a problem. Some days the "Resolve" button is my only source of job satisfaction.

alkaramba 14y ago

>4 TB of data? I didn't realize Kiln's gotten that big already. When this whole thing is sorted out, @gecko @kevingessner - would love to see a "State of the Kiln" post and some stats!