8 ms·

There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights

by gecko 14y ago

There's a little bit more information in the last status post, plus a picture of Fog Creek cofounder Michael Pryor hauling 5-gallon jugs of diesel up 17 flights of stairs. (We have other employees down there, too, and I know Stack Exchange's NYC employees were excited to help out, so they're probably there by now, too.) http://status.fogcreek.com/2012/10/diesel-bucket-brigade-maintains-services.html http://status.fogcreek.com/2012/10/diesel-bucket-brigade-mai...

ivany 14y ago
Are there any safety issues associated with hauling a bucket of diesel up a staircase? I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers?
- gecko 14y ago
  Are there any safety issues associated with hauling a bucket of diesel up a staircase? It's flammable and slippery. Wear boots, and I suggest this is a good day to quit smoking. I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers? They are open buckets we store the cleaning supplies for the fish tank in. And also now gasoline.
  - meaty 14y ago
    A million times this. Diesel is extremely dangerous stuff. I nearly blew myself up a few years ago by not cleaning out a diesel tank with detergent properly that I was welding. Diesel residue vapourised and went boom. Fortunately there wasn't enough in it to create a large explosion but it required new underpants. To add to that, my only car accident was due to having diesel on my shoes. My foot slipped off the brake and I rear ended some poor guy.
    - js2 14y ago
      After Hurricane Andrew, my father and I needed to siphon gasoline from one of our cars to use in our generator. Lacking a proper container, we siphoned the gas into a 5 gallon plastic water cooler bottle. The gasoline promptly ate through the bottle's seam and spilled out, dissolving a good size chunk of our asphalt driveway. Lesson learned: use only approved storage containers for gasoline.
  - gadders 14y ago
    Good on you for getting up and working to keeps things running.
  - mseebach2 14y ago
    > They are open buckets So one guy slips and there's five gallons of diesel covering what I assume is the escape route of a NYC high rise?
    - alexfoo 14y ago
      Indeed, can't think of it as anything other than irresponsible, bordering on downright dangerous. Hacking software is one thing, but when that goes wrong it just leaves crashed servers, unresponsive web apps and 404/500 errors. And, before someone mentinos it, hauling diesel up stairs to keep a hospital generator going (as mentioned in another story) is a whole different situation. Patients are a bit more important than a bunch of servers. But you'd still evacuate all non-essential staff and patients if you were doing that given that it presents a significant danger if something were to go wrong; especially with the emergency services already dealing with a huge workload.
    - Fjolle 14y ago
      It's diesel. You'd have to heat it before you would be able to ignite it. But yes :)
      - andyjohnson0 14y ago
        Ignition isn't the only risk. Diesel causes severe skin irritation. I'm not sure about the effects of inhaling fumes from split uncombusted diesel, but it can't be good for a person's lungs. Potential hazmat incident.
        TylerE 14y ago
        Diesel doesn't vaporize to any significant degree at temperatures experienced on Earth.
      - AdamN 14y ago
        Exactly. You can't light diesel on fire with just a match. You need more heat or pressure. However, the burn from gasoline or diesel on your body is intense. I was once filling an old junker with gas when it spurted out from that tank and all over me. At first it was funny and felt nice and cold on the body. Then the burning sensation started and it took a solid 25 minutes in a cold shower to calm the sting.
  - eliajf 14y ago
    I would highly recommend no longer using those buckets for fish tank cleaning. At least not for putting water back in the tank. :-)
Killah911 14y ago
Wow, that's going beyond the extra mile. I'm pretty far from NYC but if I were closer I'd give you guys a hand. In lieu of that, any possibility of a "Trello Hurricane fund" which we can donate to? Would give appreciative users like me a chance to help pitch in for a team and product which has already helped so much. Maybe it'll amount to just enough for lunch or something, but I for one would feel much better knowing that I contributed a little...
- d3ad1ysp0rk 14y ago
  I think there are many, many better places to donate than a profitable company who had to do a little hard labor because they didn't have a second datacenter like everyone suggests.
  - Killah911 14y ago
    The point is, as a developer myself I feel their pain, sometimes shit just hits the fan. As a user, not only is their service amazing, it is integral in running my little startup. I'm not saying monetary compensation is the only way to say thank you, but I think it would be nice if others like me got to say thank you in a slightly more substantial way, for the guys going the beyond the extra mile. There are services that I pay for which were down and they didn't even bother. More like buying a friend who helps you move some lunch. I'm all for donating to red cross and other charitable organizations, but that doesn't mean you can't show anyone else appreciation just because there are others with needs in the world. Give to both, hopefully more to the charitable organizations and a little less for token gifts.
grey-area 14y ago
I'm sure you've thought of this, but now that you're up and running, isn't it possible to buy/hire a pump, run it off the electricity from the already running generators, and pump it up from street level instead, instead of walking up stairs with buckets?
- Dobbs 14y ago
  The pump is underwater.
  - culturestate 14y ago
    OP is suggesting using a new pump to pump from the trucks, in lieu of buckets - not the pumps that are submerged.
- maxerickson 14y ago
  They probably (only) need a couple of gallons per hour. If they can't tie into their existing fuel infrastructure, hauling it up the ~150 to 200 feet of stairs bit by bit is probably easier than trying to source and install something temporary.
- gecko 14y ago
  They've thought of that. Kind of fun to find new sets of industrial pumps post-hurricane though.
- jellicle 14y ago
  It's very difficult to pump liquids up long distances. You can't suck liquids up more than 25 feet high, and while you can push it higher, you need a big pump and strong hoses/pipes, etc. Practically what you end up with is a series of smaller pumps. The water supply to tall buildings is a difficult bit of engineering on its own. Anyway, the short answer is no, it's not at all easy to pump anything up 17 stories. You're talking about 8 or more pumps and appropriate hoses curling up the staircases, making them impassable... but people need those stairs.
dbecker 14y ago
Perhaps the Fog Creek staff is just really fit, but it seems like it would have made sense to hire some burly mover dudes to haul gasoline up stairs.
- gadders 14y ago
  Play some Zumba music and charge for it.
RobAley 14y ago
When FogBugz on Demand launched, Joel touted 2 data centers[1] , in NY and LA, "for the once-in-a-lifetime case of an entire data center blowing up". What happened to the second data center? [1] http://webcache.googleusercontent.com/search?q=cache:lHEK939AKiEJ:www.joelonsoftware.com/items/2007/07/09.html+&cd=5&hl=en&ct=clnk&gl=us&client=safari http://webcache.googleusercontent.com/search?q=cache:lHEK939...
- andyjohnson0 14y ago
  No need to post a link to the Google cache. The original source [1] contains the same text. I don't think that there is much risk that they might ninja-edit the five year-old announcement to change the claim about redundancy. My guess is that the second data centre was there, but failover didn't work for some reason. It would be good to know the reason, though. [1] http://www.joelonsoftware.com/items/2007/07/09.html http://www.joelonsoftware.com/items/2007/07/09.html Edit: Politeness
  - RobAley 14y ago
    Apologies, the link was copied from an unrelated post elsewhere and I didn't really look at the URL, no slight of hand was expected by them. I think your guess is probably wrong though, in a status post 2 days ago they listed[1] the precautions they were taking so that they could be "confident that all of our services will remain available to our customers throughout the weather", and they did not refer to another data center, only "Our data center". That would be the ideal place to mention any fall-back centre. They are also resolutely not replying to public posts on various boards questioning these issues. I am cutting them some slack in that they have a lot to attend to right now, although the fact they are managing to find time to reply to some threads (like the one in which this is posted) is a tad sniffy. Once they're back up and running I'm sure they'll update us all on why their redundancy failed or magically wasn't there. [1] http://status.fogcreek.com/2012/10/feelin-fine-no-expected-downtime-due-to-hurricane-sandy.html http://status.fogcreek.com/2012/10/feelin-fine-no-expected-d...
    - andyjohnson0 14y ago

by gecko 14y ago

ivany 14y ago

Are there any safety issues associated with hauling a bucket of diesel up a staircase? I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers?

gecko 14y ago
Are there any safety issues associated with hauling a bucket of diesel up a staircase? It's flammable and slippery. Wear boots, and I suggest this is a good day to quit smoking. I know it's not as volatile as gasoline but it's still not exactly the most fire-safe substance. Are they literally 5-gallon pails, or are they sealed containers? They are open buckets we store the cleaning supplies for the fish tank in. And also now gasoline.
- meaty 14y ago
  A million times this. Diesel is extremely dangerous stuff. I nearly blew myself up a few years ago by not cleaning out a diesel tank with detergent properly that I was welding. Diesel residue vapourised and went boom. Fortunately there wasn't enough in it to create a large explosion but it required new underpants. To add to that, my only car accident was due to having diesel on my shoes. My foot slipped off the brake and I rear ended some poor guy.
  - js2 14y ago
    After Hurricane Andrew, my father and I needed to siphon gasoline from one of our cars to use in our generator. Lacking a proper container, we siphoned the gas into a 5 gallon plastic water cooler bottle. The gasoline promptly ate through the bottle's seam and spilled out, dissolving a good size chunk of our asphalt driveway. Lesson learned: use only approved storage containers for gasoline.
- gadders 14y ago
  Good on you for getting up and working to keeps things running.
- mseebach2 14y ago
  > They are open buckets So one guy slips and there's five gallons of diesel covering what I assume is the escape route of a NYC high rise?
  - alexfoo 14y ago
    Indeed, can't think of it as anything other than irresponsible, bordering on downright dangerous. Hacking software is one thing, but when that goes wrong it just leaves crashed servers, unresponsive web apps and 404/500 errors. And, before someone mentinos it, hauling diesel up stairs to keep a hospital generator going (as mentioned in another story) is a whole different situation. Patients are a bit more important than a bunch of servers. But you'd still evacuate all non-essential staff and patients if you were doing that given that it presents a significant danger if something were to go wrong; especially with the emergency services already dealing with a huge workload.
  - Fjolle 14y ago
    It's diesel. You'd have to heat it before you would be able to ignite it. But yes :)
    - andyjohnson0 14y ago
      Ignition isn't the only risk. Diesel causes severe skin irritation. I'm not sure about the effects of inhaling fumes from split uncombusted diesel, but it can't be good for a person's lungs. Potential hazmat incident.
      - TylerE 14y ago
        Diesel doesn't vaporize to any significant degree at temperatures experienced on Earth.
    - AdamN 14y ago
      Exactly. You can't light diesel on fire with just a match. You need more heat or pressure. However, the burn from gasoline or diesel on your body is intense. I was once filling an old junker with gas when it spurted out from that tank and all over me. At first it was funny and felt nice and cold on the body. Then the burning sensation started and it took a solid 25 minutes in a cold shower to calm the sting.
- eliajf 14y ago
  I would highly recommend no longer using those buckets for fish tank cleaning. At least not for putting water back in the tank. :-)

Killah911 14y ago

Wow, that's going beyond the extra mile. I'm pretty far from NYC but if I were closer I'd give you guys a hand. In lieu of that, any possibility of a "Trello Hurricane fund" which we can donate to? Would give appreciative users like me a chance to help pitch in for a team and product which has already helped so much. Maybe it'll amount to just enough for lunch or something, but I for one would feel much better knowing that I contributed a little...

d3ad1ysp0rk 14y ago
I think there are many, many better places to donate than a profitable company who had to do a little hard labor because they didn't have a second datacenter like everyone suggests.
- Killah911 14y ago
  The point is, as a developer myself I feel their pain, sometimes shit just hits the fan. As a user, not only is their service amazing, it is integral in running my little startup. I'm not saying monetary compensation is the only way to say thank you, but I think it would be nice if others like me got to say thank you in a slightly more substantial way, for the guys going the beyond the extra mile. There are services that I pay for which were down and they didn't even bother. More like buying a friend who helps you move some lunch. I'm all for donating to red cross and other charitable organizations, but that doesn't mean you can't show anyone else appreciation just because there are others with needs in the world. Give to both, hopefully more to the charitable organizations and a little less for token gifts.

grey-area 14y ago

I'm sure you've thought of this, but now that you're up and running, isn't it possible to buy/hire a pump, run it off the electricity from the already running generators, and pump it up from street level instead, instead of walking up stairs with buckets?

Dobbs 14y ago
The pump is underwater.
- culturestate 14y ago
  OP is suggesting using a new pump to pump from the trucks, in lieu of buckets - not the pumps that are submerged.
maxerickson 14y ago
They probably (only) need a couple of gallons per hour. If they can't tie into their existing fuel infrastructure, hauling it up the ~150 to 200 feet of stairs bit by bit is probably easier than trying to source and install something temporary.
gecko 14y ago
They've thought of that. Kind of fun to find new sets of industrial pumps post-hurricane though.
jellicle 14y ago
It's very difficult to pump liquids up long distances. You can't suck liquids up more than 25 feet high, and while you can push it higher, you need a big pump and strong hoses/pipes, etc. Practically what you end up with is a series of smaller pumps. The water supply to tall buildings is a difficult bit of engineering on its own. Anyway, the short answer is no, it's not at all easy to pump anything up 17 stories. You're talking about 8 or more pumps and appropriate hoses curling up the staircases, making them impassable... but people need those stairs.

dbecker 14y ago

Perhaps the Fog Creek staff is just really fit, but it seems like it would have made sense to hire some burly mover dudes to haul gasoline up stairs.

gadders 14y ago
Play some Zumba music and charge for it.

RobAley 14y ago

When FogBugz on Demand launched, Joel touted 2 data centers[1] , in NY and LA, "for the once-in-a-lifetime case of an entire data center blowing up". What happened to the second data center? [1] http://webcache.googleusercontent.com/search?q=cache:lHEK939AKiEJ:www.joelonsoftware.com/items/2007/07/09.html+&cd=5&hl=en&ct=clnk&gl=us&client=safari http://webcache.googleusercontent.com/search?q=cache:lHEK939...

andyjohnson0 14y ago
No need to post a link to the Google cache. The original source [1] contains the same text. I don't think that there is much risk that they might ninja-edit the five year-old announcement to change the claim about redundancy. My guess is that the second data centre was there, but failover didn't work for some reason. It would be good to know the reason, though. [1] http://www.joelonsoftware.com/items/2007/07/09.html http://www.joelonsoftware.com/items/2007/07/09.html Edit: Politeness
- RobAley 14y ago
  Apologies, the link was copied from an unrelated post elsewhere and I didn't really look at the URL, no slight of hand was expected by them. I think your guess is probably wrong though, in a status post 2 days ago they listed[1] the precautions they were taking so that they could be "confident that all of our services will remain available to our customers throughout the weather", and they did not refer to another data center, only "Our data center". That would be the ideal place to mention any fall-back centre. They are also resolutely not replying to public posts on various boards questioning these issues. I am cutting them some slack in that they have a lot to attend to right now, although the fact they are managing to find time to reply to some threads (like the one in which this is posted) is a tad sniffy. Once they're back up and running I'm sure they'll update us all on why their redundancy failed or magically wasn't there. [1] http://status.fogcreek.com/2012/10/feelin-fine-no-expected-downtime-due-to-hurricane-sandy.html http://status.fogcreek.com/2012/10/feelin-fine-no-expected-d...
  - andyjohnson0 14y ago