10 ms·

We’re Fucked, It’s Over: Coming Back from the Brink

351 points by rajbala 13y ago

Susan_we 13y ago
I really admire Scott from what he's written. He can be an inch close and still get out of WFIO situation. And he seems to know the difference when it's doomed like with e-commerce business. So keep rowing, Scott!
espressopowered 13y ago
Nice article. I particularly agree with the point on leadership. If you can motivate those around you, people perform so much better and can sometimes come up with better solutions. Doesn't always guarantee you coming back from the brink, but stands you in good stead.
pgrote 13y ago
"It's never really as bad as it seems." Very true. Many times when something happens the automatic response is to focus on the worst case scenario instead of the immediate problem at hand. Also, I couldn't select any text to copy and paste.
- mathattack 13y ago
  "It's never really as bad as it seems." A very good point for our personal lives too.
- not_paul_graham 13y ago
  This is great advice even for students because sometimes the things that younger people (anecdotal: me) give a lot of importance to and feel terrible for screwing up end up not mattering at all. 1. Things like validation from peers / popular kids 2. Screwing up on that major test that sometimes "defines" where you end up. A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly). 3. The academic thing is worse if you are artistically inclined and have science and math thrown down your throat telling you that humanities are for those that can't cut it in science (I know friends that have had a variation of this conversation with their kin/parents, etc) quite true in countries like India / Pakistan, Korea and perhaps even China. 4. Humiliation/shame from something that seems drastic in the moment (Getting rejected by the cute girl in front of all your friends and classmates. Yea that sucks). Getting labeled sucks too, a lot of girls can be mean to their own gender at that age which is absolutely disgusting. Unfortunately for most people it is something that they only figure out in hindsight.
  - NAFV_P 13y ago
    $ A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly). Heavy debts are a significant contributor to suicide rates, fear and shame play a large part. The potential collapse of companies would heighten this risk, so here's some statistics on the Great Depression (bottom of the page): http://www.shmoop.com/great-depression/statistics.html http://www.shmoop.com/great-depression/statistics.html
  - aestra 13y ago
    >A lot of suicides result from academic pressure and failure I was wondering if this was actually true or not, turns out that there is some evidence suggesting it. http://apt.rcpsych.org/content/8/6/418.short http://apt.rcpsych.org/content/8/6/418.short http://www.tandfonline.com/doi/abs/10.1080/13811110590904016#.UzGOaVcZGC8 http://www.tandfonline.com/doi/abs/10.1080/13811110590904016...
  - rk17 13y ago
    Great comment. Off-topic: funny name.
  - Fomite 13y ago
    I had an Economics professor who gave a small lecture to our class after we got our midterm grades back. "This midterm is one third of your grade. This class is one of four you are taking this semester. This semester is one of eight you will take at $University. And while people will tell you otherwise, your time in college is a small part of your life. It will be okay."
- mildtrepidation 13y ago
  This is not true. It's almost always true in most contexts, but to say that it's a fact is to encourage a false sense of security... sometimes things are that bad, and it's important to be able to realize and accept that, otherwise you never even give yourself the option to try and improve them.
- vvvv 13y ago
  It's never so bad, that it can't get worse ;]
- ams6110 13y ago
  Just the point I came to comment on. There have been many times in my life where things seemed to be completely falling apart. But I have learned that I almost always paint a much bleaker picture than what actually ends up happening. Life goes on.
- jakobe 13y ago
  > Also, I couldn't select any text to copy and paste. Seems to be a Safari bug. I had the same thing, after loading the website I couldn't even select text on HN anymore. Restarting Safari fixed it.
mathattack 13y ago
Great points, especially leading from the front when times are tough. I didn't think Hotmail would have survived the outages, but they came out just fine.
cousin_it 13y ago
Can't select any text to copy/paste, can't click any of the links in the post or sidebar. #welcometothenewweb
- deleted 13y ago
  [deleted]
- mmastrac 13y ago
  I'm guessing it's some sort of layout thing gone horribly wrong. Not really the new web to blame here -- just poor testing.
- userbinator 13y ago
  Site works fine without JS, and I can do all of those things too. Although all those extra parameters in the URL are a little odd.
  - aestra 13y ago
    For what its worth, the page works fine without all the parameters. Why they are in the submitted link, I don't know.
    - Two9A 13y ago
      The utm_* parameters tend to be used for source tracking; the important ones here being "content" and "campaign". Of course, this particular link doesn't refer to HN in its campaign code, so their statistics are going to get a little confused.
- cyphax 13y ago
  I had this problem also but I was logged in to wordpress.com (or .org, I don't even know, but it had the black bar at the top from WordPress) and when I signed out, that problem went away immediately.
- gabemart 13y ago
  A transparent portion of the #comments div is covering the body text and sidebar. If you change #comments position from relative to initial, things work as expected. I think the problem is that the sidebar is floated and the float isn't cleared. Setting a clear: both on #comments prevents the div from overlapping the body and sidebar.
JoeAltmaier 13y ago
Some survivor bias there, but an inspiring message: always continue as if you're going to survive. All the real survivors did that.
deleted 13y ago
[deleted]
JonLim 13y ago
> Get all the brains around the table An important addition to this: get everyone involved, but also listen to their ideas. Having been a part of a sinking ship, the most frustrating thing about having everyone pour their blood, sweat, and tears into reviving the company is when the top brass decides to ignore all of the hard work and carry on with their own ideas that were never cleared with anyone else. They were their own iceberg.
randall 13y ago
This is essentially the definition of survivorship bias. http://www.fastcodesign.com/1671172/how-a-story-from-world-war-ii-shapes-facebook-today http://www.fastcodesign.com/1671172/how-a-story-from-world-w... "In WWII, Allied bombers were key to strategic attacks, yet these lumbering giants were constantly shot down over enemy territory. The planes needed more armor, but armor is heavy. So extra plating could only go where the planes were being shot the most."... "[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes." Having experienced 2 "WFIO" things in the last 6 months, I agree that entrepreneurs need to be resilient, but I think it's more valuable to read post mortems than these "we almost died" posts because they prove more instructional. For instance, Aaron's post about Tutorspree is a really useful post for me, although it definitely has less of a feel good vibe to it. http://www.aaronkharris.com/when-seo-fails-single-channel-dependency-and-the-end-of-tutorspree http://www.aaronkharris.com/when-seo-fails-single-channel-de...
- jacobparker 13y ago
  Relevant Hark! A Vagrant http://www.harkavagrant.com/index.php?id=206 http://www.harkavagrant.com/index.php?id=206 The sentiment seems very popular in the start-up scene :)
  - twic 13y ago
    May i say how wonderful it is too see a relevant Hark! rather than a relevant XKCD for a change. Class, tonight's homework is to find a relevant Dinosaur Comics. For extra credit, a relevant Achewood.
    - kstop 13y ago
      http://www.achewood.com/index.php?date=05312002 http://www.achewood.com/index.php?date=05312002 (I don't do regular credit.)
    - taybin 13y ago
      You have excellent taste in comics.
mkrecny 13y ago
Fresh out of college I was responsible for a WFIO at a 40 person company - ultimately had to leave. http://edu.mkrecny.com/thoughts/how-i-fired-myself http://edu.mkrecny.com/thoughts/how-i-fired-myself "Um those [backups] got really expensive, so we stopped doing them about a month ago" sounds painfully familiar.
- thejosh 13y ago
  I remember reading a while ago and cringing at this so bad, this needs to be a cautionary tale of version control, development/staging/production environments and ... backups.
  - mkrecny 13y ago
    "The Startup CTO's Guide to ..." ?
    - thejosh 13y ago
      all it contains is the word: "backups".
      - angersock 13y ago
        Multiple times, for redundancy. It should probably also contain the word "documentation of system configurations".
      - sliverstorm 13y ago
        But why would a rockstar-ninja-pirate ever need backups?
        FireBeyond 13y ago
        Backups are not MVP! First to market!
- gabemart 13y ago
sswaner 13y ago
The same principles were true and applicable in a much larger near-implosion that I experienced from 2008-2012 working for a "too big to fail" financial company. Strong, out-front leadership, a strong team led the way. It took time for us to realize that it wasn't as bad as it seemed. We had to learn that the external pressure from the media, public and even friends and family was only a distraction. In this situation it was quickly apparent that my role was to focus on a solution, not dwell on the problem or the cause. This should be true for everyone on the team, unless you are the specific individual who caused a global financial meltdown, deleted all the email or caused a critical failure.
- tsunamifury 13y ago
  Actually that was a problem due to the culture of stupidity and willful ignorance of risk, not some keyboard mash that caused the markets to crash. It might be one of the rare situations where it was so bad, that after stabilizing the situation, many of the responsible parties needed to be put down. I know from my experience working with Wachovia, the gross incompetence, system wide narrow minded greed and the fuck-everyone-but-me attitude was so bad it was mind boggling. It was not a mistake, this was a concequence of the culture and goals of those companies.
- rhizome 13y ago
  It took time for us to realize that it wasn't as bad as it seemed. I'm not sure this example is relevant, since this realization would not have been possible without a massive leg-up and pat on the head from your national government. That is: "too big to fail" negates "near-implosion."
michaelochurch 13y ago
The leader needs to be the first one there, the last one to leave, and be willing to do anything it takes – like answer customer care calls or personally drive a replacement part to an irate customer. Nothing is beneath a leader in times of crisis. I like this, because it says two things at once, both true. First, if you're in a position of ownership/authority but you're not willing or able to do the grunt work, you won't have the credibility for long. People will follow orders out of fear of getting fired, but you'll never get more than the bare minimum. There's a point where leaders are replaced or outnumbered by true executives (lazy, bikeshedding, rent-seeking parasites) and after that point, the organization can't even motivate shit to stink. Second, it might be that it's just not worth it to lead from the front. That means that you're not really in a position of leadership. You might be a middle manager who realizes that the people above you will never buy in to what you're doing. Then you'll probably lose that desire to make those sacrifices. That's fine. You shouldn't tie yourself to the mast, at that point, because you're not really in a leadership role anyway.
orthishappended 13y ago
The most interesting take away from that was how resilient companies appear to be despite their leaders best efforts to sabotage everything. I think the real lesson is (and I don't like it): if you want your company to survive you need to be good at spin. In the case of hotmail, how did he ever let a situation arise where one bug in a nightly cron script could obliterate his users emails?! No backups because they cost too much?? "the sun, the moon and the stars lined up against us for it to happen" No, you were just reckless and too busy chasing growth. With Ironport, he says his whole business was built on the back of anti-spam yet they were totally unprepared when their partner who provided the anti-spam tech they used pulled the rug from under them (despite knowing full well this day would come). The real lesson here should be: try not to outsource the key component upon which your entire business depends. I had to laugh at the end when he says the only time he's had a failure was when he was using his own limited resources to fund a business and not investor money. Well yes... mistakes are easier to hide/absorb when you're swimming in cash. Not so much when things are tight and you can't afford to learn after the fuck up.
dsugarman 13y ago
how did you build the spam filter? bayesian classifier?
- herdrick 13y ago
  No, not in 1997.
eterpstra 13y ago
"Nothing is beneath a leader in times of crisis" I've worked for people that believe this, and others that don't. Nothing builds morale and loyalty like rolling up your sleeves and doing the hard stuff with your employees. I once had a project manager who couldn't write a lick of code, but whenever a deadline approached or something blew up, he'd be right there at 10pm with everyone else bringing in fresh pizza, testing whatever he could, and giving pep talks. Contrast this with someone that says, "get this done by tomorrow, OK?" and then walks out the door. Who do you think is going to have a higher performing team in the long run?
- jzwinck 13y ago
  In the truly long run, one of those teams will either burn out or die of a collective heart attack from all the late nights eating pizza at its desk. Still, it's nice when someone chips in to do additional testing before release.
- incision 13y ago
  >"Who do you think is going to have a higher performing team in the long run?" The project manager who manages the team, schedule and risks such that things don't routinely blow and run down to the wire would be preferable to either.
  - eterpstra 13y ago
    lol. In a perfect world where bugs never happen, estimates are always perfect, and last-minute top-down requirements changes never occur, I would totally agree with you.
- digikata 13y ago
  I'd rather be working for the project manager that tells everybody to go home at a reasonable hour, get a good nights sleep, and come back fresh in the morning. That's the most productive. It's amazing how often an intractable bug in the evening becomes an, "ah hah, it's x" in the morning.
TheCraiggers 13y ago
Am I the only one that thought the query string was actually more interesting than the article? ?utm_content=bufferc2fec&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer I'm glad that ycombinator isn't listed as the source, at least. I think it would be fairly interesting to see what kind of data they have about various social medias and link propagation. And also, a slight sickening sensation that marketers are doing this. Not that I'm surprised, it's just that I'm sure they have papers written saying that "Twitter has a 64% link click rate if you write your post like X, whereas Facebook can achieve as high as 70% if you do Y." I know it has been going on for years on the web, and decades for other marketing, but here it's plain to see. It fills me with unease, like I'm no longer in control and what I like in life is already planned out.
- showerst 13y ago
  those are standard google analytics campaign tags, they probably (automatically?) built the link that way before they originally shared it on twitter, and then someone cut & pasted that link to share on HN.
- michaelbuckbee 13y ago
  This is a pretty standard Google Analytics Tracking URL [1] - setting aside the Googleness of it (which raises a different set of issues) I look at it like the modern equivalent of an Apache Access Log. Also, not sure if this will make you feel better, but most places that are doing this type of in-depth analytics, etc. are much more often trying to bend themselves to your will ("what feature, benefit or product that people will get value out of") than to trick you into liking something you otherwise wouldn't. One way of interpreting that query string is as the answer to an implicit survey question of "How would you like to get the stories we write?" the answer in this case being: "Twitter" [2]. 1 - https://support.google.com/analytics/answer/1033867?hl=en https://support.google.com/analytics/answer/1033867?hl=en 2 - Obviously in this case it's pretty well statistically messed up by that link being posted to HN, but you get the point
  - eli 13y ago
    I think the "Googleness" is in your head. Those parameters were defined long before Google bought Urchin.
mfrommil 13y ago
Both Hotmail & Ironport reacted with a customer-centric focus (e.g. Hotmail: "How do we recover their emails? What else can we provide customers in the future? What steps will we take so this never happens again?") I would guess a lot of companies in a WFIO situation that take a CYA approach rather than customer-first face a worse fate.
zxcvvcxz 13y ago
Question - why did IronPort cancel the contract with their customers if they just got the feature implemented that they seemed to want? Was it a done deal - i.e. they were losing those customers anyways? Also, survivorship bias much? Two examples --> conclusion? Was nice hearing about those company's histories though, didn't know about the Hotmail thing.
- nedwin 13y ago
  I asked Scott the same question on Twitter. His response: "The Brightmail agreement was 80% went to them, 20% to us. IronPort A/S was 100% to us. Customers liked BM and we were unproven." https://twitter.com/W_ScottWeiss/status/448509947731804160 https://twitter.com/W_ScottWeiss/status/448509947731804160
- chrisweekly 13y ago
  "Instead of Symantec canceling the contract, we went on the offensive and faxed a letter to all of our customers cancelling the contract with THEM [Ed.:SYMANTEC]– a position of strength." I initially misread that too. He's actually saying they faxed a notice to their customers that they were canceling the contract with Symantec. It's worded poorly.
malanj 13y ago
I had serious WFIO case a few years ago. We were building a crypto solution on J2ME phones. The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app. As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo. After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked. We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough. We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.
- saryant 13y ago
  I ran into something similar a few months ago. There's a bug in Oracle JDK 7u45 that causes SSL handshakes to fail ~5% of the time due to a bug in that version's Diffie-Hellman cypher suite. Tracking that bug down was a fun three days.
caycep 13y ago
Hm, WFIO is kind of like "welping a fleet" in EVE...
resu_nimda 13y ago
I don't understand the meaning of "crack the egg with a sledgehammer." I would think it means "to use way more force than is necessary," but the context is more along the lines of "get the problem solved at all costs." I also followed the link to the other article where that phrase is used[1], and found something a bit concerning: "Nawaf moved the entire engineering team over to work on it. He called them all in to work nights and weekends until it was fixed, [...] Nawaf saved our bacon." Umm, what about those engineers? I'll give the benefit of the doubt that they were rewarded appropriately, but the wording here seems to almost deliberately stoke the developers vs. management flames, especially coming from a VC... [1] http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/ http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/
lauradhamilton 13y ago
"We're Fucked, It's Over" strikes me as a pretty dramatic response to a technology vendor failing to renew a contract. Unless they have extra-special access to customers or customer data, whatever they do can be replicated in-house.
Mz 13y ago
I am disappointed to see the top comment here criticizes this as "survivorship bias." Maybe the survivors have something useful to communicate that can help others survive what looks "impossible" when you first run into it? Maybe that's the point? Though this probably shouldn't be exactly a surprise to me given how much shit people give me any time I try to talk about getting well after doctor's basically wrote me off for dead in some sense. No one wants to learn from that either and I honest to god don't get it.
loladesoto 13y ago
if it's never as bad as it seems: does this make a case for optimism or realism on a team? regardless, that's precisely why you need a cofounder who balances you. whether it's a different perspective, questioning some of your choices, or holding you accountable (and holding you up): you need it. you'll make an order of magnitude more poor choices on your own.
ssully 13y ago
My palms got sweaty just reading the hotmail portion. I think someone could make a really great horror book that is just a collection of stories like this. I know it would give me nightmares.
ajsoh 13y ago
Fascinating to read a 1st person account of a near-death moment. "Companies are damn resilient" - I think entrepreneurs are damn resilient, too, for making it through crises like this over & over...

351 points by rajbala 13y ago

I really admire Scott from what he's written. He can be an inch close and still get out of WFIO situation. And he seems to know the difference when it's doomed like with e-commerce business. So keep rowing, Scott!

espressopowered 13y ago

Nice article. I particularly agree with the point on leadership. If you can motivate those around you, people perform so much better and can sometimes come up with better solutions. Doesn't always guarantee you coming back from the brink, but stands you in good stead.

pgrote 13y ago

"It's never really as bad as it seems." Very true. Many times when something happens the automatic response is to focus on the worst case scenario instead of the immediate problem at hand. Also, I couldn't select any text to copy and paste.

mathattack 13y ago
"It's never really as bad as it seems." A very good point for our personal lives too.
not_paul_graham 13y ago
This is great advice even for students because sometimes the things that younger people (anecdotal: me) give a lot of importance to and feel terrible for screwing up end up not mattering at all. 1. Things like validation from peers / popular kids 2. Screwing up on that major test that sometimes "defines" where you end up. A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly). 3. The academic thing is worse if you are artistically inclined and have science and math thrown down your throat telling you that humanities are for those that can't cut it in science (I know friends that have had a variation of this conversation with their kin/parents, etc) quite true in countries like India / Pakistan, Korea and perhaps even China. 4. Humiliation/shame from something that seems drastic in the moment (Getting rejected by the cute girl in front of all your friends and classmates. Yea that sucks). Getting labeled sucks too, a lot of girls can be mean to their own gender at that age which is absolutely disgusting. Unfortunately for most people it is something that they only figure out in hindsight.
- NAFV_P 13y ago
  $ A lot of suicides result from academic pressure and failure (plus the environment/parents hinting that you can't succeed in life if you can't do X, Y or Z properly). Heavy debts are a significant contributor to suicide rates, fear and shame play a large part. The potential collapse of companies would heighten this risk, so here's some statistics on the Great Depression (bottom of the page): http://www.shmoop.com/great-depression/statistics.html http://www.shmoop.com/great-depression/statistics.html
- aestra 13y ago
  >A lot of suicides result from academic pressure and failure I was wondering if this was actually true or not, turns out that there is some evidence suggesting it. http://apt.rcpsych.org/content/8/6/418.short http://apt.rcpsych.org/content/8/6/418.short http://www.tandfonline.com/doi/abs/10.1080/13811110590904016#.UzGOaVcZGC8 http://www.tandfonline.com/doi/abs/10.1080/13811110590904016...
- rk17 13y ago
  Great comment. Off-topic: funny name.
- Fomite 13y ago
  I had an Economics professor who gave a small lecture to our class after we got our midterm grades back. "This midterm is one third of your grade. This class is one of four you are taking this semester. This semester is one of eight you will take at $University. And while people will tell you otherwise, your time in college is a small part of your life. It will be okay."
mildtrepidation 13y ago
This is not true. It's almost always true in most contexts, but to say that it's a fact is to encourage a false sense of security... sometimes things are that bad, and it's important to be able to realize and accept that, otherwise you never even give yourself the option to try and improve them.
vvvv 13y ago
It's never so bad, that it can't get worse ;]
ams6110 13y ago
Just the point I came to comment on. There have been many times in my life where things seemed to be completely falling apart. But I have learned that I almost always paint a much bleaker picture than what actually ends up happening. Life goes on.
jakobe 13y ago
> Also, I couldn't select any text to copy and paste. Seems to be a Safari bug. I had the same thing, after loading the website I couldn't even select text on HN anymore. Restarting Safari fixed it.

mathattack 13y ago

Great points, especially leading from the front when times are tough. I didn't think Hotmail would have survived the outages, but they came out just fine.

cousin_it 13y ago

Can't select any text to copy/paste, can't click any of the links in the post or sidebar. #welcometothenewweb

deleted 13y ago
[deleted]
mmastrac 13y ago
I'm guessing it's some sort of layout thing gone horribly wrong. Not really the new web to blame here -- just poor testing.
userbinator 13y ago
Site works fine without JS, and I can do all of those things too. Although all those extra parameters in the URL are a little odd.
- aestra 13y ago
  For what its worth, the page works fine without all the parameters. Why they are in the submitted link, I don't know.
  - Two9A 13y ago
    The utm_* parameters tend to be used for source tracking; the important ones here being "content" and "campaign". Of course, this particular link doesn't refer to HN in its campaign code, so their statistics are going to get a little confused.
cyphax 13y ago
I had this problem also but I was logged in to wordpress.com (or .org, I don't even know, but it had the black bar at the top from WordPress) and when I signed out, that problem went away immediately.
gabemart 13y ago
A transparent portion of the #comments div is covering the body text and sidebar. If you change #comments position from relative to initial, things work as expected. I think the problem is that the sidebar is floated and the float isn't cleared. Setting a clear: both on #comments prevents the div from overlapping the body and sidebar.

JoeAltmaier 13y ago

Some survivor bias there, but an inspiring message: always continue as if you're going to survive. All the real survivors did that.

deleted 13y ago

[deleted]

JonLim 13y ago

> Get all the brains around the table An important addition to this: get everyone involved, but also listen to their ideas. Having been a part of a sinking ship, the most frustrating thing about having everyone pour their blood, sweat, and tears into reviving the company is when the top brass decides to ignore all of the hard work and carry on with their own ideas that were never cleared with anyone else. They were their own iceberg.

randall 13y ago

This is essentially the definition of survivorship bias. http://www.fastcodesign.com/1671172/how-a-story-from-world-war-ii-shapes-facebook-today http://www.fastcodesign.com/1671172/how-a-story-from-world-w... "In WWII, Allied bombers were key to strategic attacks, yet these lumbering giants were constantly shot down over enemy territory. The planes needed more armor, but armor is heavy. So extra plating could only go where the planes were being shot the most."... "[A guy in charge] said the military didn’t need to reinforce the spots that had bullet holes. They needed to reinforce the spots that didn’t have bullet holes." Having experienced 2 "WFIO" things in the last 6 months, I agree that entrepreneurs need to be resilient, but I think it's more valuable to read post mortems than these "we almost died" posts because they prove more instructional. For instance, Aaron's post about Tutorspree is a really useful post for me, although it definitely has less of a feel good vibe to it. http://www.aaronkharris.com/when-seo-fails-single-channel-dependency-and-the-end-of-tutorspree http://www.aaronkharris.com/when-seo-fails-single-channel-de...

jacobparker 13y ago
Relevant Hark! A Vagrant http://www.harkavagrant.com/index.php?id=206 http://www.harkavagrant.com/index.php?id=206 The sentiment seems very popular in the start-up scene :)
- twic 13y ago
  May i say how wonderful it is too see a relevant Hark! rather than a relevant XKCD for a change. Class, tonight's homework is to find a relevant Dinosaur Comics. For extra credit, a relevant Achewood.
  - kstop 13y ago
    http://www.achewood.com/index.php?date=05312002 http://www.achewood.com/index.php?date=05312002 (I don't do regular credit.)
  - taybin 13y ago
    You have excellent taste in comics.

mkrecny 13y ago

Fresh out of college I was responsible for a WFIO at a 40 person company - ultimately had to leave. http://edu.mkrecny.com/thoughts/how-i-fired-myself http://edu.mkrecny.com/thoughts/how-i-fired-myself "Um those [backups] got really expensive, so we stopped doing them about a month ago" sounds painfully familiar.

thejosh 13y ago
I remember reading a while ago and cringing at this so bad, this needs to be a cautionary tale of version control, development/staging/production environments and ... backups.
- mkrecny 13y ago
  "The Startup CTO's Guide to ..." ?
  - thejosh 13y ago
    all it contains is the word: "backups".
    - angersock 13y ago
      Multiple times, for redundancy. It should probably also contain the word "documentation of system configurations".
    - sliverstorm 13y ago
      But why would a rockstar-ninja-pirate ever need backups?
      - FireBeyond 13y ago
        Backups are not MVP! First to market!
gabemart 13y ago

sswaner 13y ago

The same principles were true and applicable in a much larger near-implosion that I experienced from 2008-2012 working for a "too big to fail" financial company. Strong, out-front leadership, a strong team led the way. It took time for us to realize that it wasn't as bad as it seemed. We had to learn that the external pressure from the media, public and even friends and family was only a distraction. In this situation it was quickly apparent that my role was to focus on a solution, not dwell on the problem or the cause. This should be true for everyone on the team, unless you are the specific individual who caused a global financial meltdown, deleted all the email or caused a critical failure.

tsunamifury 13y ago
Actually that was a problem due to the culture of stupidity and willful ignorance of risk, not some keyboard mash that caused the markets to crash. It might be one of the rare situations where it was so bad, that after stabilizing the situation, many of the responsible parties needed to be put down. I know from my experience working with Wachovia, the gross incompetence, system wide narrow minded greed and the fuck-everyone-but-me attitude was so bad it was mind boggling. It was not a mistake, this was a concequence of the culture and goals of those companies.
rhizome 13y ago
It took time for us to realize that it wasn't as bad as it seemed. I'm not sure this example is relevant, since this realization would not have been possible without a massive leg-up and pat on the head from your national government. That is: "too big to fail" negates "near-implosion."

michaelochurch 13y ago

The leader needs to be the first one there, the last one to leave, and be willing to do anything it takes – like answer customer care calls or personally drive a replacement part to an irate customer. Nothing is beneath a leader in times of crisis. I like this, because it says two things at once, both true. First, if you're in a position of ownership/authority but you're not willing or able to do the grunt work, you won't have the credibility for long. People will follow orders out of fear of getting fired, but you'll never get more than the bare minimum. There's a point where leaders are replaced or outnumbered by true executives (lazy, bikeshedding, rent-seeking parasites) and after that point, the organization can't even motivate shit to stink. Second, it might be that it's just not worth it to lead from the front. That means that you're not really in a position of leadership. You might be a middle manager who realizes that the people above you will never buy in to what you're doing. Then you'll probably lose that desire to make those sacrifices. That's fine. You shouldn't tie yourself to the mast, at that point, because you're not really in a leadership role anyway.

orthishappended 13y ago

The most interesting take away from that was how resilient companies appear to be despite their leaders best efforts to sabotage everything. I think the real lesson is (and I don't like it): if you want your company to survive you need to be good at spin. In the case of hotmail, how did he ever let a situation arise where one bug in a nightly cron script could obliterate his users emails?! No backups because they cost too much?? "the sun, the moon and the stars lined up against us for it to happen" No, you were just reckless and too busy chasing growth. With Ironport, he says his whole business was built on the back of anti-spam yet they were totally unprepared when their partner who provided the anti-spam tech they used pulled the rug from under them (despite knowing full well this day would come). The real lesson here should be: try not to outsource the key component upon which your entire business depends. I had to laugh at the end when he says the only time he's had a failure was when he was using his own limited resources to fund a business and not investor money. Well yes... mistakes are easier to hide/absorb when you're swimming in cash. Not so much when things are tight and you can't afford to learn after the fuck up.

dsugarman 13y ago

how did you build the spam filter? bayesian classifier?

herdrick 13y ago
No, not in 1997.

eterpstra 13y ago

"Nothing is beneath a leader in times of crisis" I've worked for people that believe this, and others that don't. Nothing builds morale and loyalty like rolling up your sleeves and doing the hard stuff with your employees. I once had a project manager who couldn't write a lick of code, but whenever a deadline approached or something blew up, he'd be right there at 10pm with everyone else bringing in fresh pizza, testing whatever he could, and giving pep talks. Contrast this with someone that says, "get this done by tomorrow, OK?" and then walks out the door. Who do you think is going to have a higher performing team in the long run?

jzwinck 13y ago
In the truly long run, one of those teams will either burn out or die of a collective heart attack from all the late nights eating pizza at its desk. Still, it's nice when someone chips in to do additional testing before release.
incision 13y ago
>"Who do you think is going to have a higher performing team in the long run?" The project manager who manages the team, schedule and risks such that things don't routinely blow and run down to the wire would be preferable to either.
- eterpstra 13y ago
  lol. In a perfect world where bugs never happen, estimates are always perfect, and last-minute top-down requirements changes never occur, I would totally agree with you.
digikata 13y ago
I'd rather be working for the project manager that tells everybody to go home at a reasonable hour, get a good nights sleep, and come back fresh in the morning. That's the most productive. It's amazing how often an intractable bug in the evening becomes an, "ah hah, it's x" in the morning.

TheCraiggers 13y ago

Am I the only one that thought the query string was actually more interesting than the article? ?utm_content=bufferc2fec&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer I'm glad that ycombinator isn't listed as the source, at least. I think it would be fairly interesting to see what kind of data they have about various social medias and link propagation. And also, a slight sickening sensation that marketers are doing this. Not that I'm surprised, it's just that I'm sure they have papers written saying that "Twitter has a 64% link click rate if you write your post like X, whereas Facebook can achieve as high as 70% if you do Y." I know it has been going on for years on the web, and decades for other marketing, but here it's plain to see. It fills me with unease, like I'm no longer in control and what I like in life is already planned out.

showerst 13y ago
those are standard google analytics campaign tags, they probably (automatically?) built the link that way before they originally shared it on twitter, and then someone cut & pasted that link to share on HN.
michaelbuckbee 13y ago
This is a pretty standard Google Analytics Tracking URL [1] - setting aside the Googleness of it (which raises a different set of issues) I look at it like the modern equivalent of an Apache Access Log. Also, not sure if this will make you feel better, but most places that are doing this type of in-depth analytics, etc. are much more often trying to bend themselves to your will ("what feature, benefit or product that people will get value out of") than to trick you into liking something you otherwise wouldn't. One way of interpreting that query string is as the answer to an implicit survey question of "How would you like to get the stories we write?" the answer in this case being: "Twitter" [2]. 1 - https://support.google.com/analytics/answer/1033867?hl=en https://support.google.com/analytics/answer/1033867?hl=en 2 - Obviously in this case it's pretty well statistically messed up by that link being posted to HN, but you get the point
- eli 13y ago
  I think the "Googleness" is in your head. Those parameters were defined long before Google bought Urchin.

mfrommil 13y ago

Both Hotmail & Ironport reacted with a customer-centric focus (e.g. Hotmail: "How do we recover their emails? What else can we provide customers in the future? What steps will we take so this never happens again?") I would guess a lot of companies in a WFIO situation that take a CYA approach rather than customer-first face a worse fate.

zxcvvcxz 13y ago

Question - why did IronPort cancel the contract with their customers if they just got the feature implemented that they seemed to want? Was it a done deal - i.e. they were losing those customers anyways? Also, survivorship bias much? Two examples --> conclusion? Was nice hearing about those company's histories though, didn't know about the Hotmail thing.

nedwin 13y ago
I asked Scott the same question on Twitter. His response: "The Brightmail agreement was 80% went to them, 20% to us. IronPort A/S was 100% to us. Customers liked BM and we were unproven." https://twitter.com/W_ScottWeiss/status/448509947731804160 https://twitter.com/W_ScottWeiss/status/448509947731804160
chrisweekly 13y ago
"Instead of Symantec canceling the contract, we went on the offensive and faxed a letter to all of our customers cancelling the contract with THEM [Ed.:SYMANTEC]– a position of strength." I initially misread that too. He's actually saying they faxed a notice to their customers that they were canceling the contract with Symantec. It's worded poorly.

malanj 13y ago

I had serious WFIO case a few years ago. We were building a crypto solution on J2ME phones. The solution was days away from roll-out with our first big corporate client. Late on a Friday afternoon we were busy with final field testing - paying students a few $ to use their phones to test the app. As the test data came in we realised we had a major problem: a small % of the phones weren't returning the correct test vectors for hashing algo. After checking for obvious user error we came to the conclusion that something big was broken. Specific firmware sets didn't execute the crypto part of the code correctly. The entire value-proposition was that it works on every phone that can run an app, so it was a pretty big deal. I thought we were totally fucked. We didn't sleep for two days and finally found the bug in the way the phones implement a bit-shift operation (doesn't carry a bit about 1/10000000 times). Then had to figure out a workaround that was still fast enough. We shipped a fixed version before the end of the weekend, but I wouldn't wish that kind of stress on anyone.

saryant 13y ago
I ran into something similar a few months ago. There's a bug in Oracle JDK 7u45 that causes SSL handshakes to fail ~5% of the time due to a bug in that version's Diffie-Hellman cypher suite. Tracking that bug down was a fun three days.

caycep 13y ago

Hm, WFIO is kind of like "welping a fleet" in EVE...

resu_nimda 13y ago

I don't understand the meaning of "crack the egg with a sledgehammer." I would think it means "to use way more force than is necessary," but the context is more along the lines of "get the problem solved at all costs." I also followed the link to the other article where that phrase is used[1], and found something a bit concerning: "Nawaf moved the entire engineering team over to work on it. He called them all in to work nights and weekends until it was fixed, [...] Nawaf saved our bacon." Umm, what about those engineers? I'll give the benefit of the doubt that they were rewarded appropriately, but the wording here seems to almost deliberately stoke the developers vs. management flames, especially coming from a VC... [1] http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/ http://scott.a16z.com/2014/02/03/harvey-keitel-ceo/

lauradhamilton 13y ago

"We're Fucked, It's Over" strikes me as a pretty dramatic response to a technology vendor failing to renew a contract. Unless they have extra-special access to customers or customer data, whatever they do can be replicated in-house.

Mz 13y ago

I am disappointed to see the top comment here criticizes this as "survivorship bias." Maybe the survivors have something useful to communicate that can help others survive what looks "impossible" when you first run into it? Maybe that's the point? Though this probably shouldn't be exactly a surprise to me given how much shit people give me any time I try to talk about getting well after doctor's basically wrote me off for dead in some sense. No one wants to learn from that either and I honest to god don't get it.

loladesoto 13y ago

if it's never as bad as it seems: does this make a case for optimism or realism on a team? regardless, that's precisely why you need a cofounder who balances you. whether it's a different perspective, questioning some of your choices, or holding you accountable (and holding you up): you need it. you'll make an order of magnitude more poor choices on your own.

ssully 13y ago

My palms got sweaty just reading the hotmail portion. I think someone could make a really great horror book that is just a collection of stories like this. I know it would give me nightmares.

ajsoh 13y ago

Fascinating to read a 1st person account of a near-death moment. "Companies are damn resilient" - I think entrepreneurs are damn resilient, too, for making it through crises like this over & over...