9 ms·

Peter Molyneux’s staff work 36 hours non-stop

108 points by beedogs 14y ago

rprasad 14y ago
Judging from the reviews for Curiosity in the Google Play store, it was not time well-spent. (I would provide my own comments on the game, but it insists on crashing before I can do anything with it.) If it is a joke...Mr. Molyneux would have better spent the time on fixing his abysmally bug-ridden game.
damian2000 14y ago
My advice is for Peter to have a read of "Peopleware: Productive Projects and Teams" by DeMarco & Lister.
- markokocic 14y ago
  My advice to _anyone_ in our field is to read this book. Even if you yourself are not manager, and never have managed any team, it will at least make it easier to spot some common pitfalls and be aware of the consequences before it's too late.
jpxxx 14y ago
Industry bullshit artist approaching has-been status... super high-concept game design without any actual game... his nephew as producer... five staff total working on a massively parallel realtime cross-platform title with novel mechanics... What could possibly go wrong? popcorn.gif
- supersaiyan 14y ago
  Peter Molyneux in a nutshell : http://www.youtube.com/watch?v=DkGMY63FF3Q http://www.youtube.com/watch?v=DkGMY63FF3Q But aside from the hate 22cans is doing some impressive stuff
- davedx 14y ago
  There's as much game in it as 90% of what's on the app stores. Make of that what you will...
astrodust 14y ago
Unless you're battle-hardened and able to endure multiple 36-hour runs in a single week, which few of us can withstand before something completely snaps, this is probably the worst possible idea. You do a big push like this as a last resort, not as standard practice. It reminds me of an anecdote about Icelanders and how they do their off-road driving: You keep your vehicle in two wheel drive. If you get stuck you can always use the four wheel drive to get back out. If you're always in four wheel drive, when you're stuck you're really stuck.
- biscarch 14y ago
  Nowww you knowww But seriously. Interesting point about Icelander off-road driving style. It makes sense for terrain that could stick you while in four wheel drive, but it's never occurred to me to drive like that. Ex: Chicago snow storm about 2 years ago, I just drove super-careful but many people had to abandon their cars on the side of the road.
- brey 14y ago
  heh, cute analogy, and I agree with the sentiment. but having just returned from an Iceland 4x4 trip in places you absolutely need 12 inches of ground clearance and a snorkel, four wheel drive gets turned on the moment you start on F-roads. you really don't want to be slipping into a fjord on icy gravel roads because your front wheels don't have any power. if you get to a technically hard part, you engage diff lock and your low range transfer box.
jrockway 14y ago
The author of these tweets should read a book about literacy. He could learn something worthy of cerlibretion.
BklynJay 14y ago
On what planet does this make sense? Here on Earth, this is generally considered "unwise."
dpeck 14y ago
I like Mr Molyneux's imagination, but I have to say that I enjoy his deliverables significantly less. Whether intentionally hyped or not he has become, in my mind, quite the bullshitter. Seeing this I'm not surprised by the disconnect between his vision and products. I'd advise anyone against purchasing something from his group without having a very generous return policy.
- damian2000 14y ago
  I read here (see below link) they are soliciting user donations in order to speed up the bug fixing of the server code... so presumably to help pay the overtime of the coders. First time I've heard that a game user has been asked to pay to fix bugs in their game. I congratulate Molyneux on his chutzpah. "The popularity of the app has now forced Molyneux to ask for donations so that they can help improve the experience, but we’re not too certain how well this plea will be received." -- http://www.phonesreview.co.uk/2012/11/13/curiosity-app-game-by-22-cans-peter-molyneux-social-cube/ http://www.phonesreview.co.uk/2012/11/13/curiosity-app-game-...
  - jpatokal 14y ago
    Srsly? The Curiosity app is free. The donate button isn't even in the app itself, but on the developer's web page.
  - rhizome 14y ago
    Don't trust an owner who won't put their own money into their business.
- ekianjo 14y ago
  Molyneux has stopped delivering on his promises long, long ago. I think it was from Fable that I noticed that the most. He proposed an open world with all your choices having an impact on all characters and nature around, but it ended up simply a series of binary choices between good and evil. Hardly revolutionary. Well I enjoyed Populous, Dungeon Keeper and Theme Park. After it all went down the drain.
bryceneal 14y ago
You're making a lot of assumptions here based on a tweet. Also calling these employees "idiots" is pretty irresponsible in itself.
- beedogs 14y ago
  Frankly I just copy-pasted the title from the post. And it looks like it's been 'corrected' now.
wallflower 14y ago
The danger of incurring so much sleep debt so quickly is that you will start paying compound interest on it immediately - in terms of ability to avoid accidents, struggling with mental coherence, handling relations with loved ones, out-of-it attitude, and dealing with unexpected situations gracefully.
jpatokal 14y ago
Jeebus, what a way to turn a molehill into a mountain. @22cans: In fact @pmolyneux BEGGED the team to go home but they wouldn't have any of it :) https://twitter.com/22Cans/status/268387590368030721 https://twitter.com/22Cans/status/268387590368030721 22 Cans is a tiny startup and they were expecting 10,000 concurrent users tops -- they're at 2 million and counting now, and they have to capitalize on that momentum and not piss off their obviously fanatical players. So they pulled an all-nighter to build and migrate from a single server to a clustered architecture, and hell yeah, if I was in that position I'd do that too. Seriously, is there another option? If you think the answer is "clock out at 5:01 PM and go home", you shouldn't be in a startup. Obviously pulling 36 hour shifts on a regular basis is not sustainable, but this is precisely the type of situation where it's warranted.
- wallflower 14y ago
  Keeping the servers up under heavy load is part of the startup archetype. You have to cross the Rubicon. > The servers melted. Systrom and Krieger worked a straight 24 hours to keep the app running—in that period 25,000 iPhone users downloaded the free service. http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagrams-kevin-systrom-the-stanford-millionaire-machine-strikes-again/3/ http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagr...
- fatbird 14y ago
  The question is why did they assume 10k concurrent users, or plan for that to be the cap handled under the current architecture? Any competent project manager should have said "let's pretend we're massively successful... how will it work then?" and have some contingency plan in place so that they're not trying "experimental" fixes. The flaw here isn't the heroic efforts, it's that heroic efforts were needed. In the days of cloud based architectures, it's a lot easier to be ready for massive horizontal scaling than it used to be, so much so that being unready is malpractice.
  - Negitivefrags 14y ago
    Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of money (and more importantly time) designing for load they never needed to handle before release. In about a month my company will be releasing an online game. We think we will get about 15-20k concurrent users in the first few days, but we will be testing the realm for as high as 50k concurrent users using load testing bots. What will happen if we suddenly get 100k or 200k concurrent users? Even throwing more hardware at the problem might not solve it immediately if there is some new bottleneck we haven't found before. Testing the realm for more than 50k concurrents and then spending engineering effort required to make it work will consume time and effort that we can't afford to spend. All for an outcome that is unlikely to occur.
mikeklaas 14y ago
When we launched Zite, I (along with several other engineers), were awake for about the same amount of time keeping the servers alive. Sometimes a heroic emergency effort is required. As long as it is (extraordinarily) rare, it's not in itself sign of a dysfunctional organization.
- hrktb 14y ago
  > heroic emergency effort Completely agree that it'something required sometimes. But it's nothing heroic, it is a (temporary I hope) dysfunctionment, and shouldn't be put in midly positive light. Planning failed (not assuming you might have 2m users is sensible, but it's still a failure in it's way), and shit hit the fans. It's the kind of failure you might wish you had, but next time you should think hard about a way to deal with this kind of situation with minimal emegency, and no need for 'heroes'.
cwh 14y ago
as one of two 'server coders' that scaled Draw Something from 0 to 35 million DAU, a 36 hour day sounds nice.
- miahi 14y ago
  What was your motivation?
  - cwh 14y ago
    quite simply, to see just how far we could take it.
ispivey 14y ago
For all of you crapping on Peter and his team: What if the employees were founders with equal shares of the company, and they launched a product and it fell over? Would you blame them for pulling an all-nighter to scale it up in the face of unexpected traffic? Should they give up at 10pm? Midnight? 5AM? If they're not equal partners, should they immediately give up? What if they own 10%? 1%? What if they don't have much on their resumes and are excited to make this a successful project, instead of a laughingstock that fell over in the face of scaling challenges? What if they care about each other and want to make this a success for their friends and coworkers? What's the function into which you plug all these variables and decide when to go home? It's not black-and-white, pun intended, and if you think it is you're being obtuse. I hope they feel happy and like they accomplished a lot in the face of long odds, and you should too.
- Deestan 14y ago
  I'd rather have a colleague come in to work drunk than pull an all-nighter to work on core functionality. Yes, we really need to get this thing improved as soon as possible, but having effectively brain-damaged people introduce bugs all night and then be utterly useless the next day is not helping. Take your financial lumps and get over it instead of burning out everyone and ruining the codebase. These kind of masochistic hero stories perpetuate the destructive belief that every hour your face is planted in front of the monitor is equally productive.
- vidarh 14y ago
  If someone I worked with tried pulling a 36 hour shift I'd physically throw them out of the building. In my mind, odds are 90%+ that'd they'd do more damage (more bugs, slowing down other people) than they'd do good in aggregate over that 36 window, and odds that they'd be more productive in a single 36 hour bout than doing shifts with proper sleep windows is pretty much 0. More than 14 hours or so in a single stretch is highly unlikely to be beneficial to the company/project, with rest in the middle. More than just being in line with research, this is in line with what I've seen over and over again in real projects, where best project during crunch time is generally had by ensuring people leave and rest. Sure, do longer hours for a short period. Sure, ask people to cut down on time off or alternate work/sleep shifts. But the moment your developers hits 18-20 hours, I could talk most of them into making horrifically bad decisions and have them think they were logically sound - your ability to reason goes out the window pretty quickly. I don't want people in that kind of state anywhere near projects I work on, because I've seen what they can, and will, do in the belief they're helping out. > What if they don't have much on their resumes and are excited to make this a successful project, instead of a laughingstock that fell over in the face of scaling challenges? What if they care about each other and want to make this a success for their friends and coworkers? Then learning to understand that there's nothing smart about pulling stunts like this unless you're doing menial work that you can still do reasonably reliably with next to no concentration or short term memory is even more important. > I hope they feel happy and like they accomplished a lot in the face of long odds, and you should too. No, I shouldn't, because if they feel that way they've learned the wrong lesson. They'd also most likely be wrong in terms of their relative productivity with more rest. Unless they were all on illegal stimulants, or his staff consists of mutants, it is simply not worth spending much time even considering the option that this might somehow have been the best way for them to spend their time.
gobengo 14y ago
No one should be pressured into this type of work, and it's important to always maintain that. However, if you haven't worked a 48hr+ stretch in a young startup, check your pace and/or commitment. It may be fine, just think about it.
seivan 14y ago
I'm a gamer and this is very subjective of course but Peter Molyneux hasn't actually done any good games. Overrated junk ~ and it's not his works, it is the engineers, designers, illustrators, sound engineers and actors who've done the game. He just makes enough noise to get attention and praise.
- vacri 14y ago
  His early games were interesting, fun, and original, pretty much all the Bullfrog catalog. I'm also not sure you can get away with saying it's not his work when he's the lead designer and executive producer on those titles. Yes, he didn't do it alone, but those roles stamp a considerable amount of personality onto the product. Lionhead produced nothing but crap.
politician 14y ago
Anyone else find Curiosity intriguing not for the game (which is arguably about vanity), but for the technical brain-trust that operating such a service creates? Here's hoping they all write books...
JanezStupar 14y ago
I did a 36 hour push once. I did it because there was really no other way to achieve the deadline, since there was about three days worth of bug squashing to reach a deadline and I only had two days left. I am proud that I managed to pull it off, however I tell it as a tale of caution not as a tale of inspiration. The work I did was objectively atrocious, by the 24 hour mark the simplest operation took me 5-10 minutes. My short term memory was completely gone by then and I had to rely on writing down each step of an operation as simple as refactoring a block of code into its own function, just to be able to pull it of in a "reasonable time frame". Thus I would call bullshit on anyone claiming that his team made "huge improvements" and that after such a push some piece of software "started behaving like it should". Unless that means that they introduced a plethora of bugs that they would be busy fixing for months from now.
- babebridou 14y ago
  The experience of going for an all-nighter is akin to watching a full season of 24 in one go, only you're the protagonist. It's memorable, and yes, especially if you manage to pull it off. I tell it as a tale of inspiration: do it, once. Whether it's good or bad for business/health/morale is a whole other story. Hint: it's not good.
  - JonnieCache 14y ago
    One of the best things about going to university is you can get all of that stuff out of the way without hurting anyone else.
    - anthonyb 14y ago
      Well, until you graduate and continue your old habits...
  - TeMPOraL 14y ago
    That's what Global Game Jam is for, it's a 48h competition for writing games. I attended two of those, and during those 48h took maybe total of 8h naps here and there. Amazing experience. But yes, really, at some points your brain just stops working.
MortenK 14y ago
To "just work harder", in this case 36 hours, is the easy decision to make when something goes wrong. It smells deliciously of heroics, bravado and top performer mentality. Most importantly, the soul drenching, exhausting late hours, gives the illusion that "we are doing all that is humanly possible to resolve the problem". How tough are we, right? But as the author points out, heaps of literature clearly show that "crunching", like it's affectionately called in the games industry, is contra-productive. One guy working 8 hours, going home for a good sleep, and coming back to work an additional 8 hours will vastly outperform the developer-turned-zombie that's been at it for 24 hours straight. At that point, short term memory is gone, stupid mistakes (blunders even) are rampant. Fixing simple issues takes extraordinary amounts of time, because the ability to focus is all but lost. You effectively have tunnel vision and your reactions and logical skills are reduced to that of a drunk. This is not anecdotal - there are an immense amounts of studies done on the subject. But time and again, the knowledge gets discarded because it's just easier to "work through the night" at a frantic pace, rather than take the necessary step back, let people rest and do the work in an orderly and planned manner. By burning the candle in both ends, at least no one can challenge your commitment to solving the problem. While it may sound counter-intuitive to go to bed while the house is on fire, it is, at least if 40 years of research is to be believed, actually the only correct thing to do in a software development crisis. Note this doesn't mean "stick 100% to 8 hours and always go home at 17.01", but it does mean if you are still there at 23.00 you are much better off going home, getting sleep and resuming work when well rested. The games industry seem to be especially vulnerable to this problem. It might be due to the general inexperience level caused by games development being a young mans game, or it might be due to a failure to understand that games, while glorified in the industry, is "just" software, and therefore also subject to all the established rules of software development. In any case, I agree with the author that Molyneux should know better than to tweet from the rooftops how many hours everybody is working, but I guess he is not thinking entirely straight after 36 hours of work.
- jacquesm 14y ago
  It depends on whether or not a fix is perceived to be further away than what you can accomplish in that one night. Being down for yet another business day when you don't have to be ('house on fire') is usually not an option. There is also a class of problems (chasing subtle bugs) that you're much better equipped to tackle when you come in fresh in the morning or when you're actually not looking directly for them, the flip side is there are those that it takes forever to get the mental model set up that maps the circumstances of the bug with enough resolution that you have a chance of cracking them that it pays off to just sit there until it's done, even if that means staying up well past your bed time. The rule seems to be 'go home if you can no longer positively contribute towards solving the problem', not just 'go home'.
michaelgrafl 14y ago
I'm a big proponent of working (i.e. laboring) as little as you possibly can in order to leave as much room as possible for convivial activities and personal growth. But I think it is disingenuous in this case to paint Molyneux as a slave driver. They were ill-prepared for the launch and stuck together to scoop the water out of the boat. It's probably a bad idea to do it for 36-hours straight, but it's understandable. I would certainly stick my teeth in there till the storm's passed. Especially in a company this small, where the line between collegiality and friendship are probably quite blurred. I'm much more interested in what the RSI is a result of. That shit needs to be taken care of.
borplk 14y ago
It's all our own fault. We glorify the noodle-eating couch-sleeping code-generating-machine hacker and let the corporations profit by exchanging our health and personal lives for stupid hacker-perks which costs the companies nothing. Woohoo! come hack for us, we give you a fancy chair and large monitors to stare at and even food! what else do you want out of your life? We need to see that we are being taken advantage of. We have raised expectations so high, that you can't land a decent job unless you have dedicated your evenings to patching bugs in open source projects for free. What do you think that says to an employer? Sorry to break it to you but it roughly goes like this, "The poor bastard has no life. When he goes home, he does the same crap for fun. He'll be more than happy if I pay him 2 peanuts" Have you asked yourself why other professionals don't have "open source" like we do? If programmers are expected to work for fun to develop your email client, web browser, server softwares etc... it is only fair if lawyers were also expected to contribute to cases for free. I'm too damn tired of this crap. We have put ourselves into an infinite loop of competition that only hurts ourselves and brings profit to the corporatoins. The guy next to you has to sacrifice his life because he needs to compete with you to stay relevant and you do for the same reason and so on... We are smarter than this. We deserver better than this. We need to bring this to a stop.
- philbarr 14y ago
  In other professional fields (like with lawyers), doing work for free is called "pro bono". I would think that if you're being interviewed for lawyer position they would ask you how much pro bono work you've done. So I disagree that doing this stuff for free is inherently bad, but I fully agree with the sentiments of the rest of your post. Particularly this: "come hack for us, we give you a fancy chair and large monitors to stare at and even food! what else do you want out of your life?" An old manager of mine was once on the phone to a recruitment agency trying to hire some developers complaining that they hadn't made progress, at one point he screamed, "it's software development, it's not fucking rocket science!" He did that in a room full of developers. For some reason many non-technical people get to become managers of technical people and have that kind of opinion. This then leads to the, "well if it doesn't work you can stay here all night until it does" mentality.
jheriko 14y ago
... and to think all they needed to do was hire one real programmer in the first place.
antidoh 14y ago
We hope none of those people are driving when they finally go home.
lyricalpolymath 14y ago
Does anyone know to what kind of books or reference is the author talking about when talking about the "15 years of free literature on how to run large-scale online projects" I've seen in the comments that damian2000 (http://news.ycombinator.com/item?id=4781706 http://news.ycombinator.com/item?id=4781706) has suggested "Peopleware: Productive Projects and Teams" by DeMarco & Lister. any other suggestions?

108 points by beedogs 14y ago

Judging from the reviews for Curiosity in the Google Play store, it was not time well-spent. (I would provide my own comments on the game, but it insists on crashing before I can do anything with it.) If it is a joke...Mr. Molyneux would have better spent the time on fixing his abysmally bug-ridden game.

damian2000 14y ago

My advice is for Peter to have a read of "Peopleware: Productive Projects and Teams" by DeMarco & Lister.

markokocic 14y ago
My advice to _anyone_ in our field is to read this book. Even if you yourself are not manager, and never have managed any team, it will at least make it easier to spot some common pitfalls and be aware of the consequences before it's too late.

jpxxx 14y ago

Industry bullshit artist approaching has-been status... super high-concept game design without any actual game... his nephew as producer... five staff total working on a massively parallel realtime cross-platform title with novel mechanics... What could possibly go wrong? popcorn.gif

supersaiyan 14y ago
Peter Molyneux in a nutshell : http://www.youtube.com/watch?v=DkGMY63FF3Q http://www.youtube.com/watch?v=DkGMY63FF3Q But aside from the hate 22cans is doing some impressive stuff
davedx 14y ago
There's as much game in it as 90% of what's on the app stores. Make of that what you will...

astrodust 14y ago

Unless you're battle-hardened and able to endure multiple 36-hour runs in a single week, which few of us can withstand before something completely snaps, this is probably the worst possible idea. You do a big push like this as a last resort, not as standard practice. It reminds me of an anecdote about Icelanders and how they do their off-road driving: You keep your vehicle in two wheel drive. If you get stuck you can always use the four wheel drive to get back out. If you're always in four wheel drive, when you're stuck you're really stuck.

biscarch 14y ago
Nowww you knowww But seriously. Interesting point about Icelander off-road driving style. It makes sense for terrain that could stick you while in four wheel drive, but it's never occurred to me to drive like that. Ex: Chicago snow storm about 2 years ago, I just drove super-careful but many people had to abandon their cars on the side of the road.
brey 14y ago
heh, cute analogy, and I agree with the sentiment. but having just returned from an Iceland 4x4 trip in places you absolutely need 12 inches of ground clearance and a snorkel, four wheel drive gets turned on the moment you start on F-roads. you really don't want to be slipping into a fjord on icy gravel roads because your front wheels don't have any power. if you get to a technically hard part, you engage diff lock and your low range transfer box.

jrockway 14y ago

The author of these tweets should read a book about literacy. He could learn something worthy of cerlibretion.

BklynJay 14y ago

On what planet does this make sense? Here on Earth, this is generally considered "unwise."

dpeck 14y ago

I like Mr Molyneux's imagination, but I have to say that I enjoy his deliverables significantly less. Whether intentionally hyped or not he has become, in my mind, quite the bullshitter. Seeing this I'm not surprised by the disconnect between his vision and products. I'd advise anyone against purchasing something from his group without having a very generous return policy.

damian2000 14y ago
I read here (see below link) they are soliciting user donations in order to speed up the bug fixing of the server code... so presumably to help pay the overtime of the coders. First time I've heard that a game user has been asked to pay to fix bugs in their game. I congratulate Molyneux on his chutzpah. "The popularity of the app has now forced Molyneux to ask for donations so that they can help improve the experience, but we’re not too certain how well this plea will be received." -- http://www.phonesreview.co.uk/2012/11/13/curiosity-app-game-by-22-cans-peter-molyneux-social-cube/ http://www.phonesreview.co.uk/2012/11/13/curiosity-app-game-...
- jpatokal 14y ago
  Srsly? The Curiosity app is free. The donate button isn't even in the app itself, but on the developer's web page.
- rhizome 14y ago
  Don't trust an owner who won't put their own money into their business.
ekianjo 14y ago
Molyneux has stopped delivering on his promises long, long ago. I think it was from Fable that I noticed that the most. He proposed an open world with all your choices having an impact on all characters and nature around, but it ended up simply a series of binary choices between good and evil. Hardly revolutionary. Well I enjoyed Populous, Dungeon Keeper and Theme Park. After it all went down the drain.

bryceneal 14y ago

You're making a lot of assumptions here based on a tweet. Also calling these employees "idiots" is pretty irresponsible in itself.

beedogs 14y ago
Frankly I just copy-pasted the title from the post. And it looks like it's been 'corrected' now.

wallflower 14y ago

The danger of incurring so much sleep debt so quickly is that you will start paying compound interest on it immediately - in terms of ability to avoid accidents, struggling with mental coherence, handling relations with loved ones, out-of-it attitude, and dealing with unexpected situations gracefully.

jpatokal 14y ago

Jeebus, what a way to turn a molehill into a mountain. @22cans: In fact @pmolyneux BEGGED the team to go home but they wouldn't have any of it :) https://twitter.com/22Cans/status/268387590368030721 https://twitter.com/22Cans/status/268387590368030721 22 Cans is a tiny startup and they were expecting 10,000 concurrent users tops -- they're at 2 million and counting now, and they have to capitalize on that momentum and not piss off their obviously fanatical players. So they pulled an all-nighter to build and migrate from a single server to a clustered architecture, and hell yeah, if I was in that position I'd do that too. Seriously, is there another option? If you think the answer is "clock out at 5:01 PM and go home", you shouldn't be in a startup. Obviously pulling 36 hour shifts on a regular basis is not sustainable, but this is precisely the type of situation where it's warranted.

wallflower 14y ago
Keeping the servers up under heavy load is part of the startup archetype. You have to cross the Rubicon. > The servers melted. Systrom and Krieger worked a straight 24 hours to keep the app running—in that period 25,000 iPhone users downloaded the free service. http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagrams-kevin-systrom-the-stanford-millionaire-machine-strikes-again/3/ http://www.forbes.com/sites/stevenbertoni/2012/08/01/instagr...
fatbird 14y ago
The question is why did they assume 10k concurrent users, or plan for that to be the cap handled under the current architecture? Any competent project manager should have said "let's pretend we're massively successful... how will it work then?" and have some contingency plan in place so that they're not trying "experimental" fixes. The flaw here isn't the heroic efforts, it's that heroic efforts were needed. In the days of cloud based architectures, it's a lot easier to be ready for massive horizontal scaling than it used to be, so much so that being unready is malpractice.
- Negitivefrags 14y ago
  Assuming massive success is a stupid move. The majority of the time you are not that successful. The ground is littered by projects that spent vast sums of money (and more importantly time) designing for load they never needed to handle before release. In about a month my company will be releasing an online game. We think we will get about 15-20k concurrent users in the first few days, but we will be testing the realm for as high as 50k concurrent users using load testing bots. What will happen if we suddenly get 100k or 200k concurrent users? Even throwing more hardware at the problem might not solve it immediately if there is some new bottleneck we haven't found before. Testing the realm for more than 50k concurrents and then spending engineering effort required to make it work will consume time and effort that we can't afford to spend. All for an outcome that is unlikely to occur.

mikeklaas 14y ago

When we launched Zite, I (along with several other engineers), were awake for about the same amount of time keeping the servers alive. Sometimes a heroic emergency effort is required. As long as it is (extraordinarily) rare, it's not in itself sign of a dysfunctional organization.

hrktb 14y ago
> heroic emergency effort Completely agree that it'something required sometimes. But it's nothing heroic, it is a (temporary I hope) dysfunctionment, and shouldn't be put in midly positive light. Planning failed (not assuming you might have 2m users is sensible, but it's still a failure in it's way), and shit hit the fans. It's the kind of failure you might wish you had, but next time you should think hard about a way to deal with this kind of situation with minimal emegency, and no need for 'heroes'.

cwh 14y ago

as one of two 'server coders' that scaled Draw Something from 0 to 35 million DAU, a 36 hour day sounds nice.

miahi 14y ago
What was your motivation?
- cwh 14y ago
  quite simply, to see just how far we could take it.

ispivey 14y ago

For all of you crapping on Peter and his team: What if the employees were founders with equal shares of the company, and they launched a product and it fell over? Would you blame them for pulling an all-nighter to scale it up in the face of unexpected traffic? Should they give up at 10pm? Midnight? 5AM? If they're not equal partners, should they immediately give up? What if they own 10%? 1%? What if they don't have much on their resumes and are excited to make this a successful project, instead of a laughingstock that fell over in the face of scaling challenges? What if they care about each other and want to make this a success for their friends and coworkers? What's the function into which you plug all these variables and decide when to go home? It's not black-and-white, pun intended, and if you think it is you're being obtuse. I hope they feel happy and like they accomplished a lot in the face of long odds, and you should too.

Deestan 14y ago
I'd rather have a colleague come in to work drunk than pull an all-nighter to work on core functionality. Yes, we really need to get this thing improved as soon as possible, but having effectively brain-damaged people introduce bugs all night and then be utterly useless the next day is not helping. Take your financial lumps and get over it instead of burning out everyone and ruining the codebase. These kind of masochistic hero stories perpetuate the destructive belief that every hour your face is planted in front of the monitor is equally productive.
vidarh 14y ago
If someone I worked with tried pulling a 36 hour shift I'd physically throw them out of the building. In my mind, odds are 90%+ that'd they'd do more damage (more bugs, slowing down other people) than they'd do good in aggregate over that 36 window, and odds that they'd be more productive in a single 36 hour bout than doing shifts with proper sleep windows is pretty much 0. More than 14 hours or so in a single stretch is highly unlikely to be beneficial to the company/project, with rest in the middle. More than just being in line with research, this is in line with what I've seen over and over again in real projects, where best project during crunch time is generally had by ensuring people leave and rest. Sure, do longer hours for a short period. Sure, ask people to cut down on time off or alternate work/sleep shifts. But the moment your developers hits 18-20 hours, I could talk most of them into making horrifically bad decisions and have them think they were logically sound - your ability to reason goes out the window pretty quickly. I don't want people in that kind of state anywhere near projects I work on, because I've seen what they can, and will, do in the belief they're helping out. > What if they don't have much on their resumes and are excited to make this a successful project, instead of a laughingstock that fell over in the face of scaling challenges? What if they care about each other and want to make this a success for their friends and coworkers? Then learning to understand that there's nothing smart about pulling stunts like this unless you're doing menial work that you can still do reasonably reliably with next to no concentration or short term memory is even more important. > I hope they feel happy and like they accomplished a lot in the face of long odds, and you should too. No, I shouldn't, because if they feel that way they've learned the wrong lesson. They'd also most likely be wrong in terms of their relative productivity with more rest. Unless they were all on illegal stimulants, or his staff consists of mutants, it is simply not worth spending much time even considering the option that this might somehow have been the best way for them to spend their time.

gobengo 14y ago

No one should be pressured into this type of work, and it's important to always maintain that. However, if you haven't worked a 48hr+ stretch in a young startup, check your pace and/or commitment. It may be fine, just think about it.

seivan 14y ago

I'm a gamer and this is very subjective of course but Peter Molyneux hasn't actually done any good games. Overrated junk ~ and it's not his works, it is the engineers, designers, illustrators, sound engineers and actors who've done the game. He just makes enough noise to get attention and praise.

vacri 14y ago
His early games were interesting, fun, and original, pretty much all the Bullfrog catalog. I'm also not sure you can get away with saying it's not his work when he's the lead designer and executive producer on those titles. Yes, he didn't do it alone, but those roles stamp a considerable amount of personality onto the product. Lionhead produced nothing but crap.

politician 14y ago

Anyone else find Curiosity intriguing not for the game (which is arguably about vanity), but for the technical brain-trust that operating such a service creates? Here's hoping they all write books...

JanezStupar 14y ago

I did a 36 hour push once. I did it because there was really no other way to achieve the deadline, since there was about three days worth of bug squashing to reach a deadline and I only had two days left. I am proud that I managed to pull it off, however I tell it as a tale of caution not as a tale of inspiration. The work I did was objectively atrocious, by the 24 hour mark the simplest operation took me 5-10 minutes. My short term memory was completely gone by then and I had to rely on writing down each step of an operation as simple as refactoring a block of code into its own function, just to be able to pull it of in a "reasonable time frame". Thus I would call bullshit on anyone claiming that his team made "huge improvements" and that after such a push some piece of software "started behaving like it should". Unless that means that they introduced a plethora of bugs that they would be busy fixing for months from now.

babebridou 14y ago
The experience of going for an all-nighter is akin to watching a full season of 24 in one go, only you're the protagonist. It's memorable, and yes, especially if you manage to pull it off. I tell it as a tale of inspiration: do it, once. Whether it's good or bad for business/health/morale is a whole other story. Hint: it's not good.
- JonnieCache 14y ago
  One of the best things about going to university is you can get all of that stuff out of the way without hurting anyone else.
  - anthonyb 14y ago
    Well, until you graduate and continue your old habits...
- TeMPOraL 14y ago
  That's what Global Game Jam is for, it's a 48h competition for writing games. I attended two of those, and during those 48h took maybe total of 8h naps here and there. Amazing experience. But yes, really, at some points your brain just stops working.

MortenK 14y ago

To "just work harder", in this case 36 hours, is the easy decision to make when something goes wrong. It smells deliciously of heroics, bravado and top performer mentality. Most importantly, the soul drenching, exhausting late hours, gives the illusion that "we are doing all that is humanly possible to resolve the problem". How tough are we, right? But as the author points out, heaps of literature clearly show that "crunching", like it's affectionately called in the games industry, is contra-productive. One guy working 8 hours, going home for a good sleep, and coming back to work an additional 8 hours will vastly outperform the developer-turned-zombie that's been at it for 24 hours straight. At that point, short term memory is gone, stupid mistakes (blunders even) are rampant. Fixing simple issues takes extraordinary amounts of time, because the ability to focus is all but lost. You effectively have tunnel vision and your reactions and logical skills are reduced to that of a drunk. This is not anecdotal - there are an immense amounts of studies done on the subject. But time and again, the knowledge gets discarded because it's just easier to "work through the night" at a frantic pace, rather than take the necessary step back, let people rest and do the work in an orderly and planned manner. By burning the candle in both ends, at least no one can challenge your commitment to solving the problem. While it may sound counter-intuitive to go to bed while the house is on fire, it is, at least if 40 years of research is to be believed, actually the only correct thing to do in a software development crisis. Note this doesn't mean "stick 100% to 8 hours and always go home at 17.01", but it does mean if you are still there at 23.00 you are much better off going home, getting sleep and resuming work when well rested. The games industry seem to be especially vulnerable to this problem. It might be due to the general inexperience level caused by games development being a young mans game, or it might be due to a failure to understand that games, while glorified in the industry, is "just" software, and therefore also subject to all the established rules of software development. In any case, I agree with the author that Molyneux should know better than to tweet from the rooftops how many hours everybody is working, but I guess he is not thinking entirely straight after 36 hours of work.

jacquesm 14y ago
It depends on whether or not a fix is perceived to be further away than what you can accomplish in that one night. Being down for yet another business day when you don't have to be ('house on fire') is usually not an option. There is also a class of problems (chasing subtle bugs) that you're much better equipped to tackle when you come in fresh in the morning or when you're actually not looking directly for them, the flip side is there are those that it takes forever to get the mental model set up that maps the circumstances of the bug with enough resolution that you have a chance of cracking them that it pays off to just sit there until it's done, even if that means staying up well past your bed time. The rule seems to be 'go home if you can no longer positively contribute towards solving the problem', not just 'go home'.

michaelgrafl 14y ago

I'm a big proponent of working (i.e. laboring) as little as you possibly can in order to leave as much room as possible for convivial activities and personal growth. But I think it is disingenuous in this case to paint Molyneux as a slave driver. They were ill-prepared for the launch and stuck together to scoop the water out of the boat. It's probably a bad idea to do it for 36-hours straight, but it's understandable. I would certainly stick my teeth in there till the storm's passed. Especially in a company this small, where the line between collegiality and friendship are probably quite blurred. I'm much more interested in what the RSI is a result of. That shit needs to be taken care of.

borplk 14y ago

It's all our own fault. We glorify the noodle-eating couch-sleeping code-generating-machine hacker and let the corporations profit by exchanging our health and personal lives for stupid hacker-perks which costs the companies nothing. Woohoo! come hack for us, we give you a fancy chair and large monitors to stare at and even food! what else do you want out of your life? We need to see that we are being taken advantage of. We have raised expectations so high, that you can't land a decent job unless you have dedicated your evenings to patching bugs in open source projects for free. What do you think that says to an employer? Sorry to break it to you but it roughly goes like this, "The poor bastard has no life. When he goes home, he does the same crap for fun. He'll be more than happy if I pay him 2 peanuts" Have you asked yourself why other professionals don't have "open source" like we do? If programmers are expected to work for fun to develop your email client, web browser, server softwares etc... it is only fair if lawyers were also expected to contribute to cases for free. I'm too damn tired of this crap. We have put ourselves into an infinite loop of competition that only hurts ourselves and brings profit to the corporatoins. The guy next to you has to sacrifice his life because he needs to compete with you to stay relevant and you do for the same reason and so on... We are smarter than this. We deserver better than this. We need to bring this to a stop.

philbarr 14y ago
In other professional fields (like with lawyers), doing work for free is called "pro bono". I would think that if you're being interviewed for lawyer position they would ask you how much pro bono work you've done. So I disagree that doing this stuff for free is inherently bad, but I fully agree with the sentiments of the rest of your post. Particularly this: "come hack for us, we give you a fancy chair and large monitors to stare at and even food! what else do you want out of your life?" An old manager of mine was once on the phone to a recruitment agency trying to hire some developers complaining that they hadn't made progress, at one point he screamed, "it's software development, it's not fucking rocket science!" He did that in a room full of developers. For some reason many non-technical people get to become managers of technical people and have that kind of opinion. This then leads to the, "well if it doesn't work you can stay here all night until it does" mentality.

jheriko 14y ago

... and to think all they needed to do was hire one real programmer in the first place.

antidoh 14y ago

We hope none of those people are driving when they finally go home.

lyricalpolymath 14y ago

Does anyone know to what kind of books or reference is the author talking about when talking about the "15 years of free literature on how to run large-scale online projects" I've seen in the comments that damian2000 (http://news.ycombinator.com/item?id=4781706 http://news.ycombinator.com/item?id=4781706) has suggested "Peopleware: Productive Projects and Teams" by DeMarco & Lister. any other suggestions?