13 ms·
Reddit Is Down To One Developer
- jprobert 16y agoThis makes me shiver in my boots. I couldn't imagine having to run that site myself. Neil is inline for a big pay raise.
- redthrowaway 16y agoHe is, but he's only been there a couple months.
- thezilch 16y agoHe doesn't run it himself. Reddit has two sysops and Amazon AWS staff. I won't pass judgement on Neil's performance or compensation -- we shouldn't speculate, but it'd be surprising to see a developer of only 4-5 months get a big raise.
- sokoloff 16y agoSupply and demand is an important factor. If CN demands the services of a developer with experience in the reddit codebase at reddit and the worldwide supply of developers able and willing to supply that need is only Neil, it wouldn't be unreasonable business to give him a raise, short tenure being somewhat irrelevant if that's the situation.
- jedsmith 16y agoThat's a bit unfair to jedberg, who actually runs the site with his operations team of two (one of them a new hire). This article and the comments are putting far too much weight on the developer, as if Reddit is a site that's constantly iterating new features. That's not to downplay his contribution, but: * He was hired in November. * He works on code, and didn't get woken up during the last major downtime until the end (from their post mortem, presumably due to the DB eating itself). The word 'even' in "we *even* woke up spladug" is telling, that indicates to me that he doesn't wake up often. * Reddit is pretty much a done product, and the last features I can think of were meta (Gold comes to mind) He has a positive outlook[1], anyway, which the TechCrunch piece didn't seem to notice. Doing operations for a site the size of Reddit with a staff of 2, on AWS, is far more impressive to me. Keeping an operation like Reddit afloat in front of that many uniques must be a pretty significant challenge. There's not a lot of depth for on-call, so I feel for their sleep. On top of that, jedberg is the sole voice actively fighting for Reddit's reputation in public, since (let's be honest) the exodus looks a little suspect. Watching his comments[2] in the face of a perceivably sinking ship -- not to mention former employees who are saying quite a bit in front of an impressionable user base[3] -- is interesting, to say the least. They're definitely undergoing a challenge, and I'm interested to see how they come out of it. (Aside: Today has apparently been discuss-Reddit day, looking at my threads.) [1]: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l7i4m?context=1 http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d... [2]: http://www.reddit.com/r/reddit.com/comments/g6d7i/three_former_reddit_employees_ketralnis/c1lagp0 http://www.reddit.com/r/reddit.com/comments/g6d7i/three_form... [3]: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_down_for_6_of_the_last_24_hours/c1l6ykx http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...
- jarek 16y agoSorry! I accidentally downvoted while intending to upvote. Tricksy fingers...
- starnix17 16y agoWhat happens when the last one leaves? Is Conde just going to run off to Elance or something? I can't even find a jobs page on Reddit. At least it's open source.
- mcantelon 16y agoIronically, a month ago they were looking for devs and asking them to invest considerable time writing demo apps (front end engineers, for example, were tasked with writing an Javascript Reddit clone using HTML5 storage to keep track of votes, etc.) to prove their worth: http://www.reddit.com/r/blog/comments/fjgit/reddit_is_doubling_the_size_of_its_programming/ http://www.reddit.com/r/blog/comments/fjgit/reddit_is_doubli... My guess is this may have repelled some experienced devs that, otherwise, might have applied.
- JoachimSchipper 16y agoI thought their backend challenge looked reasonable, though: given a timestamped log file, return all entries since a certain time (efficiently!)
- redthrowaway 16y agoWith a couple of caveats, iirc. There were a few idiosyncrasies with the log files that needed to be dealt with.
- mattm 16y agoWow, sometimes companies are just out of their minds with the hoops they put up. I would definitely not invest that much time into a job application, especially before an initial interview to know if I even want to work there.
- mechanical_fish 16y agoThis application is the initial interview. You now know that you don't want to work there. Saved both you and Reddit a lot of time. From the application: Well, first off, it's supposed to be fun. If you don't see the joy in either of these puzzles, please don't apply. Seems fair to me. Set high standards. You can always lower them later.
- jarek 16y agoUnless they're working on new features which are needed tomorrow, one dev seems fine in the interim. The number of sysadmins is more important to keeping a site running, but of course that's not what's in TC's headline.
- deleted 16y ago[deleted]
- jedberg 16y agoBoth are important. A lot of work of a reddit programmer is finding bugs that come about because of the unreliable disks and also finding scaling inflection points and fixing them before they take the site down.
- thezilch 16y agoReddit's sysadmins neither have to maintain DC racking (nor travel), maintain hardware lifecycles, nor maintain vendor relationships. They certainly appear up to their teeth in working around AWS (EBS) deficiencies. They're not going to control a lot of their hardware, and most "fixing" of AWS pitfalls are going to come in the form of software. Now if we are talking devops, I can agree. Engineering teams this small need to wear many hats; I think jedberg has been said to have helped in development efforts, if my memory of past blogs serves right.
- jedberg 16y agoYes, I do dev too. I just haven't done it in a long time. A lot of our sysadmin time is spent tracking down bugs in the stuff we use (like Cassandra) and doing tuning and automation. Also, I do a lot of the other business stuff, so at best I'm a part time sysadmin right now. We do pretty much do "devops" though. A lot of the work of Neil, our programmer, is tracking down bugs due to data inconsistencies due to hardware problems.
- lowglow 16y agoThat's why they need someone from the start-up world -- both a systems guy/girl and a hacker guy/girl.
- citizenkeys 16y agoIn theory, if they can get by temporarily with just one engineer then the site was engineered well to begin with. There's some good management lessons from "The Mythical Man-Month" being applied here: http://en.wikipedia.org/wiki/The_Mythical_Man-Month http://en.wikipedia.org/wiki/The_Mythical_Man-Month
- redthrowaway 16y agoI wish the TC article had done more than merely mention that keltranis is "joining Alexis". Hipmunk now has both reddit cofounders, their first hire in keysersosa, and their most senior programmer in keltranis. I wouldn't be surprised to see jedberg join them before too long, and then hipmunk would have almost all of the old reddit team on board.
- jedberg 16y agoMe too. It is a total dis-service to hipmunk to not mention Chris or Steve. But for now I think their tech team is full, so they won't be needing my services. :)
- redthrowaway 16y agoDoes that mean you'd consider joining them if they asked you to? Also, given the concerns expressed at [1], once you get a healthy dev team would you guys look into maintaining redditOSS a little more thoroughly? I imagine personnel shortage has made that impossible to date, but if you had a sizeable group is that something you'd do? [1]http://www.deserettechnology.com/journal/reddit-the-open-source-software http://www.deserettechnology.com/journal/reddit-the-open-sou...
- jedberg 16y ago> once you get a healthy dev team would you guys look into maintaining redditOSS a little more thoroughly? I imagine personnel shortage has made that impossible to date, but if you had a sizeable group is that something you'd do? Yes, very much. We would like to make the open source software MUCH easier to run, so that it is easier for others to develop on. It will probably be one of the first projects for a new hire, because it will make it easier for THEM to develop too.
- rjhackin 16y agoThis is not good given that they just had a recent 6 hour downtime. Why would folks at Conde let this happen ?
- lowglow 16y agoI think a large company such as conde nast has a problem with engineering talent retention -- hence the move to AWS maybe?
- staunch 16y agoI know that at least one of the motivations for using AWS was that they didn't have to go through purchasing process of Conde Nast which was a huge roadblock. I think that's actually one of AWS' biggest strengths. Once you get a corporate card in the account it's easy for the devs to buy what they need.
- cookiecaper 16y agoConde really doesn't know what to do with reddit. They probably acquired it "knowing" they had to have something in "the social media space" (the acquisition predates the salesman's current buzzword of the hour, "the cloud") so they could compete in the "digital age" and all of that, but since they've acquired it my understanding is they've basically starved it and reddit has been running on fumes until they implemented Reddit Gold last summer. They're keeping it because they think they need something on this new-fangled internet stuff, but they don't what to do with it, and it's not generating much revenue, so they're just trying to leave things alone until they figure out something. Note that this is just speculation based on what I've read, I don't have any inside knowledge.
- JoachimSchipper 16y agoApparently, they have enough money but internal politics prevent them from hiring more people (it's a different budget...) Hence AWS, which isn't that cheap but does reduce personnel costs. See the big reddit thread on this topic, or HN's thread about that reddit thread.
- grncdr 16y agoOT but I found this amusing: competitor Digg (which has 42 employees and has raised over _$40_ in funding)
- swirlee 16y agoThat's as many as four tens!
- diptanu 16y agoHN seems to be doing fine with just one awesome developer :-)
- waterlesscloud 16y agoYeah, but look who he is.
- alnayyir 16y agoNot accurate.
- cookiecaper 16y agoHN has been struggling to keep up with load recently. Remember the pagination of comments a few weeks back needed to keep the server from falling over? Yeah. I think news.arc/HN could really do with some extra scalability work; it still slows down significantly at peak times.
- bluesnowmonkey 16y agoServer? There's one server?
- tomjen3 16y agoYes one server, but that isn't entirely fair because pg has Robert Morris to tweak it. And he is a kick ass sysadmin.
- phillco 16y agoInteresting. This sure is boosting their pageviews/developer ratio.
- redthrowaway 16y agoWhich is currently sitting at 1.1B : 1.
- gersh 16y agoHow many developers does hacker news have?
- shii 16y agortm and pg, plus a few mods as far as I can tell.
- cookiecaper 16y agoHN is a much, much simpler program, both in terms of features and internals. Internally, Reddit is a gigantic mess.
- adrianN 16y agoIt also doesn't have a gazillion pageviews a month and daily downtime.
- shii 16y agoIsn't HN basically news.arc[1] plus some custom mods and caching with content/stories saved as flatfiles? Reedit has many more sophisticated load issues to about. The two aren't really comparable it seems. [1]: https://github.com/nex3/arc/raw/042990a55d8c880a0a88476eb52b169e47f35d32/news.arc https://github.com/nex3/arc/raw/042990a55d8c880a0a88476eb52b...
- cookiecaper 16y agoYeah, that was my point. Reddit is much more complex and much messier.
- nkassis 16y agoThis is the second TC article today I see where they claim news site like digg or reddit are in competition with twitter. I don't think they understand that it's two very different type of stuff that is complementary. Or they are paid to say twitter in every article.
- colkassad 16y agoTech journalism has a hard on for Twitter and it's not so surprising why. Twitter is all about self promoters. I see people using reddit all the time...I have never seen Twitter in the wild.
- anacrolix 16y agoScrap it. Rewrite. It can't be that complicated and if it is, it needs to go.
- marcc 16y agoRarely a good answer. All of that complexity wasn't added for fun. Making a comment like this really shows ignorance in software development. While we all want to scrap and rewrite some projects, most of us know that it isn't as easy as it looks and you will be spending your time on the wrong thing. Also, I don't have any idea what to make of your comment "...and if it is, it needs to go.". Are you just trolling or do you really suggest that they shut the site down because it's complex?
- devinrhode2 16y agototally agree.
- grammr 16y agoI think he's suggesting that if any part of the reddit codebase is too complex, then it needs to be ripped out.
- PonyGumbo 16y agoA sizable portion of reddit's recent success is owed to Digg's decision to do just this.
- pdaviesa 16y agoIt appears that Reddit's days are numbered. Sure they may be able to get by with one developer for a while, the product is fairly mature and content is driven by the user base. But how long can a website remain stagnant, with no innovation until the user base starts to migrate to the next best thing? Also, being owned by a large, old media company doesn't necessarily give one much confidence that Reddit is backed by visionaries who are willing to push the envelope and take risks.
- Raphael 16y agoAnd what is the next best thing in your opinion? It's one of the few places that does nested comments properly and the lightweight up/down voting always keeps it interesting. The only thing I can see replacing it is an outright clone, but no one's stepped up and done it in all the years Reddit has been around.
- officemonkey 16y ago>And what is the next best thing in your opinion? I'll know it when I see it. I was a daily user of Slashdot from 2001 to 2006 and then I found Reddit. I completely abandoned Slashdot less than a year later. What could beat reddit? Well, if there was a facebook-y site which allowed me to associate with people who had my interests (rather than family, friends, and people I went to college/high school), I'd be down with that. (I love my mom, but I don't think she cares about Minecraft, old-school RPGs, French pop songs, Asa Akira, and Jules Verne novels.) Twitter is close, but the character limit, url shorteners, and "twitter-speak" (intentional bad spelling/grammar in order to "get under the limit") just make my teeth itch.
- trotsky 16y agoI took a job some time ago (back around when the internet was morphing from uucp to ppp) as a consultant to an "independent agency of the United States government" which seems to enjoy placing big blue boxes all over the landscape. It was my first real job outside of the university system, so it is safe to say that I was rather green. This job basically involved being an apprentice to someone who oversaw the code supplied by vendors that sorted a tremendous amount of non machine readable mail, i.e. it assisted humans using chord keyboards sorting mail with pneumatic arms and big chains and slots. Back then, a fair amount of mail traveled this way - stuff your grandmother wrote that couldn't be OCR'd, postcards, etc. The whole apparatus was similar in size to a semi trailer [1]. Each station dealt with a piece of mail every second (give or take), with two fifteen minute breaks and a half hour for lunch. This was around the time of the postal shootings [2]. Two weeks into the job, my boss quit. Just never showed up again. Not exactly sure what happened. Apparently, I had become the most knowledgeable person in the organization on this software which was currently being rolled out widely. Unsurprisingly, there were some serious code quality issues. This was QNX real time sorta unix in C and ASM with RTOS daughter boards. The vendor's code jockeys, apparently, had become unavailable. Main takeaways: Systems seem to keep working despite everything looking very fragile. Read tons of code. If it's working, don't poke it. A year later I was able to make some form of industrial dance music by coordinating the openings and closings of the mail slots and the chain with the software. Bad situations can sometimes lead to greater clarity and new opportunities. [1] http://www.usps.com/postalhistory/images/photogallery/equipmentpix/APPP136_fuji280cropped.jpg http://www.usps.com/postalhistory/images/photogallery/equipm... [2] https://secure.wikimedia.org/wikipedia/en/wiki/Going_postal https://secure.wikimedia.org/wikipedia/en/wiki/Going_postal
- bhrgunatha 16y agoIt seems like a common situation in the tech world for those with a little capability to deal with some legacy system that you somehow get lumped with or become the "expert". Maybe it's inevitable, people do get bored with doing the same things all the time. I'd love to hear some of the music that came out of your experience - anywhere to listen to it?
- heed 16y agoHmm.. Would now be a good time to build a new reddit?
- aberkowitz 16y agoBuilding a new reddit is easy - building a community, and scaling to support 1.1 billion views a month is not.
- fedd 16y agomy bet is that there is some 'technical debt', so the devs leave when said 'business is over your tech details'! :)
- dhughes 16y agoWouldn't it be surreal if Kevin Rose went to Reddit?
- jrockway 16y agoThis is what happens when your primary demographic is people with no jobs and no money -- no ad revenue because nobody clicks the ads, and no possibility of selling subscriptions. Compare Reddit's target demographic with Bloomberg's target demographic. I bet Bloomberg has more than one developer. (Incidentally, what a scam. Bloomberg terminals cost some insane monthly fee, and they have ads! And all I've ever seen anyone do with them is check sports scores and read celebrity gossip. Google should have put them out of business years ago.)
- shalmanese 16y agoBloomberg's annual revenue is about 25% of the entire journalism industry's.
- jedberg 16y agoIt's not a money problem. We've got plenty of money. We just weren't allowed to spend it on developers until recently. Now we have a bunch of open developer positions that we are in the middle of hiring for, and will probably have more soon.
- quicksilver03 16y agoI'm sorry, but if you have money and you're not allowed to spend it on what you do need the most, then actually you don't have money and you've definitely got a money problem.
- dabeeeenster 16y agoIf you ever get the chance to sit on the shoulder of a Bloomberg power user I would recommend doing so. In the right hands they are mind-bendingly powerful. Also, a large part of the cost of the terminal is the data feeds.
- barista 16y agoDoes this mean that the social new bubble has finaly burst with reddit down to one person and digg winding down?