10 ms·

As Digg Struggles, VP Of Engineering Is Shown The Door

143 points by icey 16y ago

megaman821 16y ago
This article is a fine example of the poor logic that pisses me off nearly every day. Just because Digg v4, which heavily uses Cassandra, can't be used as a Cassandra success story does not automatically mean you can assume the opposite, that it is a Cassandra failure story. There is no indication why Digg has been down so often and there is really no conclusions to draw as of yet on the technology they use.
- fleitz 16y ago
  The article basically comes right out and says it. You just need to read between the lines a little bit. From the article: Quinn was the main champion of moving over to Cassandra, say our sources. Now the site is taking a huge hit, at least in the short term, because of that decision and/or how it was implemented, and Quinn is paying for it with his job. It's always a toss up between whether it was implemented correctly or not. The correct course of action of course would have been to slowly move the site over to the new technology piece by piece rather than a wholesale switchover. The risk is in the migration strategy not the technology picked. They could have been equally stupid switching over to a new architecture with mysql.
  - jshen 16y ago
    I'm not sure I'd use tech crunch as a reliable source.
rbranson 16y ago
Queue NoACID haters declaring this the final nail in the coffin.
- dangrossman 16y ago
  This isn't Digg.
heffay 16y ago
I'm really interested to see what comes of this and what went wrong. It sounds like (from reading his blog) they were making a lot of customizations to Cassandra?
- pquerna 16y ago
  They have made many feature additions to Cassandra. They haven't said much about the details of why they are having trouble. It could be a core Cassandra problem, something they added, or completely unrelated to Cassandra; But the internet doesn't care. It's drama at its finest.
  - ergo98 16y ago
    "It could be a core Cassandra problem, something they added, or completely unrelated to Cassandra; But the internet doesn't care. It's drama at its finest" Digg made a big deal about their move to Cassandra (just as Digg's move to Cassandra was used to legitimize Cassandra, and by correlation NoSQL, among a wide range of zealots), going back over a year. The thing about talking big like that is that it often comes back to bite you in the ass if things don't go well. If Digg quietly released a new version that worked more reliably and provided a better experience, they would have been in a perfect position to pontificate on technology.
madridorama 16y ago
Digg turned into nothing but a paid distribution and SEO engine anyway
- leon_ 16y ago
  when was it something different?
- blasdel 16y ago
  Digg was something else before?
  - josefresco 16y ago
    Yeah originally it was Slashdot but better, more current and user powered (which was a new concept in the mid 2000's). What it turned into was a traffic generator for a small group of power users who fed the community the lowest common denominator of content.
    - jshen 16y ago
      User powered will always turn into lowest common denominator if the user base grows beyond a niche
agentultra 16y ago
This man's reputation is on the line. I hope they release more details on what exactly it is that is causing the problem. As it stands he appears to be an unfortunate scapegoat.
- heffay 16y ago
  Hopefully he creates a blog post explaining the disaster. He seems to regularly update that site
leon_ 16y ago
digg was dead to me when they introduced their stupid "digg bar" that framed every website.
mikeryan 16y ago
I actually wondered if they were running into issues with Cassandra. I'm not a NoSQL hater - but its still pretty bleeding edge and it always seemed like making it your core db was super risky. arg - really want more insight, maybe Quinn will elaborate now that he's gone.
- leon_ 16y ago
  I've seen the "digg nation" video where they elaborate the whole cassandra thing; Kevin Rose said they had data corruption bugs a few days before deployment of the new system. I don't know but it sounds pretty insane to me to take an unproven, untested beta technology and make it the backbone of your site.
narrator 16y ago
Reddit runs on Cassandra + PostgreSQL. They use Cassandra as a key/value store and not as the primary database though.
- cookiecaper 16y ago
  They are planning to transition to Cassandra as the primary database. PostgreSQL is not used in a relational way on reddit -- it is used as a makeshift k-v store. You can take a look at this yourself as reddit is fully open-source: http://github.com/reddit http://github.com/reddit .
  - zzzeek 16y ago
    > They are planning to transition to Cassandra as the primary database. do you have a source for that ? > it is used as a makeshift k-v store That is very true, they should switch. It just seemed like they had no plan to do so.
    - bbatsell 16y ago
      I could swear an admin said it with some pretty fair definitiveness; I thought I remembered that it was ketralnis, but this was the best I could find from him, and it's not nearly as definite as my memory says. http://www.reddit.com/r/programming/comments/bcqhi/reddits_now_running_on_cassandra/c0m3rs9 http://www.reddit.com/r/programming/comments/bcqhi/reddits_n...
    - ketralnis 16y ago
      > > They are planning to transition to Cassandra as the primary database. > do you have a source for that ? Yes. Me.
  - pavs 16y ago
    According to reddit admins some of the recent downtimes in the last few months were due to cassandra and they were having some bad performance and stability issues. I can look it up if anyone is interested but it will be a bit of a work, since they stated it (several times) in the comment section and perhaps once on a blog post too.
btipling 16y ago
For some reason this reminds me of the "no one has ever been fired for choosing IBM." slogan. I guess maybe there is some truth to it. Seems like new tech needs to start slow on big places, only startups really have the freedom to risk it all.
- aaronblohowiak 16y ago
  your post implies digg isn't a startup.
  - soofaloofa 16y ago
    Digg isn't a startup. It started in 2004 and has been mature for about 5 years. They are a business now. And a big one at that.
    - points 16y ago
      For the 40m investment, I wonder how much profit they make. They seem to have a very large staff.
      - potatolicious 16y ago
        I have questions about what they're doing with all that staff, especially now that it's become apparent that Reddit is doing similar (if not larger) numbers with a handful of guys (literally, countable on one hand). Evidently Digg is better at the monetization game than Reddit has been, so more sales staff I can understand... but devs?
        Devilboy 16y ago
        I can understand that Digg needs more people than Reddit (even though their traffic numbers are about the same) but TEN TIMES more people? That's just not sensible. I'm sure Digg can make a nice profit without further growth if they just got rid of about half their staff.
cookiecaper 16y ago
Is there any reason that Cassandra is the focus of this article? It is really silly and irresponsible to peg a nascent project like that without any reasoning or sources. I'm sure something changed besides just a Cassandra rollout, and wasn't Digg using it on v3 too? I think Cassandra is pretty well tested. There have been lots of super-large-scale deployments. It just seems lame to blame it on that, but I guess maybe their anonymous sources inside Digg revealed it? But then we'd hope they'd know if the problem was with the datastore or the implementation.
- riffraff 16y ago
  as far as I remember: yes they were using it on v3 too, for some things
  - sfrench 16y ago
    Cassandra was used for a single feature on v3 called "Green Badges". I believe there is a post on the digg blog about the project, in particular.
  - wsongk 16y ago
    About a year ago, they were using it to store a list of friends of viewing user who dugg the viewing news. original blog post: http://about.digg.com/blog/looking-future-cassandra http://about.digg.com/blog/looking-future-cassandra Treads on HN: http://news.ycombinator.com/item?id=813528 http://news.ycombinator.com/item?id=813528
- sliverstorm 16y ago
  Placing all the blame on Cassandra helps Digg justify letting go the VP, since he was allegedly the guy who pushed it.
  - TheCondor 16y ago
    So what's the digg story here? I get that some VP suggested a new buzzwordy technology, they gave him enough rope to hang himself and he did and left the company with a broken pile of crap. It could happen, if you have a healthy company you give trust to people. That it got this far doesn't speak well of the rest of management. It doesn't speak really well of the rest of the team either. Shouldn't there have been some circuit breakers or something? Digg isn't a poor, bring-your-own-laptop startup. They've got resources, they've had substantial investment. They can afford to build and test software and I know of no real marketing reason they had to push something untested out. Rose could go out and say it wasn't done, it's going to take more time. How does Rose keep his job? Wasn't he this VP's boss? And it's single technology? That couldn't have been vetted and tested independently of all of digg? Really? And MongoDB, or hadoop, or one of the dozens of other nosqls wouldn't work, either? You do what you have to do and there is never a 'truth' with VPs and CxOs are canned, but it all doesn't float with me, just looks like another over valued and under talented company, got lucky and the blind squirrel found the nut, there isn't gonna be a second act. Maybe it just seems low brow to me, name and finger the guy, blame the opensource tool you use, never explain or elaborate why you launched anyways when you were fixing bugs in the tool at the 11th hour.
houseabsolute 16y ago
I wonder what issues they are running into that a dark launch would not have found. As I have discovered painfully in my own projects, making big changes without a rollback plan is usually a bad idea, and it sounds like this is no exception.
- gphil 16y ago
  Yeah, it sounds like the launch process is more at fault than the technology for this if there's really no way to rollback to stability.
pclark 16y ago
Unless Digg cannot roll out new features due to Cassandra (eg: a) Cassandra woes taking all dev team time, or b) tech limitations prevent features) it seems highly unlikely that cassandra is the reason why Digg 4 is grating with users initially. It's a change of features/product rather than technology that is the problem.
- enjo 16y ago
  I think the folks at Digg where prepared for the heartburn associated with the content changes. They weren't ready for some very serious issues with the implementation. Digg has definitely seen a lot of downtime over the last week or two.
- mikeryan 16y ago
  Digg has had some significant downtime during this transition (Can't blame Cassandra - but can't clear it either). Issues with product features are one thing - but nothing probably got more people jumping to a different ship (reddit) then not being able to find a good lolcat when you're bored, and your favorite news aggregator is down. Hmm - I wonder if fark has seen the same uptick as reddit has.
deleted 16y ago
[deleted]
parfe 16y ago
I have to say I like the new Digg. Sad this guy is getting his career stomped on over it. Lot of people complain that the new site lets big sites submit their own content automatically, but how was it any different with a middle-man user submitting it himself?
- gamble 16y ago
  The real problem is that v4 removed the illusion that average users had an influence on the rankings, without altering the fact that they don't.
  - hop 16y ago
    Yeah, I may be blind, but how do you even get to the upcoming stories?
    - gamble 16y ago
      You can't right now. It's on the list of things they haven't finished reimplementing from v3.
amix 16y ago
Like Joel Spolsky said it: "They followed the single worst strategic mistake that any software company can make: They decided to rewrite everything from scratch" - http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html I think rewriting from scratch is the core of their problem and not really Cassandra. Gradually going over to Cassandra would have been a much better idea.
- gaiusparx 16y ago
  The other problem is they have been hyping V4 for a long time by making beta testing invite only and hard to get. It would have been better if they can have a public V4 beta running in parallel with V3 since it is a rewrite.
  - dawie 16y ago
    Even better: Have a link that says: "Try the new Beta" at the top like gmail and google docs, with a "Go back to Old Version". Implementing this should not be to hard and it allows them to get some feedback.
    - bryanh 16y ago
      As a developer myself, I love hearing: > Implementing this should not be to hard...
    - pilif 16y ago
      Considering the completely different backends of v3 and v4, I think, this would have been incredibly hard to implement. At least if changes in one system should end up in the other. In this case you would not just have to write scrips to one-time migrate all needed data from v3 mysql to v4 Cassandra. No. You would have to build a mechanism that doesn't just do it in both directions, it would also have to work at near-realtime speed. If you then need consistency of the transferred data, this quickly gets impossible. Try finding a way to ensure consistency between these two completely different architectures. In the end, most of v3 would have needed to be rewritten for a parallel use to be possible, at which point you don't really gain much. "not hard to implement" - sigh Disclaimer: I don't work at digg and I don't know more about their backend than the rest of the public. I did however just get around doing something like what you describe and there it wad "just" a different schema on the same database backend and even just that would have been hell
mcantelon 16y ago
Reddit went through similar issues a few months back (downtime, slowness, etc.), but they overcame these issues without turfing people. My guess is Digg pushed the engineering VP out to make the investors happy rather than to actually move forward.
- pavs 16y ago
  How does it make investors happy? Why would investors care what kind of database you use?
  - fookyong 16y ago
    it's an attempt to control the PR message. by doing this, they are laying the blame on one person so the media can stop hating Digg and start hating the ex-VP of Engineering who "killed Digg". of course, whether he was actually responsible in some way is something we may never know. for all we know he may have been completely against releasing v4 but was vetoed by Rose et al. or on the other hand he may have overpromised and under-delivered, putting the company in jeopardy, in which case he deserves to be let go. it's all speculation until we hear an official comment from either side.
  - code_duck 16y ago
    The typical investor doesn't take the time to really understand the companies or technical issues, unfortunately. They see 'there was a problem, management fired the guy who was at fault' and are happy.
zephjc 16y ago
Digg v4 has added back the features I liked after taking them away - mostly the Upcoming section, which tends to have interesting things that never make it to the front page, and the setting that allows you to default to Top Mews, instead of things I have already read ('My News'). For most users, change in features is what drove them away, not necessarily the spotty QOS (though that was pretty bad for a while).
mminolt 16y ago
Ok Digg is having a hard time. I feel like I witness a strange blood thirsty tone in some of the comments though. I wish Digg the best and I hope they can prove all the skeptics wrong. It's going to be tough though!
nl 16y ago
Digg's put out a lot of information about their Cassandra implementation, eg https://nosqleast.com/2009/slides/sarkissian-cassandra.pdf https://nosqleast.com/2009/slides/sarkissian-cassandra.pdf (from November 2009). Back then they seemed to have a rather sensible migration strategy (ie, basically running the new Cassandra back-end in parallel with the MySQL backend). It seems to me that it was the v4 upgrade that broke, not Cassandra alone. It's possible their frustrations with Cassandra were more long term, and the fact the v4 upgrade didn't go well was the last straw. For example, Digg has done a lot of work on Cassandra internals and tools. If you are using a new, open source product you kind of expect that, but it's possible the expense of that didn't seem like good value once v4 started to get into trouble,
c00p3r 16y ago
Let me guess - the problem is about the difference between theory and practice. In theory, Java is great and Cassandra is great. In practice - Java under a heavy load is a disaster, because it was never designed for it, and Cassandra is a just a hype and propaganda. Face the reality - it doesn't work in production as it supposed to - as a primary storage engine. People at the Digg aren't amateur idiots, so I think they do everything as it described in docs, but the damn thing just doesn't work.
- nl 16y ago
  > In practice - Java under a heavy load is a disaster Google's heavy use of serverside Java would indicate otherwise. > because it was never designed for it Yes it was. Java has a lot of problems, but one thing that isn't a problem is heavy load. > Cassandra is a just a hype and propaganda Facebook seems to be using it ok.
  - c00p3r 16y ago
    Google probably use server side Java without a really heavy load, and just add more servers to keep the load low. System which was designed for being isolated from an OS (leave alone hardware) will have a bottleneck exactly in this level. Facebook doesn't use it as a primary storage with a high load.
    - cameronh90 16y ago
      What would you prefer to write web applications in? C++? Most languages used in web programming are divorced from the hardware, and either use a virtual machine (e.g. Java, .NET) or an interpreter (PHP, Python, Perl, Ruby). Virtually no-one uses a low level language for web programming, and the benchmarks say that the virtual machines are faster than interpreters. Java is very fast and efficient once it's running, but initial start-up time is often slower than interpreted languages due to the JIT compilation, but servers rarely "start up". If you're really hitting a barrier with Java's performance, just as with other non-native languages, you can write the performance critical section in C. http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php http://shootout.alioth.debian.org/u32q/which-programming-lan...
Dervish 16y ago
Digg 4.x is like Star Wars Galaxies NGE.

143 points by icey 16y ago

This article is a fine example of the poor logic that pisses me off nearly every day. Just because Digg v4, which heavily uses Cassandra, can't be used as a Cassandra success story does not automatically mean you can assume the opposite, that it is a Cassandra failure story. There is no indication why Digg has been down so often and there is really no conclusions to draw as of yet on the technology they use.

fleitz 16y ago
The article basically comes right out and says it. You just need to read between the lines a little bit. From the article: Quinn was the main champion of moving over to Cassandra, say our sources. Now the site is taking a huge hit, at least in the short term, because of that decision and/or how it was implemented, and Quinn is paying for it with his job. It's always a toss up between whether it was implemented correctly or not. The correct course of action of course would have been to slowly move the site over to the new technology piece by piece rather than a wholesale switchover. The risk is in the migration strategy not the technology picked. They could have been equally stupid switching over to a new architecture with mysql.
- jshen 16y ago
  I'm not sure I'd use tech crunch as a reliable source.

rbranson 16y ago

Queue NoACID haters declaring this the final nail in the coffin.

dangrossman 16y ago
This isn't Digg.

heffay 16y ago

I'm really interested to see what comes of this and what went wrong. It sounds like (from reading his blog) they were making a lot of customizations to Cassandra?

pquerna 16y ago
They have made many feature additions to Cassandra. They haven't said much about the details of why they are having trouble. It could be a core Cassandra problem, something they added, or completely unrelated to Cassandra; But the internet doesn't care. It's drama at its finest.
- ergo98 16y ago
  "It could be a core Cassandra problem, something they added, or completely unrelated to Cassandra; But the internet doesn't care. It's drama at its finest" Digg made a big deal about their move to Cassandra (just as Digg's move to Cassandra was used to legitimize Cassandra, and by correlation NoSQL, among a wide range of zealots), going back over a year. The thing about talking big like that is that it often comes back to bite you in the ass if things don't go well. If Digg quietly released a new version that worked more reliably and provided a better experience, they would have been in a perfect position to pontificate on technology.

madridorama 16y ago

Digg turned into nothing but a paid distribution and SEO engine anyway

leon_ 16y ago
when was it something different?
blasdel 16y ago
Digg was something else before?
- josefresco 16y ago
  Yeah originally it was Slashdot but better, more current and user powered (which was a new concept in the mid 2000's). What it turned into was a traffic generator for a small group of power users who fed the community the lowest common denominator of content.
  - jshen 16y ago
    User powered will always turn into lowest common denominator if the user base grows beyond a niche

agentultra 16y ago

This man's reputation is on the line. I hope they release more details on what exactly it is that is causing the problem. As it stands he appears to be an unfortunate scapegoat.

heffay 16y ago
Hopefully he creates a blog post explaining the disaster. He seems to regularly update that site

leon_ 16y ago

digg was dead to me when they introduced their stupid "digg bar" that framed every website.

mikeryan 16y ago

I actually wondered if they were running into issues with Cassandra. I'm not a NoSQL hater - but its still pretty bleeding edge and it always seemed like making it your core db was super risky. arg - really want more insight, maybe Quinn will elaborate now that he's gone.

leon_ 16y ago
I've seen the "digg nation" video where they elaborate the whole cassandra thing; Kevin Rose said they had data corruption bugs a few days before deployment of the new system. I don't know but it sounds pretty insane to me to take an unproven, untested beta technology and make it the backbone of your site.

narrator 16y ago

Reddit runs on Cassandra + PostgreSQL. They use Cassandra as a key/value store and not as the primary database though.

cookiecaper 16y ago
They are planning to transition to Cassandra as the primary database. PostgreSQL is not used in a relational way on reddit -- it is used as a makeshift k-v store. You can take a look at this yourself as reddit is fully open-source: http://github.com/reddit http://github.com/reddit .
- zzzeek 16y ago
  > They are planning to transition to Cassandra as the primary database. do you have a source for that ? > it is used as a makeshift k-v store That is very true, they should switch. It just seemed like they had no plan to do so.
  - bbatsell 16y ago
    I could swear an admin said it with some pretty fair definitiveness; I thought I remembered that it was ketralnis, but this was the best I could find from him, and it's not nearly as definite as my memory says. http://www.reddit.com/r/programming/comments/bcqhi/reddits_now_running_on_cassandra/c0m3rs9 http://www.reddit.com/r/programming/comments/bcqhi/reddits_n...
  - ketralnis 16y ago
    > > They are planning to transition to Cassandra as the primary database. > do you have a source for that ? Yes. Me.
- pavs 16y ago
  According to reddit admins some of the recent downtimes in the last few months were due to cassandra and they were having some bad performance and stability issues. I can look it up if anyone is interested but it will be a bit of a work, since they stated it (several times) in the comment section and perhaps once on a blog post too.

btipling 16y ago

For some reason this reminds me of the "no one has ever been fired for choosing IBM." slogan. I guess maybe there is some truth to it. Seems like new tech needs to start slow on big places, only startups really have the freedom to risk it all.

aaronblohowiak 16y ago
your post implies digg isn't a startup.
- soofaloofa 16y ago
  Digg isn't a startup. It started in 2004 and has been mature for about 5 years. They are a business now. And a big one at that.
  - points 16y ago
    For the 40m investment, I wonder how much profit they make. They seem to have a very large staff.
    - potatolicious 16y ago
      I have questions about what they're doing with all that staff, especially now that it's become apparent that Reddit is doing similar (if not larger) numbers with a handful of guys (literally, countable on one hand). Evidently Digg is better at the monetization game than Reddit has been, so more sales staff I can understand... but devs?
      - Devilboy 16y ago
        I can understand that Digg needs more people than Reddit (even though their traffic numbers are about the same) but TEN TIMES more people? That's just not sensible. I'm sure Digg can make a nice profit without further growth if they just got rid of about half their staff.

cookiecaper 16y ago

Is there any reason that Cassandra is the focus of this article? It is really silly and irresponsible to peg a nascent project like that without any reasoning or sources. I'm sure something changed besides just a Cassandra rollout, and wasn't Digg using it on v3 too? I think Cassandra is pretty well tested. There have been lots of super-large-scale deployments. It just seems lame to blame it on that, but I guess maybe their anonymous sources inside Digg revealed it? But then we'd hope they'd know if the problem was with the datastore or the implementation.

riffraff 16y ago
as far as I remember: yes they were using it on v3 too, for some things
- sfrench 16y ago
  Cassandra was used for a single feature on v3 called "Green Badges". I believe there is a post on the digg blog about the project, in particular.
- wsongk 16y ago
  About a year ago, they were using it to store a list of friends of viewing user who dugg the viewing news. original blog post: http://about.digg.com/blog/looking-future-cassandra http://about.digg.com/blog/looking-future-cassandra Treads on HN: http://news.ycombinator.com/item?id=813528 http://news.ycombinator.com/item?id=813528
sliverstorm 16y ago
Placing all the blame on Cassandra helps Digg justify letting go the VP, since he was allegedly the guy who pushed it.
- TheCondor 16y ago
  So what's the digg story here? I get that some VP suggested a new buzzwordy technology, they gave him enough rope to hang himself and he did and left the company with a broken pile of crap. It could happen, if you have a healthy company you give trust to people. That it got this far doesn't speak well of the rest of management. It doesn't speak really well of the rest of the team either. Shouldn't there have been some circuit breakers or something? Digg isn't a poor, bring-your-own-laptop startup. They've got resources, they've had substantial investment. They can afford to build and test software and I know of no real marketing reason they had to push something untested out. Rose could go out and say it wasn't done, it's going to take more time. How does Rose keep his job? Wasn't he this VP's boss? And it's single technology? That couldn't have been vetted and tested independently of all of digg? Really? And MongoDB, or hadoop, or one of the dozens of other nosqls wouldn't work, either? You do what you have to do and there is never a 'truth' with VPs and CxOs are canned, but it all doesn't float with me, just looks like another over valued and under talented company, got lucky and the blind squirrel found the nut, there isn't gonna be a second act. Maybe it just seems low brow to me, name and finger the guy, blame the opensource tool you use, never explain or elaborate why you launched anyways when you were fixing bugs in the tool at the 11th hour.

houseabsolute 16y ago

I wonder what issues they are running into that a dark launch would not have found. As I have discovered painfully in my own projects, making big changes without a rollback plan is usually a bad idea, and it sounds like this is no exception.

gphil 16y ago
Yeah, it sounds like the launch process is more at fault than the technology for this if there's really no way to rollback to stability.

pclark 16y ago

Unless Digg cannot roll out new features due to Cassandra (eg: a) Cassandra woes taking all dev team time, or b) tech limitations prevent features) it seems highly unlikely that cassandra is the reason why Digg 4 is grating with users initially. It's a change of features/product rather than technology that is the problem.

enjo 16y ago
I think the folks at Digg where prepared for the heartburn associated with the content changes. They weren't ready for some very serious issues with the implementation. Digg has definitely seen a lot of downtime over the last week or two.
mikeryan 16y ago
Digg has had some significant downtime during this transition (Can't blame Cassandra - but can't clear it either). Issues with product features are one thing - but nothing probably got more people jumping to a different ship (reddit) then not being able to find a good lolcat when you're bored, and your favorite news aggregator is down. Hmm - I wonder if fark has seen the same uptick as reddit has.

deleted 16y ago

[deleted]

parfe 16y ago

I have to say I like the new Digg. Sad this guy is getting his career stomped on over it. Lot of people complain that the new site lets big sites submit their own content automatically, but how was it any different with a middle-man user submitting it himself?

gamble 16y ago
The real problem is that v4 removed the illusion that average users had an influence on the rankings, without altering the fact that they don't.
- hop 16y ago
  Yeah, I may be blind, but how do you even get to the upcoming stories?
  - gamble 16y ago
    You can't right now. It's on the list of things they haven't finished reimplementing from v3.

amix 16y ago

Like Joel Spolsky said it: "They followed the single worst strategic mistake that any software company can make: They decided to rewrite everything from scratch" - http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html I think rewriting from scratch is the core of their problem and not really Cassandra. Gradually going over to Cassandra would have been a much better idea.

gaiusparx 16y ago
The other problem is they have been hyping V4 for a long time by making beta testing invite only and hard to get. It would have been better if they can have a public V4 beta running in parallel with V3 since it is a rewrite.
- dawie 16y ago
  Even better: Have a link that says: "Try the new Beta" at the top like gmail and google docs, with a "Go back to Old Version". Implementing this should not be to hard and it allows them to get some feedback.
  - bryanh 16y ago
    As a developer myself, I love hearing: > Implementing this should not be to hard...
  - pilif 16y ago
    Considering the completely different backends of v3 and v4, I think, this would have been incredibly hard to implement. At least if changes in one system should end up in the other. In this case you would not just have to write scrips to one-time migrate all needed data from v3 mysql to v4 Cassandra. No. You would have to build a mechanism that doesn't just do it in both directions, it would also have to work at near-realtime speed. If you then need consistency of the transferred data, this quickly gets impossible. Try finding a way to ensure consistency between these two completely different architectures. In the end, most of v3 would have needed to be rewritten for a parallel use to be possible, at which point you don't really gain much. "not hard to implement" - sigh Disclaimer: I don't work at digg and I don't know more about their backend than the rest of the public. I did however just get around doing something like what you describe and there it wad "just" a different schema on the same database backend and even just that would have been hell

mcantelon 16y ago

Reddit went through similar issues a few months back (downtime, slowness, etc.), but they overcame these issues without turfing people. My guess is Digg pushed the engineering VP out to make the investors happy rather than to actually move forward.

pavs 16y ago
How does it make investors happy? Why would investors care what kind of database you use?
- fookyong 16y ago
  it's an attempt to control the PR message. by doing this, they are laying the blame on one person so the media can stop hating Digg and start hating the ex-VP of Engineering who "killed Digg". of course, whether he was actually responsible in some way is something we may never know. for all we know he may have been completely against releasing v4 but was vetoed by Rose et al. or on the other hand he may have overpromised and under-delivered, putting the company in jeopardy, in which case he deserves to be let go. it's all speculation until we hear an official comment from either side.
- code_duck 16y ago
  The typical investor doesn't take the time to really understand the companies or technical issues, unfortunately. They see 'there was a problem, management fired the guy who was at fault' and are happy.

zephjc 16y ago

Digg v4 has added back the features I liked after taking them away - mostly the Upcoming section, which tends to have interesting things that never make it to the front page, and the setting that allows you to default to Top Mews, instead of things I have already read ('My News'). For most users, change in features is what drove them away, not necessarily the spotty QOS (though that was pretty bad for a while).

mminolt 16y ago

Ok Digg is having a hard time. I feel like I witness a strange blood thirsty tone in some of the comments though. I wish Digg the best and I hope they can prove all the skeptics wrong. It's going to be tough though!

nl 16y ago

Digg's put out a lot of information about their Cassandra implementation, eg https://nosqleast.com/2009/slides/sarkissian-cassandra.pdf https://nosqleast.com/2009/slides/sarkissian-cassandra.pdf (from November 2009). Back then they seemed to have a rather sensible migration strategy (ie, basically running the new Cassandra back-end in parallel with the MySQL backend). It seems to me that it was the v4 upgrade that broke, not Cassandra alone. It's possible their frustrations with Cassandra were more long term, and the fact the v4 upgrade didn't go well was the last straw. For example, Digg has done a lot of work on Cassandra internals and tools. If you are using a new, open source product you kind of expect that, but it's possible the expense of that didn't seem like good value once v4 started to get into trouble,

c00p3r 16y ago

Let me guess - the problem is about the difference between theory and practice. In theory, Java is great and Cassandra is great. In practice - Java under a heavy load is a disaster, because it was never designed for it, and Cassandra is a just a hype and propaganda. Face the reality - it doesn't work in production as it supposed to - as a primary storage engine. People at the Digg aren't amateur idiots, so I think they do everything as it described in docs, but the damn thing just doesn't work.

nl 16y ago
> In practice - Java under a heavy load is a disaster Google's heavy use of serverside Java would indicate otherwise. > because it was never designed for it Yes it was. Java has a lot of problems, but one thing that isn't a problem is heavy load. > Cassandra is a just a hype and propaganda Facebook seems to be using it ok.
- c00p3r 16y ago
  Google probably use server side Java without a really heavy load, and just add more servers to keep the load low. System which was designed for being isolated from an OS (leave alone hardware) will have a bottleneck exactly in this level. Facebook doesn't use it as a primary storage with a high load.
  - cameronh90 16y ago
    What would you prefer to write web applications in? C++? Most languages used in web programming are divorced from the hardware, and either use a virtual machine (e.g. Java, .NET) or an interpreter (PHP, Python, Perl, Ruby). Virtually no-one uses a low level language for web programming, and the benchmarks say that the virtual machines are faster than interpreters. Java is very fast and efficient once it's running, but initial start-up time is often slower than interpreted languages due to the JIT compilation, but servers rarely "start up". If you're really hitting a barrier with Java's performance, just as with other non-native languages, you can write the performance critical section in C. http://shootout.alioth.debian.org/u32q/which-programming-languages-are-fastest.php http://shootout.alioth.debian.org/u32q/which-programming-lan...

Dervish 16y ago

Digg 4.x is like Star Wars Galaxies NGE.