6 ms·

Like Joel Spolsky said it: "They followed the single worst strategic mistake that any software company can make: They decided to rewrite everything from scratc

by amix 16y ago

Like Joel Spolsky said it: "They followed the single worst strategic mistake that any software company can make: They decided to rewrite everything from scratch" - http://www.joelonsoftware.com/articles/fog0000000069.html http://www.joelonsoftware.com/articles/fog0000000069.html I think rewriting from scratch is the core of their problem and not really Cassandra. Gradually going over to Cassandra would have been a much better idea.

gaiusparx 16y ago
The other problem is they have been hyping V4 for a long time by making beta testing invite only and hard to get. It would have been better if they can have a public V4 beta running in parallel with V3 since it is a rewrite.
- dawie 16y ago
  Even better: Have a link that says: "Try the new Beta" at the top like gmail and google docs, with a "Go back to Old Version". Implementing this should not be to hard and it allows them to get some feedback.
  - bryanh 16y ago
    As a developer myself, I love hearing: > Implementing this should not be to hard...
  - pilif 16y ago
    Considering the completely different backends of v3 and v4, I think, this would have been incredibly hard to implement. At least if changes in one system should end up in the other. In this case you would not just have to write scrips to one-time migrate all needed data from v3 mysql to v4 Cassandra. No. You would have to build a mechanism that doesn't just do it in both directions, it would also have to work at near-realtime speed. If you then need consistency of the transferred data, this quickly gets impossible. Try finding a way to ensure consistency between these two completely different architectures. In the end, most of v3 would have needed to be rewritten for a parallel use to be possible, at which point you don't really gain much. "not hard to implement" - sigh Disclaimer: I don't work at digg and I don't know more about their backend than the rest of the public. I did however just get around doing something like what you describe and there it wad "just" a different schema on the same database backend and even just that would have been hell
    - MartinCron 16y ago
      In a way, the one-time migration might be harder than near-realtime bidirectional synchronization. That way, you could move portions of users to the new system and back as needed. A sudden leap from one backend to another is like jumping over the grand canyon on a motorcycle. Personally, I would rather build a bridge.
      - pilif 16y ago
        For the concurrent configuration to work consistently, digg would probably have to port v3 to the new backend. Now in the current case, the most issues apparently come from the non-working backend as opposed to the changed featureset. So while they could have run the two versions in parallel, they would not have gained anything. Likely, this was their rationale behind not doing so in the first place.
        MartinCron 16y ago
        No, for the current configuration to work, you would have to create a way to move data in both directions from the "old" backend to the "new" backend. Ideally, you would slice up the beast so you didn't have to do this with everything all at once. It's not a system that would be ideal for something transactional like a bank, but it may have been possible for an organization like Digg.
- blantonl 16y ago
  you can't do a parallel beta test of two functionally different sites. You would end up with two separate data stores, workflows, and ultimately a confused user base. Never mind the fact that the user base is confused by the rollout. And never mind the fact that Google has forever damaged the term "Beta" in the minds of general Web consumers.
  - patio11 16y ago
    And never mind the fact that Google has forever damaged the term "Beta" in the minds of general Web consumers. The general web consumer still thinks betas are a type of fish. Not really relevant to the median Digg user, though, since they are not the general Web user.
plnewman 16y ago
Actually if you google "digg cassandra" you'll get a number of blog posts dating back to 2009, describing different things they use Cassandra for.
binspace 16y ago
Yep, big bang rewrites are always scary. Doing something more incremental would have been safer. One way to "safely" do a big-bang change would have been to use both architectures in parallel and sync back and forth. That way, if the new architecture fails, you still have the old one. Of course, that is lots of effort, and restricts new features (which may be a good thing anyway).
squidsoup 16y ago
I've heard this time and time again, but what if your application is a genuine ball of mud? Would you really not advocate a rewrite for an unmaintainable spaghetti classic asp app still in production today?
- dasil003 16y ago
  I'd say it depends on if your engineering talent is vastly superior today than what it was when it was written. Even then it's so risky. It's probably always better to do it incrementally even if it takes twice as long to do so, because you can maintain working software and fix bugs as you go. The approach I would take is to get the minimum set of engineers who know the most about each major aspect of the code, and put their heads together on what the ideal architecture would be. But rather than building it from scratch, figure out how to implement just one of those pieces now. That way you can decrease entropy in the codebase piecemeal without chucking out all the code at once, which is no doubt full of forgotten assumptions that no one will remember until it's too late.
- MartinCron 16y ago
  I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system. Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events. A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame that so many people think it's a good idea, even in 2010.
  - davidw 16y ago
    > incremental rewrite of parts of the unmaintainable spaghetti classic Sounds good in theory. In practice? Part of the problem with many big ball of mud systems is that all the parts depend on and talk to all the other parts. Want to fix that horrid DB schema? You'll have to rewrite all the code that talks to it, or rewrite it to talk to an intermediary. Want to rewrite that horrid bit of code that's called foobar_20040623 ("foobar" has been changed, but yes, I saw this in some PHP code...), you'll have to find out what all it interacts with, and likely redo that too.
Yaa101 16y ago
No, what they done wrong is not to make a proper 2 way conversion tool for their database backup. Starting from scratch is no problem if you got your core data covered, then you can always revert back. Don't ever develop yourself into a one way street!!
mmaunder 16y ago
This is my all time favorite Spolsky blog entry. This lesson has saved me a few times. The broader lesson is that new and shiny in software isn't better and is often worse.

by amix 16y ago

gaiusparx 16y ago

The other problem is they have been hyping V4 for a long time by making beta testing invite only and hard to get. It would have been better if they can have a public V4 beta running in parallel with V3 since it is a rewrite.

dawie 16y ago
Even better: Have a link that says: "Try the new Beta" at the top like gmail and google docs, with a "Go back to Old Version". Implementing this should not be to hard and it allows them to get some feedback.
- bryanh 16y ago
  As a developer myself, I love hearing: > Implementing this should not be to hard...
- pilif 16y ago
  Considering the completely different backends of v3 and v4, I think, this would have been incredibly hard to implement. At least if changes in one system should end up in the other. In this case you would not just have to write scrips to one-time migrate all needed data from v3 mysql to v4 Cassandra. No. You would have to build a mechanism that doesn't just do it in both directions, it would also have to work at near-realtime speed. If you then need consistency of the transferred data, this quickly gets impossible. Try finding a way to ensure consistency between these two completely different architectures. In the end, most of v3 would have needed to be rewritten for a parallel use to be possible, at which point you don't really gain much. "not hard to implement" - sigh Disclaimer: I don't work at digg and I don't know more about their backend than the rest of the public. I did however just get around doing something like what you describe and there it wad "just" a different schema on the same database backend and even just that would have been hell
  - MartinCron 16y ago
    In a way, the one-time migration might be harder than near-realtime bidirectional synchronization. That way, you could move portions of users to the new system and back as needed. A sudden leap from one backend to another is like jumping over the grand canyon on a motorcycle. Personally, I would rather build a bridge.
    - pilif 16y ago
      For the concurrent configuration to work consistently, digg would probably have to port v3 to the new backend. Now in the current case, the most issues apparently come from the non-working backend as opposed to the changed featureset. So while they could have run the two versions in parallel, they would not have gained anything. Likely, this was their rationale behind not doing so in the first place.
      - MartinCron 16y ago
        No, for the current configuration to work, you would have to create a way to move data in both directions from the "old" backend to the "new" backend. Ideally, you would slice up the beast so you didn't have to do this with everything all at once. It's not a system that would be ideal for something transactional like a bank, but it may have been possible for an organization like Digg.
blantonl 16y ago
you can't do a parallel beta test of two functionally different sites. You would end up with two separate data stores, workflows, and ultimately a confused user base. Never mind the fact that the user base is confused by the rollout. And never mind the fact that Google has forever damaged the term "Beta" in the minds of general Web consumers.
- patio11 16y ago
  And never mind the fact that Google has forever damaged the term "Beta" in the minds of general Web consumers. The general web consumer still thinks betas are a type of fish. Not really relevant to the median Digg user, though, since they are not the general Web user.

plnewman 16y ago

Actually if you google "digg cassandra" you'll get a number of blog posts dating back to 2009, describing different things they use Cassandra for.

binspace 16y ago

Yep, big bang rewrites are always scary. Doing something more incremental would have been safer. One way to "safely" do a big-bang change would have been to use both architectures in parallel and sync back and forth. That way, if the new architecture fails, you still have the old one. Of course, that is lots of effort, and restricts new features (which may be a good thing anyway).

squidsoup 16y ago

I've heard this time and time again, but what if your application is a genuine ball of mud? Would you really not advocate a rewrite for an unmaintainable spaghetti classic asp app still in production today?

dasil003 16y ago
I'd say it depends on if your engineering talent is vastly superior today than what it was when it was written. Even then it's so risky. It's probably always better to do it incrementally even if it takes twice as long to do so, because you can maintain working software and fix bugs as you go. The approach I would take is to get the minimum set of engineers who know the most about each major aspect of the code, and put their heads together on what the ideal architecture would be. But rather than building it from scratch, figure out how to implement just one of those pieces now. That way you can decrease entropy in the codebase piecemeal without chucking out all the code at once, which is no doubt full of forgotten assumptions that no one will remember until it's too late.
MartinCron 16y ago
I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system. Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events. A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame that so many people think it's a good idea, even in 2010.
- davidw 16y ago
  > incremental rewrite of parts of the unmaintainable spaghetti classic Sounds good in theory. In practice? Part of the problem with many big ball of mud systems is that all the parts depend on and talk to all the other parts. Want to fix that horrid DB schema? You'll have to rewrite all the code that talks to it, or rewrite it to talk to an intermediary. Want to rewrite that horrid bit of code that's called foobar_20040623 ("foobar" has been changed, but yes, I saw this in some PHP code...), you'll have to find out what all it interacts with, and likely redo that too.

Yaa101 16y ago

No, what they done wrong is not to make a proper 2 way conversion tool for their database backup. Starting from scratch is no problem if you got your core data covered, then you can always revert back. Don't ever develop yourself into a one way street!!

mmaunder 16y ago

This is my all time favorite Spolsky blog entry. This lesson has saved me a few times. The broader lesson is that new and shiny in software isn't better and is often worse.