11 ms·

Code icebergs

136 points by bjplink 16y ago

dhruvbird 16y ago
Hey it's great you got a name for them now! You won't believe how many times I've been in a discussion about something and the other person has said "oh that's easy to do" or "it can be done in a few hours" when in fact if they were to go into the details, they would see the hiding devil...
- praptak 16y ago
  My personal red-flag phrase is "You just need to ..." Favorite occurence: "You just need to build a state machine." Yeah, saving the whole browser-side state (did I mention third-party GUI components?) of an application and reestablishing the server-side session state to match it is really easy with this piece of sage advice.
  - damncabbage 16y ago
    "Just" is unofficially a banned word around here. :)
bediger 16y ago
I think this is one way that J.P. Lewis' "Large Limits to Software Estimation" (http://scribblethink.org/Work/Softestim/softestim.html http://scribblethink.org/Work/Softestim/softestim.html) comes out in reality. Naturally, there's an uncountable infinity of ways that comes out in reality, but Code Icebergs are a common way.
deleted 16y ago
[deleted]
marcusbooster 16y ago
Every now and again I'll remember that guy who said he was going to build a Stack Overflow clone in a weekend. I still chuckle.
- vidar 16y ago
  I had the same thought, Stack Overflow is the canonical iceberg in this context. I hope this will pull the blinders from some peoples eyes.
- jules 16y ago
  This might not be such a fantasy as you think it is. Yes, it will be very unpolished. But you can get the basics in place in a couple of hours: login, questions, answers and voting. I agree though that to get it to the level of stackoverflow will be very hard indeed in a weekend ;)
  - nl 16y ago
    Anything is easy if you ignore the hard parts.
- w1ntermute 16y ago
  http://news.ycombinator.com/item?id=678501 http://news.ycombinator.com/item?id=678501 For those of us who weren't around at that time or have forgotten. ktharavaad agrees to attempt it in a grandchild post.
  - robryan 16y ago
    Doesn't ook like he got to far, nothing of substance in the google code he setup for the project.
moondistance 16y ago
On the topic of mining Wikipedia, DBpedia (http://dbpedia.org http://dbpedia.org) is a fantastic source for structured Wikipedia content. Extracting data is pretty easy with SPARQL. Freebase (http://freebase.com http://freebase.com) isn't bad, either.
- jules 16y ago
  The DBpedia license would force him to license his code under GPL or similar, no?
  - ddorian 16y ago
    if he chooses to release the code
  - moondistance 16y ago
    I haven't read the GNU Free Documentation License (http://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License http://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_...), but I don't think so. "DBpedia is derived from Wikipedia and is distributed under the same licensing terms as Wikipedia itself" http://wiki.dbpedia.org/Datasets#h18-19 http://wiki.dbpedia.org/Datasets#h18-19 Please correct me if I'm mistaken!
pmjordan 16y ago
I've probably fallen for these more often than I'd like to admit. Yet sometimes a naïve "how hard could it be" is exactly the right approach that leads to unexpectedly simple solutions.
amix 16y ago
I think great products tend to have a lot of complexity, but most of the complexity is hidden away from the user. This picture sums up this thought (and it mimics Gabriel's iceberg metaphor quite well): http://amix.dk/blog/post/19555#The-essence-of-minimal-product-design http://amix.dk/blog/post/19555#The-essence-of-minimal-produc...
sdrinf 16y ago
Code iceberg is in the eye of the beholder. Recently started bizdev-people consistently underestimate the time requirements for certain well-exercised tasks. Some of the most common icebergs are: -form validation (seriously -one of the most highly exercised user-interaction paths; it's all over the place, and scales semi-exponentially with the number of fields) -search ("how hard could it be? you just put an input form there, then figure out what the user thought, then display it" -exact quote) -anything that has to process natural language. I mean everything. Wanna split up a text into sentences? How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders? Generating a definite article ("a", "an") before a noun? Keep in mind that 1,2,@,$,=, and other characters might also be valid noun first-letters :) etc. In my experience, the best anti-iceberg pattern is to follow a portfolio approach, and for each requirements which smells like iceberg, have a fallback plan in place -ie. after N hours of sunken investment, execution shifts to plan B. Usually works out much better, than banging away on the same problem for days.
- praptak 16y ago
  "How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders?" Fun fact: there exists a convention stipulating a double space after a period that ends a sentence. Not that I'm advocating relying on this for any serious purposes
  - RickHull 16y ago
    There is also a convention of capitalizing the first letter of a sentence. The example text demonstrates that corpus quality (forgive the fuzzy terminology) trumps clean, simple rules.
    - billybob 16y ago
      Yep. First letter of a sentence, or title, or any other proper noun. So... how can your program tell which it is? The correct algorithm approaches AI in complexity.
      - 16y ago
edanm 16y ago
Steve Yegge once wrote a post on this sort of thing. It's called "Have you ever legalized marijuana?", but it doesn't focus on marijuana all that much. Well worth the read: http://steve-yegge.blogspot.com/2009/04/have-you-ever-legalized-marijuana.html http://steve-yegge.blogspot.com/2009/04/have-you-ever-legali....
- Nitramp 16y ago
  Except that his conclusion is way off for the Marijuana part. There are lots of countries where Marijuana is legal and/or tolerated, so apparently the legal systems are a lot more flexible than the US, or Yegge's estimates for refactoring costs in the legal system are way off.
bialecki 16y ago
Twitter is one of my favorite examples of this. Lots of people look at it and say, "it's so easy implement to build." I'd like to see them scale it to millions of users.
- robryan 16y ago
  I'd be surprised if anyone thought building Twitter with similar scalability would be easy. Sure while you cold fit it into 1 mySQL DB it would be easy to get the basic features down with a much simpler UI and no API. Past that though they would be vastly underestimating the work.
- boulderdash 16y ago
  scaling twitter is easy. it is getting the users that is hard. I can say this because I was part of a service in 1999 that had more users and volume (bytes/hits/etc.) yet didn't have nearly the same issues.
  - nl 16y ago
    I can say this because I was part of a service in 1999 that had more users and volume (bytes/hits/etc.) yet didn't have nearly the same issues. Not to be rude, but I doubt it. Twitter currently has 175 million users[1]. Estimates in 1999 for the online population of the entire internet were 259 million, with 110 million in the US [2]. In 1999, I imagine Yahoo and maybe a couple of other sites (Microsoft/Excite/AOL/Lycos?) were getting similar traffic numbers to what Twitter does today. BUT the scaling is very different, because Twitter requires fan-out of messages, which none of those sites did. [1]: http://www.pcmag.com/article2/0,2817,2371826,00.asp http://www.pcmag.com/article2/0,2817,2371826,00.asp [2]: http://web.archive.org/web/20000208141450/http://c-i-a.com/199911iu.htm http://web.archive.org/web/20000208141450/http://c-i-a.com/1...
gfodor 16y ago
Another class of code icebergs are numerical algorithms. Often a few dozen lines of Matlab or R can be the result of months of effort. Failed approaches, tolerance thresholds, manual data cleansing, and more can all end up living as a few lines of math.
- leif 16y ago
  Most math takes the form of icebergs. :-)
deleted 16y ago
[deleted]
patio11 16y ago
I always annotate these on development plans with HBG: Here Be Dragons.
- caf 16y ago
  Enquiring minds want to know: Why does G denote Dragons?
  - patio11 16y ago
    iPad typing is not recommended before morning coffee.
callmevlad 16y ago
A great example comes from 37signals - http://answers.37signals.com/basecamp/52-how-can-i-move-or-copy-to-do-lists-between-projects http://answers.37signals.com/basecamp/52-how-can-i-move-or-c... - look for Sam Stephenson's reply (from Sept 8th) in which he explains all the edge cases below the surface.
Hexstream 16y ago
Try to beat this one: I spent 4 months, full-time (I think I was a bit depressive and unproductive, though (that might have to do with the difficulty of the problem a bit)) to make a goddamn "error message merging" system where you specify some merge rules for error messages, and then said messages are "merged" efficiently at runtime (with another of my adaptations of the awesome Rete algorithm). For instance, as a trivial example merging "Please enter your Username." and "Please enter your Password." could yield "Please enter your Username and Password." Merging error messages efficiently with a great concise syntax looked SO EASY =/ I was wondering why the hell no websites (that I know of) do that because it's a pretty obvious feature to me... Well, now I know. People don't really mind that much about these things (maybe they just have low expectations) AND it's really hard to implement. I finally made it, right now the implementation is utter crap, with some missing features and some bugs but the general architecture is there and works. I'll clean it up and document it within a few months, probably. It's one of the hardest things I ever made so far in programming.
- GFischer 16y ago
  Heh, I was asked to do the same thing for the current system/website we're building here at the corporation I work for. Fortunately we were able to convince them it wasn't time well spent, but it would be neat.
derefr 16y ago
I think a code iceberg is actually a symptom of either working at too low a level, or relying on a library with missing/broken features. You see libfoo, and you think "great! I'll use that to implement foobar-ization functionality!" but then, after playing with it for a bit, you realize that foobar-ization actually requires you to do all sorts of crazy things with the output of libfoo before you can use it in XYZApp. Now, you can put all those crazy foobar-izing things into XYZApp, and that'll work—but they should really either go into libfoo itself, or into a new library (libfoobarize) that uses libfoo. This is the case with the example in the article: DuckDuckGo shouldn't be parsing Wikipedia to make its own abstracts. MediaWiki already creates abstracts—they're just bad abstracts. The correct thing to do, since MediaWiki is just a regular ol' FOSS project, is to write a patch that makes MediaWiki spit out good abstracts, that are actually trivial to use in DuckDuckGo. Or, even better, if you know MediaWiki cares about having good abstracts, just submit it as an issue to their tracker and let them do it for you. In other words, repeat the programmer's litany to stave off NIH: "It's not my job. I shall buy, not build. 80% of the features at 20% of the cost. Don't ask a question, send a message. No god-objects. Encapsulate, encapsulate, encapsulate." Note that, of course, there are cases where there really is no libfoo—but then you're doing something totally new, and you can tell the client right up-front "no one's ever done this before, so we have to schedule time for R&D before we can even tell you how much time this feature will take." There is also the case where the only libfoo/libfoobarize is a proprietary one used by the people you're trying to steal market-share from by implementing this feature, in which case you can tell your client "we know it's possible, but we don't know how long it took them to build it. What we do know is that no one else has yet copied them, which means that foobar-ization isn't trivial. It'll probably take a while."
rams 16y ago
Icebergs are a good analogy for technical debt as well. The hackish stuff below the surface, often done against better advice invariably causes serious damage.
staktrace 16y ago
There's the classic rule of thumb from Brooks' Mythical Man Month: turning a project into a product takes 9x the effort.

136 points by bjplink 16y ago

Hey it's great you got a name for them now! You won't believe how many times I've been in a discussion about something and the other person has said "oh that's easy to do" or "it can be done in a few hours" when in fact if they were to go into the details, they would see the hiding devil...

praptak 16y ago
My personal red-flag phrase is "You just need to ..." Favorite occurence: "You just need to build a state machine." Yeah, saving the whole browser-side state (did I mention third-party GUI components?) of an application and reestablishing the server-side session state to match it is really easy with this piece of sage advice.
- damncabbage 16y ago
  "Just" is unofficially a banned word around here. :)

bediger 16y ago

I think this is one way that J.P. Lewis' "Large Limits to Software Estimation" (http://scribblethink.org/Work/Softestim/softestim.html http://scribblethink.org/Work/Softestim/softestim.html) comes out in reality. Naturally, there's an uncountable infinity of ways that comes out in reality, but Code Icebergs are a common way.

deleted 16y ago

[deleted]

marcusbooster 16y ago

Every now and again I'll remember that guy who said he was going to build a Stack Overflow clone in a weekend. I still chuckle.

vidar 16y ago
I had the same thought, Stack Overflow is the canonical iceberg in this context. I hope this will pull the blinders from some peoples eyes.
jules 16y ago
This might not be such a fantasy as you think it is. Yes, it will be very unpolished. But you can get the basics in place in a couple of hours: login, questions, answers and voting. I agree though that to get it to the level of stackoverflow will be very hard indeed in a weekend ;)
- nl 16y ago
  Anything is easy if you ignore the hard parts.
w1ntermute 16y ago
http://news.ycombinator.com/item?id=678501 http://news.ycombinator.com/item?id=678501 For those of us who weren't around at that time or have forgotten. ktharavaad agrees to attempt it in a grandchild post.
- robryan 16y ago
  Doesn't ook like he got to far, nothing of substance in the google code he setup for the project.

moondistance 16y ago

On the topic of mining Wikipedia, DBpedia (http://dbpedia.org http://dbpedia.org) is a fantastic source for structured Wikipedia content. Extracting data is pretty easy with SPARQL. Freebase (http://freebase.com http://freebase.com) isn't bad, either.

jules 16y ago
The DBpedia license would force him to license his code under GPL or similar, no?
- ddorian 16y ago
  if he chooses to release the code
- moondistance 16y ago
  I haven't read the GNU Free Documentation License (http://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_Documentation_License http://en.wikipedia.org/wiki/Wikipedia:Text_of_the_GNU_Free_...), but I don't think so. "DBpedia is derived from Wikipedia and is distributed under the same licensing terms as Wikipedia itself" http://wiki.dbpedia.org/Datasets#h18-19 http://wiki.dbpedia.org/Datasets#h18-19 Please correct me if I'm mistaken!

pmjordan 16y ago

I've probably fallen for these more often than I'd like to admit. Yet sometimes a naïve "how hard could it be" is exactly the right approach that leads to unexpectedly simple solutions.

amix 16y ago

I think great products tend to have a lot of complexity, but most of the complexity is hidden away from the user. This picture sums up this thought (and it mimics Gabriel's iceberg metaphor quite well): http://amix.dk/blog/post/19555#The-essence-of-minimal-product-design http://amix.dk/blog/post/19555#The-essence-of-minimal-produc...

sdrinf 16y ago

Code iceberg is in the eye of the beholder. Recently started bizdev-people consistently underestimate the time requirements for certain well-exercised tasks. Some of the most common icebergs are: -form validation (seriously -one of the most highly exercised user-interaction paths; it's all over the place, and scales semi-exponentially with the number of fields) -search ("how hard could it be? you just put an input form there, then figure out what the user thought, then display it" -exact quote) -anything that has to process natural language. I mean everything. Wanna split up a text into sentences? How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders? Generating a definite article ("a", "an") before a noun? Keep in mind that 1,2,@,$,=, and other characters might also be valid noun first-letters :) etc. In my experience, the best anti-iceberg pattern is to follow a portfolio approach, and for each requirements which smells like iceberg, have a fallback plan in place -ie. after N hours of sunken investment, execution shifts to plan B. Usually works out much better, than banging away on the same problem for days.

praptak 16y ago
"How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders?" Fun fact: there exists a convention stipulating a double space after a period that ends a sentence. Not that I'm advocating relying on this for any serious purposes
- RickHull 16y ago
  There is also a convention of capitalizing the first letter of a sentence. The example text demonstrates that corpus quality (forgive the fuzzy terminology) trumps clean, simple rules.
  - billybob 16y ago
    Yep. First letter of a sentence, or title, or any other proper noun. So... how can your program tell which it is? The correct algorithm approaches AI in complexity.
    - 16y ago

edanm 16y ago

Steve Yegge once wrote a post on this sort of thing. It's called "Have you ever legalized marijuana?", but it doesn't focus on marijuana all that much. Well worth the read: http://steve-yegge.blogspot.com/2009/04/have-you-ever-legalized-marijuana.html http://steve-yegge.blogspot.com/2009/04/have-you-ever-legali....

Nitramp 16y ago
Except that his conclusion is way off for the Marijuana part. There are lots of countries where Marijuana is legal and/or tolerated, so apparently the legal systems are a lot more flexible than the US, or Yegge's estimates for refactoring costs in the legal system are way off.

bialecki 16y ago

Twitter is one of my favorite examples of this. Lots of people look at it and say, "it's so easy implement to build." I'd like to see them scale it to millions of users.

robryan 16y ago
I'd be surprised if anyone thought building Twitter with similar scalability would be easy. Sure while you cold fit it into 1 mySQL DB it would be easy to get the basic features down with a much simpler UI and no API. Past that though they would be vastly underestimating the work.
boulderdash 16y ago
scaling twitter is easy. it is getting the users that is hard. I can say this because I was part of a service in 1999 that had more users and volume (bytes/hits/etc.) yet didn't have nearly the same issues.
- nl 16y ago
  I can say this because I was part of a service in 1999 that had more users and volume (bytes/hits/etc.) yet didn't have nearly the same issues. Not to be rude, but I doubt it. Twitter currently has 175 million users[1]. Estimates in 1999 for the online population of the entire internet were 259 million, with 110 million in the US [2]. In 1999, I imagine Yahoo and maybe a couple of other sites (Microsoft/Excite/AOL/Lycos?) were getting similar traffic numbers to what Twitter does today. BUT the scaling is very different, because Twitter requires fan-out of messages, which none of those sites did. [1]: http://www.pcmag.com/article2/0,2817,2371826,00.asp http://www.pcmag.com/article2/0,2817,2371826,00.asp [2]: http://web.archive.org/web/20000208141450/http://c-i-a.com/199911iu.htm http://web.archive.org/web/20000208141450/http://c-i-a.com/1...

gfodor 16y ago

Another class of code icebergs are numerical algorithms. Often a few dozen lines of Matlab or R can be the result of months of effort. Failed approaches, tolerance thresholds, manual data cleansing, and more can all end up living as a few lines of math.

leif 16y ago
Most math takes the form of icebergs. :-)

deleted 16y ago

[deleted]

patio11 16y ago

I always annotate these on development plans with HBG: Here Be Dragons.

caf 16y ago
Enquiring minds want to know: Why does G denote Dragons?
- patio11 16y ago
  iPad typing is not recommended before morning coffee.

callmevlad 16y ago

A great example comes from 37signals - http://answers.37signals.com/basecamp/52-how-can-i-move-or-copy-to-do-lists-between-projects http://answers.37signals.com/basecamp/52-how-can-i-move-or-c... - look for Sam Stephenson's reply (from Sept 8th) in which he explains all the edge cases below the surface.

Hexstream 16y ago

Try to beat this one: I spent 4 months, full-time (I think I was a bit depressive and unproductive, though (that might have to do with the difficulty of the problem a bit)) to make a goddamn "error message merging" system where you specify some merge rules for error messages, and then said messages are "merged" efficiently at runtime (with another of my adaptations of the awesome Rete algorithm). For instance, as a trivial example merging "Please enter your Username." and "Please enter your Password." could yield "Please enter your Username and Password." Merging error messages efficiently with a great concise syntax looked SO EASY =/ I was wondering why the hell no websites (that I know of) do that because it's a pretty obvious feature to me... Well, now I know. People don't really mind that much about these things (maybe they just have low expectations) AND it's really hard to implement. I finally made it, right now the implementation is utter crap, with some missing features and some bugs but the general architecture is there and works. I'll clean it up and document it within a few months, probably. It's one of the hardest things I ever made so far in programming.

GFischer 16y ago
Heh, I was asked to do the same thing for the current system/website we're building here at the corporation I work for. Fortunately we were able to convince them it wasn't time well spent, but it would be neat.

derefr 16y ago

I think a code iceberg is actually a symptom of either working at too low a level, or relying on a library with missing/broken features. You see libfoo, and you think "great! I'll use that to implement foobar-ization functionality!" but then, after playing with it for a bit, you realize that foobar-ization actually requires you to do all sorts of crazy things with the output of libfoo before you can use it in XYZApp. Now, you can put all those crazy foobar-izing things into XYZApp, and that'll work—but they should really either go into libfoo itself, or into a new library (libfoobarize) that uses libfoo. This is the case with the example in the article: DuckDuckGo shouldn't be parsing Wikipedia to make its own abstracts. MediaWiki already creates abstracts—they're just bad abstracts. The correct thing to do, since MediaWiki is just a regular ol' FOSS project, is to write a patch that makes MediaWiki spit out good abstracts, that are actually trivial to use in DuckDuckGo. Or, even better, if you know MediaWiki cares about having good abstracts, just submit it as an issue to their tracker and let them do it for you. In other words, repeat the programmer's litany to stave off NIH: "It's not my job. I shall buy, not build. 80% of the features at 20% of the cost. Don't ask a question, send a message. No god-objects. Encapsulate, encapsulate, encapsulate." Note that, of course, there are cases where there really is no libfoo—but then you're doing something totally new, and you can tell the client right up-front "no one's ever done this before, so we have to schedule time for R&D before we can even tell you how much time this feature will take." There is also the case where the only libfoo/libfoobarize is a proprietary one used by the people you're trying to steal market-share from by implementing this feature, in which case you can tell your client "we know it's possible, but we don't know how long it took them to build it. What we do know is that no one else has yet copied them, which means that foobar-ization isn't trivial. It'll probably take a while."

rams 16y ago

Icebergs are a good analogy for technical debt as well. The hackish stuff below the surface, often done against better advice invariably causes serious damage.

staktrace 16y ago

There's the classic rule of thumb from Brooks' Mythical Man Month: turning a project into a product takes 9x the effort.