4 ms·

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process. FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least according to Alexa). I'm not speaking from inexperience: if this problem is hard for 37signals, it's because of failures of the implementation, not because the problem is intrinsically hard. Engineers who don't recognize that are not ones whose competence I would put trust in.

kristiandupont 16y ago
Should you implement something with an array or a linked list? That depends. Should it be fast to locate an element in the middle? Or should it be cheap to add another element regardless of the size? Ideally, both, but you weigh the pros and cons and choose one. And even if it is the right choice, you might still run into situations where the other choice would have been better and people will comment that your decision was stupid.
- vog 16y ago
  This is not necessarily about whether the denormalized database structure is appropriate. It is more about: Why the h... doesn't their ORM, or even better their database, already take care of cascading updates? Why does it have to be implemented manually for every code that updates something? (... which is of course expensive and prone to errors)
  - gaius 16y ago
    Exactly. I can imagine that they thought they were being very clever using ORM + MySQL instead of writing SQL to run on a more mature RDBMS. Well, decisions like that often result in you painting yourself into a corner. There's a reason that MySQL is free yet people still pay for Oracle...
    - vog 16y ago
      I was not thinking of Oracle here. PostgreSQL is free software and offers almost everything you expect from a real database. So I guess the main reason people pay for Oracle is because they only know about MySQL and not about PostgreSQL.
      - gaius 16y ago
        You're probably right. I've never understood why Postgres has such a small mindshare. Between Postgres and SQLite, there ought to be no ecological niche for MySQL to exist...
        jemfinch 16y ago
        MySQL does (did, at least; Postgres has been improving) replication better. Replication is really important for high availability, which almost everyone needs. The irony, of course, is that a single Postgres server is frequently (IME) more reliable than multiple MySQL servers even when the latter is setup for HA.
        vog 16y ago
        In addition, replication is the next thing where PostgreSQL is vastly improving (according to their road map). Also note that if you want to trade some of the ACID properties for better performance and replication, the so-called NoSQL databases (CouchDB etc) seem to be a better trade-off than, say, MySQL with MyISAM instead of InnoDB.
- jemfinch 16y ago
  Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence. While I fully agree that trade offs like yours do occur in our profession, their mere existence is no argument against my initial non-trust of 37signals' competence: you need to actually demonstrate that this problem is one of those cases, and I think that would be difficult to do. The problem of associating entities in a way that their associations can be modified easily and consistently (i.e., normalization) is effectively a solved problem. Just like the programmer complaining that his program is slow because he's doing a linear search through a linked list, a BaseCamp programmer complain that a requested feature is difficult to implement because their data is in an abnormal form does not inspire confidence, and certainly gives no reason to trust his competence.
  - ellyagg 16y ago
    > Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence In Erlang, the most common data structure is the linked list. There's no way to search them other than linearly. You are claiming that in general we should suspect that Erlang programmers are incompetent?
    - Dylan16807 16y ago
      Only the ones that won't switch to a log-performance data structure when it's causing them trouble.
  - ericd 16y ago
    If the worst case size of that linked list is small, then I would say that I would trust that programmer more than the one that throws every searched linked list into a tree structure regardless of size. In the case of sharding by project, it actually seems like a decent move given that a single Basecamp account has potentially large space requirements (75 GB), and you can't realistically provision enough space on a box to meet all the needs of a bunch of expanding accounts on that box. Being able to throw an entire project onto a new server and assuming that everything associated with a project is on that one server are both nice simplifying assumptions which seem like they would strike a nice balance. Please stop painting everything with broad black and white strokes, programming and business in general are frequently more about striking the right balance of compromises than about knowing the "best" way to do things. There's usually not a universal "best". 37Signals seem to be doing a marvelous job balancing those compromises between their various disciplines, especially given their massive success with customers. Edit: This is assuming they're storing attachment files on the same machine, which might very well be wrong. It would be pretty hard for one group to get up to 75GB of todo's and other text content...
- anthonyb 16y ago
  Basecamp is written in Ruby, so the choice is really a hash or a list. Either way, it's not likely to be your bottleneck in a web app.
dasil003 16y ago
Pick 10,000 random features and aspects of a software program to be implemented over a 10 year period. Some architectural choices will benefit some and harm others. There is no choice that will make "all easy problems easy". You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. Do you really not see that?
- jemfinch 16y ago
  > You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. The problem is easy. Seriously, it really is. Most of the programmers here have probably done something almost exactly like it ten or more times in their relatively short careers; I know I have. There is no real intrinsic difficulty to the problem. I'm not saying that the programmers are 37signals were incompetent. I'm saying that I have do not trust that they are competent. I have no reason, especially in the face of this evidence, to believe that they're competent. They made a successful software product: so have lots of other programmers of questionable competency. The fact remains that at least in this example, one of their programmers points at an intrinsically easy problem and says, "This is really hard because I'd have to do a lot of stuff that I really shouldn't have to do" except he doesn't seem to realize that he shouldn't have do those things. So tell me, in face of that obvious fact, why should I trust 37signals' engineers' competence?
  - dasil003 16y ago
    The problem is easy. Seriously, it really is. It's easy in the simplified mental model that you have constructed devoid of real world context. Any engineer worth their salt knows the devil's in the details. I'm going to stop arguing now because you're not even responding to my actual argument. It's not a zen thing, you should be able to get it if you actually read my comments, but just in case you need a koan, ponder this fact that I would bet my life savings on: There exists a potential feature which could be implemented in Basecamp faster than in your product (and vice versa).

kristiandupont 16y ago

Should you implement something with an array or a linked list? That depends. Should it be fast to locate an element in the middle? Or should it be cheap to add another element regardless of the size? Ideally, both, but you weigh the pros and cons and choose one. And even if it is the right choice, you might still run into situations where the other choice would have been better and people will comment that your decision was stupid.

vog 16y ago
This is not necessarily about whether the denormalized database structure is appropriate. It is more about: Why the h... doesn't their ORM, or even better their database, already take care of cascading updates? Why does it have to be implemented manually for every code that updates something? (... which is of course expensive and prone to errors)
- gaius 16y ago
  Exactly. I can imagine that they thought they were being very clever using ORM + MySQL instead of writing SQL to run on a more mature RDBMS. Well, decisions like that often result in you painting yourself into a corner. There's a reason that MySQL is free yet people still pay for Oracle...
  - vog 16y ago
    I was not thinking of Oracle here. PostgreSQL is free software and offers almost everything you expect from a real database. So I guess the main reason people pay for Oracle is because they only know about MySQL and not about PostgreSQL.
    - gaius 16y ago
      You're probably right. I've never understood why Postgres has such a small mindshare. Between Postgres and SQLite, there ought to be no ecological niche for MySQL to exist...
      - jemfinch 16y ago
        MySQL does (did, at least; Postgres has been improving) replication better. Replication is really important for high availability, which almost everyone needs. The irony, of course, is that a single Postgres server is frequently (IME) more reliable than multiple MySQL servers even when the latter is setup for HA.
        vog 16y ago
        In addition, replication is the next thing where PostgreSQL is vastly improving (according to their road map). Also note that if you want to trade some of the ACID properties for better performance and replication, the so-called NoSQL databases (CouchDB etc) seem to be a better trade-off than, say, MySQL with MyISAM instead of InnoDB.
jemfinch 16y ago
Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence. While I fully agree that trade offs like yours do occur in our profession, their mere existence is no argument against my initial non-trust of 37signals' competence: you need to actually demonstrate that this problem is one of those cases, and I think that would be difficult to do. The problem of associating entities in a way that their associations can be modified easily and consistently (i.e., normalization) is effectively a solved problem. Just like the programmer complaining that his program is slow because he's doing a linear search through a linked list, a BaseCamp programmer complain that a requested feature is difficult to implement because their data is in an abnormal form does not inspire confidence, and certainly gives no reason to trust his competence.
- ellyagg 16y ago
  > Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence In Erlang, the most common data structure is the linked list. There's no way to search them other than linearly. You are claiming that in general we should suspect that Erlang programmers are incompetent?
  - Dylan16807 16y ago
    Only the ones that won't switch to a log-performance data structure when it's causing them trouble.
- ericd 16y ago
  If the worst case size of that linked list is small, then I would say that I would trust that programmer more than the one that throws every searched linked list into a tree structure regardless of size. In the case of sharding by project, it actually seems like a decent move given that a single Basecamp account has potentially large space requirements (75 GB), and you can't realistically provision enough space on a box to meet all the needs of a bunch of expanding accounts on that box. Being able to throw an entire project onto a new server and assuming that everything associated with a project is on that one server are both nice simplifying assumptions which seem like they would strike a nice balance. Please stop painting everything with broad black and white strokes, programming and business in general are frequently more about striking the right balance of compromises than about knowing the "best" way to do things. There's usually not a universal "best". 37Signals seem to be doing a marvelous job balancing those compromises between their various disciplines, especially given their massive success with customers. Edit: This is assuming they're storing attachment files on the same machine, which might very well be wrong. It would be pretty hard for one group to get up to 75GB of todo's and other text content...
anthonyb 16y ago
Basecamp is written in Ruby, so the choice is really a hash or a list. Either way, it's not likely to be your bottleneck in a web app.

dasil003 16y ago

Pick 10,000 random features and aspects of a software program to be implemented over a 10 year period. Some architectural choices will benefit some and harm others. There is no choice that will make "all easy problems easy". You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. Do you really not see that?

jemfinch 16y ago
> You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. The problem is easy. Seriously, it really is. Most of the programmers here have probably done something almost exactly like it ten or more times in their relatively short careers; I know I have. There is no real intrinsic difficulty to the problem. I'm not saying that the programmers are 37signals were incompetent. I'm saying that I have do not trust that they are competent. I have no reason, especially in the face of this evidence, to believe that they're competent. They made a successful software product: so have lots of other programmers of questionable competency. The fact remains that at least in this example, one of their programmers points at an intrinsically easy problem and says, "This is really hard because I'd have to do a lot of stuff that I really shouldn't have to do" except he doesn't seem to realize that he shouldn't have do those things. So tell me, in face of that obvious fact, why should I trust 37signals' engineers' competence?
- dasil003 16y ago
  The problem is easy. Seriously, it really is. It's easy in the simplified mental model that you have constructed devoid of real world context. Any engineer worth their salt knows the devil's in the details. I'm going to stop arguing now because you're not even responding to my actual argument. It's not a zen thing, you should be able to get it if you actually read my comments, but just in case you need a koan, ponder this fact that I would bet my life savings on: There exists a potential feature which could be implemented in Basecamp faster than in your product (and vice versa).