4 ms·

Everybody at a company experiencing scaling issues feels the way you do. The argument however is if the past went the way you wish it did, arguably the company

by gfodor 6y ago

Everybody at a company experiencing scaling issues feels the way you do. The argument however is if the past went the way you wish it did, arguably the company you currently work for would not exist, because they would have lacked the knowledge that you have now to know which areas were the ones that would matter. Absent that knowledge, they would have built “scalable” systems that still had bottlenecks they could not yet see and also would have adopted extraneous technology that ultimately would be thrown out as the product discarded the parts that ended up not being critical to the value proposition uncovered. And of course, this whole process would have burned time, energy, and cash, not to mention shifted the nature of hires and the composition of the team itself, such that in sum it may have been a sufficient opportunity cost to jeopardize the whole business.

cortesoft 6y ago
Yeah, I know this is true, and we always try to remember to understand that the engineering decisions that were made at the time were made for good reasons given the circumstances we were facing. It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time. Still wish we had made some different ones, though!
- josephg 6y ago
  Hear hear! I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Spoilers: That never happened - the longer we used mongo, the more expensive migrating became. We never did manage to justify the migration. As far as I know the team is still stuck with it, and paying the price. (If you're reading this and you know who you are, I'm sorry!)
  - rubyist5eva 6y ago
    Never use MongoDB. Ever. It's a rule of thumb that hasn't led me astray yet. http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/ http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...
    - dominotw 6y ago
      > It's a rule of thumb that hasn't led me astray yet. How can you know if your decision is correct if you never use it in the first place. Isn't it like only implementing A version of A/B and concluding that not implementing B version was a good decision.
    - jcpst 6y ago
      Meh. I read the article, and I agree with it. But that is not what you are saying. Don’t use it for relational data. Don’t use it as your primary data store. At the enterprise I work at, I’ve had success delivering features where data science would have some job that aggregates production data and dumps it to kafka. Then we’ll map it and dump it in mongo. Stick a .net core microservice on top for retrieval. Works great, has none of the problems mentioned in that article. Of course, this might not make sense if you’re not big enough to where you need kafka, kubernetes, etc But “Never use it” is certainly not good advice.
    - karmakaze 6y ago
      This part is totally misguided. > What’s missing from MongoDB is a SQL-style join operation, which is the ability to write one query that mashes together the activity stream and all the users that the stream references. Because MongoDB doesn’t have this ability, you end up manually doing that mashup in your application code, instead. Even with a normalized relational database one wouldn't make a single query to return rows from a join of tables T1, T2, T3, etc when there are repeating relations such as the users in the social graph. This might be done sometimes paying for the duplication to save a network round trip, that's all and no different.
      - debaserab2 6y ago
        A list of activities joined to users wouldn't be a problem for an RDBMS. I'm not sure what you mean by "repeating relations".
        karmakaze 6y ago
        The deep relations (7-table join) example in the link is something I wouldn't do (including the repeated 'user' entities) with an RDBMS, given that I expect some elements of some tables to appear multiple times, that's a waste of I/O and hard to cache. > Users have friends, friends have posts, posts have comments and likes, each comment has one commenter and each like has one liker. I would likely gather all the instances of User id in each context (user, friend, commenter, and liker) and query it separately from the JOIN query. Depends on your data which way works out better on average/worst-cases.
  - zozbot234 6y ago
    > I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Yeah, that can't end well. It's a bit ironic from parent poster's viewpoint but "boring", well-understood technology is generally also the easiest to migrate away from. That's one key reason to choose it at the outset even over more "innovative" and perhaps more "ergonomic" choices like mongo, that in reality aren't nearly as simple to deal with.
deepGem 6y ago
The key part you highlight is that of hiring. If you want to be scale ready from day 1, you may not find the scale out talent because they either are working on large scale problems or would want to work on large scale problems. In your early days, you are mostly hunting for a product market fit.So your problem statement will not even be relevant to the hires. This is the best case, the worst case is that the scale out thinking hampers your product market fit progress. That's the risk you run. So, while it seems to have gone scale out on day 1 in hindsight, it's always always better to start small and scale out as needs arise, even if that means re-architecting the whole stack. It's painful, but doable. It's also a reasonable challenge to entice new hires. You have scaled out a system from degree 1 to degree m. Now scale that out from m to r++. Something like that.
hodgesrm 6y ago
On balance it's probably more important to create (a) stable public APIs backed by tests and (b) an automated deployment process than to get the internal implementation right from the start. These two things help any system at any stage. They also make it easier to handle migration if that's necessary.

by gfodor 6y ago

cortesoft 6y ago

Yeah, I know this is true, and we always try to remember to understand that the engineering decisions that were made at the time were made for good reasons given the circumstances we were facing. It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time. Still wish we had made some different ones, though!

josephg 6y ago
Hear hear! I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Spoilers: That never happened - the longer we used mongo, the more expensive migrating became. We never did manage to justify the migration. As far as I know the team is still stuck with it, and paying the price. (If you're reading this and you know who you are, I'm sorry!)
- rubyist5eva 6y ago
  Never use MongoDB. Ever. It's a rule of thumb that hasn't led me astray yet. http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/ http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...
  - dominotw 6y ago
    > It's a rule of thumb that hasn't led me astray yet. How can you know if your decision is correct if you never use it in the first place. Isn't it like only implementing A version of A/B and concluding that not implementing B version was a good decision.
  - jcpst 6y ago
    Meh. I read the article, and I agree with it. But that is not what you are saying. Don’t use it for relational data. Don’t use it as your primary data store. At the enterprise I work at, I’ve had success delivering features where data science would have some job that aggregates production data and dumps it to kafka. Then we’ll map it and dump it in mongo. Stick a .net core microservice on top for retrieval. Works great, has none of the problems mentioned in that article. Of course, this might not make sense if you’re not big enough to where you need kafka, kubernetes, etc But “Never use it” is certainly not good advice.
  - karmakaze 6y ago
    This part is totally misguided. > What’s missing from MongoDB is a SQL-style join operation, which is the ability to write one query that mashes together the activity stream and all the users that the stream references. Because MongoDB doesn’t have this ability, you end up manually doing that mashup in your application code, instead. Even with a normalized relational database one wouldn't make a single query to return rows from a join of tables T1, T2, T3, etc when there are repeating relations such as the users in the social graph. This might be done sometimes paying for the duplication to save a network round trip, that's all and no different.
    - debaserab2 6y ago
      A list of activities joined to users wouldn't be a problem for an RDBMS. I'm not sure what you mean by "repeating relations".
      - karmakaze 6y ago
        The deep relations (7-table join) example in the link is something I wouldn't do (including the repeated 'user' entities) with an RDBMS, given that I expect some elements of some tables to appear multiple times, that's a waste of I/O and hard to cache. > Users have friends, friends have posts, posts have comments and likes, each comment has one commenter and each like has one liker. I would likely gather all the instances of User id in each context (user, friend, commenter, and liker) and query it separately from the JOIN query. Depends on your data which way works out better on average/worst-cases.
- zozbot234 6y ago
  > I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Yeah, that can't end well. It's a bit ironic from parent poster's viewpoint but "boring", well-understood technology is generally also the easiest to migrate away from. That's one key reason to choose it at the outset even over more "innovative" and perhaps more "ergonomic" choices like mongo, that in reality aren't nearly as simple to deal with.

deepGem 6y ago

The key part you highlight is that of hiring. If you want to be scale ready from day 1, you may not find the scale out talent because they either are working on large scale problems or would want to work on large scale problems. In your early days, you are mostly hunting for a product market fit.So your problem statement will not even be relevant to the hires. This is the best case, the worst case is that the scale out thinking hampers your product market fit progress. That's the risk you run. So, while it seems to have gone scale out on day 1 in hindsight, it's always always better to start small and scale out as needs arise, even if that means re-architecting the whole stack. It's painful, but doable. It's also a reasonable challenge to entice new hires. You have scaled out a system from degree 1 to degree m. Now scale that out from m to r++. Something like that.

hodgesrm 6y ago

On balance it's probably more important to create (a) stable public APIs backed by tests and (b) an automated deployment process than to get the internal implementation right from the start. These two things help any system at any stage. They also make it easier to handle migration if that's necessary.