4 ms·

To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewrit

To be honest the article does realize this, first blaming it on the poor hindsight from original developer (co-founder) and in the conclusion about maybe rewriting the whole thing. It seemed to be all about how to extract the most performance from the lemon they had to deal with. I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/ https://rachelbythebay.com/w/2020/03/07/costly/

lmilcin 5y ago
I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes. This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything. I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing. > poor hindsight from original developer (co-founder) Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less? It is not poor choice, it is lack of competency. You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.
- nomdep 5y ago
  So do you think using Django is stupid? I guess you think the same about every product that uses Ruby on Rails?
  - lmilcin 5y ago
    No, Django is not stupid. It is the decision to choose it to run load that will require 1000s of servers when it could be handled with 5-10 servers in another technology without more development effort.
    - mst 5y ago
      I doubt they expected that level of request load that early on - I imagine the technology choice was made significantly before the whole pandemic thing started.
- taylorhughes 5y ago
  (CH employee here) The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that gives you the best shot at capturing the lightning. Django seemed to help!
  - lmilcin 5y ago
    This is just silly excuse. The job of the cofounder is also to anticipate possible risks. And building your company on an astronomically inefficient technology sounds like a huge risk to me. Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided. Django has nothing special in it that would allow building applications faster than in a lot other frameworks that are also much more efficient. So it is just a matter of simple choice. Nobody expects people to write webapps in C++ or Rust. Just don't choose technology that is famous for being inefficient.
    - jimsimmons 5y ago
      Python is not astronomically inefficient. Instagram serves like a billion users with it. Job of a cofounder is to build what people want. You can always scale in Silicon Valley by hiring people like you. You can’t build another viral app like clubhouse by hiring from the same crowd. This may hurt you but the truth is scaling and software engineering is highly commoditised. That’s the whole point of being in the valley. You can hire people for such things and forget about it. Clubhouse is not a tech company. They don’t have to care about being the best at infra
      - lmilcin 5y ago
        > Python is not astronomically inefficient. Well, it is. It is a fact. https://rachelbythebay.com/w/2020/03/07/costly/ https://rachelbythebay.com/w/2020/03/07/costly/ > Clubhouse is not a tech company. When you spin 1000s of nodes you need some tech competency. Or in other words, if it blew one day and there would be a link to writeup on HN, people would be asking "They had 1000s of servers and nobody competent to maintain it?"
- themarkers 5y ago
  You are right if it’s a technical driving thing. But most are not that case. CPU is much cheaper for scaling a business.

lmilcin 5y ago

I don't know Python or how complex their domain is but the number of workers suggests to me it is not that complex and their application spends most of its time switching contexts and in inefficient frameworks. Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes. This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything. I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing. > poor hindsight from original developer (co-founder) Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less? It is not poor choice, it is lack of competency. You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.

nomdep 5y ago
So do you think using Django is stupid? I guess you think the same about every product that uses Ruby on Rails?
- lmilcin 5y ago
  No, Django is not stupid. It is the decision to choose it to run load that will require 1000s of servers when it could be handled with 5-10 servers in another technology without more development effort.
  - mst 5y ago
    I doubt they expected that level of request load that early on - I imagine the technology choice was made significantly before the whole pandemic thing started.
taylorhughes 5y ago
(CH employee here) The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that gives you the best shot at capturing the lightning. Django seemed to help!
- lmilcin 5y ago
  This is just silly excuse. The job of the cofounder is also to anticipate possible risks. And building your company on an astronomically inefficient technology sounds like a huge risk to me. Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided. Django has nothing special in it that would allow building applications faster than in a lot other frameworks that are also much more efficient. So it is just a matter of simple choice. Nobody expects people to write webapps in C++ or Rust. Just don't choose technology that is famous for being inefficient.
  - jimsimmons 5y ago
    Python is not astronomically inefficient. Instagram serves like a billion users with it. Job of a cofounder is to build what people want. You can always scale in Silicon Valley by hiring people like you. You can’t build another viral app like clubhouse by hiring from the same crowd. This may hurt you but the truth is scaling and software engineering is highly commoditised. That’s the whole point of being in the valley. You can hire people for such things and forget about it. Clubhouse is not a tech company. They don’t have to care about being the best at infra
    - lmilcin 5y ago
      > Python is not astronomically inefficient. Well, it is. It is a fact. https://rachelbythebay.com/w/2020/03/07/costly/ https://rachelbythebay.com/w/2020/03/07/costly/ > Clubhouse is not a tech company. When you spin 1000s of nodes you need some tech competency. Or in other words, if it blew one day and there would be a link to writeup on HN, people would be asking "They had 1000s of servers and nobody competent to maintain it?"
themarkers 5y ago
You are right if it’s a technical driving thing. But most are not that case. CPU is much cheaper for scaling a business.