7 ms·

I like this. I recall a startup some years back with a vision to build something complex. The founder sat down with everyone and said that we had an important d

I like this. I recall a startup some years back with a vision to build something complex. The founder sat down with everyone and said that we had an important decision to make: Do we build something that grows or do we build something that evolves? Neither way forward was presented as an obvious win. Building something that grows meant overengineering and trying to forecast the future. Building something that evolves meant constantly rewriting things, taking time away from new development. Twitter seems to have taken the "evolution" route, and it has served them well. I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time.

div 15y ago
Building something that evolves feels like the smarter intuitive decision. It allows you to focus on features and pivoting over ceremony (boilerplatey / architecture stuff) and when you finally start hitting those performance limits, you have a luxury problem. It will also be more apparent where to invest time spent scaling your product, whereas if you try to optimize out of the gate, you may still hit unforeseen performance issues.
- kenjackson 15y ago
  It allows you to focus on features and pivoting over ceremony This always sounds nice, but I've found that ceremony almost never really reduces productivity by much. The real wins/losses almost never have to do with ceremony related features of language, but rather architectural and framework components. You build something that evolves by bringing in the smallest architecture and least fx components and then build. The language you use may dictate the fx components to some extent -- otherwise it's typically just to make developers feel happy (which is important, but really is just about morale more than anything truly inherent in the productivity of the language).
  - div 15y ago
    When I said ceremony, I did mean working around architectural up-front decisions. I was arguing in favor of 'growing something' as opposed to 'designing something' with regards to a hopefully growing userbase. Language choice may be of lesser importance as you say, but I do feel that some languages fit the growing strategy better while others have a more design up front feel to them.
nupark2 15y ago
I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time. I'm not sure why you think that, or why you think Twiiter had a 'grow' vs 'evolve' strategy. Their approach seemed not nearly so self-aware, and resulted in serious, core deficiencies that required considerable time and expense to correct. From an external position, it very much did not look like a rational considered approach to solving he problem through 'evolution'.
- raganwald 15y ago
  I never suggested they were self-aware, only that they evolved. Therefore, I do not suggest that they decided anything. All evolution strategies involve serious, core deficiencies, that's the trade-off, that's why it isn't 100% obviously better to evolve than to grow. But I do suggest that evolution--however it came about at Twitter HQ--has worked out for them.
  - deleted 15y ago
    [deleted]
angryasian 15y ago
don't know why everyone assumes jvm language = over engineering. I agree that some jvm language code can be more robust, but the reality is most of us should live in our test suites. I would even say that Java test suites start and run faster than the standard rails tests ( without all the things to make it faster like spork). The frameworks argument of configuration vs convention I think is moot as most jvm language frameworks are taking a more conventional approach, but allowing for greater configuration now. on the other side the reality of a startup is you're going to push out so much code / features / product based on demand so fast that a lot of the things that you consider building for the future will be thrown out the window pretty fast. Just build it, see if it works, and move to the next thing.
eropple 15y ago
These aren't incompatible, though. Building something that "grows", even to very large scale, is not that difficult, and still leaves the door open to evolutionary improvements and changes to the fundamental architecture over time if they become necessary. The key is, when designing your growth architecture (and I don't consider it overengineering--a few simple decisions allow you to scale pretty well to a fairly large load), is to code to interface rather than code to implementation. If things evolve/change over time, you've already got half the work done for you because things don't spontaneously break on change.
- raganwald 15y ago
  It is incredibly easy to design for growth when looking in a rear view mirror. For example, it is easy NOW to see that Twitter's problem was scaling, and that they should have planned to grow their scalability while evolving their user experience. However, at the time it might have gone the other way: Perhaps their user base and volume might have increased at a slower place, leaving plenty of time to evolve their scalability, while their user experience might have required relentless change. In which case, they should have planned for their user experience to "scale" rather than their infrastructure.
  - eropple 15y ago
    True. That said, designing for scalability isn't that hard, and does not take that much more time than just bunging out whatever comes first to mind.
    - jshen 15y ago
      You're assuming that you know ahead of time what the bottlenecks are, and more importantly, which trade offs to make, because scaling is almost always about trade offs. A scalable solution is usually a more rigid solution as well, and that can be a problem for a startup that may need to try a few ideas before they hit on what works.
- dasil003 15y ago
  This is armchair architect thinking. Sure, coding to the interface makes scaling easier, it also makes maintenance, testing and everything else easier: it's just a good idea. But the devil is in the details. When you have to scale like Twitter you are getting into the territory where you have to do a lot of legwork just to figure out how to scale a particular interface. Performance and consistency issues start to gnaw away at the ideal interface you carefully designed back when you were 20k users and 10 reqs/s.
grok2 15y ago
Twitter's approach (atleast in the early stages) has been more of Darwinian evolution ("Shit, we have to do something about this or we're screwed") rather than planned evolution :-). It has probably served them well -- the growth they saw in usage meant they could without thinking too much whether something was worth it, evolve to more sophisticated solutions as time went by.
bborud 15y ago
The problem with Java is mainly that people think they have to use all the horribly complex, intrusive, badly written, underperforming frameworks that exist in the Java sphere. And then, of course, you will inevitably be screwed. To this day it takes real effort to convince Java programmers that a lot of "best practices" are anything but. For instance I still see Java programmers use frameworks that require them to maintain mountains of flimsy XML configs for things that they ought to have done in code. Both to get the benefit of letting the compiler do the work of weeding out boneheaded errors and to get rid of unnecessary flexibility that just leads to more work and more confusing, hard to read code. Java is a great language in which you can be very productive. But productivity means that you have to crack down on people who drag along J2EE-crap, or whatever crap was invented to make J2EE-crap slightly less crap. It also means you have to mentor people actively and harass anyone who even thinks of doing in XML what can be accomplished much faster in testable, compiled Java code.
- nerd_in_rage 15y ago
  Yep, I saw this same thing. Mountains of XML deployment descriptors / "component wiring" (some "Spring" garbage), needlessly verbose and usually unneeded mapper classes, useless interfaces (my rule is if you are only going to have ONE implementation save the interface for later after you know you got it right and need it), extra layers that do nothing (except "map", of course), etc. kill me. Makes me want to kill myself.
  - bluelu 15y ago
    That's one of the reasons I left my first and last employer. Tons of compnontent wiring written in xml, tons of configuration directives, but the end only the defaults were used anyway. Impossible to do updates because everything could break.
  - GrooveStomp 15y ago
    This exactly describes my very brief experience working with Java professionally. Back in university it was no big deal - the differences between C++ and Java weren't that big. In a production environment, though; ... XML, XML, XML! All the XML-wrapper tools for Java are supposedly the reason why there is so much XML config stuff going on - but the result is that you end up writing soo much code in XML instead of the actual language that you're supposed to be using. It's like somebody looked quickly at the MVC pattern, got it all wrong, and now Java has very tight coupling between XML and Java code for any production environment that hopes to leverage existing tools.
6ren 15y ago
By "something that grows", I think you mean "something that scales". It confused me, because Brooks talks about "growing' software (as opposed to building/planning it), which is I think what you mean by "something that evolves". FWIW, I think the standard wisdom today is to evolve (your term) software (e.g. agile ideas of YAGNI, DTSTTCPW), and only to scale/plan it if you have a very clear idea of what you're doing (e.g. frozen specs, which exist in some government/military domains; long-term standards; mathematics) - and you also know how to do it, having done it a few times before.
amurmann 15y ago
I don't think Twitter would have existed, if they had had to take the "grow"/plan for scale path. As I understand, Twitter originally was little tool, that the founder just wanted to have for them selves. If they had thought, they would have to take months to implement it in a scalable version right away, they might not have done it. I also think it's a good idea to throw a bunch of small, simple stuff on the wall and see what sticks. If everything is planned to be a big engineering effort, made to scale to millions of users, you spend a lot of time building stuff nobody will care about. That's why frameworks like Rails are great. They probably aren't perfect for your use case, esp. once you want to start scaling to crazy user numbers (although high user number will probably work very well), but they help you get a useable version out of the door crazy fast, compared to an all custom implementation.

div 15y ago

Building something that evolves feels like the smarter intuitive decision. It allows you to focus on features and pivoting over ceremony (boilerplatey / architecture stuff) and when you finally start hitting those performance limits, you have a luxury problem. It will also be more apparent where to invest time spent scaling your product, whereas if you try to optimize out of the gate, you may still hit unforeseen performance issues.

kenjackson 15y ago
It allows you to focus on features and pivoting over ceremony This always sounds nice, but I've found that ceremony almost never really reduces productivity by much. The real wins/losses almost never have to do with ceremony related features of language, but rather architectural and framework components. You build something that evolves by bringing in the smallest architecture and least fx components and then build. The language you use may dictate the fx components to some extent -- otherwise it's typically just to make developers feel happy (which is important, but really is just about morale more than anything truly inherent in the productivity of the language).
- div 15y ago
  When I said ceremony, I did mean working around architectural up-front decisions. I was arguing in favor of 'growing something' as opposed to 'designing something' with regards to a hopefully growing userbase. Language choice may be of lesser importance as you say, but I do feel that some languages fit the growing strategy better while others have a more design up front feel to them.

nupark2 15y ago

I'm not at all sure they would be this successful if they'd tried to build on the JVM right from the start using the technologies available at the time. I'm not sure why you think that, or why you think Twiiter had a 'grow' vs 'evolve' strategy. Their approach seemed not nearly so self-aware, and resulted in serious, core deficiencies that required considerable time and expense to correct. From an external position, it very much did not look like a rational considered approach to solving he problem through 'evolution'.

raganwald 15y ago
I never suggested they were self-aware, only that they evolved. Therefore, I do not suggest that they decided anything. All evolution strategies involve serious, core deficiencies, that's the trade-off, that's why it isn't 100% obviously better to evolve than to grow. But I do suggest that evolution--however it came about at Twitter HQ--has worked out for them.
- deleted 15y ago
  [deleted]

angryasian 15y ago

don't know why everyone assumes jvm language = over engineering. I agree that some jvm language code can be more robust, but the reality is most of us should live in our test suites. I would even say that Java test suites start and run faster than the standard rails tests ( without all the things to make it faster like spork). The frameworks argument of configuration vs convention I think is moot as most jvm language frameworks are taking a more conventional approach, but allowing for greater configuration now. on the other side the reality of a startup is you're going to push out so much code / features / product based on demand so fast that a lot of the things that you consider building for the future will be thrown out the window pretty fast. Just build it, see if it works, and move to the next thing.

eropple 15y ago

These aren't incompatible, though. Building something that "grows", even to very large scale, is not that difficult, and still leaves the door open to evolutionary improvements and changes to the fundamental architecture over time if they become necessary. The key is, when designing your growth architecture (and I don't consider it overengineering--a few simple decisions allow you to scale pretty well to a fairly large load), is to code to interface rather than code to implementation. If things evolve/change over time, you've already got half the work done for you because things don't spontaneously break on change.

raganwald 15y ago
It is incredibly easy to design for growth when looking in a rear view mirror. For example, it is easy NOW to see that Twitter's problem was scaling, and that they should have planned to grow their scalability while evolving their user experience. However, at the time it might have gone the other way: Perhaps their user base and volume might have increased at a slower place, leaving plenty of time to evolve their scalability, while their user experience might have required relentless change. In which case, they should have planned for their user experience to "scale" rather than their infrastructure.
- eropple 15y ago
  True. That said, designing for scalability isn't that hard, and does not take that much more time than just bunging out whatever comes first to mind.
  - jshen 15y ago
    You're assuming that you know ahead of time what the bottlenecks are, and more importantly, which trade offs to make, because scaling is almost always about trade offs. A scalable solution is usually a more rigid solution as well, and that can be a problem for a startup that may need to try a few ideas before they hit on what works.
dasil003 15y ago
This is armchair architect thinking. Sure, coding to the interface makes scaling easier, it also makes maintenance, testing and everything else easier: it's just a good idea. But the devil is in the details. When you have to scale like Twitter you are getting into the territory where you have to do a lot of legwork just to figure out how to scale a particular interface. Performance and consistency issues start to gnaw away at the ideal interface you carefully designed back when you were 20k users and 10 reqs/s.

grok2 15y ago

Twitter's approach (atleast in the early stages) has been more of Darwinian evolution ("Shit, we have to do something about this or we're screwed") rather than planned evolution :-). It has probably served them well -- the growth they saw in usage meant they could without thinking too much whether something was worth it, evolve to more sophisticated solutions as time went by.

bborud 15y ago

The problem with Java is mainly that people think they have to use all the horribly complex, intrusive, badly written, underperforming frameworks that exist in the Java sphere. And then, of course, you will inevitably be screwed. To this day it takes real effort to convince Java programmers that a lot of "best practices" are anything but. For instance I still see Java programmers use frameworks that require them to maintain mountains of flimsy XML configs for things that they ought to have done in code. Both to get the benefit of letting the compiler do the work of weeding out boneheaded errors and to get rid of unnecessary flexibility that just leads to more work and more confusing, hard to read code. Java is a great language in which you can be very productive. But productivity means that you have to crack down on people who drag along J2EE-crap, or whatever crap was invented to make J2EE-crap slightly less crap. It also means you have to mentor people actively and harass anyone who even thinks of doing in XML what can be accomplished much faster in testable, compiled Java code.

nerd_in_rage 15y ago
Yep, I saw this same thing. Mountains of XML deployment descriptors / "component wiring" (some "Spring" garbage), needlessly verbose and usually unneeded mapper classes, useless interfaces (my rule is if you are only going to have ONE implementation save the interface for later after you know you got it right and need it), extra layers that do nothing (except "map", of course), etc. kill me. Makes me want to kill myself.
- bluelu 15y ago
  That's one of the reasons I left my first and last employer. Tons of compnontent wiring written in xml, tons of configuration directives, but the end only the defaults were used anyway. Impossible to do updates because everything could break.
- GrooveStomp 15y ago
  This exactly describes my very brief experience working with Java professionally. Back in university it was no big deal - the differences between C++ and Java weren't that big. In a production environment, though; ... XML, XML, XML! All the XML-wrapper tools for Java are supposedly the reason why there is so much XML config stuff going on - but the result is that you end up writing soo much code in XML instead of the actual language that you're supposed to be using. It's like somebody looked quickly at the MVC pattern, got it all wrong, and now Java has very tight coupling between XML and Java code for any production environment that hopes to leverage existing tools.

6ren 15y ago

By "something that grows", I think you mean "something that scales". It confused me, because Brooks talks about "growing' software (as opposed to building/planning it), which is I think what you mean by "something that evolves". FWIW, I think the standard wisdom today is to evolve (your term) software (e.g. agile ideas of YAGNI, DTSTTCPW), and only to scale/plan it if you have a very clear idea of what you're doing (e.g. frozen specs, which exist in some government/military domains; long-term standards; mathematics) - and you also know how to do it, having done it a few times before.

amurmann 15y ago

I don't think Twitter would have existed, if they had had to take the "grow"/plan for scale path. As I understand, Twitter originally was little tool, that the founder just wanted to have for them selves. If they had thought, they would have to take months to implement it in a scalable version right away, they might not have done it. I also think it's a good idea to throw a bunch of small, simple stuff on the wall and see what sticks. If everything is planned to be a big engineering effort, made to scale to millions of users, you spend a lot of time building stuff nobody will care about. That's why frameworks like Rails are great. They probably aren't perfect for your use case, esp. once you want to start scaling to crazy user numbers (although high user number will probably work very well), but they help you get a useable version out of the door crazy fast, compared to an all custom implementation.