6 ms·

I used to be very interested in Software Architecture, in fact I've read many of the papers cited here. When I did a startup many years ago, I committed the mi

by Maro 9y ago

I used to be very interested in Software Architecture, in fact I've read many of the papers cited here. When I did a startup many years ago, I committed the mistake of paying too much attention to the architecture of the software [1] I was writing, and not enough attention to the product/customer side of it. The last couple of years I've been de-emphasizing software architecture as an interest, and have been paying much more attention to how product teams build successful products, what the patterns are, etc. I was lucky enough to work at Facebook for a while and got to see (and learn) a very successful working model of product development. So, while I'm not saying that software architecture is not important (it is), also pay attention to the product/customer side: what choices (software, organizational, hiring, business) allow you to move fast and iterate, to release early and often, to run A/B tests, etc. I think good software engineers are just as much product guys (and data guys) as they are software guys. - [1] https://github.com/scalien/scaliendb https://github.com/scalien/scaliendb

Clubber 9y ago
Agreed, good architecture doesn't come into play until you need to add a lot of complex features or scale significantly. Initially architecture doesn't really mean squat. I would concentrate on making the codebase flexible, but that's about it. I've regretted making some of my software with a cool but complex architecture when I should have been focused on what the product does.
- humanrebar 9y ago
  > I would concentrate on making the codebase flexible, but that's about it Arguably that's the property of good architecture. Standardizing your entire codebase also works, but only as long as those initial standards stay smart (1), and even then, that relies on discipline, tools, and incentives that are typically not all in place. (1) A standard might be "all webservers will be written in Java 6". The benefits of choosing that standard tend to be front-loaded with a gradual decline into net-negative with no affordable path to a better standard.
  - qznc 9y ago
    To design your software perfectly requires to know the future. You want the things that will change flexible and the rest simple. In practice, you will not hit the sweet spot and either over-engineer too much or hardcode too much (or both in different parts).
    - gregmac 9y ago
      I tend to place the most emphasis on loose coupling, and being very strict about interfaces between components -- both in terms of documentation and addition/changes. Ideally documentation includes unit or integration tests. This applies to separate high level services as well as it does to internal code structure (classes/modules/etc). Well documented and tested interfaces of a component make that component easy to replace or rewrite if needed. Nearly anytime I have worked on or been involved with a project where I or someone else tried to design specifically for a vague future requirement that wasn't entirely clear or guaranteed, it didn't work out: either the requirement never happened, or was so materially different that what we wrote was wrong. As a result, the code we wrote went unused or worse, just got in the way and ended up as unnecessary technical debt.
- madisonmay 9y ago
  In Knuth's words: "Premature optimization is the root of all evil". Although I guess in this context, you're emphasizing that it's more valuable to solve the right problem poorly than it is to solve the wrong problem well.
  - Clubber 9y ago
    I didn't say anything about solving the right problem poorly, but you are probably speaking a truth, depending on what poorly means. It certainly isn't ideal. I've seen solutions designed so poorly that when it came time to add features that paying clients wanted, it was significantly more difficult (significant risk) than if it was designed well. I'd say solve the right problem well, without spending time on requirements you don't yet have. These would include a large scalability requirement and over-modularization, like breaking out 10 "micro services."
thewarrior 9y ago
Could you elaborate on the insights you gained ? At least on a high level . It would help a lot of people here .
- Maro 9y ago
  I think most of the things people know, it's covered in books like Lean Startup, etc. It's a bit like eating well and working out. We all know we should do it, but most people don't actually eat well and work out. Then, when you see somebody who does it and looks great, you ask them, "What's your secret?". But it's not a secret, it's just that most people don't do it, because it's hard :) One story: When I was at FB, I happen to know that a team of size S conducted X experiments in 6 months (I can't disclose the number). As it happens, I have worked in similar size teams in other companies, and there the number was ~X/20, and sometimes 0. It don't matter how good your software architecture is, if you're trying out just 1 thing instead of 20 things... I call this velocity. Another good story: many semi-successful companies end up in a place where there is an initial product/software which gets them a lot of growth, and then ~5 years into the startup, the bigger, more mature, bigger team decided that the old legacy code is holding them back, and they're going to REWRITE IT. Maybe in some fancy new language, or a fancy new architecture like micro-services. The estimate is 6-9 months to get to first light. But it will probably end up taking 3-5 years, because the legacy had a lot of fine-tuning in it, and it turns out many of the problems are hard to fix, moving the production to a new thing is REALLY HARD, and all those fancy new technologies are actually far from perfect, plus the current team doesn't have a lot of experience with it. Compare to this what Facebook did with its PHP codebase: at some point it became a bottleneck (the crappy language and the runtime speed), but there was a never a from-the-ground-rewrite. Instead they ended up writing several iterations of better runtimes, and since they were changing the runtime anyway, they "fixed up" the language (while keeping it mostly backward compatible). The new language is called Hack and the new runtime is called HHVM. The cool thing is, in all this time, there wasn't a rewrite, so they were able to keep shipping new features on Facebook, run A/B tests, iterate on the product. Compare this to one of the companies I worked at, where they did a rewrite , and now customers have to chose between the old and a new thing, it's not transparent, because <software issues>. There's a book about Hack/HHVM, one of the Facebook guys wrote it, iirc the first chapter is about this whole story. See this blog post for links: http://bytepawn.com/hack-hhvm-second-system-effect.html http://bytepawn.com/hack-hhvm-second-system-effect.html In general, the principles I've seen to work really well: - write good code, but don't do big rewrites - cont. delivery: always be shipping to master and production in small increments (don't have big git branches that aren't in production, at the end it will be scary to merge it and put it into production) - cont. integration: have tests and run them on every commit (you probably need some testing nazis to enforce this...) - if you (your team) can't write a good monolith, you (your team) also can't write a good MSA - invest heavily in linting and other automated ways to catch and conform code when it is committed - programming language doesn't matter that much, just pick one for each domain (eg. web, mobile, etc), and stop thinking about it, and don't let people waste their time on arguing over it too much; instead invest heavily in tooling that supports all the other aspects (like code reviews, perf, experimentation, etc) - 1 other person should review and okay the code before it goes into production - aggressivley remove obstacles from shipping stuff to production - make it easy to run experiments, and run a lot of experiments - don't hire people who just want to write code, or think their job stops there
ChuckMcM 9y ago
I had a similar journey although came away from that point headed in a slightly different direction. As you have studied the processes by which software is developed have you noticed a correlation between developing pieces that are architecturally important (many things are touched by the choices) and planning investment by the team?
- Maro 9y ago
  What I've seen is that most orgs, once they reach a certain size (100-200 people), start to spend a significant time planning. Part of this is human nature: once there are enough people, there will be managers, and plans are something managers show their hire ups to prove they're in control and doing a good job. But, I don't want to sound cynical, clearly planning is useful. I used to think that spending too much time on planning is a waste of time , but then I noticed that teams at Facebook also spend a lot of time at the beginning of each half (6 mo) doing planning. However, I wasn't able to pick up tricks or patterns that made FB good at this, other than obvious/useless things like all the people making the plans were really smart / domain experts. I did notice that the plans/planning process was very lose, there was no methodology, often the outcome was a bunch of bullet points. And they were very conscious about the value of the plan: they knew that if things go well and they move quickly, probably a lot of things will change in 3-4 months, and they will diverge from the plan. The value is thinking things through, agreeing on goals, spinning up teams, building things (eg. dashboards) to track progress, etc. Another story about the value of planning: at another company we hired a manager, and one of the things this guy did was introduce Capacity Planning. At the core it's a simple thing: make a big matrix, where the rows are the projects people want to work on, and the columns are the available people/teams. Collect all the projects, and make people put down their estimates for each (requires X time from team Y, etc). The value is, it's a very explicit way to show what people want to work on, obviously it will be more than you can actually do. So then you can make a very explicit choice about what you're doing and what you're not doing in the next X months. Back when we did this, there was an interesting insight the first time around: since we could count man days, we learned we're spending iirc ~70% of our time keeping the lights on (infra, etc), and only ~30% working on new things. But despite this insight, I thought it's a waste of time, it's too slow/formal/sluggish. But then I saw FB also spends significant time on planning (although it's much less formal, because they can get away with it, for a long list of reasons), so I thought okay, spending time planning is probably the way to go. Then I went to a company which was suffering from a focus problem. Every 2 weeks they'd have a prioritization meeting, where projects would get re-prioritized, which leads to obvious problems. Here I realized, for this problem, Capacity Planning would be a good thing, because one of the things it gets you is a long(er) term commitment to certain projects. Overall, my takeaway is to spend 10-20% of time on planning and related meta-activities. Be conscious that the value is in thinking things through etc, and the actual path may diverge, but that's okay. Depending on the culture, there are a number of things that planning gives you, eg. a commitment to work on X for Y time. Make metrics and dashboards to track things, but don't overdo it.
gobengo 9y ago
> I think good software engineers are just as much product guys (and data guys) as they are software guys. ya often times they are women too
- wwosik 9y ago
  pretty sure 'guy' in modern American English is gender- and age-less. As in two 70 old ladies could be addressed as "you guys" too. Surprised me a lot when I was there
  - seanmcdirmid 9y ago
    It depends on what region you are in. Guy in the UK I think means male as in Guy Fawkes. In the USA, Guy is often genderless except in the south where gal can be used without being very awkward. In general, you should just steer clear of the word, however, as opinions about it are diverse.
    - OJFord 9y ago
      > Guy in the UK I think means male as in Guy Fawkes. That's a name. Nobody's going to think you're only talking about men named 'Guy'. Context is king (ooor queen!) - it can certainly be either alone, or mixed.
      - seanmcdirmid 9y ago
        http://www.word-detective.com/2009/08/guy/ http://www.word-detective.com/2009/08/guy/
    - Silhouette 9y ago
      Guy isn't necessarily male in the UK, except when used as a name and maybe in the context of "guys and girls". "Hi guys" is a common greeting here when addressing a mixed group.
  - madisonmay 9y ago
    Like most language, meaning is dependent on context. But in general it seems to be good practice to steer clear from pronouns associated with a particular gender when there are other perfectly good terms to be used (In this context, "people", "folks", or even "focused".)
kenver 9y ago
There’s a big difference between being tunnel visioned on architecture and not getting anything done, and the very common scenario of not seeming to care at all. Having been a contractor at many companies Im pretty sick and saddened of seeing how common it is for every layer of the product to be implemented as a global state Singleton.
Silhouette 9y ago
I was lucky enough to work at Facebook for a while and got to see (and learn) a very successful working model of product development. Facebook has been extremely successful commercially, but I think it's dangerous to read too much into how a unicorn like that develops its software, partly because there is survivorship bias at work here, and partly because the most important things Facebook has achieved have relatively little to do with software. Facebook's golden goose is the network effect. Once it reached a critical mass of users, it was all but unstoppable. Arguably its most impressive technical feat was achieving enough scalability in its infrastructure that it could keep up with that many users and that much data. That was a remarkable success story by any standard, but while it surely has a software element, no doubt it involved much more than just code. On the other hand, to a first approximation Facebook has had infinite resources for most of its existence. It operates in an online environment where problems can be fixed even in production. And it doesn't really do anything that is going to cause catastrophic, unfixable consequences if something does break for a while. That is a list of luxuries that few software development teams enjoy, and what works if you can make those assumptions won't necessarily be a good idea if you can't. As you say, you do have to pay attention to other factors as well, but there are not many organisations that have as much room to manoeuvre on the software side as Facebook does.
- Maro 9y ago
  When you're looking at templates for success, then looking at the successful companies (survivorship bias) is a good start. The fallacy is when it turns out that there are lots of other companies doing the same thing, but those companies aren't successful. This is the problem with all those blog posts examining the daily habits of highly successful people (like, they wake up early, or, they eat energy bars to save time, etc). Clearly, just doing these things won't make you a billionaire. For what I wrote, I believe this doesn't apply, because of the stark difference between the culture I experienced at FB and other companies. (Admittedly, this is not a stat.sign. sample.) So in my experience it's _not_ true that other companies are doing the same thing... Having said that, you're right that FB is a one in a million company, and probably nobody reading this will be the next M.Zuckerberg... But still, if you want to be the best in your domain (eg. best todo app), I strongly think these are good patterns to follow.

by Maro 9y ago

Clubber 9y ago

Agreed, good architecture doesn't come into play until you need to add a lot of complex features or scale significantly. Initially architecture doesn't really mean squat. I would concentrate on making the codebase flexible, but that's about it. I've regretted making some of my software with a cool but complex architecture when I should have been focused on what the product does.

humanrebar 9y ago
> I would concentrate on making the codebase flexible, but that's about it Arguably that's the property of good architecture. Standardizing your entire codebase also works, but only as long as those initial standards stay smart (1), and even then, that relies on discipline, tools, and incentives that are typically not all in place. (1) A standard might be "all webservers will be written in Java 6". The benefits of choosing that standard tend to be front-loaded with a gradual decline into net-negative with no affordable path to a better standard.
- qznc 9y ago
  To design your software perfectly requires to know the future. You want the things that will change flexible and the rest simple. In practice, you will not hit the sweet spot and either over-engineer too much or hardcode too much (or both in different parts).
  - gregmac 9y ago
    I tend to place the most emphasis on loose coupling, and being very strict about interfaces between components -- both in terms of documentation and addition/changes. Ideally documentation includes unit or integration tests. This applies to separate high level services as well as it does to internal code structure (classes/modules/etc). Well documented and tested interfaces of a component make that component easy to replace or rewrite if needed. Nearly anytime I have worked on or been involved with a project where I or someone else tried to design specifically for a vague future requirement that wasn't entirely clear or guaranteed, it didn't work out: either the requirement never happened, or was so materially different that what we wrote was wrong. As a result, the code we wrote went unused or worse, just got in the way and ended up as unnecessary technical debt.
madisonmay 9y ago
In Knuth's words: "Premature optimization is the root of all evil". Although I guess in this context, you're emphasizing that it's more valuable to solve the right problem poorly than it is to solve the wrong problem well.
- Clubber 9y ago
  I didn't say anything about solving the right problem poorly, but you are probably speaking a truth, depending on what poorly means. It certainly isn't ideal. I've seen solutions designed so poorly that when it came time to add features that paying clients wanted, it was significantly more difficult (significant risk) than if it was designed well. I'd say solve the right problem well, without spending time on requirements you don't yet have. These would include a large scalability requirement and over-modularization, like breaking out 10 "micro services."

thewarrior 9y ago

Could you elaborate on the insights you gained ? At least on a high level . It would help a lot of people here .

Maro 9y ago
I think most of the things people know, it's covered in books like Lean Startup, etc. It's a bit like eating well and working out. We all know we should do it, but most people don't actually eat well and work out. Then, when you see somebody who does it and looks great, you ask them, "What's your secret?". But it's not a secret, it's just that most people don't do it, because it's hard :) One story: When I was at FB, I happen to know that a team of size S conducted X experiments in 6 months (I can't disclose the number). As it happens, I have worked in similar size teams in other companies, and there the number was ~X/20, and sometimes 0. It don't matter how good your software architecture is, if you're trying out just 1 thing instead of 20 things... I call this velocity. Another good story: many semi-successful companies end up in a place where there is an initial product/software which gets them a lot of growth, and then ~5 years into the startup, the bigger, more mature, bigger team decided that the old legacy code is holding them back, and they're going to REWRITE IT. Maybe in some fancy new language, or a fancy new architecture like micro-services. The estimate is 6-9 months to get to first light. But it will probably end up taking 3-5 years, because the legacy had a lot of fine-tuning in it, and it turns out many of the problems are hard to fix, moving the production to a new thing is REALLY HARD, and all those fancy new technologies are actually far from perfect, plus the current team doesn't have a lot of experience with it. Compare to this what Facebook did with its PHP codebase: at some point it became a bottleneck (the crappy language and the runtime speed), but there was a never a from-the-ground-rewrite. Instead they ended up writing several iterations of better runtimes, and since they were changing the runtime anyway, they "fixed up" the language (while keeping it mostly backward compatible). The new language is called Hack and the new runtime is called HHVM. The cool thing is, in all this time, there wasn't a rewrite, so they were able to keep shipping new features on Facebook, run A/B tests, iterate on the product. Compare this to one of the companies I worked at, where they did a rewrite , and now customers have to chose between the old and a new thing, it's not transparent, because <software issues>. There's a book about Hack/HHVM, one of the Facebook guys wrote it, iirc the first chapter is about this whole story. See this blog post for links: http://bytepawn.com/hack-hhvm-second-system-effect.html http://bytepawn.com/hack-hhvm-second-system-effect.html In general, the principles I've seen to work really well: - write good code, but don't do big rewrites - cont. delivery: always be shipping to master and production in small increments (don't have big git branches that aren't in production, at the end it will be scary to merge it and put it into production) - cont. integration: have tests and run them on every commit (you probably need some testing nazis to enforce this...) - if you (your team) can't write a good monolith, you (your team) also can't write a good MSA - invest heavily in linting and other automated ways to catch and conform code when it is committed - programming language doesn't matter that much, just pick one for each domain (eg. web, mobile, etc), and stop thinking about it, and don't let people waste their time on arguing over it too much; instead invest heavily in tooling that supports all the other aspects (like code reviews, perf, experimentation, etc) - 1 other person should review and okay the code before it goes into production - aggressivley remove obstacles from shipping stuff to production - make it easy to run experiments, and run a lot of experiments - don't hire people who just want to write code, or think their job stops there

ChuckMcM 9y ago

I had a similar journey although came away from that point headed in a slightly different direction. As you have studied the processes by which software is developed have you noticed a correlation between developing pieces that are architecturally important (many things are touched by the choices) and planning investment by the team?

Maro 9y ago
What I've seen is that most orgs, once they reach a certain size (100-200 people), start to spend a significant time planning. Part of this is human nature: once there are enough people, there will be managers, and plans are something managers show their hire ups to prove they're in control and doing a good job. But, I don't want to sound cynical, clearly planning is useful. I used to think that spending too much time on planning is a waste of time , but then I noticed that teams at Facebook also spend a lot of time at the beginning of each half (6 mo) doing planning. However, I wasn't able to pick up tricks or patterns that made FB good at this, other than obvious/useless things like all the people making the plans were really smart / domain experts. I did notice that the plans/planning process was very lose, there was no methodology, often the outcome was a bunch of bullet points. And they were very conscious about the value of the plan: they knew that if things go well and they move quickly, probably a lot of things will change in 3-4 months, and they will diverge from the plan. The value is thinking things through, agreeing on goals, spinning up teams, building things (eg. dashboards) to track progress, etc. Another story about the value of planning: at another company we hired a manager, and one of the things this guy did was introduce Capacity Planning. At the core it's a simple thing: make a big matrix, where the rows are the projects people want to work on, and the columns are the available people/teams. Collect all the projects, and make people put down their estimates for each (requires X time from team Y, etc). The value is, it's a very explicit way to show what people want to work on, obviously it will be more than you can actually do. So then you can make a very explicit choice about what you're doing and what you're not doing in the next X months. Back when we did this, there was an interesting insight the first time around: since we could count man days, we learned we're spending iirc ~70% of our time keeping the lights on (infra, etc), and only ~30% working on new things. But despite this insight, I thought it's a waste of time, it's too slow/formal/sluggish. But then I saw FB also spends significant time on planning (although it's much less formal, because they can get away with it, for a long list of reasons), so I thought okay, spending time planning is probably the way to go. Then I went to a company which was suffering from a focus problem. Every 2 weeks they'd have a prioritization meeting, where projects would get re-prioritized, which leads to obvious problems. Here I realized, for this problem, Capacity Planning would be a good thing, because one of the things it gets you is a long(er) term commitment to certain projects. Overall, my takeaway is to spend 10-20% of time on planning and related meta-activities. Be conscious that the value is in thinking things through etc, and the actual path may diverge, but that's okay. Depending on the culture, there are a number of things that planning gives you, eg. a commitment to work on X for Y time. Make metrics and dashboards to track things, but don't overdo it.

gobengo 9y ago

> I think good software engineers are just as much product guys (and data guys) as they are software guys. ya often times they are women too

wwosik 9y ago
pretty sure 'guy' in modern American English is gender- and age-less. As in two 70 old ladies could be addressed as "you guys" too. Surprised me a lot when I was there
- seanmcdirmid 9y ago
  It depends on what region you are in. Guy in the UK I think means male as in Guy Fawkes. In the USA, Guy is often genderless except in the south where gal can be used without being very awkward. In general, you should just steer clear of the word, however, as opinions about it are diverse.
  - OJFord 9y ago
    > Guy in the UK I think means male as in Guy Fawkes. That's a name. Nobody's going to think you're only talking about men named 'Guy'. Context is king (ooor queen!) - it can certainly be either alone, or mixed.
    - seanmcdirmid 9y ago
      http://www.word-detective.com/2009/08/guy/ http://www.word-detective.com/2009/08/guy/
  - Silhouette 9y ago
    Guy isn't necessarily male in the UK, except when used as a name and maybe in the context of "guys and girls". "Hi guys" is a common greeting here when addressing a mixed group.
- madisonmay 9y ago
  Like most language, meaning is dependent on context. But in general it seems to be good practice to steer clear from pronouns associated with a particular gender when there are other perfectly good terms to be used (In this context, "people", "folks", or even "focused".)

kenver 9y ago

There’s a big difference between being tunnel visioned on architecture and not getting anything done, and the very common scenario of not seeming to care at all. Having been a contractor at many companies Im pretty sick and saddened of seeing how common it is for every layer of the product to be implemented as a global state Singleton.

Silhouette 9y ago

I was lucky enough to work at Facebook for a while and got to see (and learn) a very successful working model of product development. Facebook has been extremely successful commercially, but I think it's dangerous to read too much into how a unicorn like that develops its software, partly because there is survivorship bias at work here, and partly because the most important things Facebook has achieved have relatively little to do with software. Facebook's golden goose is the network effect. Once it reached a critical mass of users, it was all but unstoppable. Arguably its most impressive technical feat was achieving enough scalability in its infrastructure that it could keep up with that many users and that much data. That was a remarkable success story by any standard, but while it surely has a software element, no doubt it involved much more than just code. On the other hand, to a first approximation Facebook has had infinite resources for most of its existence. It operates in an online environment where problems can be fixed even in production. And it doesn't really do anything that is going to cause catastrophic, unfixable consequences if something does break for a while. That is a list of luxuries that few software development teams enjoy, and what works if you can make those assumptions won't necessarily be a good idea if you can't. As you say, you do have to pay attention to other factors as well, but there are not many organisations that have as much room to manoeuvre on the software side as Facebook does.

Maro 9y ago
When you're looking at templates for success, then looking at the successful companies (survivorship bias) is a good start. The fallacy is when it turns out that there are lots of other companies doing the same thing, but those companies aren't successful. This is the problem with all those blog posts examining the daily habits of highly successful people (like, they wake up early, or, they eat energy bars to save time, etc). Clearly, just doing these things won't make you a billionaire. For what I wrote, I believe this doesn't apply, because of the stark difference between the culture I experienced at FB and other companies. (Admittedly, this is not a stat.sign. sample.) So in my experience it's _not_ true that other companies are doing the same thing... Having said that, you're right that FB is a one in a million company, and probably nobody reading this will be the next M.Zuckerberg... But still, if you want to be the best in your domain (eg. best todo app), I strongly think these are good patterns to follow.