6 ms·

How I organise a large project

34 points by vegancap 12y ago

chriswarbo 12y ago
This looks somewhat similar to what we're doing at my current job. We've writing APIs for our internal systems to decouple things a bit, and have ended up with a frontend router, a "middle-end" message queue and backend scripts to do the processing. However, our Vagrant setup just took a massive turn for the worse: due to some compatability issues, these VMs have now all been hooked together using Puppet! During provisioning, if the git repos for the other machines aren't checked out to a particular location, everything dies; which seems like a really cargo-culty choice to me. I'd be interested in doing things like this with Nix: Disnix to define the relationships between the servers and NixOps to define the VM provisioning. That may be because I'm a functional programming nut who's just switched to NixOS though ;)
- vegancap 12y ago
  I've only been using this structure for say, an app and an API. I haven't tried it with say caching vm's or a message server thrown in the mix. Your idea about using Nix might be worth me looking into next! It kinda feels like spinning plates trying to organise vm's and codebases haha. I thought this structure might be a good start for new comers though :)
dbpatterson 12y ago
A large project? I'm not sure what's particularly large about this. Perhaps it should be "how I organize a project I'm going to have running on a server at some point"?
- vegancap 12y ago
  By large I just meant a project which houses multiple components. Of course the term 'large' is subjective.
paukiatwee 12y ago
The structure petty interesting, but which language/framework do you use? I think different language/framework might have their standard.
- vegancap 12y ago
  This was about a PHP/Symfony API and a angularjs/phonegap app. I've split it between a PHP app, Node app and a phonegap app previously as well and it felt like a nice workflow and I felt like I was getting more done. You're right it might not be ideal for everything. But for new comers I think it's a solid place to start. Especially for PHP, which can get a bit messy!
  - paukiatwee 12y ago
    Never really work on PHP before, but I think you should include those info so people like me will understand this setup is optimize for which language.
Ixiaus 12y ago
I've learned my lesson trying to structure "large projects" inside of a single "project" or logical repository. A developer I hired at one point had read an article written by someone in the OSS community promulgating the virtues of treating your "projects" like OSS projects and if you have a "large project" breaking it down into smaller constituent projects with their own source repository, dependency defs, README's, tests, Vagrant / Docker environments etc... This fits well with the ideas behind small composable units of work and it also increases clarity. [EDIT] Essentially, if you have a large project, it's probably time to re-think how you're coupling or de-coupling functional units. The goal being lots of small, documented, and tested units composed to create a "large software product".
- vegancap 12y ago
  Yeah I agree! I used to commit everything into one repo, and I found I was wasting a lot of time downloading thousands of files and vm's etc when I only needed to work on say one component. I think this structure lets you take either approach, needs some tweaking though for sure.
liotier 12y ago
In the name of all corporate grunts, thanks for reminding of how subjective 'large' is...
- vegancap 12y ago
  Haha not a problem! I've worked with a lot bigger (being a corporate grunt also), but this was just a starting point. In theory you could extend this structure without things getting too messy :)
BillinghamJ 12y ago
Imo, it's better to keep all components separated in their own repos for any project which can really be described as 'large'. What if, for example, you completely rewrite your backend system without changing its API spec at all? Having the history of the backend and mobile app linked becomes a bit odd at this point. What if you're using a system which deploys your repo? It may expect certain things to be in the root of the repo, and you certainly don't want your mobile app code to end up on your application servers. Honestly, I'm not really aware of any instance where it is actually better to lump together multiple parts of a system into one repository. Maybe you made some bad architectural decisions which makes it easier to do so, but I don't consider that 'better'.
- taeric 12y ago
  This does go counter to the way that it is described that both Google and Facebook have been reported to do things. The specific problem usually comes down to the extra work necessary to correlate changes between repositories.
  - BillinghamJ 12y ago
    I kinda get the impression that some of the very large, very fast growing companies have ended up like that by necessity because they've dug themselves into a hole they can't get out of. e.g. Facebook building HHVM instead of not using PHP.
    - taeric 12y ago
      I do not at all disagree with this impression. The question, though, is why are the major successes counter to the "best practice" that so many folk like to cite? Consider, even the linux kernel is counter to the best practices as they were thought at the time. I also realize that they could just be an unrelated data point. I just have grown weary of every success story I know about being "in spite of terrible practices." At some point, the best practices theory bows to the results of actual practices.
tootie 12y ago
Are the API and app not separate deploys?
_random_ 12y ago
Pretty standard for ASP.NET Web API + Angular projects.

34 points by vegancap 12y ago

This looks somewhat similar to what we're doing at my current job. We've writing APIs for our internal systems to decouple things a bit, and have ended up with a frontend router, a "middle-end" message queue and backend scripts to do the processing. However, our Vagrant setup just took a massive turn for the worse: due to some compatability issues, these VMs have now all been hooked together using Puppet! During provisioning, if the git repos for the other machines aren't checked out to a particular location, everything dies; which seems like a really cargo-culty choice to me. I'd be interested in doing things like this with Nix: Disnix to define the relationships between the servers and NixOps to define the VM provisioning. That may be because I'm a functional programming nut who's just switched to NixOS though ;)

vegancap 12y ago
I've only been using this structure for say, an app and an API. I haven't tried it with say caching vm's or a message server thrown in the mix. Your idea about using Nix might be worth me looking into next! It kinda feels like spinning plates trying to organise vm's and codebases haha. I thought this structure might be a good start for new comers though :)

dbpatterson 12y ago

A large project? I'm not sure what's particularly large about this. Perhaps it should be "how I organize a project I'm going to have running on a server at some point"?

vegancap 12y ago
By large I just meant a project which houses multiple components. Of course the term 'large' is subjective.

paukiatwee 12y ago

The structure petty interesting, but which language/framework do you use? I think different language/framework might have their standard.

vegancap 12y ago
This was about a PHP/Symfony API and a angularjs/phonegap app. I've split it between a PHP app, Node app and a phonegap app previously as well and it felt like a nice workflow and I felt like I was getting more done. You're right it might not be ideal for everything. But for new comers I think it's a solid place to start. Especially for PHP, which can get a bit messy!
- paukiatwee 12y ago
  Never really work on PHP before, but I think you should include those info so people like me will understand this setup is optimize for which language.

Ixiaus 12y ago

I've learned my lesson trying to structure "large projects" inside of a single "project" or logical repository. A developer I hired at one point had read an article written by someone in the OSS community promulgating the virtues of treating your "projects" like OSS projects and if you have a "large project" breaking it down into smaller constituent projects with their own source repository, dependency defs, README's, tests, Vagrant / Docker environments etc... This fits well with the ideas behind small composable units of work and it also increases clarity. [EDIT] Essentially, if you have a large project, it's probably time to re-think how you're coupling or de-coupling functional units. The goal being lots of small, documented, and tested units composed to create a "large software product".

vegancap 12y ago
Yeah I agree! I used to commit everything into one repo, and I found I was wasting a lot of time downloading thousands of files and vm's etc when I only needed to work on say one component. I think this structure lets you take either approach, needs some tweaking though for sure.

liotier 12y ago

In the name of all corporate grunts, thanks for reminding of how subjective 'large' is...

vegancap 12y ago
Haha not a problem! I've worked with a lot bigger (being a corporate grunt also), but this was just a starting point. In theory you could extend this structure without things getting too messy :)

BillinghamJ 12y ago

Imo, it's better to keep all components separated in their own repos for any project which can really be described as 'large'. What if, for example, you completely rewrite your backend system without changing its API spec at all? Having the history of the backend and mobile app linked becomes a bit odd at this point. What if you're using a system which deploys your repo? It may expect certain things to be in the root of the repo, and you certainly don't want your mobile app code to end up on your application servers. Honestly, I'm not really aware of any instance where it is actually better to lump together multiple parts of a system into one repository. Maybe you made some bad architectural decisions which makes it easier to do so, but I don't consider that 'better'.

taeric 12y ago
This does go counter to the way that it is described that both Google and Facebook have been reported to do things. The specific problem usually comes down to the extra work necessary to correlate changes between repositories.
- BillinghamJ 12y ago
  I kinda get the impression that some of the very large, very fast growing companies have ended up like that by necessity because they've dug themselves into a hole they can't get out of. e.g. Facebook building HHVM instead of not using PHP.
  - taeric 12y ago
    I do not at all disagree with this impression. The question, though, is why are the major successes counter to the "best practice" that so many folk like to cite? Consider, even the linux kernel is counter to the best practices as they were thought at the time. I also realize that they could just be an unrelated data point. I just have grown weary of every success story I know about being "in spite of terrible practices." At some point, the best practices theory bows to the results of actual practices.

tootie 12y ago

Are the API and app not separate deploys?

_random_ 12y ago

Pretty standard for ASP.NET Web API + Angular projects.