6 ms·

I got side-tracked by the talk of version control. Do they use a single repo for everything? It seems like it from they way they talk. In that case, I am sur

I got side-tracked by the talk of version control. Do they use a single repo for everything? It seems like it from they way they talk. In that case, I am surprised git is a good fit. SVN might have been better, though some commercial solutions like ClearCase or Perforce would actually be right for that sort of work-load.

daenney 10y ago
To me it seems like they're using a single repository for all their DNS infrastructure, not a monorepo for all of Spotify.
paulddraper 10y ago
Not sure, but FYI a single git repo is very doable for an org like Spotify, if you want to make it work. You have a few choices. (1) Keep binary/data files elsewhere. (2) Keep large files in Large File Store/Annex. (3) Use Git Virtual File System by MS.
- JoshTriplett 10y ago
  No matter which of those paths you take, you also need to have an automatic merge/rebase bot, because in a sufficiently large organization with a single repository, by the time you finish pulling to ensure that your change fast-forwards, someone else will have already pushed another change.
  - paulddraper 10y ago
    Few repos have a commit every couple seconds. They exist, but you have to be monumentally huge to have that.
    - JoshTriplett 10y ago
      Perhaps not every few seconds, but it will likely take more than a few seconds to "git pull --rebase", review the result, re-check the build and smoke tests, and then push. (Or, alternatively, to merge into master, build and test the result, and push.) And "every minute or two* is not at all unlikely for a company-wide shared repository during at least part of the day. Much easier to let a bot do either a cherry-pick or merge, including a build and smoke test.
      - paulddraper 10y ago
        git pull [--rebase] && git push takes a few seconds. Doing it server-side (say, through any off-the-shelf code review system) is even faster. If you're insisting on running tests every single change before fast-forwarding into trunk....yes that will get prohibitive very fast. A bot would hardly help though. If you have 4 commits a minute applied and verified serially, you need to build and test in 15 seconds.
        saryant 10y ago
        git pull doesn't take just a few a seconds on huge repos. Pulling down a ~day's worth of changes takes several minutes in ours.
        JoshTriplett 10y ago
        Even if you don't run tests, at a bare minimum you always need to build-test. master should always build. With a warm ccache, that shouldn't take excessively long, but long enough that you still want to do it server-side. Assuming you've locally build-tested each commit, the bot only needs to build-test the combination. And some bots even pull in a set of merges at a time, and keep them if they all pass, only falling back to serial testing if they fail.
blixt 10y ago
Spotify mostly uses independent repos for all code, but some configurations (e.g., DNS) are contained within a single repo. Some big companies do successfully use Git for monorepos though (e.g., Microsoft – search for "GVFS"). Note: I haven't been there for two years so of course some things may have changed.
raverbashing 10y ago
I assume you never used CC but it's a gigantic pile of crap. It is completely worthless. Sold on golf courses to clueless higher ups, because most developers hate it But in general I've never seen any commercial source control system beat an open source I've.
- daxelrod 10y ago
  Perforce is mostly a better Subversion. The two have an extremely similar model, but Perforce handles merges much better and gives more flexibility in slicing and dicing your local view of the repo (extremely useful for huge monorepos). Disclaimer: it's been several years since I've used Subversion, this may have changed.

daenney 10y ago

To me it seems like they're using a single repository for all their DNS infrastructure, not a monorepo for all of Spotify.

paulddraper 10y ago

Not sure, but FYI a single git repo is very doable for an org like Spotify, if you want to make it work. You have a few choices. (1) Keep binary/data files elsewhere. (2) Keep large files in Large File Store/Annex. (3) Use Git Virtual File System by MS.

JoshTriplett 10y ago
No matter which of those paths you take, you also need to have an automatic merge/rebase bot, because in a sufficiently large organization with a single repository, by the time you finish pulling to ensure that your change fast-forwards, someone else will have already pushed another change.
- paulddraper 10y ago
  Few repos have a commit every couple seconds. They exist, but you have to be monumentally huge to have that.
  - JoshTriplett 10y ago
    Perhaps not every few seconds, but it will likely take more than a few seconds to "git pull --rebase", review the result, re-check the build and smoke tests, and then push. (Or, alternatively, to merge into master, build and test the result, and push.) And "every minute or two* is not at all unlikely for a company-wide shared repository during at least part of the day. Much easier to let a bot do either a cherry-pick or merge, including a build and smoke test.
    - paulddraper 10y ago
      git pull [--rebase] && git push takes a few seconds. Doing it server-side (say, through any off-the-shelf code review system) is even faster. If you're insisting on running tests every single change before fast-forwarding into trunk....yes that will get prohibitive very fast. A bot would hardly help though. If you have 4 commits a minute applied and verified serially, you need to build and test in 15 seconds.
      - saryant 10y ago
        git pull doesn't take just a few a seconds on huge repos. Pulling down a ~day's worth of changes takes several minutes in ours.
      - JoshTriplett 10y ago
        Even if you don't run tests, at a bare minimum you always need to build-test. master should always build. With a warm ccache, that shouldn't take excessively long, but long enough that you still want to do it server-side. Assuming you've locally build-tested each commit, the bot only needs to build-test the combination. And some bots even pull in a set of merges at a time, and keep them if they all pass, only falling back to serial testing if they fail.

blixt 10y ago

Spotify mostly uses independent repos for all code, but some configurations (e.g., DNS) are contained within a single repo. Some big companies do successfully use Git for monorepos though (e.g., Microsoft – search for "GVFS"). Note: I haven't been there for two years so of course some things may have changed.

raverbashing 10y ago

I assume you never used CC but it's a gigantic pile of crap. It is completely worthless. Sold on golf courses to clueless higher ups, because most developers hate it But in general I've never seen any commercial source control system beat an open source I've.

daxelrod 10y ago
Perforce is mostly a better Subversion. The two have an extremely similar model, but Perforce handles merges much better and gives more flexibility in slicing and dicing your local view of the repo (extremely useful for huge monorepos). Disclaimer: it's been several years since I've used Subversion, this may have changed.