21 ms·

To me the beauty of git stems from the fact that it is an implementation of a functional data structure. It‘s a tree index that is read-only, and updating it in

To me the beauty of git stems from the fact that it is an implementation of a functional data structure. It‘s a tree index that is read-only, and updating it involves creating a complete copy of the tree and giving it a new name. Then the only challenge is to make that copy as cheap as possible - for which the tree lends itself, as only the nodes on the path to the root need to get updated. As a result, you get lock-free transactions (branches) and minimal overhead. And through git‘s pointer-to-parent commit you get full lineage. It is so beautiful in fact that when I think about systems that need to maintain long-running state in concurrent environments, my first reaction is ”split up the state into files, and maintain it through git(hub)“.

WhompingWindows 7y ago
Wow, that is a beautiful post, thank you for writing it out that way...it makes me pine for VCS in my job. Can you or someone else reflect on my file system? I work for the government doing statistical analysis of healthcare data, and there is no VCS where I code, other than how you name the files and where you put them in folders and how you back them up manually. I am facing a major data-branching event where I'm going from ~40 scripts (R, SQL, STATA) on one dataset, to then three overlapping but different datasets and having ~100 scripts. I just don't know if my brain and wits are up to the task of maintaining these 3 branches with 3 different languages, given all I have is folder/file names and my knowledge reservoir and memory... I know this is a perfect use case for git, but I've never used it before and no one else in my department uses it. I don't know if I have the time and energy left at this job to implement a new system of VCS AND reproduce my code for 3 different-but-similar projects. Burnout approaches...
- caymanjim 7y ago
  You can use Git on your own without anyone else being affected. It doesn't require a server to add benefit. Learn to work with it and then introduce your coworkers later.
  - akhilcacharya 7y ago
    It’s possible you can’t install it in the computing environment
    - ShteiLoups 7y ago
      Sublime-merge (the Gui git client from the sublime text people) is available in a portable version, and so can be run as a .exe from the filesystem, or a mountable drive. Comes with its own git binary. The GUI is stunningly beautiful and functional, and there are more than enough keyboard shortcuts to keep things snappy once you're in the flow. I used to live and die by the terminal, now I am in love with sublime merge. I used the portable version for a job where I didn't have install rights to the corporate laptop, and it preserved my workflow and kept me sane during my dev work. The portable version can run a little slow, but it's a pretty good solution.
    - jrumbut 7y ago
      I'm in a similar situation and the entire git for windows setup (including git bash that works beautifully with things like Windows network drives!) can be used without ever needing admin privileges. So I not only have git but also vim and perl and the whole *nix kit I was so sorely missing. Some truly locked down environments may not allow it but if the poster has other open source tools like R they can probably run .exe files.
  - shantly 7y ago
    I've done exactly this ~4 years ago when I briefly worked at a place that used Subversion, after an acquisition. I wanted to be able to dick around in my own branches, with proper diffing and tracking and all, without updating the server, which appeared to be impossible (more or less). There was a git-to-svn I could use but considering how easy it was to screw up other people's state in Subversion, it made me nervous. So I just worked in my own, local git then copied the files to SVN when ready to commit something worth sharing.
- londons_explore 7y ago
  Tell management that your current approach isn't going to work for much longer, and say you have some ideas that might improve the situation. Get your department to pay for you and ~2 colleagues to go on a git training course for a few days. As well as teaching you how to use git, it'll give you some time with an expert to look at your problem, and give you some relaxation time helping the burnout, and with 3 of you on the course, you'll likely get buy-in for a new setup. Beware that git isn't a silver bullet. While it solves a bunch of issues, it causes many new ones - especially when you have lots of people who aren't knowledgeable about version control using it. I wish git had better integration with 'regular files' - ie. so that Mary in the marketing department can update a readme file without having to learn a totally new way of working. I wish you could "mount" a git repo as a drive in Windows, and all changes would be auto-committed to a branch as soon as a file is saved, and that branch were auto-merged to master as long as tests pass. Then people without git knowledge can work as before.
  - 1996 7y ago
    > wish you could "mount" a git repo as a drive in Windows, and all changes would be auto-committed to a branch as soon as a file is saved, and that branch were auto-merged to master as long as tests pass. Then people without git knowledge can work as before. Cool idea for a project
    - remram 7y ago
      Isn't that what GVFS is? https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit
      - 1996 7y ago
        Cool find, didn't know about that. Does it only check files passing tests? I read quickly and didn't see that
- saalweachter 7y ago
  git is actually pretty easy to drop into a terrible methodology without too much disruption. git works by creating its own .git directory wherever you create a new git repository, but doesn't touch the files and directories outside of it until you tell it to. So you can have a directory of old code and you just cd to it and run 'git init', and now you have a git repository in the same directory. It won't be managing any of the files yet, but it will technically be there. Because git is just a bunch of extra data in a .git directory, and because git is also built as a distributed VCS, the "make a copy of a directory to back it up" methodology actually works pretty OK with git. Ideally you should be using 'git clone' to copy your directories and 'git pull' to keep them in sync, but if you just Control-C Control-V your source code directory, git will actually be just fine with that, and you can still later use git to sync the changes between those two directories. I'm not going to put a full git tutorial into this post, about how you add files to the repository and make commits, but I just want to convey that while git has a justifiable reputation for sometimes devolving into arcane incantations -- it's actually low effort to get started and you only need to learn three or five commands to get like 95% of the value from it. Once you learn those three or five commands, you'll find yourself running 'git init' in nearly every directory you make -- for your random scripts, for your free time coding projects, for your free time creative writing projects -- and you'll even find it easy to use on horrible "27 directory copies of the source code with 14 file renames" projects where none of your teammates use git; you can use git yourself in such cases without adding any real friction, and it still helps you even if your teammates just copy your code directory or send you copies of their code directories. EDIT: One other note: git can also go away easily if you decide you don't like it. You don't need to run git commands to create, edit, copy or otherwise modify the files in your code base, like you do with some other source control systems, so if you can just forget it is there if you are busy and don't want to worry about it, and then later go ahead and add or commit all of your work. If you really don't like it, you just stop running git commands and you're no longer using it: you don't need to 'export' or 'ungitify' your code base. So it's pretty low-risk in that way as well.
  - shantly 7y ago
    Other cool things about git being "just a directory full of files": - you can put the git directory somewhere other than in your working directory, if you really want to. Or reference a bunch of .git directories in a series of commands without having to change your current directory. Sometimes this is handy (usually for automation or something like that). - If you're nervous about some command you're about to run—something that might screw up your git tree—just copy the .git directory somewhere else first. You can copy it back to entirely restore your state before the command, no need to figure out how to reverse what you did (assuming it's even possible).
- taco_emoji 7y ago
  Git being distributed means you can use it without any centralized "master"--your local repository contains the entire history. And if git seems too difficult to start with, Subversion can also "host" a repository on the file system, in a directory separate from your working directory.
  - tempguy9999 7y ago
    Agreed with this, SVN (short for subversion) is a good alternative. I understood and was comfortable with SVN within a few minutes (using the TortoiseGit front-end, which I highly recommend). I wrestled with git for months and at the end still feel I haven't subdued it properly. I can use it reliably but SVN is just so much friendlier. So my suggestion is go with SVN + TortoiseGit. SVN is your butler. Git is a hydra that can do so much, once you've tied it down and cut off its thrashing heads. It's not just me, our whole (small) company moved to it it burnt too much of our time and mental resources. Edit: after learning TortoiseGit, learn the SVN command line commands (it's easy), and learn ASAP how to make backups of your repository!
    - tasuki 7y ago
      SVN is easy, Git is simple. Getting started with SVN is very quick, but once you need to peek under the hood, you'll find out it's super complicated inside. Git is just the other way around: the interface is a mess, but the internals are simple and beautiful. Once you understand four concepts (blobs, trees, commits, refs), the rest falls into place. Recommended four page intro to git internals: https://www.chromium.org/developers/fast-intro-to-git-internals https://www.chromium.org/developers/fast-intro-to-git-intern...
      - tempguy9999 7y ago
        I'll check your link, thanks. Could you explain what you mean by svn being super complicated inside? I presume you mean from a user's not a programmer's perspective; I never found it confusing, ever. It has it's flaws (tags are writable, unless that's been cured) but it's really pretty good, and far better than git for a beginner IMO.
- mooneater 7y ago
  I faced a moment like this, where I realised I needed git to survive a big set of changes. (Though I was on SVN before, which was better than nothing but a far cry from git). However, branching may not be the ideal solution given how you describe your issue. With git branches, we typically dont want to run something then switch branches then run something else. I would say branches are primarily for organizing sets of changes over time. If you have multiple datasets with similarities, what you may need more than git is refactoring and design patterns. To handle the common data in a common way, and then cleanly organize the differences. That said I would still definitely want all scripts in git. It is not that hard to learn, lean on someone you know or email me if you need to.
- x0x0 7y ago
  git for ML projects with data: https://dvc.org https://dvc.org In particular, dvc carefully handles large binary files.
- deleted 7y ago
  [deleted]
Swizec 7y ago
Fyi this is called a merkle tree. Also the data structure that blockchain uses and a couple other protocols. They’re wonderful and surprisingly easy to implement. https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree
- hyperpallium 7y ago
  Wow, 1979. And patented.
  - that_jojo 7y ago
    Most ideas in CS are old as dirt. There was a flurry of theory advancements mid-century and a lot of work since then has been putting those ideas into practice.
    - agumonkey 7y ago
      Even back in renaissance days with the most advanced mathematics there were unknown prior art
    - hyperpallium 7y ago
      I have a theory this is a consequence of Euclid not being taught anymore.
    - Accujack 7y ago
      ..so right about the time computers were invented, people figured out how to use them and did some basic research.
  - tinus_hn 7y ago
    Patents from 1979 have expired a long time ago.
BenoitEssiambre 7y ago
I sometimes like to explain things the other way around in terms of functional programming immutability being like version control for program state. I rarely use functional programming but I certainly see its appeal for certain things. I think the concept of immutability in functional programming confuses people. It really clicked for me when I stopped thinking of it in terms of things not being able to change and started instead to think of it in terms of each version of things having different names, somewhat like commits in version control. Functional programming makes explicit, not only which variables you are accessing, but which version of it. It may seem like you are copying variables every time you want to modify them but really you are just giving different mutations, different names. This doesn't mean things are actually copied in memory. Like git, the compiler doesn't need to make full copies for every versions. If it sees that you are not going to reference a particular mutation, it might just physically overwrite it with the next mutation. In the background "var a=i, var b=a+j", might compile as something like "var b = i; b+=j";
duxup 7y ago
I encountered a large company where they had a private git server for their engineering teams. Over time someone discovered that the number of repositories and usage was much greater than they expected. What they found was that non engineering folks who had contact with engineering had asked questions about how they manage their code, what branches were, and etc. Some friendly engineering teams had explained, then some capable non engineering employees discovered that the server was open to anyone with a login (as far as creating and managing your own repositories) and capable employees had started using it to manage their own files. The unexpected users mostly used it on a per user basis (not as a team) as the terminology tripped up / slowed down a lot of non engineering folks, but individuals really liked it. IT panicked and wanted to lock it down but because engineering owned it ... they just didn't care / nothing was done. They were a cool team.
- GhettoMaestro 7y ago
  Chad engineers v. The virgin CISO
- jniedrauer 7y ago
  Unfortunately git does not handle binary files elegantly (unless you use git-lfs). You can inflate storage rapidly by, say, editing a 10M zip file a few times. I've had to GC more than one repo where someone accidentally added an innocuous binary file, and the next thing you know the repo has exceeded 2G of storage space.
  - db48x 7y ago
    Luckily storage is getting cheaper. I do wish someone hadn't checked a custom-built nginx binary into ours though.
    - apostacy 7y ago
      Even with infinite storage, having lots of blobs can make a repo unmanageable. In order to get an 8GB repo onto github, I had to make temporary branches and and push them incrementally. I highly recommend git-annex. It is like git-lfs but a bit less mature but much more powerful. Especially good if you don't want to set up a centralized lfs server.
maxerickson 7y ago
The real genius of git is the clear, concise user interface for that data structure. (for those with a severe sugar hangover, I'm being a little bit sarcastic)
- afiori 7y ago
  I especially like the intuitive order of command line arguments
  - Iwan-Zotow 7y ago
    some intuition you have...
- miohtama 7y ago
  Quoting myself from yesterday: https://news.ycombinator.com/threads?id=miohtama https://news.ycombinator.com/threads?id=miohtama "Git is what a version control UX would look like if it were written by kernel developers who only knew Perl and C" Back in a day we had Subversion, Mercurial, Bazaar, some others. I used all of these. All of them were more coherent than Git. However they were slower - but not much - and they were not used by the most popular software project in the world. Then, GitHub popularized git and Github become well funded enough to take over the software development world. Bitbucket, now Atlassian, started as a hosted Mercurial repos. Bazaar was DVCS for Ubuntu, developed by Ubuntu folks. Will we see another DVCS ever again? I hope yes. Now all software developers with less than 5 years of experience are using Google and GitHub as the user interface for Git. Git's cognitive burden is terrible and can be solved. However Git authors themselves are not priorising this. Software development industry could gain a lot of productivity in the form of more sane de facto version control system, with saner defaults and better discoverability.
  - ryl00 7y ago
    > However they were slower - but not much I don't think we should underestimate how much performance differences can color opinions here, especially for something like CLI tools that are used all the time. Little cuts add up. At work I use both git and bazaar, and bazaar's sluggishness makes me tend to avoid it when possible. I recall Mercurial recently announced an attempt to rewrite core(?) parts in Rust, because python was just not performant enough.
localhost 7y ago
It seems like git can also be used as an immutable database as well. Does anyone here have experience / comments on using git as a database backend? EDIT: I found a couple of interesting references for folks who may be curious about this as well. I especially like [2] for its diagrams. [1] https://stackoverflow.com/questions/20151158/using-git-repository-as-a-database-backend https://stackoverflow.com/questions/20151158/using-git-repos... [2] https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/ https://www.kenneth-truyers.net/2016/10/13/git-nosql-databas...
- cmarschner 7y ago
  It seems you would want a hosted git installation like github / gitlab or VSO for that. The main concern to me is that API keys are usually too limiting (or too expensive). E.g. if you don't want to manage local state by having a working directory you need to do many API calls to make a change (create the objects, create the trees, then create the commit, then change the ref) so that it is almost more worthwhile to use the same data structure on a more generic backend like a NoSQL key-value store. I haven't done this though (though it's been on my wish list for long). What it also doesn't give you for free is sensible search indexes. I do think though that combining it with a search index could be very powerful.
combatentropy 7y ago
". . . unlike every single horror I've ever witnessed when looking closer at SCM products, git actually has a simple design, with stable and reasonably well-documented data structures. In fact, I'm a huge proponent of designing your code around the data, rather than the other way around, and I think it's one of the reasons git has been fairly successful. . . . "I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships." --- Linus Torvalds, https://lwn.net/Articles/193245/ https://lwn.net/Articles/193245/
- gimboland 7y ago
  That last comment is absolutely golden. Once upon a time I had the privilege to spend a few years working in Swansea University's compsci department, which punches above its weight in theoretical computer science. One of the moments that made me the programmer I am today (whatever that's worth) came when I was meeting with the head of the department to discuss a book he was writing, and while we were discussing this very point of data vs code, I said to him, realising the importance of choosing the right structure, "so the data is central to the subject" (meaning computer science in general" — to which he replied emphathically that "the data IS the subject". That was a lightbulb moment for me. From then on I saw computer science as the study of how data is represented, and how those representations are transformed and transported — that's it, that basically covers everything. It's served me well.
  - abtinf 7y ago
    I think part of the confusion stems from the word “computer” itself. Ted Nelson makes the point that the word is an accident of history, arising because main funding came from large military computation projects. But computers don’t “compute”, they don’t do math. Computers are simplifying, integrating machines that manipulate symbols. Data (and its relationships) is the essential concept in the term “symbolic manipulator”. Code (ie a function) is the essential concept in the term “compute”.
jayd16 7y ago
Git is great but all the major drawbacks stem from this design as well? Immutability and a full copy cause public rebase headaches, extremely large repos (before LFS) and the lack of partial checkouts to name a few. Doesn't it show more of the drawbacks of this functional data structure?
- cmarschner 7y ago
  I think rebases are causing so much pain that it is better to have tool support such that they are not needed. E.g. it was a revelation to discover that github doesn't care in PRs - just do a merge with master, done. As for the working directory - yes, there could be more management around that. I'm not sure why the git community went for nested repos / submodules rather than partial checkouts. It's a different question than the data structure of the repo itself, though. Compared to other VCS it still seems miles ahead. Large repos: It seems one could alleviate that by limiting the pull history (and LFS if needed), right?
dominotw 7y ago
database version of this is datomic (closed source). Are there any others? browser version of this is datascript (oss).
rectang 7y ago
Totally agree with this point, so much that I did a presentation on "Git Data Structure Design" two months ago for Papers We Love San Diego that hammers it home over the course of 50 minutes. https://www.youtube.com/watch?v=fHSZz_Mx-Uo https://www.youtube.com/watch?v=fHSZz_Mx-Uo To become a Git power user, it is far more beneficial to learn its underlying content-addressable-store data structure rather than explore the bazillion options in its command line interface. It is surprisingly easy to create a repository manually and then to start adding "blob" files to the store!
SilasX 7y ago
But git also has (and most use cases demand) the concept of tree deletion.
dreamcompiler 7y ago
It is a beautifully simple data structure. I just wish its UI was that simple.
Upvoter33 7y ago
eh, it's just copy-on-write - an ancient OS technique. You can call it functional if you like but the idea is old and has been applied very generally for decades.
WalterBright 7y ago
If I had known that from the start, I would have found git much easier to learn!

WhompingWindows 7y ago

Wow, that is a beautiful post, thank you for writing it out that way...it makes me pine for VCS in my job. Can you or someone else reflect on my file system? I work for the government doing statistical analysis of healthcare data, and there is no VCS where I code, other than how you name the files and where you put them in folders and how you back them up manually. I am facing a major data-branching event where I'm going from ~40 scripts (R, SQL, STATA) on one dataset, to then three overlapping but different datasets and having ~100 scripts. I just don't know if my brain and wits are up to the task of maintaining these 3 branches with 3 different languages, given all I have is folder/file names and my knowledge reservoir and memory... I know this is a perfect use case for git, but I've never used it before and no one else in my department uses it. I don't know if I have the time and energy left at this job to implement a new system of VCS AND reproduce my code for 3 different-but-similar projects. Burnout approaches...

caymanjim 7y ago
You can use Git on your own without anyone else being affected. It doesn't require a server to add benefit. Learn to work with it and then introduce your coworkers later.
- akhilcacharya 7y ago
  It’s possible you can’t install it in the computing environment
  - ShteiLoups 7y ago
    Sublime-merge (the Gui git client from the sublime text people) is available in a portable version, and so can be run as a .exe from the filesystem, or a mountable drive. Comes with its own git binary. The GUI is stunningly beautiful and functional, and there are more than enough keyboard shortcuts to keep things snappy once you're in the flow. I used to live and die by the terminal, now I am in love with sublime merge. I used the portable version for a job where I didn't have install rights to the corporate laptop, and it preserved my workflow and kept me sane during my dev work. The portable version can run a little slow, but it's a pretty good solution.
  - jrumbut 7y ago
    I'm in a similar situation and the entire git for windows setup (including git bash that works beautifully with things like Windows network drives!) can be used without ever needing admin privileges. So I not only have git but also vim and perl and the whole *nix kit I was so sorely missing. Some truly locked down environments may not allow it but if the poster has other open source tools like R they can probably run .exe files.
- shantly 7y ago
  I've done exactly this ~4 years ago when I briefly worked at a place that used Subversion, after an acquisition. I wanted to be able to dick around in my own branches, with proper diffing and tracking and all, without updating the server, which appeared to be impossible (more or less). There was a git-to-svn I could use but considering how easy it was to screw up other people's state in Subversion, it made me nervous. So I just worked in my own, local git then copied the files to SVN when ready to commit something worth sharing.
londons_explore 7y ago
Tell management that your current approach isn't going to work for much longer, and say you have some ideas that might improve the situation. Get your department to pay for you and ~2 colleagues to go on a git training course for a few days. As well as teaching you how to use git, it'll give you some time with an expert to look at your problem, and give you some relaxation time helping the burnout, and with 3 of you on the course, you'll likely get buy-in for a new setup. Beware that git isn't a silver bullet. While it solves a bunch of issues, it causes many new ones - especially when you have lots of people who aren't knowledgeable about version control using it. I wish git had better integration with 'regular files' - ie. so that Mary in the marketing department can update a readme file without having to learn a totally new way of working. I wish you could "mount" a git repo as a drive in Windows, and all changes would be auto-committed to a branch as soon as a file is saved, and that branch were auto-merged to master as long as tests pass. Then people without git knowledge can work as before.
- 1996 7y ago
  > wish you could "mount" a git repo as a drive in Windows, and all changes would be auto-committed to a branch as soon as a file is saved, and that branch were auto-merged to master as long as tests pass. Then people without git knowledge can work as before. Cool idea for a project
  - remram 7y ago
    Isn't that what GVFS is? https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit
    - 1996 7y ago
      Cool find, didn't know about that. Does it only check files passing tests? I read quickly and didn't see that
saalweachter 7y ago
git is actually pretty easy to drop into a terrible methodology without too much disruption. git works by creating its own .git directory wherever you create a new git repository, but doesn't touch the files and directories outside of it until you tell it to. So you can have a directory of old code and you just cd to it and run 'git init', and now you have a git repository in the same directory. It won't be managing any of the files yet, but it will technically be there. Because git is just a bunch of extra data in a .git directory, and because git is also built as a distributed VCS, the "make a copy of a directory to back it up" methodology actually works pretty OK with git. Ideally you should be using 'git clone' to copy your directories and 'git pull' to keep them in sync, but if you just Control-C Control-V your source code directory, git will actually be just fine with that, and you can still later use git to sync the changes between those two directories. I'm not going to put a full git tutorial into this post, about how you add files to the repository and make commits, but I just want to convey that while git has a justifiable reputation for sometimes devolving into arcane incantations -- it's actually low effort to get started and you only need to learn three or five commands to get like 95% of the value from it. Once you learn those three or five commands, you'll find yourself running 'git init' in nearly every directory you make -- for your random scripts, for your free time coding projects, for your free time creative writing projects -- and you'll even find it easy to use on horrible "27 directory copies of the source code with 14 file renames" projects where none of your teammates use git; you can use git yourself in such cases without adding any real friction, and it still helps you even if your teammates just copy your code directory or send you copies of their code directories. EDIT: One other note: git can also go away easily if you decide you don't like it. You don't need to run git commands to create, edit, copy or otherwise modify the files in your code base, like you do with some other source control systems, so if you can just forget it is there if you are busy and don't want to worry about it, and then later go ahead and add or commit all of your work. If you really don't like it, you just stop running git commands and you're no longer using it: you don't need to 'export' or 'ungitify' your code base. So it's pretty low-risk in that way as well.
- shantly 7y ago
  Other cool things about git being "just a directory full of files": - you can put the git directory somewhere other than in your working directory, if you really want to. Or reference a bunch of .git directories in a series of commands without having to change your current directory. Sometimes this is handy (usually for automation or something like that). - If you're nervous about some command you're about to run—something that might screw up your git tree—just copy the .git directory somewhere else first. You can copy it back to entirely restore your state before the command, no need to figure out how to reverse what you did (assuming it's even possible).
taco_emoji 7y ago
Git being distributed means you can use it without any centralized "master"--your local repository contains the entire history. And if git seems too difficult to start with, Subversion can also "host" a repository on the file system, in a directory separate from your working directory.
- tempguy9999 7y ago
  Agreed with this, SVN (short for subversion) is a good alternative. I understood and was comfortable with SVN within a few minutes (using the TortoiseGit front-end, which I highly recommend). I wrestled with git for months and at the end still feel I haven't subdued it properly. I can use it reliably but SVN is just so much friendlier. So my suggestion is go with SVN + TortoiseGit. SVN is your butler. Git is a hydra that can do so much, once you've tied it down and cut off its thrashing heads. It's not just me, our whole (small) company moved to it it burnt too much of our time and mental resources. Edit: after learning TortoiseGit, learn the SVN command line commands (it's easy), and learn ASAP how to make backups of your repository!
  - tasuki 7y ago
    SVN is easy, Git is simple. Getting started with SVN is very quick, but once you need to peek under the hood, you'll find out it's super complicated inside. Git is just the other way around: the interface is a mess, but the internals are simple and beautiful. Once you understand four concepts (blobs, trees, commits, refs), the rest falls into place. Recommended four page intro to git internals: https://www.chromium.org/developers/fast-intro-to-git-internals https://www.chromium.org/developers/fast-intro-to-git-intern...
    - tempguy9999 7y ago
      I'll check your link, thanks. Could you explain what you mean by svn being super complicated inside? I presume you mean from a user's not a programmer's perspective; I never found it confusing, ever. It has it's flaws (tags are writable, unless that's been cured) but it's really pretty good, and far better than git for a beginner IMO.
mooneater 7y ago
I faced a moment like this, where I realised I needed git to survive a big set of changes. (Though I was on SVN before, which was better than nothing but a far cry from git). However, branching may not be the ideal solution given how you describe your issue. With git branches, we typically dont want to run something then switch branches then run something else. I would say branches are primarily for organizing sets of changes over time. If you have multiple datasets with similarities, what you may need more than git is refactoring and design patterns. To handle the common data in a common way, and then cleanly organize the differences. That said I would still definitely want all scripts in git. It is not that hard to learn, lean on someone you know or email me if you need to.
x0x0 7y ago
git for ML projects with data: https://dvc.org https://dvc.org In particular, dvc carefully handles large binary files.
deleted 7y ago
[deleted]

Swizec 7y ago

Fyi this is called a merkle tree. Also the data structure that blockchain uses and a couple other protocols. They’re wonderful and surprisingly easy to implement. https://en.wikipedia.org/wiki/Merkle_tree https://en.wikipedia.org/wiki/Merkle_tree

hyperpallium 7y ago
Wow, 1979. And patented.
- that_jojo 7y ago
  Most ideas in CS are old as dirt. There was a flurry of theory advancements mid-century and a lot of work since then has been putting those ideas into practice.
  - agumonkey 7y ago
    Even back in renaissance days with the most advanced mathematics there were unknown prior art
  - hyperpallium 7y ago
    I have a theory this is a consequence of Euclid not being taught anymore.
  - Accujack 7y ago
    ..so right about the time computers were invented, people figured out how to use them and did some basic research.
- tinus_hn 7y ago
  Patents from 1979 have expired a long time ago.

BenoitEssiambre 7y ago

I sometimes like to explain things the other way around in terms of functional programming immutability being like version control for program state. I rarely use functional programming but I certainly see its appeal for certain things. I think the concept of immutability in functional programming confuses people. It really clicked for me when I stopped thinking of it in terms of things not being able to change and started instead to think of it in terms of each version of things having different names, somewhat like commits in version control. Functional programming makes explicit, not only which variables you are accessing, but which version of it. It may seem like you are copying variables every time you want to modify them but really you are just giving different mutations, different names. This doesn't mean things are actually copied in memory. Like git, the compiler doesn't need to make full copies for every versions. If it sees that you are not going to reference a particular mutation, it might just physically overwrite it with the next mutation. In the background "var a=i, var b=a+j", might compile as something like "var b = i; b+=j";

duxup 7y ago

I encountered a large company where they had a private git server for their engineering teams. Over time someone discovered that the number of repositories and usage was much greater than they expected. What they found was that non engineering folks who had contact with engineering had asked questions about how they manage their code, what branches were, and etc. Some friendly engineering teams had explained, then some capable non engineering employees discovered that the server was open to anyone with a login (as far as creating and managing your own repositories) and capable employees had started using it to manage their own files. The unexpected users mostly used it on a per user basis (not as a team) as the terminology tripped up / slowed down a lot of non engineering folks, but individuals really liked it. IT panicked and wanted to lock it down but because engineering owned it ... they just didn't care / nothing was done. They were a cool team.

GhettoMaestro 7y ago
Chad engineers v. The virgin CISO
jniedrauer 7y ago
Unfortunately git does not handle binary files elegantly (unless you use git-lfs). You can inflate storage rapidly by, say, editing a 10M zip file a few times. I've had to GC more than one repo where someone accidentally added an innocuous binary file, and the next thing you know the repo has exceeded 2G of storage space.
- db48x 7y ago
  Luckily storage is getting cheaper. I do wish someone hadn't checked a custom-built nginx binary into ours though.
  - apostacy 7y ago
    Even with infinite storage, having lots of blobs can make a repo unmanageable. In order to get an 8GB repo onto github, I had to make temporary branches and and push them incrementally. I highly recommend git-annex. It is like git-lfs but a bit less mature but much more powerful. Especially good if you don't want to set up a centralized lfs server.

maxerickson 7y ago

The real genius of git is the clear, concise user interface for that data structure. (for those with a severe sugar hangover, I'm being a little bit sarcastic)

afiori 7y ago
I especially like the intuitive order of command line arguments
- Iwan-Zotow 7y ago
  some intuition you have...
miohtama 7y ago
Quoting myself from yesterday: https://news.ycombinator.com/threads?id=miohtama https://news.ycombinator.com/threads?id=miohtama "Git is what a version control UX would look like if it were written by kernel developers who only knew Perl and C" Back in a day we had Subversion, Mercurial, Bazaar, some others. I used all of these. All of them were more coherent than Git. However they were slower - but not much - and they were not used by the most popular software project in the world. Then, GitHub popularized git and Github become well funded enough to take over the software development world. Bitbucket, now Atlassian, started as a hosted Mercurial repos. Bazaar was DVCS for Ubuntu, developed by Ubuntu folks. Will we see another DVCS ever again? I hope yes. Now all software developers with less than 5 years of experience are using Google and GitHub as the user interface for Git. Git's cognitive burden is terrible and can be solved. However Git authors themselves are not priorising this. Software development industry could gain a lot of productivity in the form of more sane de facto version control system, with saner defaults and better discoverability.
- ryl00 7y ago
  > However they were slower - but not much I don't think we should underestimate how much performance differences can color opinions here, especially for something like CLI tools that are used all the time. Little cuts add up. At work I use both git and bazaar, and bazaar's sluggishness makes me tend to avoid it when possible. I recall Mercurial recently announced an attempt to rewrite core(?) parts in Rust, because python was just not performant enough.

localhost 7y ago

It seems like git can also be used as an immutable database as well. Does anyone here have experience / comments on using git as a database backend? EDIT: I found a couple of interesting references for folks who may be curious about this as well. I especially like [2] for its diagrams. [1] https://stackoverflow.com/questions/20151158/using-git-repository-as-a-database-backend https://stackoverflow.com/questions/20151158/using-git-repos... [2] https://www.kenneth-truyers.net/2016/10/13/git-nosql-database/ https://www.kenneth-truyers.net/2016/10/13/git-nosql-databas...

cmarschner 7y ago
It seems you would want a hosted git installation like github / gitlab or VSO for that. The main concern to me is that API keys are usually too limiting (or too expensive). E.g. if you don't want to manage local state by having a working directory you need to do many API calls to make a change (create the objects, create the trees, then create the commit, then change the ref) so that it is almost more worthwhile to use the same data structure on a more generic backend like a NoSQL key-value store. I haven't done this though (though it's been on my wish list for long). What it also doesn't give you for free is sensible search indexes. I do think though that combining it with a search index could be very powerful.

combatentropy 7y ago

". . . unlike every single horror I've ever witnessed when looking closer at SCM products, git actually has a simple design, with stable and reasonably well-documented data structures. In fact, I'm a huge proponent of designing your code around the data, rather than the other way around, and I think it's one of the reasons git has been fairly successful. . . . "I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships." --- Linus Torvalds, https://lwn.net/Articles/193245/ https://lwn.net/Articles/193245/

gimboland 7y ago
That last comment is absolutely golden. Once upon a time I had the privilege to spend a few years working in Swansea University's compsci department, which punches above its weight in theoretical computer science. One of the moments that made me the programmer I am today (whatever that's worth) came when I was meeting with the head of the department to discuss a book he was writing, and while we were discussing this very point of data vs code, I said to him, realising the importance of choosing the right structure, "so the data is central to the subject" (meaning computer science in general" — to which he replied emphathically that "the data IS the subject". That was a lightbulb moment for me. From then on I saw computer science as the study of how data is represented, and how those representations are transformed and transported — that's it, that basically covers everything. It's served me well.
- abtinf 7y ago
  I think part of the confusion stems from the word “computer” itself. Ted Nelson makes the point that the word is an accident of history, arising because main funding came from large military computation projects. But computers don’t “compute”, they don’t do math. Computers are simplifying, integrating machines that manipulate symbols. Data (and its relationships) is the essential concept in the term “symbolic manipulator”. Code (ie a function) is the essential concept in the term “compute”.

jayd16 7y ago

Git is great but all the major drawbacks stem from this design as well? Immutability and a full copy cause public rebase headaches, extremely large repos (before LFS) and the lack of partial checkouts to name a few. Doesn't it show more of the drawbacks of this functional data structure?

cmarschner 7y ago
I think rebases are causing so much pain that it is better to have tool support such that they are not needed. E.g. it was a revelation to discover that github doesn't care in PRs - just do a merge with master, done. As for the working directory - yes, there could be more management around that. I'm not sure why the git community went for nested repos / submodules rather than partial checkouts. It's a different question than the data structure of the repo itself, though. Compared to other VCS it still seems miles ahead. Large repos: It seems one could alleviate that by limiting the pull history (and LFS if needed), right?

dominotw 7y ago

database version of this is datomic (closed source). Are there any others? browser version of this is datascript (oss).

rectang 7y ago

Totally agree with this point, so much that I did a presentation on "Git Data Structure Design" two months ago for Papers We Love San Diego that hammers it home over the course of 50 minutes. https://www.youtube.com/watch?v=fHSZz_Mx-Uo https://www.youtube.com/watch?v=fHSZz_Mx-Uo To become a Git power user, it is far more beneficial to learn its underlying content-addressable-store data structure rather than explore the bazillion options in its command line interface. It is surprisingly easy to create a repository manually and then to start adding "blob" files to the store!

SilasX 7y ago

But git also has (and most use cases demand) the concept of tree deletion.

dreamcompiler 7y ago

It is a beautifully simple data structure. I just wish its UI was that simple.

Upvoter33 7y ago

eh, it's just copy-on-write - an ancient OS technique. You can call it functional if you like but the idea is old and has been applied very generally for decades.

WalterBright 7y ago

If I had known that from the start, I would have found git much easier to learn!