11 ms·

Show HN: A version control system based on rsync

249 points by zdgeier 4y ago

Hi everyone! I'm trying to create a version control system that solves some of the problems that Git and other version control software has when working in a team. Let me know if you have any feedback!

sigmonsays 4y ago
how does this even work with merges or conflicts? How do you review changes easily? It seems like a lot is actually missing. Rolling hashes over blocks is not exactly revision control.
- zdgeier 4y ago
  Yeah this project is still pretty early in development. The main algorithm described on the site is implemented but there's a lot missing that developers would expect from a system like this. Trying to make a version control system and hosting platform in my free time while working a full-time job is not exactly an easy task haha. I plan on adding more features in the next few months.
quintussss 4y ago
Would be nice if you could talk about the problems you had with git and how jamsync solves them.
- zdgeier 4y ago
  Hi! I decided to link to the About page since this content was better suited for HN. You can read more about the problems Jamsync plans to solve on the homepage at jamsync.dev/ . Thanks for the suggestion!
- kokanee 4y ago
  I think a lot of folks in this thread are forgetting the first day they spent using git. It's kind of like asking why someone would create a text editor in a world where vim exists, since vim can already do everything you need. Sure, but it takes months to become proficient with and years to master. Years ago I worked at a company that had a team of front end developers that used Dropbox for version control instead of git. Conflicts were prevented by the default "this file has changed on disk, are you sure you want to save?" warning in their IDEs. It worked incredibly well for that team, and they never suffered a single gitastrophe.
jerpint 4y ago
One thing that could be useful for this is writing code locally and testing it on a remote server. I assume the simple act of editing reflects the change on the other machine? No need to commit, push, then pull to do a one-liner experimental change
- zdgeier 4y ago
  Thanks for checking this out! Yep that's where I'm planning on going with this. I think this could even be extended to modifying and deploying code without having the push/commit/pull process that we have now.
  - rickydroll 4y ago
    I would encourage you to explore this more thoroughly. I frequently find myself working with three machines at the same time. My Windows box because of needing to use speech recognition in conjunction with copilot and VS code. My Linux laptop for local endpoint, and a VM endpoint somewhere (the old man says standing in the yard shaking his fist at the cloud) Far too often I find myself making all my changes on my Windows box, or copying them to one destination, the other, or both. Then bug fixing on the local machine as part of my test cycle. Then I have a few moments of confusion when I try to remember what I changed where and how to pull them all back over to my Windows box. So yeah, something like your proposed tool would be quite useful.
    - zdgeier 4y ago
      Yeah definitely found myself having some of the same pain points when working on multiple machines. I'll keep this in mind!
    - thefreeman 4y ago
      It’s pretty trivial to setup syncthing for your working directory and have it sync between all 3 of your machines. you don’t need version control for that just file system syncing.
      - 4y ago
aliqot 4y ago
> We're currently not taking contributions but feel free to email us for feature requests. why
- samwillis 4y ago
  Open source != open contribution or project The most well known project following this is SQLite. There are many reasons to do this: - ensuring that the copyright is fully owned - ensuring security by reducing the likelihood of someone submitting secretly vulnerable code - not wanting the overhead of running an Open Source Project - wanting ultimate control on the architecture of the tool - early stage of a rapidly changing codebase There is a danger that "gate keeping" the concept of open contribution == open source, reduces the motivation of people to open source their code. All open source, no matter the contribution policy, should be applauded.
- zdgeier 4y ago
  Still in the early stages of development with code changing rapidly. Not sure if it would be the best idea to get other people working on things at this stage, but I'm open to suggestions!
  - riceart 4y ago
    My suggestion: keep doing what you’re comfortable with. Having source available is better than not, full stop - and should be commended. Don’t succumb to peer pressure. If you want to make a “community” project do it on your own terms.
- deleted 4y ago
  [deleted]
harryvederci 4y ago
Nice, I hope you won't get an avalanche of comments here by grumpy old devs that are afraid this will somehow mean they'll have to learn something new. I think it's great to look into possibilities of doing things in a better way, even if the majority of people think the current way is the only correct way.
- zdgeier 4y ago
  Thanks! Haha yeah my plan is to certainly not try to replace any existing tools but provide options for working faster, especially for new developers.
- tambourine_man 4y ago
  I think I qualify as an old developer. I love learning new exciting things that I know I’ll use for the rest of my live. C, Vim, Bash, HTML, CSS, JavaScript will all outlive everyone here. It’s a pretty safe bet. I hate the amount of useless knowledge I accumulated for obsolete unexciting things. I feel like Git is not the final answer to version control, it had a lot of great ideas, but it’s not even good enough for some things and too complicated for most others. My hope is that it becomes like SVN, mostly legacy stuff, and that we can build something new with the lessons learned from Git. I wish Jamsync luck, it looks interesting and I freaking love rsync.
  - mbork_pl 4y ago
    I guess Git suffers from a similar problem as (La)TeX: it's a truly great prototype which sadly wasn't thrown away to build something better. Don't get me wrong, I like both TeX and Git _very_ much. (I even co-authored a LaTeX textbook.) I also have a lot of respect for DEK and LT. But they were trailblazers (especially DEK), and so they did a lot of things not knowing their true impact – and sometimes this means these decisions were far from optimal. (The case of TeX is even worse because the machines of the time were very limited compared to what we have today.)
    - tambourine_man 4y ago
samwillis 4y ago
What's the particular use case for this tool? Would it be better for projects with large static/binary assets, such as game dev or CAD?
- zdgeier 4y ago
  Yes, versioning large files is certainly one use case that Jamsync solves (and was the original problem that caused me to develop this). More use cases are on the homepage, but I think that preventing merge conflicts could be interesting. Typically a source of merge conflicts is not pulling/rebasing frequently enough which Jamsync plans to solve by pulling in changes as they happen on the current branch and rebasing branches as changes happen on the "main" branch. This means that branches are constantly being merged and reduces the risk that you base your changes off of old code.
  - johnwalkr 4y ago
    It's a bit more complicated for 3D CAD (I'm talking about the mechanical engineering type here) than just rebasing frequently. I'm going to write a long comment because I hope someone will solve my problem.. I'm looking for a good tool for 3D CAD version control because the available ones are all expensive and focus on features for managers, not engineers (like approval workflows). I was excited by Jamsync, but I don't think this will work well for a few reasons. First, CAD software generally modifies every file you open, unless it is read-only. This is mainly because even rotating a model causes a file change. Second, it's really common for multiple engineers to work in the same assembly, even if they are working in different parts. Third, merging of the CAD files, which are binary, isn't usually possible even when using the same vendor for both CAD and version control. So all you can do is select a version to keep, you can't merge. It's annoying because often one person's real work is overwritten just because someone else had a stale copy of an assembly file, and saved over the real work with something trivial like a view rotation. Only there's no easy way even compare two versions so the most common scenario is redoing the work. CAD won't generally continuously load any changes like in the example on your website, because engineers have assemblies open, for the entire day, with all or most files in a project, and these are loaded into memory once and locked on the disk. All this means that one feature is 100% required: (a) explicit checkout and checkin of the files you want to work on, only allowed by one user at a time. It's almost a requirement (b) to do this with a plugin in the CAD software itself, because hierarchies within CAD assembly files are used to select, open, etc files, and because CAD files are often just names with a consecutive numbering scheme. It's quite painful to have to use a different interface to locate and checkout/checkin files. If you can offer these two things you may have a decent market for small companies using CAD. Self-hosting (c) is a requirement for many industries. A file viewer (d) is nice to have but not at all required, and every other software attempts to add JIRA-type features (e) which frankly just becomes another place other than the correct place to document things. The only software I've seen which really hit the mark was grabcad workbench, which is unfortunately being discontinued[1]. It did just a, b, and d and it was free too! The offerings from actual CAD vendors are way too heavy, need dedicated roles to maintain them, and cost more than CAD licenses. Kenesto[2] does a, b, d and and Bild[3] does a, d only. Kenesto looks good but I've heard complaints that it feels barely maintained. Bild looked promising but it's slow and not only is it lacking b, a plugin for CAD, you have to use its webview application which offers no search, preview or sorting options so you have spend your time scrolling through a list of files. And of course, both offer way too many unwanted features. Everyone selling version control software for CAD doesn't seem to realize that we don't want to them to compete with Siemens on features. We want a few features done well, and already have tools or workflows for tickets, approvals, commenting, etc. [1] https://blog.grabcad.com/blog/2022/08/09/a-fond-farewell-to-grabcad-workbench/ https://blog.grabcad.com/blog/2022/08/09/a-fond-farewell-to-... [2] https://www.kenesto.com/buynow https://www.kenesto.com/buynow [3] https://www.getbild.com/plans https://www.getbild.com/plans
forgotpwd16 4y ago
Question, due to a recent thread on an rsync-like CDC-based tool, for this problem how rsync compares to CDC?
- zdgeier 4y ago
  Thanks for bringing this up! I somehow missed this. Yes I plan on implementing a CDC-based approach in the future but for now I'm just using a naive fixed chunk size.
brendoncarroll 4y ago
This is a very similar to one of my projects "Got". https://github.com/gotvc/got https://github.com/gotvc/got The algorithms it uses are superior to rsync and git in a few ways. It comes short on features, especially for software development compared to Git. The motivation is more for personal file storage. I notice you're using Go and AGPL licensed, so you could borrow any of Got's libraries without issue. (Got is GPL licensed.) Definitely reach out in a GitHub issue.
- zdgeier 4y ago
  Ah awesome! I'll check this out
  - zdgeier 4y ago
    Took a look and this is really cool. Will definitely keep this is mind. One major difference and area I'm focused on in this project is hosting. I would argue that there are currently much better VCS options for projects than Git (like yours and Fossil) but the reason these haven't taken off is that Github and Gitlab offer unmatched hosting and collaboration tools for these projects. Would be curious to know your thoughts/approach to this!
    - brendoncarroll 4y ago
      My approach to hosting with Got has been to make it easy and secure for users to host from any machine. INET256 solves that problem nicely. If you have access to an INET256 network, then all you have to do is swap addresses and two Got instances can communicate. https://github.com/inet256/inet256 https://github.com/inet256/inet256 Also, end-to-end encryption is table stakes. Any data that leaves the user needs to be encrypted in transit, and if it hangs around away from the user, at rest.
- kamranjon 4y ago
  "The algorithms it uses are superior to rsync and git in a few ways." - Would you mind explaining this?
replwoacause 4y ago
This looks great. Keep going!
- zdgeier 4y ago
  Thanks!
remram 4y ago
An option for mounting the files with fuse and/or doing a sparse checkout would probably be good. Otherwise your working copy may use a lot of disk space with redundant data.
- zdgeier 4y ago
  Thanks! Yeah planning on adding the ability to mount specific directories so people don't have to pull down large assets they don't need, for example in gamedev or machine learning applications.
tuukkah 4y ago
It's not yet clear to me whether Jamsync has the concept of commits and tracks the history of locally made changes. (Rsync obviously hasn't and doesn't.) Does Jamsync have 3-way merging (e.g. of local and remote changes on top of a shared ancestor)?
- zdgeier 4y ago
  I'm still figuring out semantics, but currently a "commit" is made every time you make a change to a file which is synced to the remote. Every time you run `jam`, you push a commit. Or if you leave `jam` running, commits will be made each time you save. 3-way merging will be possible in the future, but will be a little different than most VCS since there is no history stored locally.
  - jeremyjh 4y ago
    So how do you communicate with other developers the intention of a commit or set of commits, like you would with commit messages in git? How do you rewrite/cleanup history?
    - zdgeier 4y ago
      This is still in-development but my plan is to essentially have "branches" which are merged and create a single "commit" on the main branch. You'll have the option to add information to a branch and a merge message if you would like, but it won't be required. I don't have any approaches on rewriting/cleaning up history right now since I think managing code history is pretty low-value for most teams. In my experience, it's far better to add another commit than it is to try to change history. Would be curious to know your thoughts though!
      - jacobsenscott 4y ago
        This sounds great. Your VCS should fade into the background, rather than be front and center like git. Nobody should be breaking their flow, staging, committing, pushing as an ersatz backup, and then going back to coding. Teams shouldn't be bikeshedding on merge vs rebase, many commits vs one commit, etc. They should just be coding. However I can see where a sort of "checkpoint" feature before performing a "hold my beer" change makes sense. A commit in some sense. `jam checkpoint` ... code ... `jam revert` You should just be coding, and when the thing your are coding is done just do a single "squash merge (in git speak)" to the main branch.
zabzonk 4y ago
so is this kind of a better replacement for subversion? thinking about large file support. i'm not aware of major problems that git and/or hg have when working with a team.
- zdgeier 4y ago
  Yes, that's one way to think about it. You could probably say it's closer to Google Drive than a VCS right now but my idea is to make a VCS that's more collaborative than the options currently available. Some people will have no issues with the current VCS that we have now but many developers that I've talked to have expressed frustration with the general user experience of VCSs and when merge conflicts occur.
richardwhiuk 4y ago
I'm curious whether there are common operations which cause extremely expensive resyncs - it's unclear, depending on the hash. e.g. does adding a space at the start of a file causing all of the blocks to be misaligned?
- zdgeier 4y ago
  The rsync paper https://www.andrew.cmu.edu/course/15-749/READINGS/required/cas/tridgell96.pdf https://www.andrew.cmu.edu/course/15-749/READINGS/required/c... is better than me at explaining this, but the rolling hash is the key to solving this misalignment problem that you describe. Essentially, using the rolling hash enables you to detect when a block as been misaligned (bytes added at the beginning of the file for example). There's no way to cause expensive resyncs in this case since it rolls over every byte in the file and reuses blocks on any alignment, even with data inserted in between.
Dave3of5 4y ago
If two users are editing the same file this looks like you'll see my edits in your file in near real-time. This is being used to do way with merging / merge conflicts. A few observations: * I think this makes two devs changing the file basically unusable as you're changing the file and potentially the same line simultaneous which could cause me to overwrite your changes without realising * It still presumable doesn't get away from the fact I can edit the file with the jam command not running then run it after in which it'll just overwrite my local changes onto the remote without merging * What does it do if my editor has changes in the change buffer that aren't saved and an update comes in on the same line, then I save. Will my now out of date change buffer overwrite the server version ? I don't think you can do away with merging and merge conflicts it's a vital part of source control so that two devs can work on the same file simultaneously. How often does the thing sync, none of the demos really show this. The demo's showed the thing running very often i.e. every few seconds. Does that mean if I'm making large changes to a file it automatically upload every time I save the file ? I definitely do not want that as typically when editing code as I edit I will save something that is in a partially broken / incomplete state. It also precludes the main CI/CD workflows which will normally trigger from a dev uploading something they think will work. Where is the actual source, the source page links me to here: https://jamsync.dev/download https://jamsync.dev/download This just gives a link to a .zip which is presumably out of date now. I presume you are trying to self host this but since you haven't ironed out the details it would be better to use an open source code tool like github /gitlab ...etc for the time being. You are allowing me to register / login to an "account" on something with basically no terms of service or privacy policy. Not only are you not GDPR compliant but more importantly at the moment you are not limiting your liability! The fact you've stuck this up without limiting your liability you are now partially liable for whatever stuff people are putting up on here baffles me. Seriously for your own good please get some terms of service with at least limitation of liability. You'll have to also sort the problem of copyright as I wouldn't push code to a service that attempts to claim any copyright of that which I assume you're not doing but I would need that in writing. Given the frequency of commits that this solution will introduce you'll need to offer way more than 5GB of hosting. How will that version slider work when I have 500k commits. Even squashing branches, the repo I'm working on at the moment has 10k commits. This is a team of about 30-40 devs and each day is about 15ish PRs. With Jamsync syncing every save I could easily see this going up to millions of commits.
billconan 4y ago
reminds me of https://xethub.com/assets/docs/xet-specifics/how-xet-deduplication-works https://xethub.com/assets/docs/xet-specifics/how-xet-dedupli...
kypro 4y ago
I have nothing much to add about the tool, but just want to say this is some beautiful documentation.
- zdgeier 4y ago
  Thanks :) I unfortunately can't take the credit for this beautiful theme. Credit goes to https://radoslawkoziel.pl/ https://radoslawkoziel.pl/ who made https://github.com/panr/hugo-theme-terminal https://github.com/panr/hugo-theme-terminal which I based this off of.
  - joshspankit 4y ago
    I can’t help suggesting to fill the top bar left to right as the project progresses to 1.0
hardwaregeek 4y ago
There seem to be a few projects that try to be a live, collaborative git. Perhaps it’s just me, but I don’t necessarily think I’d want this for my main system. Maybe a short term collaboration/pair programming session, but not in general. I like how git is not immediate. I like that I can take time before sharing my code to clean it up. The issues that a tool like this purports to solve are not issues to me. Live collaboration is a great idea, but maybe not as a version control system. But hey if other people get use out of it, why not!
- zdgeier 4y ago
  Appreciate the feedback! Will keep this use case in mind, this could definitely work as a layer on top of Git when people need to pair program or preview code.
- andix 4y ago
  What I always wanted to have is a tool that syncs my uncommitted changes of git repos between all my devices. So that I can switch from the desktop to the laptop without committing, pushing and pulling. Right where I stopped. Also sync not yet pushed commits, I may want to rebase/edit before pushing.
  - pmontra 4y ago
    By uncommitted changes do you mean edited files? Then something like syncthing or anything functionally similar to Dropbox.
  - klabb3 4y ago
    100x yes. I agree with parent that pushing live changes is way to intrusive, but a synced working/staging directory could be incredibly helpful. I don’t see any fundamental reason why git or, even 3p tooling for git, couldn’t support this use case using something like private, ephemeral, auto-synced branches.
  - nvissari 4y ago
    Here's the roundabout way I sync uncommitted changes. Push everything to a new tmp branch. Then on the other system, fetch and cherry-pick the last commit id, then reset HEAD~1. Boom uncommitted changes synced.
frogperson 4y ago
I don't think I would want this for code, but it sounds really interesting for a shared drive. I would love to version control the dir where my team dumps all of its docs and spreadsheets and everything else.
- zdgeier 4y ago
  Thanks for the feedback! What about a version control system would be valuable for storing your documents? I'm curious why you couldn't use something like Google Drive/Dropbox.
  - jbverschoor 4y ago
    Security, local copies, and limited restoration options
account-5 4y ago
How does this compare with fossil? I use fossil for personal projects.
- zdgeier 4y ago
  There are some similarities but Fossil still has a push/pull/commit flow, although it does provide syncing features to automatically pull commits. Also, I don't think it has great support for large files. Ultimately, the priorities of Fossil are going to be different than Jamsync, since Jamsync is not decentralized. Being centralized means that we'll be able to do some different things, like live file editing, but with some downsides related to replication and distribution.
KerrAvon 4y ago
I don't understand the use case. Very large file support aside, are you saying you've observed people looking at git and say "I wish I could use this like I use a chat server?"
- zdgeier 4y ago
  Haha appreciate you taking a look. One use case is to prevent merge conflicts, especially when working on a fast-moving project. Merge conflicts generally occur when you make changes on some old code. Jamsync constantly "rebases" your changes on top of new changes so that your code will always be up to date. Obviously some merge conflicts will still occur but much less so in this approach.
  - zdgeier 4y ago
    More details about the problems I'm trying to solve are on the homepage if you're curious.
  - kccqzy 4y ago
    How is that different from running `git pull` or the equivalent faster, maybe on a timer?
    - zdgeier 4y ago
      This could work for the pulling side but really what jamsync does is both pull and push. Having changes that are always present in the remote means that merge conflicts are less likely. If you were to do this with Git, you would also have to automate the git add, commit, and push steps which would ruin the log in addition to being pretty slow.
      - yencabulator 4y ago
        That sounds like one developer being in the middle of an edit breaks the project immediately for everyone?
  - 4y ago
ipid 4y ago
I wonder whether this system works for zip-compressed files, such as large .docx, .npz, etc. These are common large files, but it seems that rsync cannot handle compressed files. Would Jamsync solve this problem?
- zdgeier 4y ago
  Not currently, but it would definitely be interesting to have an unzip done automatically based on the file type. Then you would be able to track the contents efficiently, rather than having large binary blobs with large diffs. I haven't heard of other version control systems out there doing this.
- antod 4y ago
  Compression formats can be made to handle rsync, even if it isn't a thing out of the box - in my ignorance gzip is the only example I'm aware of. https://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/ https://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/
kinleyd 4y ago
Sounds very useful to me. rsync is my go to tool - I use it to manually maintain versions of my workstation as I find it the most reliable and usable among all options I've tried.
deleted 4y ago
[deleted]
joveian 4y ago
Two comments from a quick look: 1) I recall a comment a while back from the xdelta author about how most history tools store a sequence of changes from nothing (as it sounds like yours does), but this isn't ideal since usually you want a recent version which then needs to be produced from the potentially long sequence of changes. Caching various revisions could help in your case but that is less than ideal when the files are large. 2) It isn't clear how you go from live editing type thing to revision control. If you haven't yet, you might want to look at the Aegis paper[1] (it is short) for a quick overview of the framework it used to form a revision control system from a collection of more specific tools that could be customized. That type of framework might go well with your rapid branch synchronization. Sadly the author, Peter Miller[2], died in 2014. As far as I know no one picked up development (it isn't the easiest name to search for :/). The User Guide [3] from the documentation page [4] gives a bit more info about the model. [1] https://aegis.sourceforge.net/auug93.pdf https://aegis.sourceforge.net/auug93.pdf [2] https://en.wikipedia.org/wiki/Peter_Miller_(software_engineer) https://en.wikipedia.org/wiki/Peter_Miller_(software_enginee... [3] https://aegis.sourceforge.net/user-guide.pdf https://aegis.sourceforge.net/user-guide.pdf [4] https://aegis.sourceforge.net/documents.html https://aegis.sourceforge.net/documents.html
- zdgeier 4y ago
  Thanks for this! 1) Agreed, caching might not be great for large files. I don't think this is the only way though. I think there is another approach, that I haven't explored, which is storing a pointer to the last known Data block in each Range block. This means that you can just jump directly to the actual data without having to go through this regeneration or caching process. The space cost of this would be pretty low, one int per block, but it would mean that you wouldn't need to iterate through every change. Still figuring this out though and caching would be the best option if this doesn't work. 2) Yes I haven't totally figured this out yet. I'll give this a look, thanks!
pmeunier 4y ago
Btw, Pijul (https://pijul.org https://pijul.org) already does this for binary files and large files. But it also has other advantages such as easier workflows, provably correct merge and cherry-picking.
JoeyBananas 4y ago
It might be better to limit the scope to "Live editing" or "source code syncing" rather than "version control".
- zdgeier 4y ago
  Yeah I agree that live editing is probably a more appropriate name right now but I think there are so many tools that already do that right now and haven't really made much impact. I don't see a large value in the live editing aspect when it comes to developing and deploying faster, even though this system will support it. The live editing is most valuable when preventing merge conflicts and working on several machines. Definitely open to your thoughts though.

249 points by zdgeier 4y ago

sigmonsays 4y ago

how does this even work with merges or conflicts? How do you review changes easily? It seems like a lot is actually missing. Rolling hashes over blocks is not exactly revision control.

zdgeier 4y ago
Yeah this project is still pretty early in development. The main algorithm described on the site is implemented but there's a lot missing that developers would expect from a system like this. Trying to make a version control system and hosting platform in my free time while working a full-time job is not exactly an easy task haha. I plan on adding more features in the next few months.

quintussss 4y ago

Would be nice if you could talk about the problems you had with git and how jamsync solves them.

zdgeier 4y ago
Hi! I decided to link to the About page since this content was better suited for HN. You can read more about the problems Jamsync plans to solve on the homepage at jamsync.dev/ . Thanks for the suggestion!
kokanee 4y ago
I think a lot of folks in this thread are forgetting the first day they spent using git. It's kind of like asking why someone would create a text editor in a world where vim exists, since vim can already do everything you need. Sure, but it takes months to become proficient with and years to master. Years ago I worked at a company that had a team of front end developers that used Dropbox for version control instead of git. Conflicts were prevented by the default "this file has changed on disk, are you sure you want to save?" warning in their IDEs. It worked incredibly well for that team, and they never suffered a single gitastrophe.

jerpint 4y ago

One thing that could be useful for this is writing code locally and testing it on a remote server. I assume the simple act of editing reflects the change on the other machine? No need to commit, push, then pull to do a one-liner experimental change

zdgeier 4y ago
Thanks for checking this out! Yep that's where I'm planning on going with this. I think this could even be extended to modifying and deploying code without having the push/commit/pull process that we have now.
- rickydroll 4y ago
  I would encourage you to explore this more thoroughly. I frequently find myself working with three machines at the same time. My Windows box because of needing to use speech recognition in conjunction with copilot and VS code. My Linux laptop for local endpoint, and a VM endpoint somewhere (the old man says standing in the yard shaking his fist at the cloud) Far too often I find myself making all my changes on my Windows box, or copying them to one destination, the other, or both. Then bug fixing on the local machine as part of my test cycle. Then I have a few moments of confusion when I try to remember what I changed where and how to pull them all back over to my Windows box. So yeah, something like your proposed tool would be quite useful.
  - zdgeier 4y ago
    Yeah definitely found myself having some of the same pain points when working on multiple machines. I'll keep this in mind!
  - thefreeman 4y ago
    It’s pretty trivial to setup syncthing for your working directory and have it sync between all 3 of your machines. you don’t need version control for that just file system syncing.
    - 4y ago

aliqot 4y ago

> We're currently not taking contributions but feel free to email us for feature requests. why

samwillis 4y ago
Open source != open contribution or project The most well known project following this is SQLite. There are many reasons to do this: - ensuring that the copyright is fully owned - ensuring security by reducing the likelihood of someone submitting secretly vulnerable code - not wanting the overhead of running an Open Source Project - wanting ultimate control on the architecture of the tool - early stage of a rapidly changing codebase There is a danger that "gate keeping" the concept of open contribution == open source, reduces the motivation of people to open source their code. All open source, no matter the contribution policy, should be applauded.
zdgeier 4y ago
Still in the early stages of development with code changing rapidly. Not sure if it would be the best idea to get other people working on things at this stage, but I'm open to suggestions!
- riceart 4y ago
  My suggestion: keep doing what you’re comfortable with. Having source available is better than not, full stop - and should be commended. Don’t succumb to peer pressure. If you want to make a “community” project do it on your own terms.
deleted 4y ago
[deleted]

harryvederci 4y ago

Nice, I hope you won't get an avalanche of comments here by grumpy old devs that are afraid this will somehow mean they'll have to learn something new. I think it's great to look into possibilities of doing things in a better way, even if the majority of people think the current way is the only correct way.

zdgeier 4y ago
Thanks! Haha yeah my plan is to certainly not try to replace any existing tools but provide options for working faster, especially for new developers.
tambourine_man 4y ago
I think I qualify as an old developer. I love learning new exciting things that I know I’ll use for the rest of my live. C, Vim, Bash, HTML, CSS, JavaScript will all outlive everyone here. It’s a pretty safe bet. I hate the amount of useless knowledge I accumulated for obsolete unexciting things. I feel like Git is not the final answer to version control, it had a lot of great ideas, but it’s not even good enough for some things and too complicated for most others. My hope is that it becomes like SVN, mostly legacy stuff, and that we can build something new with the lessons learned from Git. I wish Jamsync luck, it looks interesting and I freaking love rsync.
- mbork_pl 4y ago
  I guess Git suffers from a similar problem as (La)TeX: it's a truly great prototype which sadly wasn't thrown away to build something better. Don't get me wrong, I like both TeX and Git _very_ much. (I even co-authored a LaTeX textbook.) I also have a lot of respect for DEK and LT. But they were trailblazers (especially DEK), and so they did a lot of things not knowing their true impact – and sometimes this means these decisions were far from optimal. (The case of TeX is even worse because the machines of the time were very limited compared to what we have today.)
  - tambourine_man 4y ago

samwillis 4y ago

What's the particular use case for this tool? Would it be better for projects with large static/binary assets, such as game dev or CAD?

zdgeier 4y ago
Yes, versioning large files is certainly one use case that Jamsync solves (and was the original problem that caused me to develop this). More use cases are on the homepage, but I think that preventing merge conflicts could be interesting. Typically a source of merge conflicts is not pulling/rebasing frequently enough which Jamsync plans to solve by pulling in changes as they happen on the current branch and rebasing branches as changes happen on the "main" branch. This means that branches are constantly being merged and reduces the risk that you base your changes off of old code.
- johnwalkr 4y ago
  It's a bit more complicated for 3D CAD (I'm talking about the mechanical engineering type here) than just rebasing frequently. I'm going to write a long comment because I hope someone will solve my problem.. I'm looking for a good tool for 3D CAD version control because the available ones are all expensive and focus on features for managers, not engineers (like approval workflows). I was excited by Jamsync, but I don't think this will work well for a few reasons. First, CAD software generally modifies every file you open, unless it is read-only. This is mainly because even rotating a model causes a file change. Second, it's really common for multiple engineers to work in the same assembly, even if they are working in different parts. Third, merging of the CAD files, which are binary, isn't usually possible even when using the same vendor for both CAD and version control. So all you can do is select a version to keep, you can't merge. It's annoying because often one person's real work is overwritten just because someone else had a stale copy of an assembly file, and saved over the real work with something trivial like a view rotation. Only there's no easy way even compare two versions so the most common scenario is redoing the work. CAD won't generally continuously load any changes like in the example on your website, because engineers have assemblies open, for the entire day, with all or most files in a project, and these are loaded into memory once and locked on the disk. All this means that one feature is 100% required: (a) explicit checkout and checkin of the files you want to work on, only allowed by one user at a time. It's almost a requirement (b) to do this with a plugin in the CAD software itself, because hierarchies within CAD assembly files are used to select, open, etc files, and because CAD files are often just names with a consecutive numbering scheme. It's quite painful to have to use a different interface to locate and checkout/checkin files. If you can offer these two things you may have a decent market for small companies using CAD. Self-hosting (c) is a requirement for many industries. A file viewer (d) is nice to have but not at all required, and every other software attempts to add JIRA-type features (e) which frankly just becomes another place other than the correct place to document things. The only software I've seen which really hit the mark was grabcad workbench, which is unfortunately being discontinued[1]. It did just a, b, and d and it was free too! The offerings from actual CAD vendors are way too heavy, need dedicated roles to maintain them, and cost more than CAD licenses. Kenesto[2] does a, b, d and and Bild[3] does a, d only. Kenesto looks good but I've heard complaints that it feels barely maintained. Bild looked promising but it's slow and not only is it lacking b, a plugin for CAD, you have to use its webview application which offers no search, preview or sorting options so you have spend your time scrolling through a list of files. And of course, both offer way too many unwanted features. Everyone selling version control software for CAD doesn't seem to realize that we don't want to them to compete with Siemens on features. We want a few features done well, and already have tools or workflows for tickets, approvals, commenting, etc. [1] https://blog.grabcad.com/blog/2022/08/09/a-fond-farewell-to-grabcad-workbench/ https://blog.grabcad.com/blog/2022/08/09/a-fond-farewell-to-... [2] https://www.kenesto.com/buynow https://www.kenesto.com/buynow [3] https://www.getbild.com/plans https://www.getbild.com/plans

forgotpwd16 4y ago

Question, due to a recent thread on an rsync-like CDC-based tool, for this problem how rsync compares to CDC?

zdgeier 4y ago
Thanks for bringing this up! I somehow missed this. Yes I plan on implementing a CDC-based approach in the future but for now I'm just using a naive fixed chunk size.

brendoncarroll 4y ago

This is a very similar to one of my projects "Got". https://github.com/gotvc/got https://github.com/gotvc/got The algorithms it uses are superior to rsync and git in a few ways. It comes short on features, especially for software development compared to Git. The motivation is more for personal file storage. I notice you're using Go and AGPL licensed, so you could borrow any of Got's libraries without issue. (Got is GPL licensed.) Definitely reach out in a GitHub issue.

zdgeier 4y ago
Ah awesome! I'll check this out
- zdgeier 4y ago
  Took a look and this is really cool. Will definitely keep this is mind. One major difference and area I'm focused on in this project is hosting. I would argue that there are currently much better VCS options for projects than Git (like yours and Fossil) but the reason these haven't taken off is that Github and Gitlab offer unmatched hosting and collaboration tools for these projects. Would be curious to know your thoughts/approach to this!
  - brendoncarroll 4y ago
    My approach to hosting with Got has been to make it easy and secure for users to host from any machine. INET256 solves that problem nicely. If you have access to an INET256 network, then all you have to do is swap addresses and two Got instances can communicate. https://github.com/inet256/inet256 https://github.com/inet256/inet256 Also, end-to-end encryption is table stakes. Any data that leaves the user needs to be encrypted in transit, and if it hangs around away from the user, at rest.
kamranjon 4y ago
"The algorithms it uses are superior to rsync and git in a few ways." - Would you mind explaining this?

replwoacause 4y ago

This looks great. Keep going!

zdgeier 4y ago
Thanks!

remram 4y ago

An option for mounting the files with fuse and/or doing a sparse checkout would probably be good. Otherwise your working copy may use a lot of disk space with redundant data.

zdgeier 4y ago
Thanks! Yeah planning on adding the ability to mount specific directories so people don't have to pull down large assets they don't need, for example in gamedev or machine learning applications.

tuukkah 4y ago

It's not yet clear to me whether Jamsync has the concept of commits and tracks the history of locally made changes. (Rsync obviously hasn't and doesn't.) Does Jamsync have 3-way merging (e.g. of local and remote changes on top of a shared ancestor)?

zdgeier 4y ago
I'm still figuring out semantics, but currently a "commit" is made every time you make a change to a file which is synced to the remote. Every time you run `jam`, you push a commit. Or if you leave `jam` running, commits will be made each time you save. 3-way merging will be possible in the future, but will be a little different than most VCS since there is no history stored locally.
- jeremyjh 4y ago
  So how do you communicate with other developers the intention of a commit or set of commits, like you would with commit messages in git? How do you rewrite/cleanup history?
  - zdgeier 4y ago
    This is still in-development but my plan is to essentially have "branches" which are merged and create a single "commit" on the main branch. You'll have the option to add information to a branch and a merge message if you would like, but it won't be required. I don't have any approaches on rewriting/cleaning up history right now since I think managing code history is pretty low-value for most teams. In my experience, it's far better to add another commit than it is to try to change history. Would be curious to know your thoughts though!
    - jacobsenscott 4y ago
      This sounds great. Your VCS should fade into the background, rather than be front and center like git. Nobody should be breaking their flow, staging, committing, pushing as an ersatz backup, and then going back to coding. Teams shouldn't be bikeshedding on merge vs rebase, many commits vs one commit, etc. They should just be coding. However I can see where a sort of "checkpoint" feature before performing a "hold my beer" change makes sense. A commit in some sense. `jam checkpoint` ... code ... `jam revert` You should just be coding, and when the thing your are coding is done just do a single "squash merge (in git speak)" to the main branch.

zabzonk 4y ago

so is this kind of a better replacement for subversion? thinking about large file support. i'm not aware of major problems that git and/or hg have when working with a team.

zdgeier 4y ago
Yes, that's one way to think about it. You could probably say it's closer to Google Drive than a VCS right now but my idea is to make a VCS that's more collaborative than the options currently available. Some people will have no issues with the current VCS that we have now but many developers that I've talked to have expressed frustration with the general user experience of VCSs and when merge conflicts occur.

richardwhiuk 4y ago

I'm curious whether there are common operations which cause extremely expensive resyncs - it's unclear, depending on the hash. e.g. does adding a space at the start of a file causing all of the blocks to be misaligned?

zdgeier 4y ago
The rsync paper https://www.andrew.cmu.edu/course/15-749/READINGS/required/cas/tridgell96.pdf https://www.andrew.cmu.edu/course/15-749/READINGS/required/c... is better than me at explaining this, but the rolling hash is the key to solving this misalignment problem that you describe. Essentially, using the rolling hash enables you to detect when a block as been misaligned (bytes added at the beginning of the file for example). There's no way to cause expensive resyncs in this case since it rolls over every byte in the file and reuses blocks on any alignment, even with data inserted in between.

Dave3of5 4y ago

If two users are editing the same file this looks like you'll see my edits in your file in near real-time. This is being used to do way with merging / merge conflicts. A few observations: * I think this makes two devs changing the file basically unusable as you're changing the file and potentially the same line simultaneous which could cause me to overwrite your changes without realising * It still presumable doesn't get away from the fact I can edit the file with the jam command not running then run it after in which it'll just overwrite my local changes onto the remote without merging * What does it do if my editor has changes in the change buffer that aren't saved and an update comes in on the same line, then I save. Will my now out of date change buffer overwrite the server version ? I don't think you can do away with merging and merge conflicts it's a vital part of source control so that two devs can work on the same file simultaneously. How often does the thing sync, none of the demos really show this. The demo's showed the thing running very often i.e. every few seconds. Does that mean if I'm making large changes to a file it automatically upload every time I save the file ? I definitely do not want that as typically when editing code as I edit I will save something that is in a partially broken / incomplete state. It also precludes the main CI/CD workflows which will normally trigger from a dev uploading something they think will work. Where is the actual source, the source page links me to here: https://jamsync.dev/download https://jamsync.dev/download This just gives a link to a .zip which is presumably out of date now. I presume you are trying to self host this but since you haven't ironed out the details it would be better to use an open source code tool like github /gitlab ...etc for the time being. You are allowing me to register / login to an "account" on something with basically no terms of service or privacy policy. Not only are you not GDPR compliant but more importantly at the moment you are not limiting your liability! The fact you've stuck this up without limiting your liability you are now partially liable for whatever stuff people are putting up on here baffles me. Seriously for your own good please get some terms of service with at least limitation of liability. You'll have to also sort the problem of copyright as I wouldn't push code to a service that attempts to claim any copyright of that which I assume you're not doing but I would need that in writing. Given the frequency of commits that this solution will introduce you'll need to offer way more than 5GB of hosting. How will that version slider work when I have 500k commits. Even squashing branches, the repo I'm working on at the moment has 10k commits. This is a team of about 30-40 devs and each day is about 15ish PRs. With Jamsync syncing every save I could easily see this going up to millions of commits.

billconan 4y ago

reminds me of https://xethub.com/assets/docs/xet-specifics/how-xet-deduplication-works https://xethub.com/assets/docs/xet-specifics/how-xet-dedupli...

kypro 4y ago

I have nothing much to add about the tool, but just want to say this is some beautiful documentation.

zdgeier 4y ago
Thanks :) I unfortunately can't take the credit for this beautiful theme. Credit goes to https://radoslawkoziel.pl/ https://radoslawkoziel.pl/ who made https://github.com/panr/hugo-theme-terminal https://github.com/panr/hugo-theme-terminal which I based this off of.
- joshspankit 4y ago
  I can’t help suggesting to fill the top bar left to right as the project progresses to 1.0

hardwaregeek 4y ago

There seem to be a few projects that try to be a live, collaborative git. Perhaps it’s just me, but I don’t necessarily think I’d want this for my main system. Maybe a short term collaboration/pair programming session, but not in general. I like how git is not immediate. I like that I can take time before sharing my code to clean it up. The issues that a tool like this purports to solve are not issues to me. Live collaboration is a great idea, but maybe not as a version control system. But hey if other people get use out of it, why not!

zdgeier 4y ago
Appreciate the feedback! Will keep this use case in mind, this could definitely work as a layer on top of Git when people need to pair program or preview code.
andix 4y ago
What I always wanted to have is a tool that syncs my uncommitted changes of git repos between all my devices. So that I can switch from the desktop to the laptop without committing, pushing and pulling. Right where I stopped. Also sync not yet pushed commits, I may want to rebase/edit before pushing.
- pmontra 4y ago
  By uncommitted changes do you mean edited files? Then something like syncthing or anything functionally similar to Dropbox.
- klabb3 4y ago
  100x yes. I agree with parent that pushing live changes is way to intrusive, but a synced working/staging directory could be incredibly helpful. I don’t see any fundamental reason why git or, even 3p tooling for git, couldn’t support this use case using something like private, ephemeral, auto-synced branches.
- nvissari 4y ago
  Here's the roundabout way I sync uncommitted changes. Push everything to a new tmp branch. Then on the other system, fetch and cherry-pick the last commit id, then reset HEAD~1. Boom uncommitted changes synced.

frogperson 4y ago

I don't think I would want this for code, but it sounds really interesting for a shared drive. I would love to version control the dir where my team dumps all of its docs and spreadsheets and everything else.

zdgeier 4y ago
Thanks for the feedback! What about a version control system would be valuable for storing your documents? I'm curious why you couldn't use something like Google Drive/Dropbox.
- jbverschoor 4y ago
  Security, local copies, and limited restoration options

account-5 4y ago

How does this compare with fossil? I use fossil for personal projects.

zdgeier 4y ago
There are some similarities but Fossil still has a push/pull/commit flow, although it does provide syncing features to automatically pull commits. Also, I don't think it has great support for large files. Ultimately, the priorities of Fossil are going to be different than Jamsync, since Jamsync is not decentralized. Being centralized means that we'll be able to do some different things, like live file editing, but with some downsides related to replication and distribution.

KerrAvon 4y ago

I don't understand the use case. Very large file support aside, are you saying you've observed people looking at git and say "I wish I could use this like I use a chat server?"

zdgeier 4y ago
Haha appreciate you taking a look. One use case is to prevent merge conflicts, especially when working on a fast-moving project. Merge conflicts generally occur when you make changes on some old code. Jamsync constantly "rebases" your changes on top of new changes so that your code will always be up to date. Obviously some merge conflicts will still occur but much less so in this approach.
- zdgeier 4y ago
  More details about the problems I'm trying to solve are on the homepage if you're curious.
- kccqzy 4y ago
  How is that different from running `git pull` or the equivalent faster, maybe on a timer?
  - zdgeier 4y ago
    This could work for the pulling side but really what jamsync does is both pull and push. Having changes that are always present in the remote means that merge conflicts are less likely. If you were to do this with Git, you would also have to automate the git add, commit, and push steps which would ruin the log in addition to being pretty slow.
    - yencabulator 4y ago
      That sounds like one developer being in the middle of an edit breaks the project immediately for everyone?
- 4y ago

ipid 4y ago

I wonder whether this system works for zip-compressed files, such as large .docx, .npz, etc. These are common large files, but it seems that rsync cannot handle compressed files. Would Jamsync solve this problem?

zdgeier 4y ago
Not currently, but it would definitely be interesting to have an unzip done automatically based on the file type. Then you would be able to track the contents efficiently, rather than having large binary blobs with large diffs. I haven't heard of other version control systems out there doing this.
antod 4y ago
Compression formats can be made to handle rsync, even if it isn't a thing out of the box - in my ignorance gzip is the only example I'm aware of. https://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/ https://beeznest.wordpress.com/2005/02/03/rsyncable-gzip/

kinleyd 4y ago

Sounds very useful to me. rsync is my go to tool - I use it to manually maintain versions of my workstation as I find it the most reliable and usable among all options I've tried.

deleted 4y ago

[deleted]

joveian 4y ago

Two comments from a quick look: 1) I recall a comment a while back from the xdelta author about how most history tools store a sequence of changes from nothing (as it sounds like yours does), but this isn't ideal since usually you want a recent version which then needs to be produced from the potentially long sequence of changes. Caching various revisions could help in your case but that is less than ideal when the files are large. 2) It isn't clear how you go from live editing type thing to revision control. If you haven't yet, you might want to look at the Aegis paper[1] (it is short) for a quick overview of the framework it used to form a revision control system from a collection of more specific tools that could be customized. That type of framework might go well with your rapid branch synchronization. Sadly the author, Peter Miller[2], died in 2014. As far as I know no one picked up development (it isn't the easiest name to search for :/). The User Guide [3] from the documentation page [4] gives a bit more info about the model. [1] https://aegis.sourceforge.net/auug93.pdf https://aegis.sourceforge.net/auug93.pdf [2] https://en.wikipedia.org/wiki/Peter_Miller_(software_engineer) https://en.wikipedia.org/wiki/Peter_Miller_(software_enginee... [3] https://aegis.sourceforge.net/user-guide.pdf https://aegis.sourceforge.net/user-guide.pdf [4] https://aegis.sourceforge.net/documents.html https://aegis.sourceforge.net/documents.html

zdgeier 4y ago
Thanks for this! 1) Agreed, caching might not be great for large files. I don't think this is the only way though. I think there is another approach, that I haven't explored, which is storing a pointer to the last known Data block in each Range block. This means that you can just jump directly to the actual data without having to go through this regeneration or caching process. The space cost of this would be pretty low, one int per block, but it would mean that you wouldn't need to iterate through every change. Still figuring this out though and caching would be the best option if this doesn't work. 2) Yes I haven't totally figured this out yet. I'll give this a look, thanks!

pmeunier 4y ago

Btw, Pijul (https://pijul.org https://pijul.org) already does this for binary files and large files. But it also has other advantages such as easier workflows, provably correct merge and cherry-picking.

JoeyBananas 4y ago

It might be better to limit the scope to "Live editing" or "source code syncing" rather than "version control".

zdgeier 4y ago
Yeah I agree that live editing is probably a more appropriate name right now but I think there are so many tools that already do that right now and haven't really made much impact. I don't see a large value in the live editing aspect when it comes to developing and deploying faster, even though this system will support it. The live editing is most valuable when preventing merge conflicts and working on several machines. Definitely open to your thoughts though.