7 ms·

Instead of teaching employees to use git, why don't you put all of your data files on S3 and enable versioning? If the non-engineers are just using git to uplo

Instead of teaching employees to use git, why don't you put all of your data files on S3 and enable versioning? If the non-engineers are just using git to upload and version json files full of data without modifying the code that reads them, uploading to s3 seems like a much safer and easier way to accomplish the same goal (without building CRUD interfaces to handle it).

jopsen 9y ago
you won't get global versioning, issue integration, PRs, etc..
- learc83 9y ago
  You're essentially replacing a database and a CRUD interface, which doesn't have any of those features either. I can't see how any of that is worth the extra time spent teaching git, or the extra complexity that comes along with merging your code and data into one repository.
  - danpalmer 9y ago
    A python file with a list in it, is significantly simpler than a remotely stored data file, which brings with it parsing, error handling, issues developing offline, and still requires AWS permission understanding and some S3 teaching. Git, and GitHub, at the level we teach them, are not very complex.
codemac 9y ago
It is an incredible failure of modern version control systems that distributed branching means that they're impossible for novices to use. However, if you see github.com as a proprietary UI for git, it looks almost exactly like putting your files on S3 with some CRUD UI around it.
- learc83 9y ago
  They're still teaching them about branches, pull requests, commit messages, merging, and build statuses. Not to mention that mixing data and code into the same versioning system adds complexity that I don't think is maintainable. There is a reason we tend to version data separately from code.
  - rpedela 9y ago
    What complexity is added by keeping data in the tree? If we are talking about GB+ datasets, then yes that shouldn't be stored in the tree. But a small list of products that needs to be manually updated regardless of where it is stored? That is perfectly fine to store in the tree and I think the author was talking about that sort of data. As always, it depends on the data. Sometimes git is fine. Sometimes you need S3. Sometimes Postgres. There is no one size fits all solution.
    - learc83 9y ago
      What complexity is added by keeping data in the tree? Polluting the commit logs with thousands of extra commit messages if you're dealing with fast changing data for one. If a developer is working on a "data file" that happens to contain logic (its going to happen no matter how diligent you are about keeping them separate), dealing with unnecessary merging from the constant edits is another. >That is perfectly fine to store in the tree and I think the author was talking about that sort of data. The individual datasets may be small, but from reading the authors comments here, the scope of what they're storing this way seems to be way beyond what I'd feel comfortable managing this way.
  - madeofpalk 9y ago
    No, they’re not. We’ve used this approach before we were able to intergrate required CMS, and it was fine. Our Product Owner would go into the YAML files in Github, click "Edit", make the changes, then hit the big green ”Create Pull Request" button. GitHub actually makes it fairly easy to do this flow and comes with sane default commit messages.
tootie 9y ago
That's the niche that Box, Dropbox, GDrive, OneDrive and others are already living in. In fact, there is new generation of tools that offer way more value adds for niche uses like Zeplin for designers.
danpalmer 9y ago
Many of these files in our case are imported in Python so need to be in the codebase and committed. Some of them are templates which also need to be in the repositories.
- learc83 9y ago
  >Some of them are templates which also need to be in the repositories. Letting people who are editing templates use git seems perfectly fine to me. Using git to version constantly changing data is what I think isn't maintainable. I think at some point that polluting the commit logs with data changes, dealing with merging issues, and training everyone who wants to change data on git imposes a hard limit on scaleability for this approach. You may never hit that limit, but I don't think this would work for very large teams, and I suspect that if you git to the point that it stops working, it will be very difficult to sort it out.
  - danpalmer 9y ago
    I agree, the point here is to push off the cost of building the interfaces and making the data fully dynamic until they are actually needed, because we often find they aren't. As for the commit pollution, I don't see this as much of a problem. In the long term we're always going to have lots of commits!
cortesoft 9y ago
Because developers are already using git, so why have separate tools for similar use cases?
- wyclif 9y ago
  Because non-technical staff are expected to access, interact, and keep up to date with developer workflow using git now. Seems like abstracting some of the higher-order functionality away while adding project management amenities and making git friendly for those people might be an opportunity. For instance, I'm thinking of products like Conveyor: https://wildbit.com/blog/2017/02/14/our-next-product-conveyor https://wildbit.com/blog/2017/02/14/our-next-product-conveyo...
  - wyclif 9y ago
    EDIT: meant to say "GitHub" instead of git.
chaostheory 9y ago
Branching, assuming that non-engineers are using a good GUI, is a really useful concept and idea for people to be exposed to. It helps people take more intellectual risks safely and cheap

jopsen 9y ago

you won't get global versioning, issue integration, PRs, etc..

learc83 9y ago
You're essentially replacing a database and a CRUD interface, which doesn't have any of those features either. I can't see how any of that is worth the extra time spent teaching git, or the extra complexity that comes along with merging your code and data into one repository.
- danpalmer 9y ago
  A python file with a list in it, is significantly simpler than a remotely stored data file, which brings with it parsing, error handling, issues developing offline, and still requires AWS permission understanding and some S3 teaching. Git, and GitHub, at the level we teach them, are not very complex.

codemac 9y ago

It is an incredible failure of modern version control systems that distributed branching means that they're impossible for novices to use. However, if you see github.com as a proprietary UI for git, it looks almost exactly like putting your files on S3 with some CRUD UI around it.

learc83 9y ago
They're still teaching them about branches, pull requests, commit messages, merging, and build statuses. Not to mention that mixing data and code into the same versioning system adds complexity that I don't think is maintainable. There is a reason we tend to version data separately from code.
- rpedela 9y ago
  What complexity is added by keeping data in the tree? If we are talking about GB+ datasets, then yes that shouldn't be stored in the tree. But a small list of products that needs to be manually updated regardless of where it is stored? That is perfectly fine to store in the tree and I think the author was talking about that sort of data. As always, it depends on the data. Sometimes git is fine. Sometimes you need S3. Sometimes Postgres. There is no one size fits all solution.
  - learc83 9y ago
    What complexity is added by keeping data in the tree? Polluting the commit logs with thousands of extra commit messages if you're dealing with fast changing data for one. If a developer is working on a "data file" that happens to contain logic (its going to happen no matter how diligent you are about keeping them separate), dealing with unnecessary merging from the constant edits is another. >That is perfectly fine to store in the tree and I think the author was talking about that sort of data. The individual datasets may be small, but from reading the authors comments here, the scope of what they're storing this way seems to be way beyond what I'd feel comfortable managing this way.
- madeofpalk 9y ago
  No, they’re not. We’ve used this approach before we were able to intergrate required CMS, and it was fine. Our Product Owner would go into the YAML files in Github, click "Edit", make the changes, then hit the big green ”Create Pull Request" button. GitHub actually makes it fairly easy to do this flow and comes with sane default commit messages.

tootie 9y ago

That's the niche that Box, Dropbox, GDrive, OneDrive and others are already living in. In fact, there is new generation of tools that offer way more value adds for niche uses like Zeplin for designers.

danpalmer 9y ago

Many of these files in our case are imported in Python so need to be in the codebase and committed. Some of them are templates which also need to be in the repositories.

learc83 9y ago
>Some of them are templates which also need to be in the repositories. Letting people who are editing templates use git seems perfectly fine to me. Using git to version constantly changing data is what I think isn't maintainable. I think at some point that polluting the commit logs with data changes, dealing with merging issues, and training everyone who wants to change data on git imposes a hard limit on scaleability for this approach. You may never hit that limit, but I don't think this would work for very large teams, and I suspect that if you git to the point that it stops working, it will be very difficult to sort it out.
- danpalmer 9y ago
  I agree, the point here is to push off the cost of building the interfaces and making the data fully dynamic until they are actually needed, because we often find they aren't. As for the commit pollution, I don't see this as much of a problem. In the long term we're always going to have lots of commits!

cortesoft 9y ago

Because developers are already using git, so why have separate tools for similar use cases?

wyclif 9y ago
Because non-technical staff are expected to access, interact, and keep up to date with developer workflow using git now. Seems like abstracting some of the higher-order functionality away while adding project management amenities and making git friendly for those people might be an opportunity. For instance, I'm thinking of products like Conveyor: https://wildbit.com/blog/2017/02/14/our-next-product-conveyor https://wildbit.com/blog/2017/02/14/our-next-product-conveyo...
- wyclif 9y ago
  EDIT: meant to say "GitHub" instead of git.

chaostheory 9y ago

Branching, assuming that non-engineers are using a good GUI, is a really useful concept and idea for people to be exposed to. It helps people take more intellectual risks safely and cheap