5 ms·

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated f

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

b3n 4y ago
Note that these files are not statically generated; they are translation files, generated by translators.
sdflhasjd 4y ago
Yes, the article glosses over that a little bit, and it's an unusual decision.
snicker7 4y ago
As a counter-point, package managers generate “lock files” that are designed to be tracked in VC. For these translation files, I’d imagine there may be occasional work to modify them even after they are initially generated.
onion2k 4y ago
They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.
- fomine3 4y ago
  I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.
  - kadoban 4y ago
    The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.
    - Gigachad 4y ago
      Depends how much generated stuff is there. We have our graphql schema in git even though its auto generated via a library. But its useful in PRs to see exactly how the schema changed as a result of the root change.
      - kadoban 4y ago
        Yeah if there's not a lot of it, and if it's easy to regenerate, it can be fine.
    - _ikke_ 4y ago
      You may want to set the '-diff' attribute for these files so that git will not show diffs for these, instead it will show 'Binary files differ'. There is also '-merge', which will cause git to not attempt to merge the contents, but just ask you to pick a side. The challenge however is then verifying the contents of these files in things like merge requests. See https://gitirc.eu/gitattributes.html https://gitirc.eu/gitattributes.html
- robertlagrant 4y ago
  It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?
  - kstenerud 4y ago
    > place into an artifact repository This adds yet another moving part to the system, and another place things can go wrong. > generated and consumed in CI as part of generating a final build output This can get quite slow, and on larger projects you have to expend a lot of effort to keep build times reasonable. Also, if you're serving a library for public consumption, you generally don't want to add the burden of extra build steps for the user to follow before they can use it. If it can all be automated to the point of invisibility to the user that's fine, but often it can't.
  - arinlen 4y ago
    > It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. This is not surprising at all. In fact, it's quite standard to commit string translations. Just because you can run the code generation/string replacement step as part of the build that does not mean it's a good idea to generate everything from scratch at every single build. String translations hardly change once they are introduced, running the build step takes significant amounts of time, and if anything fails then your product can break in critical and hard to notice ways.
    - robertlagrant 4y ago
      I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea; why is it not in this case? 2) these are not - to my understanding - the raw resource files, but rather machine-generated intermediate files. This is why it's about caching, rather than minimal source files. Additionally, to respond to your comment, if string translations don't change much then it may be possible to push them out as an internal 3rd-party library, and then they're even quicker to build.
- haizzz 4y ago
  Sorry I've been a bit misleading. These xlf files aren't generated, they're just not interacted with by engineers but they're still created and edited by humans as translations. We want to keep track of them so that if we deploy a different commit, the texts and translations in other languages will match
wichert 4y ago
Reading the article they are not generated files, but files that are never touched by developers. Translators will work with those files. I expect that for translators they have a different sparse checkout that only fetches .xlf files for their target languages.
- onion2k 4y ago
  xlf files are usually generated. They're XML. No one wants to write that by hand.
  - Xylakant 4y ago
    .docx files are archives of xml-files. No one wants to write that by hand. Or, in more words: The format of the files is just the representation on disk - it’s not directly connected to how the files are generated or edited. XML files can be written by hand with suitable editor support.
    - onion2k 4y ago
      I think we have different ideas of what "written by hand" means. Someone making a Word doc is not writing XML by hand.
  - lultimouomo 4y ago
    Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand. I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in. Translations files, even if they are not created with a plain text editor but with some other tool that handles the XML layer, are clearly not generated, as long as the translation is done by a human.
    - onion2k 4y ago
      For the app I work at the moment we use https://lokalise.com/ https://lokalise.com/. We add translation strings to a SaaS app, and then the translation team translate them. I've written a build tool that downloads the translation JSON files from the API using the CLI, or as part of our CI process. Other teams have tools that download their language packs for different iOS and Android apps. The translations are versioned in Lokalise and we using a branching strategy to manage the work. Lokalise has an option to generate xlf files (and JSON, xliff, arb, etc). This is a very typical workflow. Most people are not out there modifying xlf files by opening them in a text editor. For a start, translations usually aren't done by developers. (Huge shoutout to Lokalise btw. I can highly recommend it. It makes building a multi-lingual app across different platforms so much easier.)
epage 4y ago
I've been switching a lot of generated files to being checked in (with CI verifying they haven't drifted from the source). The primary motivation has been performance. For example, in Rust code, it means I don't need to foist the code-gen process and all the dependencies needed for it on dependent crates. I've seen this play out similar in other build systems and circumstances. The key is the data needs to be independent of other factors (like the system doing the generation) and the rate of change of the code-gen source and generator has to be relatively low.
AlfeG 4y ago
We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.
- jokethrowaway 4y ago
  If they're generated you can just re-generate from source every time you need to track changes. You're using git as a cache. You don't need to version a cache.
  - gustavohenke 4y ago
    Regenerating certain things might be fast, but some might not be. Hundreds of engineers pushing code and having to wait for these to be regenerated both locally and on CI means that caching is quite cheap after all.

From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.

b3n 4y ago

Note that these files are not statically generated; they are translation files, generated by translators.

sdflhasjd 4y ago

Yes, the article glosses over that a little bit, and it's an unusual decision.

snicker7 4y ago

As a counter-point, package managers generate “lock files” that are designed to be tracked in VC. For these translation files, I’d imagine there may be occasional work to modify them even after they are initially generated.

onion2k 4y ago

They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.

fomine3 4y ago
I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.
- kadoban 4y ago
  The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.
  - Gigachad 4y ago
    Depends how much generated stuff is there. We have our graphql schema in git even though its auto generated via a library. But its useful in PRs to see exactly how the schema changed as a result of the root change.
    - kadoban 4y ago
      Yeah if there's not a lot of it, and if it's easy to regenerate, it can be fine.
  - _ikke_ 4y ago
    You may want to set the '-diff' attribute for these files so that git will not show diffs for these, instead it will show 'Binary files differ'. There is also '-merge', which will cause git to not attempt to merge the contents, but just ask you to pick a side. The challenge however is then verifying the contents of these files in things like merge requests. See https://gitirc.eu/gitattributes.html https://gitirc.eu/gitattributes.html
robertlagrant 4y ago
It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. Is there a reason why that type of file couldn't be better place into an artifact repository, or just generated and consumed in CI as part of generating a final build output?
- kstenerud 4y ago
  > place into an artifact repository This adds yet another moving part to the system, and another place things can go wrong. > generated and consumed in CI as part of generating a final build output This can get quite slow, and on larger projects you have to expend a lot of effort to keep build times reasonable. Also, if you're serving a library for public consumption, you generally don't want to add the burden of extra build steps for the user to follow before they can use it. If it can all be automated to the point of invisibility to the user that's fine, but often it can't.
- arinlen 4y ago
  > It's still surprising to have any generated things there. E.g. you could make the same case for keeping built binaries in Git as well. This is not surprising at all. In fact, it's quite standard to commit string translations. Just because you can run the code generation/string replacement step as part of the build that does not mean it's a good idea to generate everything from scratch at every single build. String translations hardly change once they are introduced, running the build step takes significant amounts of time, and if anything fails then your product can break in critical and hard to notice ways.
  - robertlagrant 4y ago
    I'm not saying don't have the translations at all. I'm saying: 1) caching things in git in general is a bad idea; why is it not in this case? 2) these are not - to my understanding - the raw resource files, but rather machine-generated intermediate files. This is why it's about caching, rather than minimal source files. Additionally, to respond to your comment, if string translations don't change much then it may be possible to push them out as an internal 3rd-party library, and then they're even quicker to build.
haizzz 4y ago
Sorry I've been a bit misleading. These xlf files aren't generated, they're just not interacted with by engineers but they're still created and edited by humans as translations. We want to keep track of them so that if we deploy a different commit, the texts and translations in other languages will match

wichert 4y ago

Reading the article they are not generated files, but files that are never touched by developers. Translators will work with those files. I expect that for translators they have a different sparse checkout that only fetches .xlf files for their target languages.

onion2k 4y ago
xlf files are usually generated. They're XML. No one wants to write that by hand.
- Xylakant 4y ago
  .docx files are archives of xml-files. No one wants to write that by hand. Or, in more words: The format of the files is just the representation on disk - it’s not directly connected to how the files are generated or edited. XML files can be written by hand with suitable editor support.
  - onion2k 4y ago
    I think we have different ideas of what "written by hand" means. Someone making a Word doc is not writing XML by hand.
- lultimouomo 4y ago
  Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand. I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in. Translations files, even if they are not created with a plain text editor but with some other tool that handles the XML layer, are clearly not generated, as long as the translation is done by a human.
  - onion2k 4y ago
    For the app I work at the moment we use https://lokalise.com/ https://lokalise.com/. We add translation strings to a SaaS app, and then the translation team translate them. I've written a build tool that downloads the translation JSON files from the API using the CLI, or as part of our CI process. Other teams have tools that download their language packs for different iOS and Android apps. The translations are versioned in Lokalise and we using a branching strategy to manage the work. Lokalise has an option to generate xlf files (and JSON, xliff, arb, etc). This is a very typical workflow. Most people are not out there modifying xlf files by opening them in a text editor. For a start, translations usually aren't done by developers. (Huge shoutout to Lokalise btw. I can highly recommend it. It makes building a multi-lingual app across different platforms so much easier.)

epage 4y ago

I've been switching a lot of generated files to being checked in (with CI verifying they haven't drifted from the source). The primary motivation has been performance. For example, in Rust code, it means I don't need to foist the code-gen process and all the dependencies needed for it on dependent crates. I've seen this play out similar in other build systems and circumstances. The key is the data needs to be independent of other factors (like the system doing the generation) and the rate of change of the code-gen source and generator has to be relatively low.

AlfeG 4y ago

We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.

jokethrowaway 4y ago
If they're generated you can just re-generate from source every time you need to track changes. You're using git as a cache. You don't need to version a cache.
- gustavohenke 4y ago
  Regenerating certain things might be fast, but some might not be. Hundreds of engineers pushing code and having to wait for these to be regenerated both locally and on CI means that caching is quite cheap after all.