4 ms·

Sorry, can you explain this? > Not all files can be committed to GitHub ... very long lines ... VCS directories Wait so github will reject a push if one of th

Sorry, can you explain this? > Not all files can be committed to GitHub ... very long lines ... VCS directories Wait so github will reject a push if one of the commits contains a file with very long lines in it?

FridgeSeal 3y ago
Maybe it’s a protection for windows users? Windows has some arbitrary and reasonably small path limit IIRC.
- Thorrez 3y ago
  The path length limit only applies to files and directory names, it doesn't limit how long the lines of text can be in the content of a file.
jwilk 3y ago
Maybe they meant very long names?
orf 3y ago
Not quite: the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing. The “very long lines” exclusion is for text files that are very large but contain less than 5 or so lines. If I remember correctly there are some “py-armour” files that are basically just big dumps of base64 encoded python byte code on two or so lines. These are likely unique files and so won’t compress well, which bloats the size of the repositories. If you’re interested you can use the SQL console on the datasets page to take a look at the specific files.
- KirillPanov 3y ago
  > The “very long lines” exclusion is for text files that are very large but contain less than 5 or so lines. And github will block a push containing a commit which creates one of these files? And git itself does not?
  - orf 3y ago
    No, if you check above: "the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing." Github has a max size limit per repository. You can push a 150gb git repository to github: it won't stop you, but you'll get a message from support telling you remove it within 48 hours. By "heuristically excluding some text files that won’t compress well based on their content and size before committing", we are able to keep the size bound of the repositories within those set by Github whilst keeping the majority of useful code. We have excluded 26,610 .py files for being too large, whilst keeping 414,228,665 .py files. The average size for an excluded .py file is 15 MiB, vs ~10KiB for included files. 1. https://play.clickhouse.com/play?user=play#c2VsZWN0IHNraXBfcmVhc29uLAogICAgICAgY291bnQoKikgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFzIHRvdGFsX2ZpbGVzLAogICAgICAgZm9ybWF0UmVhZGFibGVTaXplKHN1bShzaXplKSkgICAgICAgICAgICAgICAgICAgIGFzIHRvdGFsX3NpemUsCiAgICAgICBmb3JtYXRSZWFkYWJsZVNpemUocXVhbnRpbGUoMC45KShzaXplKSkgYXMgcDkwX3NpemUsCiAgICAgICBmb3JtYXRSZWFkYWJsZVNpemUoYXZnKHNpemUpKSBhcyBhdmVyYWdlX3NpemUKZnJvbSBweXBpCndoZXJlIGFycmF5RWxlbWVudChzcGxpdEJ5Q2hhcignLicsIGFycmF5RWxlbWVudChzcGxpdEJ5Q2hhcignLycsIHBhdGgpLCAtMSkpLCAtMSkgPSAncHknCmdyb3VwIGJ5IDEKb3JkZXIgYnkgc3VtKHNpemUpIGRlc2MKbGltaXQgMTA7Cg== https://play.clickhouse.com/play?user=play#c2VsZWN0IHNraXBfc...

FridgeSeal 3y ago

Maybe it’s a protection for windows users? Windows has some arbitrary and reasonably small path limit IIRC.

Thorrez 3y ago
The path length limit only applies to files and directory names, it doesn't limit how long the lines of text can be in the content of a file.

jwilk 3y ago

Maybe they meant very long names?

orf 3y ago

Not quite: the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing. The “very long lines” exclusion is for text files that are very large but contain less than 5 or so lines. If I remember correctly there are some “py-armour” files that are basically just big dumps of base64 encoded python byte code on two or so lines. These are likely unique files and so won’t compress well, which bloats the size of the repositories. If you’re interested you can use the SQL console on the datasets page to take a look at the specific files.

KirillPanov 3y ago
> The “very long lines” exclusion is for text files that are very large but contain less than 5 or so lines. And github will block a push containing a commit which creates one of these files? And git itself does not?
- orf 3y ago
  No, if you check above: "the aim is to keep the size of the repositories within a manageable bound. This is achieved by heuristically excluding some text files that won’t compress well based on their content and size before committing." Github has a max size limit per repository. You can push a 150gb git repository to github: it won't stop you, but you'll get a message from support telling you remove it within 48 hours. By "heuristically excluding some text files that won’t compress well based on their content and size before committing", we are able to keep the size bound of the repositories within those set by Github whilst keeping the majority of useful code. We have excluded 26,610 .py files for being too large, whilst keeping 414,228,665 .py files. The average size for an excluded .py file is 15 MiB, vs ~10KiB for included files. 1. https://play.clickhouse.com/play?user=play#c2VsZWN0IHNraXBfcmVhc29uLAogICAgICAgY291bnQoKikgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFzIHRvdGFsX2ZpbGVzLAogICAgICAgZm9ybWF0UmVhZGFibGVTaXplKHN1bShzaXplKSkgICAgICAgICAgICAgICAgICAgIGFzIHRvdGFsX3NpemUsCiAgICAgICBmb3JtYXRSZWFkYWJsZVNpemUocXVhbnRpbGUoMC45KShzaXplKSkgYXMgcDkwX3NpemUsCiAgICAgICBmb3JtYXRSZWFkYWJsZVNpemUoYXZnKHNpemUpKSBhcyBhdmVyYWdlX3NpemUKZnJvbSBweXBpCndoZXJlIGFycmF5RWxlbWVudChzcGxpdEJ5Q2hhcignLicsIGFycmF5RWxlbWVudChzcGxpdEJ5Q2hhcignLycsIHBhdGgpLCAtMSkpLCAtMSkgPSAncHknCmdyb3VwIGJ5IDEKb3JkZXIgYnkgc3VtKHNpemUpIGRlc2MKbGltaXQgMTA7Cg== https://play.clickhouse.com/play?user=play#c2VsZWN0IHNraXBfc...