5 ms·

Using strace to figure out how Git push over SSH works

111 points by luu 11y ago

krallja 11y ago
Cool hack. Another way to solve the mystery is to read the book: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols#The-Smart-Protocol https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protoc...
- shurcooL 11y ago
  As someone who implemented git over SSH in Go recently, I found that page helpful but frustratingly high-level of an overview. It didn't answer some lower-level questions I had.
  - js2 11y ago
    You can try https://github.com/git/git/tree/master/Documentation/technical https://github.com/git/git/tree/master/Documentation/technic... But sometimes you have to roll up your sleeves and read the source. I also found the git mailing list to be pretty helpful.
ctur 11y ago
strace is very powerful, but probably not the tool I would jump to at first to solve this mystery. Instead, just go to the source code. Being able to read, interpret, understand, and, eventually, modify other peoples' code is a foundational skill both for industry and open source work. Reading code is often harder than writing code, too, so it's a worthwhile outlet for your natural curiosity, too. (For this particular mystery, you also could have tried Git's own tracing via setting GIT_TRACE=1 in your environment; many tools, especially ones that work over a network, offer some kind of debugging or tracing). I tend to keep a copy of glibc, Python, coreutils, openssl, and binutils laying around for random grep'ing. I also sometimes keep other more specific tools and libraries laying around, depending on what I'm working on. And, of course, whatever code you use at your place of work. If you want to have some fun, strace strace to see how it works. Or strace lsof, or ps, or df, or other tools that inspect the state of your system.
- agumonkey 11y ago
  Painfully hard for me, I end up mapping the whole things to have a full view of it. Maybe with time efficiency comes.
- jo909 11y ago
  strace is a very good first tool to use. It's extremely quick and easy to run, and gives a wonderful view of how the process interacts with the system or other processes. You also notice quickly if you can't find your answer with strace, if what you are interested in is not done with a syscall. In the source code you are looking at layers over layers of abstraction, variables being passed around and mangled and substituted into strings, maybe get into a dependency tree of various libraries. Not to say that you are wrong, the source the ultimate most powerful tool and absolutely you should learn to use it. But I disagree with what tool to use first, if you have zero knowledge about the process.
- dspillett 11y ago
  > strace is very powerful, but probably not the tool I would jump to at first to solve this mystery. Instead, just go to the source code. Though from a learning perspective, it might be useful to play with something like git using strace then compare with what you find in the code. This could help you learn how to interpret results when you don't have access to the code when diagnosing something else later.
lamby 11y ago
strace is definitely a fun toy. Here's my favourite hack: https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-progress-bar-like-wget https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-prog...
- fit2rule 11y ago
  Definitely a cute hack if you don't have the pv tool around, but if you do: better to use pv than have an strace hackery dependency, imho. Still, a very good way to get started with understanding what strace is doing ..
- david-given 11y ago
  That's one of the most evil things I've seen for a long time (and I have done many evil things).
- nikanj 11y ago
  Most versions of cp also respond to ctrl-t by printing current statistics
  - raimue 11y ago
    On *BSD and OS X the keyboard command Ctrl-T sends a SIGINFO to all processes connected to the tty. Many tools such as cp, mv and even dd display their progress when they receive a SIGINFO. This signal does not exist on Linux.
- dspillett 11y ago
  Or you could replace "cp" with "rsync --progess" for everything. rsync has become a habit for me and I only use cp these days when that isn't available (or in scripts that might one day run where rsync isn't installed) or for "cp -al" (which I could possibly do with rsync too). But the strace-cp-progress hack is an interesting snippet for understanding some of the things you can do with strace.
  - vincentkriek 11y ago
js2 11y ago
Then at some point they wanted to add some metadata to the protocol without breaking compatibility with older versions of git. They came up with a nifty hack that exploits C’s null-terminated strings: add the metadata after a null byte but before the newline. I believe this is the commit which made that change, very early in Git's history: http://article.gmane.org/gmane.comp.version-control.git/14051 http://article.gmane.org/gmane.comp.version-control.git/1405...
aji 11y ago
The hack is cool, but the post's title could use a lot of work. The article doesn't really talk about how git push over SSH works, and strace isn't even mentioned outside of the first paragraph.
erikb 11y ago
terminating lines in streams with null bytes is not really a git specific hack btw. Many shell tools have options for that as well and you should always use them if you pipe stuff from one program to another. Don't remember the reason though.

111 points by luu 11y ago

Cool hack. Another way to solve the mystery is to read the book: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols#The-Smart-Protocol https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protoc...

shurcooL 11y ago
As someone who implemented git over SSH in Go recently, I found that page helpful but frustratingly high-level of an overview. It didn't answer some lower-level questions I had.
- js2 11y ago
  You can try https://github.com/git/git/tree/master/Documentation/technical https://github.com/git/git/tree/master/Documentation/technic... But sometimes you have to roll up your sleeves and read the source. I also found the git mailing list to be pretty helpful.

ctur 11y ago

strace is very powerful, but probably not the tool I would jump to at first to solve this mystery. Instead, just go to the source code. Being able to read, interpret, understand, and, eventually, modify other peoples' code is a foundational skill both for industry and open source work. Reading code is often harder than writing code, too, so it's a worthwhile outlet for your natural curiosity, too. (For this particular mystery, you also could have tried Git's own tracing via setting GIT_TRACE=1 in your environment; many tools, especially ones that work over a network, offer some kind of debugging or tracing). I tend to keep a copy of glibc, Python, coreutils, openssl, and binutils laying around for random grep'ing. I also sometimes keep other more specific tools and libraries laying around, depending on what I'm working on. And, of course, whatever code you use at your place of work. If you want to have some fun, strace strace to see how it works. Or strace lsof, or ps, or df, or other tools that inspect the state of your system.

agumonkey 11y ago
Painfully hard for me, I end up mapping the whole things to have a full view of it. Maybe with time efficiency comes.
jo909 11y ago
strace is a very good first tool to use. It's extremely quick and easy to run, and gives a wonderful view of how the process interacts with the system or other processes. You also notice quickly if you can't find your answer with strace, if what you are interested in is not done with a syscall. In the source code you are looking at layers over layers of abstraction, variables being passed around and mangled and substituted into strings, maybe get into a dependency tree of various libraries. Not to say that you are wrong, the source the ultimate most powerful tool and absolutely you should learn to use it. But I disagree with what tool to use first, if you have zero knowledge about the process.
dspillett 11y ago
> strace is very powerful, but probably not the tool I would jump to at first to solve this mystery. Instead, just go to the source code. Though from a learning perspective, it might be useful to play with something like git using strace then compare with what you find in the code. This could help you learn how to interpret results when you don't have access to the code when diagnosing something else later.

lamby 11y ago

strace is definitely a fun toy. Here's my favourite hack: https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-progress-bar-like-wget https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-prog...

fit2rule 11y ago
Definitely a cute hack if you don't have the pv tool around, but if you do: better to use pv than have an strace hackery dependency, imho. Still, a very good way to get started with understanding what strace is doing ..
david-given 11y ago
That's one of the most evil things I've seen for a long time (and I have done many evil things).
nikanj 11y ago
Most versions of cp also respond to ctrl-t by printing current statistics
- raimue 11y ago
  On *BSD and OS X the keyboard command Ctrl-T sends a SIGINFO to all processes connected to the tty. Many tools such as cp, mv and even dd display their progress when they receive a SIGINFO. This signal does not exist on Linux.
dspillett 11y ago
Or you could replace "cp" with "rsync --progess" for everything. rsync has become a habit for me and I only use cp these days when that isn't available (or in scripts that might one day run where rsync isn't installed) or for "cp -al" (which I could possibly do with rsync too). But the strace-cp-progress hack is an interesting snippet for understanding some of the things you can do with strace.
- vincentkriek 11y ago

js2 11y ago

Then at some point they wanted to add some metadata to the protocol without breaking compatibility with older versions of git. They came up with a nifty hack that exploits C’s null-terminated strings: add the metadata after a null byte but before the newline. I believe this is the commit which made that change, very early in Git's history: http://article.gmane.org/gmane.comp.version-control.git/14051 http://article.gmane.org/gmane.comp.version-control.git/1405...

aji 11y ago

The hack is cool, but the post's title could use a lot of work. The article doesn't really talk about how git push over SSH works, and strace isn't even mentioned outside of the first paragraph.

erikb 11y ago

terminating lines in streams with null bytes is not really a git specific hack btw. Many shell tools have options for that as well and you should always use them if you pipe stuff from one program to another. Don't remember the reason though.