9 ms·

How to Quickly and Correctly Generate a Git Log in HTML

94 points by foob 9y ago

falsedan 9y ago
git log --pretty=format:"%H%x00%s" | sed 's/&/\&/g; s/</\</g; s/>/\>/g; s/"/\"/g; s/'"'"'/\'/g; s@$.*$\x0$.*$@<tr><th>\1</th><td>\2</td></tr>@' You could do the dumb html entifying in a real language. The article's solution is a straw man, since it's promoting their personal language. Why did they see \x01 & \x02 as possible sentinels but not nulls? python is fine with nulls…
- kccqzy 9y ago
  People always underestimate the power of utilities specified in POSIX. When you want to do text processing without dependencies (which seem to be the goal here), just use awk.
- js2 9y ago
  cgi.escape is just this: def escape(s, quote=None): '''Replace special characters "&", "<" and ">" to HTML-safe sequences. If the optional flag quote is true, the quotation mark character (") is also translated.''' s = s.replace("&", "&") # Must be done first! s = s.replace("<", "<") s = s.replace(">", ">") if quote: s = s.replace('"', """) return s And the author would simplify the job as you point out by escaping the git log output first, then wrapping it in HTML. You don't even need the null byte. %H is a hex sequence, so even a space separator would be fine. This would do: git log --pretty='%H %s' | sed -e ' s/&/\&/g; # must be done first s/</\</g; s/>/\>/g; s/^$[0-9a-f][0-9a-f]*$ $.*$/<tr><td>\1<\/td><td>\2<\/td><\/tr>/' That said: it's best not to be cavalier about dealing with user input. This is sufficient only if the output is going into the body of the HTML document. You have to be careful about where the output is being placed: https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%29_Prevention_Cheat_Sheet#Why_Can.27t_I_Just_HTML_Entity_Encode_Untrusted_Data.3F https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%...
  - chubot 9y ago
    I addressed this here -- that solution doesn't really work with multiple fields that contain spaces: https://www.reddit.com/r/commandline/comments/719pm2/how_to_quickly_and_correctly_generate_a_git_log/dn9klp1/ https://www.reddit.com/r/commandline/comments/719pm2/how_to_...
    - js2 9y ago
      I admit at that point you might want to break out perl: git log --pretty='format:%H%x00%an%x00%aE%x00%s' | perl -F'\0' -ane ' chomp; print "<tr>\n"; foreach(@F) { s|&|&|g; s|<|<|g; s|>|>|g; print "<td>$_</td>\n"; } print "</tr>\n"' :)
- chubot 9y ago
  My solution is easier with respect to multiple fields that contain spaces. The real example has both the description and committer name. And you can add and remove fields without changing the second part of the pipeline -- you just have to add 0x00 and 0x01 in the format string. Similar qusetion here: https://www.reddit.com/r/commandline/comments/719pm2/how_to_quickly_and_correctly_generate_a_git_log/dn9klp1/ https://www.reddit.com/r/commandline/comments/719pm2/how_to_... Also, I've used that kind of sed, and it looks horrible. It makes shell look bad.
  - falsedan 9y ago
    > You could do the dumb html entifying in a real language and xargs -0 -n2 printf '<tr><th>%s</th><td>%s</td></tr>'
Sir_Cmpwn 9y ago
I don't see what was wrong with the first solution. Keep it simple!
- qznc 9y ago
  If you use < or & in your text, it is not HTML5. Neither XHTML. However, it is probably HTML4, isn't it?
  - Sir_Cmpwn 9y ago
    Probably. In any case, cross that bridge when you get to a release that has one of those symbols in the changelog.
    - yorwba 9y ago
      Did you overlook the part where <& appears in commit messages?
      - Sir_Cmpwn 9y ago
        No?
        yorwba 9y ago
        That's what was wrong with the first solution. The bridge was crossed just when it should have been.
        Sir_Cmpwn 9y ago
        Ah, I misread the article.
mattacular 9y ago
>Some programmers might stop here and say, Let's switch to a real programming language. Do it the right way. Isn't using Python switching to a real programming language?
- knodi123 9y ago
  depends on how elitist you are
- hyperpape 9y ago
  It's a little misleading how it's written. He said "use a real language", but the real distinction was using a library for the git api. He's only using the Python stdlib for text manipulation as part of a pipeline.
jacobparker 9y ago
Reinventing escaping over and over again (which bash scripts in particular seem to encourage) is a suckers game. It's difficult to get right and if you're constantly redoing it you're eventually going to make a mistake. I've worked in web security and it's sad to see how likely it is for people with good intentions to mess this up. I'm glad the author basically came to this conclusion. The winning strategy is to use a library/framework/whatever for embedding user-provided content into HTML. Sane HTML template libraries will do this. That library has had more time to get it right. Furthermore a well designed API will clearly indicate what is trusted vs. untrusted data and all untrusted data is properly encoded before being embedded. See the "Security Model" section of golangs HTML templates below. An alternative to using the git tools which is appropriate for serious work (shell pipelines are great for prototyping) is libgit2. It has bindings for many languages. It's very easy to use (sometimes (not always) easier than the CLI) and often much higher performance vs. big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) An example set of tools: https://golang.org/pkg/html/template/ https://golang.org/pkg/html/template/ + https://github.com/libgit2/git2go https://github.com/libgit2/git2go . It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment.
- falsedan 9y ago
  > An alternative to using the git tools which is appropriate for serious work This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. > big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. > It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment. I would say the ease depends on the relative difficulty and functionality of the languages. Casting bash as the insane bad language doesn't win you any points: we all know bash is a hot mess of a garbage fire, now let's use it to do useful good work. I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process.
stephenr 9y ago
How is it that git still doesn't have machine readable output built in?
- Sean1708 9y ago
  Isn't that what the --porcelain option[0] is for? And for commands that aren't porcelain, what's wrong with --oneline or --format? [0]: https://stackoverflow.com/questions/6976473/what-does-the-term-porcelain-mean-in-git#6978402 https://stackoverflow.com/questions/6976473/what-does-the-te...
- sevensor 9y ago
  In what respect is --pretty=format not producing machine-readable output? Is putting null bytes between the fields not machine-readable enough for you? You want to use 0x1d instead? You could do that.
  - stephenr 9y ago
    In the way that you have to futz around like this with null byte separators to try and avoid escaping issues. Svn had a flag to output resonses in xml specifically for machine consumption. No ambiguity about what it would provide, or if it was escaped or how to handle things like new lines etc.
- IshKebab 9y ago
  Because if you want machine readable output you shouldn't be parsing the output of other commands - you should link with libgit2. It's actually very easy.
  - sevensor 9y ago
    I'd quibble with your premise that inserting unambiguous delimiters doesn't make a machine-readable text format, but I'm so happy to have learned about the existence of libgit2 that I won't press the issue. Machine-readable text is great and all, but libgit2.so is even better.
whipoodle 9y ago
I don't want to use an API and do it the right way. That's too complicated, poindexter! (50 lines of garbage script follow)
pixelbeat__ 9y ago
Also consider https://www.pixelbeat.org/scripts/ansi2html.sh https://www.pixelbeat.org/scripts/ansi2html.sh for the general case of (colored) output to html conversion
no_protocol 9y ago
`gitweb` is a server that comes with your git install. The `gitweb` web interface includes both a log and shortlog view for repositories. You can probably use those to some benefit. This seems to be the source of the shortlog command: https://github.com/git/git/blob/master/gitweb/gitweb.perl#L5889 https://github.com/git/git/blob/master/gitweb/gitweb.perl#L5...
tzs 9y ago
The underlying problem with the first, simple, approach is that the template it is using to get things from git, "<tr> <td>%H</td> <td>%s</td> <tr>" interpolates values that need to be escaped, but includes literal text that must not be escaped. (My guess is that the author meant "</tr>" for the last element, but the article says "<tr>" so I'm going with that). The author's approach to deal with that is to mark the places in the template where escaping will be needed, and then make and use an escaping tool that recognizes those marks and just escapes the marked segments. A simpler approach is to eliminate the underlying problem. For getting the data out of git use a template where the literal text is safe to escape, such as this: "%H,%s" The escaping can then be done by a tool that escapes its entire input. That will leave the comma from the template alone, and will not introduce any new commas. The interpolation of %s might have introduced commas, but they will all be after the literal comma from the template. The interpolation of %H will not introduce commas. The output from the escaper can then be transformed into the final output by replacing the first "," with "</td> <td>", prepending "<tr> <td>", and appending "</td> <tr>". All of these are simple in a shell pipeline using sed.
- chubot 9y ago
  Several people brought up alternative solutions like this, and I addressed them: https://news.ycombinator.com/item?id=15295556 https://news.ycombinator.com/item?id=15295556 Summary: I may have oversimplified the problem in my example. I think my solution is nicer for the real problem. It has fewer assumptions and will "scale up" to more fields with arbitrary text. I want to write the escaping ONCE, not modify it every time I change the format of the output table.
deleted 9y ago
[deleted]
masukomi 9y ago
why do people insist upon reinventing the wheel badly: git log --color=always <whatever funky coloring, options, etc you want> | aha > git_log.html side note: aha is not installed by default on macOS but homebrew will fix that for you. Also, it has many color and styling options.
- chubot 9y ago
  It doesn't look like it has hyperlinks: https://github.com/theZiz/aha https://github.com/theZiz/aha Compare: http://www.oilshell.org/release/0.1.0/changelog.html http://www.oilshell.org/release/0.1.0/changelog.html Hyperlinks are the whole point of HTML :)
dahart 9y ago
You can skip having to escape any characters or worry if the content is correct, if you put an unformatted git log into a script tag, and then line split and set the content of each element via a JS call. I just tried it, and it works beautifully, no problems with illegal characters. What's wrong with this? It'd be super easy to extend if you want columns or colors or links... <script id='gitlog' type='text'> c0c3150f5 09 - 15 dahart Color widget!, #1 improving < hsv > && things [Finishes #8736345] \m/ '",.;:%$#@* </script> <div id='lines'></div> $('#gitlog').html().split('\n').forEach(line => { $('#lines').append($('<div class="line"/>').text(line)) })

94 points by foob 9y ago

git log --pretty=format:"%H%x00%s" | sed 's/&/\&/g; s/</\</g; s/>/\>/g; s/"/\"/g; s/'"'"'/\'/g; s@$.*$\x0$.*$@<tr><th>\1</th><td>\2</td></tr>@' You could do the dumb html entifying in a real language. The article's solution is a straw man, since it's promoting their personal language. Why did they see \x01 & \x02 as possible sentinels but not nulls? python is fine with nulls…

kccqzy 9y ago
People always underestimate the power of utilities specified in POSIX. When you want to do text processing without dependencies (which seem to be the goal here), just use awk.
js2 9y ago
cgi.escape is just this: def escape(s, quote=None): '''Replace special characters "&", "<" and ">" to HTML-safe sequences. If the optional flag quote is true, the quotation mark character (") is also translated.''' s = s.replace("&", "&") # Must be done first! s = s.replace("<", "<") s = s.replace(">", ">") if quote: s = s.replace('"', """) return s And the author would simplify the job as you point out by escaping the git log output first, then wrapping it in HTML. You don't even need the null byte. %H is a hex sequence, so even a space separator would be fine. This would do: git log --pretty='%H %s' | sed -e ' s/&/\&/g; # must be done first s/</\</g; s/>/\>/g; s/^$[0-9a-f][0-9a-f]*$ $.*$/<tr><td>\1<\/td><td>\2<\/td><\/tr>/' That said: it's best not to be cavalier about dealing with user input. This is sufficient only if the output is going into the body of the HTML document. You have to be careful about where the output is being placed: https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%29_Prevention_Cheat_Sheet#Why_Can.27t_I_Just_HTML_Entity_Encode_Untrusted_Data.3F https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%...
- chubot 9y ago
  I addressed this here -- that solution doesn't really work with multiple fields that contain spaces: https://www.reddit.com/r/commandline/comments/719pm2/how_to_quickly_and_correctly_generate_a_git_log/dn9klp1/ https://www.reddit.com/r/commandline/comments/719pm2/how_to_...
  - js2 9y ago
    I admit at that point you might want to break out perl: git log --pretty='format:%H%x00%an%x00%aE%x00%s' | perl -F'\0' -ane ' chomp; print "<tr>\n"; foreach(@F) { s|&|&|g; s|<|<|g; s|>|>|g; print "<td>$_</td>\n"; } print "</tr>\n"' :)
chubot 9y ago
My solution is easier with respect to multiple fields that contain spaces. The real example has both the description and committer name. And you can add and remove fields without changing the second part of the pipeline -- you just have to add 0x00 and 0x01 in the format string. Similar qusetion here: https://www.reddit.com/r/commandline/comments/719pm2/how_to_quickly_and_correctly_generate_a_git_log/dn9klp1/ https://www.reddit.com/r/commandline/comments/719pm2/how_to_... Also, I've used that kind of sed, and it looks horrible. It makes shell look bad.
- falsedan 9y ago
  > You could do the dumb html entifying in a real language and xargs -0 -n2 printf '<tr><th>%s</th><td>%s</td></tr>'

Sir_Cmpwn 9y ago

I don't see what was wrong with the first solution. Keep it simple!

qznc 9y ago
If you use < or & in your text, it is not HTML5. Neither XHTML. However, it is probably HTML4, isn't it?
- Sir_Cmpwn 9y ago
  Probably. In any case, cross that bridge when you get to a release that has one of those symbols in the changelog.
  - yorwba 9y ago
    Did you overlook the part where <& appears in commit messages?
    - Sir_Cmpwn 9y ago
      No?
      - yorwba 9y ago
        That's what was wrong with the first solution. The bridge was crossed just when it should have been.
        Sir_Cmpwn 9y ago
        Ah, I misread the article.

mattacular 9y ago

>Some programmers might stop here and say, Let's switch to a real programming language. Do it the right way. Isn't using Python switching to a real programming language?

knodi123 9y ago
depends on how elitist you are
hyperpape 9y ago
It's a little misleading how it's written. He said "use a real language", but the real distinction was using a library for the git api. He's only using the Python stdlib for text manipulation as part of a pipeline.

jacobparker 9y ago

Reinventing escaping over and over again (which bash scripts in particular seem to encourage) is a suckers game. It's difficult to get right and if you're constantly redoing it you're eventually going to make a mistake. I've worked in web security and it's sad to see how likely it is for people with good intentions to mess this up. I'm glad the author basically came to this conclusion. The winning strategy is to use a library/framework/whatever for embedding user-provided content into HTML. Sane HTML template libraries will do this. That library has had more time to get it right. Furthermore a well designed API will clearly indicate what is trusted vs. untrusted data and all untrusted data is properly encoded before being embedded. See the "Security Model" section of golangs HTML templates below. An alternative to using the git tools which is appropriate for serious work (shell pipelines are great for prototyping) is libgit2. It has bindings for many languages. It's very easy to use (sometimes (not always) easier than the CLI) and often much higher performance vs. big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) An example set of tools: https://golang.org/pkg/html/template/ https://golang.org/pkg/html/template/ + https://github.com/libgit2/git2go https://github.com/libgit2/git2go . It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment.

falsedan 9y ago
> An alternative to using the git tools which is appropriate for serious work This belittles their and anyone else's bash scripts for git as 'toys'. That's unfair, and completely unnecessary for your point. > big shell pipelines (operating on text gets slow pretty fast, and often you end up using xargs...) Big shell pipelines are actually blazingly fast, since each command in the pipeline runs in parallel. > It's not as succinct as a bash script but it's easier to build something that's correct. Use the shell to prototype, build it right in a saner environment. I would say the ease depends on the relative difficulty and functionality of the languages. Casting bash as the insane bad language doesn't win you any points: we all know bash is a hot mess of a garbage fire, now let's use it to do useful good work. I would struggle to justify the work I'd need to do to write the equivalent in C (or python, where it would be slower) for one command that's only run in the release process.

stephenr 9y ago

How is it that git still doesn't have machine readable output built in?

Sean1708 9y ago
Isn't that what the --porcelain option[0] is for? And for commands that aren't porcelain, what's wrong with --oneline or --format? [0]: https://stackoverflow.com/questions/6976473/what-does-the-term-porcelain-mean-in-git#6978402 https://stackoverflow.com/questions/6976473/what-does-the-te...
sevensor 9y ago
In what respect is --pretty=format not producing machine-readable output? Is putting null bytes between the fields not machine-readable enough for you? You want to use 0x1d instead? You could do that.
- stephenr 9y ago
  In the way that you have to futz around like this with null byte separators to try and avoid escaping issues. Svn had a flag to output resonses in xml specifically for machine consumption. No ambiguity about what it would provide, or if it was escaped or how to handle things like new lines etc.
IshKebab 9y ago
Because if you want machine readable output you shouldn't be parsing the output of other commands - you should link with libgit2. It's actually very easy.
- sevensor 9y ago
  I'd quibble with your premise that inserting unambiguous delimiters doesn't make a machine-readable text format, but I'm so happy to have learned about the existence of libgit2 that I won't press the issue. Machine-readable text is great and all, but libgit2.so is even better.

whipoodle 9y ago

I don't want to use an API and do it the right way. That's too complicated, poindexter! (50 lines of garbage script follow)

pixelbeat__ 9y ago

Also consider https://www.pixelbeat.org/scripts/ansi2html.sh https://www.pixelbeat.org/scripts/ansi2html.sh for the general case of (colored) output to html conversion

no_protocol 9y ago

`gitweb` is a server that comes with your git install. The `gitweb` web interface includes both a log and shortlog view for repositories. You can probably use those to some benefit. This seems to be the source of the shortlog command: https://github.com/git/git/blob/master/gitweb/gitweb.perl#L5889 https://github.com/git/git/blob/master/gitweb/gitweb.perl#L5...

tzs 9y ago

The underlying problem with the first, simple, approach is that the template it is using to get things from git, "<tr> <td>%H</td> <td>%s</td> <tr>" interpolates values that need to be escaped, but includes literal text that must not be escaped. (My guess is that the author meant "</tr>" for the last element, but the article says "<tr>" so I'm going with that). The author's approach to deal with that is to mark the places in the template where escaping will be needed, and then make and use an escaping tool that recognizes those marks and just escapes the marked segments. A simpler approach is to eliminate the underlying problem. For getting the data out of git use a template where the literal text is safe to escape, such as this: "%H,%s" The escaping can then be done by a tool that escapes its entire input. That will leave the comma from the template alone, and will not introduce any new commas. The interpolation of %s might have introduced commas, but they will all be after the literal comma from the template. The interpolation of %H will not introduce commas. The output from the escaper can then be transformed into the final output by replacing the first "," with "</td> <td>", prepending "<tr> <td>", and appending "</td> <tr>". All of these are simple in a shell pipeline using sed.

chubot 9y ago
Several people brought up alternative solutions like this, and I addressed them: https://news.ycombinator.com/item?id=15295556 https://news.ycombinator.com/item?id=15295556 Summary: I may have oversimplified the problem in my example. I think my solution is nicer for the real problem. It has fewer assumptions and will "scale up" to more fields with arbitrary text. I want to write the escaping ONCE, not modify it every time I change the format of the output table.

deleted 9y ago

[deleted]

masukomi 9y ago

why do people insist upon reinventing the wheel badly: git log --color=always <whatever funky coloring, options, etc you want> | aha > git_log.html side note: aha is not installed by default on macOS but homebrew will fix that for you. Also, it has many color and styling options.

chubot 9y ago
It doesn't look like it has hyperlinks: https://github.com/theZiz/aha https://github.com/theZiz/aha Compare: http://www.oilshell.org/release/0.1.0/changelog.html http://www.oilshell.org/release/0.1.0/changelog.html Hyperlinks are the whole point of HTML :)

dahart 9y ago

You can skip having to escape any characters or worry if the content is correct, if you put an unformatted git log into a script tag, and then line split and set the content of each element via a JS call. I just tried it, and it works beautifully, no problems with illegal characters. What's wrong with this? It'd be super easy to extend if you want columns or colors or links... <script id='gitlog' type='text'> c0c3150f5 09 - 15 dahart Color widget!, #1 improving < hsv > && things [Finishes #8736345] \m/ '",.;:%$#@* </script> <div id='lines'></div> $('#gitlog').html().split('\n').forEach(line => { $('#lines').append($('<div class="line"/>').text(line)) })