8 ms·

YAPF – A formatter for Python files

142 points by sudmishra 12y ago

B4CKlash 12y ago
>YAPF is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.
- yeukhon 12y ago
  Sounds like one of those "20%" kind of project and Google GitHubers just happened to have the permission to release this project under google/ wow ...
  - nostrademons 12y ago
    True, but that category also includes protobufs, LevelDB, Snappy, Gumbo, Guice, re2, gtest, and Angular (initially; it soon became an official thing when it got popular). While the "official" open-source projects are things like Chrome, Android, GWT, Go, Dart, gRPC, and Bazel. Paul Graham once said that you should use the tools that programmers build to solve their own problems, not the tools that big corporations build to solve their ideas of what other peoples' problems are. That doesn't mean every little library is a good one (Sturgeon's Law applies - 90% of everything is crap), but it does mean that being "official" is usually a negative signal on product quality. This applies to other companies' code as well; I've never used a buggy piece of shit quite so bad as Sun's JSF (which was supposed to be the "official" way to build webapps with Java, circa 2004-2006), while BSD and Linux continue to be great pieces of software 25 years later.
fweespeech 12y ago
> YAPF is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google. Just in case anyone thought "by Google" meant "a Google product" like me.
rbanffy 12y ago
--style STYLE specify formatting style: either a style name (for example "pep8" or "google"), or the name of a file with style settings. pep8 is the default. > pep8 is the default. I can breathe again.
- rudolf0 12y ago
  "Don't be evil" still sometimes holds true. :)
- rcthompson 12y ago
  I don't think there was any other option. If the default wasn't pep8, the tool would have no chance of seeing wide adoption in the Python community, unless someone forked it and changed the default to pep8.
baq 12y ago
i use autopep8 and haven't been thrilled. it gets the job done, but i'll switch in a heartbeat if this is better. we'll see.
- eliben 12y ago
  yapf's approach is, philosiphically, different from autopep8. yapf doesn't just fix pep8 violations. It takes a look at your whole code, and reformats it to a canonical form. If you've ever used "go fmt" or "clang-format" (for C/C++), this is the same idea
- morbo_yapf 12y ago
  YAPF is still brand spanking new, so please use it and report any bugs you find. We mention in the documentation that data literals are a sticking point to most people and most automatic formatters. Even clang-format tends to balk on them. We try our best at them, but it might be best to just disable formatting them if they are already formatted to something you like. :-)
bkcooper 12y ago
I'm sure there are some cases where this is useful, but I'm not really sold. The pitch at the beginning is "satisfying PEP 8 doesn't mean it looks good!" But then it seems most of the changes cleaned up in the code (indentation level, indents on split lines, spacing between operators) are just PEP 8 violations. I think you could find a more convincing demonstration. The warning about choking on large data literals (which are probably one of the places where prettifying would be most useful, at least to me) also seems ominous. edit: for my personal use, I tend to use flycheck with flake8 in emacs. This keeps me honest. Is the primary use case for something like this cleaning your own code or other people's?
- ngoldbaum 12y ago
  I do this too but I really don't like the noticeable lag that gets introduced as flycheck shells out to python and parses my file every time I change something.
  - klibertp 12y ago
    Are you sure the lag is because of Python? I'd check flymake options first, it may wait for some time after you stop typing and only then start linting. This delay is a trade-off: too short and you're going to have currently edited line marked with red most of the time, too long and you need to wait to see if everything is alright.
    - ngoldbaum 12y ago
      Lazy HN for the win! Thanks for the suggestion.
- bpicolo 12y ago
  The point is there are things that don't violate pep8 that are still atrocious for example: some_function(long_variable_name_and_stuff, long_variable_name_and_stuff, long_variable_name_and_stuff) The lower indentation level doesnt actually matter at all says flake8 edit: To below, that's possible. I can't remember the exact case right now but there are things along these lines that are valid and shouldn't be. Multiple possible indentation formats is not necessarily something you want to allow as an org, which is where something like YAPF comes into play
mas1n 12y ago
Interesting to see the formatter is written using 2 space indentation when both pep8 and google's style guides say 4.
- Pirate-of-SV 12y ago
  I think the original Python Google Style guide did use 2 spaces. https://code.google.com/p/soc/wiki/PythonStyleGuide#Indentation https://code.google.com/p/soc/wiki/PythonStyleGuide#Indentat... And it seems like that's used in YAPF as well: https://github.com/google/yapf/blob/master/yapf/yapflib/style.py#L122 https://github.com/google/yapf/blob/master/yapf/yapflib/styl...
  - geoelectric 12y ago
    This was submitted not long ago, and that was pointed out there too. https://news.ycombinator.com/item?id=9260786 https://news.ycombinator.com/item?id=9260786 I suggested to the author that they restore "google" style to the public one and rename the existing one to "chromium" style, since that's the most prominent public project still on Google's internal style.
Pirate-of-SV 12y ago
I've been using YAPF for a while and I really like the idea. Be aware that there are still some nasty bugs that haven't been fixed yet so be careful if you use it for something important. Do anyone of you guys know what the name YAPF comes from? (The only thing I can think of is "Yet Another Python Formatter", but I'm just guessing)
- eliben 12y ago
  Indeed, it's just "Yet Another Python Formatter"
sergiotapia 12y ago
Also see go fmt
- pekk 12y ago
  Thought experiment: wouldn't it be just awesome if Google decided to promote an alternative to the format used by go fmt, so that newbies could learn that instead and then have a constant conflict with the core Go community? edit: this is directly relevant to the thread, which is about Python formatting and Google promoting its weird style
zk00006 12y ago
Is it possible to YAPF it in Sublime?
- jkane 11y ago
  For sublime 2 yes: https://github.com/jason-kane/PyYapf https://github.com/jason-kane/PyYapf It isn't in Sublime package control yet.
- morbo_yapf 12y ago
  We need to have plugins for all editors. I'm not familiar with Sublime. But I would welcome people submitting plugins for it. :-)
Fuzzwah 12y ago
I've got a python project where I've been pretty naughty with long lines containing sql queries. I found that running this tool over it resulted in some strange new lines. For example, this was my old hideously long line: query_qual = "INSERT OR REPLACE INTO Qualifying (poleid, seasonid, race_week_num, carclassid, pole) values (%s%s, %s, %s, %s, %s)" % (race[u'seasonid'], race[u'race_week_num'], race[u'seasonid'], race[u'race_week_num'], race[u'carclassid'], pole) Changed to this: query_qual = "INSERT OR REPLACE INTO Qualifying (poleid, seasonid, race_week_num, carclassid, pole) values (%s%s, %s, %s, %s, %s)" % ( race[u'seasonid'], race[u'race_week_num'], race[ u'seasonid' ], race[u'race_week_num'], race[u'carclassid'], pole)
- Pirate-of-SV 12y ago
  You could try set the line length penalty so something really high. That would probably sort things out.
- spott 12y ago
  newlines need to be escaped somehow (two newlines seems to do it), otherwise they are combined into the same line: Every word in this sentence was entered on a different line.
- eliben 12y ago
  Feel free to submit a Github issue for yapf
- hyperpallium 12y ago
  To my eye, HN has rendered them identically. You can preserve literal whitespace on HN by indenting by two spaces (and a newline to separate the code formatting from ordinary text). it appears like this
teddyh 12y ago
There seems to be a problem parsing backslash escapes: echo 'foo="\\"' | PYTHONPATH=/tmp/yapf/yapf python /tmp/yapf Raises a lib2to3.pgen2.parse.ParseError. Also crashes on \n, but not, strangely, on \t. I’m guessing that the backslashes get interpreted twice, somehow.
- eliben 12y ago
  Are you sure this is not a shell artifact? Does the same problem happen when there's a \\ in a file? In any case, feel free to open a Github issue for yapf, we'll take a look
  - teddyh 12y ago
    Yes, it was when parsing a file I encountered it. The one-liner shell command is just a minimal repro of the bug. Apparently I need to “Sign in” to report a Github issue, so I won’t.
    - eliben 12y ago
      Sorry, I can't reproduce this. I'm trying on this file: https://gist.github.com/eliben/9727758f4847d2e7d86e https://gist.github.com/eliben/9727758f4847d2e7d86e And yapf runs just fine on it. Does this file work for you?
      - teddyh 12y ago
        Nope. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/yapf/yapf/__main__.py", line 18, in <module> sys.exit(yapf.main(sys.argv)) File "/tmp/yapf/yapf/__init__.py", line 104, in main verify=args.verify)) File "/tmp/yapf/yapf/yapflib/yapf_api.py", line 99, in FormatCode tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n') File "/tmp/yapf/yapf/yapflib/pytree_utils.py", line 100, in ParseCodeToTree tree = parser_driver.parse_string(code, debug=False) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string return self.parse_tokens(tokens, debug) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 71, in parse_tokens if p.addtoken(type, value, (prefix, start)): File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 116, in addtoken ilabel = self.classify(type, value, context) File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 172, in classify raise ParseError("bad token", type, value, context) lib2to3.pgen2.parse.ParseError: bad token: type=55, value=u' ', context=('', (1, 5))
alok-g 12y ago
Off-topic, but why is no space before opening parenthesis the norm in programming ("class foo(object):") when normal English usage is to have one? I know myself as the only one who puts this space.
- jdnier 12y ago
  I think of it as indicating the function name and parameter list are a unit (it's foo(object), not foo (object)). I've always liked that style. When I see people write 'some %s %s text ' %('x', 'y') my first thought is "% is not not a function". I'm reminded also of (and agree with) the Crockford convention in JavaScript recommending a space in function () {...} but not in function foo(object) {...} But conventions are are a choice made; most times it's easier to adopt them and move on.
  - alok-g 12y ago
    >> But conventions are are a choice made; most times it's easier to adopt them and move on. Which brings me to a related question. Just like color-schemes are preferences in editors and IDEs, why isn't such spacing too. Then it would not be a "choice made" a priori for all individuals.
    - alexhill 12y ago
      Colour schemes aren't part of the text itself. However, if your IDE could automatically display your code formatted as you like while keeping the underlying text file aligned with some convention, and make that not a nightmare while editing...well, I think you're onto something there.
    - zo1 12y ago
      Because colors can be altered in any end-reader's IDE/text editor. Indentation and spacing, however, can not. At least, not safely yet. I suppose you could run all the code you receive and have to read through a code-formatter such as YAPF. That way, no matter what "way" someone writes their code, when you read it, it'll be in your comfortable/preferred format.
deleted 12y ago
[deleted]
pdknsk 12y ago
This passed HN duplicate detector, despite same URL and submission title. https://news.ycombinator.com/item?id=9260786 https://news.ycombinator.com/item?id=9260786
kolanos 12y ago
I definitely would like to see configuration options for the vertical alignment of arguments. If the arguments don't fit within the line length then having the option to put them on separate lines (still pep8) would be nice.

142 points by sudmishra 12y ago

>YAPF is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

yeukhon 12y ago
Sounds like one of those "20%" kind of project and Google GitHubers just happened to have the permission to release this project under google/ wow ...
- nostrademons 12y ago
  True, but that category also includes protobufs, LevelDB, Snappy, Gumbo, Guice, re2, gtest, and Angular (initially; it soon became an official thing when it got popular). While the "official" open-source projects are things like Chrome, Android, GWT, Go, Dart, gRPC, and Bazel. Paul Graham once said that you should use the tools that programmers build to solve their own problems, not the tools that big corporations build to solve their ideas of what other peoples' problems are. That doesn't mean every little library is a good one (Sturgeon's Law applies - 90% of everything is crap), but it does mean that being "official" is usually a negative signal on product quality. This applies to other companies' code as well; I've never used a buggy piece of shit quite so bad as Sun's JSF (which was supposed to be the "official" way to build webapps with Java, circa 2004-2006), while BSD and Linux continue to be great pieces of software 25 years later.

fweespeech 12y ago

> YAPF is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google. Just in case anyone thought "by Google" meant "a Google product" like me.

rbanffy 12y ago

--style STYLE specify formatting style: either a style name (for example "pep8" or "google"), or the name of a file with style settings. pep8 is the default. > pep8 is the default. I can breathe again.

rudolf0 12y ago
"Don't be evil" still sometimes holds true. :)
rcthompson 12y ago
I don't think there was any other option. If the default wasn't pep8, the tool would have no chance of seeing wide adoption in the Python community, unless someone forked it and changed the default to pep8.

baq 12y ago

i use autopep8 and haven't been thrilled. it gets the job done, but i'll switch in a heartbeat if this is better. we'll see.

eliben 12y ago
yapf's approach is, philosiphically, different from autopep8. yapf doesn't just fix pep8 violations. It takes a look at your whole code, and reformats it to a canonical form. If you've ever used "go fmt" or "clang-format" (for C/C++), this is the same idea
morbo_yapf 12y ago
YAPF is still brand spanking new, so please use it and report any bugs you find. We mention in the documentation that data literals are a sticking point to most people and most automatic formatters. Even clang-format tends to balk on them. We try our best at them, but it might be best to just disable formatting them if they are already formatted to something you like. :-)

bkcooper 12y ago

I'm sure there are some cases where this is useful, but I'm not really sold. The pitch at the beginning is "satisfying PEP 8 doesn't mean it looks good!" But then it seems most of the changes cleaned up in the code (indentation level, indents on split lines, spacing between operators) are just PEP 8 violations. I think you could find a more convincing demonstration. The warning about choking on large data literals (which are probably one of the places where prettifying would be most useful, at least to me) also seems ominous. edit: for my personal use, I tend to use flycheck with flake8 in emacs. This keeps me honest. Is the primary use case for something like this cleaning your own code or other people's?

ngoldbaum 12y ago
I do this too but I really don't like the noticeable lag that gets introduced as flycheck shells out to python and parses my file every time I change something.
- klibertp 12y ago
  Are you sure the lag is because of Python? I'd check flymake options first, it may wait for some time after you stop typing and only then start linting. This delay is a trade-off: too short and you're going to have currently edited line marked with red most of the time, too long and you need to wait to see if everything is alright.
  - ngoldbaum 12y ago
    Lazy HN for the win! Thanks for the suggestion.
bpicolo 12y ago
The point is there are things that don't violate pep8 that are still atrocious for example: some_function(long_variable_name_and_stuff, long_variable_name_and_stuff, long_variable_name_and_stuff) The lower indentation level doesnt actually matter at all says flake8 edit: To below, that's possible. I can't remember the exact case right now but there are things along these lines that are valid and shouldn't be. Multiple possible indentation formats is not necessarily something you want to allow as an org, which is where something like YAPF comes into play

mas1n 12y ago

Interesting to see the formatter is written using 2 space indentation when both pep8 and google's style guides say 4.

Pirate-of-SV 12y ago
I think the original Python Google Style guide did use 2 spaces. https://code.google.com/p/soc/wiki/PythonStyleGuide#Indentation https://code.google.com/p/soc/wiki/PythonStyleGuide#Indentat... And it seems like that's used in YAPF as well: https://github.com/google/yapf/blob/master/yapf/yapflib/style.py#L122 https://github.com/google/yapf/blob/master/yapf/yapflib/styl...
- geoelectric 12y ago
  This was submitted not long ago, and that was pointed out there too. https://news.ycombinator.com/item?id=9260786 https://news.ycombinator.com/item?id=9260786 I suggested to the author that they restore "google" style to the public one and rename the existing one to "chromium" style, since that's the most prominent public project still on Google's internal style.

Pirate-of-SV 12y ago

I've been using YAPF for a while and I really like the idea. Be aware that there are still some nasty bugs that haven't been fixed yet so be careful if you use it for something important. Do anyone of you guys know what the name YAPF comes from? (The only thing I can think of is "Yet Another Python Formatter", but I'm just guessing)

eliben 12y ago
Indeed, it's just "Yet Another Python Formatter"

sergiotapia 12y ago

Also see go fmt

pekk 12y ago
Thought experiment: wouldn't it be just awesome if Google decided to promote an alternative to the format used by go fmt, so that newbies could learn that instead and then have a constant conflict with the core Go community? edit: this is directly relevant to the thread, which is about Python formatting and Google promoting its weird style

zk00006 12y ago

Is it possible to YAPF it in Sublime?

jkane 11y ago
For sublime 2 yes: https://github.com/jason-kane/PyYapf https://github.com/jason-kane/PyYapf It isn't in Sublime package control yet.
morbo_yapf 12y ago
We need to have plugins for all editors. I'm not familiar with Sublime. But I would welcome people submitting plugins for it. :-)

Fuzzwah 12y ago

I've got a python project where I've been pretty naughty with long lines containing sql queries. I found that running this tool over it resulted in some strange new lines. For example, this was my old hideously long line: query_qual = "INSERT OR REPLACE INTO Qualifying (poleid, seasonid, race_week_num, carclassid, pole) values (%s%s, %s, %s, %s, %s)" % (race[u'seasonid'], race[u'race_week_num'], race[u'seasonid'], race[u'race_week_num'], race[u'carclassid'], pole) Changed to this: query_qual = "INSERT OR REPLACE INTO Qualifying (poleid, seasonid, race_week_num, carclassid, pole) values (%s%s, %s, %s, %s, %s)" % ( race[u'seasonid'], race[u'race_week_num'], race[ u'seasonid' ], race[u'race_week_num'], race[u'carclassid'], pole)

Pirate-of-SV 12y ago
You could try set the line length penalty so something really high. That would probably sort things out.
spott 12y ago
newlines need to be escaped somehow (two newlines seems to do it), otherwise they are combined into the same line: Every word in this sentence was entered on a different line.
eliben 12y ago
Feel free to submit a Github issue for yapf
hyperpallium 12y ago
To my eye, HN has rendered them identically. You can preserve literal whitespace on HN by indenting by two spaces (and a newline to separate the code formatting from ordinary text). it appears like this

teddyh 12y ago

There seems to be a problem parsing backslash escapes: echo 'foo="\\"' | PYTHONPATH=/tmp/yapf/yapf python /tmp/yapf Raises a lib2to3.pgen2.parse.ParseError. Also crashes on \n, but not, strangely, on \t. I’m guessing that the backslashes get interpreted twice, somehow.

eliben 12y ago
Are you sure this is not a shell artifact? Does the same problem happen when there's a \\ in a file? In any case, feel free to open a Github issue for yapf, we'll take a look
- teddyh 12y ago
  Yes, it was when parsing a file I encountered it. The one-liner shell command is just a minimal repro of the bug. Apparently I need to “Sign in” to report a Github issue, so I won’t.
  - eliben 12y ago
    Sorry, I can't reproduce this. I'm trying on this file: https://gist.github.com/eliben/9727758f4847d2e7d86e https://gist.github.com/eliben/9727758f4847d2e7d86e And yapf runs just fine on it. Does this file work for you?
    - teddyh 12y ago
      Nope. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/yapf/yapf/__main__.py", line 18, in <module> sys.exit(yapf.main(sys.argv)) File "/tmp/yapf/yapf/__init__.py", line 104, in main verify=args.verify)) File "/tmp/yapf/yapf/yapflib/yapf_api.py", line 99, in FormatCode tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n') File "/tmp/yapf/yapf/yapflib/pytree_utils.py", line 100, in ParseCodeToTree tree = parser_driver.parse_string(code, debug=False) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string return self.parse_tokens(tokens, debug) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 71, in parse_tokens if p.addtoken(type, value, (prefix, start)): File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 116, in addtoken ilabel = self.classify(type, value, context) File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 172, in classify raise ParseError("bad token", type, value, context) lib2to3.pgen2.parse.ParseError: bad token: type=55, value=u' ', context=('', (1, 5))

alok-g 12y ago

Off-topic, but why is no space before opening parenthesis the norm in programming ("class foo(object):") when normal English usage is to have one? I know myself as the only one who puts this space.

jdnier 12y ago
I think of it as indicating the function name and parameter list are a unit (it's foo(object), not foo (object)). I've always liked that style. When I see people write 'some %s %s text ' %('x', 'y') my first thought is "% is not not a function". I'm reminded also of (and agree with) the Crockford convention in JavaScript recommending a space in function () {...} but not in function foo(object) {...} But conventions are are a choice made; most times it's easier to adopt them and move on.
- alok-g 12y ago
  >> But conventions are are a choice made; most times it's easier to adopt them and move on. Which brings me to a related question. Just like color-schemes are preferences in editors and IDEs, why isn't such spacing too. Then it would not be a "choice made" a priori for all individuals.
  - alexhill 12y ago
    Colour schemes aren't part of the text itself. However, if your IDE could automatically display your code formatted as you like while keeping the underlying text file aligned with some convention, and make that not a nightmare while editing...well, I think you're onto something there.
  - zo1 12y ago
    Because colors can be altered in any end-reader's IDE/text editor. Indentation and spacing, however, can not. At least, not safely yet. I suppose you could run all the code you receive and have to read through a code-formatter such as YAPF. That way, no matter what "way" someone writes their code, when you read it, it'll be in your comfortable/preferred format.

deleted 12y ago

[deleted]

pdknsk 12y ago

This passed HN duplicate detector, despite same URL and submission title. https://news.ycombinator.com/item?id=9260786 https://news.ycombinator.com/item?id=9260786

kolanos 12y ago

I definitely would like to see configuration options for the vertical alignment of arguments. If the arguments don't fit within the line length then having the option to put them on separate lines (still pep8) would be nice.