5 ms·

Are you sure this is not a shell artifact? Does the same problem happen when there's a \\ in a file? In any case, feel free to open a Github issue for yapf, we'

by eliben 12y ago

Are you sure this is not a shell artifact? Does the same problem happen when there's a \\ in a file? In any case, feel free to open a Github issue for yapf, we'll take a look

teddyh 12y ago
Yes, it was when parsing a file I encountered it. The one-liner shell command is just a minimal repro of the bug. Apparently I need to “Sign in” to report a Github issue, so I won’t.
- eliben 12y ago
  Sorry, I can't reproduce this. I'm trying on this file: https://gist.github.com/eliben/9727758f4847d2e7d86e https://gist.github.com/eliben/9727758f4847d2e7d86e And yapf runs just fine on it. Does this file work for you?
  - teddyh 12y ago
    Nope. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/yapf/yapf/__main__.py", line 18, in <module> sys.exit(yapf.main(sys.argv)) File "/tmp/yapf/yapf/__init__.py", line 104, in main verify=args.verify)) File "/tmp/yapf/yapf/yapflib/yapf_api.py", line 99, in FormatCode tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n') File "/tmp/yapf/yapf/yapflib/pytree_utils.py", line 100, in ParseCodeToTree tree = parser_driver.parse_string(code, debug=False) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string return self.parse_tokens(tokens, debug) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 71, in parse_tokens if p.addtoken(type, value, (prefix, start)): File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 116, in addtoken ilabel = self.classify(type, value, context) File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 172, in classify raise ParseError("bad token", type, value, context) lib2to3.pgen2.parse.ParseError: bad token: type=55, value=u' ', context=('', (1, 5))
    - eliben 12y ago
      Interesting. Can you cite the exact version of Python 2.7 you're using? Mine is 2.7.6 I'm asking because bugs get fixed in lib2to3 occasionally, and we see differences between minor Python versions in this regard.
      - teddyh 12y ago
        Python 2.7.3, the normal one from the current Debian stable.
        eliben 12y ago
        The time diff between 2.7.3 and 2.7.6 is a year and a half, and lib2to3 got a bunch of fixed merge in during that period. I peeked at the CPython logs, and there's a number of changes so this bug may very well have been plugged. I guess we should be recommending the latest Python 2.7 possible, since it's very difficult for us to work around lib2to3 bugs in yapf - we're basically limited to whatever it can parse. Some things can be monkey-patched, but core parsing bugs are challenging.

by eliben 12y ago

Are you sure this is not a shell artifact? Does the same problem happen when there's a \\ in a file? In any case, feel free to open a Github issue for yapf, we'll take a look

teddyh 12y ago

Yes, it was when parsing a file I encountered it. The one-liner shell command is just a minimal repro of the bug. Apparently I need to “Sign in” to report a Github issue, so I won’t.

eliben 12y ago
Sorry, I can't reproduce this. I'm trying on this file: https://gist.github.com/eliben/9727758f4847d2e7d86e https://gist.github.com/eliben/9727758f4847d2e7d86e And yapf runs just fine on it. Does this file work for you?
- teddyh 12y ago
  Nope. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/tmp/yapf/yapf/__main__.py", line 18, in <module> sys.exit(yapf.main(sys.argv)) File "/tmp/yapf/yapf/__init__.py", line 104, in main verify=args.verify)) File "/tmp/yapf/yapf/yapflib/yapf_api.py", line 99, in FormatCode tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n') File "/tmp/yapf/yapf/yapflib/pytree_utils.py", line 100, in ParseCodeToTree tree = parser_driver.parse_string(code, debug=False) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string return self.parse_tokens(tokens, debug) File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 71, in parse_tokens if p.addtoken(type, value, (prefix, start)): File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 116, in addtoken ilabel = self.classify(type, value, context) File "/usr/lib/python2.7/lib2to3/pgen2/parse.py", line 172, in classify raise ParseError("bad token", type, value, context) lib2to3.pgen2.parse.ParseError: bad token: type=55, value=u' ', context=('', (1, 5))
  - eliben 12y ago
    Interesting. Can you cite the exact version of Python 2.7 you're using? Mine is 2.7.6 I'm asking because bugs get fixed in lib2to3 occasionally, and we see differences between minor Python versions in this regard.
    - teddyh 12y ago
      Python 2.7.3, the normal one from the current Debian stable.
      - eliben 12y ago
        The time diff between 2.7.3 and 2.7.6 is a year and a half, and lib2to3 got a bunch of fixed merge in during that period. I peeked at the CPython logs, and there's a number of changes so this bug may very well have been plugged. I guess we should be recommending the latest Python 2.7 possible, since it's very difficult for us to work around lib2to3 bugs in yapf - we're basically limited to whatever it can parse. Some things can be monkey-patched, but core parsing bugs are challenging.