4 ms·

Simpler example of the issue: >>> 1or 2 1

mixmastamyk 6y ago
Looks like the parser knows numbers end when the digits end so splits them into two tokens. (Ignoring the optional [lL] long suffix before Python 3) One of those things that looks helpful at first glance, but is problematic in the long run. Throwing a syntax error immediately would lead to more robust/maintainable code.
- pdw 6y ago
  Though you don't want to throw a syntax error on "1+2"
  - mixmastamyk 6y ago
    Yes, the difference between a letter character and punctuation/operator character. There's an argument to be made that disallowing it could help in the long run as well, but Python has never been so strict. So I'd stop short of recommending it here. Perhaps in a bondage and discipline language. :-D
  - _kst_ 6y ago
    1+2 is a valid expression consisting of 3 tokens, "1", "+", and "2". In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of them can be adjacent to an operator. The odd thing here is "0xfor" being tokenized as "0xf" and "or". In C, for example, "0xfor" is an invalid token.
    - toast0 6y ago
      > In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of them can be adjacent to an operator. The odd thing here is "0xfor" being tokenized as "0xf" and "or" 0xf is a numeric literal, and or is an operator, so as you said, they can be adjacent. or isn't an operator in C. of course, if it were 0xfand ... you have a fun problem, because it could be 0xfa nd, but nd isn't an operator (that I know of), or 0xf and. Ideally, syntax shouldn't be ambiguous.
      - _kst_ 6y ago
        I should have stated that more clearly. In Python, "or" is both an operator and a keyword. In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of then can be adjacent to a token that consists of punctuation characters. (In some languages, most or all operators spelled using punctuation characters, which is what I had in mind when I wrote my previous comment.)
      - fanf2 6y ago
        or is an operator in C++ and if you #include <iso646.h> in C
        _kst_ 6y ago
        > or is an operator in C++ Yes. > and if you #include <iso646.h> in C Sort of. As a preprocessor token, it's an identifier that happens to be the name of a macro. As a token (after preprocessing), it becomes the "||" operator.
    - deleted 6y ago
      [deleted]
    - deleted 6y ago
      [deleted]
  - xapata 6y ago
    My preference would be that the syntax requires spaces in that case. Oh well, I've always got the option of writing my own programming language.
- GeorgeTirebiter 6y ago
  You have to recognize the specific case to throw the syntax error. Sounds like they're going to fix that. I guess people don't remember PL/I very much: PL/I keywords are not reserved words, so it is possible to use them in a program in other than their keyword context. DO DO=1 TO TO BY BY;END=END+1;END; is a valid program.
  - Someone 6y ago
    For those wondering, the ideas behind that choice is that you can’t expect anybody to know all keywords. PL/I had many built-in functions that all were keywords, so it had many keywords (https://www.cs.vu.nl/grammarware/browsable/os-pli-v2r3/#Keywords https://www.cs.vu.nl/grammarware/browsable/os-pli-v2r3/#Keyw... lists hundreds) and even if you do, future versions of the language may include more keywords.
Nokinside 6y ago
Clearly a lexer error.
- creata 6y ago
  Why is it "clearly" an error? This syntax has been around forever and is widely used by code golfers, so if it were unintentional, I'm sure the Python devs would've fixed it by now.
  - hvdijk 6y ago
    It no longer matters if it was originally unintentional. If it's in active use now, so even if was originally by accident it cannot be fixed without breaking backwards compatibility.
    - tekromancr 6y ago
      Eh, python breaks backwards compatibility between versions all the time. And I don't know if "it's used by codegolfers" is a legitimate reason to keep a language feature. That said, I have a really hard time imagining this issue coming up in regular usage of the language so I don't really have much of an opinion should they choose to keep or drop it.
dagss 6y ago
There's the additional thing with x not being defined, not not being evaluated, which I found surprising at least. So perhaps: >>> 1or x # without x existing 1

Looks like the parser knows numbers end when the digits end so splits them into two tokens. (Ignoring the optional [lL] long suffix before Python 3) One of those things that looks helpful at first glance, but is problematic in the long run. Throwing a syntax error immediately would lead to more robust/maintainable code.

pdw 6y ago
Though you don't want to throw a syntax error on "1+2"
- mixmastamyk 6y ago
  Yes, the difference between a letter character and punctuation/operator character. There's an argument to be made that disallowing it could help in the long run as well, but Python has never been so strict. So I'd stop short of recommending it here. Perhaps in a bondage and discipline language. :-D
- _kst_ 6y ago
  1+2 is a valid expression consisting of 3 tokens, "1", "+", and "2". In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of them can be adjacent to an operator. The odd thing here is "0xfor" being tokenized as "0xf" and "or". In C, for example, "0xfor" is an invalid token.
  - toast0 6y ago
    > In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of them can be adjacent to an operator. The odd thing here is "0xfor" being tokenized as "0xf" and "or" 0xf is a numeric literal, and or is an operator, so as you said, they can be adjacent. or isn't an operator in C. of course, if it were 0xfand ... you have a fun problem, because it could be 0xfa nd, but nd isn't an operator (that I know of), or 0xf and. Ideally, syntax shouldn't be ambiguous.
    - _kst_ 6y ago
      I should have stated that more clearly. In Python, "or" is both an operator and a keyword. In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of then can be adjacent to a token that consists of punctuation characters. (In some languages, most or all operators spelled using punctuation characters, which is what I had in mind when I wrote my previous comment.)
    - fanf2 6y ago
      or is an operator in C++ and if you #include <iso646.h> in C
      - _kst_ 6y ago
        > or is an operator in C++ Yes. > and if you #include <iso646.h> in C Sort of. As a preprocessor token, it's an identifier that happens to be the name of a macro. As a token (after preprocessing), it becomes the "||" operator.
  - deleted 6y ago
    [deleted]
  - deleted 6y ago
    [deleted]
- xapata 6y ago
  My preference would be that the syntax requires spaces in that case. Oh well, I've always got the option of writing my own programming language.
GeorgeTirebiter 6y ago
You have to recognize the specific case to throw the syntax error. Sounds like they're going to fix that. I guess people don't remember PL/I very much: PL/I keywords are not reserved words, so it is possible to use them in a program in other than their keyword context. DO DO=1 TO TO BY BY;END=END+1;END; is a valid program.
- Someone 6y ago
  For those wondering, the ideas behind that choice is that you can’t expect anybody to know all keywords. PL/I had many built-in functions that all were keywords, so it had many keywords (https://www.cs.vu.nl/grammarware/browsable/os-pli-v2r3/#Keywords https://www.cs.vu.nl/grammarware/browsable/os-pli-v2r3/#Keyw... lists hundreds) and even if you do, future versions of the language may include more keywords.

Nokinside 6y ago

Clearly a lexer error.

creata 6y ago
Why is it "clearly" an error? This syntax has been around forever and is widely used by code golfers, so if it were unintentional, I'm sure the Python devs would've fixed it by now.
- hvdijk 6y ago
  It no longer matters if it was originally unintentional. If it's in active use now, so even if was originally by accident it cannot be fixed without breaking backwards compatibility.
  - tekromancr 6y ago
    Eh, python breaks backwards compatibility between versions all the time. And I don't know if "it's used by codegolfers" is a legitimate reason to keep a language feature. That said, I have a really hard time imagining this issue coming up in regular usage of the language so I don't really have much of an opinion should they choose to keep or drop it.

dagss 6y ago

There's the additional thing with x not being defined, not not being evaluated, which I found surprising at least. So perhaps: >>> 1or x # without x existing 1