11 ms·
In Python, `[0xfor x in (1, 2, 3)]` returns `[15]`
- adpirz 6y agoInterpreted as: [0xf or x in (1,2,3)] 0xf = Hex for 15 Expression is evaluated as a boolean
- kristaps 6y agoWhy is it parsed like that and not as invalid integer literal "0xfor"? edit: 'cause it's a bug! https://bugs.python.org/issue43833 https://bugs.python.org/issue43833
- pdw 6y agoFor pretty much the same reason that "1+2" is not parsed as an invalid integer.
- _kst_ 6y ago"1+2" is a valid expression consisting of the three tokens "1", "+", and "2". It's permitted to omit whitespace between those tokens. The question is whether "0xfor" is a valid way to represent the two tokens "0xf", "or". My reading of the rules is that it isn't, and that whitespace is required.
- _kst_ 6y agoI read the rules incorrectly. It appears that "0xfor" is a valid way to represent the two tokens "0xf" and "or". The rule, which I initially misinterpreted, is: > Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens). Since "0xfor" cannot be interpreted as a different (single) token, whitespace is not needed. (I'm not a fan of this particular consequence of that rule, and I suspect it was not intended, but it says what it says.)
- mjs7231 6y agoShouldn't this be considered a bug in Python? Why does it even try to evaluate 0xfor without the space? Trying a few other things.. * 0xfor1 evaluates. * 1or 2 evaluates. * 1or2 doesn't. * ''or'foo' evaluates. This is gross.
- kristaps 6y agoNot by design, fortunately: https://bugs.python.org/issue43833 https://bugs.python.org/issue43833
- xxpor 6y agoThat hasn't been confirmed. Are we sure that it's not an inherent ambiguity in the grammar?
- joshuamorton 6y agoThat's not totally clear. A bug being filed doesn't mean it's accepted. And this has been (ab)used for quite some time in various python codegolf. See https://codegolf.stackexchange.com/a/56 https://codegolf.stackexchange.com/a/56 from 2011.
- Sohcahtoa82 6y agoIt is 100% by design. It's even documented. https://docs.python.org/3/reference/lexical_analysis.html#whitespace-between-tokens https://docs.python.org/3/reference/lexical_analysis.html#wh... This is not a bug.
- _kst_ 6y agoThe cited documentation says: > Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens). The two tokens in this case are "0xf" and "or". Their concatenation cannot be interpreted as a different token, because "0xfor" is not a valid token. Therefore, if I'm reading the rule correctly, whitespace is needed in this case. "0xffor" is another interesting case. It's also not a valid token, but it could be interpreted as two tokens in two different ways: "0xf" "for" or "0xff" "or". (Python does the latter. I presume it uses something like C's "maximal munch" rule.)
- deleted 6y ago[deleted]
- TrackerFF 6y agoAnyone know why it's legal to use logic operators like that in Python? That is, 0or, 0xor, 0and,etc. in my view, this should clearly be a syntax error. edit: as noted by the parent comment, 0xor != 0 xor, but 0x or. But this seems to hold true for all data types.
- deleted 6y ago[deleted]
- mywittyname 6y agoOther languages don't enforce whitespace around logical operators. Like `0xf||1+2` is perfectly valid javascript. I can see how this could slip by, it's possible that original python used a non-alphabetical token as the logical operator, then it was swapped out at some later time for or and and.
- wyldfire 6y agoHere's the AST: >>> ast.dump(ast.parse('[0xfor x in (1,2,3)]')) "Module(body=[Expr(value=List(elts=[BoolOp(op=Or(), values=[Num(n=15), Compare(left=Name(id='x', ctx=Load()), ops=[In()], comparators=[Tuple(elts=[Num(n=1), Num(n=2), Num(n=3)], ctx=Load())])])], ctx=Load()))])"
- tgv 6y agoA bit of a miss for a language that puts whitespace central in the syntax.
- goto11 6y ago"2+2" and "2 + 2" is the same in Python, so whitespace is not significant between otherwise unambigous tokens in Python. Whitespace in Python is significant to indicate statements and block.
- commandlinefan 6y agoI initially assumed the missing space was a typo in the headline, so I tried [0x for x in (1,2,3)] and got "invalid hexadecimal literal" as I would have expected. Took me a few minutes to realize the typo was intentional. I still can't figure out what the author was trying to do when he stumbled onto that, though...
- tsimionescu 6y agoMy guess is that they were trying to do soemthing like [x for x in some_list], but had some unfortunate typos.
- commandlinefan 6y agoYeah maybe he was converting a tuple to a list, but that still seems like a weird way to do it.
- Dylan16807 6y agoThink of it as a simplified version to demonstrate the issue.
- wyldfire 6y agoThis is the kind of thing you could end up with from an automated test case reducer. Start with a typo that causes odd behavior (which might be interpreted as a CPython 'bug') and use something like creduce to whittle it down to the minimal case.
- nickysielicki 6y agoJust to elaborate, it short circuits, so x need not be defined. [sielicki@dogfruit ~]$ python3.10 -c 'print(0xf or some_undefined_variable in some_undefined_set)' 15
- wyldfire 6y agoThis is the kinda thing that helps illustrate just how powerfully dynamic Python is. Almost nothing is statically bound. Super cool if you want to write code that inspects some dynamic state and has special behavior. But the cost is that there's very little static checking capability. This code "should" result in a NameError or something similar. But resolving that identifier to a name won't happen until we decide to evaluate the right-hand side of the `or`. Thankfully, most python static checkers make some mostly-sane assumptions and will flag this code as an error, despite the subtle possibility of legally creating this name using some other Python/CPython features. Type annotations are a great idea but in practice I don't see them used much. Should be great for dev teams working on bigger Python projects.
- layer8 6y agoThis illustrates why lexer tokens should preferably not be defined as exactly what the language allows. Instead the internal lexer definition should include invalid tokens (like “0xfor”) that are only sorted out in a later step (in this case, when actually converting it to an integer value).
- fanf2 6y agoYes, that is basically how C works, though the C standard describes it in a weird way. There is the notion of a “preprocessor number” which starts with a digit and allows any characters that can appear in a number or identifier. At a later step in parsing it is converted to a strict number, which would fail in a case like 0xfor.
- brundolf 6y agoThere's no ambiguity though, right? [0x for x in (1,2,3)] is nonsense (one typo off from something legitimate, but nonsense regardless). I can't think of another way someone might expect this to parse.
- Skunkleton 6y agoI would have expected the english boolean operators to require whitespace to parse correctly, personally. "0xf||x" makes sense. "0xforx" makes no sense.
- Sohcahtoa82 6y ago"0xforx" would be parsed as "0xf orx" which would be a syntax error.
- Skunkleton 6y agoYeah, I realized after my edit window had expired. Should have been "0xf|| x" and "0xfor x".
- moefh 6y agoIf you want confusion with a valid expression: [0x1for x in (1,2,3)] looks like it should parse as [0x1 for x in (1,2,3)] but instead parses as [0x1f or x in (1,2,3)]
- nickysielicki 6y agoThis is better than the original post.
- CameronNemo 6y agoSimpler example of the issue: >>> 1or 2 1
- mixmastamyk 6y agoLooks like the parser knows numbers end when the digits end so splits them into two tokens. (Ignoring the optional [lL] long suffix before Python 3) One of those things that looks helpful at first glance, but is problematic in the long run. Throwing a syntax error immediately would lead to more robust/maintainable code.
- pdw 6y agoThough you don't want to throw a syntax error on "1+2"
- mixmastamyk 6y agoYes, the difference between a letter character and punctuation/operator character. There's an argument to be made that disallowing it could help in the long run as well, but Python has never been so strict. So I'd stop short of recommending it here. Perhaps in a bondage and discipline language. :-D
- _kst_ 6y ago1+2 is a valid expression consisting of 3 tokens, "1", "+", and "2". In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of them can be adjacent to an operator. The odd thing here is "0xfor" being tokenized as "0xf" and "or". In C, for example, "0xfor" is an invalid token.
- toast0 6y ago> In most languages, numeric literals, identifiers, and keywords cannot be adjacent, but any of them can be adjacent to an operator. The odd thing here is "0xfor" being tokenized as "0xf" and "or" 0xf is a numeric literal, and or is an operator, so as you said, they can be adjacent. or isn't an operator in C. of course, if it were 0xfand ... you have a fun problem, because it could be 0xfa nd, but nd isn't an operator (that I know of), or 0xf and. Ideally, syntax shouldn't be ambiguous.
- obi1kenobi 6y agoCorresponding bug report: https://bugs.python.org/issue43833 https://bugs.python.org/issue43833
- noobermin 6y agoFor Christ's sake this isn't a bug.
- Sohcahtoa82 6y agoThis isn't a bug. In fact, it's documented behavior: https://docs.python.org/3/reference/lexical_analysis.html#whitespace-between-tokens https://docs.python.org/3/reference/lexical_analysis.html#wh...
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- xenonite 6y ago`0xbor(11)` is 11 `11-0xbor-11` is -11 and `11-0xbor-0xbor-0xbor-11` is -11, too
- css 6y ago> Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens). https://docs.python.org/3/reference/lexical_analysis.html#whitespace-between-tokens https://docs.python.org/3/reference/lexical_analysis.html#wh...
- tannhaeuser 6y agoI find that to be a really odd choice when OTOH tabs/indents are required for block structure.
- RHSeeger 6y agoPython: Semantic whitespace is the one true way Also Python: Meh, you can leave out the whitespace, I'll figure it out
- notacoward 6y agoSimilarly, Python is supposed to be a language that's easy to learn and easy to read code in. Pythonistas take pride in that, as IMO they should. With that in mind, preserving a counterintuitive behavior that exists solely because of how the lexer was implemented seems inconsistent at best. A language with a more relaxed attitude toward inscrutable brevity - e.g. Perl - can use that excuse. I don't think Python should.
- mywittyname 6y agoI first learned Python in the 2.6 days, and have been developing in it fairly regularly over the 15 or so years since. I've never encountered this issue before. The reason I've never encountered this issue, or even needed to know it was an issue is because I use good development tools. 1) Pycharm highlighted the or in orange, making it clear that it was being interpreted as a keyword, 2) I use a linter (as we all should) which explicitly highlights the lack of whitespace around the token as an issue (PEP 8: E225), and 3) I use a code formatter (which we all should) which, again, highlights this statement as requests that I fix it. Python has a lot of flaws. This one is down near the bottom of ones that are even really worth talking about.
- maccard 6y agoReminds me of the "goes to"[0] operator in c (and c++) int x = 10; while ( x --> 0) [0] https://stackoverflow.com/questions/1642028/what-is-the-operator-in-c-c https://stackoverflow.com/questions/1642028/what-is-the-oper...
- 9front 6y agoReverse order and you get: >>> [x in (1,2,3) or 0xf] [True]
- noobermin 6y agono that would return an error unless x is defined earlier
- phsilva 6y ago>>> 0xfor whatever 15
- jVinc 6y agoTrying to clarify it a bit, it gets interpreted as: "[0xf or (x in (1,2,3))]" and the "or" short circuits, never evaluating the second part (which would have given an uninitialized variable error). This therefore evaluates to [0xf] or just [15] once you convert the hex notation to an integer. If you neglect the outer list that's only there to fool you into thinking this has something to do with list comprehensions, same as the x, and if you don't use hex to make it seem like there is a "for" in the middle, it boils down to something like: "15or whatever()" which doesn't seem all that confusing, even if "whatever" uses an uninitialized variable like x, or is undefined, because we're in Python and it's only evaluated when it runs. Then we are left with the 'confusion' boiling down to a Wat about why it's legal to do "15or 3" without space before or, and especially why "x0for 3" works. This is documented as other comments mention and is due to the way the parsing works. I don't know if the parser could be changed to require a leading space before the or operator, but it's pretty clear to me that this is only confusing if you intentionally do you very best to try to add confusing structure around it in an attempt to fool the reader into thinking something very different and weird is going on.
- takeda 6y agoYeah, this also evaluates to [15] and is easier to understand what's going on: [15or x in (1, 2, 3)] As another person pointed out, Python's lexer sees 0xfor (or 15or) and splits it into "0xf", "or" ("15", "or" for the other case) then parser processes it as usual.
- d23 6y ago> it's pretty clear to me that this is only confusing if you intentionally do you very best to try to add confusing structure around it in an attempt to fool the reader into thinking something very different and weird is going on. Sure, but that's half the fun :-) I frequently see Ned's name pop up with interesting things like this.
- sco1 6y ago> it's pretty clear to me that this is only confusing if you intentionally do you very best to try to add confusing structure around it in an attempt to fool the reader into thinking something very different and weird is going on. This has come up from a Core Developer before [1] so it's not just code golfers having a laugh. Yes, the docs note the tokenization behavior [2], but Guido's response today in the above mail thread is also pretty unambiguous: > I would totally make that a SyntaxError, and backwards compatibility be damned. 1: https://mail.python.org/archives/list/python-dev@python.org/thread/D2WPCITHG2LBQAP7DBTC6CY26WQUBAKP/#D2WPCITHG2LBQAP7DBTC6CY26WQUBAKP https://mail.python.org/archives/list/python-dev@python.org/... 2: https://docs.python.org/3/reference/lexical_analysis.html#whitespace-between-tokens https://docs.python.org/3/reference/lexical_analysis.html#wh...
- fennecfoxen 6y agocould be much worse # launch debugger with the program provided inline, consisting of the statement `1`. # this program is ignored and we just mess with some variables. % perl -de1 Loading DB routines from perl5db.pl version 1.57 Editor support available. Enter h or 'h h' for help, or 'man perldebug' for more help. main::(-e:1): 1 DB<1> @a = (1..4) DB<2> x \@a 0 ARRAY(0x7faf890418f8) 0 1 1 2 2 3 3 4 DB<3> x scalar @a 0 4 DB<4> sub foo { return (1..4) } DB<5> @b = foo() DB<6> x scalar @b 0 4 DB<7> x scalar foo() 0 '' yes, that's the empty string, aka false, because the .. operator returns entirely different things in scalar context so you can write code like while (<STDIN>) { next if /^BEGIN$/ .. /^END$/; # ... }
- bjornorn 6y ago>>> [0xfor d or cambridge] [15]
- deleted 6y ago[deleted]