7 ms·

Your 2nd error isn't possible in Python, so I'm not sure what you did there. Regarding the first, sure, it is a bug that was annoying to catch. But, having an `

Your 2nd error isn't possible in Python, so I'm not sure what you did there. Regarding the first, sure, it is a bug that was annoying to catch. But, having an `Iterable` interface in Python is also really neat and useful if used responsibly. If you're programming regularly in Python, you are accustomed to the tradeoffs that come with a dynamic programming language and no static types, and you can still avoid issues like the one above. Right off the top of my head, using type hints with a decent IDE or an assert statement would likely have caught the issue. I'm not saying that Python doesn't have issues (all languages do), but I don't see the error noted above as any sort of deal breaker. On the other hand, if you're only ever going to use Python like a strongly typed language without taking any advantage of its dynamic characteristics, then I can see why it would seem as a total downgrade compared to languages like Java.

CydeWeys 7y ago
I didn't explain the second one well. Here's some exact code. group_keys = ... if not isinstance(group_keys, list): groups_keys = [ group_keys ] So rather than listifying the non-list variable, it was creating a new variable. The cause of this bug is that Python doesn't distinguish between declaring new variables and overwriting existing ones.
- sreque 7y ago
  I still don't think you quite understand what's going on here. Python wouldn't create a new variable in this case. I It would re-assign the value represented by the variable you already assigned once. I agree that it would have been better if Python had explicit variable declarations (this is one of the few things I think Perl got right.). On the other hand, Ruby made this same mistake. If you wrote this code in Javascript you wouldn't get an error, but you would in fact have two different variables. For instance, this code runs for me using node 8: var fun = (bool) => { var x = 1; if (bool) { var x = 2; x += 1; } else { x += 1; } console.log("x=" + x); } fun(1); fun(0);
  - CydeWeys 7y ago
    I don't understand why you're accusing me of being irrational. These seem like very rational problems to have with Python. They literally caused me bugs that cost me time to deal with that I wouldn't have faced in other languages. You're also assuming that I don't have the same problems with Ruby or JavaScript. I do. The exact same critique could be made of them as well, but they're not the subject of this thread; Python is.
    - sreque 7y ago
      My bad. I didn't realize you were against all dynamic languages in particular. FWIW I prefer Java and static types as well, but as far as scripting languages go, I think Python is pretty great.
    - rohan1024 7y ago
      You can't argue with someone who has chosen to overlook your viewpoint. I've ran into the same issues while writing python code. People who are newly picking up python are especially prone to these kind of bugs. Also, with python I have to spend lot of time to figure out what went wrong in my code as compared to other languages. People who have been using python for long have wired there brain to avoid such pitfalls and now they happily defend it.
      - sreque 7y ago
        I don't think what you're saying is true. I already said I think it would have been better if Python and Ruby had explicit variable declarations. But, if this is your biggest issue with a language and it's ecosystem, then IMO that language is doing pretty well. I would rather, for instance, have to deal with implicit variable declarations in Python that the gigantic mess of Java frameworks that have been invented to "reduce boilerplate", such as Spring/Guice, AspectJ, Hibernate, etc.
  - winstonewert 7y ago
    > I still don't think you quite understand what's going on here. Python wouldn't create a new variable in this case. I It would re-assign the value represented by the variable you already assigned once. Uh. no. he typoed the reassignment, so it wouldn't re-assign the value. > So, of three of the most popular dynamic languages, Python, Ruby, and Javascript, none of them would have helped you catch this kind of error at script-parsing time. So again, it seems like you have an irrational dislike for Python, all things considered. Sure, but he's made it clear he likes Java. Fundamentally he's against dynamic typing, so of course he doesn't like any of the dynamic languages.
- jldugger 7y ago
  Well, this should have been caught as an unused assignment in static analysis. A whole ton of languages allow this situation, so I'm not gonna ding Python too hard for that one. However, here's a related but different python gotcha: if foo(a): v = list(bar(a)) for i in v: print i In this example, v is only defined inside the if. Due to python's limited scopes, v is also valid outside the if, but only has an assignment when foo(a) is True. When foo(a) is false, the for loop throws an NameError. And yes, a coworker wrote code that accidentally implemented this antipattern, albeit much more spread out in the code base. This is clearly a bug in the code, yet no static analysis tools I've tried have successfully discovered it. There's a bug in pylint that's been marked WONTFIX because it requires a wildly different implementation. At a language level, it feels weird that if blocks aren't a scope level for new variables. If you want to reference v outside the if loop, declare / assign it outside the loop first.
  - takeda 7y ago
    mypy was able to detect these kinds of issues for me
  - joshuamorton 7y ago
    Indeed, as another user mentions, mypy will detect this issue, as will pytype, even without any annotations.
    - jldugger 7y ago
      Interesting, I had not looked at these because I'm not interested in volunteering to add type notations to our code base.
  - marmaduke 7y ago
    I'm retty sure PyCharm catches that 2nd one with a warning
- takeda 7y ago
  I see this often and it is a bad pattern that people do. Typically type checked languages wouldn't even allow you to do. If you would use mypy for type checking it wouldn't like it because you're redefining the type of a variable. Best practices would suggest you use a different variable for the conversion if you must, but ideally you should just make the function accept list as an argument. If you're really worried about passing something else than a list, you should use type annotations to tell type checker what it is. If you want to add extra runtime check then do: assert isinstance(group_keys, list) You can complain that Python allowed you to something dangerous, but you have tools to help you avoid it and this flexibility is what makes tools like SQLAlchemy so powerful.
- joshuamorton 7y ago
  FYI, the google style guide (or maybe the internal only version) suggests to avoid initialize-then-assign in favor of single assignment form: unclear_type_thing = ... if isinstance(unclear_type_thing, list): group_keys = unclear_type_thing else: group_keys = [unclear_type_thing] statically avoids this problem. In general, prefer immutable variables where possible. Single-assignment form is nice for a lot of reasons, not the least of which is that it avoids this particular gotcha. And I should add that the "right" way to do this would be to factor this out to a function: group_keys = coerce_to_list(...) is much clearer than either block, and avoids the possibility of the issue.
  - CydeWeys 7y ago
    All of these things are true, but they require a non-trivial level of experience and discipline to avoid most potential gotchas. Your average Python project on the Web isn't written to this level of quality, and when people are learning programming using Python in school they certainly aren't there yet, and are gonna hit all kinds of problems related to this stuff. But is there a way to force immutable variables in Python? You can easily still end up in the same situation when you typo something (easy to do when plurals are involved), and then end up reassigning something when you meant to create a new variable.
    - _coveredInBees 7y ago
      I don't think that's fair to be honest. If you had simply used Pycharm with default settings you would have easily caught the first bug due to the linting. It's a fair complaint, but this specific bug is easy to catch using any modern Python IDE.
      - CydeWeys 7y ago
        I've never found the "Use this specific IDE" defense particularly valid, considering that many IDEs don't have these features and that in other languages the compiler itself protects you. Needless to say, I was not using Pycharm for this development, nor am I likely to install an entire IDE just for a small change I'm making on a random project. It's a non-trivial burden to configure and learn an entire IDE, vs just using what I already know (which is often just emacs).
mnky9800n 7y ago
Maybe they could write unit tests to make sure what's being passed is lists and strings. But that's probably crazy.
- CydeWeys 7y ago
  The values were coming from a YAML deserializer, for what it's worth.
- cc81 7y ago
  That would not catch the bug if the input is not under is control. You could as well say "Just check if the object is a string" in the method, which would work but the point was rather that it is difficult to notice if you did not think about it. Compared to other languages that would crash or not compile instead.
  - CydeWeys 7y ago
    Yeah, the input isn't really under control because it's coming from deserializing a YAML file. It worked for the exact type of input I was expecting, namely, when you configure a specific value as a list, but it wasn't working for anything else. And YAML has plenty of types it can split out, so my naive fix still only handled lists and strings properly!
    - winstonewert 7y ago
      Yeah, YAML deserialization is the worst case scenario for dynamic typing. In most situations, types are pretty consistent and assuming you run your code at least once, you'll find most errors. But with YAML deserialization all bets are off. YAML is even worse then JSON for this because seemingly minor changes in the YAML can change the shape of the data. I've had success validating such data against a schema, so I know it had consistent type structure before working with it.
- human20190310 7y ago
  You can write a unit test for anything you can think of, of course. But a strongly-typed language will catch such errors automatically (and for free) at compile time, even if you didn't anticipate the failure case.
btilly 7y ago
I disbelieve. And I disbelieve despite being a fan of dynamic languages. The tradeoff is that dynamic languages are faster to develop, more concise, but more expensive in maintenance exactly because of issues like this. The data that I base this opinion on is an unpublished internal report from nearly a decade ago at Google quantifying costs of projects of different size in their different languages. Which was Java, C++, and Python. Python had the lowest initial development, and the highest maintenance costs. That is why Google then moved to Go as a replacement for Python. It was good for the same things that they used Python for, but being statically typed, its maintenance costs were lower.
- _coveredInBees 7y ago
  I can believe that. But for a lot of people, the lower initial development time/cost aspect matters a lot. If I had Google resources, sure, I'd Go with other languages perhaps, but you can still write high-quality and capable software in Python. And while the batteries included aspect of Python is not everyone's cup of tea, I personally find it quite handy to have that so I don't have to waste a ton of time evaluating different libs to do fairly standard things. To be clear - I'm not trying to say that Python is better in any objective way. Ultimately, I think people should use the tools they have available and prefer, to build what they want.
  - btilly 7y ago
    But for a lot of people, the lower initial development time/cost aspect matters a lot. As I said, I'm a fan of dynamic languages. :-) One of the top ways that startups fail is failing to build what they need to build quickly enough. Maintenance costs only matter if you succeed in the first place. Using dynamic languages is therefore a good fit. But, even if you're not Google, if you're writing software and have the luxury of paying attention to the lifetime costs of the project up front, you should choose a statically typed language.
27182818284 7y ago
>Iterable` interface in Python is also really neat and useful if used responsibly. Honestly this was a major attraction to python for me a decade plus ago as a student when I started learning--even when I used it irresponsibly. There are so many small tasks where you just kinda have to iterate over 100-1000 items that you're not worried about big-O or anything like that—you just want to iterate and work on a collection quickly for some task in the office.

CydeWeys 7y ago

I didn't explain the second one well. Here's some exact code. group_keys = ... if not isinstance(group_keys, list): groups_keys = [ group_keys ] So rather than listifying the non-list variable, it was creating a new variable. The cause of this bug is that Python doesn't distinguish between declaring new variables and overwriting existing ones.

sreque 7y ago
I still don't think you quite understand what's going on here. Python wouldn't create a new variable in this case. I It would re-assign the value represented by the variable you already assigned once. I agree that it would have been better if Python had explicit variable declarations (this is one of the few things I think Perl got right.). On the other hand, Ruby made this same mistake. If you wrote this code in Javascript you wouldn't get an error, but you would in fact have two different variables. For instance, this code runs for me using node 8: var fun = (bool) => { var x = 1; if (bool) { var x = 2; x += 1; } else { x += 1; } console.log("x=" + x); } fun(1); fun(0);
- CydeWeys 7y ago
  I don't understand why you're accusing me of being irrational. These seem like very rational problems to have with Python. They literally caused me bugs that cost me time to deal with that I wouldn't have faced in other languages. You're also assuming that I don't have the same problems with Ruby or JavaScript. I do. The exact same critique could be made of them as well, but they're not the subject of this thread; Python is.
  - sreque 7y ago
    My bad. I didn't realize you were against all dynamic languages in particular. FWIW I prefer Java and static types as well, but as far as scripting languages go, I think Python is pretty great.
  - rohan1024 7y ago
    You can't argue with someone who has chosen to overlook your viewpoint. I've ran into the same issues while writing python code. People who are newly picking up python are especially prone to these kind of bugs. Also, with python I have to spend lot of time to figure out what went wrong in my code as compared to other languages. People who have been using python for long have wired there brain to avoid such pitfalls and now they happily defend it.
    - sreque 7y ago
      I don't think what you're saying is true. I already said I think it would have been better if Python and Ruby had explicit variable declarations. But, if this is your biggest issue with a language and it's ecosystem, then IMO that language is doing pretty well. I would rather, for instance, have to deal with implicit variable declarations in Python that the gigantic mess of Java frameworks that have been invented to "reduce boilerplate", such as Spring/Guice, AspectJ, Hibernate, etc.
- winstonewert 7y ago
  > I still don't think you quite understand what's going on here. Python wouldn't create a new variable in this case. I It would re-assign the value represented by the variable you already assigned once. Uh. no. he typoed the reassignment, so it wouldn't re-assign the value. > So, of three of the most popular dynamic languages, Python, Ruby, and Javascript, none of them would have helped you catch this kind of error at script-parsing time. So again, it seems like you have an irrational dislike for Python, all things considered. Sure, but he's made it clear he likes Java. Fundamentally he's against dynamic typing, so of course he doesn't like any of the dynamic languages.
jldugger 7y ago
Well, this should have been caught as an unused assignment in static analysis. A whole ton of languages allow this situation, so I'm not gonna ding Python too hard for that one. However, here's a related but different python gotcha: if foo(a): v = list(bar(a)) for i in v: print i In this example, v is only defined inside the if. Due to python's limited scopes, v is also valid outside the if, but only has an assignment when foo(a) is True. When foo(a) is false, the for loop throws an NameError. And yes, a coworker wrote code that accidentally implemented this antipattern, albeit much more spread out in the code base. This is clearly a bug in the code, yet no static analysis tools I've tried have successfully discovered it. There's a bug in pylint that's been marked WONTFIX because it requires a wildly different implementation. At a language level, it feels weird that if blocks aren't a scope level for new variables. If you want to reference v outside the if loop, declare / assign it outside the loop first.
- takeda 7y ago
  mypy was able to detect these kinds of issues for me
- joshuamorton 7y ago
  Indeed, as another user mentions, mypy will detect this issue, as will pytype, even without any annotations.
  - jldugger 7y ago
    Interesting, I had not looked at these because I'm not interested in volunteering to add type notations to our code base.
- marmaduke 7y ago
  I'm retty sure PyCharm catches that 2nd one with a warning
takeda 7y ago
I see this often and it is a bad pattern that people do. Typically type checked languages wouldn't even allow you to do. If you would use mypy for type checking it wouldn't like it because you're redefining the type of a variable. Best practices would suggest you use a different variable for the conversion if you must, but ideally you should just make the function accept list as an argument. If you're really worried about passing something else than a list, you should use type annotations to tell type checker what it is. If you want to add extra runtime check then do: assert isinstance(group_keys, list) You can complain that Python allowed you to something dangerous, but you have tools to help you avoid it and this flexibility is what makes tools like SQLAlchemy so powerful.
joshuamorton 7y ago
FYI, the google style guide (or maybe the internal only version) suggests to avoid initialize-then-assign in favor of single assignment form: unclear_type_thing = ... if isinstance(unclear_type_thing, list): group_keys = unclear_type_thing else: group_keys = [unclear_type_thing] statically avoids this problem. In general, prefer immutable variables where possible. Single-assignment form is nice for a lot of reasons, not the least of which is that it avoids this particular gotcha. And I should add that the "right" way to do this would be to factor this out to a function: group_keys = coerce_to_list(...) is much clearer than either block, and avoids the possibility of the issue.
- CydeWeys 7y ago
  All of these things are true, but they require a non-trivial level of experience and discipline to avoid most potential gotchas. Your average Python project on the Web isn't written to this level of quality, and when people are learning programming using Python in school they certainly aren't there yet, and are gonna hit all kinds of problems related to this stuff. But is there a way to force immutable variables in Python? You can easily still end up in the same situation when you typo something (easy to do when plurals are involved), and then end up reassigning something when you meant to create a new variable.
  - _coveredInBees 7y ago
    I don't think that's fair to be honest. If you had simply used Pycharm with default settings you would have easily caught the first bug due to the linting. It's a fair complaint, but this specific bug is easy to catch using any modern Python IDE.
    - CydeWeys 7y ago
      I've never found the "Use this specific IDE" defense particularly valid, considering that many IDEs don't have these features and that in other languages the compiler itself protects you. Needless to say, I was not using Pycharm for this development, nor am I likely to install an entire IDE just for a small change I'm making on a random project. It's a non-trivial burden to configure and learn an entire IDE, vs just using what I already know (which is often just emacs).

mnky9800n 7y ago

Maybe they could write unit tests to make sure what's being passed is lists and strings. But that's probably crazy.

CydeWeys 7y ago
The values were coming from a YAML deserializer, for what it's worth.
cc81 7y ago
That would not catch the bug if the input is not under is control. You could as well say "Just check if the object is a string" in the method, which would work but the point was rather that it is difficult to notice if you did not think about it. Compared to other languages that would crash or not compile instead.
- CydeWeys 7y ago
  Yeah, the input isn't really under control because it's coming from deserializing a YAML file. It worked for the exact type of input I was expecting, namely, when you configure a specific value as a list, but it wasn't working for anything else. And YAML has plenty of types it can split out, so my naive fix still only handled lists and strings properly!
  - winstonewert 7y ago
    Yeah, YAML deserialization is the worst case scenario for dynamic typing. In most situations, types are pretty consistent and assuming you run your code at least once, you'll find most errors. But with YAML deserialization all bets are off. YAML is even worse then JSON for this because seemingly minor changes in the YAML can change the shape of the data. I've had success validating such data against a schema, so I know it had consistent type structure before working with it.
human20190310 7y ago
You can write a unit test for anything you can think of, of course. But a strongly-typed language will catch such errors automatically (and for free) at compile time, even if you didn't anticipate the failure case.

btilly 7y ago

I disbelieve. And I disbelieve despite being a fan of dynamic languages. The tradeoff is that dynamic languages are faster to develop, more concise, but more expensive in maintenance exactly because of issues like this. The data that I base this opinion on is an unpublished internal report from nearly a decade ago at Google quantifying costs of projects of different size in their different languages. Which was Java, C++, and Python. Python had the lowest initial development, and the highest maintenance costs. That is why Google then moved to Go as a replacement for Python. It was good for the same things that they used Python for, but being statically typed, its maintenance costs were lower.

_coveredInBees 7y ago
I can believe that. But for a lot of people, the lower initial development time/cost aspect matters a lot. If I had Google resources, sure, I'd Go with other languages perhaps, but you can still write high-quality and capable software in Python. And while the batteries included aspect of Python is not everyone's cup of tea, I personally find it quite handy to have that so I don't have to waste a ton of time evaluating different libs to do fairly standard things. To be clear - I'm not trying to say that Python is better in any objective way. Ultimately, I think people should use the tools they have available and prefer, to build what they want.
- btilly 7y ago
  But for a lot of people, the lower initial development time/cost aspect matters a lot. As I said, I'm a fan of dynamic languages. :-) One of the top ways that startups fail is failing to build what they need to build quickly enough. Maintenance costs only matter if you succeed in the first place. Using dynamic languages is therefore a good fit. But, even if you're not Google, if you're writing software and have the luxury of paying attention to the lifetime costs of the project up front, you should choose a statically typed language.

27182818284 7y ago

>Iterable` interface in Python is also really neat and useful if used responsibly. Honestly this was a major attraction to python for me a decade plus ago as a student when I started learning--even when I used it irresponsibly. There are so many small tasks where you just kinda have to iterate over 100-1000 items that you're not worried about big-O or anything like that—you just want to iterate and work on a collection quickly for some task in the office.