20 ms·

A bite of Python

334 points by ashitlerferad 10y ago

alex-yo 10y ago
I wouldn't call it 'traps'. I would call it 'read and understand documentation before writing code' like: what is 'is' operator, or how floats behave in EVERY programming language, or why you should sanitize EVERY user input. So, basically, I can write such a list for every language I know.
- baq 10y ago
  which is great and you totally should. not everyone knows about those things.
  - userbinator 10y ago
    I think the point of the parent comment is that they should. I'm not even a regular user of Python (or dynamic languages in general), and yet the list of things in the article doesn't seem all that surprising.
- d33 10y ago
  I'd argue that a good programming language should rarely force you to go back to the docs. Anyway, for a list of weird language properties, check out this tag: https://stackoverflow.com/questions/tagged/hidden-features?sort=votes&pageSize=50 https://stackoverflow.com/questions/tagged/hidden-features?s...
- blowski 10y ago
  Relying on developers to read and remember every bit of documentation for every bit of code is more likely to end up with insecure code compared to introducing sane defaults with an explicit, expressive API.
  - SEJeff 10y ago
    And the TL;DNR being: developers can't be expected to do a good job. Some of them in fact will do a terrible job. This can be said for every industry involving people.
    - guyzmo 10y ago
      The sad truth is: s/Some/Most/
    - afarrell 10y ago
      And the way we handle that is by designing systems to compensate for the fallibility of humans so that the human-computer system is more robust as a whole.
    - hueving 10y ago
      Which is why any sane industry has lots of safety involved. We don't just shrug every time someone gets electrocuted to death and say "they forgot part c page 4 of the operations manual which indicates that the off switch doesn't work on tuesdays".
      - deleted 10y ago
        [deleted]
- hacker42 10y ago
  That's like saying people shouldn't read FAQs because they should rather read the documentation. These things aren't actually mutually exclusive.
viraptor 10y ago
If you're interested in reviewing Python code for potential security issues, here's a related project: https://github.com/openstack/bandit https://github.com/openstack/bandit (I'm one of the devs) It will actually pick up a number of security issues listed in the post. It's useful in real world too - led to a number of CVEs being reported.
- ubernostrum 10y ago
  I looked at Bandit earlier this year, but had to put it down when I discovered it didn't have a way to fill in default config -- the instant I specified anything in the config file, I had to supply a complete config, including literally every single check it's capable of doing, because Bandit would discard all its defaults on encountering that single line of custom config. Don't suppose you know if that's gotten better?
  - viraptor 10y ago
    It's not ideal - the config file is still a complete list, but there are a few things you can do. - `bandit-config-generator` will give you a file filled with the current/default configuration, so it's a simple way to start with the defaults and modify just what you need - if you just need to enable/disable tests rather than reconfigure, you can do that in command line options - if you want to get rid of specific warning, you can mark the line with "# nosec" in the source Merging various configs is possible, but rather complex to implement considering we aim for the config to be a complete description that won't ever need to change between versions. If none of the above workarounds solve your use case, feel free to report an issue. (https://bugs.launchpad.net/bandit https://bugs.launchpad.net/bandit) I can't guarantee how/whether we'll fix this, but we'd definitely like to know what the problem is and how you're trying to use Bandit.
    - wyldfire 10y ago
      FYI when using pypy bandit fails to discover tests for some reason (could be a pypy bug?). In this case, the default output generated by bandit-config-generator is mostly empty and bandit fails to parse it. It doesn't indicate what the nature of the parse failure is (what line, what rule(s) were violated), even with verbose mode. The readme doesn't cover the format of the config file. Is it YAML/TOML/JSON/other?
michaelfeathers 10y ago
The behavior of 'assert' is not an anomaly. It comes from 'design by contract.' Assert is primarily meant to be documentation of constraints in code and secondarily a way of catching errors during development. "Contract conditions should never be violated during execution of a bug-free program. Contracts are therefore typically only checked in debug mode during software development. Later at release, the contract checks are disabled to maximize performance." - https://en.wikipedia.org/wiki/Design_by_contract https://en.wikipedia.org/wiki/Design_by_contract
- blowski 10y ago
  It's not an anomaly, but it can be a surprise for people who don't understand what it does. For example, they use asserts for validation and then the validation doesn't work in production. It's absolutely right the way it works, but it's still a gotcha for the audience this blog post is aimed at.
  - coldtea 10y ago
    >It's not an anomaly, but it can be a surprise for people who don't understand what it does. That's almost a tautology though.
    - blowski 10y ago
      The point is that they think they understand it, because most of Python behaves the same in development and production. You see this function called 'assert', it gives the right error at the right time, and all is good. Then you push it to production and it stops throwing errors. Eventually, you read the manual and it tells you that this specific function is ignored in production. This is a surprise because, say, print doesn't behave like that.
      - coldtea 10y ago
        That only happens if you have different production and development environment settings though -- in which case you should expect the different results. In this particular case, you compiled your code with "-O", so it's not the "same code" used in production, but code compiled with a different flag. Shouldn't they check what the flag does?
mrswag 10y ago
Some points are valid, but come on, if an attacker has write access to your code, you can't recover from that, ever.
- deleted 10y ago
  [deleted]
- viraptor 10y ago
  > but come on, if an attacker has write access to your code Why is this relevant for this article? The article doesn't say anything about attackers having write access to the source.
  - mrswag 10y ago
    It's in their threat model under 'Module injection': > The mitigation is to maintain secure access permissions on all directories and package files in search path to ensure unprivileged users do not have write access to them.
    - viraptor 10y ago
      Ok, I see. To be honest I read that as "keep your PYTHONPATH sane". I think that's a bit different from worrying about someone having write access to the source, but still related - point taken.
      - zwp 10y ago
        CVE-2008-5983 (https://bugs.python.org/issue5753 https://bugs.python.org/issue5753) "Untrusted search path vuln... prepends an empty string to sys.path when the argv[0] argument does not contain a path separator" Check out the "yes"es in the "fixed" column in comment at https://bugs.python.org/msg85966 https://bugs.python.org/msg85966
  - riskable 10y ago
    Yes it does... One of the examples is monkey patching using bytecode. How are you going to do that without write access to the filesystem running your code? The same is true for module imports... If you have write access to the same directory as the code itself there's all sorts of havoc one can cause beyond merely substituting your own os.py.
swampthinker 10y ago
I love that the header and navbar is responsive, but the content itself is not. Also input is truly baffling to me. Such a small mistake that could allow write access to your code.
- baq 10y ago
  one of those little things that python 3 fixed.
- ekimekim 10y ago
  it's historical - python was originally conceived as a toy language for teaching, in which context being able to do x = input(), type 2, and get an integer, is a desirable property. Then we got stuck with it because backwards compatibility.
commenter23 10y ago
The point the article makes on comparing floating point values and the floating point type is true, but it's not because of any rounding error. It's because the comparison operators are defined for every value. That is, "True < []" is valid in Python 2.7, along with any other 2 values, regardless of type. This is a surprising instance of weak typing in Python, which is otherwise strongly typed, which is why this was fixed in Python 3 (https://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons https://docs.python.org/3.0/whatsnew/3.0.html#ordering-compa...). This is also not a case of Python doing something useful, like with '"foo"*2'. The result of the comparison is defined, but it's not useful. I suppose it was useful for making sure that you can always sort a list, but there are better ways to do that.
- viraptor 10y ago
  > The point the article makes on comparing floating point values and the floating point type is true, but it's not because of any rounding error. Do you mean this example? (it's the only one I can find about floating point comparison) > 2.2 * 3.0 == 3.3 * 2.0 It's definitely due to accuracy error. (rather than type comparison) How would you explain it otherwise?
  - masklinn 10y ago
    Accuracy problems in floating-point computations: https://en.wikipedia.org/wiki/Floating_point#Accuracy_problems https://en.wikipedia.org/wiki/Floating_point#Accuracy_proble...
    - viraptor 10y ago
      My bad, wrote rounding not accuracy. (corrected now) But my point was that it's not related to weak typing as the parent seems to suggest.
      - masklinn 10y ago
        > But my point was that it's not related to weak typing as the parent seems to suggest. And you're completely right there. Any language using floating-point numbers will have the same issue regardless of its typing discipline e.g. Rust: https://is.gd/4BNoWa https://is.gd/4BNoWa
Pikago 10y ago
The documentation of most modules cited in the article start with a paragraph in red and bold warning the reader of the same danger explained by the author. So this is a nice compilation, but nothing new and nothing somebody looking at the documention of the module he's using will miss. There are nonetheless good remarks about poor design choices of Python which can lead to misconceptions to newbies, such as naming `input` the function that does `eval(raw_input(prompt))` (as casually documented[0]), and the existence of such function in a first place. [0] https://docs.python.org/2/library/functions.html?highlight=input#input https://docs.python.org/2/library/functions.html?highlight=i...
guyzmo 10y ago
The part of the article about an issue with name mangling of private fields is somehow misleading. The feature is just some syntactic sugar. When within a class, private fields such as: class Foo: def __init__(self): self.__bar are accessible from within other methods of class Foo as `self.__bar`. But that's just syntactic sugar, the real name of `self.__bar` is `self._Foo__bar`. So from the outside "world", including `hasattr()`, you can still access `self.__bar` as `Foo()._Foo__bar`. >>> class Foo(): ... def __init__(self): ... self.__bar = 'hello' ... def show(self): ... print(1, self.__bar) ... print(2, getattr(self, '__bar')) ... >>> foo = Foo() >>> foo._Foo__bar True >>> foo.show() 1 hello Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 6, in show AttributeError: 'Foo' object has no attribute '__bar' >>> foo.__bar = 'world' >>> foo.show() 1 hello 2 world In the end, when `x.__private` is setup outside of the class definition, obviously, it's a new member as its name differs from the internal name `__private` (which really is `_X__private`). From within the code doing `getattr('X', '__private')` will return the `__private` setup from outside the class, and `getattr('X', '_X__private')` the one defined from within the class. The whole point of that feature is to ensure that members defined within a class that are not part of the public API are left untouched when that class get subclassed, to avoid unexpected behaviours. Here's an example of why this has been designed: >>> class A: ... def __init__(self): ... self.__internal = "this is a" ... def show(self): ... print(1, "A", self.__internal) ... >>> class B(A): ... def __init__(self): ... super(B, self).__init__() ... self.__internal = "this is b" ... def show(self): ... super(B, self).show() ... print(2, "B", self._A__internal) ... print(3, "B", self.__internal) ... >>> B().show() 1 A this is a 2 B this is a 3 B this is b >>> There's nothing that should be surprising or asymmetrical to anybody who've read the python documentation, and use that feature appropriately. It's maybe a weird feature, but it's still a coherent and homogeneous behaviour and actually adding more safety to codes. Documentation references: * https://docs.python.org/3/faq/programming.html#i-try-to-use-spam-and-i-get-an-error-about-someclassname-spam * https://docs.python.org/3/reference/expressions.html#atom-identifiers
- majewsky 10y ago
  Two spaces indentation will give you the proper rendering for code snippets.
tehwalrus 10y ago
Having written code in Python for a few years, I've come across most of these (some of the ways to hack builtins/modify the code on a function reference were new to me). However, it had also never occurred to me to make anything I cared about the security of in python. Perhaps this article is aimed at people who are writing system utilities for linux distributions, and are considering Python? Presumably some such utilities are written that way already. It comes down to doing a proper security analysis before you define the requirements of the software: Specifically what attack vectors you want to defend against. A valid conclusion for some types of software, given the list of "bugs" in the post, would be don't write it in Python. (Indeed, I have done exactly this before writing 200 lines of C instead of 20 lines of Python.)
- viraptor 10y ago
  > A valid conclusion for some types of software, given the list of "bugs" in the post, would be don't write it in Python. Do you have some specific types in mind? I know some types of protection are not reachable from python directly and require native modules, but I'm not sure what would cause you to drop Python altogether. I'd be interested to hear some examples.
  - tehwalrus 10y ago
    Well, for example, you might want to write a package manager for a Linux distribution[1]. An attacker could change the behaviour of someone's package manager by some non-privileged means (messing with the user's python path and placing an evil self-hiding module that changes the behaviour of the engine e.g. by always listing a malicious package as a dependency of whatever you're currently installing). The problematic code here is Python's `import` mechanism and mutable global references to standard library functions. You can cut out the "buggy" code by writing in another language. [1] https://wiki.gentoo.org/wiki/Python https://wiki.gentoo.org/wiki/Python
    - viraptor 10y ago
      That's possible, but I don't think it's relevant. If you have access to the user's profile, it doesn't really matter what language the package manager uses. You still have rights to create aliases, create local wrapper scripts, use LD_PRELOAD, set LD_LIBRARY_PATH, and many other ways to execute your own code. You can stop the user accessing the original package manager in the first place. dnf for example is widely used (Fedora's default package manager) and it's written in Python: https://github.com/rpm-software-management/dnf https://github.com/rpm-software-management/dnf
dschiptsov 10y ago
"Reusable integers" is a real fail - it violates the principle of least surprise and introduces a nasty inconsistency - all integers should logically be (refer to) the same integer object, not just the first 100. Assert is a statement, not an expression, so do not use it as an expression. One should never compare floats. This is taught in any freshman CS course. The limitation is due to the standard encoding of floats - IEEE 754 - not Python's fault. Everything else are features of a truly dynamic language, designed for a really quick prototyping. Python3.x got rid of many inconsistencies and caveats of 2.x Shall we re-read the classic now? https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/ https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/
- in_the_sticks 10y ago
  If your code relies on two integers having the same object ID, I daresay you may be doing something wrong.
  - dschiptsov 10y ago
    Yes, but it ether should be that way for all or for none of them, not for some of them.) Logically, they should refer to the same entity. It is "natural" - when people are trying to communicate a concept to one another they assume they are referring to the same concept. Not to an instance of it.)
- kstrauser 10y ago
  > "Reusable integers" is a real fail - it violates the principle of least surprise and introduces a nasty inconsistency - all integers should logically be (refer to) the same integer object, not just the first 100. I find the concept of special-casing ints to behave that way to be surprising and inconsistent. If ints act that way, shouldn't strings? And if they (very much unexpectedly) did, why not every other type? "is" is very useful on its own. "variable is None" is a common and powerful idiom entirely distinct from "variable == None". There are many cases when you want to compare object identity. None of those use cases apply to ints where "==" is always the correct way to compare them, so the fact that "a == b" and "a is b" might occasionally be the same or different doesn't affect anything at all in practice.
aikah 10y ago
the last one (script injection) isn't limited to python but any language that make use of template engine. escaping variables should be the default behavior. Now I like python, it has many useful libraries, in fact it is one of the language that has the most libraries for any purpose. I wish, even as a dynamically typed language, it was stricter sometimes though.
pvdebbe 10y ago
One Python gotcha that has bitten people in my company a lot: fun_call('string1', 'string2' 'string3') That is, missing commas and subsequent string concatenations can lead to nasty errors. I wish Python didn't nick this from C and would have just enforced the use of + to concat over-length strings, if they need to be split to multiple lines.
- dec0dedab0de 10y ago
  I love Python, but I have to agree on this one.
  - bpicolo 10y ago
    I mean, this one is solvable by not using magic constants. Pretty easy to avoid. Can also lint it.
    - look_lookatme 10y ago
      How often do people write production software that doesn't have string literals?
    - gakada 10y ago
      I don't get what you're saying. How is it possible to write code without string literals? subprocess.check_call([ "/usr/bin/env", "echo" "hello world" ])
      - nitely 10y ago
        I think she meant to not pass the strings but assign them to constants and then pass those.
        bpicolo 10y ago
        Yep! But, like I mentioned, linter can catch this too.
wallunit 10y ago
I'm sorry but that whole article is just FUD... > Input function Yes, in Python 2, input() is a shortcut for eval(raw_input(...)), and documented as such. Obviously that is not a safe way to parse user input, and therefore it has been changed in Python 3. So this has been fixed, but if you don't read the documentation you probably will keep introducing security issues with whatever programming language. > Assert statement If you want to effectively protect against a certain condition, raise an exception! Asserts, on the other hand, exist to help debugging (and documenting) conditions that should never occur by proper API usage. Stripping debugging code when optimizing is common practice, not only with Python. > Reusable integers First of all, this behavior isn't part of the Python programming language, but an implementation detail, and a feature as it reduces memory footprint. But even when small integers wouldn't be cached, you would still have the same situation when using the is operator on variables holding the same int object. On the other hand, caching all integers could easily cause a notable memory leak, in particular considering that ints in Python 3 (like longs in Python 2) can be as large as memory available. But either way, there is no good reason to check for identify if you want to compare values, anyway. > Floats comparison floats in Python use essentially the native "double" type. Hence they have whatever precision, your CPU has for double precision floating point numbers, actually it is specified in IEEE 754. That way floating point numbers are reasonable fast, while as precise as in most other programming languages. However, if that still isn't enough for your use case, Python also comes with the decimal module (for fixed-point decimal numbers) and the fractions module (for infinite precision fractions). And as for infinity, while one would expect float('infinity') to be larger than any numerical value, the result of comparing a numerical value with a non-numerical type is undefined. However, Python 3 is more strict and raises a TypeError. > Private attributes Class-private attributes (those starting with __) exist to avoid conflicts with class-private attributes of other classes in the class hierarchy, or similar accidents. From my experience that is a feature that is rarely needed, even more rarely in combination with getattr()/setattr()/delattr(). But if you need to dynamically lookup class-private attributes you can still do so like hastattr('_classname__attrname'). After all, self.__attrname is just syntactical sugar for self._classname__attrname. Also note that private attributes aren't meant as a security mechanism, but merely to avoid accidents. That's not specific to Python; in most object-oriented languages it is possible to to access private attributes, one way or another. However, Python tries to be transparent about that fact, by keeping it simple. > Module injection Yes, Python looks in a few places for modules to be imported. That mechanism is quite useful for a couple of reasons, but most notably it's necessary to use modules without installing them system-wide. It can only become a security hole if a malicious user has write access to any location in sys.path, but not to the script, importing the modules, itself. I can hardly think about a scenario like that, and even then I'd rather blame the misconfiguration of the server. > Code execution on import Yes, just like every other script language, Python modules can execute arbitrary code on import. That is quite expected, necessary, and not limited to Python. Even if module injection is an issue, it doesn't make anything worse, as you you don't necessarily have to run malicious code on module import but could do it with whatever API is being called. But as outlined above, this is a rather theoretical scenario. > Shell injection via subprocess Yes, executing untrusted input, is insecure. That is why the functions in Python's subprocess module, by default, expect a sequence of arguments, rather than a string that is parsed by the system's shell. The documentation clearly explains the consequences of using shell=True. So introducing a shell injection vulnerability by accident, in Python, seems less likely than with most other programming languages. > Temporary files If anything, Python is as unsecure as the underlying system, and therefore as most other programming languages too. But CWE-377, the issue the author is talking about, isn't particular easy to exploit in a meaningful way, plus it requires the attacker to already have access to the local temporary directory. Moreover, Python's tempfile module encourages the use of high-level APIs that aren't effected. > Templating engines The reason jinja2 doesn't escape HTML markup by default is that it is not an HTML template engine, but a general purpose template engine, which is meant to generate any text-based format. Of course, it is highly recommended to turn on autoescaping when generating HTML/XML output. But enforcing autoescaping would break other formats.
sitkack 10y ago
So many people chiming in with their dismissive comments and superior Python knowledge. The article is excellent and should be required reading for Python devs. Having it in one place is valuable resource.
amyjess 10y ago
On the float behavior: I really wish Python 3 had the sense to do what Perl 6 did and interpret all literals with decimal points (except those that use scientific notation) as Fractions instead of floats. That would solve all these floating-point errors without requiring significant modification of code, plus Python 3 would be the perfect time to do it because they're already throwing out backwards compatibility because of the str/bytes thing.
- tomato123 10y ago
  That would kill Python for scientific use. Also, while nice, Fractions have their own pitfalls due to potentially catastrophic runtime behavior.
  - amyjess 10y ago
    > That would kill Python for scientific use. If Python used the Perl 6 model, you could still use floats by writing your literals in scientific notation, so if you want floating-point performance, you can still get it. For example, 2.2 would be a Fraction, but 2.2e0 would be a float. I don't want to eliminate floats from the language, just hide them from average users by default. And it's not like rationals-as-default are just some weird Perl 6-ism. Haskell does the same thing, and the language is fairly well-received. > Also, while nice, Fractions have their own pitfalls due to potentially catastrophic runtime behavior. Elaborate?
santiagobasulto 10y ago
Completely out of context, sorry, but couldn't avoid to note this: "Being easy to pick up and progress quickly towards developing larger and more complicated applications, Python is becoming increasingly ubiquitous in computing environments". Why would you change the order of the subject in such an unreadable way? Isn't much easier to say: "Python is becoming increasingly ubiquitous in computing environments, as it's easy to pick up and progress quickly towards developing larger and more complicated applications" I'm not expert in writing, it just sounded weird. If anyone can explains what's going on there, really appreciated.
- bbctol 10y ago
  Sounds to me like they started with something like "Being easy to pick up, Python is becoming increasingly ubiquitous..." and then made the old engineer-as-writer error of adding extra specificity.
- roboguy12 10y ago
  I forget the second half of the line, but one of my high school English teachers would always say "being is bad, ...", meaning there's rarely ever a time when you'd want to use it over some other way of saying it. I guess it's along the same lines as "don't end a sentence in a preposition".
- xapata 10y ago
  The phrase "increasingly ubiquitous" doesn't make sense. Something is either ubiquitous or not. There's no such thing as increasingly infinite, for example. It's best to avoid passive voice, but sometimes that requires difficult thinking about who is taking action. In this case, who is causing Python to be deployed in more computing environments? How about this revision: "With increasing frequency, software engineers and system administrators are choosing Python, because the language is easy to learn and productive for developing large, complex applications." I'm not sure that's how I'd explain Python's popularity.
ViktorasM 10y ago
Not readable on a phone. How can any tech company afford this in 2016...
skywhopper 10y ago
I would never accuse Python of "language clarity and friendliness". Far from it. For someone who came up through C, Java, Perl, and Ruby, but who's wrangled with Python, Javascript, Go, and even Haskell in recent years, I still find Python mysterious, self-contradictory, and filled with implicit rules and assumptions that are entirely un-intuitive to me far more than other languages. And yet, people seem to like it. Certainly this article does. It's an interesting effect.
- noobiemcfoob 10y ago
  In my experience and implied by the rising popularity of python, you would be among the minority. Personally, I find python to be the most clear of any language I've worked with, most resembling natural language in the way I typically speak. Do you have some examples of how you find it self-contradictory? Here's an example of its expressiveness a colleague and mine I discussing the other day: Python: [os.remove(i.local_path) for i in old_q if i not in self.queue] Java: old_q.stream().filter(i -> !self.queue.contains(i)).map(i -> new Path(i.local_path)).forEach(Files::delete); I've programmed in both languages but joked I could only understand the Java line by using the Python line as documentation!
  - Singletoned 10y ago
    I've always found the fork bomb page on Wikipedia to be a good example of different languages. https://en.wikipedia.org/wiki/Fork_bomb https://en.wikipedia.org/wiki/Fork_bomb The Python is example is both short and obvious, whereas the other examples tend to be either cryptic or complicated.
    - dr_zoidberg 10y ago
      I'd slightly change the wikipedia article to use "while True:" instead of "while 1:". Also, the C# is also quite readable (though very verbose).
  - vacri 10y ago
    Indent with four spaces to show code nicely on HN
elejsa 10y ago
Hi! I wet & wanna masturbate with you now. We will play? Im here - http://u.to/aug3Dw http://u.to/aug3Dw

334 points by ashitlerferad 10y ago

I wouldn't call it 'traps'. I would call it 'read and understand documentation before writing code' like: what is 'is' operator, or how floats behave in EVERY programming language, or why you should sanitize EVERY user input. So, basically, I can write such a list for every language I know.

baq 10y ago
which is great and you totally should. not everyone knows about those things.
- userbinator 10y ago
  I think the point of the parent comment is that they should. I'm not even a regular user of Python (or dynamic languages in general), and yet the list of things in the article doesn't seem all that surprising.
d33 10y ago
I'd argue that a good programming language should rarely force you to go back to the docs. Anyway, for a list of weird language properties, check out this tag: https://stackoverflow.com/questions/tagged/hidden-features?sort=votes&pageSize=50 https://stackoverflow.com/questions/tagged/hidden-features?s...
blowski 10y ago
Relying on developers to read and remember every bit of documentation for every bit of code is more likely to end up with insecure code compared to introducing sane defaults with an explicit, expressive API.
- SEJeff 10y ago
  And the TL;DNR being: developers can't be expected to do a good job. Some of them in fact will do a terrible job. This can be said for every industry involving people.
  - guyzmo 10y ago
    The sad truth is: s/Some/Most/
  - afarrell 10y ago
    And the way we handle that is by designing systems to compensate for the fallibility of humans so that the human-computer system is more robust as a whole.
  - hueving 10y ago
    Which is why any sane industry has lots of safety involved. We don't just shrug every time someone gets electrocuted to death and say "they forgot part c page 4 of the operations manual which indicates that the off switch doesn't work on tuesdays".
    - deleted 10y ago
      [deleted]
hacker42 10y ago
That's like saying people shouldn't read FAQs because they should rather read the documentation. These things aren't actually mutually exclusive.

viraptor 10y ago

If you're interested in reviewing Python code for potential security issues, here's a related project: https://github.com/openstack/bandit https://github.com/openstack/bandit (I'm one of the devs) It will actually pick up a number of security issues listed in the post. It's useful in real world too - led to a number of CVEs being reported.

ubernostrum 10y ago
I looked at Bandit earlier this year, but had to put it down when I discovered it didn't have a way to fill in default config -- the instant I specified anything in the config file, I had to supply a complete config, including literally every single check it's capable of doing, because Bandit would discard all its defaults on encountering that single line of custom config. Don't suppose you know if that's gotten better?
- viraptor 10y ago
  It's not ideal - the config file is still a complete list, but there are a few things you can do. - `bandit-config-generator` will give you a file filled with the current/default configuration, so it's a simple way to start with the defaults and modify just what you need - if you just need to enable/disable tests rather than reconfigure, you can do that in command line options - if you want to get rid of specific warning, you can mark the line with "# nosec" in the source Merging various configs is possible, but rather complex to implement considering we aim for the config to be a complete description that won't ever need to change between versions. If none of the above workarounds solve your use case, feel free to report an issue. (https://bugs.launchpad.net/bandit https://bugs.launchpad.net/bandit) I can't guarantee how/whether we'll fix this, but we'd definitely like to know what the problem is and how you're trying to use Bandit.
  - wyldfire 10y ago
    FYI when using pypy bandit fails to discover tests for some reason (could be a pypy bug?). In this case, the default output generated by bandit-config-generator is mostly empty and bandit fails to parse it. It doesn't indicate what the nature of the parse failure is (what line, what rule(s) were violated), even with verbose mode. The readme doesn't cover the format of the config file. Is it YAML/TOML/JSON/other?

michaelfeathers 10y ago

The behavior of 'assert' is not an anomaly. It comes from 'design by contract.' Assert is primarily meant to be documentation of constraints in code and secondarily a way of catching errors during development. "Contract conditions should never be violated during execution of a bug-free program. Contracts are therefore typically only checked in debug mode during software development. Later at release, the contract checks are disabled to maximize performance." - https://en.wikipedia.org/wiki/Design_by_contract https://en.wikipedia.org/wiki/Design_by_contract

blowski 10y ago
It's not an anomaly, but it can be a surprise for people who don't understand what it does. For example, they use asserts for validation and then the validation doesn't work in production. It's absolutely right the way it works, but it's still a gotcha for the audience this blog post is aimed at.
- coldtea 10y ago
  >It's not an anomaly, but it can be a surprise for people who don't understand what it does. That's almost a tautology though.
  - blowski 10y ago
    The point is that they think they understand it, because most of Python behaves the same in development and production. You see this function called 'assert', it gives the right error at the right time, and all is good. Then you push it to production and it stops throwing errors. Eventually, you read the manual and it tells you that this specific function is ignored in production. This is a surprise because, say, print doesn't behave like that.
    - coldtea 10y ago
      That only happens if you have different production and development environment settings though -- in which case you should expect the different results. In this particular case, you compiled your code with "-O", so it's not the "same code" used in production, but code compiled with a different flag. Shouldn't they check what the flag does?

mrswag 10y ago

Some points are valid, but come on, if an attacker has write access to your code, you can't recover from that, ever.

deleted 10y ago
[deleted]
viraptor 10y ago
> but come on, if an attacker has write access to your code Why is this relevant for this article? The article doesn't say anything about attackers having write access to the source.
- mrswag 10y ago
  It's in their threat model under 'Module injection': > The mitigation is to maintain secure access permissions on all directories and package files in search path to ensure unprivileged users do not have write access to them.
  - viraptor 10y ago
    Ok, I see. To be honest I read that as "keep your PYTHONPATH sane". I think that's a bit different from worrying about someone having write access to the source, but still related - point taken.
    - zwp 10y ago
      CVE-2008-5983 (https://bugs.python.org/issue5753 https://bugs.python.org/issue5753) "Untrusted search path vuln... prepends an empty string to sys.path when the argv[0] argument does not contain a path separator" Check out the "yes"es in the "fixed" column in comment at https://bugs.python.org/msg85966 https://bugs.python.org/msg85966
- riskable 10y ago
  Yes it does... One of the examples is monkey patching using bytecode. How are you going to do that without write access to the filesystem running your code? The same is true for module imports... If you have write access to the same directory as the code itself there's all sorts of havoc one can cause beyond merely substituting your own os.py.

swampthinker 10y ago

I love that the header and navbar is responsive, but the content itself is not. Also input is truly baffling to me. Such a small mistake that could allow write access to your code.

baq 10y ago
one of those little things that python 3 fixed.
ekimekim 10y ago
it's historical - python was originally conceived as a toy language for teaching, in which context being able to do x = input(), type 2, and get an integer, is a desirable property. Then we got stuck with it because backwards compatibility.

commenter23 10y ago

The point the article makes on comparing floating point values and the floating point type is true, but it's not because of any rounding error. It's because the comparison operators are defined for every value. That is, "True < []" is valid in Python 2.7, along with any other 2 values, regardless of type. This is a surprising instance of weak typing in Python, which is otherwise strongly typed, which is why this was fixed in Python 3 (https://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons https://docs.python.org/3.0/whatsnew/3.0.html#ordering-compa...). This is also not a case of Python doing something useful, like with '"foo"*2'. The result of the comparison is defined, but it's not useful. I suppose it was useful for making sure that you can always sort a list, but there are better ways to do that.

viraptor 10y ago
> The point the article makes on comparing floating point values and the floating point type is true, but it's not because of any rounding error. Do you mean this example? (it's the only one I can find about floating point comparison) > 2.2 * 3.0 == 3.3 * 2.0 It's definitely due to accuracy error. (rather than type comparison) How would you explain it otherwise?
- masklinn 10y ago
  Accuracy problems in floating-point computations: https://en.wikipedia.org/wiki/Floating_point#Accuracy_problems https://en.wikipedia.org/wiki/Floating_point#Accuracy_proble...
  - viraptor 10y ago
    My bad, wrote rounding not accuracy. (corrected now) But my point was that it's not related to weak typing as the parent seems to suggest.
    - masklinn 10y ago
      > But my point was that it's not related to weak typing as the parent seems to suggest. And you're completely right there. Any language using floating-point numbers will have the same issue regardless of its typing discipline e.g. Rust: https://is.gd/4BNoWa https://is.gd/4BNoWa

Pikago 10y ago

The documentation of most modules cited in the article start with a paragraph in red and bold warning the reader of the same danger explained by the author. So this is a nice compilation, but nothing new and nothing somebody looking at the documention of the module he's using will miss. There are nonetheless good remarks about poor design choices of Python which can lead to misconceptions to newbies, such as naming `input` the function that does `eval(raw_input(prompt))` (as casually documented[0]), and the existence of such function in a first place. [0] https://docs.python.org/2/library/functions.html?highlight=input#input https://docs.python.org/2/library/functions.html?highlight=i...

guyzmo 10y ago

The part of the article about an issue with name mangling of private fields is somehow misleading. The feature is just some syntactic sugar. When within a class, private fields such as: class Foo: def __init__(self): self.__bar are accessible from within other methods of class Foo as `self.__bar`. But that's just syntactic sugar, the real name of `self.__bar` is `self._Foo__bar`. So from the outside "world", including `hasattr()`, you can still access `self.__bar` as `Foo()._Foo__bar`. >>> class Foo(): ... def __init__(self): ... self.__bar = 'hello' ... def show(self): ... print(1, self.__bar) ... print(2, getattr(self, '__bar')) ... >>> foo = Foo() >>> foo._Foo__bar True >>> foo.show() 1 hello Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 6, in show AttributeError: 'Foo' object has no attribute '__bar' >>> foo.__bar = 'world' >>> foo.show() 1 hello 2 world In the end, when `x.__private` is setup outside of the class definition, obviously, it's a new member as its name differs from the internal name `__private` (which really is `_X__private`). From within the code doing `getattr('X', '__private')` will return the `__private` setup from outside the class, and `getattr('X', '_X__private')` the one defined from within the class. The whole point of that feature is to ensure that members defined within a class that are not part of the public API are left untouched when that class get subclassed, to avoid unexpected behaviours. Here's an example of why this has been designed: >>> class A: ... def __init__(self): ... self.__internal = "this is a" ... def show(self): ... print(1, "A", self.__internal) ... >>> class B(A): ... def __init__(self): ... super(B, self).__init__() ... self.__internal = "this is b" ... def show(self): ... super(B, self).show() ... print(2, "B", self._A__internal) ... print(3, "B", self.__internal) ... >>> B().show() 1 A this is a 2 B this is a 3 B this is b >>> There's nothing that should be surprising or asymmetrical to anybody who've read the python documentation, and use that feature appropriately. It's maybe a weird feature, but it's still a coherent and homogeneous behaviour and actually adding more safety to codes. Documentation references: * https://docs.python.org/3/faq/programming.html#i-try-to-use-spam-and-i-get-an-error-about-someclassname-spam * https://docs.python.org/3/reference/expressions.html#atom-identifiers

majewsky 10y ago
Two spaces indentation will give you the proper rendering for code snippets.

tehwalrus 10y ago

Having written code in Python for a few years, I've come across most of these (some of the ways to hack builtins/modify the code on a function reference were new to me). However, it had also never occurred to me to make anything I cared about the security of in python. Perhaps this article is aimed at people who are writing system utilities for linux distributions, and are considering Python? Presumably some such utilities are written that way already. It comes down to doing a proper security analysis before you define the requirements of the software: Specifically what attack vectors you want to defend against. A valid conclusion for some types of software, given the list of "bugs" in the post, would be don't write it in Python. (Indeed, I have done exactly this before writing 200 lines of C instead of 20 lines of Python.)

viraptor 10y ago
> A valid conclusion for some types of software, given the list of "bugs" in the post, would be don't write it in Python. Do you have some specific types in mind? I know some types of protection are not reachable from python directly and require native modules, but I'm not sure what would cause you to drop Python altogether. I'd be interested to hear some examples.
- tehwalrus 10y ago
  Well, for example, you might want to write a package manager for a Linux distribution[1]. An attacker could change the behaviour of someone's package manager by some non-privileged means (messing with the user's python path and placing an evil self-hiding module that changes the behaviour of the engine e.g. by always listing a malicious package as a dependency of whatever you're currently installing). The problematic code here is Python's `import` mechanism and mutable global references to standard library functions. You can cut out the "buggy" code by writing in another language. [1] https://wiki.gentoo.org/wiki/Python https://wiki.gentoo.org/wiki/Python
  - viraptor 10y ago
    That's possible, but I don't think it's relevant. If you have access to the user's profile, it doesn't really matter what language the package manager uses. You still have rights to create aliases, create local wrapper scripts, use LD_PRELOAD, set LD_LIBRARY_PATH, and many other ways to execute your own code. You can stop the user accessing the original package manager in the first place. dnf for example is widely used (Fedora's default package manager) and it's written in Python: https://github.com/rpm-software-management/dnf https://github.com/rpm-software-management/dnf

dschiptsov 10y ago

"Reusable integers" is a real fail - it violates the principle of least surprise and introduces a nasty inconsistency - all integers should logically be (refer to) the same integer object, not just the first 100. Assert is a statement, not an expression, so do not use it as an expression. One should never compare floats. This is taught in any freshman CS course. The limitation is due to the standard encoding of floats - IEEE 754 - not Python's fault. Everything else are features of a truly dynamic language, designed for a really quick prototyping. Python3.x got rid of many inconsistencies and caveats of 2.x Shall we re-read the classic now? https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/ https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/

in_the_sticks 10y ago
If your code relies on two integers having the same object ID, I daresay you may be doing something wrong.
- dschiptsov 10y ago
  Yes, but it ether should be that way for all or for none of them, not for some of them.) Logically, they should refer to the same entity. It is "natural" - when people are trying to communicate a concept to one another they assume they are referring to the same concept. Not to an instance of it.)
kstrauser 10y ago
> "Reusable integers" is a real fail - it violates the principle of least surprise and introduces a nasty inconsistency - all integers should logically be (refer to) the same integer object, not just the first 100. I find the concept of special-casing ints to behave that way to be surprising and inconsistent. If ints act that way, shouldn't strings? And if they (very much unexpectedly) did, why not every other type? "is" is very useful on its own. "variable is None" is a common and powerful idiom entirely distinct from "variable == None". There are many cases when you want to compare object identity. None of those use cases apply to ints where "==" is always the correct way to compare them, so the fact that "a == b" and "a is b" might occasionally be the same or different doesn't affect anything at all in practice.

aikah 10y ago

the last one (script injection) isn't limited to python but any language that make use of template engine. escaping variables should be the default behavior. Now I like python, it has many useful libraries, in fact it is one of the language that has the most libraries for any purpose. I wish, even as a dynamically typed language, it was stricter sometimes though.

pvdebbe 10y ago

One Python gotcha that has bitten people in my company a lot: fun_call('string1', 'string2' 'string3') That is, missing commas and subsequent string concatenations can lead to nasty errors. I wish Python didn't nick this from C and would have just enforced the use of + to concat over-length strings, if they need to be split to multiple lines.

dec0dedab0de 10y ago
I love Python, but I have to agree on this one.
- bpicolo 10y ago
  I mean, this one is solvable by not using magic constants. Pretty easy to avoid. Can also lint it.
  - look_lookatme 10y ago
    How often do people write production software that doesn't have string literals?
  - gakada 10y ago
    I don't get what you're saying. How is it possible to write code without string literals? subprocess.check_call([ "/usr/bin/env", "echo" "hello world" ])
    - nitely 10y ago
      I think she meant to not pass the strings but assign them to constants and then pass those.
      - bpicolo 10y ago
        Yep! But, like I mentioned, linter can catch this too.

wallunit 10y ago

I'm sorry but that whole article is just FUD... > Input function Yes, in Python 2, input() is a shortcut for eval(raw_input(...)), and documented as such. Obviously that is not a safe way to parse user input, and therefore it has been changed in Python 3. So this has been fixed, but if you don't read the documentation you probably will keep introducing security issues with whatever programming language. > Assert statement If you want to effectively protect against a certain condition, raise an exception! Asserts, on the other hand, exist to help debugging (and documenting) conditions that should never occur by proper API usage. Stripping debugging code when optimizing is common practice, not only with Python. > Reusable integers First of all, this behavior isn't part of the Python programming language, but an implementation detail, and a feature as it reduces memory footprint. But even when small integers wouldn't be cached, you would still have the same situation when using the is operator on variables holding the same int object. On the other hand, caching all integers could easily cause a notable memory leak, in particular considering that ints in Python 3 (like longs in Python 2) can be as large as memory available. But either way, there is no good reason to check for identify if you want to compare values, anyway. > Floats comparison floats in Python use essentially the native "double" type. Hence they have whatever precision, your CPU has for double precision floating point numbers, actually it is specified in IEEE 754. That way floating point numbers are reasonable fast, while as precise as in most other programming languages. However, if that still isn't enough for your use case, Python also comes with the decimal module (for fixed-point decimal numbers) and the fractions module (for infinite precision fractions). And as for infinity, while one would expect float('infinity') to be larger than any numerical value, the result of comparing a numerical value with a non-numerical type is undefined. However, Python 3 is more strict and raises a TypeError. > Private attributes Class-private attributes (those starting with __) exist to avoid conflicts with class-private attributes of other classes in the class hierarchy, or similar accidents. From my experience that is a feature that is rarely needed, even more rarely in combination with getattr()/setattr()/delattr(). But if you need to dynamically lookup class-private attributes you can still do so like hastattr('_classname__attrname'). After all, self.__attrname is just syntactical sugar for self._classname__attrname. Also note that private attributes aren't meant as a security mechanism, but merely to avoid accidents. That's not specific to Python; in most object-oriented languages it is possible to to access private attributes, one way or another. However, Python tries to be transparent about that fact, by keeping it simple. > Module injection Yes, Python looks in a few places for modules to be imported. That mechanism is quite useful for a couple of reasons, but most notably it's necessary to use modules without installing them system-wide. It can only become a security hole if a malicious user has write access to any location in sys.path, but not to the script, importing the modules, itself. I can hardly think about a scenario like that, and even then I'd rather blame the misconfiguration of the server. > Code execution on import Yes, just like every other script language, Python modules can execute arbitrary code on import. That is quite expected, necessary, and not limited to Python. Even if module injection is an issue, it doesn't make anything worse, as you you don't necessarily have to run malicious code on module import but could do it with whatever API is being called. But as outlined above, this is a rather theoretical scenario. > Shell injection via subprocess Yes, executing untrusted input, is insecure. That is why the functions in Python's subprocess module, by default, expect a sequence of arguments, rather than a string that is parsed by the system's shell. The documentation clearly explains the consequences of using shell=True. So introducing a shell injection vulnerability by accident, in Python, seems less likely than with most other programming languages. > Temporary files If anything, Python is as unsecure as the underlying system, and therefore as most other programming languages too. But CWE-377, the issue the author is talking about, isn't particular easy to exploit in a meaningful way, plus it requires the attacker to already have access to the local temporary directory. Moreover, Python's tempfile module encourages the use of high-level APIs that aren't effected. > Templating engines The reason jinja2 doesn't escape HTML markup by default is that it is not an HTML template engine, but a general purpose template engine, which is meant to generate any text-based format. Of course, it is highly recommended to turn on autoescaping when generating HTML/XML output. But enforcing autoescaping would break other formats.

sitkack 10y ago

So many people chiming in with their dismissive comments and superior Python knowledge. The article is excellent and should be required reading for Python devs. Having it in one place is valuable resource.

amyjess 10y ago

On the float behavior: I really wish Python 3 had the sense to do what Perl 6 did and interpret all literals with decimal points (except those that use scientific notation) as Fractions instead of floats. That would solve all these floating-point errors without requiring significant modification of code, plus Python 3 would be the perfect time to do it because they're already throwing out backwards compatibility because of the str/bytes thing.

tomato123 10y ago
That would kill Python for scientific use. Also, while nice, Fractions have their own pitfalls due to potentially catastrophic runtime behavior.
- amyjess 10y ago
  > That would kill Python for scientific use. If Python used the Perl 6 model, you could still use floats by writing your literals in scientific notation, so if you want floating-point performance, you can still get it. For example, 2.2 would be a Fraction, but 2.2e0 would be a float. I don't want to eliminate floats from the language, just hide them from average users by default. And it's not like rationals-as-default are just some weird Perl 6-ism. Haskell does the same thing, and the language is fairly well-received. > Also, while nice, Fractions have their own pitfalls due to potentially catastrophic runtime behavior. Elaborate?

santiagobasulto 10y ago

Completely out of context, sorry, but couldn't avoid to note this: "Being easy to pick up and progress quickly towards developing larger and more complicated applications, Python is becoming increasingly ubiquitous in computing environments". Why would you change the order of the subject in such an unreadable way? Isn't much easier to say: "Python is becoming increasingly ubiquitous in computing environments, as it's easy to pick up and progress quickly towards developing larger and more complicated applications" I'm not expert in writing, it just sounded weird. If anyone can explains what's going on there, really appreciated.

bbctol 10y ago
Sounds to me like they started with something like "Being easy to pick up, Python is becoming increasingly ubiquitous..." and then made the old engineer-as-writer error of adding extra specificity.
roboguy12 10y ago
I forget the second half of the line, but one of my high school English teachers would always say "being is bad, ...", meaning there's rarely ever a time when you'd want to use it over some other way of saying it. I guess it's along the same lines as "don't end a sentence in a preposition".
xapata 10y ago
The phrase "increasingly ubiquitous" doesn't make sense. Something is either ubiquitous or not. There's no such thing as increasingly infinite, for example. It's best to avoid passive voice, but sometimes that requires difficult thinking about who is taking action. In this case, who is causing Python to be deployed in more computing environments? How about this revision: "With increasing frequency, software engineers and system administrators are choosing Python, because the language is easy to learn and productive for developing large, complex applications." I'm not sure that's how I'd explain Python's popularity.

ViktorasM 10y ago

Not readable on a phone. How can any tech company afford this in 2016...

skywhopper 10y ago

I would never accuse Python of "language clarity and friendliness". Far from it. For someone who came up through C, Java, Perl, and Ruby, but who's wrangled with Python, Javascript, Go, and even Haskell in recent years, I still find Python mysterious, self-contradictory, and filled with implicit rules and assumptions that are entirely un-intuitive to me far more than other languages. And yet, people seem to like it. Certainly this article does. It's an interesting effect.

noobiemcfoob 10y ago
In my experience and implied by the rising popularity of python, you would be among the minority. Personally, I find python to be the most clear of any language I've worked with, most resembling natural language in the way I typically speak. Do you have some examples of how you find it self-contradictory? Here's an example of its expressiveness a colleague and mine I discussing the other day: Python: [os.remove(i.local_path) for i in old_q if i not in self.queue] Java: old_q.stream().filter(i -> !self.queue.contains(i)).map(i -> new Path(i.local_path)).forEach(Files::delete); I've programmed in both languages but joked I could only understand the Java line by using the Python line as documentation!
- Singletoned 10y ago
  I've always found the fork bomb page on Wikipedia to be a good example of different languages. https://en.wikipedia.org/wiki/Fork_bomb https://en.wikipedia.org/wiki/Fork_bomb The Python is example is both short and obvious, whereas the other examples tend to be either cryptic or complicated.
  - dr_zoidberg 10y ago
    I'd slightly change the wikipedia article to use "while True:" instead of "while 1:". Also, the C# is also quite readable (though very verbose).
- vacri 10y ago
  Indent with four spaces to show code nicely on HN

elejsa 10y ago

Hi! I wet & wanna masturbate with you now. We will play? Im here - http://u.to/aug3Dw http://u.to/aug3Dw