12 ms·
Be Careful with Python's New-Style String Format
- deleted 10y ago[deleted]
- atabdiwa 10y ago... Uncontrolled format string bugs? In 2016? Really? Someone would fall for that? ..
- the_mitsuhiko 10y agoOld style format strings are perfectly safe for arbitrary user input and commonly used as such.
- dom0 10y agoUser-provided format strings seem rather rare to me ; at least in scenarios where the user doesn't have full access anyway (eg. controlling output formatting of some command).
- coldtea 10y agoErr, why would you allow for the user to enter arbitrary format strings in the first place? Might as well write "be careful about eval of arbitrary user provided strings".
- tedunangst 10y ago"Customize the look of your blog by editing these templates."
- coldtea 10y agoShouldn't that be based not on the built-in string capabilities, but on a template engine, that must also take care for much more narrow cases of abuse (e.g. SQL injection, or access to only of a restricted number of helper functions within the template)?
- tedunangst 10y agoMany templates allow accessing fields. There's no risk of SQL injection, you're only echoing the output back to the user.
- yawaramin 10y agoDidn't you just point out that some of those fields could be sensitive, like password info?
- icebraining 10y agoThe author discovered this exploit in the template engine he wrote (Jinja2).
- nine_k 10y agoI'll consider switching to a different engine.
- jessaustin 10y agoPerhaps one that hasn't had this exploit fixed yet?
- coldtea 10y agoWhy would a template engine allow for free access to the native platforms string capabilities in the first place?
- Animats 10y agoInternationalization, usually. Word order differs between languages, and this kind of format allows reordering the inserted values.
- zbraniecki 10y agoInternationalization should never use arbitrary string or system level formatting. Look at MessageFormat or L20n. Yeah, I know that it's more complex than what you think you need (I've hear the "let's just use JS template literals" so many times) but you actually do.
- coldtea 10y agoYes, but that's meant for trusted translators of the UI, nor arbitrary users, right?
- icebraining 10y agoIn some applications, users can change the their own translations, which are stored in the DB, to allow for easier customization.
- coldtea 10y agoStill, no reason to save them in the platforms native string format and with full reign over native strings.
- azernik 10y agoIn practice, translators are often externally contracted; the amount of work required to translate any given language is not large enough to make these full-time positions. So only trusted so far.
- Manishearth 10y agoA lot of folks do use lang format strings for i18n, but they really shouldn't. They won't be able to handle pluralization and other oddities.
- brianwawok 10y agoTemplates. Very useful in many places. Although if you are using templates in Python, you are likely using jinja2. And they just "fixed" the bug. So you are good to go if your upgrade.
- dom0 10y agoAlthough this only applies to format-string evaluation in Jinja2, which would look something like {{ "Hello {user.name}".format(user) }} which is really kinda obscure, especially since Jinja already has an explicit |format using the very safe / limited (but generally sufficient) alternative %(key)s syntax (also suitable for i18n, to some extend).
- jlarocco 10y agoYeah, that was my first thought, too, and it's not very fair to blame Python's new style string format. Treating user input as a format string has always been a bad idea in every language and library that use them.
- thinkpad20 10y agoNot necessarily true. Type systems can ensure safety without too much effort. For example in Haskell, you might have some sort of format function with the type signature -- Given appropriate Error and Context types format :: Context -> Text -> Either Error Text Which would be guaranteed to be safe (there is no way this function can execute arbitrary code or read from your file system, short of egregious abuses of unsafePerformIO in its implementation). Note that the signature I wrote above takes an explicit context (presumably some sort of mapping of available variables), which removes the need for supporting arbitrary attribute access in the first place, and is more explicit to boot. There is no reason other languages or libraries could not implement a function with the same degree of safety, although their type systems might not be able to guarantee its safety. In fact I would wager that such pitfalls are only likely to be found in dynamic languages, which support runtime eval (which is, after all, the root of the described problem).
- dlbucci 10y agoI do love writing python, but it's pretty shocking when I find out you can write something like `event.__init__.__globals__[CONFIG][SECRET_KEY]`. That language just does not care about privacy or information hiding at all, I guess.
- deleted 10y ago[deleted]
- hakanderyal 10y agoPython's general mentality can be summed as "We are all consenting adults here". If you want to access some part of program or data, you generally can, but the burden of not breaking anything while messing with internals is on you. This is a great power, but also can become an unlimited source of bugs.
- deleted 10y ago[deleted]
- brianwawok 10y agoYou can do the same thing in Java with reflection. Don't let users run untrusted code. Full stop. If you need templating use a sandbox like jinja2.
- Godel_unicode 10y agoThe discovered vulnerability manifested itself in jinja2. Edit: release notes https://www.palletsprojects.com/blog/jinja-281-released/ https://www.palletsprojects.com/blog/jinja-281-released/
- brianwawok 10y agoExactly it's already fixed. It's in the article.
- jsjohnst 10y agoI'd love to hear your perspectives on Ruby if this bothers you that much.
- Animats 10y agoNo, Rust does not have the ability to access any variable in the program via a format string. Rust has this: format!("{argument}", argument = "test"); // => "test" That's just named arguments to the format. Also, that's a macro; it's expanded at compile time. Python's approach is lame. It should have used something with a limited list of named arguments, or maybe a dict.
- gizmo686 10y agoNeither does python. The problem being discussed here is that the format string can access and attribute of the object passed in. From PEP3101 [0]: " Unlike some other programming languages, you cannot embed arbitrary expressions in format strings. This is by design - the types of expressions that you can use is deliberately limited. Only two operators are supported: the '.' (getattr) operator, and the '[]' (getitem) operator. The reason for allowing these operators is that they don't normally have side effects in non-pathological code." [0] https://www.python.org/dev/peps/pep-3101/ https://www.python.org/dev/peps/pep-3101/
- echelon 10y ago> Neither does python. I would argue that being able to access any global value is a sufficient enough concern.
- deleted 10y ago[deleted]
- zardeh 10y agoThe security issue here isn't with f strings, which oddly enough, don't have this flaw (because they are literals only). This has to do with new style string formatting, which does not have access to globals.
- detaro 10y ago> don't normally have side effects in non-pathological code so, "be careful with it if you can't make sure code isn't non-pathological or friendly", what the article says.
- ak217 10y agoUntil now I thought the new features were confined to `f""` format strings. Edit: As correctly pointed out, this feature has been around since the introduction of str.format(). So this warning applies to all Python versions.
- DasIch 10y agoThis is not a new feature.
- tedunangst 10y agoThis is not a new feature.
- throwaway91111 10y agoThis existed in python 2.6; it's not a new feature by any means.
- tedunangst 10y agoMaybe a more fully worked example is needed. You're making a blog hosting service as a service service. Bloggers have different ideas about what page titles should be. Post Title Blog Name: Post Title Blog Name - Post Title Post Title - Blog Name Blog Name ----embdash---- Post Title ~~~ xXx Post Title xXx ~~~ It's a little overwhelming to put every possibility in a dropdown, so you allow the user to specify a format string. title = userformats.title.fmt(post) This doesn't look so very dangerous. And then the user can say "{post.title} - {post.blog.title}" "{post.title}: Another fine post by "{post.author}" "~~~ xXx {post.blog.__init__.dbconnection.__keys__.password} xXx ~~~" And then oops.
- bubblesocks 10y agoIsn't ingesting user input directly considered a bad idea all around though?
- foolfoolz 10y agoingesting raw user input is good emitting raw user input is bad
- peller 10y agoIsn't SQL injection caused by ingesting raw user input though? Seems to me you always have to be careful with user-supplied data.
- meowface 10y ago"Ingesting raw user input is good if you only use it to interface with other systems that provide a way to separate data from instructions or at least escape strings."
- foolfoolz 10y agosql injection is commonly caused by combining your query with its related data parameters in unsafe ways. you are emitting raw user input you received to another program, the database, it's your responsibility to give this to the DB safely. you still have to be careful, and when you follow all the right best practices you can safely ingest raw user input. I've worked at a company that escaped user input before inserting into the DB. it's a horrible nightmare I don't think anyone should have to experience.
- Cyph0n 10y agoScala's solution to string formatting is the best out there imo. You can put basically any code into the format placeholder. http://docs.scala-lang.org/overviews/core/string-interpolation.html http://docs.scala-lang.org/overviews/core/string-interpolati...
- UnoriginalGuy 10y agoI don't really understand what you mean. They're pointing out unintended consequences of allowing arbitrary string formats. You're linking to one you call "better" which would allow even more powerful misuse of arbitrary string formats. And while Scala's string formatting looks cool, it hardly seems like your reply applies to the topic/thread. In fact ironically it seems to point out that this would be an even bigger issue in Scala.
- merb 10y ago> be an even bigger issue in Scala. not with string interpolation in scala. it only works on compile time so it's not possible to actually take user input strings (without runtime reflection of course) > it hardly seems like your reply applies to the topic/thread yes string interpolation (scala) has less todo with it. it can't be compared to the python one.
- kazinator 10y agoWhat the Python code is doing is arguably a form of run-time reflection. Format strings contain embedded variables and expression, and the format function evaluates them: hence reflection. Format strings occurring as literals in Python source could be treated at compile time, like in Scala.
- icebraining 10y agoPython's string interpolation only works on compile time as well¹, this isn't interpolation, it's a runtime method in the string class, which Scala also has, in fact, with the same name (.format). ¹ (compilation in CPython generally occurs during initialization of the execution, though you can pre-compile files manually)
- agumonkey 10y agoOnce again strings and io formatting making fun of us.
- agentgt 10y agoThis is actually a problem with a lot of languages that allow runtime template like String interpolation. For example Groovy on the JVM has GStrings which one can do fairly nasty things. As well it actually is fairly hard to lock down most of the template languages on the JVM for user templates. (If you are going to allow user templates I recommend one of the Java Mustache-like implementations).
- innocentoldguy 10y agoWhile I agree this is a possible attack vector, I think it is extremely unlikely, at least in the localization realm, for several reasons: 1. The localization company should never know what programming language you're using. 2. You shouldn't give localization companies direct access to your internal strings. More than hacking your code, they're almost guaranteed to screw the formatting up. 3. Typically, translators are hired for their native language abilities, and not for their technical prowess. I've met precious few who knew how to open a text editor, let alone hack your product via its strings. I worked with Python and I18N/L10N for about 15 years. The way I always handled localization was to parse all our strings into a PostgreSQL database, and then provide a web interface for translators to do their work. This interface provided translators with the full-context of the strings they were translating, which internal strings often don't, prevented the inclusion of certain characters and keywords, and kept the translators from screwing up the formatting. By doing it this way, we got much better translations, and our internal strings were never out of our control.
- the_mitsuhiko 10y agoMost internationalization these days goes through things like transifex. Becoming a translator on a specific project is not completely unlikely if you want to target it.
- bubblesocks 10y agoI find it depends on the project and the localization company. I've used big ones, like Lingotek and LION (or whatever they're called now), but there are still a lot of small companies that do things manually, and for a lot less.
- kazinator 10y agoSane Lisp approach: provide easily analyzable target syntax. This is the TXR Lisp interactive listener of TXR 163. Use the :quit command or type Ctrl-D on empty line to exit. 1> (defvar foo 42) foo 2> `@(list foo) ... @foo` "42 ... 42" What is that backticked literal? Let's quote it: 3> '`@(list foo) ... @foo` `@(list foo) ... @foo` Hmm, prints back in same form. Probably syntactic sugar for a list; what is in the car? cdr? 4> (car '`@(list foo) ... @foo`) sys:quasi 5> (cdr '`@(list foo) ... @foo`) ((list foo) " ... " @foo) What if we type in this syntax ourselves: 6> '(sys:quasi (list foo) " ... " (sys:var foo)) `@(list foo) ... @foo` Prints as the notation! To sandbox this, we just have to walk a list and enforce some rule. For instance, the rule might be that all elements after sys:quasi must be string object or else (sys:var sym) forms, where sym is a symbol on some allowed list. Thus (list foo) would be banned. A custom interpreter which calculates the output string while enforcing the check is trivial to write as a one-liner. Of course if your program just eval such an untrusted quasiliteral, it has access to the dynamic/global evironment: `Mouhaha! @(file-get-string "/etc/passwd")` Very convenient for some attacker. :)
- shakna 10y agoIf you're LISP has first class environments, then this becomes much easier, and safer. (eval (read (get-some-user-string)) (let [ (post-title post-title) (post-date post-date) (post-author post-author) (fmt string-append) ] (null-environment))) eval is only executed in the context of some environment, and the only things it has access to are some strings, and a string concatenation procedure. It doesn't have access to lambda, quasiliterals or quotes, or anything of the like. I do have code like this in production, because eval only looks up within it's environment... Which is empty apart from the bound let. No closure access.
- swift 10y agoLua has this feature as well and it's fantastically useful for this kind of use-case, or more generally anywhere where you want to allow scripting.
- ryanmccullagh 10y agoThis is the same principle for not passing user input to the first argument of `printf(3)` in C. Coming from C, I would never allow the user control of the format string if I were writing code.
- cyphar 10y agoThe reasoning why printf strings in C are unsafe is not applicable in python though. You cannot access memory directly, and %n doesn't exist (nor other other fun POSIX requirements that write to random places on the stack). In fact, Python's C-style formatting strings are immune to this problem (even the extended form that allows you to name arguments).
- jlarocco 10y agoThat's pointless nitpicking. The point is that user specified format strings allow the user to access things they shouldn't be able to access. It's the same in every language that has format strings.
- cyphar 10y ago> That's pointless nitpicking. The point is that user specified format strings allow the user to access things they shouldn't be able to access. It's the same in every language that has format strings. It's not nitpicking. Go's format strings don't allow you to access stuff you shouldn't be able to. In fact, Go's templating language[1] doesn't allow you to access anything that you haven't explicitly provided to the template instance. Allowing users to format text is not an impossible problem. It's the fault of language creators that they don't make this easier. [1]: https://golang.org/pkg/text/template/ https://golang.org/pkg/text/template/
- nhumrich 10y agoThis article makes you thing the new f-strings will be discussed. What is really being discussed is 'string'.format(), which isn't new in any way.
- eternauta3k 10y agoIt's referred to in the documentation as 'New-style', with the old style being the % operator.
- tedmiston 10y agoYes but quite confusing when we have another new style string format actively being discussed in recent months vs this "new-style formatting" which has existed since 2006. It's much clearer to just refer to it as str.format.
- halomru 10y agoIt's what invariably happens when you incorporate words like "new", "super" or "high definition" in the name of something. I'm still hoping that at some point our industry learns that lesson.
- smitherfield 10y agoIt does feel a little (unintentionally?) clickbaity.
- anderskaseorg 10y agoThe proposed idea of relying on undocumented internals and blacklisting attribute names to securely sandbox formatting strings is _really_ _dangerous_. Never do that in production code! Language expansions could render your sandbox unsafe at any time. You can write your own safe formatting engine in much less code. def safe_format(fmt, **args): return re.sub(r'\{([^{}]*)\}', lambda m: args.get(m.group(1)), fmt) safe_format('{foo} {bar}', foo='Hello', bar='world') # Hello world Add bells and whistles as desired.
- tedunangst 10y agoThe first bell added is likely to be field access.
- cyphar 10y agoWhich immediately becomes a security risk because classes can overload __getattr__ and __getitem__. So now you're executing random code. Of course, %s does the same thing (__str__) but at least %s doesn't take any arguments.
- sirclueless 10y agoThe assumption behind string.format and friends is that people who override __getattr__ and __getitem__ know what they're doing. Which is probably a reasonably safe tradeoff in order to allow attribute access. I'm not worried about executing arbitrary code, because it's not user-supplied and it's normal for a templating library to access attributes of objects you pass to it (if it was unsafe you shouldn't have done that). This is more of a snafu with the number of sensitive and powerful attributes available by default on Python objects.
- anderskaseorg 10y agoRead the article. The danger is not __getattr__ and __getitem__ but rather things like event.__init__.__globals__[CONFIG][SECRET_KEY]. There is no such thing as an object where arbitrary attribute access is safe, because every object has a constructor function and every function references the globals dict.
- peterwaller 10y agoReminds me of the turing complete interpreter lurking in libc's printf. https://github.com/HexHive/printbf https://github.com/HexHive/printbf
- stevebmark 10y agoDoesn't this apply to any language that has string interpolation, Ruby, Python, Javascript Perl, etc? And doesn't it not really matter, because it's not realistic to use a dynamic string template in a program?
- Manishearth 10y agoRuby format strings are like old-style Python format strings. They only map to names or positional arguments. Perl also just maps to positional arguments, like printf. The most you can do is mess up the string by moving around positional args. JS format strings get evaluated at "compile time", as do Ruby interpolation literals. You cannot have a string with format specifiers in it, you can have a backtick-delimited string literal with format specifiers in it. The formatting is done when the interpreter encounters this literal; the user can't input such a value since these literals aren't a different kind of value. > because it's not realistic Done reasonably often for i18n. Not a good i18n solution, but for many it's good enough :|
- libeclipse 10y agoI tend to use the old style ("%s" % "lel"). Just wondering, does this affect that too?
- antsar 10y agoAttribute access doesn't seem to work with old-style formatting, so I would think it's safe. >>> class foo: ... def __init__(self, **kwargs): ... self.__dict__.update(kwargs) ... >>> my_foo = foo(secret="bazinga") >>> >>> "%(foo)s" % {'foo': my_foo} '<__main__.foo instance at 0x1101d23f8>' >>> >>> "%(foo.secret)s" % {'foo': my_foo} Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 'foo.secret'
- bubblesocks 10y agoThe nice thing about the new format is that you can reverse the order of the variables inside the string, which is nice when you're localizing strings into languages with a reversed sentence structures, like Japanese. I haven't done a ton of Python, but the old style won't let you do that, will it?
- Franciscouzo 10y agoReversing the order is not possible, but you can use a dict instead: >>> "%(foo)s" % {"foo": "bar"} 'bar'
- libeclipse 10y agoHeyyyyyy I did not know that. TIL, thank you.
- dozzie 10y agoFunny. I thought Python implemented all the quirks of the original printf() formatting, including using arguments at specific positions. Apparently it didn't (and arguably it's a good thing). $ printf '%2$s %2$s %1$s\n' aaa bbb bbb bbb aaa
- 10y ago
- foota 10y agoWow, I had no idea you could access attributes from within format strings.
- Manishearth 10y agoYeah, I never liked Python's "new-style" format because of this. It didn't occur to me that you could use field access to access globals (never done enough Python metaprogramming to mess with the reflection stuff), but I was afraid of arbitrary getters being invoked. In general I'm very wary of runtime string formatting. Strings tend to be untrusted input with a large degree of freedom. format strings are almost always known at compile time (and more trustworthy). If your interpolation system is more than simply mapping keys to values or positions, you should probably restrict it to compile time. Feel free to expose a harder-to-use runtime API. Rust has compile-time format strings, for example. They're not as powerful as `str.format`, but they could be without there being security issues. JS has a different syntax for format literals. Regular strings cannot be "formatted", you must specify a string literal with backticks and that gets converted to a string value when the interpreter gets there. These literals can execute arbitrary code, but since it's just literals there's no way for an untrusted string to get in there. One main use case for runtime string formatting is i18n. But that really should use a different solution. Most string formatting APIs are geared for programmer convenience -- the programmer is writing the code and the string. The scales shift for translators, who are only writing the strings. They don't need things like field access and stuff. Besides, most string formatting APIs are inadequate for i18n. Not if you want to handle stuff like pluralization (http://mlocati.github.io/cldr-to-gettext-plural-rules/ http://mlocati.github.io/cldr-to-gettext-plural-rules/). Another use case is template engines and stuff like that. In that case, field access is useful, but you probably should exert more control on these things (which is exactly what jinja2 seems to be doing here) At one point I toyed with an idea for a super-type-safe template engine in Rust. It would validate the templates at compile time, and additionally ensure that the right types are in the right places. For example, it could ensure that strings that get interpolated with the HTML are either "trusted", escaped, or otherwise XSS-sanitized (using the type system to mark such types). Similarly, url attributes (href, etc) can only have URLs that have been checked for `javascript:`. Never got around to writing it, sadly.
- yladiz 10y agoSo is the issue that it's a problem to let the format string be controlled arbitrarily? It's good to warn users around it because some may not know the dangers, but in general you don't trust user input, so I don't really see this as something you need to build something custom around, rather just be careful and follow good practices. It should be understood not to pass user data directly to internal code without sanitizing it. Proposing some custom code that uses undocumented internal features is overkill and also dangerous since things that aren't documented/internal can change suddenly.
- geofft 10y ago> It should be understood not to pass user data directly to internal code without sanitizing it. But how do you sanitize it? If you hadn't read this article and were told to sanitize a format string, how would you implement that? Unless you know exactly what the problem is, you're basically reduced to writing your own string formatter. (Though see 'anderskaseorg's example, it's a lot less code if you do.)
- tedmiston 10y agoI think asking how to sanitize it is really asking the wrong question. We shouldn't sanitize a format string from a user in the first place. We should get input and generate a format string rather than allow one to be supplied "raw". For example, using re.sub as another commenter pointed out. An even more Pythonic solution would be to use the built-in string.Template [1] which at a glance appears to be safe against this kind of attack (but I haven't tested in code). I'm not sure why Armin didn't mention string.Template in his post other than that it is much more restricted than str.format, however that kind of seems like the point to me. [1]: https://docs.python.org/3.4/library/string.html#template-strings https://docs.python.org/3.4/library/string.html#template-str...
- daveguy 10y agoSince 2.4: https://docs.python.org/2/library/string.html#template-strings https://docs.python.org/2/library/string.html#template-strin...
- unscaled 10y agoI was always puzzled by Python's insistence to forego string interpolation until the latest version. Runtime string formatting, even if done safely (e.g. .NET's String.Format() which doesn't have property access AFAIK), can still cause unexpected exceptions at the very least, and suffers from inferior performance.
- Pxtl 10y agoOdd they reference c# since c# does not have this feature in its traditional format strings - they don't allow you to access arbitrary members. C# recently added string interpolation, which does allow arbitrary code, but string interpolation itself is compiled C# code and can't be stored like a format string. Personally I use mustache when i need format-string-like-behavior from semi-trusted users.
- chrisdone 10y ago> So what do you do if you do need to let someone else provide format strings? You can use the somewhat undocumented internals to change the behavior. Well, that's just sloppy and shame on you for exposing programming internals to a user in a service. What did you expect? Write your own format string parser and stop being lazy.
- bbcbasic 10y agoPure functions ftw
- babyrainbow 10y agoDiscussion when this pep when it was accepted and people handwaving and down voting arguments against it. https://www.reddit.com/r/Python/comments/3k6qi8/pep_498_approved/ https://www.reddit.com/r/Python/comments/3k6qi8/pep_498_appr...
- tedunangst 10y agoAnd again, not the feature under discussion.
- babyrainbow 10y agoNot sure what you mean...
- tzury 10y agoIf within the context that parse and execute user input, sensitive data is available, then what this has to do with New-Style String Format? I mean, {event.__init__.__globals__[CONFIG][SECRET_KEY]} Shall not be available to any function call, whatsoever!
- FeepingCreature 10y agoIf you are making a new language, the easy way to fix this is make formatting a property of your strings, not a runtime function. Ie. instead of having "foo {bar}".format(bar=bar), have "foo {bar}" be equivalent to "foo "+bar. This sidesteps the problem because only literal strings are formatted.
- angusp 10y agoIt's a well known C pattern that you should never trust a user supplied format string, E.g. printf(arg) vs printf("%s", arg). The same applies here
- iamsk 10y agoThis should be fixed built-in, like how sql injection fixed.