11 ms·
All things considered, this is pretty cool. Basically, this replaces db.execute("QUERY WHERE name = ?", (name,)) with db.execute(t"QUERY WHERE name =
by serbuvlad 1y ago
All things considered, this is pretty cool. Basically, this replaces
db.execute("QUERY WHERE name = ?", (name,))
with
db.execute(t"QUERY WHERE name = {name}")
Does the benefit from this syntactic sugar outweigh the added complexity of a new language feature? I think it does in this case for two reasons:
1. Allowing library developers to do whatever they want with {} expansions is a good thing, and will probably spawn some good uses.
2. Generalizing template syntax across a language, so that all libraries solve this problem in the same way, is probably a good thing.
- pinoy420 1y agoNow instead of being explicit all it takes is someone unfamiliar with t strings (which will be almost everyone - still few know about f strings and their formatting capabilities) to use an f instead and you are in for a bad time.
- falcor84 1y agoThat is an issue, but essentially it boils down to the existing risk of unknowledgeable people not escaping untrusted inputs. The solution should be more education and better tooling (linters, SAST), and t-strings are likely to help with both.
- masklinn 1y agot-strings allow building APIs which don't accept strings at all (or require some sort of opt-in), and will always error on such. That's the boon. Having to write cr.execute(t"...") even when there's nothing to format in is not a big imposition.
- deleted 1y ago[deleted]
- mcintyre1994 1y agoAny sane library will just error when you pass a string to a function that expects a template though. And that library will have types too so your IDE tells you before you get that far.
- Dx5IQ 1y agoSuch library functions tend to also accept a string as a valid input. E.g. db.execute from the GP usually works with strings to allow non-parametrized SQL queries.
- kccqzy 1y agoThe library should just refuse strings. If a non parametrized query is desired, it could require the user to supply a t-string with no {}.
- dragonwriter 1y ago> Such library functions tend to also accept a string as a valid input. Also? They tend only to accept a string (possibly with some additional arguments, if there is an in-library way to handle parameterization) as input, because Template literally hasn't been an option. New APIs designed with Template available will look different.
- _Algernon_ 1y agoThis would break backwardcompatibility pretty hard. In many cases it may not be worth it.
- eichin 1y agoBut now at least the language has the necessary rope (and an opportunity for a cultural push to insist on it.)
- hombre_fatal 1y agoJavascript already has prior art here. A library can extend an existing database library like 'pg' so that PgClient#query() and PgPool#query() require string template statements. That way 'pg' can continue working with strings, and people who want nice templated strings can use the small extension library, and the small extension library makes it impossible to accidentally pass strings into the query functions.
- sanderjd 1y agoNo, because they don't return a string, so good library authors will raise a type error when that happens, for exactly this reason.
- hackrmn 1y agoI suppose lack of overlap in the "interface surface" (attributes, including callables) between `str` and `Template` should nip the kind of issue in the bud -- being passed a `Template` and needing to actually "instantiate" it -- accessing `strings` and `values` attributes on the passed object, will likely fail at runtime when attempted on a string someone passed instead (e.g. confusing a `t`-string with an `f`-string)?
- Tenoke 1y agoI don't see what it adds over f-string in that example?
- teruakohatu 1y agoIf I pass an f-string to a method, it just sees a string. If I pass a t-string the method can decide how to process the t-string.
- evertedsphere 1y agosafety against sql injection
- ds_ 1y agoThe execute function can recognize it as a t-string and prevent SQL injection if the name is coming from user input. f-strings immediately evaluate to a string, whereas t-strings evaluate to a template object which requires further processing to turn it into a string.
- Tenoke 1y agoThen the useful part is the extra execute function you have to write (it's not just a substitute like in the comment) and an extra function can confirm the safety of a value going into a f-string just as well. I get the general case, but even then it seems like an implicit anti-pattern over doing db.execute(f"QUERY WHERE name = {safe(name)}")
- ubercore 1y agoProblem with that example is where do you get `safe`? Passing a template into `db.execute` lets the `db` instance handle safety specifically for the backend it's connected to. Otherwise, you'd need to create a `safe` function with a db connection to properly sanitize a string. And further, if `safe` just returns a string, you still lose out on the ability for `db.execute` to pass the parameter a different way -- you've lost the information that a variable is being interpolated into the string.
- NewEntryHN 1y agoAssuming you also need to format non-values in the SQL (e.g. column names), how does the `execute` function is supposed to make the difference between stuff that should be formatted in the string vs a parametrized value?
- masklinn 1y agoSame as currently: the library provides some sort of `Identifier` wrapper you can apply to those.
- NewEntryHN 1y agoFair enough. It would be nice if Python allowed to customize the formatting options after `:` This way you could encode such identifier directly in the t-string variable rather than with some "out-of-band" logic.
- mcintyre1994 1y agoThe article does mention that the function receiving the template has access to those formatting options for each interpolation, so presumably you could abuse the ones that are available for that purpose?
- masklinn 1y ago> Fair enough. It would be nice if Python allowed to customize the formatting options after `:` It does, the `Interpolation` object contains an arbitrary `format_spec` string: https://peps.python.org/pep-0750/#the-interpolation-type https://peps.python.org/pep-0750/#the-interpolation-type However I think using the format spec that way would be dubious and risky, because it makes the sink responsible for whitelisting values, and that means any processing between the source and sink becomes a major risk. It's the same issue as HTML templates providing `raw` output, now you have to know to audit any modification to the upstream values which end there, which is a lot harder to do than when "raw markup" values are reified. > rather than with some "out-of-band" logic. It's the opposite, moving it to the format spec is out of band because it's not attached to values, it just says "whatever value is here is safe", which is generally not true. Unless you use the format spec as a way to signal that a term should use identifier escaping rules rather than value escaping rules (something only the sink knows), and an `Identifier` wrapper remains a way to bypass that.
- amelius 1y agoOne thing it misses is compile-time checks for e.g. the format spec.
- karamanolev 1y agoDoesn't all of Python miss that, having (close to) no compile time?
- amelius 1y agoPython does some checks before it runs code. E.g.: print("hello") def f(): nonlocal foo gives: SyntaxError: no binding for nonlocal 'foo' found before printing hello, and note that f() wasn't even called.
- nomel 1y agoI think it's just giving an error because a valid AST can't be made, which means valid bytecode can't be made. "<word> <word>" is only valid syntax if one is a reserved word. `nonlocal(foo)` is just fine, of course.
- pansa2 1y ago> "<word> <word>" is only valid syntax if one is a reserved word. `nonlocal` is a keyword
- zahlman 1y agoNo, it gives an error because `nonlocal foo` requests that the name `foo` be looked up in a closure, but `f` doesn't have such a closure (the `foo` defined outside the function is global instead). `nonlocal` is the same sort of keyword as `global` but for enclosing functions instead of the global namespace; see also https://stackoverflow.com/questions/1261875 https://stackoverflow.com/questions/1261875 .
- nomel 1y ago
- rwmj 1y agoI did a safe OCaml implementation of this about 20 years ago, the latest version being here: https://github.com/darioteixeira/pgocaml https://github.com/darioteixeira/pgocaml Note that the variables are safely and correctly interpolated at compile time. And it's type checked across the boundary too, by checking (at compile time) the column types with the live database.
- tasuki 1y agoYes, what you did is strictly more powerful than what the Python people did. And you did it 20 years ago. Well done, have an upvote. And yet, here we are in 2025 with Python popularity growing unstoppably and (approximately) no one caring about OCaml (and all the other languages better than Python). It makes me sad.
- sanderjd 1y agoNetwork effects are a beast! But my two cents is that we're pretty lucky it's python that has taken off like a rocket. It's not my favorite language, but there are far worse that it could have been.
- psychoslave 1y agoYou mean like Cobol? Oh wait!
- rwmj 1y agoI'm switching between C, OCaml, Python, bash & Rust roughly equally every day (to a lesser extent, Perl as well). Not everything is what gets on the front page of HN.
- skeledrew 1y agoIt's interesting how the majority has explicitly chosen NOT to use the "better" languages. Is the majority really that bad in their judgment? Or is it that "better" is actually defined by adoption over time?
- tetha 1y agoOr you could use this in a library like sh with sh(t"stat {some_file}") With t-strings you could run proper escaping over the contents of `some_file` before passing it to a shell. I'd have to take a look at the order things happen in shell, but you might even be able to increase security/foot-gun-potential a little bit here by turning this into something like `stat "$( base64 -d [base64 encoded content of some_file] )"`.
- nhumrich 1y agoYou should check out PEP 787
- tetha 1y agoHmm, PEP-787 has some interesting discussions around it. I'll have to sort my thoughts on these aspects a bit.
- pauleveritt 1y agoWe really should just point most of these comments at that PEP. Thanks for getting it out so fast.
- zahlman 1y agoOh! I missed this one because I've been looking specifically at the Packaging forum rather than the PEPs forum. This looks like a brilliant use case. (I'm aiming for wide compatibility - back to 3.6 - with my current projects, but I look forward to trying this out if and when it's accepted and implemented.) Now if only the overall `subprocess` interface weren't so complex....
- Flimm 1y agoPEP 787 – Safer subprocess usage using t-strings https://peps.python.org/pep-0787/ https://peps.python.org/pep-0787/
- dhruvrajvanshi 1y agoNot Python but this is exactly the idea behind zx https://github.com/google/zx https://github.com/google/zx
- mikeholler 1y agoA potential concern is how close this looks to the pattern they're trying to override. db.execute(f"QUERY WHERE name = {name}") versus db.execute(t"QUERY WHERE name = {name}")
- fzzzy 1y agoBut won't the f string version fail loudly because there's no name parameter?
- benwilber0 1y agothe {name} parameter is in the locals() dict like it always is
- fzzzy 1y agoGood point. Perhaps the database api could refuse strings and require Templates.
- bshacklett 1y agoThat’s a big breaking change around a brand new feature. I’m sure it could be done well, but it gives me the shivers.
- daedrdev 1y agomuch better would be execute_template(t"...")
- dragonwriter 1y agoYou add a new API that takes templates only leaving the existing API in place. You (some releases later) deprecate the string API. You (some releases later, with clear advance warning of when it is coming) actually remove the deprecated API. "It's a big breaking change around a brand new feature", yeah, so you don't break anything around a brand new feature, it's not like this kind of transition is a new concept.
- benwilber0 1y agoAren't there other benefits to server-side parameter binding besides just SQL-injection safety? For instance, using PG's extended protocol (binary) instead of just raw SQL strings. Caching parameterized prepared statements, etc. Also: db.execute(t"QUERY WHERE name = {name}") Is dangerously close to: db.execute(f"QUERY WHERE name = {name}") A single character difference and now you've just made yourself trivially injectible. I don't think this new format specifier is in any way applicable to SQL queries.
- deleted 1y ago[deleted]
- VWWHFSfQ 1y ago> I don't think this new format specifier is in any way applicable to SQL queries. Agree. And the mere presence of such a feature will trigger endless foot-gunning across the Python database ecosystem.
- masklinn 1y ago> Aren't there other benefits to server-side parameter binding besides just SQL-injection safety? For instance, using PG's extended protocol (binary) instead of just raw SQL strings. Caching parameterized prepared statements, etc. All of which can be implemented on top of template strings. > A single character difference and now you've just made yourself trivially injectible. It's not just a one character difference, it's a different type. So `db.execute` can reject strings both statically and dynamically. > I don't think Definitely true. > this new format specifier is in any way applicable to SQL queries. It's literally one of PEP 750's motivations.
- VWWHFSfQ 1y ago> It's literally one of PEP 750's motivations. Python is notorious for misguided motivations. We're not "appealing to authority" here. We're free to point out when things are goofy.
- willcipriano 1y agofrom string.templatelib import Template def execute(query: Template) Should allow for static analysis to prevent this issue if you run mypy as part of your pr process. That would be in addition to doing any runtime checks.
- VWWHFSfQ 1y ago> Allowing library developers to do whatever they want with {} expansions is a good thing, and will probably spawn some good uses. I completely disagree with this. Look what happened to Log4J when it was given similar freedoms.
- serbuvlad 1y agoI think this would have solved the log4j vulnerability, no? As I understand it, log4j allowed malicious ${} expansion in any string passed to logging functions. So logging user generated code at all would be a security hole. But Python's t-strings purposely _do not_ expand user code, they only expand the string literal.
- int_19h 1y agoPython is not the first one to get this feature. It's been present in JS for some time now, and before that in C# (not sure if that's the origin or they also borrowed it from somewhere). Python adopted it based in part on successful experience in those other languages.
- serbuvlad 1y agoThat's really cool. I don't use JS or C#, so I wasn't aware of this, but it's a good idea.
- zahlman 1y ago3. It prevents the developer from trying db.execute(f"QUERY WHERE name = {name}") or db.execute("QUERY WHERE name = %s" % name, ()) or other ways of manually interpolating the string - because `db.execute` can flag a `TypeError` if given a string (no matter how it was constructed) rather than a `Template` instance.
- jimwhite 1y agoYes and your example is the hero case because it isn't just sugar. A t-string implementation for SQL will of course escape the values which is a common security issue. https://xkcd.com/327/ https://xkcd.com/327/
- hombre_fatal 1y agoNo, a t-string returns a Template which is basically { strings: str[], values: any[] }. So you would write db.execute(template) to turn template t"... where id = {id}" into a parameterized structure like ("... where id = ?", id).