7 ms·

PEP 750: Tag Strings for Writing Domain-Specific Languages

96 points by lumpa 2y ago

formerly_proven 2y ago
Yikes. Don't get me wrong, I totally understand the reasoning why this would be useful (though I violently disagree with the idea of deferring the evaluation of the contained expressions), but it's also so very kitchensinky and adds so little over just calling a function (which doesn't require a 20-page explainer, as everyone already knows how function calls work). It also promotes using what looks like string interpolation (and what might be string interpolation, you can't tell at the "call site") for things which we know string interpolation is the wrong tool. The API also seems really, I dunno, weird to me. The string is split around interpolations and verbatim portions result in one argument, which is "string-like", while interpolations become four-tuple-like (one of which is a lambda, which you call to perform the deferred interpolation). This seems really awkward to me for building stuff like the suggested use cases of XML/HTML or SQL templating. Also the scoping rules of this are a special case which doesn't appear in regular Python code so far: "The use of annotation scope means it’s not possible to fully desugar interpolations into Python code. Instead it’s as if one is writing interpolation_lambda: tag, not lambda: tag, where a hypothetical interpolation_lambda keyword variant uses annotation scope instead of the standard function scope." -- i.e. it's "as if you wrapped all interpolation expressions in a lambda: <expr>, except it uses different scoping rules".
- masklinn 2y ago
  > This seems really awkward to me for building stuff like the suggested use cases of XML/HTML or SQL templating. Compared to what? At the end of the day you're still doing string formatting, if you want parsing, then you'd feed the item into a parser, which this doesn't preclude. The interface sounds a lot better than JS's anyway, as that completely separates the literal strings and the interpolations so you have to re-intersperse them which is muggy. > interpolations become four-tuple-like They become an Interpolation object, which can be unpacked if you find that more convenient, but you can access the members if you prefer: - 0 is getvalue is the callable to retrieve the evaluated expression - 1 is expr is the raw text form of the expression - 2 is conv is the !conversion tag (s, r, or a) - 3 is format_spec
  - jimbaker 2y ago
    The intent here is to support the following approach for tag function authors: 1. Parse to an AST, generally using an off-the-shelf parser. In practice, it's possible to rewrite interpolations with a placeholder suitable for a given language, eg x$Nx for HTML. Of course if that doesn't actually work, you might have to write/modify an existing parser. Hopefully we can cleverly avoid this extra work. 2. Walk/compile the AST, filling interpolations, but taking in account the context. This can for example take the form of building appropriate query strings that avoid Bobby Tables SQL injection, whether by mapping to SQL placeholders or with appropriate quoting (such as for a column or table name). 3. Memoize these steps, much as we see with builtin DSLs in Python, like the re module; see https://github.com/python/cpython/blob/3.12/Lib/re/__init__.py#L331 https://github.com/python/cpython/blob/3.12/Lib/re/__init__.... We do plan to make this easier/faster by supporting getting the original source of the template string (Template.source), vs the *args approach we show in the PEP at the start of this discussion (this will become Template.args instead; Template here is the proposed protocol of the object passed in). Related is my post here: https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408/108 https://discuss.python.org/t/pep-750-tag-strings-for-writing...
dmart 2y ago
I have to admit that at first glance I don’t like this. These seem to be essentially normal str -> Any functions, with some naming limitations due to the existing string prefixes special-cased in the language. I don’t feel like adding this additional complexity is worth being able to save two parentheses per function call.
- Epa095 2y ago
  This was my first thought as well. But an important difference is that the arguments are not eagerly evaluated, but they are passed as lambdas which can be evaluated if desired. This means that it can be used for example in log messages (if you don't want to evaluate the string at the wrong log levels). But is it worth it for that? Idk.
  - masklinn 2y ago
    Even if eager evaluation it's already a very compelling way of managing basically every lightweight "templating" for safety: e.g. embedded dynamic HTML or SQL. `markupsafe` is great, but it's way too easy to perform formatting before calling it, especially with f-strings. That f-strings were "static" was by far my biggest criticism of it, given how useful I find JS's template strings. And this proposal seems like a straight up better version of template strings: - the static strings and interpolations are not split and don't have to be awkwardly re-interpsersed which I've never found 100% trouble and 0% utility - the lazy evaluation means it can be used for things like logging (which really want lazy evaluation), or meta-programmation (because you can introspect the callables, and you get the expression text)
    - codethief 2y ago
      > - the lazy evaluation means it can be used for things like logging (which really want lazy evaluation) Could you elaborate? I would find it rather surprising if my log messages don't contain the data at the very moment I invoke the logger.
      - 2y ago
ziml77 2y ago
It seems the purpose of this proposal is to have a way to essentially have custom string interpolation. I don't think that's necessarily a bad idea on its own, but this syntax feels out of place to me. Instead, why not add a single new string prefix, like "l" for "lazy"? So, f"hello {name}" would immediately format it while l"hello {name}" would produce an object which contains a template and the captured variables. Then their example would be called like: greet(l"hello {name}").
TwentyPosts 2y ago
This like a bad idea on the first glance? Maybe I don't get the whole pitch here? It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call. The laziness might be interesting, but I feel like "lazy strings" might be all that's needed here. Laziness and validation (or custom string formatting logic) are separate concerns and should be separated.
- masklinn 2y ago
  > It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call. That could also be leveraged at f-strings themselves. > Laziness and validation (or custom string formatting logic) are separate concerns and should be separated. In which case the one to move out is the laziness not the customised interpolation. Because the latter is the one that's necessary for safer dynamic SQL or HTML or whatever.
behnamoh 2y ago
I want this in Python: https://codecodeship.com/blog/2024-06-03-curl_req https://codecodeship.com/blog/2024-06-03-curl_req From the article: """ ~CURL[curl https://catfact.ninja/fact] |> Req.request!() This is actual code; you can run this. It will convert the curl command into a Req request and you will get a response back. This is really great, because we have been able to increase the expressiveness of the language. """
- zelphirkalt 2y ago
  And somewhere there is the logic implemented to know what ~CURL is and what needs to happen with the part in square brackets, how it needs to be translated into a request call. And if that is merely an arguments equals arguments thingy, then it would be kind of useless, since there are usually more things one specifies when making a request from inside Python. Things like headers or whether or not to verify TLS certs, and that is not a 1 to 1 mapping between curl arguments and a requests call. So I remain doubtful, as long as no way is shown to me, how the user of the language can define this syntactic abstraction themselves, which is unlikely to happen or exist in Python.
Too 2y ago
Looks good. Would have been nice if they included a way to express type checking of the format_spec. That’s going to be an unnecessary source of runtime errors.
- 12_throw_away 2y ago
  self-deleted
  - layer8 2y ago
    https://news.ycombinator.com/item?id=36674260 https://news.ycombinator.com/item?id=36674260
jtwaleson 2y ago
For people adding insightful critique on the PEP on HN (I saw some on this thread already), please ensure your opinion is represented in the PEP thread itself too.
- 0cf8612b2e1e 2y ago
  Given the names attached to the proposal, is this PEP actually up for debate? Most of the critiques I am reading (with which I fully agree) is that this is more complexity in the language without sufficient payoff. There are now how many stupid or fancy ways to construct a Python string with a variable?
- akklrgaG 2y ago
  Why would anyone do this? You risk being banned and defamed if you lack deference or exceed your allotment of five messages per thread. Moreover, Python has resume-driven development and people with a financial or reputational interest will get their new toys in no matter what. You would just contribute to the appearance of democracy.
deleted 2y ago
[deleted]
mrweasel 2y ago
My issue with this that is will eventually sneak into libraries and the users of that library would be expected to use these tag strings all over the place to utilize the library. This prevents people from having a uniform coding style and make code harder to read. The concern isn't having features that will make it easier to write DSLs, my problem is that people will misuse it in regular Python projects. I know that one of the authors are Guido, but I'm not buying the motivation. Jinja2 and Django template are pretty much just using Python, it's not really much of an issue, and I don't believe that business logic should exist in your templates anyway. As for the SQL argument, it will still be possible for people to mess it up, even with Tag Strings, unless you completely remove all legacy code. The issue here isn't that the existing facilities aren't good enough, it's that many developers aren't aware of the concepts, like prepared statements. If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings? Obviously Guido is a better developer than me any day, so I might be completely wrong, but this doesn't feel right. I've seen tools developed to avoid just doing the proper training, and the result is always worse.
- masklinn 2y ago
  > If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings? Because you've deprecated the "bare string" interface so they can't use that anymore, or it's hidden deep into the utility modules.
  - mrweasel 2y ago
    But you could do that already, could you not? Django does. Its just not really SQL anymore then. Someone else also pointed out that you could just do this with functions. It seems like a very fancy way of avoiding using (). I don't know, maybe show me how that would solve the issue of unsafe SQL and I'd be more easily convinced.
    - hombre_fatal 2y ago
      Because it turns into a parameterized SQL query. This already exists in the Javascript ecosystem: sql`select * from users where id = ${id}` Turns into: { query: 'select * from users where id = $1', values: [id] } So if you tried an injection like this: sql`select * from users ${'where id = 3'}` It turns into an invalid statement since "where id = 3" cannot exist as a parameterized value for the same reason this doesn't work: { query: 'select * from users $1', values: ['where id = 3'] } Where you go from here is to offer a query(statement) function that requires the use of the tag string so that you can't accidentally pass in a normal string-interpolated string. Examples: - slonik: https://github.com/gajus/slonik?tab=readme-ov-file#protecting-against-unsafe-value-interpolation https://github.com/gajus/slonik?tab=readme-ov-file#protectin... - postgres.js: https://github.com/porsager/postgres?tab=readme-ov-file#queries https://github.com/porsager/postgres?tab=readme-ov-file#quer...
ianbicking 2y ago
I LOVE tagged templates in JavaScript. But in Python I could also imagine YET ANOTHER constant prefix, like t"", that returns a "template" object of some sort, and then you could do html(t"") or whatever. That is, it would be just like f"" but return an object and not a string so you could get at the underlying values. Like in JavaScript the ability to see the original backslash escaping would be nice, and as an improvement over JavaScript the ability to see the expressions as text would also be nice. But the deferred evaluation seems iffy to me. Like I can see all the cool things one might do with it, but it also means you can't understand the evaluation without understanding the tag implementation. Also I don't think deferred evaluation is enough to make this an opportunity for a "full" DSL. Something like a for loop requires introducing new variables local to the template/language, and that's really beyond what this should be, or what deferred evaluation would allow.
- pauleveritt 2y ago
  Cool to see you jump in, Ian. I don't particularly mind the prefix thing. It came up in the PEP discussion, as did choice of backticks to indicate this is different. But JS template literals -> tagged template literals shows, you can get from A to B without a fundamental change. I'm very interested though in the deferred part. I agree that there is complexity. I weigh that, though, against the complexity of existing Python HTML templating, where finding out what just happened is...harder. I think we can get a TSX-level of DX out of this. And maybe a Lit level of composition. Agree that it is non-zero complexity.
  - ianbicking 2y ago
    Hey Paul! I think JSX is an example of the somewhat crude but practical use of simple execution patterns. For instance if you have a loop you do: return <ol> {items.map((item, i) => <li key={i}>{item}</li>)} </ol>; Which isn't really templating at all, but just the ability to use inline expressions and easily construct objects does get the job done. Or in a SQL builder with JavaScript tagged templates, I do: exec(sql` SELECT * FROM a_table WHERE category = ${category} ${subcategory ? sql`AND subcategory=${subcategory}` : sql``} `) That is, I nest tagged templates to handle different logic conditions and loops. If there's deferred execution, it's done with ?: and .map() – though these very long expressions don't work nearly as well in Python. (List comprehension is in some ways better than .map()/.filter(), but not for very large expressions like in a JSX template.)
spankalee 2y ago
I tried skimming the PEP while I could, but it seems like this might be missing a couple of the features that make JS tagged template literals work so well: - tags get a strings array that's referentially stable across invocations. This can function as a cache key to cache strings parsing work. - tags can return any kind of value, not just a string. Often you need to give structured data to another cooperating API. Deferred evaluation of expressions is very cool, and would be really useful for reactive use-cases, assuming they can be evaluated multiple times.
- svieira 2y ago
  The string array is not referentially stable across invocations and cannot be because there is a single argument array containing both the "static" bits and the "dynamic" bits. So you can't use it the way that JS' `static_strings` argument can be used as a key in a `WeakMap`. Tags can return any kind of value, so there's that. Deferred evaluations can be evaluated multiple times and is in fact one of the biggest foot-guns in this API (in my opinion).
  - spankalee 2y ago
    I wonder if each of the string args is stable? Then you could just use the first as the key. The JS API where the strings are separate might seem awkward at first, but it ends up being a really elegant design. Deferred evaluation is really powerful and I wish JS had it. It's one of the reasons why Solid has a custom JSX compiler and doesn't use tagged literals... to get the-evaluation you need to user to pass in closures, which is cumbersome.
    - Nullabillity 2y ago
      Python does string interning, so that seems.. dangerous.
      - spankalee 2y ago
        Most managed languages do string interning. But the object passed as the string value doesn't seem to be just a string. It's wrapped in another object, which could have callsite identity.
treyd 2y ago
I can't help but believe that this is introducing more spooky action at a distance and is bound to be abused. Is it really more usable this way? Do they have any concrete and practical examples where this improves readability?
Hamuko 2y ago
I hate the idea of reusing the existing string/bytes prefixes for something that is completely different. How is someone expected to know that br"" is inherent Python syntax and my"" is essentially an user-defined function? And the only way to ever add a new prefix into the language (like f"" was added quite recently) is to wait until Python 4, at which point we'll need 3to4 to automatically rename all of your old tag strings that are now conflicting and people will bitch about how badly major Python upgrades suck.
zoogeny 2y ago
I've seen this feature used responsibly and to good effect in a few TypeScript projects so I understand why it would be desirable in Python.
- pauleveritt 2y ago
  Hi, I'm one of the PEP authors. This is precisely my primary interest. TSX makes for a good HTML templating experience. Python templating approaches usually create a parallel universe of scope, syntax, imports, linting, type checking, IDE assistance, etc. where little of Python tooling exists.
tofflos 2y ago
I would have loved to see Java introduce something similar to the IntelliJ @Language-annotation in the standard library but maybe they'll figure out the sweet spot in a future String Templating JEP. @Language("application/sql") String query = "SELECT 1"; @Language("application/graphql+json") String query = """ query HeroNameAndFriends { hero { name friends { name } } } """;
- neonsunset 2y ago
  This is exactly how raw string literals together with StringSyntaxAttribute work in C#. It is very useful in e.g Regex syntax highlighting.
DataDive 2y ago
Excellent idea, I don't get the criticism, If a syntax such as f"{variable}" is already a feature - and turned out to be a popular one - why shouldn't we be able to add our own custom "f"s? Because that is what this is about. It might make generating output even simpler. I applaud the idea and am pleased to see that Python keeps innovating!
- RhysU 2y ago
  f("Consider...") greet("Hello {name}") What was wrong with the standard way to write function application? Python is sufficiently dynamic that an implementation of greet(...) can look up one level to resolve {name}, right? That's why Python will forever run like a dog. Might as take advantage of it to build such capabilities in user space. This crap is going to end up inside f-strings inside tag-strings inside f-strings inside... We have a language. Don't extend it to express what it's perfectly capable of expressing already.
  - DataDive 2y ago
    Your reply appears to indicate that you do not properly understand the new proposed feature. It is most certainly not just about dropping two parentheses. > Tag strings extract more than just a callable from the Interpolation. They also provide Python string formatting info, as well as the original text. The feature is akin to moving print from a keyword to a function. That change also made a huge difference in that it unified the output stream and avoided having undefined objects like a "print" keyword. Here, you can think of the feature as moving an "f" string from a hardcoded, predetermined definition to a generalizable and programmable behavior. If "f" strings have become so popular so quickly it means they addressed a pressing need. It is logical to assume that a programmable version of an "f" string would be even more useful.
    - pauleveritt 2y ago
      (PEP co-author here.) You've described it well. As the "How to teach it section" emphasizes, we'd like consumers of tag functions to just think of it as an f-string with other stuff that happens before evaluation. From their POV, inside the quotes, what you know about f-strings, you know here as well.
kbd 2y ago
At least in the spirit of "the language shouldn't be able to define things the user can't" (see: Java string concatenation) this seems like a good change.
samatman 2y ago
I think this will turn out well. Julia has had this forever as string macros, and it has worked out rather nicely, features like `r"\d+"` for regex, and `raw"strings"` are just string macros. The set of all useful custom literal strings isn't bounded, so a lightweight mechanism to define them and make use of the results is a good thing. Another kitchen sink to add to Python's world-class kitchen sink collection.
orbisvicis 2y ago
This seems similar to my protostrings library [1] which I wrote years ago and mostly forgot about till now. 1. https://protostrings.readthedocs.io/en/latest/index.xhtml https://protostrings.readthedocs.io/en/latest/index.xhtml iirc I wanted to encode state in a recursive-descent parser without additional complexity to the parser. Similar in purpose not design; protostrings provides lazy and context-sensitive strings from the bottom up rather than this top-down template-style proposal, which I feel addresses most of the concerns here.
cr125rider 2y ago
This seems very unpythonic in the way that it breaks the one best way to do things adage. It’s syntax sugar for a function call. Just keep it a function call if needed.
pansa2 2y ago
Off-topic, but when did Python become so... verbose? From the PEP: def greet(*args: Decoded | Interpolation) -> str: result = [] for arg in args: match arg: case Decoded() as decoded: result.append(decoded) case Interpolation() as interpolation: value = interpolation.getvalue() result.append(value.upper()) return f"{''.join(result)}!" Isn't that just this? def greet(*args): def convert(arg): return arg.getvalue().upper() if hasattr(arg, getvalue) else arg return ''.join(convert(arg) for arg in args) + '!'
- orbisvicis 2y ago
  Guido has been pretty vocal about his preference for loops over comprehensions, so it's just a matter of personal preference. But they're also trying to use the newer pattern matching rather than duck-typing, and comprehensions don't support pattern matching (isinstance doesn't count). The pattern matching stuff is neat but seems pretty half-baked. Only available to imperative code and restricted in what can be matched. I wish they'd finish fleshing it out because right now it feels tacky. Actually my number one wish forever has been that functions would become actual first-class objects, able to be defined and manipulated during runtime at the syntax level. Would have been great for so much of my code, and the original decorator module. Things are better with the new inspect module but it's still ridiculously clunky.
Groxx 2y ago
... is this any different than a function like this: greet("hello {world}") which walks the call stack to find the variables, and uses them as locals / arguments? If so: why not just do that? I would expect the performance to be kinda terrible compared to a language-intrinsic, but this hardly seems like a thing worth truly optimizing. And if it's too costly at runtime, someone can implement a parse-time AST rewriter, like many Python things already do. Heck, that example's `assert` is probably using pytest (because everyone uses pytest) and it's doing exactly this already, and it isn't a language feature, it's just a normal library using normal Python features.
- jimbaker 2y ago
  Walking the call stack implies using dynamic scope, which has hard edges, vs lexical scope. See my answer earlier in this thread https://news.ycombinator.com/item?id=41272285 https://news.ycombinator.com/item?id=41272285 It's been nearly 23 years, but Python 2.2 fixed this issue (https://docs.python.org/3/whatsnew/2.2.html#pep-227-nested-scopes https://docs.python.org/3/whatsnew/2.2.html#pep-227-nested-s...), and it's also why JavaScript added let (and const). f-string support also uses lexical scope, and it's an important part of its success.
Spivak 2y ago
The much bigger feature here is buried under the DSL stuff. Python is effectively implementing a method of lazy evaluation of function arguments! I thought I would never see the day! It's crazy that if this PEP is accepted, functions in Python will actually be a special case of f-strings. I hope they eventually grant this power to regular functions because otherwise I know folks will end up myfunc"{arg1},{arg2}" to get that feature.
nope96 2y ago
Does this mean I can write a print function and be able to print"hello world" without parentheses again, like in python 2.x ?
agumonkey 2y ago
Seems like many languages are allowing compile time interception (zig, es, now python)

96 points by lumpa 2y ago

Yikes. Don't get me wrong, I totally understand the reasoning why this would be useful (though I violently disagree with the idea of deferring the evaluation of the contained expressions), but it's also so very kitchensinky and adds so little over just calling a function (which doesn't require a 20-page explainer, as everyone already knows how function calls work). It also promotes using what looks like string interpolation (and what might be string interpolation, you can't tell at the "call site") for things which we know string interpolation is the wrong tool. The API also seems really, I dunno, weird to me. The string is split around interpolations and verbatim portions result in one argument, which is "string-like", while interpolations become four-tuple-like (one of which is a lambda, which you call to perform the deferred interpolation). This seems really awkward to me for building stuff like the suggested use cases of XML/HTML or SQL templating. Also the scoping rules of this are a special case which doesn't appear in regular Python code so far: "The use of annotation scope means it’s not possible to fully desugar interpolations into Python code. Instead it’s as if one is writing interpolation_lambda: tag, not lambda: tag, where a hypothetical interpolation_lambda keyword variant uses annotation scope instead of the standard function scope." -- i.e. it's "as if you wrapped all interpolation expressions in a lambda: <expr>, except it uses different scoping rules".

masklinn 2y ago
> This seems really awkward to me for building stuff like the suggested use cases of XML/HTML or SQL templating. Compared to what? At the end of the day you're still doing string formatting, if you want parsing, then you'd feed the item into a parser, which this doesn't preclude. The interface sounds a lot better than JS's anyway, as that completely separates the literal strings and the interpolations so you have to re-intersperse them which is muggy. > interpolations become four-tuple-like They become an Interpolation object, which can be unpacked if you find that more convenient, but you can access the members if you prefer: - 0 is getvalue is the callable to retrieve the evaluated expression - 1 is expr is the raw text form of the expression - 2 is conv is the !conversion tag (s, r, or a) - 3 is format_spec
- jimbaker 2y ago
  The intent here is to support the following approach for tag function authors: 1. Parse to an AST, generally using an off-the-shelf parser. In practice, it's possible to rewrite interpolations with a placeholder suitable for a given language, eg x$Nx for HTML. Of course if that doesn't actually work, you might have to write/modify an existing parser. Hopefully we can cleverly avoid this extra work. 2. Walk/compile the AST, filling interpolations, but taking in account the context. This can for example take the form of building appropriate query strings that avoid Bobby Tables SQL injection, whether by mapping to SQL placeholders or with appropriate quoting (such as for a column or table name). 3. Memoize these steps, much as we see with builtin DSLs in Python, like the re module; see https://github.com/python/cpython/blob/3.12/Lib/re/__init__.py#L331 https://github.com/python/cpython/blob/3.12/Lib/re/__init__.... We do plan to make this easier/faster by supporting getting the original source of the template string (Template.source), vs the *args approach we show in the PEP at the start of this discussion (this will become Template.args instead; Template here is the proposed protocol of the object passed in). Related is my post here: https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408/108 https://discuss.python.org/t/pep-750-tag-strings-for-writing...

dmart 2y ago

I have to admit that at first glance I don’t like this. These seem to be essentially normal str -> Any functions, with some naming limitations due to the existing string prefixes special-cased in the language. I don’t feel like adding this additional complexity is worth being able to save two parentheses per function call.

Epa095 2y ago
This was my first thought as well. But an important difference is that the arguments are not eagerly evaluated, but they are passed as lambdas which can be evaluated if desired. This means that it can be used for example in log messages (if you don't want to evaluate the string at the wrong log levels). But is it worth it for that? Idk.
- masklinn 2y ago
  Even if eager evaluation it's already a very compelling way of managing basically every lightweight "templating" for safety: e.g. embedded dynamic HTML or SQL. `markupsafe` is great, but it's way too easy to perform formatting before calling it, especially with f-strings. That f-strings were "static" was by far my biggest criticism of it, given how useful I find JS's template strings. And this proposal seems like a straight up better version of template strings: - the static strings and interpolations are not split and don't have to be awkwardly re-interpsersed which I've never found 100% trouble and 0% utility - the lazy evaluation means it can be used for things like logging (which really want lazy evaluation), or meta-programmation (because you can introspect the callables, and you get the expression text)
  - codethief 2y ago
    > - the lazy evaluation means it can be used for things like logging (which really want lazy evaluation) Could you elaborate? I would find it rather surprising if my log messages don't contain the data at the very moment I invoke the logger.
    - 2y ago

ziml77 2y ago

It seems the purpose of this proposal is to have a way to essentially have custom string interpolation. I don't think that's necessarily a bad idea on its own, but this syntax feels out of place to me. Instead, why not add a single new string prefix, like "l" for "lazy"? So, f"hello {name}" would immediately format it while l"hello {name}" would produce an object which contains a template and the captured variables. Then their example would be called like: greet(l"hello {name}").

TwentyPosts 2y ago

This like a bad idea on the first glance? Maybe I don't get the whole pitch here? It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call. The laziness might be interesting, but I feel like "lazy strings" might be all that's needed here. Laziness and validation (or custom string formatting logic) are separate concerns and should be separated.

masklinn 2y ago
> It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call. That could also be leveraged at f-strings themselves. > Laziness and validation (or custom string formatting logic) are separate concerns and should be separated. In which case the one to move out is the laziness not the customised interpolation. Because the latter is the one that's necessary for safer dynamic SQL or HTML or whatever.

behnamoh 2y ago

I want this in Python: https://codecodeship.com/blog/2024-06-03-curl_req https://codecodeship.com/blog/2024-06-03-curl_req From the article: """ ~CURL[curl https://catfact.ninja/fact] |> Req.request!() This is actual code; you can run this. It will convert the curl command into a Req request and you will get a response back. This is really great, because we have been able to increase the expressiveness of the language. """

zelphirkalt 2y ago
And somewhere there is the logic implemented to know what ~CURL is and what needs to happen with the part in square brackets, how it needs to be translated into a request call. And if that is merely an arguments equals arguments thingy, then it would be kind of useless, since there are usually more things one specifies when making a request from inside Python. Things like headers or whether or not to verify TLS certs, and that is not a 1 to 1 mapping between curl arguments and a requests call. So I remain doubtful, as long as no way is shown to me, how the user of the language can define this syntactic abstraction themselves, which is unlikely to happen or exist in Python.

Too 2y ago

Looks good. Would have been nice if they included a way to express type checking of the format_spec. That’s going to be an unnecessary source of runtime errors.

12_throw_away 2y ago
self-deleted
- layer8 2y ago
  https://news.ycombinator.com/item?id=36674260 https://news.ycombinator.com/item?id=36674260

jtwaleson 2y ago

For people adding insightful critique on the PEP on HN (I saw some on this thread already), please ensure your opinion is represented in the PEP thread itself too.

0cf8612b2e1e 2y ago
Given the names attached to the proposal, is this PEP actually up for debate? Most of the critiques I am reading (with which I fully agree) is that this is more complexity in the language without sufficient payoff. There are now how many stupid or fancy ways to construct a Python string with a variable?
akklrgaG 2y ago
Why would anyone do this? You risk being banned and defamed if you lack deference or exceed your allotment of five messages per thread. Moreover, Python has resume-driven development and people with a financial or reputational interest will get their new toys in no matter what. You would just contribute to the appearance of democracy.

deleted 2y ago

[deleted]

mrweasel 2y ago

My issue with this that is will eventually sneak into libraries and the users of that library would be expected to use these tag strings all over the place to utilize the library. This prevents people from having a uniform coding style and make code harder to read. The concern isn't having features that will make it easier to write DSLs, my problem is that people will misuse it in regular Python projects. I know that one of the authors are Guido, but I'm not buying the motivation. Jinja2 and Django template are pretty much just using Python, it's not really much of an issue, and I don't believe that business logic should exist in your templates anyway. As for the SQL argument, it will still be possible for people to mess it up, even with Tag Strings, unless you completely remove all legacy code. The issue here isn't that the existing facilities aren't good enough, it's that many developers aren't aware of the concepts, like prepared statements. If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings? Obviously Guido is a better developer than me any day, so I might be completely wrong, but this doesn't feel right. I've seen tools developed to avoid just doing the proper training, and the result is always worse.

masklinn 2y ago
> If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings? Because you've deprecated the "bare string" interface so they can't use that anymore, or it's hidden deep into the utility modules.
- mrweasel 2y ago
  But you could do that already, could you not? Django does. Its just not really SQL anymore then. Someone else also pointed out that you could just do this with functions. It seems like a very fancy way of avoiding using (). I don't know, maybe show me how that would solve the issue of unsafe SQL and I'd be more easily convinced.
  - hombre_fatal 2y ago
    Because it turns into a parameterized SQL query. This already exists in the Javascript ecosystem: sql`select * from users where id = ${id}` Turns into: { query: 'select * from users where id = $1', values: [id] } So if you tried an injection like this: sql`select * from users ${'where id = 3'}` It turns into an invalid statement since "where id = 3" cannot exist as a parameterized value for the same reason this doesn't work: { query: 'select * from users $1', values: ['where id = 3'] } Where you go from here is to offer a query(statement) function that requires the use of the tag string so that you can't accidentally pass in a normal string-interpolated string. Examples: - slonik: https://github.com/gajus/slonik?tab=readme-ov-file#protecting-against-unsafe-value-interpolation https://github.com/gajus/slonik?tab=readme-ov-file#protectin... - postgres.js: https://github.com/porsager/postgres?tab=readme-ov-file#queries https://github.com/porsager/postgres?tab=readme-ov-file#quer...

ianbicking 2y ago

I LOVE tagged templates in JavaScript. But in Python I could also imagine YET ANOTHER constant prefix, like t"", that returns a "template" object of some sort, and then you could do html(t"") or whatever. That is, it would be just like f"" but return an object and not a string so you could get at the underlying values. Like in JavaScript the ability to see the original backslash escaping would be nice, and as an improvement over JavaScript the ability to see the expressions as text would also be nice. But the deferred evaluation seems iffy to me. Like I can see all the cool things one might do with it, but it also means you can't understand the evaluation without understanding the tag implementation. Also I don't think deferred evaluation is enough to make this an opportunity for a "full" DSL. Something like a for loop requires introducing new variables local to the template/language, and that's really beyond what this should be, or what deferred evaluation would allow.

pauleveritt 2y ago
Cool to see you jump in, Ian. I don't particularly mind the prefix thing. It came up in the PEP discussion, as did choice of backticks to indicate this is different. But JS template literals -> tagged template literals shows, you can get from A to B without a fundamental change. I'm very interested though in the deferred part. I agree that there is complexity. I weigh that, though, against the complexity of existing Python HTML templating, where finding out what just happened is...harder. I think we can get a TSX-level of DX out of this. And maybe a Lit level of composition. Agree that it is non-zero complexity.
- ianbicking 2y ago
  Hey Paul! I think JSX is an example of the somewhat crude but practical use of simple execution patterns. For instance if you have a loop you do: return <ol> {items.map((item, i) => <li key={i}>{item}</li>)} </ol>; Which isn't really templating at all, but just the ability to use inline expressions and easily construct objects does get the job done. Or in a SQL builder with JavaScript tagged templates, I do: exec(sql` SELECT * FROM a_table WHERE category = ${category} ${subcategory ? sql`AND subcategory=${subcategory}` : sql``} `) That is, I nest tagged templates to handle different logic conditions and loops. If there's deferred execution, it's done with ?: and .map() – though these very long expressions don't work nearly as well in Python. (List comprehension is in some ways better than .map()/.filter(), but not for very large expressions like in a JSX template.)

spankalee 2y ago

I tried skimming the PEP while I could, but it seems like this might be missing a couple of the features that make JS tagged template literals work so well: - tags get a strings array that's referentially stable across invocations. This can function as a cache key to cache strings parsing work. - tags can return any kind of value, not just a string. Often you need to give structured data to another cooperating API. Deferred evaluation of expressions is very cool, and would be really useful for reactive use-cases, assuming they can be evaluated multiple times.

svieira 2y ago
The string array is not referentially stable across invocations and cannot be because there is a single argument array containing both the "static" bits and the "dynamic" bits. So you can't use it the way that JS' `static_strings` argument can be used as a key in a `WeakMap`. Tags can return any kind of value, so there's that. Deferred evaluations can be evaluated multiple times and is in fact one of the biggest foot-guns in this API (in my opinion).
- spankalee 2y ago
  I wonder if each of the string args is stable? Then you could just use the first as the key. The JS API where the strings are separate might seem awkward at first, but it ends up being a really elegant design. Deferred evaluation is really powerful and I wish JS had it. It's one of the reasons why Solid has a custom JSX compiler and doesn't use tagged literals... to get the-evaluation you need to user to pass in closures, which is cumbersome.
  - Nullabillity 2y ago
    Python does string interning, so that seems.. dangerous.
    - spankalee 2y ago
      Most managed languages do string interning. But the object passed as the string value doesn't seem to be just a string. It's wrapped in another object, which could have callsite identity.

treyd 2y ago

I can't help but believe that this is introducing more spooky action at a distance and is bound to be abused. Is it really more usable this way? Do they have any concrete and practical examples where this improves readability?

Hamuko 2y ago

I hate the idea of reusing the existing string/bytes prefixes for something that is completely different. How is someone expected to know that br"" is inherent Python syntax and my"" is essentially an user-defined function? And the only way to ever add a new prefix into the language (like f"" was added quite recently) is to wait until Python 4, at which point we'll need 3to4 to automatically rename all of your old tag strings that are now conflicting and people will bitch about how badly major Python upgrades suck.

zoogeny 2y ago

I've seen this feature used responsibly and to good effect in a few TypeScript projects so I understand why it would be desirable in Python.

pauleveritt 2y ago
Hi, I'm one of the PEP authors. This is precisely my primary interest. TSX makes for a good HTML templating experience. Python templating approaches usually create a parallel universe of scope, syntax, imports, linting, type checking, IDE assistance, etc. where little of Python tooling exists.

tofflos 2y ago

I would have loved to see Java introduce something similar to the IntelliJ @Language-annotation in the standard library but maybe they'll figure out the sweet spot in a future String Templating JEP. @Language("application/sql") String query = "SELECT 1"; @Language("application/graphql+json") String query = """ query HeroNameAndFriends { hero { name friends { name } } } """;

neonsunset 2y ago
This is exactly how raw string literals together with StringSyntaxAttribute work in C#. It is very useful in e.g Regex syntax highlighting.

DataDive 2y ago

Excellent idea, I don't get the criticism, If a syntax such as f"{variable}" is already a feature - and turned out to be a popular one - why shouldn't we be able to add our own custom "f"s? Because that is what this is about. It might make generating output even simpler. I applaud the idea and am pleased to see that Python keeps innovating!

RhysU 2y ago
f("Consider...") greet("Hello {name}") What was wrong with the standard way to write function application? Python is sufficiently dynamic that an implementation of greet(...) can look up one level to resolve {name}, right? That's why Python will forever run like a dog. Might as take advantage of it to build such capabilities in user space. This crap is going to end up inside f-strings inside tag-strings inside f-strings inside... We have a language. Don't extend it to express what it's perfectly capable of expressing already.
- DataDive 2y ago
  Your reply appears to indicate that you do not properly understand the new proposed feature. It is most certainly not just about dropping two parentheses. > Tag strings extract more than just a callable from the Interpolation. They also provide Python string formatting info, as well as the original text. The feature is akin to moving print from a keyword to a function. That change also made a huge difference in that it unified the output stream and avoided having undefined objects like a "print" keyword. Here, you can think of the feature as moving an "f" string from a hardcoded, predetermined definition to a generalizable and programmable behavior. If "f" strings have become so popular so quickly it means they addressed a pressing need. It is logical to assume that a programmable version of an "f" string would be even more useful.
  - pauleveritt 2y ago
    (PEP co-author here.) You've described it well. As the "How to teach it section" emphasizes, we'd like consumers of tag functions to just think of it as an f-string with other stuff that happens before evaluation. From their POV, inside the quotes, what you know about f-strings, you know here as well.

kbd 2y ago

At least in the spirit of "the language shouldn't be able to define things the user can't" (see: Java string concatenation) this seems like a good change.

samatman 2y ago

I think this will turn out well. Julia has had this forever as string macros, and it has worked out rather nicely, features like `r"\d+"` for regex, and `raw"strings"` are just string macros. The set of all useful custom literal strings isn't bounded, so a lightweight mechanism to define them and make use of the results is a good thing. Another kitchen sink to add to Python's world-class kitchen sink collection.

orbisvicis 2y ago

This seems similar to my protostrings library [1] which I wrote years ago and mostly forgot about till now. 1. https://protostrings.readthedocs.io/en/latest/index.xhtml https://protostrings.readthedocs.io/en/latest/index.xhtml iirc I wanted to encode state in a recursive-descent parser without additional complexity to the parser. Similar in purpose not design; protostrings provides lazy and context-sensitive strings from the bottom up rather than this top-down template-style proposal, which I feel addresses most of the concerns here.

cr125rider 2y ago

This seems very unpythonic in the way that it breaks the one best way to do things adage. It’s syntax sugar for a function call. Just keep it a function call if needed.

pansa2 2y ago

Off-topic, but when did Python become so... verbose? From the PEP: def greet(*args: Decoded | Interpolation) -> str: result = [] for arg in args: match arg: case Decoded() as decoded: result.append(decoded) case Interpolation() as interpolation: value = interpolation.getvalue() result.append(value.upper()) return f"{''.join(result)}!" Isn't that just this? def greet(*args): def convert(arg): return arg.getvalue().upper() if hasattr(arg, getvalue) else arg return ''.join(convert(arg) for arg in args) + '!'

orbisvicis 2y ago
Guido has been pretty vocal about his preference for loops over comprehensions, so it's just a matter of personal preference. But they're also trying to use the newer pattern matching rather than duck-typing, and comprehensions don't support pattern matching (isinstance doesn't count). The pattern matching stuff is neat but seems pretty half-baked. Only available to imperative code and restricted in what can be matched. I wish they'd finish fleshing it out because right now it feels tacky. Actually my number one wish forever has been that functions would become actual first-class objects, able to be defined and manipulated during runtime at the syntax level. Would have been great for so much of my code, and the original decorator module. Things are better with the new inspect module but it's still ridiculously clunky.

Groxx 2y ago

... is this any different than a function like this: greet("hello {world}") which walks the call stack to find the variables, and uses them as locals / arguments? If so: why not just do that? I would expect the performance to be kinda terrible compared to a language-intrinsic, but this hardly seems like a thing worth truly optimizing. And if it's too costly at runtime, someone can implement a parse-time AST rewriter, like many Python things already do. Heck, that example's `assert` is probably using pytest (because everyone uses pytest) and it's doing exactly this already, and it isn't a language feature, it's just a normal library using normal Python features.

jimbaker 2y ago
Walking the call stack implies using dynamic scope, which has hard edges, vs lexical scope. See my answer earlier in this thread https://news.ycombinator.com/item?id=41272285 https://news.ycombinator.com/item?id=41272285 It's been nearly 23 years, but Python 2.2 fixed this issue (https://docs.python.org/3/whatsnew/2.2.html#pep-227-nested-scopes https://docs.python.org/3/whatsnew/2.2.html#pep-227-nested-s...), and it's also why JavaScript added let (and const). f-string support also uses lexical scope, and it's an important part of its success.

Spivak 2y ago

The much bigger feature here is buried under the DSL stuff. Python is effectively implementing a method of lazy evaluation of function arguments! I thought I would never see the day! It's crazy that if this PEP is accepted, functions in Python will actually be a special case of f-strings. I hope they eventually grant this power to regular functions because otherwise I know folks will end up myfunc"{arg1},{arg2}" to get that feature.

nope96 2y ago

Does this mean I can write a print function and be able to print"hello world" without parentheses again, like in python 2.x ?

agumonkey 2y ago

Seems like many languages are allowing compile time interception (zig, es, now python)