6 ms·

Nick Humrich here, the author who helped rewrite PEP 501 to introduce t-strings, which was the foundation for this PEP. I am not an author on this accepted PEP,

Nick Humrich here, the author who helped rewrite PEP 501 to introduce t-strings, which was the foundation for this PEP. I am not an author on this accepted PEP, but I know this PEP and story pretty well. Let me know if you have any questions. I am super excited this is finally accepted. I started working on PEP 501 4 years ago.

_cs2017_ 2y ago
Thank you! Curious what options for deferred evalution were considered and rejected? IMHO, the main benefit of deferred evaluation isn't in the saving of a bit of code to define a deferred evaluation class, but in standardazing the API so that anyone can read the code without having to learn what it means in each project. Also: were prompt templates for LLM prompt chaining a use case that influenced the design in any way (examples being LangChain and dozens of other libraries with similar functionlity)?
- nhumrich 2y ago
  One solution that existed for a while was using the `!` operator for deferred. `t!'my defered {str}'` The main reason for non having deferred evaluation was that it over-complicated the feature quite a bit and introduces a rune. Deferred evaluation also has the potential to dramatically increase complexity for beginners in the language, as it can be confusing to follow if you dont know what is going on. Which means "deferred by default" wasnt going to be accepted. As for LLM's, it was not the main consideration, as the PEP process here started before LLM's were popular.
  - _cs2017_ 2y ago
    Ah interesting, so the complexity wasn't in the API design or implementation, but only in the additional rune? Is that really such a big cost?
- davepeck 2y ago
  > were prompt templates for LLM prompt chaining a use case that influenced the design in any way Maybe not directly, but the Python community is full of LLM users and so I think there's a general awareness of the issues.
  - andy99 2y ago
    Is there an example of how these could be used in LLM prompting?
Waterluvian 2y ago
I often read concerns that complexity keeps being added to the language with yet another flavour of string or whatnot. Given that those who author and deliberate on PEPs are, kind of by definition, experts who spend a lot of time with the language, they might struggle to grok the Python experience from the perspective of a novice or beginner. How does the PEP process guard against this bias?
- davepeck 2y ago
  You might find the Python discussion forums ([0] and [1]) interesting; conversation that guides the evolution of PEPs happens there. As Nick mentioned, PEP 750 had a long and winding road to its final acceptance; as the process wore on, and the complexities of the earliest cuts of the PEPs were reconsidered, the two converged. [0] The very first announcement: https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408 https://discuss.python.org/t/pep-750-tag-strings-for-writing... [1] Much later in the PEP process: https://discuss.python.org/t/pep750-template-strings-new-updates/71594 https://discuss.python.org/t/pep750-template-strings-new-upd...
- nhumrich 2y ago
  All discussion on PEP's happens in public forums where anyone can opine on things before they are accepted. I agree that the experts are more likely to participate in this exchange. And while this is wish-washy, I feel like the process is really intended to benefit the experts more than the novices anyways. There have been processes put into place in recent years to try to curb the difficulty of things. One of those is that all new PEPs have to include a "how can you teach this to beginers" section, as seen here on this pep: https://peps.python.org/pep-0750/#how-to-teach-this https://peps.python.org/pep-0750/#how-to-teach-this
  - Waterluvian 2y ago
    I think "how can you teach this to beginners?" is a fantastic, low-hanging fruit option for encouraging the wizards to think about that very important type of user. Other than a more broad "how is the language as a whole faring?" test, which might be done through surveys or other product-style research, I think this is just plainly a hard problem to approach, just by the nature that it's largely about user experience.
patrec 2y ago
My memory is that ES6's template strings preceded f-strings. If that is correct, do you happen to know why python was saddled with f-strings, which seem like an obviously inferior design, in the first place? We are now at five largely redundant string interpolation systems (%, .format, string.Template, f-string, t-string).
- nhumrich 2y ago
  PEP 501 when originally written (not by me) was intended to be the competing standard against f-strings, and to have been more inline with ES6's template strings. There was debate between the more simple f-string PEP (PEP 498) and PEP 501. Ultimately, it was decided to go with f-strings as a less confusing, more approachable version (and also easier to implement) and to "defer" PEP 501 to "see what happens". Since then, the python internal have also changed, allowing t-strings to be even easier to implement (See PEP 701). We have seen what happens, and now its introduced. f-strings and t-strings are not competing systems. They are different. Similar to ES6 templates and namedTaggedTemplates, they are used for different things while API feels similar intentionally. f-strings are not inferior to t-strings, they are better for most use cases of string templating where what you really want, is just a string.
  - patrec 2y ago
    Thanks! > they are better for most use cases of string templating where what you really want, is just a string. I think use cases where you want to unconditionally bash a string together are rare. I'd bet that in > 80% of cases the "just a string" really is just a terrible representation for what really is either some tree (html, sql, python, ...) structure or at least requires lazy processing (logging, where you only want to pay for the expensive string formatting and generation if you run at the log level or higher that the relevant logging line is meant to operate).
    - ii41 1y ago
      If the "just a string" is html, sql, python, etc., the programmer is making a mistake as string interpolation shouldn't be used for those things in the first place, and I really hope < 80% cases of "just a string" are among those. "just a string" where the string really is just a string is really, really common though. Python is the go-to language for a lot of people (including me) if they just want to bash together a one-time-use script that will never see input from other people, and f-strings are incredibly useful in such cases for generating filenames, debugging messages, etc.
bjourne 2y ago
Does Python really need yet another type of string literal? I feel like while templating is a good addition to the standard library, it's not something that needs syntactic support. t"blah blah" is just an alias for Template("blah blah", context), isn't it?
- nhumrich 2y ago
  yes, it does actually need syntax support. In order for it to work, you need to preserve which parts of the string are static (hard coded) and which parts are dynamic (likely user input). Which you can only do at a syntax level. You could potentially do it by hand, using placeholders, like with `%`, but as we now live in f-string world, we have something better. The syntax highlighting and ergonomics of f-strings are so good, devs prefer it in most cases. The idea is to make the most ergonomic thing, also the safest thing. By decreasing ergonomics, you reduce the adoption of safer symantics.
  - bjourne 2y ago
    That's why I specified the context argument. Something like Template("{name} {address}", dict(name = ..., address = ...)) would be exactly equivalent to t"{name} {address}" assuming those variables are fetched from the local scope.
    - nhumrich 2y ago
      Yes, which is essentially how SQLalchemy works today. You can still put strings in the context though, so for more complex things, it's turtles all the way down. Also, as f-srrings are more ergonomic, people now reach for them, even when they shouldn't
    - thayne 2y ago
      So now you are repeating the name of each interpolated value three times (once in the template string, once for the key in the dict, once for the actual value). Yes, you can do that, but a t-string is much more ergonomic, and IMO, more readable .
frainfreeze 2y ago
Nice work on PEP 501! Probably a silly question, but how comes PEP 292 isn't mentioned anywhere in PEP 750?
- davepeck 2y ago
  My hope is to write some new documentation as 3.14 nears release that explains the (growing) constellation of string formatting mechanisms in Python and describes when they might each be useful. They overlap to some degree, but each has a unique twist that makes them useful in different situations. PEP 292 is going nowhere and is used, for instance, in really powerful libraries like `flufl.i18n`
  - sevensor 2y ago
    Is a PEP 750 Template entirely different from a PEP 292 Template? I’m a bit confused about the relationship.
    - davepeck 2y ago
      Yeah, they’re unrelated. (PEP 750 places its Template and Interpolation classes in the new string.templatelib)
EndsOfnversion 2y ago
[flagged]
leobuskin 2y ago
As I understand, it may help a bit with logging performance, not sure, still trying to understand the template abilities. So, right now, you have two options to log: 1. `logger.debug(f'Processing {x}')` - looks great, but evaluates anyway, even if logging level > `logging.DEBUG`; 2. `logger.debug('Processing %s', x)` - won't evaluate till necessary. What would be the approach with t-strings in this case? Would we get any benefits?
- davepeck 2y ago
  T-strings, like f-strings, are eagerly evaluated -- so in this sense, no, there's no benefit.
  - trashburger 2y ago
    Not quite; the interpolations are not eagerly stringified which is the potentially expensive part. In this sense it's kind of a middle ground between the two approaches.
    - davepeck 2y ago
      Sure, good point; I generally think of evaluation, not stringification, as the likely dominant expense. But maybe it’s sometimes the other way around?
- bcoates 2y ago
  The expression (x) is eagerly evaluated in both cases, cuz that's how Python works. You can defer the format call but Python fundamentally doesn't have an equivalent of lazy/compile time flag argument evaluation and this doesn't change that. For a logger t-strings are mostly just a more pleasant and less bug-prone syntax for #2
smnrchrds 2y ago
Thank you for your work on this topic and for answering questions here. I have a question: is there a way to avoid the security issues with string formatting described here? It seems like all (most?) string formatting options suffer from the same issue. https://lucumr.pocoo.org/2016/12/29/careful-with-str-format/ https://lucumr.pocoo.org/2016/12/29/careful-with-str-format/

_cs2017_ 2y ago

Thank you! Curious what options for deferred evalution were considered and rejected? IMHO, the main benefit of deferred evaluation isn't in the saving of a bit of code to define a deferred evaluation class, but in standardazing the API so that anyone can read the code without having to learn what it means in each project. Also: were prompt templates for LLM prompt chaining a use case that influenced the design in any way (examples being LangChain and dozens of other libraries with similar functionlity)?

nhumrich 2y ago
One solution that existed for a while was using the `!` operator for deferred. `t!'my defered {str}'` The main reason for non having deferred evaluation was that it over-complicated the feature quite a bit and introduces a rune. Deferred evaluation also has the potential to dramatically increase complexity for beginners in the language, as it can be confusing to follow if you dont know what is going on. Which means "deferred by default" wasnt going to be accepted. As for LLM's, it was not the main consideration, as the PEP process here started before LLM's were popular.
- _cs2017_ 2y ago
  Ah interesting, so the complexity wasn't in the API design or implementation, but only in the additional rune? Is that really such a big cost?
davepeck 2y ago
> were prompt templates for LLM prompt chaining a use case that influenced the design in any way Maybe not directly, but the Python community is full of LLM users and so I think there's a general awareness of the issues.
- andy99 2y ago
  Is there an example of how these could be used in LLM prompting?

Waterluvian 2y ago

I often read concerns that complexity keeps being added to the language with yet another flavour of string or whatnot. Given that those who author and deliberate on PEPs are, kind of by definition, experts who spend a lot of time with the language, they might struggle to grok the Python experience from the perspective of a novice or beginner. How does the PEP process guard against this bias?

davepeck 2y ago
You might find the Python discussion forums ([0] and [1]) interesting; conversation that guides the evolution of PEPs happens there. As Nick mentioned, PEP 750 had a long and winding road to its final acceptance; as the process wore on, and the complexities of the earliest cuts of the PEPs were reconsidered, the two converged. [0] The very first announcement: https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408 https://discuss.python.org/t/pep-750-tag-strings-for-writing... [1] Much later in the PEP process: https://discuss.python.org/t/pep750-template-strings-new-updates/71594 https://discuss.python.org/t/pep750-template-strings-new-upd...
nhumrich 2y ago
All discussion on PEP's happens in public forums where anyone can opine on things before they are accepted. I agree that the experts are more likely to participate in this exchange. And while this is wish-washy, I feel like the process is really intended to benefit the experts more than the novices anyways. There have been processes put into place in recent years to try to curb the difficulty of things. One of those is that all new PEPs have to include a "how can you teach this to beginers" section, as seen here on this pep: https://peps.python.org/pep-0750/#how-to-teach-this https://peps.python.org/pep-0750/#how-to-teach-this
- Waterluvian 2y ago
  I think "how can you teach this to beginners?" is a fantastic, low-hanging fruit option for encouraging the wizards to think about that very important type of user. Other than a more broad "how is the language as a whole faring?" test, which might be done through surveys or other product-style research, I think this is just plainly a hard problem to approach, just by the nature that it's largely about user experience.

patrec 2y ago

My memory is that ES6's template strings preceded f-strings. If that is correct, do you happen to know why python was saddled with f-strings, which seem like an obviously inferior design, in the first place? We are now at five largely redundant string interpolation systems (%, .format, string.Template, f-string, t-string).

nhumrich 2y ago
PEP 501 when originally written (not by me) was intended to be the competing standard against f-strings, and to have been more inline with ES6's template strings. There was debate between the more simple f-string PEP (PEP 498) and PEP 501. Ultimately, it was decided to go with f-strings as a less confusing, more approachable version (and also easier to implement) and to "defer" PEP 501 to "see what happens". Since then, the python internal have also changed, allowing t-strings to be even easier to implement (See PEP 701). We have seen what happens, and now its introduced. f-strings and t-strings are not competing systems. They are different. Similar to ES6 templates and namedTaggedTemplates, they are used for different things while API feels similar intentionally. f-strings are not inferior to t-strings, they are better for most use cases of string templating where what you really want, is just a string.
- patrec 2y ago
  Thanks! > they are better for most use cases of string templating where what you really want, is just a string. I think use cases where you want to unconditionally bash a string together are rare. I'd bet that in > 80% of cases the "just a string" really is just a terrible representation for what really is either some tree (html, sql, python, ...) structure or at least requires lazy processing (logging, where you only want to pay for the expensive string formatting and generation if you run at the log level or higher that the relevant logging line is meant to operate).
  - ii41 1y ago
    If the "just a string" is html, sql, python, etc., the programmer is making a mistake as string interpolation shouldn't be used for those things in the first place, and I really hope < 80% cases of "just a string" are among those. "just a string" where the string really is just a string is really, really common though. Python is the go-to language for a lot of people (including me) if they just want to bash together a one-time-use script that will never see input from other people, and f-strings are incredibly useful in such cases for generating filenames, debugging messages, etc.

bjourne 2y ago

Does Python really need yet another type of string literal? I feel like while templating is a good addition to the standard library, it's not something that needs syntactic support. t"blah blah" is just an alias for Template("blah blah", context), isn't it?

nhumrich 2y ago
yes, it does actually need syntax support. In order for it to work, you need to preserve which parts of the string are static (hard coded) and which parts are dynamic (likely user input). Which you can only do at a syntax level. You could potentially do it by hand, using placeholders, like with `%`, but as we now live in f-string world, we have something better. The syntax highlighting and ergonomics of f-strings are so good, devs prefer it in most cases. The idea is to make the most ergonomic thing, also the safest thing. By decreasing ergonomics, you reduce the adoption of safer symantics.
- bjourne 2y ago
  That's why I specified the context argument. Something like Template("{name} {address}", dict(name = ..., address = ...)) would be exactly equivalent to t"{name} {address}" assuming those variables are fetched from the local scope.
  - nhumrich 2y ago
    Yes, which is essentially how SQLalchemy works today. You can still put strings in the context though, so for more complex things, it's turtles all the way down. Also, as f-srrings are more ergonomic, people now reach for them, even when they shouldn't
  - thayne 2y ago
    So now you are repeating the name of each interpolated value three times (once in the template string, once for the key in the dict, once for the actual value). Yes, you can do that, but a t-string is much more ergonomic, and IMO, more readable .

frainfreeze 2y ago

Nice work on PEP 501! Probably a silly question, but how comes PEP 292 isn't mentioned anywhere in PEP 750?

davepeck 2y ago
My hope is to write some new documentation as 3.14 nears release that explains the (growing) constellation of string formatting mechanisms in Python and describes when they might each be useful. They overlap to some degree, but each has a unique twist that makes them useful in different situations. PEP 292 is going nowhere and is used, for instance, in really powerful libraries like `flufl.i18n`
- sevensor 2y ago
  Is a PEP 750 Template entirely different from a PEP 292 Template? I’m a bit confused about the relationship.
  - davepeck 2y ago
    Yeah, they’re unrelated. (PEP 750 places its Template and Interpolation classes in the new string.templatelib)

EndsOfnversion 2y ago

[flagged]

leobuskin 2y ago

As I understand, it may help a bit with logging performance, not sure, still trying to understand the template abilities. So, right now, you have two options to log: 1. `logger.debug(f'Processing {x}')` - looks great, but evaluates anyway, even if logging level > `logging.DEBUG`; 2. `logger.debug('Processing %s', x)` - won't evaluate till necessary. What would be the approach with t-strings in this case? Would we get any benefits?

davepeck 2y ago
T-strings, like f-strings, are eagerly evaluated -- so in this sense, no, there's no benefit.
- trashburger 2y ago
  Not quite; the interpolations are not eagerly stringified which is the potentially expensive part. In this sense it's kind of a middle ground between the two approaches.
  - davepeck 2y ago
    Sure, good point; I generally think of evaluation, not stringification, as the likely dominant expense. But maybe it’s sometimes the other way around?
bcoates 2y ago
The expression (x) is eagerly evaluated in both cases, cuz that's how Python works. You can defer the format call but Python fundamentally doesn't have an equivalent of lazy/compile time flag argument evaluation and this doesn't change that. For a logger t-strings are mostly just a more pleasant and less bug-prone syntax for #2

smnrchrds 2y ago

Thank you for your work on this topic and for answering questions here. I have a question: is there a way to avoid the security issues with string formatting described here? It seems like all (most?) string formatting options suffer from the same issue. https://lucumr.pocoo.org/2016/12/29/careful-with-str-format/ https://lucumr.pocoo.org/2016/12/29/careful-with-str-format/